Skip to content

Commit

Permalink
Merge branch 'main' into update-server-infer-docs
Browse files Browse the repository at this point in the history
  • Loading branch information
mgoin authored Apr 17, 2024
2 parents 9dc5c75 + 1ff44cd commit e6e728c
Show file tree
Hide file tree
Showing 3 changed files with 13 additions and 10 deletions.
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ For documentation edits, include:

## Question or Problem

- Sign up or log in to our [**Neural Magic Community Slack**](https://join.slack.com/t/discuss-neuralmagic/shared_invite/zt-q1a1cnvo-YBoICSIw3L1dmQpjBeDurQ). We are growing the community member by member and happy to see you there. Post all other questions including support or how to contribute. Don’t forget to search through existing discussions to avoid duplication! Thanks!
- Sign up or log in to our [**Neural Magic Community Slack**](https://neuralmagic.com/community/). We are growing the community member by member and happy to see you there. Post all other questions including support or how to contribute. Don’t forget to search through existing discussions to avoid duplication! Thanks!

Post all other questions including support or how to contribute. Don’t forget to search through existing discussions to avoid duplication! Thanks!

Expand Down
12 changes: 6 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,16 +16,16 @@ limitations under the License.


<div style="display: flex; flex-direction: column; align-items: center;">
<h1>
<img alt="tool icon" src="https://raw.githubusercontent.com/neuralmagic/deepsparse/main/docs/old/source/icon-deepsparse.png" />
&nbsp;&nbsp;DeepSparse
<h1 style="display: flex; align-items: center;" >
<img width="60" height="60" alt="tool icon" src="https://neuralmagic.com/wp-content/uploads/2024/03/icon_DeepSparse-005.png" />
<span>&nbsp;&nbsp;DeepSparse</span>
</h1>
<h4>Sparsity-aware deep learning inference runtime for CPUs</h4>
<div align="center">
<a href="https://docs.neuralmagic.com/deepsparse/">
<img alt="Documentation" src="https://img.shields.io/badge/documentation-darkred?&style=for-the-badge&logo=read-the-docs" height="20" />
</a>
<a href="https://join.slack.com/t/discuss-neuralmagic/shared_invite/zt-q1a1cnvo-YBoICSIw3L1dmQpjBeDurQ/">
<a href="https://neuralmagic.com/community/">
<img alt="Slack" src="https://img.shields.io/badge/slack-purple?style=for-the-badge&logo=slack" height="20" />
</a>
<a href="https://github.com/neuralmagic/deepsparse/issues/">
Expand Down Expand Up @@ -74,7 +74,7 @@ print(pipeline(prompt, max_new_tokens=75).generations[0].text)
# Sparsity is the property of a matrix or other data structure in which a large number of elements are zero and a smaller number of elements are non-zero. In the context of machine learning, sparsity can be used to improve the efficiency of training and prediction.
```

> [Check out the `TextGeneration` documentation for usage details.](https://github.com/neuralmagic/deepsparse/blob/main/docs/llms/text-generation-pipeline.md)
Check out the [`TextGeneration` documentation for usage details](https://github.com/neuralmagic/deepsparse/blob/main/docs/llms/text-generation-pipeline.md) and get the [latest sparsified LLMs on our HF Collection](https://huggingface.co/collections/neuralmagic/deepsparse-sparse-llms-659d61e81774dd48343642bf).

### Sparsity :handshake: Performance

Expand Down Expand Up @@ -213,7 +213,7 @@ Confirm that telemetry is shut off through info logs streamed with engine invoca
### Get In Touch

- [Contribution Guide](https://github.com/neuralmagic/deepsparse/blob/main/CONTRIBUTING.md)
- [Community Slack](https://join.slack.com/t/discuss-neuralmagic/shared_invite/zt-q1a1cnvo-YBoICSIw3L1dmQpjBeDurQ)
- [Community Slack](https://neuralmagic.com/community/)
- [GitHub Issue Queue](https://github.com/neuralmagic/deepsparse/issues)
- [Subscribe To Our Newsletter](https://neuralmagic.com/subscribe/)
- [Blog](https://www.neuralmagic.com/blog/)
Expand Down
9 changes: 6 additions & 3 deletions src/deepsparse/transformers/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -151,9 +151,12 @@ https://sparsezoo.neuralmagic.com/?useCase=text_generation)
```python
from deepsparse import Pipeline

opt_pipeline = Pipeline.create(task="opt", model_path="zoo:opt-1.3b-opt_pretrain-quantW8A8")
llama_pipeline = Pipeline.create(
task="text-generation",
model_path="zoo:llama2-7b-ultrachat200k_llama2_pretrain-pruned50_quantized"
)

inference = opt_pipeline("Who is the president of the United States?")
inference = llama_pipeline("Who is the president of the United States?")

>> 'The president of the United States is the head of the executive branch of government...'
```
Expand All @@ -163,7 +166,7 @@ Spinning up:
```bash
deepsparse.server \
--task text-generation \
--model_path zoo:opt-1.3b-opt_pretrain-pruned50_quantW8A8
--model_path zoo:llama2-7b-ultrachat200k_llama2_pretrain-pruned50_quantized
```

Making a request:
Expand Down

0 comments on commit e6e728c

Please sign in to comment.