Skip to content

Commit

Permalink
Polish CoreML Llama Doc (pytorch#5745)
Browse files Browse the repository at this point in the history
Summary:
Document
* The export python script arguments to export llama for coreml
* The cmake argument to compile Mac runner with coreml backend

Minor fix: Remove outdated arguments

Pull Request resolved: pytorch#5745

Reviewed By: kirklandsign

Differential Revision: D63652220

Pulled By: cccclai

fbshipit-source-id: 1703158b4e2b31cc29d08e9262e56d564a90e539
  • Loading branch information
yifan_shen3 authored and facebook-github-bot committed Sep 30, 2024
1 parent 9c38cf7 commit 418c4c3
Showing 1 changed file with 6 additions and 0 deletions.
6 changes: 6 additions & 0 deletions examples/models/llama2/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -332,6 +332,8 @@ Note for Mac users: There's a known linking issue with Xcode 15.1. Refer to the
For Llama2 and stories models, pass the converted `tokenizer.bin` file instead of `tokenizer.model`.
To build for CoreML backend and validate on Mac, replace `-DEXECUTORCH_BUILD_XNNPACK=ON` with `-DEXECUTORCH_BUILD_COREML=ON`
## Step 5: Run benchmark on Android phone
**1. Build llama runner binary for Android**
Expand Down Expand Up @@ -418,6 +420,10 @@ for each backend ([CoreML](https://pytorch.org/executorch/main/build-run-coreml.
The iOS LLAMA app supports the CoreML and MPS model and the Android LLAMA app supports the QNN model. On Android, it also allow to cross compiler the llama runner binary, push to the device and run.
For CoreML, there are 2 additional optional arguments:
* `--coreml-ios`: Specify the minimum iOS version to deploy (and turn on available optimizations). E.g. `--coreml-ios 18` will turn on [in-place KV cache](https://developer.apple.com/documentation/coreml/mlstate?language=objc) and [fused scaled dot product attention kernel](https://apple.github.io/coremltools/source/coremltools.converters.mil.mil.ops.defs.html#coremltools.converters.mil.mil.ops.defs.iOS18.transformers.scaled_dot_product_attention) (the resulting model will then need at least iOS 18 to run, though)
* `--coreml-quantize`: Use [quantization tailored for CoreML](https://apple.github.io/coremltools/docs-guides/source/opt-quantization-overview.html). E.g. `--coreml-quantize b4w` will perform per-block 4-bit weight-only quantization in a way tailored for CoreML
# What is coming next?
## Quantization
- Enabling FP16 model to leverage smaller groupsize for 4-bit quantization.
Expand Down

0 comments on commit 418c4c3

Please sign in to comment.