Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add quickstart instructions for iree_tests #306

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Add quickstart instructions for iree_tests #306

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions iree_tests/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,39 @@
# IREE Tests

## Quickstart instructions
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This duplicates the setup at https://github.com/nod-ai/SHARK-TestSuite/tree/main/iree_tests#common-venv-setup-with-deps . Want to merge the two?

  • Keep it as "quickstart" near the top
  • Keep as a single code block for easier copy/paste
  • Add a line or two for local build dir and $PATH vs python packages
  • Move the overview text ("A conftest.py file collects test cases from subdirectories, wrapping ...") to the newly labeled "About" section


```
#!/bin/bash
Comment on lines +5 to +6
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: add an explicit language when known to code blocks, to help markdown renderers apply syntax highlighting

Suggested change
```
#!/bin/bash
```bash
#!/bin/bash

set -e

cd ~/SHARK-TestSuite/

# Create and activate a virtual environment
python3 -m venv .venv
source .venv/bin/activate

# Install required packages
pip install -r iree_tests/requirements.txt
pip install ./common_tools/
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm considering folding common_tools back into iree_tests to this extra setup step isn't needed, FYI.


# Optional: Install IREE compiler and runtime if you don't want to use a local copy
# pip install --find-links https://iree.dev/pip-release-links.html iree-compiler iree-runtime --upgrade

# Install Git LFS
git lfs install
Comment on lines +22 to +23
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It helps to run this before cloning the repo IIRC. I'm not sure if installing git lfs will automatically fetch blobs if you had previously cloned/fetched without git lfs.


# Optional: Download large remote files required for certain tests
# python download_remote_files.py

# Run a simple test to verify setup
pytest iree_tests/simple

echo "Setup complete! You can now run IREE tests."
echo "Example: pytest iree_tests -n auto"
Comment on lines +31 to +32
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We rarely ever run all tests with the same config file. Consider the simple case of CPU:

The ONNX tests run single threaded (--device=local-sync') and have lists of XFAIL'd ONNX tests. The model tests run multithreaded (--device=local-task), enable some baseline optimizations (--iree-llvmcpu-target-cpu-features=host`), and have lists of XFAIL'd model tests.

I'd put a command here that is expected to work out of the box. Maybe something more like pytest iree_tests -n auto -k "test_sub_" (see "Run a filtered subset of tests" further down in this file) or pytest iree_tests/onnx -n auto.

```

## About

This directory contains generated test suites for running through IREE's
compiler and runtime tools.

Expand Down
Loading