Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Task]: Add unit tests for Engine.export #2273

Open
ashwinvaidya17 opened this issue Aug 26, 2024 · 4 comments · May be fixed by #2438
Open

[Task]: Add unit tests for Engine.export #2273

ashwinvaidya17 opened this issue Aug 26, 2024 · 4 comments · May be fixed by #2438
Labels
Good First Issue Issues that can be picked up by someone unfamiliar with the repo and would like to contribute.

Comments

@ashwinvaidya17
Copy link
Collaborator

What is the motivation for this task?

Currently, export types like NNCF and POT do not fall under testing. This leads to outdated docstrings and broken functionality.

Describe the solution you'd like

The test can be as simple as optimizing a pre-trained Padim model. We already have the trained checkpoint available in a fixture.

Additional context

No response
@ashwinvaidya17 ashwinvaidya17 added the Good First Issue Issues that can be picked up by someone unfamiliar with the repo and would like to contribute. label Aug 26, 2024
@mastaan66
Copy link

Can you provide more context regarding the issue?

@sky0walker99
Copy link

I would like to work on this issue.

@samet-akcay
Copy link
Contributor

samet-akcay commented Nov 14, 2024 •
edited
Loading

@sky0walker99, sure, thanks for your interest. @ashwinvaidya17, can you provide an acceptance criteria for @sky0walker99

@ashwinvaidya17
Copy link
Collaborator Author

OpenVINO export also supports post training optimization, and neural network compression. These can be enabled by passing additional parameters to Engine.export.
While we have tests for Engine.export

anomalib/tests/integration/cli/test_cli.py

Line 159 in bcc0b43

def test_export(

The tests do not check these two features of OpenVINO export. Since the API has changed over time, the PTQ and NNCF CLI invocation has diverged from that mentioned here in the docstrings.

anomalib/src/anomalib/engine/engine.py

Line 937 in bcc0b43

anomalib export --model Padim --export_type openvino --ckpt_path <PATH_TO_CHECKPOINT> \

Also, since these don't fall under the coverage tests, we don't come to know when they break.

The solution to address this can be as simple as extending the OpenVINO export tests to add POT and NNCF parameters to the list so that we can then test 1. Normal OpenVINO export, 2. All the combination of compression types listed here

anomalib/src/anomalib/deploy/export.py

Line 36 in bcc0b43

class CompressionType(str, Enum):
. There is no need to check the accuracy or train the model from scratch. We can just use the existing checkpoint, and export the model loaded from it. The idea is to check if the model is saved to the file system.

@sky0walker99 sky0walker99 linked a pull request Nov 26, 2024 that will close this issue
Open
9 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Good First Issue Issues that can be picked up by someone unfamiliar with the repo and would like to contribute.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

🔨 Added Tests for POT and NNCF Compression Types sky0walker99/anomalib
4 participants
@ashwinvaidya17 @samet-akcay @sky0walker99 @mastaan66