-
Notifications
You must be signed in to change notification settings - Fork 694
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[PIMO] speed up #2379
base: main
Are you sure you want to change the base?
[PIMO] speed up #2379
Conversation
Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>
Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>
Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>
Signed-off-by: jpcbertoldo <24547377+jpcbertoldo@users.noreply.github.com>
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
@@ -68,15 +69,18 @@ def pimo_curves( | |||
_validate.has_at_least_one_normal_image(masks) | |||
|
|||
image_classes = images_classes_from_masks(masks) | |||
anomaly_maps_normal_images = anomaly_maps[image_classes == 0] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
anomaly_maps_normal_images = anomaly_maps[image_classes == 0] | |
normal_anomaly_maps = anomaly_maps[image_classes == 0] |
@@ -276,6 +275,73 @@ def aupimo_scores( | |||
# =========================================== AUX =========================================== | |||
|
|||
|
|||
def _binary_search_threshold_at_fpr_target( | |||
anomaly_maps_normals: torch.Tensor, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
anomaly_maps_normals: torch.Tensor, | |
normal_anomaly_maps: torch.Tensor, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, this is a nice optimization. Just out of curiosity, do you have some numbers on the amount of speed up is achieved by this change?
fpr_target: float | torch.Tensor, | ||
maximum_iterations: int = 300, | ||
) -> float: | ||
"""Binary search of threshold that achieves the given shared FPR level. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It would be good to add a more detailed description to this docstring, explaining why we need to apply the binary search and how it is performed. This would be useful for future reference.
Looks like one of the pimo notebook tests are failing |
📝 Description
without
numba
, aupimo became annoyingly slow for full resolution test, so this idea show speed it up by removing unnecessayr computationthis parameter is what mostly makes it so inefficient (
num_thresholds = 300_000
)anomalib/src/anomalib/metrics/pimo/functional.py
Line 118 in 1465b05
it has to be so big to make sure that there will be enough points in the AUC integration within the integration range
the current implementation thresholds the anomaly score maps from their min to max value, which is the inefficient because we only need it in a much smaller range
strategy to improve it:
- use binary search to find the thresholds corresponding to the fpr integration bounds
- compute the binary classification curves within those bounds
- decrease the number of thresholds from
300_000
to300
✨ Changes
Select what type of change your PR is:
✅ Checklist
Before you submit your pull request, please make sure you have completed the following steps:
For more information about code review checklists, see the Code Review Checklist.