[Question]: How to work with winCLIP on custom dataset? (also a call for better examples)

[brechtBDCK]

Describe the bug

Following the recent advance in Anomaly classification/segmentation, I wanted to try the new winCLIP model, of which the anomalib library also has an implementation. As this is a zero-shot model, i would assume that this would easily work out of the box. Anyone have any experience of how to test zero-shot or to give a couple "normal/healthy" images for few-shot? I can't get it working. I would assume that (as it's zero-shot) i would be able to just input some custom defect images? I also can't figure out the few-shot using some normal/healthy images from my own dataset.

Here is my current code:

from anomalib.models.image import WinClip
from anomalib.engine import Engine


# Import the datamodule
from anomalib.data import Folder
from anomalib.data.utils import TestSplitMode

# Create the datamodule
datamodule = Folder(
    name="lasercut_plank",
    root="./DATA_0shot",
    normal_dir="normal",
    test_split_mode=TestSplitMode.NONE
)

# Setup the datamodule
datamodule.setup()

# Access the datasets
train_dataset = datamodule.train_data

# Access the dataloaders
train_dataloader = datamodule.train_dataloader()

# Create the model and engine
model = WinClip(class_name="lasercut_plank")
engine = Engine(task="segmentation")

# Train a Patchcore model on the given datamodule
engine.train(datamodule=datamodule, model=model)

Dataset

Other (please specify in the text field below)

Model

Other (please specify in the field below)

Steps to reproduce the behavior

None

OS information

OS information:
WSL ubuntu on windows 11

Name: anomalib
Version: 1.2.0

Expected behavior

none

Screenshots

No response

Pip/GitHub

pip

What version/branch did you use?

No response

Configuration YAML

none

Logs

none

Code of Conduct

• I agree to follow this project's Code of Conduct

[djdameln]

Hi, you are right, WinClip is a zero-shot model, so in theory, you can use it out-of-the box to generate some predictions on local images.

However, the full answer is a bit more nuanced. WinClip, like many other anomaly detection models, does not produce binary normal-vs-anomalous classification labels or segmentation masks, but instead produces real-valued anomaly scores (image-level) and anomaly maps (pixel-level). To convert the anomaly scores to labels, we need to compare them to some known threshold value (and assign a normal label to all scores below the threshold, and an anomalous label to all scores above the threshold).

When using WinClip out-of-the-box to generate predictions on a new dataset, we don't know which threshold value we should use. This is why we usually run a validation sequence on a small subset of our data which contains some normal and anomalous samples. During this validation sequence, we adaptively compute the optimal threshold value by maximizing the F1 score over the validation set. The validation sequence also collects some statistics used for normalizing the raw anomaly scores and maps to the [0, 1] range.

Now let's get back to your use-case. If retrieving just the raw anomaly scores is sufficient for your use-case, you could use WinClip to generate predictions by directly calling the engine's predict method, which allows passing a path to an image or folder of images. The following would be sufficient to yield the predictions:

from anomalib.models import WinClip
from anomalib.engine import Engine

engine = Engine(task="classification")
model = WinClip(class_name="lasercut_plank")

predictions = engine.predict(model, data_path="./DATA_0shot/normal")

You can access the raw predictions by inspecting the predictions variable.

When running predict, the engine also writes visualizations of the predictions to the results folder in your working directory. By looking at the visualizations, you can easily confirm that the model is not able to make sense of the raw anomaly scores. The labels will likely be inaccurate and the heatmaps flat. This is because of the lack of adequate threshold and normalization statistics needed to convert the raw predictions to meaningful results.

To improve this, you will have to provide Anomalib with some normal and anomalous validation images that can be used for normalization and thresholding. Here's some example code illustrating how this could be achieved.

from anomalib.models import WinClip
from anomalib.engine import Engine
from anomalib.data import Folder

engine = Engine(task="classification")
model = WinClip(class_name="lasercut_plank")

datamodule = Folder(
    name="lasercut_plank",
    root="./DATA_0shot",
    normal_dir="normal",
    abnormal_dir="abnormal",  # or wherever your abnormal images are located
    task="classification",   # we don't have ground-truth masks, so we can't use the segmentation task type
)

engine.validate(model, datamodule=datamodule)
engine.predict(model, data_path="path/to/your/query/images")

Few-shot mode

To run WinClip in few-shot mode, simply set the k_shot parameter to the desired number of reference images (e.g. 1 for 1-shot or 2 for 2-shot), and pass a path to a folder of normal reference images using the few_shot_source parameter. Here's an example how to run the model in 2-shot mode:

model = WinClip(
    class_name="lasercut_plank",
    k_shot=2,
    few_shot_source="path/to/normal/reference/images",
)

[brechtBDCK]

Thanks for the clear reply! I actually do have some ground truth segmentation masks (only of the anomalous images). Could you give a quick idea of how that would look in code?

[djdameln]

If you have ground truth masks that's great. You can run the model in segmentation mode and the validation sequence will compute separate pixel-level threshold and normalization stats. Just pass the mask path when instantiating the datamodule, and set the task to "segmentation". Note that segmentation is the default task type in Anomalib, so you could also just omit the parameter from the constructor call. The following would be sufficient:

engine = Engine()
model = WinClip(class_name="lasercut_plank")

datamodule = Folder(
    name="lasercut_plank",
    root="./DATA_0shot",
    normal_dir="normal",
    abnormal_dir="abnormal",  # or wherever your abnormal images are located
    mask_dir="path/to/your/masks".  # can be relative to root or absolute
)

engine.validate(model, datamodule=datamodule)
engine.predict(model, data_path="path/to/your/query/images")

[brechtBDCK]

Are there any requirements for the masks i.e. naming convention or black/white convention? Mine are black background + white mask, named 14_mask.png (for image 14.png). Currently my "predicted mask" output is either fully white or black.

from anomalib.models import WinClip
from anomalib.engine import Engine
from anomalib.data import Folder

engine = Engine()
model = WinClip(
    class_name="lasercut_plank",
    # k_shot=5,
    # few_shot_source="./DATA_0shot/normal",
)
datamodule = Folder(
    name="lasercut_plank",
    root="./DATA_0shot",
    normal_dir="normal",
    abnormal_dir="abnormal",  # or wherever your abnormal images are located
    mask_dir="ground_truth",  # or wherever your ground-truth masks are located
)

engine.validate(model, datamodule=datamodule)
engine.predict(model, data_path="./DATA_0shot/mix_for_testing")

[djdameln]

The Folder datamodule should be able to find your mask files, as long as the image file name (without file extension) of the images occurs in the mask file name, so "14.png" and "14_mask.png" should work fine. Binary masks with black background/white mask should also be fine. It's hard to tell what is going wrong without seeing your dataset. It's possible that WinClip just does not perform well on your dataset.

One thing you could do is see if you can get some sensible results with our toy "hazelnut" dataset (available for download here), to confirm that your setup is correct. Just use one defect type (e.g. crack) for validation and the other defect type (colour) for inference, and you should be able to see some fairly good predictions.

[samet-akcay]

Thanks @djdameln for the nice explanation. Would it be an idea to rename few_shot_source to k_shot_source for consistency?

[djdameln]

I would prefer to keep it as few_shot_source. Because k in k_shot can be 0, while the few_shot prefix signals that it only applies for k>=1