CPU significantly outperforming GPU performance

[lewisjluck]

Hi anomalib community!

Firstly, would like to thank the team - truly innovative piece of software and looks extremely useful to many applications.

I am experimenting in attempting to apply anomalib to an industrial process, checking to see if it can detect stray fallen objects on a conveyor belt covering mounds of material (from simulated images). In doing so, I have attempted to run inference on a dataset of 294 images using both the "accelerator: cpu" and "accelerator: gpu" options in my .yaml configuration file. I would expect the gpu to be much faster, but on this testing set I get an average of 96ms per sample for CPU, and 336ms per sample for GPU. I have a NVIDIA GeForce RTX 3070. When checking my system diagnostics, predictably using the CPU option consumes over 90% of the CPU, but when using the GPU option, about 50% of my GPU and 50% of my CPU is used simultaenously. Is this perhaps a situation where memory is being copied constantly between RAM and the GPU, meaning that the parallelisation benefit of GPUs are compromised? This significant of a performance decrease using a GPU is surprising to me (I would expect it to be faster than the CPU!).

I am running inference by the suggest command line interface using tools/inference/lightning_inference.py . Inference to my knowledge is running correctly, and my model trained with no issues on a dataset of about 1500 normal images.

Should I perhaps try another inference model like OpenVINO? May this fix my issue?

Fair warning, I am quite inexperienced in anomalib, and the torch semantics in general, so if I am missing something obvious please feel free to point out my shortsightedness.

Thank you!

[blaz-r]

Hello. Good to hear that you find Anomalib useful 😄.

In order to help you more, it'd be great to have entire config.yaml of the model you are trying to train, so please if you can share that.

[lewisjluck]

Hi!

My mistake. Please see below.

dataset:
  name: <removed>
  format: folder
  root: ../images_dataset
  normal_dir: not_item # name of the folder containing normal images.
  abnormal_dir: item # name of the folder containing abnormal images.
  task: classification # classification or segmentation
  mask_dir: null #optional
  normal_test_dir: null # optional
  extensions: null
  split_ratio: 0.2  # normal images ratio to create a test split
  seed: 0
  image_size: 256
  train_batch_size: 32
  eval_batch_size: 32
  num_workers: 16
  normalization: imagenet # data distribution to which the images will be normalized
  test_split_mode: from_dir # options [from_dir, synthetic]
  val_split_ratio: 0.5 # fraction of train/test images held out for validation (usage depends on val_split_mode)
  transform_config:
      train: null
      eval: null
  val_split_mode: from_test # determines how the validation set is created, options [same_as_test, from_test]
  tiling:
      apply: false
      tile_size: null
      stride: null
      remove_border_count: 0
      use_random_tiling: False
      random_tile_count: 16
    

model:
  name: padim
  backbone: resnet18
  pre_trained: true
  layers:
    - layer1
    - layer2
    - layer3
  normalization_method: min_max # options: [none, min_max, cdf]

metrics:
  image:
    - F1Score
    - AUROC
    - Accuracy
  threshold:
    method: adaptive #options: [adaptive, manual]
    manual_image: null
    manual_pixel: null

visualization:
  show_images: False # show images on the screen
  save_images: True # save images to the file system
  log_images: True # log images to the available loggers (if any)
  image_save_path: null # path to which images will be saved
  mode: full # options: ["full", "simple"]

project:
  seed: 42
  path: ./results
  unique-dir: true

logging:
  logger: [] # options: [comet, tensorboard, wandb, csv] or combinations.
  log_graph: false # Logs the model graph to respective logger.

optimization:
  export_mode: null # options: torch, onnx, openvino

# PL Trainer Args. Don't add extra parameter here.
trainer:
  enable_checkpointing: true
  default_root_dir: null
  gradient_clip_val: 0
  gradient_clip_algorithm: norm
  num_nodes: 1
  devices: 1
  enable_progress_bar: true
  overfit_batches: 0.0
  track_grad_norm: -1
  check_val_every_n_epoch: 1 # Don't validate before extracting features.
  fast_dev_run: false
  accumulate_grad_batches: 1
  max_epochs: 1
  min_epochs: null
  max_steps: -1
  min_steps: null
  max_time: null
  limit_train_batches: 1.0
  limit_val_batches: 1.0
  limit_test_batches: 1.0
  limit_predict_batches: 1.0
  val_check_interval: 1.0 # Don't validate before extracting features.
  log_every_n_steps: 50
  accelerator: "cpu" # <"cpu", "gpu", "tpu", "ipu", "hpu", "auto">
  strategy: null
  sync_batchnorm: false
  precision: 32
  enable_model_summary: true
  num_sanity_val_steps: 0
  profiler: null
  benchmark: false
  deterministic: false
  reload_dataloaders_every_n_epochs: 0
  auto_lr_find: false
  replace_sampler_ddp: true
  detect_anomaly: false
  auto_scale_batch_size: [false](url)
  plugins: null
  move_metrics_to_cpu: false
  multiple_trainloader_mode: max_size_cycle

[blaz-r]

I'm not that familiar with the code, as it seems most of computation is done on GPU. But judging from the code, embeddings of every batch are put on cpu so I think this might be the reason.

[lewisjluck]

Ah yes I see that now. Unfortunate that it can have such a negative impact to render using a GPU useless. Something to dig into a little more maybe. Thanks very much for your help!