integrations/weights-biases/

[giscus[bot]]

integrations/weights-biases/

Discover how to train your YOLOv8 models efficiently with Weights & Biases. This guide walks through integrating Weights & Biases with YOLOv8 to enable seamless experiment tracking, result visualization, and model explainability.

https://docs.ultralytics.com/integrations/weights-biases/

[J0ELh]

Is it possible to remove logging and only log progress in the command line? For my training process yolov8 seems to automatically connect with my wandb account and start logging. This process is very time-consuming so I'd like to know whether there's a flag like log = False or something like that.

[J0ELh]

^ this is because I'm done experimenting and need quick retraining options now.

[glenn-jocher]

Hey there! 👋 If you're looking to disable automatic logging to Weights & Biases (W&B) during your YOLOv8 training to speed things up, you can do so by setting the wandb argument to False when initiating your training command. Here's how you can do it:

yolo detect train data=coco128.yaml model=yolov8n.pt epochs=100 wandb=False

This will prevent YOLOv8 from connecting to your W&B account and logging the training progress there, allowing for quicker retraining options as per your requirement. Happy training! 🚀

[danielst4]

Hey there, I am trying to make a sweep using WandB but cant quite put together where to insert what into the YOLO call, to use different hyperparameters and track them. Even though I found a site where one has done this he too doesnt describe what he had to do. I am not sure if it is more complicated than I think and if I really have to insert stuff into the training function or if there is an easy way to achieve this.
I would be really greatful for help to know what exactly is necessary to make a WandB hyperparameter sweep with YOLO
Thanks
Daniel

[pderrenger]

Hey Daniel 👋,

Great to hear you're exploring hyperparameter sweeps with WandB and YOLO! It's simpler than it might seem at first. Here's a quick guide to get you started:

1. Initialize WandB: Before starting your training, make sure to log in to WandB in your script or notebook using:

   import wandb
   wandb.login()

2. Set Up the Sweep: Define your sweep configuration in WandB, specifying the hyperparameters you want to sweep over. This is usually done in a YAML file or directly in your script. For example:

   method: random
   metric:
     name: val/box_loss
     goal: minimize
   parameters:
     learning_rate:
       min: 0.0001
       max: 0.001
     batch_size:
       values: [16, 32, 64]

3. Integrate Sweep with YOLO: Use the wandb.integration.ultralytics.add_wandb_callback function to integrate WandB with your YOLO training. This will automatically log metrics, model checkpoints, and more to your WandB dashboard.

   from ultralytics import YOLO
   from wandb.integration.ultralytics import add_wandb_callback
   
   # Initialize WandB run
   wandb.init(project="your_project_name", config={
       "learning_rate": 0.001,
       "batch_size": 32,
       # Add other hyperparameters here
   })
   
   # Load YOLO model
   model = YOLO("yolov8n.pt")
   
   # Add WandB callback for logging
   add_wandb_callback(model)
   
   # Start training
   model.train(data="coco128.yaml", epochs=5, imgsz=640)

4. Start the Sweep: Once everything is set up, you can start the sweep in WandB. This will iterate over the defined hyperparameter space, automatically logging each experiment's results.

Remember, the key part is to integrate WandB with YOLO using the add_wandb_callback function and to ensure your sweep configuration in WandB is set up correctly to explore the desired hyperparameter space.

Hope this helps! If you have any more questions or run into issues, feel free to ask. Happy sweeping! 🚀

[danielst4]

Thank you Paula, this helped a lot!
I can make a sweep now but I noticed that the values of the parameters which are given from the sweep to the script dont match the parameters used.
Even though the command looks like this:
wandb.wandb_agent - INFO - About to run command: /usr/bin/env python training_rtdetr.py --batch=8 --epochs=6 --imgsz=1280 --learning_rate=0.0007013348811086657
where "training_rtdetr.py" is a script with the code you gave me. (I am working with the RTDETR but it shouldnt make a difference as the metrics and calls are the same if I understand correctly?)
But then after engine/trainer it says its using epochs=100 and it also uses that number for training.
So just to be sure about a few things, I already logged in to WandB, I wrote a script with the code from your 3., I went to wandb, created a sweep there with the code from 2. + the line program: training_rtdetr.py and in a terminal I used the resulting call wandb agent [sweepid]
Also I am kinda confused because you used "learning_rate" and "batch_size" but as far as I can tell these are not the parameters I can give for the training and they should be called "lr0" and "batch" respectively (which I both tried but they didnt work)
I hope I provided all the necessary information otherwise just tell me what you need to know

[pderrenger]

@danielst4 hey Daniel,

It sounds like you're on the right track with setting up your WandB sweep with YOLOv8! Regarding the mismatch in parameter values, it seems like there might be a discrepancy between the parameter names expected by your script and those defined in your WandB sweep configuration.

For YOLOv8, the correct parameter names for learning rate and batch size are indeed lr0 and batch, respectively. If you're using these parameters in your training_rtdetr.py script, ensure they're correctly parsed from the command line arguments. Here's a quick Python snippet using argparse for reference:

import argparse

# Parse command line arguments
parser = argparse.ArgumentParser()
parser.add_argument('--batch', type=int, help='batch size')
parser.add_argument('--epochs', type=int, help='number of epochs')
parser.add_argument('--imgsz', type=int, help='image size')
parser.add_argument('--learning_rate', type=float, help='learning rate')
args = parser.parse_args()

# Now you can use args.batch, args.epochs, etc., in your script

Make sure your WandB sweep configuration uses the same parameter names as those your script expects. For example:

program: training_rtdetr.py
method: bayes
metric:
  name: val/box_loss
  goal: minimize
parameters:
  learning_rate:
    min: 0.0001
    max: 0.001
  batch:
    values: [8, 16, 32]
  epochs:
    value: 6
  imgsz:
    value: 1280

If you've already done this and the issue persists, it might be helpful to double-check how your script handles these parameters and ensure they're being correctly applied to your YOLOv8 model configuration.

Feel free to share more details if you're still encountering issues. We're here to help! 🚀

[DungHD-1997]

from ultralytics import YOLO
from wandb.integration.ultralytics import add_wandb_callback
import wandb
def train():
wandb.login()
project_name = "test_ai"
# Step 1: Initialize a Weights & Biases run
wandb.init(project=project_name, job_type="training",name="old_data")

# Step 2: Define the YOLOv8 Model and Dataset
model_name = "yolov8x"
dataset_name = "C://Users//Administrator//project//yolov5_train_data//data//slant_2024.yaml"
model = YOLO(f"{model_name}.pt")

# Step 3: Add W&B Callback for Ultralytics
add_wandb_callback(model, enable_model_checkpointing=True)
model_name = "train_01"
# Step 4: Train and Fine-Tune the Model
model.train(project=project_name,name=model_name, data=dataset_name, epochs=100, imgsz=640, batch = 8)

# Step 5: Validate the Model
model.val()

# Step 6: Perform Inference and Log Results
#model(["path/to/image1", "path/to/image2"])

# Step 7: Finalize the W&B Run
wandb.finish()

if name == "main":
train()
I used the above code to train custom data, but when using wandb, the GPU memory continuously increases over epochs until an out of memory error appears. program has been stopped.
Error message:
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB. GPU 0 has a total capacity of 15.88 GiB of which 7.62 MiB is free. Of the allocated memory 14.53 GiB is allocated by PyTorch, and 103.44 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

[DungHD-1997]

If I don't use wandb, the GPU memory hardly changes over each epoch. Is there something wrong with my code?

[glenn-jocher]

Hey there! It looks like you're encountering a memory issue related to using Weights & Biases (W&B) with YOLOv8, specifically the GPU running out of memory during training. This can happen for a few reasons, including excessive memory allocation by PyTorch or W&B logging mechanisms. Here are a couple of suggestions that might help:

1. Reduce Batch Size: Lowering the batch size can significantly reduce memory usage. Although this might increase training time, it can help avoid out-of-memory errors.

   model.train(batch=4)  # Try reducing the batch size

2. Limit Logging Frequency: W&B logs a lot of data, which can increase memory usage. Try reducing the logging frequency or the amount of data logged.

   wandb.init(settings=wandb.Settings(sync_tensorboard=True), sync_interval=1000)

3. Enable Gradient Checkpointing: This technique reduces memory usage at the cost of a slight increase in computation time.

   # You'll need to modify your model/training script to enable this.

4. Clear CUDA Cache: Manually clearing the CUDA cache after each epoch might help release unused memory.

   import torch
   torch.cuda.empty_cache()

5. Optimize Model and DataLoader: Ensure that your DataLoader is not the source of the memory leak, and try to optimize your model's memory usage.

6. Use W&B's enable_model_checkpointing=False: Since model checkpointing can increase memory usage, try disabling it when adding the W&B callback.

   add_wandb_callback(model, enable_model_checkpointing=False)

Finally, consider profiling your code to identify specific lines or operations that cause the memory spike. PyTorch and W&B offer tools and documentation on best practices to optimize memory usage. 📘

Hope this helps! If the issue persists, feel free to reach out with more details or check the W&B and YOLOv8 docs for more insights.

[JiyongBoo]

Is it possible to train RT-DETR model with wandb? In add_wandb_callback() function, there are nothing about rt-detr.
I tried replacing yolov8 with rt-detr in the example code. In the result, I confirmed that runs.summary["Validation-Table"] and runs.summary["Train-Table"] did not appear. How do I solve this problem?

[pderrenger]

@JiyongBoo hi there! 😊 It seems you're diving into integrating RT-DETR models with Weights & Biases for tracking and logging purposes. We currently have detailed documentation around using YOLOv8 models with Weights & Biases, but I understand you're looking for something more specific to RT-DETR models.

While our add_wandb_callback() function primarily supports YOLOv8 models out of the box, adapting it for RT-DETR models might need some extra steps. Currently, the automatic logging of Validation-Table and Train-Table is tailored to the outputs of YOLOv8 models.

To work around this for RT-DETR, you might consider manually logging these tables to W&B. You can achieve this by extracting relevant metrics from the RT-DETR model's training and validation process and using wandb.log() function directly. Here's a simplified example:

import wandb

# After each validation step in your training loop
wandb.log({
    "Validation-Table": wandb.Table(dataframe=your_val_df),
    "Train-Table": wandb.Table(dataframe=your_train_df),
})

Make sure to replace your_val_df and your_train_df with your actual pandas DataFrames or data structures holding your training and validation metrics. 📊

Unfortunately, without specific support for RT-DETR in the add_wandb_callback() function, this manual step is necessary. However, it provides a flexible way to log whatever information you find most critical to your experiment tracking.

We're always working to support more models and functionalities. Stay tuned for future updates, and don't hesitate to contribute if you find a neat solution for integrating RT-DETR with W&B! Happy coding! 🚀

[TJdeWet]

Hi!

I am using wandb to do hyperparameter tuning. Is it possible to maximise validation mAP50? This seems to be maximising the training mAP50 if I am not mistaken?

metric:
name: mAP50
goal: maximise

Otherwise should I do hyperparameter using something like this instead:

metric:
name: val/box_loss
goal: minimize

Thanks!

[glenn-jocher]

Hi there!

For your hyperparameter tuning in Weights & Biases, if you're aiming to maximize validation mAP50 specifically, I recommend adjusting your metric settings accordingly. To focus on validation mAP50, your metric configuration might look something like this:

metric:
  name: val/mAP50
  goal: maximize

This setup ensures you're targeting the maximization of mAP50 on the validation data directly.

Using val/box_loss with the goal to minimize can also be effective, especially if you prefer to focus on reducing the box prediction errors during validation:

metric:
  name: val/box_loss
  goal: minimize

Both approaches are valid and can be chosen based on your specific model performance goals. Hope this helps! If you have more questions, feel free to ask. 😊

[TJdeWet]

Hi Glenn,

Thanks for getting back to me. I just tried it, and it doesn't seem that tuning to maximize val/mAP50 is working - the only metrics I see that are being logged are: cls, box, and dfl loss (both train and val), and metrics/mAP50(B), metrics/mAP50-75(B), metrics/Recall(B), and metrics/Precision(B), which I believe are the training metrics? I would like to log those same metrics for validation, and tune based on val/mAP50. Is there a way I can do this?

Thanks!

[glenn-jocher]

Hi there!

It looks like the specific validation metrics you're interested in (val/mAP50) aren't being logged automatically. To log these validation metrics, you'll need to ensure that they are explicitly set up for tracking in your training script.

Here's a general approach to ensure validation metrics like val/mAP50 are being logged and used for your hyperparameter tuning:

1. Make sure your validation function or loop is setting up Weights & Biases to log these metrics. This can usually be done with something like:

   wandb.log({'val/mAP50': your_map50_value})

2. Integrate this logging into the point in your code where validation metrics are calculated post-epoch.

Ensure that your wandb initialization and callback in your model training script are correctly implemented, as this might be the part where validation metrics get missed if not set up properly.

If you need more detailed modifications to the script, it would be helpful to see how you are currently implementing Weights & Biases and your validation procedure. This will help in suggesting a more precise code adjustment.

Thanks!

[Pushtogithub23]

I used this code, but it only logs system logs; no training metrics, media, or validation metrics are being logged as shown in the video. How can I log all the training metrics and media as demonstrated in the video? Can anyone help me with this?

from ultralytics import YOLO
from wandb.integration.ultralytics import add_wandb_callback

wandb.login()
wandb.init(project = 'Hand-Gestures-Detection', name = 'yolov8n_5',  job_type='training')

model = YOLO("yolov8n.pt")

add_wandb_callback(model, enable_model_checkpointing=True)
model.train(data="HAND-GESTURE-DETECTION-1/data.yaml", 
            epochs=5,
            imgsz = 640,
            project = "Hand-Gestures-Detection",
            name = 'yolov8n_5'
            )

[glenn-jocher]

@Pushtogithub23 to log all training metrics and media with Weights & Biases, ensure you have the latest versions of ultralytics and wandb installed. Also, verify that your Weights & Biases account is properly authenticated. For detailed setup instructions, visit the Weights & Biases integration guide.