Ultralytics
integrations/mlflow/ #13161
giscus[bot]
bot
Replies: 4 comments 19 replies
-
|
how to set mlflow to true using CLI? |
Beta Was this translation helpful? Give feedback.
-
|
even tho --backend-store-uri runs the logged experiments correctly, the "runs/mlflow.db" database folder has empty experiments and runs, this means it an Ultralytics logging issue not an mlflow issue, please advise |
Beta Was this translation helpful? Give feedback.
-
|
@pderranger, how do I get yolov8 to automatically log to sqlite database? |
Beta Was this translation helpful? Give feedback.
-
|
Hello @MoAbbasid, For logging validation runs in MLflow, you can manually log the metrics after each validation epoch. You can modify the validation loop to include MLflow logging statements similar to how you might have done it for training. Here's a basic example: import mlflow
from ultralytics import YOLO
model = YOLO('yolov8n-seg.pt')
results = model.val(data='coco128.yaml')
# Assuming 'results' contains metrics like validation loss
mlflow.log_metrics({'val_loss': results.metrics['val_loss']})Regarding your issue with setting up a SQLite database for MLflow, ensure that your MLflow server is correctly configured to use the SQLite backend. You might need to specify the SQLite URI when starting the MLflow server: mlflow server --backend-store-uri sqlite:///mlflow.db --default-artifact-root ./artifactsMake sure the path to the SQLite database ( If you need further assistance, feel free to ask! |
Beta Was this translation helpful? Give feedback.
-
|
@pderranger, I initiated the server using your code: "mlflow server --backend-store-uri sqlite:///mlflow.db --default-artifact-root ./artifacts", and I have set mlflow=True in yolo settings but after I train the results are still logged to directory runs/mlflow and not the sqlite database, |
Beta Was this translation helpful? Give feedback.
-
|
Hello @MoAbbasid, It seems like there might be a misunderstanding in setting the MLflow tracking URI. The
If you continue to face issues, please ensure that the path to the SQLite database is correct and accessible, and that there are no permission issues. Also, verify that the MLflow version you are using supports the features and configurations you are trying to use. Let me know if this helps or if you need further assistance! |
Beta Was this translation helpful? Give feedback.
-
|
@pderrenger Ive set the tracking uri correctly to point to the sqlite database but the db itself created after the Yolo training run is empty and has experiments or run, this is automatically generated by Ultralytics right? so this is an issue on Ultralytics side? |
Beta Was this translation helpful? Give feedback.
-
|
Hi @MoAbbasid, Thanks for reaching out! The creation and management of the SQLite database for MLflow is generally handled by MLflow itself rather than directly by Ultralytics. If the database is empty after a training run, it's possible that the MLflow tracking URI might not be set correctly, or the logging calls are not being executed as expected. Here are a few things you can check:
If these steps don't resolve the issue, it might be helpful to check the MLflow logs for any errors or warnings that could provide more insight. Let us know how it goes or if you need further assistance! |
Beta Was this translation helpful? Give feedback.
-
|
Hey I'm facing a strange situation between yolov8 and mlflow. import pandas as pd
from ultralytics import YOLO, settings
import os
import json
import mlflow
settings.reset()
settings.update({
'mlflow': True,
'clearml': False,
'comet': False,
'dvc': False,
'hub': False,
'neptune': False,
'raytune': False,
'tensorboard': False,
'wandb': False
})
mlflow.set_experiment("Teste 01")
mlflow.set_tracking_uri("postgresql://user:senha@localhost:5432/postgres")
model_name = "yolov8n-cls.pt"
model = YOLO(model_name)
epochs = 3
image_size = 128
with mlflow.start_run(run_name=f'{model_name}-{epochs}-{image_size}'):
model.train(name=f'{model_name}-{epochs}-{image_size}', data='dataset', epochs=epochs, imgsz=image_size, hsv_h=1, hsv_s=1, hsv_v=1)The reason I maintained some mlflow configs out of with statement is because I would like to preserve all parameters, metrics and artifacts generated automatically by yolo-mlflow settings. And the error I found was during the db creation: the experiment_id generated by yolo is bigger than serial4 type, which is the type used as postgres-mlflow default (a bigint type would solve I suppose). So, due to that, I'm facing the following error: I was wondering if I could just define the experiment_id as a serial4, maintaining it in MLFlow patterns (numerical order e.g 1, 2, 3, 4...). I thought about fix it manually by changing the type directly on my database, but I believe that there's a smarter way to fix it. Does anyone know how to solve it? |
Beta Was this translation helpful? Give feedback.
-
|
@vini-c-costa it seems the issue is related to the experiment_id exceeding the integer range in PostgreSQL. You might consider altering the database schema to use a |
Beta Was this translation helpful? Give feedback.
-
|
For multiple GPU training a YOLO model, with autolog enabled and also I manually logged a few items by using mlflow.log_xxx() and I also logged the customized PyFunc model, it turns out to be multiple runs are generated per training. I want to only have 1 run and it better have all the logged artifacts.
Thanks |
Beta Was this translation helpful? Give feedback.
-
|
my code example: |
Beta Was this translation helpful? Give feedback.
-
|
Also, is it possible to enable multi-GPU training to use the project folder on Databricks Unity Catalog Volume directory? e.g.,
|
Beta Was this translation helpful? Give feedback.
-
|
To achieve a single MLflow run during multi-GPU training, ensure manual logging is executed only once, typically on the main GPU (rank 0). For project directories, you can specify the Unity Catalog Volume directory by setting the |
Beta Was this translation helpful? Give feedback.
-
integrations/mlflow/
Uncover the utility of MLflow for effective experiment logging in your Ultralytics YOLO projects.
https://docs.ultralytics.com/integrations/mlflow/
Beta Was this translation helpful? Give feedback.
All reactions