Automatically detect deleted resources #1386
Conversation
rafaelweingartner
force-pushed
the
mark-resource-as-deleted-when-they-stop-receiving-measures
branch
2 times, most recently
from
67d710e
to
9d08cc0
Compare
|
Thanks @pedro-martins ! |
While executing some Gnocchi optimizations (gnocchixyz#1307), we noticed that some deleted/removed resources do not have the "ended_at" field with a datetime. This can cause slowness with time, as more and more "zombie" resources are left there, and this has a direct impact in the MySQL queries executed with the aggregates API. This patch introduces a new parameter called `metric_inactive_after`, which defines for how long a metric can go without receiving new datapoints until we consider it as inactive. Then, when all metrics of a resource are in inactive state, we can mark/consider the resource as removed.
rafaelweingartner
force-pushed
the
mark-resource-as-deleted-when-they-stop-receiving-measures
branch
from
7764d2c
to
6e76708
Compare
|
@pedro-martins and @chungg I have rebased this PR and it is ready for your reviews |
|
Thanks @pedro-martins for your review! |
Co-authored-by: gord chung <5091603+chungg@users.noreply.github.com>
Co-authored-by: gord chung <5091603+chungg@users.noreply.github.com>
Co-authored-by: gord chung <5091603+chungg@users.noreply.github.com>
|
@chungg thanks for your review! I have added the code changes you suggested, and there are some remarks open, which I would like to hear your response before closing them. |
|
Hello @chungg is there something else that needs to be addressed here? |
|
Hello guys, This patch is the base for some further optimizations that we are doing and that we would like to propose upstream. |
There was a problem hiding this comment.
i'm ok with this although i think it'd be better if the update logic was in sql rather than python. something like following but in orm. could also be done later.
update resource r set ended_at = %s where not exists (select 1 from metric m where m.resource_id=a.id AND m.last_measure_timestamp >= %s) and r.ended_at is null;
| if resource.ended_at is not None: | ||
| LOG.info("Resource [%s] was marked with a timestamp for the " | ||
| "'ended_at' field. However, it received a " | ||
| "measurement for metric [%s]. Therefore, we undelete " |
There was a problem hiding this comment.
| "measurement for metric [%s]. Therefore, we undelete " | |
| "measurement for metric [%s]. Therefore, restoring " |
| op.add_column( | ||
| "metric", sqlalchemy.Column( | ||
| "last_measure_timestamp", sqlalchemy.DateTime, | ||
| nullable=False, server_default=func.current_timestamp())) |
There was a problem hiding this comment.
i don't know if it makes sense to make this nullable? it's possible that this is disabled at first and gets a default value and is enabled later and immediately all of the resources can potentially get set to ended (until next update).
There was a problem hiding this comment.
actually nevermind. i just realised last_measure_timestamp is updated regardless if auto end logic is enabled or not.
This patch is created on top of #1385, #1381, #1387, and #1388. Therefore, we need to merge them first. Then, the extra commits here will disappear.
While executing some Gnocchi optimizations (#1307), we noticed that some deleted/removed resources do not have the "ended_at" field with a datetime. This can cause slowness with time, as more and more "zombie" resources are left there, and this has a direct impact in the MySQL queries executed with the aggregates API.
This patch introduces a new parameter called
metric_inactive_after, which defines for how long a metric can go without receiving new data points until we consider it as inactive. Then, when all metrics of a resource are in inactive state, we can mark/consider the resource as removed.