Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for CMS Monitoring VictoriaMetrics back-end #148

Open
wants to merge 5 commits into
base: master
Choose a base branch
from

Conversation

vkuznet
Copy link
Contributor

@vkuznet vkuznet commented Dec 15, 2019

I added support to send data to VictoriaMetrics, similar to ES, AMQ.

@vkuznet
Copy link
Contributor Author

vkuznet commented Dec 16, 2019

Even though I put http://cms-monitoring.cern.ch:30422 as default VM URL, for testing we should use port 30423, i.e. --vm_url=http://cms-monitoring.cern.ch:30423 option.

Copy link
Collaborator

@cronosnull cronosnull left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a missing import for htcondor_es.vm both in the queues.py and the history.py files. Also, the current approach is not working, it fails with "too many values to unpack" error.
The victoriametrics API expects to receive a message in a specific format, so the one created in the convert_to_json will not work.
I'm not sure if store the full documents in victoria metrics is a good approach. Instead, selecting a set of tags and metrics can work better. I think the use case for victoria metrics is closer to Influxdb than to ES, so a similar strategy should be used.

@vkuznet
Copy link
Contributor Author

vkuznet commented Jan 17, 2020

Christian, I adjusted code wrt your comments. Please review again. I made it configurable that we can pass which attributes we'd like to extract from classAds doc to send to VM. My understanding that classAds is a dict with simple key:value structure. Therefore code will allow to pass list of attributes we'd like to keep and store in VM.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants