-
Notifications
You must be signed in to change notification settings - Fork 3.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Support split align and caching for instant metric query results #11814
Conversation
Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>
Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>
Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>
Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>
Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>
Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>
bb83a3a
to
57d77a9
Compare
Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>
Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>
Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>
Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>
Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>
Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>
Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>
Fix test cases that failed with this changes Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>
Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>
Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>
1. Update both start and end when removing offset 2. Unify subqueries generation in splitalign method Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>
Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>
Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>
Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>
Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>
Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>
Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Love the detailed description and lots of tests!
Added a few comments, mostly nits. Good job Kavi!
I need to do another pass to review the tests, rest lgtm. Nice one @kavirajk ❤️ |
Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>
Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>
Caching and split align in action Do instant query for
How the query is split and cache reqs and cache hits (these logs lines are from Quey-frontend (saying 4 requests were actually made for cache, and none of those got hit)
How the above query got split and run
Do the instant query for
How the query is split and cache reqs and cache hits (these logs lines are from Quey-frontend (saying 4 requests were actually made for cache, 2 of those got hit)
How the above query got split and run (only two queries that miss cache hit, got run this time)
|
Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>
Use right split duration (new InstantSplitDuration) for instant queries Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>
grafana#11814) Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>
What this PR does / why we need it:
Follow up to metadata results caching, this PR adds support for instant metric queries. It also adds support to enable aligning the subquery for more reusability.
Config changes:
cache_instant_metric_results
- Enable/disable (default disable) - Booleaninstant_metric_results_cache
- CacheConfig to tweak (usually not needed, have sane defaults)instant_metric_query_split_align
- Enable/disable (default disable) - BooleanHow it works (without split align)
Consider following query
Query:
sum(rate({foo="bar"}[3h]))
@12:34:00
SplitInterval:
1h
So we need results from
09:34:00
to12:34:00
. (3h total)Currently, After range mapper, it splits into
Even if we remove the offset it turns into
But the problem is now eval time is not aligned. And it's mostly unlikely these subqueries are reused.
How it works (with split align)
Now consider the same exact query
Query:
sum(rate({foo="bar"}[3h]))
@12:34:00
SplitInterval:
1h
After range mapper, it splits into
And after removing the offset it tuns into (properly eval time aligned)
Now we have (2) and (3) subqueries properly aligned and highly likely be reused.
Which issue(s) this PR fixes:
Fixes #
Special notes for your reviewer:
Will have some follow up PRs (tried doing it in single PR, but turns out really hard to review and big change) to
stats.proto
)Checklist
CONTRIBUTING.md
guide (required)CHANGELOG.md
updatedadd-to-release-notes
labeldocs/sources/setup/upgrade/_index.md
production/helm/loki/Chart.yaml
and updateproduction/helm/loki/CHANGELOG.md
andproduction/helm/loki/README.md
. Example PRdeprecated-config.yaml
anddeleted-config.yaml
files respectively in thetools/deprecated-config-checker
directory. Example PR