feat: Support usage trackers for received and discarded bytes. #11840

jeschkies · 2024-01-31T15:27:02Z

What this PR does / why we need it:
Currently, Loki exposes the distributor_bytes_received_total and discarded_bytes_total metrics. These only expose the tenant and the discarded reason in the labels. In some cases it is useful to be specific to to certain streams. E.g. one might want to know how many bytes from a certain region were received and dropped. This is where usage trackers come in.

They receive the added or discarded bytes of a stream and its labels. This can be used to build a custom tracker such as Mimir's grafana/mimir#1188 which uses regex matchers and exports the active series as a Prometheus metric.

Checklist

Reviewed the CONTRIBUTING.md guide (required)
Documentation added
Tests updated
CHANGELOG.md updated
- If the change is worth mentioning in the release notes, add add-to-release-notes label
Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
For Helm chart changes bump the Helm chart version in production/helm/loki/Chart.yaml and update production/helm/loki/CHANGELOG.md and production/helm/loki/README.md. Example PR
If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR

cstyan

We should have some mechanism for cleaning these metrics up if they haven't been updated recently, in which case they're just excess cardinality.

Also, it looks like disitrbutors processing Push requests is roughly 10% of their CPU time, it would be good to know what kind of additional CPU cost we're incurring when we have some of these custom trackers for a tenant or two.

pkg/distributor/distributor.go

pkg/ingester/limiter.go

pkg/ingester/instance.go

cstyan · 2024-02-23T00:57:43Z

pkg/distributor/distributor.go

 type labelData struct {
+	ls     labels.Labels
 	labels string
 	hash   uint64
 }


could we modify this struct to just have the ls? or do we call access the labels enough that ls.String() would be too expensive?

It's really weird. We use it to override stream.Labels. I think this is because the parsing also sorts the labels.

vlad-diachenko · 2024-02-23T10:40:36Z

pkg/distributor/validator.go

 		return fmt.Errorf(validation.LineTooLongErrorMsg, maxSize, labels, len(entry.Line))
 	}

 	if len(entry.StructuredMetadata) > 0 {
 		if !ctx.allowStructuredMetadata {
 			validation.DiscardedSamples.WithLabelValues(validation.DisallowedStructuredMetadata, ctx.userID).Inc()
 			validation.DiscardedBytes.WithLabelValues(validation.DisallowedStructuredMetadata, ctx.userID).Add(float64(len(entry.Line)))
+			if v.customStreamsTracker != nil {
+				for _, matchedLbs := range ctx.customTrackerConfig.MatchTrackers(labels) {
+					v.customStreamsTracker.DiscardedBytesAdd(ctx.userID, matchedLbs, float64(len(entry.Line)))


It would be useful to pass the reason why the bytes have been discarded

vlad-diachenko · 2024-02-23T10:54:05Z

pkg/loghttp/push/otlp.go

@@ -145,6 +145,18 @@ func otlpToLokiPushRequest(ld plog.Logs, userID string, tenantsRetention Tenants
 		labelsStr := streamLabels.String()

 		lbs := modelLabelsSetToLabelsList(streamLabels)
+
+		// Init custom streams tracking
+		trackedLabels := customTrackersConfig.MatchTrackers(lbs)


should we take into account the labels from structured metadata to track them also?

vlad-diachenko · 2024-02-23T12:24:01Z

pkg/loghttp/push/otlp.go

+
+				for i := range trackedLabels {
+					stats.logLinesBytesCustomTrackers[i].Bytes[tenantsRetention.RetentionPeriodFor(userID, lbs)] += int64(len(entry.Line))
+					stats.structuredMetadataBytesCustomTrackers[i].Bytes[tenantsRetention.RetentionPeriodFor(userID, lbs)] += int64(labelsSize(entry.StructuredMetadata) - resourceAttributesAsStructuredMetadataSize - scopeAttributesAsStructuredMetadataSize)


I believe we need to include in this metric everything that is added to structured metadata

Suggested change

stats.structuredMetadataBytesCustomTrackers[i].Bytes[tenantsRetention.RetentionPeriodFor(userID, lbs)] += int64(labelsSize(entry.StructuredMetadata) - resourceAttributesAsStructuredMetadataSize - scopeAttributesAsStructuredMetadataSize)

stats.structuredMetadataBytesCustomTrackers[i].Bytes[tenantsRetention.RetentionPeriodFor(userID, lbs)] += int64(labelsSize(entry.StructuredMetadata))

vlad-diachenko · 2024-02-23T12:29:23Z

pkg/loghttp/push/push.go

 			if err != nil {
 				return nil, nil, fmt.Errorf("couldn't parse labels: %w", err)
 			}
+		}
+		trackedLabels := customTrackers.MatchTrackers(lbs)


I would call it trackers

Suggested change

trackedLabels := customTrackers.MatchTrackers(lbs)

trackers := customTrackers.MatchTrackers(lbs)

jeschkies · 2024-02-23T15:19:19Z

@vlad-diachenko and @cstyan I've changed the code to just inject a usage tracker. This is quite generic and allows us to encapsulate the labels filtering in summing in a few simple methods.

cstyan

LGTM but I still think the additional CPU usage is worth measuring

Also, it looks like disitrbutors processing Push requests is roughly 10% of their CPU time, it would be good to know what kind of additional CPU cost we're incurring when we have some of these custom trackers for a tenant or two.

jeschkies · 2024-02-24T07:56:11Z

Thanks @cstyan. I'll measure the impact when I'm adding a concrete tracker.

vlad-diachenko

Looks great 💎

…na#11840)

jeschkies added 2 commits January 31, 2024 14:11

Support custom trackers for received bytes.

154a6d1

Implement custom trackers config

79fff6b

jeschkies requested a review from gubjanos January 31, 2024 15:27

pull-request-size bot added the size/L label Jan 31, 2024

jeschkies added 9 commits February 1, 2024 12:43

Note todos

fd1ff01

Fix test

a787cf4

Move code block

ac8ae3a

Merge remote-tracking branch 'grafana/main' into karsten/custom-trackers

dcbd58e

Start tracking discarded bytes

d37343b

Calrify todo

d5be74f

Change return type

77efea5

Comment discarded bytes

51cbaa4

generate docs

b0b4c7c

github-actions bot added the type/docs Issues related to technical documentation; the Docs Squad uses this label across many repositories label Feb 2, 2024

jeschkies added 6 commits February 2, 2024 13:09

Update docs

fe34b32

Add changelog entry

be4d2c8

Merge remote-tracking branch 'grafana/main' into karsten/custom-trackers

e15746c

Test push metrics

659a9ef

Return empty trackers instead of nil

90099ef

Test oltp push

4f0697a

jeschkies marked this pull request as ready for review February 2, 2024 15:59

jeschkies requested a review from a team as a code owner February 2, 2024 15:59

jeschkies requested review from trevorwhitney and slim-bean February 2, 2024 15:59

Merge remote-tracking branch 'grafana/main' into karsten/custom-trackers

b6b47a5

cstyan reviewed Feb 20, 2024

View reviewed changes

pkg/distributor/distributor.go Outdated Show resolved Hide resolved

jeschkies added 4 commits February 20, 2024 17:35

Merge remote-tracking branch 'grafana/main' into karsten/custom-trackers

457b002

Use label names intead of regex matchers

4c3292d

correct doc

ca417c2

Test unmarshalling

ee947b3

Report on discarded stream bytes

22e1d8e

jeschkies requested a review from cstyan February 22, 2024 15:17

Merge remote-tracking branch 'grafana/main' into karsten/custom-trackers

a9a17fa

jeschkies removed the request for review from slim-bean February 22, 2024 15:22

jeschkies added 2 commits February 22, 2024 16:25

Document custom tracker structs and methods

c564f40

Rename tracker interface and methods

fede4cf

cstyan reviewed Feb 23, 2024

View reviewed changes

pkg/ingester/limiter.go Outdated Show resolved Hide resolved

cstyan reviewed Feb 23, 2024

View reviewed changes

pkg/ingester/instance.go Outdated Show resolved Hide resolved

cstyan reviewed Feb 23, 2024

View reviewed changes

vlad-diachenko reviewed Feb 23, 2024

View reviewed changes

jeschkies changed the title ~~feat: Support custom trackers for received and discarded bytes.~~ feat: Support usage trackers for received and discarded bytes. Feb 23, 2024

pull-request-size bot added size/L and removed size/XL labels Feb 23, 2024

Remove labels filtering and configuration.

9498628

github-actions bot removed the type/docs Issues related to technical documentation; the Docs Squad uses this label across many repositories label Feb 23, 2024

jeschkies added 3 commits February 23, 2024 16:07

Rename tracker

ab4f152

Correct changelog

9df9532

Inject usage tracker

c6a7cb4

jeschkies requested review from cstyan and vlad-diachenko February 23, 2024 15:18

jeschkies added 2 commits February 23, 2024 16:20

Do not presist label string

d07ac85

Correct push test

c01afd7

cstyan approved these changes Feb 23, 2024

View reviewed changes

jeschkies merged commit 6578a00 into grafana:main Feb 24, 2024
11 checks passed

vlad-diachenko reviewed Feb 26, 2024

View reviewed changes

jeschkies deleted the karsten/custom-trackers branch February 27, 2024 09:20

loki-gh-app bot mentioned this pull request Mar 27, 2024

chore(add-major-release-workflow): release 3.0.0-rc.1 #12380

Closed

rhnasc pushed a commit to inloco/loki that referenced this pull request Apr 12, 2024

feat: Support usage trackers for received and discarded bytes. (grafa…

834072d

…na#11840)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Support usage trackers for received and discarded bytes. #11840

feat: Support usage trackers for received and discarded bytes. #11840

jeschkies commented Jan 31, 2024 •

edited

Loading

cstyan left a comment

cstyan Feb 23, 2024

jeschkies Feb 23, 2024

vlad-diachenko Feb 23, 2024

vlad-diachenko Feb 23, 2024

vlad-diachenko Feb 23, 2024

vlad-diachenko Feb 23, 2024

jeschkies commented Feb 23, 2024

cstyan left a comment

jeschkies commented Feb 24, 2024

vlad-diachenko left a comment

	stats.structuredMetadataBytesCustomTrackers[i].Bytes[tenantsRetention.RetentionPeriodFor(userID, lbs)] += int64(labelsSize(entry.StructuredMetadata) - resourceAttributesAsStructuredMetadataSize - scopeAttributesAsStructuredMetadataSize)
	stats.structuredMetadataBytesCustomTrackers[i].Bytes[tenantsRetention.RetentionPeriodFor(userID, lbs)] += int64(labelsSize(entry.StructuredMetadata))

	trackedLabels := customTrackers.MatchTrackers(lbs)
	trackers := customTrackers.MatchTrackers(lbs)

feat: Support usage trackers for received and discarded bytes. #11840

feat: Support usage trackers for received and discarded bytes. #11840

Conversation

jeschkies commented Jan 31, 2024 • edited Loading

cstyan left a comment

Choose a reason for hiding this comment

cstyan Feb 23, 2024

Choose a reason for hiding this comment

jeschkies Feb 23, 2024

Choose a reason for hiding this comment

vlad-diachenko Feb 23, 2024

Choose a reason for hiding this comment

vlad-diachenko Feb 23, 2024

Choose a reason for hiding this comment

vlad-diachenko Feb 23, 2024

Choose a reason for hiding this comment

vlad-diachenko Feb 23, 2024

Choose a reason for hiding this comment

jeschkies commented Feb 23, 2024

cstyan left a comment

Choose a reason for hiding this comment

jeschkies commented Feb 24, 2024

vlad-diachenko left a comment

Choose a reason for hiding this comment

jeschkies commented Jan 31, 2024 •

edited

Loading