[exporter/elasticsearch] Limit bulk request size to avoid 413 Entity Too Large

[carsonip]

Description

Limit the bulk request size to roughly flush::bytes for sync bulk indexer.
Sync bulk indexer is used when batcher::enabled is either true or false. In order words, sync bulk indexer is not used when batcher config is undefined.
Change flush::bytes to always measure in uncompressed bytes.
Change default batcher::max_size_items to 0 as bulk request size limit is now more effectively enforced by flush::bytes.

Link to tracking issue

Fixes #36163

Testing

Modified BenchmarkExporter to run with {name: "xxlarge_batch", batchSize: 1000000}, and removed batcher::max_size_items and added a log line for compressed and uncompressed buffer size to reproduce the error.

logger.go:146: 2024-11-19T17:16:40.060Z	ERROR	Flush	{"s.bi.Len": 10382932, "s.bi.UncompressedLen": 532777786}
    logger.go:146: 2024-11-19T17:16:40.312Z	ERROR	bulk indexer flush error	{"error": "flush failed (413): [413 Request Entity Too Large] "}

With this PR, every flush logs and there is no error.

   logger.go:146: 2024-11-19T17:23:52.574Z	ERROR	Flush	{"s.bi.Len": 99148, "s.bi.UncompressedLen": 5000007}

Documentation

[carsonip]

Also benched with ~5MB vs ~1MB vs 95MB flush::bytes to an Elastic Cloud 64GB Elasticsearch, with num_workers=1:

5MB:

BenchmarkExporter/logs/otel/xxlarge_batch-16         	       1	99909472632 ns/op	     10009 events/s	13099905120 B/op	68449712 allocs/op

1MB:

BenchmarkExporter/logs/otel/xxlarge_batch-16         	       1	108315234948 ns/op	      9232 events/s	13138234072 B/op	68519404 allocs/op

95MB:

BenchmarkExporter/logs/otel/xxlarge_batch-16         	       1	97584554530 ns/op	     10248 events/s	13092218512 B/op	68444255 allocs/op

[lahsivjar]

Do we really need to change the default max size items? Other changes LGTM!

[lahsivjar]

LGTM!

[mauri870]

LGTM, thanks!

[ChrsMark]

lgtm

[ChrsMark]

nit: maybe consider moving these defaults to some constant vars at some point. (not for this PR)