Skip to content

Latest commit

 

History

History
74 lines (53 loc) · 5.68 KB

CONFIGURATION.md

File metadata and controls

74 lines (53 loc) · 5.68 KB
Raw

Recommended configurations

SASL and SSL settings are not included here. Please refer to SDK quickstarts.

Java client configuration properties

Producer and consumer configurations

Property Recommended Values Permitted Range Notes
metadata.max.age.ms ~ 180000 < 240000 Can be lowered to pick up metadata changes sooner.
connections.max.idle.ms 180000 < 240000 Azure closes inbound TCP idle > 240000 ms, which can result in sending on dead connections (shown as expired batches due to send timeout).

Producer configurations only

Producer configs can be found here.

Property Recommended Values Permitted Range Notes
max.request.size 1000000 < 1046528 The service will close connections if requests larger than 1046528 bytes are sent. This must be changed and will cause issues in high-throughput produce scenarios.
retries > 0 May require increasing delivery.timeout.ms value, see documentation.
request.timeout.ms 30000 .. 60000 > 20000 EH will internally default to a minimum of 20000 ms. While requests with lower timeout values are accepted, client behavior is not guaranteed.
metadata.max.idle.ms 180000 > 5000 Controls how long the producer will cache metadata for a topic that's idle. If the elapsed time since a topic was last produced to exceeds the metadata idle duration, then the topic's metadata is forgotten and the next access to it will force a metadata fetch request.
linger.ms > 0 For high throughput scenarios, linger value should be equal to the highest tolerable value to take advantage of batching.
delivery.timeout.ms Set according to the formula (request.timeout.ms + linger.ms) * retries.
compression.type none Compression currently not supported..

Consumer configurations only

Consumer configs can be found here.

Property Recommended Values Permitted Range Notes
heartbeat.interval.ms 3000 This is default and should not be changed.
session.timeout.ms 30000 6000 .. 300000 Start with 30000, increase if seeing frequent rebalancing due to missed heartbeats.
max.poll.interval.ms 300000 (default) >session.timeout.ms Used for rebalance timeout, so this should not be set too low. Must be greater than session.timeout.ms.

librdkafka configuration properties

The main librdkafka configuration file (link) contains extended descriptions for the properties below.

Producer and consumer configurations

Property Recommended Values Permitted Range Notes
socket.keepalive.enable true Necessary if connection is expected to idle. Azure will close inbound TCP idle > 240000 ms.
metadata.max.age.ms ~ 180000 < 240000 Can be lowered to pick up metadata changes sooner.

Producer configurations only

Property Recommended Values Permitted Range Notes
retries > 0 Default is 2. This is fine.
request.timeout.ms 30000 .. 60000 > 20000 EH will internally default to a minimum of 20000 ms. librdkafka default value is 5000, which can be problematic. While requests with lower timeout values are accepted, client behavior is not guaranteed.
partitioner consistent_random See librdkafka documentation consistent_random is default and best. Empty and null keys are handled ideally for most cases.
compression.codec none Compression currently not supported.

Consumer configurations only

Property Recommended Values Permitted Range Notes
heartbeat.interval.ms 3000 This is default and should not be changed.
session.timeout.ms 30000 6000 .. 300000 Start with 30000, increase if seeing frequent rebalancing due to missed heartbeats.
max.poll.interval.ms 300000 (default) >session.timeout.ms Used for rebalance timeout, so this should not be set too low. Must be greater than session.timeout.ms.

Further notes

Check the following table of common configuration-related error scenarios.

Symptoms Problem Solution
Offset commit failures due to rebalancing Your consumer is waiting too long in between calls to poll() and the service is kicking the consumer out of the group. You have several options:
  • increase session timeout
  • decrease message batch size to speed up processing
  • improve processing parallelization to avoid blocking consumer.poll()
Applying some combination of the three is likely wisest.
Network exceptions at high produce throughput Are you using Java client + default max.request.size? Your requests may be too large. See Java configs above.
Seeing frequent rebalancing becaues of frequent consumer leave group Check your client side logs, and you should find the log saying "Member [some member-id] sending LeaveGroup request to coordinator [xyz] due to consumer poll timeout has expired". This means the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time processing messages. There are serveral settings you can tweak:
  • Either increase max.poll.interval.ms (but then rebalance may take longer)
  • or speed up processing by reducing the maximum size of batches returned in poll() with max.poll.records (which may impact performance due to less batching)
  • or improve processing parallelization to avoid blocking consumer.poll() for too long
Applying some combination of the three is likely necessary to get the best balance for your scenario.