-
Notifications
You must be signed in to change notification settings - Fork 547
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Handle HTTP/2 stream errors in tracker client and LB state updator #1000
Conversation
d2/src/test/java/com/linkedin/d2/balancer/strategies/relative/StateUpdaterTest.java
Outdated
Show resolved
Hide resolved
d2/src/main/java/com/linkedin/d2/balancer/strategies/relative/StateUpdater.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
DegraderLoadBalancerStrategyV3 doesn't use StateUpdater. It uses DegraderTrackerClientUpdater to update the client state, which uses DegraderControl which then uses DegraderImpl.
Please adjust the logic there (put behind the config as suggested above) and you may add a UT for DegraderImpl to verify the isHigh result, which is critical to increase computed drop rate (hence quarantine).
d2/src/main/java/com/linkedin/d2/balancer/clients/TrackerClientImpl.java
Outdated
Show resolved
Hide resolved
boolean loadBalanceStreamException) | ||
{ | ||
return createTrackerClient(uri, uriProperties, serviceProperties, loadBalancerStrategyName, transportClient, | ||
SystemClock.instance(), loadBalanceStreamException); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we just propagate D2ClientConfig
everywhere to avoid having to do this each time a change needs to be made to tracker client, state updater, degrader, etc?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It will be weird for inner components like DegraderImpl.Config, DegraderTrackerClientImpl, etc. to hold a D2ClientConfig, which is the upper/larger layer that's supposed to control everything in d2 client.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In that case, can a new config object be created for the LB components such as tracker clients, state updater, etc.? It is essential for new code to be open for extensions.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
adding a StateUpdater.LoadBalanceConfig makes sense. But adding one to TrackerClientFactory will be hard since the createTrackerClient method is static. We could change that altogether for sure.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added TODOs for optimizing config management later.
Since we don't have bandwidth to address this now, and this PR is for an Incident action item, it doesn't worth holding off the merge for this optimization now.
The way the config (in D2ClientConfig) is passed in is:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
aligned that we'll leave the config optimization as a TODO for future. Merging this in as an urgent incident action item.
Summary
H/2 stream errors were seen in recent incidents and they aren't handled by the Tracker client currently. Handle
Http2Exception.StreamException
to fix this.Testing Done
Added UTs