Better handling of stream retries on fast disconnects #1006
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Copied from the Slack context (thanks Bohan for the writeup): Summary: we looked into the bad host's log and our xds client code. There is an issue in the reconnect logic. When a reconnect ads stream is closed/rejected for stream limit on an observer A, the stream ready timeout for this stream was not cancelled before it retries with another observer B. So that after the stream with observer B becomes ready, the ready timeout for observer A was executed which cleaned the stream with observer B. This leads to the client enables backup (FS) store and never tries to connect with the observer again. ``` 2024/06/20 10:07:00.204 INFO [XdsClientImpl] [Indis xDS client executor-4-1] [followfeed-query] [AAYbT3nOcB56sIf0OjkOIg==] ADS stream started, connected to server: main.indis-registry-observer.prod-lor1.atd.disco.linkedin.com:32123 2024/06/20 10:07:00.367 ERROR [XdsClientImpl] [Indis xDS client executor-4-1] [followfeed-query] [AAYbT3nQ9CeWMXpo8cQEFg==] ADS stream closed with status UNAVAILABLE: Network closed for unknown reason. Cause: null 2024/06/20 10:07:00.367 INFO [XdsClientImpl] [Indis xDS client executor-4-1] [followfeed-query] [AAYbT3nQ9CeWMXpo8cQEFg==] Retry ADS stream in 859991762 ns 2024/06/20 10:07:01.228 INFO [XdsClientImpl] [Indis xDS client executor-4-1] [followfeed-query] [AAYbT3neFIsVPcAaz2Yv3w==] ADS stream started, connected to server: main.indis-registry-observer.prod-lor1.atd.disco.linkedin.com:32123 2024/06/20 10:07:01.240 INFO [XdsClientImpl] [Indis xDS client executor-4-1] [followfeed-query] [AAYbT3neRq29/jeDmOPBqw==] Successfully established stream with ADS server: lor1-app104284.prod.linkedin.com 2024/06/20 10:07:01.240 INFO [XdsClientImpl] [Indis xDS client executor-4-1] [followfeed-query] [AAYbT3neRe10FiykxUBBow==] ADS stream ready, cancelled timeout task: true 2024/06/20 10:07:01.240 INFO [XdsLoadBalancer] [Indis xDS client executor-4-1] [followfeed-query] [AAYbT3neRe10FiykxUBBow==] Enabled primary stores 2024/06/20 10:07:02.202 WARN [XdsClientImpl] [Indis xDS client executor-4-1] [followfeed-query] [AAYbT3ns9W3jJPwtvqLIlw==] ADS stream not ready within 2000 milliseconds 2024/06/20 10:07:02.202 INFO [XdsLoadBalancer] [Indis xDS client executor-4-1] [followfeed-query] [AAYbT3ns9W3jJPwtvqLIlw==] Enabled backup stores ```
bohhyang
approved these changes
Jun 21, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Copied from the Slack context (thanks Bohan for the writeup):
Summary: we looked into the bad host's log and our xds client code. There is an issue in the reconnect logic. When a reconnect ads stream is closed/rejected for stream limit on an observer A, the stream ready timeout for this stream was not cancelled before it retries with another observer B. So that after the stream with observer B becomes ready, the ready timeout for observer A was executed which cleaned the stream with observer B. This leads to the client enables backup (FS) store and never tries to connect with the observer again.