Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix dual-read potential risk by using ThreadPool #945

Merged
merged 10 commits into from
Nov 21, 2023

Conversation

brycezhongqing
Copy link
Collaborator

@brycezhongqing brycezhongqing commented Nov 10, 2023

Background

During the dual_read mode for service discovery, the new loadBalance may impact the old loadBalance. As observed, when new_Lb.getProperites is executed, it can potentially block oldLb.getClient due to their synchronous progress. If new_Lb.getProperities times out, it leads to the blocking of oldLb, resulting in a timeout for the request. To address this, it is essential to incorporate a fallback mechanism.
image

Solution:

  • There we use the ExecutorService like ThreadPoolExecutor to decouple(async) new loadBalance and old loadBalance. For each new loadBalance getProperities task we will request a thread to process.
    image

There are some keypoints :
[1] We need to create a ExecutorService
[2] We need to write the fallback logic
[3] We shouldn't create a large task queue and we need to discard the task which could not be processed in time.
[4] We should add the relative metrics and log.
[5] We need promise the code change is backward compatible
[6] We need to provide the api to custimize the ExecutorService like ThreadPoolExecutor

Test

[✅]./gradlew build
[✅] test in toki(a test service)

@brycezhongqing brycezhongqing marked this pull request as ready for review November 10, 2023 01:02
@brycezhongqing brycezhongqing force-pushed the zhonchen/threadPool branch 2 times, most recently from b87968b to 648e7ff Compare November 11, 2023 21:01
@brycezhongqing brycezhongqing force-pushed the zhonchen/threadPool branch 4 times, most recently from ed33fc7 to db93223 Compare November 13, 2023 22:13
Copy link
Contributor

@bohhyang bohhyang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Please resolve the conflict with master (Paul's change is also bumping the minor version, so you'll need to bump to 29.48.0)

@brycezhongqing brycezhongqing merged commit a45e661 into master Nov 21, 2023
2 checks passed
@brycezhongqing brycezhongqing deleted the zhonchen/threadPool branch November 21, 2023 01:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants