Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add service/cluster-not-found count to simple load balancer jmx. And add entry-out-of-sync count to dual read monitoring. #936

Merged
merged 1 commit into from
Sep 27, 2023

Conversation

bohhyang
Copy link
Contributor

Background

We had issues where Observer accidentally deleted all cluster properties, but we weren't able to see that the client is failing to get clusters on any ingraph metric but only via digging into logs.
For dual read, we've been only monitoring when the data is mismatched, but not when data is received on side (like ZK) but not on the other side (like xDS).

Changes

  1. Added service and cluster not found counter to simple load balancer jmx, which gets incremented when timeout happens at fetching the resource.
  2. Added entry out-of-sync count to dual read monitoring, which gets incremented when data is either mis-match or received on one side but not on the other (until a new version is received on the first side), and gets decremented when the data matches.

…add entry-out-of-sync count to dual read monitoring.
Copy link
Contributor

@shivamgupta1 shivamgupta1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM otherwise

@bohhyang bohhyang merged commit 4e8dd72 into master Sep 27, 2023
2 checks passed
@bohhyang bohhyang deleted the bohan/monitorPropertyNotFound branch September 27, 2023 01:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants