Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Get ConsumerGroups information from ConsumerGroupRegister and update tracked Lag when assigned consumer stops consuming #131

Open
wants to merge 18 commits into
base: main
Choose a base branch
from
Open

Get ConsumerGroups information from ConsumerGroupRegister and update tracked Lag when assigned consumer stops consuming #131

wants to merge 18 commits into from

Commits on Feb 19, 2024

  1. Report the actual error if failing to create a new Prometheus Metric 

    Before the error was being swallowed.
    @detro
    detro committed Feb 19, 2024
    Configuration menu
    Copy the full SHA
    5782aa4 View commit details
    Browse the repository at this point in the history

  2. Implementing Hash trait for kafka_types/groups

    @detro
    detro committed Feb 19, 2024
    Configuration menu
    Copy the full SHA
    413c373 View commit details
    Browse the repository at this point in the history

  3. New ConsumerGroupsRegister 

    This is able to provide the groups and the members of each group.
Additionally, it holds the groups in memory after they shutdown/disappear, for a configurable `forget_after` `Duration`.
    @detro
    detro committed Feb 19, 2024
    Configuration menu
    Copy the full SHA
    fb54e8a View commit details
    Browse the repository at this point in the history

  4. Add --forget-group-after command line argument, to control `Consume… 

    …rGroupsRegister` `forget_after` input
    @detro
    detro committed Feb 19, 2024
    Configuration menu
    Copy the full SHA
    0182a73 View commit details
    Browse the repository at this point in the history

  5. Replace ConsumerGroups Receiver to feed into LagRegister, with … 

    …`ConsumerGroupsRegister`
    @detro
    detro committed Feb 19, 2024
    Configuration menu
    Copy the full SHA
    317e204 View commit details
    Browse the repository at this point in the history

  6. Make LagRegister forget groups that become unknown 

    Meaning, not known to the `ConsumerGroupsRegister` anymore
    @detro
    detro committed Feb 19, 2024
    Configuration menu
    Copy the full SHA
    6f6f7c3 View commit details
    Browse the repository at this point in the history

  7. Making tests start with should

    @detro
    detro committed Feb 19, 2024
    Configuration menu
    Copy the full SHA
    53bb0fe View commit details
    Browse the repository at this point in the history

  8. Squash the chrono requirement into the implementation details of `C… 

    …onsumerGroupsRegister`
    @detro
    detro committed Feb 19, 2024
    Configuration menu
    Copy the full SHA
    cd92bad View commit details
    Browse the repository at this point in the history

  9. Ensure ConsumerGroupsRegister correctly "forgets" groups not seen f… 

    …or the specified amount of time
    @detro
    detro committed Feb 19, 2024
    Configuration menu
    Copy the full SHA
    7d4c8ae View commit details
    Browse the repository at this point in the history

  10. Provide ConsumerGroupsRegister with tests for key functionality

    @detro
    detro committed Feb 19, 2024
    Configuration menu
    Copy the full SHA
    8c589e1 View commit details
    Browse the repository at this point in the history

  11. Fix logic that makes LagRegister forget groups

    @detro
    detro committed Feb 19, 2024
    Configuration menu
    Copy the full SHA
    0e9e126 View commit details
    Browse the repository at this point in the history

  12. All tests should_*

    @detro
    detro committed Feb 19, 2024
    Configuration menu
    Copy the full SHA
    5fa7c82 View commit details
    Browse the repository at this point in the history

Commits on Feb 28, 2024

  1. Cleanup input of the std::any::type_name function

    @detro
    detro committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    632193e View commit details
    Browse the repository at this point in the history

  2. Improve partition_offsets/lag_estimator internal doc

    @detro
    detro committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    8827fef View commit details
    Browse the repository at this point in the history

  3. Ignore GroupMetadata events if the Group is empty (has no members). 

    This means that the group is gone, so we don't want to remove it from the register just yet.
    @detro
    detro committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    d22848a View commit details
    Browse the repository at this point in the history

  4. Formatting and std::any::type_name import cleanups

    @detro
    detro committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    d17a43c View commit details
    Browse the repository at this point in the history

  5. Every second LagRegister will reconcile, doing 2 things 

    1. Check if `ConsumerGroupRegister` has changed (using the hash) and, if it did, updating it's internal tracked group lags
2. Check if any known `Lag` is stale, and if it is updating the lag estimation considering the latest tracked produced offsets
    @detro
    detro committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    ac35518 View commit details
    Browse the repository at this point in the history

  6. Deps upgrade

    @detro
    detro committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    68f9590 View commit details
    Browse the repository at this point in the history