Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Persistent audit log around cluster operations #135999

Open
benbardin opened this issue Nov 22, 2024 · 2 comments
Open

Persistent audit log around cluster operations #135999

benbardin opened this issue Nov 22, 2024 · 2 comments
Labels
C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) O-postmortem Originated from a Postmortem action item. T-observability

Comments

@benbardin
Copy link
Collaborator

benbardin commented Nov 22, 2024

In reviewing a recent customer incident, we realized we were dependent on the customer's logging of key operational events like node startup, shutdown, and upgrade. Future investigations would be aided by a CRDB-owned persistent log of these operations across the cluster. Ideally, this log would store:

  • Node ID
  • Timestamp
  • Event type (startup/cluster join/drain/shutdown)
  • SHA/version

Ideally, this log would never wrap - or if needed, would wrap on a Very Long timescale.
Also ideally, this log would be kept on every node for every node, so the record could include decommissioned nodes.

(Full context here.) Thank you!

cc @arulajmani @ajstorm @nicktrav

Epic CRDB-42978

@benbardin benbardin added C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) O-postmortem Originated from a Postmortem action item. T-observability labels Nov 22, 2024
@benbardin
Copy link
Collaborator Author

Oh, I see much of this information is in system.eventlog.txt! That's great. So the request here would be to expand that log slightly, and verify its wrapping behavior.

@benbardin
Copy link
Collaborator Author

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) O-postmortem Originated from a Postmortem action item. T-observability
Projects
None yet
Development

No branches or pull requests

1 participant