Determine what information is available from validator-info to determine node compliance

[lohanspies]

The objective is this exercise is to determine what information provided by the validator-info call can be used to measure compliance with the Sovrin Technical Policies

An example might be to determine if nodes have two Public IPs for node and client traffic respectively as per the technical policy.

Further analysis is required to determine what is already available vs what is missing. i.e. a Gap analysis, the outcome might be requested changes to the output provided by the validator-info call from indy-vdr

[lohanspies]

Node compliance info available with the current output of fetch_validator:

• Operating System Version
• Indy Version Installed
• Sovrin Version Installed
• Installed Packages
• Two unique public IPs for node and client traffic

Things that we can potentially add to fetch_validator output:

• Amount of RAM
• Amount of HDD space
• Amount of CPU cores
• NTP sync status
• Firewall status?

Another option might be to force Stewards to provide the output of the compliance scripts once a month?

Any thoughts/suggestions? @WadeBarnes @swcurran

[WadeBarnes]

I think it's best if we update the validator-info for the nodes to return the additional information we want. i.e incorporate the collection of the data analyzed by the Technical Verification Script

• NTP sync status

  • This can be determined via the timestamp in the response.
  • Entered Add system clock skew detection and alerting #27 to address.

• Network interfaces (IPs and Ports)

  • At the node summary there is the node and client IPs (as you mentioned). We want to ensure they are different.
  • In the response details the node reports it's internal node and client IP. We want to ensure they are different.
  • Entered Add IP address configuration analysis #28 to address.

• HDD Stats:

  • Used HDD space is reported. This should be updated to include full stats; Overall size, available (amount), used (amount and percent).
  • Entered Add more detailed hardware metrics to the collection and output of validator-info indy-node#1669 to address.

• Resources (Memory and CPU)

  • The CPU and memory use of the node processes is reported.
  • The response should be updated to provide more details (system) memory and CPU information.
  • Entered Add more detailed hardware metrics to the collection and output of validator-info indy-node#1669 to address.

• Firewall Status

  • This is a tricky one. The node would only be able to report it's own firewall setting. In many software and hardware defined environments the network security (firewalls) are defined outside the servers, so the servers themselves are unaware of the settings.

[WadeBarnes]

By incorporating the collection of the additional data analyzed by the Technical Verification Script, it would be easy to implement continuous compliance monitoring.

• Entered Add additional resource and system metrics to the collection and output of validator-info indy-node#1670, and Implement a health, diversity, and compliance analysis plug-in #29 to address.

[lohanspies]

@WadeBarnes it would be great if we can include the Technical Verification Script into validator_info.

Suggest we make this a privileged request the same as --status or detailed output from validator_info.

[lohanspies]

We will need to clone and clean the Technical Verification Script so that it can run without any user input and change the output format to JSON

[WadeBarnes]

@lohanspies, I recommend incorporating just the data collection elements of the Technical Verification Script into validator_info (of the node). The analysis portions would then be incorporated into the indy-node-monitor analysis (like detect_issues), and then summarized/reported in the status section of the output of indy-node-monitor. That way we're not imposing a specific network's requirements on anyone wishing to run a set of nodes.

By adding the additional details to the response from the node itself, the additional data would only be available to privileged requests as it is now.

• Entered Add additional resource and system metrics to the collection and output of validator-info indy-node#1670, and Implement a health, diversity, and compliance analysis plug-in #29 to address.

[lohanspies]

Agree with this approach. Suggest we think about anything else we might want/need to add to the Technical Verification Script before starting this work.

Anything on top of mind from your side? Maybe a list of outstanding patches would be a great addition. That can feed into the security analysis of an Indy Network.

Food for thought - how can we get some security metrics out of this as well? Thinking out loud here and not a well thought through idea at this stage.

[WadeBarnes]

Agreed, I think it would be good to discuss further when we're designing the updates for validator_info in indy-node. If we're already in there updating it to collect additional data, we may as well try to be as complete as possible.

• Covered in Add additional resource and system metrics to the collection and output of validator-info indy-node#1670.

[lohanspies]

Exactly. We can continue the discussion during the SC Health Workstream call today. Thank you for the feedback so far.

[lohanspies]

A few discussion items for the Steward Council Health Workstream call today:

Ensure that the technical policies already enable the extraction of this type of information.
ACLs on who is allowed to request, receive and store this information.

• Entered Ensure technical policies already enable the extraction of additional metrics for validator-info indy-node#1672 and Determine ACLs on who is allowed to request, receive and store additional metrics for validator-info indy-node#1673 to address.

Transparency on blockchain state: (This information is available, just not via a public dashboard)

• In/Out of Sync
• Ledger write consensus
• Unreachable nodes per node

This approach does have some benefits:

• Proactive monitoring and alerting for Stewards - for instance, Hey Steward A you have vulnerable packages - please upgrade.
• Enable Indy Network operators with an aggregated view on health/metrics/compliance and security

Maybe a good idea is to present the approach to Stewards to get their input as well.
Seems like there is still a need to help Stewards with detailed node monitoring.

@WadeBarnes update README to clearly indicate what analysis is available right now in indy-node-monitor; #25

We will continue the discussion before actively starting to work on these proposed changes.

[WadeBarnes]

Summary of tickets created based on the conversations here:

• Update Readme to indicate what analysis is performed by the scripts and how it's reported
• Add system clock skew detection and alerting
• Add IP address configuration analysis
• Add more detailed hardware metrics to the collection and output of validator-info
• Add additional resource and system metrics to the collection and output of validator-info
  • Implement a health, diversity, and compliance analysis plug-in
• Ensure technical policies already enable the extraction of additional metrics for validator-info
• Determine ACLs on who is allowed to request, receive and store additional metrics for validator-info

[WadeBarnes]

@lohanspies, Please review the summarized tickets to see if they adequately address what's been discussed.

[WadeBarnes]

@lohanspies, For the following tickets, who is best equipped to start this investigation, and where is the relevant resource material?

• Ensure technical policies already enable the extraction of additional metrics for validator-info
• Determine ACLs on who is allowed to request, receive and store additional metrics for validator-info

[lohanspies]

@WadeBarnes reviewed the tickets and made comments on some of them.
A key thing we might want to consider is to include information that can inform the node selection algorithm. Not sure if it belongs here but would be great to have a discussion with you on the topic.

[WadeBarnes]

@WadeBarnes reviewed the tickets and made comments on some of them.
A key thing we might want to consider is to include information that can inform the node selection algorithm. Not sure if it belongs here but would be great to have a discussion with you on the topic.

@lohanspies, It would be best to treat that as a separate topic.

[WadeBarnes]

@lohanspies, Once we feel we've addressed the topics sufficiently in the spawned tickets I'd like to close this one as complete.

[lohanspies]

@WadeBarnes sure, can we please discuss this in the SC Health call tomorrow and ensure we are aligned on all tickets?

[WadeBarnes]

Absolutely

[WadeBarnes]

Reviewed on the Health Workstream Call and agreed the tickets cover the discussed items.