add validator nvidia driver whether install success

[lengrongfu]

Fixes: #136

Add a driver-validator Daemonset workload, in the node to valid Nvidia driver when the install is a success. if install success, add node label hami.io/driver-validator="true" else hami.io/driver-validator="false", device-plugin Daemonset add a nodeSelect hami.io/driver-validator="true".

Test Result

• normal running

• reboot node
• reboot success
  • step 1: current not NVIDIA driver, so device-plugin pod not exist.

• step 2: wait for some time, device-plugin start success

[archlitchi]

In this version, we need to manually label GPU-node(label:"gpu=on") during installation, so we can safely assume that each node with label "gpu=on" is "gpu-ready".
However, in large cluster, manually label by cluster maintainers will be pain sometimes, so an auto-label strategy is needed.
I think it's better to rename "gpu=on" to "hami.io/nvidia-selector", and use .Values.devicePlugin.validator.enabled to switch between "manual-mode" and "auto-mode"(default manual mode)
Back to this PR, i suggest:

1. rename label "hami.io/driver-validator" -> "hami.io/nvidia-selector"
2. modify ".values.devicePlugin.nvidianodeSelector" entry ("gpu=on" -> "hami.io/nvidia-selector")
3. modify documents related to "gpu=on" ("gpu=on" -> "hami.io/nvidia-selector")
4. Remove field " {{- if .Values.devicePlugin.validator.enabled }}
   hami.io/driver-validator: "true"
   {{- end }}" in daemonset.yaml, since it is included in ".values.devicePlugin.nvidianodeSelector"

[lengrongfu]

Thanks for your suggestion, I have a different opinion on one point "and use .Values.devicePlugin.validator.enabled to switch between "manual-mode" and "auto-mode"(default manual mode)",

we can add .Values.devicePlugin.autoValidatorDriver this field to imple "manual-mode" and "auto-mode" , when value is true expression "auto-mode" else value is false expression "manual-mode".

[archlitchi]

Thanks for your suggestion, I have a different opinion on one point "and use .Values.devicePlugin.validator.enabled to switch between "manual-mode" and "auto-mode"(default manual mode)",

we can add .Values.devicePlugin.autoValidatorDriver this field to imple "manual-mode" and "auto-mode" , when value is true expression "auto-mode" else value is false expression "manual-mode".

agreed:), that looks better

[archlitchi]

please modify "Label your nodes" section in README.md and README_CN.md, introduce your manual and auto label strategy here

[chaunceyjiang]

It's just a question, I think this PR only introduces a small feature, but it requires a new Daemonset to implement this feature.

[CoderTH]

Can you add some test screenshots?

[lengrongfu]

Can you add some test screenshots?

Added part of the testing info.

[chaunceyjiang]

Additionally, can we start introducing keywords like HAMi and gradually phase out 4pd?

[lengrongfu]

Additionally, can we start introducing keywords like HAMi and gradually phase out 4pd?

They need to be modified together, and there are still many.

[chaunceyjiang]

I still think this PR is overly complicated. Please give me some time, I will research whether there are other implementation solutions.

[lengrongfu]

I still think this PR is overly complicated. Please give me some time, I will research whether there are other implementation solutions.

@chaunceyjiang Hi, any progress on this? Or we can merge PR first and then optimize later.

[chaunceyjiang]

/hold

In the next few days, I will continue to explore this issue.

[chaunceyjiang]

I believe we can use this initContainer .