new check: check_reboot_slurm

[martbhell]

Hi!

Another custom script we've been using for a while and it's been working quite nicely.

Tested with:

• Scientific Linux 6.7 and slurm 2.6.7
• CentOS7 and slurm 15.08.04

What it does:

• set a slurm node to drained and reason=reboot and nhc will:
  • reboot the node when it is drained
  • set it to idle when it's back online

[kcgthb]

Sounds useful, but unless I'm missing something, which is very possible, you can do pretty much the same thing natively in Slurm, with scontrol reboot_nodes [nodelist] and ReturnToService=2

[martbhell]

Nice @kcgthb I didn't know about that. One benefit of letting NHC do this is that the node must pass all the health checks before it's put online. Using NHC also adds a bit of delay while waiting for nhc to run.

[mej]

As I mentioned yesterday during my talk at the HPCAC Stanford Conference, I have had an item for "rolling reboots" on my "TODO list" for some time now, having first discussed it with someone from Compute Canada at MoabCon back in 2012 or so. I don't want to limit it to use with SLURM, so clearly it follows that the name would have to change, but I definitely want to implement something like this.

I'll provide some additional feedback once I have a chance to really dig into your patch! Thanks as always for the submission! :-)

[mej]

Due to the delay caused by my changing jobs, I'm having to bump the merging of new checks to the 1.4.4 release to ensure that they have ample baking time without delaying the 1.4.3 release any further. Hope you understand! This is still very much in plan for merging!