PriorityClass follow-ups

[olim7t]

We added support for priorityClassName in #1034.
There were additional suggestions in this comment:

That is, we don't actually check if the PriorityClass exists. Right now that could cause the Pod to have a priorityClassName that points to nothing and this probably prevents the pod from being scheduled correctly. I think this check could be implemented in cass-operator also, I have no preference to that right now.

Second is actually creating PriorityClasses in k8ssandra-operator, but I would have to think about this feature more. Most users should probably set something like preemptionPolicy to Never to prevent new Cassandra pods from killing existing pods. Right now, that would be up to the user to understand what they're doing (like the entire feature, which I think we could consider "experienced users only").

If we use this feature in downstream products, then tracking PodStatus' nominatedNodeName might make sense also. And our PodDisruptionBudget policy is probably not very compatible with this feature either. The scheduling is quite complex on some of these and the documentation is a bit hand waving in some cases "we might or might not do this and maybe something else".

┆Issue is synchronized with this Jira Story by Unito
┆Issue Number: K8OP-301