(experimental) Allow EMA on LoRA/Lycoris networks

[bghira]

• Refactor EMAModel to handle LoRA/Lycoris type networks instead of full model
• Quantise the EMAModel at startup when the trained model is as well
• Load state dictionary in a way that is compatible with quantised EMA model
• Save state dictionary in a way that is compatible with quantised EMA model
• EMA validations overhaul
  • lycoris support
  • peft standard LoRA
    • determine a reliable method for temporarily replacing LoRA with EMA for inference
  • full model checkpoint training
  • optional side-by-side validations
  • optional EMA-only validations
• Model card details for EMA
• Save/load as safetensors file instead of pytorch raw export
• Ability to export EMA model at the end of training
• Export EMA model in a way that is compatible with typical adapter loading methods
  • Full model export
  • Standard PEFT LoRA export / loading
  • Lycoris library export / loading

[bghira]

extended model benchmarking to include EMA vs Checkpoint.

to demonstrate the concept this is LR 8e-3 over a single step where EMA updates every step.

every 100th step EMA updating at the same LR for just a single step (so EMA = base model)

[bghira]

currently PEFT Standard LoRA has EMA validations forcibly disabled during startup.

the validation class doesn't correctly load/unload the EMA weights from the model. more development is required for that.

though now, Lycoris + EMA is working really nicely.