A small config file to override the GGUF metadata embedded to a model?

[Nexesenex]

Could it be possible to have an option to use a small config file to override the GGUF metadata embedded into a model, so another quantization, or even conversion in FP16 or the like, is not necessary just to change a few characters in metadata when a parameter was wrong in the config.json, or simply to change easily the number of experts used in a MOE, for example when using a GUI not supporting the command --override-kv?

[CISC]

If the GGUF is hosted on Hugging Face your easiest option is to use my GGUF Editor. :)

However it might make sense to add more options to gguf_new_metadata.py as well, I used to temporarily add some whenever I needed to modify a GGUF before. Then again cluttering the command line options too much isn't a great idea, and a config file (other than perhaps accepting the usual transformers ones) is tricky to ensure end-users use properly.

[Nexesenex]

Hey @CISC, thanks for answering me, and for your superb tool on HF, which I checked yesterday.

Locally, when using Llama.cpp, most can be done already with KV override, but the real problem is with third party tools based on LCPP not supporting the command.

The config file would be optional, editable like a config.json (and why not simply the config.json themselves?), so the end users would not see a change, except those willing to use it, and indeed based on the usual transformers parameters (and evolving with the transformers version), the same used to make the initial quant in FP16 or Q8_0 with the converting script, this in order to quickly change properly/conveniently a hardcoded parameter.

The number of experts, the positional embedding (which are sometimes badly set), the rope when no autorope is present in the tool (including Yarn, usually badly supported by third party tools), another tokenizer or tokenizer config when it's updated by the author of the model, for example, this without a lengthy process of edit, copy, or redownload on HF.

Ideally, the llama.cpp engine would check the existence of that file, named like the model but with another extension (.ggufc or simply the config .json named like the model in modelname.config.json or like the first split file of a split model), grab only the valid parameters (syntactically) within it, without even a command line needed, or one at max to disable the feature, once again to bypass third party tools not supporting that.

The idea is to save time, compute, SSD writes and bandwidth, flexibility for both power users and third party tools users, and enjoy the best quality a model can afford. I think it'd be more than worth it for many users.

[slaren]

Seems reasonable to me. A json parser is not likely to be added to the llama.cpp core library, but the model loader could check a file such as <model name.gguf>.overrides where each line is an override in the same format as accepted by --override-kv.

[Nexesenex]

@slaren: It would be absolutely great! Such implementation via an .override file would allow the same flexibility as envisioned at a minimal effort.