Error in building llama with eagle for speculative decoding

[JoJoLev]

Went through the Readme for llama 3 8b instruct on the examples/eagle. When running the convert checkpoint, I get the error: key error 'fc.bias'

I used llama3 8b instruct from hugging face with the eagle version for the eagle model directory.

Not really sure the issue? Maybe I need to recreate the eagle model?

[nv-guomingz]

Hi @JoJoLev Sorry, examples/eagle doesn't mention llama3 8b. Do we miss something?

[JoJoLev]

@nv-guomingz hello there, thanks for the quick reply! 😀

No, I don't see anything specifically calling out models? I was under the impression that those models built and in the HF repo for Eagle should work here for a quick demo/check.
Per the docs:

Limitations

EAGLE-2 is not supported.
All EAGLE choices have to have exactly the same depth as num_eagle_layers of the engine.
Pipeline parallelism is not supported.

I didn't see any other limitations listed.

Here is the HF model I listed for the eagle directory: https://huggingface.co/yuhuili/EAGLE-LLaMA3-Instruct-8B

I then added the base llama3 instruction as the model directory.

Is there a place where specific model support is called out?

[nv-guomingz]

Hi @JoJoLev vicuna-7b-v1.3 is a verified example model for Eagle feature, I think the team didn't verify the models u mentioned above.
So the failure is very possible but I consider it as a corner case.
Let's have a try and get back to you ASAP.

[nv-guomingz]

Hi @nekorobov would u please take a look this issue?