Lower reduction factor

[King4819]

Awesome work! I want to ask that how does the method work at lower reduction factor. In your paper, the lowest reduction factor is 0.45, is the method still work at lower reduction factor like 0.1 ? Hopes to get your reply, thanks !

[Mxbonn]

Awesome work! I want to ask that how does the method work at lower reduction factor. In your paper, the lowest reduction factor is 0.45, is the method still work at lower reduction factor like 0.1 ? Hopes to get your reply, thanks !

Once you go below 0.4 the accuracy really start to degrade quickly. If your model is still too large or slow at this reduction factor it is better to look for a smaller baseline and apply some moderate token merging/pruning there.

[King4819]

@Mxbonn Thanks for your reply. Despite the accuracy, I'm wonder that can the model learn to reduce its size to extreme low reduction factor (e.g., 0.1), since I encounter the problem that when I set reduction factor to 0.1, the result model is still big like its reduction factor is 0.4. I'm very curious about this problem, Hopes to get your reply, thanks !

[Mxbonn]

It should be possible to reduce to such low reduction factors, but I personally have never tried it. The reduction factor is achieved through the loss function. However the loss function is a combination of the classification loss and the FLOPs reduction factor loss. You can try to set increase the weight of the regularization by adding the --reg-scale= flag and using a larger value, the current default is 10. You can also try to increase the number of training epochs since such a drastic reduction might need more training in order converge.

[King4819]

@Mxbonn Thanks for your reply. I have tried setting reduction_target to 0.3, and the result model GFLOPs is 1.9 GFLOPs, which is not so accurate.