Rank reduction using random matrix theory

[fernando-neto-ai]

Hi! I really enjoyed the paper.
I've implemented a version that uses marchenko pastur in order to speed up the search, instead looking within a grid search.

If it's of your interest, we can join efforts.

I would be glad if you could take a look at https://github.com/cognitivecomputations/laserRMT

Congratulations for your work.

Best,

Fernando

[pratyushasharma]

Hi Fernando,

Thank you for your interest in our work!

I am excited to read your extension. I do have a few questions:

1. Does the speeded-up search still result in the same improvements we report in the paper? To show that the speeded-up search might be able to replace the full grid search, it would also be important to check if the same / better set of parameters are recovered. Do the resulting models from the modified search procedure give comparable performance improvements on the datasets tested in the paper?

2. When we perform LASER, the search for the optimal parameters is always using a validation set (that comes from the different datasets of interest). Therefore, the model is laser-ed differently for different data distributions [Table 3 in the appendix of the paper]. It seems like in your approach the data distribution is not accounted for when performing laser, is that correct?

Being able to replace the full grid search with something smarter will be super beneficial--We are excited to hear more about your work.

Best,
Pratyusha

[fernando-neto-ai]

Hi! Great to hear from you.

Addressing your questions:

1. For now, they are not comparable yet. As I haven't tested using the datasets of the benchmarks applying LASER over the layers. But I am working on modifying the code to use somehow the datasets instead of the perplexity.
2. We have observed a small increase on huggingface scores on some models, but very slight increases, as we are not modeling or maximizing any objective in terms of specific benchmark, which strongly encoraged us to keep moving forward. We calculate the perplexity using 3 datasets, to check if the proposed intervention in the model reduces them (that is our intervention acceptance rule).
3. It's been reported that, also, a considerable increase in encoder-based models (BERT, deberta, etc) was achieved - specially for retrieval models.
4. It seems that using a Marchenko-Pastur based approach may avoid excessive rank reduction, that may lead, ultimately, to information destruction and waste computational time searching for non feasible reductions. We have observed that for 7b models, it was common to destroy performance on gsm8k benchmarks - which totally makes sense according to your hypothesis of removing effects of less frequent terms, considering that probably knowledge over math is sparser. Here it seems both approaches suffer from it. In the end there is leveraging over some benchmarks and performance destruction on others.

best regards,

Fernando

[dkmisra]

This is exciting! Finding a computationally efficient way to select LASER hyperparameters is indeed an open question.

By the way, we have now put a results table on the website. If you prefer, we will be happy to paste your results with laserRMT and acknowledge/cite them to you.

My questions/comments are the same as @pratyushasharma . While I am optimistic that there are hyperparameters that do good across a range of tasks, however, by doing so, you may not see the huge gains we were seeing in the paper which we got by doing task-specific hyperparameter selection. Also, it will be nice to either try vanilla LASER with grid search for hyperparameter selection on these tasks that you have tried, or try laserRMT on the datasets in our paper.