Using Generative Pseudo Label (GPL) in 2.X

[NoahAi25]

Hello everyone,

I would like to know if the Generative Pseudo Label (GPL) method is still possible to test with Haystack. It seems that this method was available in version 1.X, but I can't find it in version 2.X.

However, when I try running the code provided in the 1.X documentation or examples, I encounter errors like this:

TypeError: SentenceTransformerTrainer.compute_loss() got an unexpected keyword argument 'num_items_in_batch'

I haven't been able to find a configuration that allows the code to run successfully from start to finish.

If you believe this method is outdated and no longer achieves state-of-the-art results in the context of out-domain fine-tuning without labels, but you know of a better alternative, I’d be grateful for your recommendations.

Thank you in advance!

[julian-risch]

Hello @NoahAi25,
Thank you for reaching out to us. GPL is currently not supported in Haystack 2.x. Haystack 2.x doesn't have any training procedures for models used in the available retrieval components, such as SentenceTransformersDocumentEmbedder or SentenceTransformersTextEmbedder. So far, we didn't get many requests for GPL and we don't have any plans to add it.

For fine-tuning outside of Haystack, for example with SentenceTransformers, I suggest searching for a public dataset from your domain or a similar domain: https://huggingface.co/datasets?other=sentence-transformers
An alternative to fine-tuning could be to use a combination of the best retrieval component you currently have and a reranking component, such as TransformersSimilarityRanker with a cross-encoder model. That should help to increase the quality of the retrieval results https://docs.haystack.deepset.ai/docs/transformerssimilarityranker