Replicating JinaV4 results using MTEB

#58
by nadshe - opened

Hi, I am attempting to replicate the results for retrieval tasks on MTEB using the implemented JinaV4 wrapper but am not able to do so. I continuously underperform reported results. Were tasks specific prompts used to obtain results? Are reported results using a single vector or multi-vector embeddings?

Extremely frustrating issue I'm facing as well. I'm trying to replicate the results of another retrieval paper & I can't even match the reported BM25 scores! (I'm getting better scores as compared to that reported lol)

Hi @nadshe ,
Thanks for reaching out. The difference in scores you are noticing is probably due to a recent change in the MTEB repository. Specifically, the prompt type was recently changed from "passage" to "document", but this change was not implemented properly for the Jina V4 wrapper. We've submitted a PR to address this, and once it's merged, you should be able to reproduce results much more closely.
Note: The ArguAna retrieval task was evaluated using the "text-matching" adapter.

Thanks @makram93 ! This seems to have resolved my replication issues. Much appreciated

nadshe changed discussion status to closed

Sign up or log in to comment