))
ColBERT + ColBERTv2: late interaction at a reasonable inference cost
Manage episode 355037184 series 3446693
Zeta Alpha에서 제공하는 콘텐츠입니다. 에피소드, 그래픽, 팟캐스트 설명을 포함한 모든 팟캐스트 콘텐츠는 Zeta Alpha 또는 해당 팟캐스트 플랫폼 파트너가 직접 업로드하고 제공합니다. 누군가가 귀하의 허락 없이 귀하의 저작물을 사용하고 있다고 생각되는 경우 여기에 설명된 절차를 따르실 수 있습니다 https://ko.player.fm/legal.
Andrew Yates (Assistant Professor at the University of Amsterdam) and Sergi Castella (Analyst at Zeta Alpha) discus the two influential papers introducing ColBERT (from 2020) and ColBERT v2 (from 2022), which mainly propose a fast late interaction operation to achieve a performance close to full cross-encoders but at a more manageable computational cost at inference; along with many other optimizations.
📄 ColBERT: "ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT" by Omar Khattab and Matei Zaharia. https://arxiv.org/abs/2004.12832
📄 ColBERTv2: "ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction" by Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. https://arxiv.org/abs/2112.01488
📄 PLAID: "An Efficient Engine for Late Interaction Retrieval" by Keshav Santhanam, Omar Khattab, Christopher Potts, and Matei Zaharia. https://arxiv.org/abs/2205.09707
📄 CEDR: "CEDR: Contextualized Embeddings for Document Ranking" by Sean MacAvaney, Andrew Yates, Arman Cohan, and Nazli Goharian. https://arxiv.org/abs/1904.07094
🪃 Feedback form: https://scastella.typeform.com/to/rg7a5GfJ
Timestamps:
00:00 Introduction
00:42 Why ColBERT?
03:34 Retrieval paradigms recap
08:04 ColBERT query formulation and architecture
09:04 Using ColBERT as a reranker or as an end-to-end retriever
11:28 Space Footprint vs. MRR on MS MARCO
12:24 Methodology: datasets and negative sampling
14:37 Terminology for cross encoders, interaction-based models, etc.
16:12 Results (ColBERT v1) on MS MARCO
18:41 Ablations on model components
20:34 Max pooling vs. mean pooling
22:54 Why did ColBERT have a big impact?
26:31 ColBERTv2: knowledge distillation
29:34 ColBERTv2: indexing improvements
33:59 Effects of clustering compression in performance
35:19 Results (ColBERT v2): MS MARCO
38:54 Results (ColBERT v2): BEIR
41:27 Takeaway: strong specially in out-of-domain evaluation
43:59 Qualitatively how do ColBERT scores look like?
46:21 What's the most promising of all current neural IR paradigms
49:34 How come there's still so much interest in Dense retrieval?
51:09 Many to many similarity at different granularities
53:44 What would ColBERT v3 include?
56:39 PLAID: An Efficient Engine for Late Interaction Retrieval
Contact: castella@zeta-alpha.com
13 에피소드
공유
