Data-driven Search Optimization, Analysing Relevance | S2 E2

How AI Is Built

In this episode, we talk data-driven search optimizations with Charlie Hull.

Charlie is a search expert from Open Source Connections. He has built Flax, one of the leading open source search companies in the UK, has written “Searching the Enterprise”, and is one of the main voices on data-driven search.

We discuss strategies to improve search systems quantitatively and much more.

Key Points:

1. Relevance in search is subjective and context-dependent, making it challenging to measure consistently.
2. Common mistakes in assessing search systems include overemphasizing processing speed and relying solely on user complaints.
3. Three main methods to measure search system performance:
   • Human evaluation
   • User interaction data analysis
   • AI-assisted judgment (with caution)
4. Importance of balancing business objectives with user needs when optimizing search results.
5. Technical components for assessing search systems:
   • Query logs analysis
   • Source data quality examination
   • Test queries and cases setup

Resources mentioned:

• Quepid: Open-source tool for search quality testing
• Haystack conference: Upcoming event in Berlin (September 30 - October 1)
• Relevance Slack community
• OpenSource Connections

Charlie Hull:

• LinkedIn
• X (Twitter)

Nicolay Gerold:

• ⁠LinkedIn⁠
• ⁠X (Twitter)

search results, search systems, assessing, evaluation, improvement, data quality, user behavior, proactive, test dataset, search engine optimization, SEO, search quality, metadata, query classification, user intent, search results, metrics, business objectives, user objectives, experimentation, continuous improvement, data modeling, embeddings, machine learning, information retrieval

00:00 Introduction
01:35 Challenges in Measuring Search Relevance
02:19 Common Mistakes in Search System Assessment
03:22 Methods to Measure Search System Performance
04:28 Human Evaluation in Search Systems
05:18 Leveraging User Interaction Data
06:04 Implementing AI for Search Evaluation
09:14 Technical Components for Assessing Search Systems
12:07 Improving Search Quality Through Data Analysis
17:16 Proactive Search System Monitoring
24:26 Balancing Business and User Objectives in Search
25:08 Search Metrics and KPIs: A Contract Between Teams
26:56 The Role of Recency and Popularity in Search Algorithms
28:56 Experimentation: The Key to Optimizing Search
30:57 Offline Search Labs and A/B Testing
34:05 Simple Levers to Improve Search
37:38 Data Modeling and Its Importance in Search
43:29 Combining Keyword and Vector Search
44:24 Bridging the Gap Between Machine Learning and Information Retrieval
47:13 Closing Remarks and Contact Information