Author(s) :

Sankalp Tank.

Volume/Issue :

Abstract :

Information retrieval (IR) methods, systems that find relevant information from large datasets, are separated into sparse and dense methods. This study investigates how these two types of methods can work in tandem to optimize speed, indexing cost, and accuracy when answering structured queries on large datasets. This research implemented and benchmarked multiple retrieval methods, ranging from sparse methods TF-IDF, BM25, and an enhanced version coined SUPER_BM25, to dense methods SPLADE and COLBERT. These methods were queried to measure the recall, indexing time, and query time of each. The results indicated that dense methods achieved fast retrieval times at the cost of precision; conversely sparse methods were incredibly accurate, but they took significantly more time. Based on these results, a funnel system of these disparate methods was created, where each method worked in tandem to optimize speed and accuracy. This funnel system reduced indexing time by 23.7% and query time by 99% when compared with COLBERT while retaining comparable recall scores. The funnel system achieved these high indexing and query speeds by having TFIDF, BM25, and SUPER_ BM25 cull 90.4% of the dataset, then giving SPLADE and COLBERT the remaining data to accurately rank it. This hybrid funnel approach presents a scalable and cost-efficient framework for real-word information retrieval, enabling faster, more accurate search across large datasets.