Computing Document Term Relevance Scores

In information retrieval and text mining, quantifying the importance of words within documents is a foundational task. Term Frequency-Inverse Document Frequency (TF-IDF) is a statistical measure that evaluates how relevant a term is to a specific document within a larger collection (corpus). This technique is widely used in search engines, document classification, and feature extraction for machine learning models.

Understanding the TF-IDF Formula

TF-IDF combines two complementary metrics:

1. Term Frequency (TF)

Term Frequency measures how often a term occurs in a document. The intuition is that the more frequently a term appears, the more important it might be to that document:

$$TF(t, d) = \frac{\text{count of term } t \text{ in document } d}{\text{total number of terms in document } d}$$

2. Inverse Document Frequency (IDF)

Inverse Document Frequency measures the discriminative power of a term across the corpus. Terms that appear in many documents (like "the", "and", "is") are less useful for distinguishing between documents:

$$IDF(t) = \ln\left(\frac{N + 1}{df(t) + 1}\right)$$

Where:

• N = Total number of documents in the corpus
• df(t) = Number of documents containing term t
• The smoothing factors (+1) prevent division by zero when a term doesn't appear in any document

3. TF-IDF Score

The final TF-IDF score is the product of these two components:

$$TF\text{-}IDF(t, d) = TF(t, d) \times IDF(t)$$

Your Task

Implement a function that calculates TF-IDF scores for a list of query terms across a corpus of documents. Your function must:

1. Handle smoothing properly: Apply the smoothing formula for IDF to prevent division by zero errors
2. Process empty documents gracefully: Documents with no words should return zero scores for all query terms
3. Handle missing query terms: Query terms that don't appear in any document should still return valid (zero) scores
4. Round results appropriately: Return all scores rounded to 5 decimal places

Implementation Notes

• Each document in the corpus is provided as a list of individual word tokens
• The query is also a list of terms to evaluate
• Return a matrix where rows correspond to documents and columns correspond to query terms

corpus = [
    ["the", "cat", "sat", "on", "the", "mat"],
    ["the", "dog", "chased", "the", "cat"],
    ["the", "bird", "flew", "over", "the", "mat"]
]
query = ["cat"]

[[0.04795], [0.05754], [0.0]]

corpus = [
    ["the", "cat", "sat", "on", "the", "mat"],
    ["the", "dog", "chased", "the", "cat"],
    ["the", "bird", "flew", "over", "the", "mat"]
]
query = ["cat", "dog", "mat"]

[[0.04795, 0.0, 0.04795], [0.05754, 0.13863, 0.0], [0.0, 0.0, 0.04795]]

corpus = [
    ["apple", "banana", "cherry"],
    ["date", "elderberry", "fig"],
    ["grape", "honeydew", "kiwi"]
]
query = ["mango"]

[[0.0], [0.0], [0.0]]