Back
+20 XP
Concept

TF-IDF Text Vectorization from Scratch

Term Frequency-Inverse Document Frequency (TF-IDF) weights word relevance: $TF(t, d) \times \log(\frac{N}{DF(t)})$. Common words across all documents receive near-zero weight, while distinctive terms dominate search relevance 📚.

python
import math
from collections import Counter

corpus = [
    ["apple", "banana", "apple"],
    ["banana", "orange"],
    ["apple", "kiwi"]
]
N = len(corpus)
df = Counter(w for doc in corpus for w in set(doc))

doc0 = corpus[0]
tf = Counter(doc0)
scores = {
    w: round((tf[w] / len(doc0)) * math.log(N / df[w]), 3)
    for w in sorted(set(doc0))
}
print(f"Doc 0 TF-IDF: {scores} 📚")

# 60. TF-IDF Text Vectorization from Scratch Term Frequency-Inverse Document Frequency (TF-IDF) weights word relevance: $TF(t, d) \times \log(\frac{N}{DF(t)})$. Common words across all documents receive near-zero weight, while distinctive terms dominate search relevance 📚. ```python import math from collections import Counter corpus = [ ["apple", "banana", "apple"], ["banana", "orange"], ["apple", "kiwi"] ] N = len(corpus) df = Counter(w for doc in corpus for w in set(doc)) doc0 = corpus[0] tf = Counter(doc0) scores = { w: round((tf[w] / len(doc0)) * math.log(N / df[w]), 3) for w in sorted(set(doc0)) } print(f"Doc 0 TF-IDF: {scores} 📚") ``` Before dense vector embeddings (like BERT and OpenAI Ada), TF-IDF powered almost every production search engine and Lucene/Elasticsearch index on the planet 🚀! --- ### Question What happens to the IDF (Inverse Document Frequency) score of a stopword that appears in every single document of a corpus? 🧠 - [ ] a. Its IDF drops to $\log(N/N) = \log(1) = 0.0$, neutralizing its influence on search ranking 📚 - [ ] b. It causes a catastrophic ZeroDivisionError during logarithmic evaluation - [ ] c. Its IDF score approaches positive infinity, overpowering distinct topical keywords - [ ] d. The token parser silently drops the entire document from memory --- ### Challenge Given corpus `[['apple', 'banana', 'apple'], ['banana', 'orange'], ['apple', 'kiwi']]`, compute TF-IDF scores for the words in `corpus[0]`. Print the dictionary with scores rounded to 3 decimal places. 🎯 <details> <summary>Solution</summary> ```python import math from collections import Counter corpus = [ ["apple", "banana", "apple"], ["banana", "orange"], ["apple", "kiwi"] ] N = len(corpus) df = Counter(w for doc in corpus for w in set(doc)) doc0 = corpus[0] tf = Counter(doc0) scores = { w: round((tf[w] / len(doc0)) * math.log(N / df[w]), 3) for w in sorted(set(doc0)) } print(scores) ``` </details>

Checkpoint

What happens to the IDF (Inverse Document Frequency) score of a stopword that appears in every single document of a corpus? 🧠

Practice

Given corpus `[['apple', 'banana', 'apple'], ['banana', 'orange'], ['apple', 'kiwi']]`, compute TF-IDF scores for the words in `corpus[0]`. Print the dictionary with scores rounded to 3 decimal places. 🎯

Terminal
01
Tap code blocks below to assemble
Available Blocks