Count distinct document terms while… — Code review | OneNoughtOne

Search, Parsing & Dependency Engines0/97 completed

Computing Document Term Relevance Scores
Probabilistic Document Relevance Scorer (BM25)
Normalize searchable Unicode words without changing source text
Normalize searchable Unicode words without changing source text
Give AND tighter binding than OR in search expressions
Give AND tighter binding than OR in search expressions
Match phrases by consecutive token positions
Match phrases by consecutive token positions
Rank with document frequencies and token lengths
Count distinct document terms while normalizing by characters
Keep snippet windows and highlights in code-point coordinates
Slice snippets by code points while retaining UTF-16 token offsets

Course overview & roadmap

101OneNoughtOne

0/97

Loading content...

Search, Parsing & Dependency Engines0/97 completed

Computing Document Term Relevance Scores
Probabilistic Document Relevance Scorer (BM25)
Normalize searchable Unicode words without changing source text
Normalize searchable Unicode words without changing source text
Give AND tighter binding than OR in search expressions
Give AND tighter binding than OR in search expressions
Match phrases by consecutive token positions
Match phrases by consecutive token positions
Rank with document frequencies and token lengths
Count distinct document terms while normalizing by characters
Keep snippet windows and highlights in code-point coordinates
Slice snippets by code points while retaining UTF-16 token offsets

Course overview & roadmap

101OneNoughtOne

0/97

101OneNoughtOne

00:00:00

Count distinct document terms while normalizing by characters

Review the independently proposed patch below, including its unchanged callers. Repeated terms are counted as multiple documents and character counts distort BM25 length normalization. Compute one document-frequency contribution per distinct term per document and normalize by token count.

Review focus: Repository Implementation

src/repositories/Corpus.java

BeforeProposed

+1−1

1616 if(!row.path("text").isTextual() || row.get("text").asText().length()>5000)throw new IllegalArgumentException("invalid-text");

1717 Document document=new Document(row.get("id").asText(),row.get("text").asText());

1818 documents.add(document);total+=document.tokens.size();

19- for(Token token:document.tokens)documentFrequency.merge(token.term(),1,Integer::sum);

19+ for(String term:document.frequencies.keySet())documentFrequency.merge(term,1,Integer::sum);

2020 }

2121 averageLength=documents.isEmpty()?1:Math.max(1,(double)total/documents.size());

2222 }

Your review

0 / 10,000 characters · A verdict needs supporting evidence.

30 more characters to give a verdict

101OneNoughtOne

00:00:00

Count distinct document terms while normalizing by characters

Review focus: Repository Implementation

src/repositories/Corpus.java

BeforeProposed

+1−1

1616 if(!row.path("text").isTextual() || row.get("text").asText().length()>5000)throw new IllegalArgumentException("invalid-text");

1717 Document document=new Document(row.get("id").asText(),row.get("text").asText());

1818 documents.add(document);total+=document.tokens.size();

19- for(Token token:document.tokens)documentFrequency.merge(token.term(),1,Integer::sum);

19+ for(String term:document.frequencies.keySet())documentFrequency.merge(term,1,Integer::sum);

2020 }

2121 averageLength=documents.isEmpty()?1:Math.max(1,(double)total/documents.size());

2222 }

Your review

0 / 10,000 characters · A verdict needs supporting evidence.

30 more characters to give a verdict