REPOSITORY CODING · MEDIUM
Normalize searchable Unicode words without changing source text
Full-width and compatibility-form words fail to match ordinary query text. Normalize token identities with NFKC and locale-independent lowercasing while keeping original source spans for display.
Repository ImplementationPositional document search laboratory
Work through a real codebase.
Choose an environment, investigate the existing implementation and use the tests to check your change. Sign in to open a workspace; execution and Ono usage follow your plan’s allowances.
Task requirements and environments
Java · 55 minute session
- Search a bounded offline corpus with distinct document IDs and at most 30 documents of 5000 UTF-16 code units. Tokenize Unicode letters, numbers and combining marks, normalize token identities with NFKC and Locale.ROOT lowercasing, and preserve original code-point source spans.
- Queries support terms, double-quoted phrases, uppercase AND/OR/NOT and parentheses. NOT binds most tightly, then AND, then OR. Explicit operators are required between expressions; escaping embedded quotes is outside this bounded grammar. Empty queries match all documents.
- Quoted phrases require consecutive token identities in order, including repeated words. Boolean negation affects matching; only positive-polarity terms contribute ranking or highlighting. A negative-only query ranks by document ID.
- Rank matching documents by descending BM25 with k1=1.2 and b=0.75, then lexical ID. Document frequency counts distinct documents containing a term, and document length and average length are token counts. Repeated positive query terms are deduplicated.
- Return at most topK matches. Snippets contain at most snippetSize code points, beginning half a window before the first positive hit (or at zero without hits). Highlight spans are zero-based, end-exclusive code-point offsets relative to that window, clipped to its visible bounds. Source text is preserved and no document code is executed.
Spring Boot · 55 minute session
- Search a bounded offline corpus with distinct document IDs and at most 30 documents of 5000 UTF-16 code units. Tokenize Unicode letters, numbers and combining marks, normalize token identities with NFKC and Locale.ROOT lowercasing, and preserve original code-point source spans.
- Queries support terms, double-quoted phrases, uppercase AND/OR/NOT and parentheses. NOT binds most tightly, then AND, then OR. Explicit operators are required between expressions; escaping embedded quotes is outside this bounded grammar. Empty queries match all documents.
- Quoted phrases require consecutive token identities in order, including repeated words. Boolean negation affects matching; only positive-polarity terms contribute ranking or highlighting. A negative-only query ranks by document ID.
- Rank matching documents by descending BM25 with k1=1.2 and b=0.75, then lexical ID. Document frequency counts distinct documents containing a term, and document length and average length are token counts. Repeated positive query terms are deduplicated.