REPOSITORY CODING · HARD
Deduplicate observations before session sample thresholds
Redelivered observations inflate session counts, while sessions exactly at minSamples are discarded. Repair deduplication before window functions and inclusive filtering after full session aggregation.
Data EngineerDevice counter session laboratory
Work through a real codebase.
Choose an environment, investigate the existing implementation and use the tests to check your change. Sign in to open a workspace; execution and Ono usage follow your plan’s allowances.
Task requirements and environments
PySpark · 80 minute session
- Sessionize offline device counter observations using real PySpark transformations. Each device/timestamp pair identifies one reading. Exact redelivery is harmless, conflicting counter values at that timestamp are invalid, and observations may arrive out of order. Counters lie in [0,modulus).
- Within each device, a monotonic counter contributes its difference. A decrease is a wrap only when previous*4 >= modulus*3 and current*4 <= modulus; then contribute modulus-previous+current. Other decreases are resets and contribute current. A first reading contributes zero.
- Start a new session only when the elapsed timestamp gap is strictly greater than maxGap. The first reading of every new session contributes zero even when a previous counter exists. Sum increments within each session, count unique observations, and report activeSeconds=end-start.
- Apply minSamples inclusively after complete session aggregation. Duplicate deliveries cannot satisfy that threshold. Return sessions sorted by device then start, with start, end, samples, increment and activeSeconds. The local Spark session is reused; no external telemetry source, platform account or database is accessed.