Journey Clustering — Unsupervised User Journeys from Clickstream
Published:
Open the interactive project page →
Overview
A production pipeline that turns raw mobile app click logs into goal-level user journeys, clusters them without labels, and maps each cluster to a business taxonomy of about 500 intents. The product team uses it to see what customers come to the app for, where they struggle, and how iOS and Android differ.
Data Pipeline
- MongoDB → Kafka → SQL. I built the sink connector that pulls clickstream topics from Kafka into SQL.
- Daily Airflow cron job. Each day it partitions, clusters, scores, names and summarises the previous day’s events. Every task runs one CLI in the same Docker image.
- Out-of-core and idempotent. PyArrow and DuckDB keep memory bounded on 250M+ rows, and versioned data and model runs make reruns safe.
ML Pipeline
- Data processing. Role-correct canonicalisation, ID masking, a 5-level semantic ladder that aligns iOS and Android, and rule-based journey segmentation.
- Model development. A multi-channel TF-IDF/SVD and behavioural embedding, HDBSCAN archetypes, a Markov chain per cluster for anomaly and next-action scoring, and taxonomy naming with a human review queue.
- Monitoring. Data and model drift signals trigger challenger retraining with promotion gates and rollback.
Results
- About 17M journeys scored and about 2,800 journey archetypes across Android and iOS.
- The same model runs on-device on Android (Kotlin + ONNX Runtime) and iOS (Swift).