Concept drift monitoring and continual learning in production AI systems: an empirical cost–benefit comparison of detection methods and adaptation strategies

Abstract

Production machine learning systems face a persistent operational challenge: the distribution of input features and the conditional distribution of labels can shift over time, eroding the predictive performance that motivated deployment. This paper conducts an empirical comparison of three widely used concept drift detectors—ADWIN, DDM, and Page–Hinkley—paired with two adaptation strategies, incremental learning and full retraining. Using two publicly available streaming benchmarks (Electricity and SEA) augmented with a synthetic noisy variant, we construct a cost–benefit framework that jointly accounts for predictive accuracy, drift-response latency, computational cost, and false-alarm rate. Across 60 controlled trials, ADWIN paired with incremental learning achieved the highest accuracy-to-cost ratio on stationary segments and gradual drifts, while DDM combined with periodic retraining reacted most decisively to abrupt shifts at the cost of higher compute. Page–Hinkley provided a useful middle ground when budget is moderately constrained. No single configuration dominated across regimes; engineers should select detectors based on the dominant drift profile of their pipeline.