Dynamic Scaling of Distributed Dataflows Under Uncertainty

Abstract

Performant Distributed Stream Processing Systems (DSPSs) are essential to processing large volumes of high-velocity data in a reliable and timely fashion. The global event stream processing industry is becoming ever more important to the world economy, projected to grow from its current $930 million valuation, to $2.4 billion by 2030. These systems commonly experience highly variable, bursty and unpredictable workloads, presenting a challenge when provisioning compute to meet the needs of these workloads. Rightsizing systems for peak demand leads to often-unacceptable financial cost, motivating the need for adaptive approaches to meet the needs of changing workloads. The choice of parallelism of workload operators are commonly governed by autoscalers, but their behaviour is often case specific and highly sensitive to the choice of tunable parameters and thresholds. Current approaches to evaluating DSPS and contemporary autoscaler performance provides limited visibility into the robustness of performance metrics, nor worst-case performance of systems under specific operating conditions. These issues present a challenge to practitioners wishing to understand the performance implications of their decisions. In this thesis, we make the following contributions: • Empirically study undesirable behaviours experienced by a state-of-the-art autoscal ing controller and contribute a categorisation of failures exhibited by autoscaling mechanisms. • Demonstrate the potential of moving average models to augment existing autoscalers to help mitigate these behaviours, successfully mitigating over 90% of undesirable extreme parallelism shifts and significantly reducing scaling-behaviour volatility. • Challenge current approaches to quantifying and measuring streaming system ro bustness, and propose the use of non-parametric goodness-of-fit tests to quantify streaming system robustness. • Investigate the potential of our suggested robustness quantifier and Response Surface Methodology (RSM) to capture the complex relationship between incoming workload characteristics and system latency, providing valuable insights into the behaviour of DSPSs under diverse operating conditions.