Enabling Efficient ML-based IoT Applications: Edge-Cloud Collaboration for Deployment, Updates, and Optimization

Abstract

The rapid integration of the physical world with the Internet through the IoT has led to a massive network of connected devices. ML has emerged as a crucial technology for processing and analyzing the vast amounts of diverse data generated by this IoT network, enabling intelligent IoT applications. The combination of ML and IoT has seen significant growth, enabling innovative use cases and leveraging cloud computing for data analysis and pattern extraction. ML-based IoT applications face challenges in effectively analyzing the vast amounts of data generated by diverse IoT devices. Transferring data to centralized cloud centers can be inefficient for timely analysis, and cloud computing may not always be suitable for emerging IoT applications. To overcome these challenges, a federated approach that combines cloud and edge resources is crucial. Edge computing brings computing operations to resource-constrained edge devices, enabling real-time responses and reducing data transmission to the cloud. Collaborating edge and cloud resources in a federated system allows for efficient data processing at the edge and complex ML algorithms in the cloud. This collaboration improves response times, reduces latency, enhances scalability, and optimizes resource utilization in ML-based IoT applications. An important challenge in edge-cloud collaboration is aggregating microservices in a way that meets application requirements such as latency, throughput, energy consumption, and model prediction accuracy. The edge-cloud collaboration aims to characterize Quality of Service (QoS) metrics based on microservice composition plans and adapt them to deployment sites, considering contextual factors and deployment locations. Furthermore, these plans need to be adaptable to fluctuations in computing environments throughout the application’s execution. A feedback-driven orchestration mechanism is necessary to detect changes in infrastructure performance and QoS metrics. In the edge-cloud computing paradigm, an additional challenge is the inconsistent model prediction performance observed in distributed environments. Models are con figured differently to accommodate resource constraints, leading to heterogeneity in model architectures and configurations. This heterogeneity can result in different out- puts from models when provided with the same input, posing a systemic problem that hinders prediction agreement within the application. Currently, there is a need for a systematic design to efficiently detect and minimize model inconsistency in distributed deep learning applications. To address the above mentioned challenges, the key contributions of this thesis are listed below: • Designing OsmoticGate, a video analytics task offloading framework that is capable of generating optimal workload balancing strategies, based on a Hierarchy Queue Model and a two-stage gradient-based algorithm. • Based on the research outcomes in OsmoticGate, implementing an online multi-agent reinforcement learning system named OsmoticGate2, which involves a co-designed algorithm and system to achieve workload balancing in dynamic and distributed deep learning (DL) applications. • Implementing DEEPCON , an adaptive deployment framework to quickly detect and improve model consistency through over-the-air parallel training and online knowledge distillation that enables teacher-student learning among all deployed models