Methodologies for Managing Big Data Analytics Pipelines for Smart City Applications

Abstract

Smart cities and Early warning systems rely on complex analytics on the data generated by sensor networks, including IoT and social media. In order to extract value from IoT and social media, a massive volume of heterogeneous data needs to be processed, stored and analysed, which demands a combination of tools from stream processing engines, data lakes and analytics tools. As the data sources are highly distributed and significant in the count, the analytics process is also distributed across different layers. The flow of data from the source and through different subsystems of the IoT and social media depends on various aspects like the frequency of observation, sampling, bandwidth availability, type of analytics processing and location of processing. In IoT networks, the analytical processing is distributed across edge and cloud data centres. The network conditions in the IoT network are prone to be highly variant as it relies on wireless networks operating on radio frequencies like cellular and WiFi. It is quite a common scenario in IoT networks to have frequent bandwidth changes while switching networks and various environmental conditions. Hence, data flow management in IoT networks is an essential aspect of managing the quality of service. This thesis addresses critical challenges in managing the data flow and analytics pipelines for IoT and social media data. The first part of the thesis explains a tool kit to perform data flow experiments using emulation of a three-layered IoT network. The second part processes algorithms to manage data flow based on the workload and bandwidth between the edge and the cloud data centres. The third part of the thesis explains the design of the data ingestion and analytics part of a landslide early warning system. This section discusses methodologies for orchestrating a real-time streaming data analytic pipeline for social media data. The main contributions of this thesis are i) A set of algorithms to manage the dataflow in IoT networks. ii) A set of tools for emulating IoT networks using edge processing hardware iii) Methodologies and a framework for the orchestration of streaming analytics pipeline for social media data used in an early warning system.