Dynamic modelling of nonlinear industrial processes using echo state network

Abstract

Accurate models are essential for the control and optimisation of industrial processes. Many chemical processes are complex and highly nonlinear and data-driven models need to be capitalised. Artificial neural networks based data-driven models are gaining popularity in chemical engineering area. Echo state network (ESN) is a recurrent neural network with a non-trainable sparse recurrent reservoir and an adaptable readout from the reservoir. Generally, the inputs weight and sparse reservoir connection weights are generated randomly. ESN is a highly effective method for the analysis and prediction of time series data and non-linear data and has been widely used in various research and application areas. However, ESN still has some drawbacks such as: incomprehensible black box properties, the reservoir connection structures and weights require multiple attempts to determine and lack of principled reservoir creation. This has raised the question of deep research of reservoir topology and how to create optimal ESN for modelling of complex chemical processes. In this thesis, a modular small world reservoir topology for ESN is proposed and it can enhance the dynamic property of the reservoir. Applications to both time series data and chemical process data show that ESN models with the proposed topology perform better than those with randomly generated topology. To further overcome the drawback of randomly generated reservoir, an adaptive genetic algorithm optimized ESN (AGA-ESN) is proposed. Genetic algorithm (GA) is population based global search and optimization algorithm. An adaptive mechanism is added to adjust the crossover and mutation probability. Four structural parameters of ESN are optimized by GA to promote the modelling performance for the specific modelling tasks. To overcome the problem of binary coding and discrete searching domain of GA, a continuous and efficient covariance matrix adaptation evolution strategy (CMA-ES) is used to optimize ESN. CMA-ES is a stochastic, derivative-free evolutionary algorithm for difficult non-linear optimization problems in continuous domain. As in AGA-ESN, the ESN reservoir parameters are optimized by CMA-ES. It is shown that CMA-ES models give more accurate predictions than GA-ESN models.In the final part of the thesis, the ability of ESN on handling multiple inputs data is enhanced by introducing an attention mechanism into ESN. The attention mechanism guarantees the ESN works with the most relevant input data by adding a weighted input scaling vector. It is shown that this modification leads to improved performance on modelling a penicillin fermentation process with complex multiple inputs, the proposed method reduced MSE by 6 times.