Please use this identifier to cite or link to this item:
Title: Dynamically Configured Stream Processing In Apache Flink - The use case of custom processing rules management and application
Authors: Andonov, Stefan 
Madjarov, Gjorgji 
Keywords: Apache Flink
stream processing
big data
stream analytics
distributed processing
Issue Date: 23-Sep-2021
Publisher: Ss Cyril and Methodius University in Skopje, Faculty of Computer Science and Engineering, Republic of North Macedonia
Series/Report no.: CIIT 2020 full papers;;22
Conference: 18th International Conference on Informatics and Information Technologies - CIIT 2021
Abstract: This paper presents advanced Apache Flink application patterns for low latency distributed data stream processing. These patterns extend the concept of statically defined data flows and allow Flink jobs to dynamically change at runtime, without downtime. The introduced patterns allow dynamic configuration and change of the application logic and processing steps for implementing complex business scenarios. Using a real-life use case scenario and dynamic processing rules configuration, we present the patterns for dynamic data partitioning, dynamic window configuration, and dynamic data aggregation. They are implemented using the high-level APIs for windowing and aggregation and the low-level process function API. The patterns are implemented using the concept of control/configuration stream and broadcast stream and the carrier of the control information, control message. The real-life use case scenario tackles the problem of processing and analyzing air pollution data obtained from different sensors located in many different locations, as well as visualization of the data in third-party software.
Appears in Collections:International Conference on Informatics and Information Technologies

Files in This Item:
File Description SizeFormat 
paper 22.pdf1.17 MBAdobe PDFView/Open
Show full item record

Page view(s)

checked on Jun 20, 2024


checked on Jun 20, 2024

Google ScholarTM


Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.