Dynamically Configured Stream Processing In Apache Flink - The use case of custom processing rules management and application
Date Issued
2021-09-23
Author(s)
Andonov, Stefan
Abstract
This paper presents advanced Apache Flink application patterns for low latency distributed data stream processing. These patterns extend the concept of statically defined data flows
and allow Flink jobs to dynamically change at runtime, without downtime. The introduced patterns allow dynamic configuration and change of the application logic and processing steps for
implementing complex business scenarios. Using a real-life use case scenario and dynamic processing rules configuration, we present the patterns for dynamic data partitioning, dynamic
window configuration, and dynamic data aggregation. They are implemented using the high-level APIs for windowing and aggregation and the low-level process function API. The patterns are
implemented using the concept of control/configuration stream and broadcast stream and the carrier of the control information, control message. The real-life use case scenario tackles the
problem of processing and analyzing air pollution data obtained from different sensors located in many different locations, as well as visualization of the data in third-party software.
and allow Flink jobs to dynamically change at runtime, without downtime. The introduced patterns allow dynamic configuration and change of the application logic and processing steps for
implementing complex business scenarios. Using a real-life use case scenario and dynamic processing rules configuration, we present the patterns for dynamic data partitioning, dynamic
window configuration, and dynamic data aggregation. They are implemented using the high-level APIs for windowing and aggregation and the low-level process function API. The patterns are
implemented using the concept of control/configuration stream and broadcast stream and the carrier of the control information, control message. The real-life use case scenario tackles the
problem of processing and analyzing air pollution data obtained from different sensors located in many different locations, as well as visualization of the data in third-party software.
File(s)![Thumbnail Image]()
Loading...
Name
paper 22.pdf
Size
1.14 MB
Format
Adobe PDF
Checksum
(MD5):11653797794dd697aa92df4503b8f5f4