Design and Implementation of Real Time Data Processing System Based on Spark Streaming
Abstract
With the rapid development of Internet technology, people have more abundant ways to obtain information through the
Internet.More and more information is spread in the network, which will produce a large amount of data.The big data type is more
complicated. In order to implement data implementation, the industry is generally combined with different data types and business
scenes, special development and design of different processing subsystems.This method has a large research cost and learning cost,
and a unified computing system platform is relatively lacking, which further makes developers unable to maintain and expand the
system formed by different sets of technology systems.Based on the above situation, this paper designs a universal real-time data
analysis and processing system based on Spark Streaming.
Internet.More and more information is spread in the network, which will produce a large amount of data.The big data type is more
complicated. In order to implement data implementation, the industry is generally combined with different data types and business
scenes, special development and design of different processing subsystems.This method has a large research cost and learning cost,
and a unified computing system platform is relatively lacking, which further makes developers unable to maintain and expand the
system formed by different sets of technology systems.Based on the above situation, this paper designs a universal real-time data
analysis and processing system based on Spark Streaming.
Keywords
Spark Streaming; Real-time data processing; The system design
Full Text:
PDFReferences
[1] Shi Wei. Design and Implementation of Real Time Data Processing System Based on Spark Stream [J]. Modern Information Technology, 2020, 4 (20): 10-12.
[2] Li Tianxi. Research and Implementation of Experimental Data Processing System based on Spark Streaming [D]. Shaanxi: Xidian University,2015.
[3] Xu Yifeng, Feng Dajun, ZHANG Hanwen, et al. Application of electronic technique,2017,43(9):98-100,105.
[4] Zhang Muwei, Dong Feng. Design of real-time data processing system for infrared camera based on FPGA [J]. Infrared,2016,37(4):1-6,32.
DOI: https://doi.org/10.18282/l-e.v10i3.2451
Refbacks
- There are currently no refbacks.
