Contextual data stream processing overview, architecture, and frameworks survey

Abstract EN

Event stream processing (ESP) is a data processing methodology which tackle online processing for a variety of events.

Recently stream processing witnessed a huge interest in both academic research and corporate use cases.

As a consequence, for the extremely huge data sources recently generated and diversely used.

Data sources vary from social media feeds, news articles, internal business transactions, IoT devices logs, .

.

.

etc.

Academically, a lot of research papers discuss how to deal with enormous cloud of events with different data structures such as text, video, logs, transactions, … etc.

Also, research is concerned with different streaming platforms technologies; and evaluates the weakness and strength points of each.

Researchers studied aside how to best utilize the platform within different use cases.

From corporate point of view, decision makers ask about how to best utilize those events with minimal delay in order to 1) uncover insights in real-time, 2) mine textual events, 3) recommend decisions.

This requires a mix of machine learning, stream and batch processing technologies which are typically optimized independently.

However, combining all technologies by building a scalable real-world application is a challenge.

In this paper, we shall discuss state of the art event stream processing technologies by summarizing definition, data flow architectures, textual use cases, frameworks and architecture best practice.

Furthermore, we would discuss how to combine event stream processing with textual events and sentiment analysis to enhance a recommendation model outcome.