Experiences with Streaming vs Micro-Batch for Online Learning
In this talk we describe our experiences with Flink for real time classification and clustering of time series data. In particular, we focus on using online machine learning to detect SLA violations from streaming data of server utilization statistics. This type of use case requires that we balance the throughput of the analytics process with the latency of the individual predictions. The most demanding use cases require both very high throughput (number of statistics processed) and very low latency (violation detection in the shortest possible time). We have experimented with both streaming in Flink and micro-batch in Spark and will discuss the benefits we have seen for using streaming.