Quantitative Impact Evaluation of an Abstraction Layer for Data Stream Processing Systems · HackerPapers discussions

Quantitative Impact Evaluation of an Abstraction Layer for Data Stream Processing Systems — HN submission

Thu, 05 Mar 2020 13:15:12 GMT

Quantitative Impact Evaluation of an Abstraction Layer for Stream Processing (2019)

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Quantitative Impact Evaluation of an Abstraction Layer for Data Stream Processing Systems — mentioned by faizshah

Wed, 18 Dec 2019 03:55:07 GMT

Check this reference out: https://arxiv.org/pdf/1907.08302.pdf “ Our benchmark results show that Apache Beam has a noticeable impact on the performance of DSPSs in almost all cases. Programs developed using Apache Beam suffered from a slowdown of up to a factor of 58 in the worst case. At the same time, there is one scenario where the query developed using Apache Beam is about as fast as its counterparts using the APIs of the corresponding DSPS. However, for most scenarios we observed a slowdown of at least a factor three.”

Paper and discussion history. A mention is not an endorsement.

Quantitative Impact Evaluation of an Abstraction Layer for Data Stream Processing Systems — mentioned by faizshah

Tue, 19 Nov 2019 19:33:27 GMT

So I have a bad history with Cloud Dataflow. I like BigQuery and Flink a lot, but a lot of companies seem to be switching over to Cloud Dataflow. In my experience Cloud Dataflow's cost is really unpredictable and DataPrep's pipelines (on DataFlow) were far more inefficient compared to an equivalent DataProc Spark pipeline, Spark on Compute Engine, or just plain BigQuery. In addition: Apache BEAM's overhead itself is a lot: https://arxiv.org/pdf/1907.08302.pdf Switching from Dataflow can incur massive cost savings: https://lugassy.net/how-moving-from-pub-sub-to-avro-saved-us... Right now I'm…

Paper and discussion history. A mention is not an endorsement.