SE-Radio Episode 272: Frances Perry on Apache Beam

Filed in Episodes by on October 25, 2016 2 Comments

francesperry100x125Jeff Meyerson talks with Frances Perry about Apache Beam, a unified batch and stream processing model. Topics include a history of batch and stream processing, from MapReduce to the Lambda Architecture to the more recent Dataflow model, originally defined in a Google paper. Dataflow overcomes the problem of event time skew by using watermarks and other methods discussed between Jeff and Frances. Apache Beam defines a way for users to define their pipelines in a way that is agnostic of the underlying execution engine, similar to how SQL provides a unified language for databases. This seeks to solve the churn and repeated work that has occurred in the rapidly evolving stream processing ecosystem.

Venue: Internet

Related Links

Facebooktwittergoogle_pluslinkedin

Tags: , , , , , , , ,