Back To Schedule
Wednesday, June 20 • 2:00pm - 2:50pm
OPEN TALK: Stream Processing is Hard: Why it's Hard to Have Nice Things that are Fast

Sign up or log in to save this to your schedule, view media, leave feedback and see who's attending!

Feedback form is now closed.

An ideal data platform would have the following characteristics: Fast, Cheap, Accurate, Horizontally Scalable, and Fault Tolerant, but it's hard to have these characteristics combined in one data platform.

 Typically, batch processing is cheap and accurate, but it sacrifices efficiency. Stream processing is fast, but there are many failure modes in stream processing, making fault tolerance more difficult to reason about. Finding the right balance between these requirements is difficult, therefore, lambda architectures are highly popular.

Rather than making one solution achieve all of these things, you can run batch and stream in parallel which allows for the entire system to achieve the required characteristics. However, this is not cheap. But, what if we could make stream processing accurate, fault tolerant, and horizontally scalable?

 This talk will do a deep dive into the complexities of performing a join across multiple large data sets and how to build a reliable stream processing platform. The motivating use case will be the core auctions join that AppNexus must perform in order to determine which auctions transacted, which sees 1.2M events/second and 600MB/sec.

avatar for Mike Freyberger

Mike Freyberger

Software Engineer, AppNexus
Mike Freyberger is a software engineer at AppNexus. Mike grew up in New Jersey, went to college in New Jersey, and lives in New Jersey.

Wednesday June 20, 2018 2:00pm - 2:50pm
Expo Stage