Back To Schedule
Wednesday, June 20 • 9:25am - 9:50am
PRO TALK: Automating LinkedIn’s Machine Learning & Data Pipelines with Workflow Engine Platform Azkaban

Sign up or log in to save this to your schedule, view media, leave feedback and see who's attending!

Feedback form is now closed.
At LinkedIn - we are have built massively scalable open source workflow engine platform (Azkaban) which handles and orchestrates almost all of LinkedIn’s offline data infrastructure from AI and deep learning to analytics and from Hadoop to Spark. There are massive benefits in having one powerful workflow engine to power all your flows. However, as companies scale and workloads differ from machine learning to analytics simple workflow engine simply does not scale. Linkedin is solving this challenge with building "workflow engine platform" - highly pluggable and extensible open source work flow engine to automate Linkedin's offline data, AI and analytics infrastructure. Azkaban is fully open source and in process of becoming Apache project. This talk covers challenges in building workflow engine platform and deploying at LinkedIn scale

avatar for Charlie Summers

Charlie Summers

Software Engineer, LinkedIn
Charlie has been working at LinkedIn on Azkaban since April 2017. He's worked on many pieces of the Azkaban Ecosystem (plugins, DSL, etc.) as well as the internal Azkaban build pipelines. Before working at LinkedIn, Charlie was a cadet at 42 Silicon Valley and before that he was supporting... Read More →
avatar for Jamie Sun

Jamie Sun

Software Engineer, LinkedIn
Jamie has been working at LinkedIn on Azkaban team since November 2016. She enjoys designing and implementing cool features for Azkaban. She has solved many practical problems for the distributed system. The initial integration test framework she developed has greatly improved the... Read More →

Wednesday June 20, 2018 9:25am - 9:50am EDT
Main Stage