AWS re:Invent 2019: Cloud-native machine learning at Lyft with AWS Batch and Amazon EKS (CON235-P)
Published on Dec 20, 2019
Flyte is the backbone for large-scale machine learning and data processing pipelines at Lyft and is used across many business-critical applications. It’s essentially a container-based workflow engine that executes 10+ million containers per month as part of thousands of workflows. This talk focuses on Flyte’s use of Amazon EKS and Kubernetes on Amazon EC2, as well as Amazon EKS, Amazon ECS, and AWS Batch to scale to millions of containers. Learn how Flyte uses AWS to offer frictionless serverless compute experiences and about Flyte’s architecture and specification language used to orchestrate compute and manage data flow across disparate systems. Finally, hear about managing large-scale Kubernetes deployments on AWS and Flyte use cases.