AWS re:Invent 2017: GPS: Real-Time Data Processing with AWS Lambda Quickly, at Scale (GPSTEC313)

Real-time data processing is a powerful technique that allows businesses to make agile automated decisions. This process is particularly powerful when applied to workloads like security, analyzing access logs, parsing audit logs, and monitoring API activity to detect behavior anomalies. Combined with automation, business can quickly take action to remediate security concerns, or even train a machine learning (ML) model. We explore different techniques for analyzing real-time streams on AWS using Lambda, Amazon Kinesis, Spark with Amazon EMR, and Amazon DynamoDB. We also cover best practices around short- and long-term storage and analysis of data and, briefly, the possibility of leveraging ML.