Amazon Managed Service for Apache Flink Features

Amazon Managed Service for Apache Flink makes it easier to build and run real-time stream processing applications using Apache Flink. Amazon Managed Service for Apache Flink provisions and configures your Flink clusters and orchestrates Flink job management. It sets up monitoring and alarms, offers auto scaling, and is architected for high availability (including availability zone failover). The service offers access to Apache Flink’s expressive APIs, and through Amazon Managed Service for Apache Flink Studio, you can interactively query data streams or launch stateful applications in only a few steps. With this managed service, you can get started with Apache Flink and quickly deploy and operate your data stream processing applications. 

With Amazon Managed Service for Apache Flink, you have access to the full range of Apache Flink’s industry-leading capabilities, including low-latency and high-throughput data processing, exactly-once processing, and durable application state. With Amazon Managed Service for Apache Flink, you can deploy secure, compliant, and highly available applications. Amazon Managed Service for Apache Flink effortlessly replicates data and workloads across multiple Availability Zones, ensuring uninterrupted performance and reliability, and without having to pay for additional capacity.

Application development is easier with Amazon Managed Service for Apache Flink because the service supports Flink’s flexible APIs in Java, Scala, Python, and SQL. Amazon Managed Service for Apache Flink integrates with hundreds of data sources and destinations, such as Amazon MSK, Amazon Kinesis Data Streams, Amazon Kinesis Data Firehose, Amazon S3, Amazon DynamoDB, JDBC connectors, and custom connectors. 

Amazon Managed Service for Apache Flink includes open source libraries such as Apache Flink, Apache Beam, Apache Zeppelin, Amazon SDKs and Amazon Web Services service integrations. Apache Flink is a framework and engine for building highly available and accurate streaming applications. Apache Beam is a unified model for defining streaming and batch data processing applications that are run across multiple runtime engines. The Amazon SDKs help take the complexity out of coding for many Amazon Web Services services by providing APIs in your preferred language, and they include Amazon libraries, code samples, and documentation. 

Amazon Managed Service for Apache Flink supports Flink’s flexible APIs in Java, Scala, Python, and SQL that are specialized for different use cases including stateful event processing, streaming ETL, and real-time analytics. With prebuilt operators and analytics capabilities, you can build an Apache Flink streaming application in hours instead of months, and the libraries are extensible, so you can perform real-time processing for a wide variety of use cases.

You can set up and integrate a data source or destination with minimal code. Use the Amazon Managed Service for Apache Flink libraries to integrate with the following Amazon Web Services services: 

• Amazon Simple Storage Service (S3)
• Amazon Managed Streaming for Apache Kafka (Amazon MSK)
• Amazon OpenSearch Service
• Amazon DynamoDB
• Amazon Kinesis Data Streams
• Amazon Kinesis Data Firehose
• Amazon CloudWatch
• Amazon Glue Schema Registry

In addition to the Amazon Web Services integrations, the Amazon Managed Service for Apache Flink libraries include more than 40 Apache Flink connectors and the ability to build custom integrations. With a few more lines of code, you can modify how each integration behaves with advanced functionality. You can also build custom integrations using a set of Apache Flink primitive types so that you can read and write from files, directories, sockets, or other sources accessed over the internet.

Amazon Managed Service for Apache Flink supports machine learning (ML) algorithms. You can create real-time applications for classification, clustering, evaluation, feature engineering recommendations, regressions, and statistics. 

Amazon Managed Service for Apache Flink is compatible with the Amazon Glue Schema Registry. With this serverless Amazon Glue feature, you can validate and control the evolution of streaming data using registered Apache Avro schemas, at no additional charge. The Schema Registry helps you manage your schemas on Amazon Managed Service for Apache Flink workloads connected to Apache Kafka, Amazon Managed Streaming for Apache Kafka (Amazon MSK), or Amazon Kinesis Data Streams, as either a source or sink connector. When data streaming applications are integrated with the Schema Registry, you can improve data quality and safeguard against unexpected changes using compatibility checks that govern schema evolution.

Using Amazon Managed Service for Apache Flink, you can build applications where processed records affect the results exactly once, referred to as exactly-once processing. Even in the case of an application disruption, such as internal service maintenance or user-initiated application update, the service ensures all data is processed and there is no duplicate data.

The service stores previous and in-progress computations, or state, in running application storage. Compare real-time and past results over any time period and achieve fast recovery during application disruptions. State is always encrypted and incrementally saved in running application storage.

You can create and delete durable application backups through API calls. Immediately restore your applications from the latest backup after a disruption, or restore your application to an earlier version.