Amazon Managed Streaming for Apache Kafka Documentation

Amazon Managed Streaming for Apache Kafka (Amazon MSK) supports the deployment of native versions of Apache Kafka so applications and tools built for Apache Kafka work with Amazon MSK.

Amazon MSK typically makes newer versions of Apache Kafka available within days of public availability.

You can upgrade Apache Kafka versions on Amazon MSK clusters, allowing you to decide when to take advantage of features and bug fixes present in new Apache Kafka versions. Amazon MSK helps with automating the deployment of version upgrades on running clusters to help customers maintain client I/O availability for customers following best practices.

With a few clicks in the console, you can create a fully managed Apache Kafka cluster that is designed to follow Apache Kafka’s deployment best practices, or you can create your own cluster using your own custom configuration. Once you create your desired configuration, Amazon MSK is designed to provision, configure, and manage the operations of your Apache Kafka cluster and Apache ZooKeeper nodes.

Apache ZooKeeper is required to run Apache Kafka, coordinate cluster tasks, and maintain state for resources interacting with the cluster. Amazon MSK is designed to manage the Apache ZooKeeper nodes for you. Each Amazon MSK cluster is designed to include the appropriate number of Apache ZooKeeper nodes for your Apache Kafka cluster.

The service is designed so that all clusters are provisioned across multiple availability zones (three availability zones is default), and are supported by systems that seek to detect and respond to issues within cluster infrastructure and Apache Kafka software. If a component fails, Amazon MSK is designed to replace it without downtime to your applications. Amazon MSK assists in the management of the availability of your Apache ZooKeeper nodes so you don’t need to start, stop, or directly access the nodes yourself. Amazon MSK is designed to deploy software patches as needed to help you keep your cluster up-to-date and running smoothly.

Amazon MSK is designed to use multi-AZ replication for high-availability.

Your Apache Kafka clusters are designed to run in an Amazon VPC managed by Amazon MSK. Your clusters are designed to be available to your own Amazon VPCs, subnets, and security groups based on the configuration you specify. The service is designed so that you can control your network configuration and IP addresses from your VPCs that are attached to your Amazon MSK resources through elastic network interfaces (ENIs).

By using IAM Access Control, you no longer need to build and run one-off access management systems to control client authentication and authorization for Apache Kafka and your clusters are secured using least-privileged permissions. You also can use SASL/SCRAM or mutual TLS authentication with Apache Kafka access control lists (ACLs).

Amazon MSK is designed to encrypt your data at rest without special configuration or third-party tools. Data can be encrypted at rest using Amazon Key Management Service (KMS) Customer Master Key (CMK), or your own CMK. Amazon MSK is also designed to encrypt data in-transit via TLS between brokers and between clients and brokers on your cluster.

You can scale your Amazon MSK clusters by changing the size or family of your Apache Kafka brokers. Changing the size or family of your brokers is a popular way to scale Amazon MSK clusters because it gives you the flexibility to adjust your MSK cluster’s compute capacity for changes in your workloads. This method can be preferred because it does not require partition reassignment which can impact Apache Kafka availability.

Amazon MSK integrates with Cruise Control, a popular open source tool for Apache Kafka that manages partition assignment on your behalf.

You can scale up the amount of storage provisioned per broker to match changes in storage requirements using the Amazon management console or Amazon CLI or you can create an auto scaling policy to expand your storage to meet your streaming requirements.

Amazon MSK is designed to deploy a best practice cluster configuration for Apache Kafka, and gives customers the ability to tune more than 30 different cluster configurations while supporting all dynamic and topic-level configurations.

You can visualize and monitor important cluster, broker, topic, consumer, and partition-level metrics using Amazon CloudWatch.

For additional information about service controls, security features and functionalities, including, as applicable, information about storing, retrieving, modifying, restricting, and deleting data, please see https://docs.amazonaws.cn/en_us/. This additional information does not form part of the Documentation for purposes of the Sinnet Customer Agreement for Amazon Web Services (Beijing Region), Western Cloud Data Customer Agreement for Amazon Web Services (Ningxia Region) or other agreement between you and Sinnet or NWCD governing your use of services of Amazon Web Services China Regions.