News

Apache Kafka-Based Data Streaming Now Available on AWS Cloud

• By David Ramel
• 06/05/2019

Amazon Web Services Inc. (AWS) announced its managed data streaming service based on Apache Kafka is now generally available.

Amazon Managed Streaming for Apache Kafka (Amazon MSK), helps enterprises who want to use the popular open source distributed streaming platform avoid the details of setting up, scaling and managing Kafka clusters for Big Data processing.

"When you run Apache Kafka on your own, you need to provision servers, configure Apache Kafka manually, replace servers when they fail, orchestrate server patches and upgrades, architect the cluster for high availability, ensure data is durably stored and secured, setup monitoring and alarms, and carefully plan scaling events to support load changes," the site says.

AWS says its new managed service handles all those details, incorporating Apache Kafka deployment best practices.

After being introduced as an open preview at the company's re:Invent 2018 conference, the cloud giant has added functionality to Amazon MSK such as:

• Encryption in-transit via TLS between clients and brokers, and between brokers
• Mutual TLS authentication using ACM private certificate authorities
• Support for Kafka version 2.1.0
• 99.9% availability SLA
• HIPAA eligible
• Cluster-wide storage scale up
• Integration with AWS CloudTrail for MSK API logging
• Cluster tagging and tag-based IAM policy application
• Defining custom, cluster-wide configurations for topics and brokers

The new managed service works with several other AWS cloud offerings, such as providing metrics in Amazon CloudWatch. AWS spokesperson Jeff Barr said support will soon be added for AWS CloudFormation, for describing and provisioning infrastructure resources in a cloud environment.

"Your MSK clusters always run within an Amazon VPC managed by the MSK service. Your MSK resources are made available to your own VPC, subnet, and security group through elastic network interfaces (ENIs) which will appear in your account," Barr said in a post that details how to work with the new service.

"Customers can create a cluster in minutes, use AWS Identity and Access Management (IAM) to control cluster actions, authorize clients using TLS private certificate authorities fully managed by AWS Certificate Manager (ACM), encrypt data in-transit using TLS, and encrypt data at rest using AWS Key Management Service (KMS) encryption keys."

Pricing is per Kafka broker-hour and per provisioned storage-hour, with AWS data transfer rates applying for data transfer in and out of MSK.