Posted On: Feb 28, 2022
Amazon Redshift launches Streaming Ingestion support for Kinesis Data Streams (KDS). Amazon Redshift’s Streaming Ingestion eliminates the need to stage data in Amazon S3 before ingesting it into Amazon Redshift, enabling customers to achieve low latency in the seconds while ingesting hundreds of megabytes of streaming data per second into their data warehouse.
Data engineers, data analysts, and big data developers are evolving their analytics from batch to real-time, adopting streaming engines like Amazon Kinesis, to implement near real-time responsive logic and analytics on streaming application data. Currently, customers who want to ingest real-time data, from services like Amazon Kinesis, into Amazon Redshift must first stage the data in Amazon S3 and use the COPY command, which achieves latency in the minutes. With the new streaming ingestion capability in Amazon Redshift, you can use SQL (Structured Query Language) within Redshift to provide the ability to connect to and directly ingest data from multiple Kinesis data streams simultaneously. Amazon Redshift streaming ingestion simplifies data pipelines by letting you create materialized views on top of streams directly. The materialized views can also include SQL transforms as part of your ELT (Extract Load Transform) pipeline. Once the materialized views are defined, you can manually refresh them to allow you to query the most recent stream data. This approach allows you to perform downstream processing and transformations of streaming data using existing tools and SQL that you are familiar with, at no additional cost.
Amazon Redshift Streaming Ingestion preview is now available in Amazon Web Services China (Beijing) Region, operated by Sinnet and Amazon Web Services China (Ningxia) Region, operated by NWCD. To get started with Amazon Redshift Streaming Ingestion, provision an Amazon Redshift cluster on the CURRENT Track and verify your cluster is running version 1.0.35480 or newer. Please note, preview features are provided primarily for evaluation and testing purposes. They should not be used in production systems or with production data. For preview terms and conditions, see Beta Service Participation in Sinnet Service Terms for Amazon Web Services (Beijing Region) or Western Cloud Data Service Terms for Amazon Web Services (Ningxia Region). For more information on how to get started with Amazon Redshift Streaming Ingestion, see the Amazon Redshift documentation page. If you have any questions or concerns, please contact the Amazon Web Services China Support.