Posted On: Sep 14, 2021
Amazon OpenSearch Service now supports data streams to help simplify management of time-series data such as logs, metrics, and traces. Data streams abstract the underlying indexes required for your time-series data, the rollover process, and the optimizations required to efficiently manage and query time-based data, reducing operational overhead. You can move your older rolled-over indexes that are part of a data stream to UltraWarm and beyond that to cold storage, helping you retain data for longer, cost-effectively.
Previously, a typical workflow to manage time-series data involved multiple steps such as creating a rollover index alias, defining a write index, and defining common mappings and settings for the backing indices. Data streams help simplify this initial setup process, and are designed to work out of the box for time-based data such as application logs that are typically append-only in nature. With data streams, once you define an index template, and configure the rollover criteria in an Index State Management (ISM) policy, you can start ingesting data straight away. Data streams can help automatically create the necessary write index, backing indices, and based on your ISM policy, will rollover indexes in a data stream from hot to warm or warm to cold.
Data Streams are supported for Amazon OpenSearch Service domains running OpenSearch 1.0, a community-driven, open source search and analytics suite derived from Apache 2.0 licensed Elasticsearch 7.10.2 & Kibana 7.10.2. To learn more about OpenSearch, please visit https://opensearch.org/.
To learn more about Data Streams, please see the documentation.
Data Streams is now available for Amazon OpenSearch Service domains in the Amazon Web Services China (Beijing) region, operated by Sinnet, and the Amazon Web Services China (Ningxia) region, operated by NWCD.