Revisit Amazon Web Services re:Invent 2024’s biggest moments and watch keynotes and innovation talks on demand
Q. What is Amazon IoT Analytics?
Amazon IoT Analytics is a fully-managed IoT analytics service that collects, pre-processes, enriches, stores, and analyzes IoT device data at scale. IoT Analytics can perform simple ad hoc queries as well as complex analysis, and is a simpler way to run IoT analytics for understanding the performance of devices, predicting device failures, and machine learning. It is designed specifically for IoT and automatically captures and stores the message timestamp making time-series analysis easy. IoT Analytics can also enrich the data with device-specific metadata such as device type and location using the Amazon IoT registry. Amazon IoT Analytics stores data in an IoT-optimized data store so you can run queries on large datasets. With Amazon IoT Analytics, you can also execute custom authored code containers. Amazon IoT Analytics automates the execution of your custom analyses created in Jupyter Notebook or your own tools (such as Matlab) for scheduled execution.
Q. How does Amazon IoT Analytics work?
Amazon IoT Analytics is fully integrated with Amazon IoT Core so it is easy to get started. First, you define a channel and select the data you want to collect so you only store and analyze the data of interest, such as sensor temperature. Once the channel is set up, you configure pipelines to process your data. Pipelines support transformations like Celsius to Fahrenheit conversion, conditional statements, message filtering, and message enrichment using external data sources and Amazon Lambda functions.
After processing the data in the pipeline, IoT Analytics stores it in an IoT-optimized data store for analysis. You can query the data using the built-in SQL query engine to answer specific business questions. For example, you may want to know how many monthly active users there are for each device in your fleet. Through integration with Amazon SageMaker, IoT Analytics supports more sophisticated analytics, like Bayesian inference and machine learning.
Q. Can I execute my custom analysis code on Amazon IoT Analytics?
Yes, you can bring your own custom analysis packaged in a container to execute on Amazon IoT Analytics. You can import your custom code container in a container data set and schedule the execution of your container.
Q. How is a SQL data set different than a container data set?
A SQL data set is similar to a materialized view from a SQL database. In fact, you create a SQL data set by applying a SQL action. SQL data sets can be generated automatically on a recurring schedule by specifying a trigger.
A container data set allows you to automatically run your analysis tools and generate results. It brings together a SQL data set as input, a Docker container with your analysis tools and needed library files, input and output variables, and an optional schedule trigger. The input and output variables tell the executable image where to get the data and store the results. The trigger can run your analysis when a SQL data set finishes creating its content or according to a time schedule expression. A container data set will automatically run, generate and then save the results of the analysis tools.
Q. What are DeltaTime Windows?
Delta windows are a series of user-defined, non-overlapping and contiguous time intervals. Delta windows allow you to create data set contents with, and perform analysis on, new data that has arrived in the data store since the last analysis. You create a Delta window by setting the DeltaTime in the filters portion of a queryAction of a data set. Basically, this allows you to filter messages that have arrived during a specific time window, so the data contained in messages from previous time windows doesn't get counted twice.
Q. How do I execute my custom code container on Amazon IoT Analytics at my preferred schedule?
You can bring your own custom analysis code packaged in a container to execute on Amazon IoT Analytics. You can create a container data set through the Amazon IoT Analytics console or API by specifying your SQL dataset, custom code container image, input and output variables and an optional schedule trigger as a parameter, where a trigger can be an event of SQL dataset content creation or a schedule. The container data set feeds the content of SQL data set to the analytical code available in the container image and automatically executes the image based on your trigger to generate insights at your preferred schedule.
Q. What retention policies do I have on my Data Stores and Channels?
You can choose to set your own data retention policies on the Data stores and Channels. You can select the retention period for any number of days. If you do not select any retention period, the service will retain your data for unlimited period.
Q. What type of message formats are supported with Amazon IoT Analytics?
Amazon IoT Analytics service supports JSON and Binary formats (Binary JSON, protocol buffers, etc.). In case the messages are binary, the first activity of your Pipeline needs to be the Lambda activity with an output of a valid JSON.
Q. Can I re-process my data from Channel to a Pipeline?
Yes, you can re-process your data from the Channel connected to the Pipeline using StartPipelineReprocessing. If you want to re-process your Channel data using another Pipeline, you will need to connect the Pipeline to your appropriate Channel.
Q. How do I get the data into Amazon IoT Analytics using the Ingestion API?
You can use the BatchPutMessage API to send your data to Amazon IoT Analytics from sources like Amazon S3, Amazon Kinesis or any other data source. You can use this APIs within your Lambda function or any other script to send the data to Amazon IoT Analytics. For more information, please refer to Send data from S3 to IoT Analytics and send data from Kinesis to IoT Analytics.
Q. Can I preview my messages in the Channel?
Yes, you can preview your messages using SampleChannelData API. This API samples up to ten messages from your Channel across all data.
Q. Can I simulate my pipeline activity?
Yes, you can use the RunPipelineActivity API or console to simulate your pipeline activity. This API takes the list of sample data (either from SampleChannelData or supplied manually) to simulate the activity and previews the results.
Q. What are the differences between Amazon IoT Analytics and Amazon Kinesis Analytics?
Amazon IoT Analytics is designed specifically for IoT and automatically captures and stores the message timestamp so it is easy to perform time-series analytics. IoT Analytics can also enrich the data with device-specific metadata such as device type and location using the Amazon IoT registry and other public data sources. IoT Analytics stores the device data in IoT-optimized data store so you can run queries on large datasets.
Amazon Kinesis Analytics is a general-purpose tool designed to easily process streaming data from IoT devices as well as other data sources in real time.
Table 1: Amazon IoT Analytics vs. Kinesis Analytics feature comparison
| Features | Amazon IoT Analytics |
Amazon Kinesis Analytics |
| Storage of time-series data | X | |
| Automatic data partitions by message timestamp and device ID | X | |
| Device-specific data enrichment | X | |
| Queries on large datasets | X | |
| Streaming analytics | X | |
| Real-time processing | Minutes or seconds latencies | Seconds or milliseconds latency |
| Time-windowed operations | X | |
| Parse unstructured data and automatically create schema | JSON and CSV | JSON and CSV |
Q. When do I use Amazon IoT Analytics and when do I use Amazon Kinesis Analytics?
You can use Amazon IoT Analytics for IoT analytics. Some use cases include understanding long-term device performance, business reporting and ad-hoc analysis, and predictive fleet maintenance. IoT Analytics is best suited for these use cases because it collects, prepares, and stores data from devices over long timeframes in an IoT-optimized data store. IoT Analytics also enriches the data with device-specific metadata such as device type and location using the Amazon IoT registry and other public data sources.
However, if you need to analyze IoT data in real-time for use cases such as device monitoring, you can use Amazon Kinesis Analytics.
Table 2: Amazon IoT Analytics vs. Kinesis Analytics use cases
| Use Case |
Amazon IoT Analytics | Amazon Kinesis Analytics |
| Understanding Long-Term Device Performance Characteristics | Yes. Enrich IoT data with IoT-specific metadata such as device type and location using Amazon IoT registry and other public data sources. For example, vineyard operators need to enrich humidity sensor data with expected rainfall at the vineyards so they know when to water crops. | No. Suited for real-time, streaming analytics. |
| Business Reporting and Ad-Hoc Analysis on IoT Data | Yes. Collect, process, and store IoT data and integrate with Amazon Web Services QuickSight to build dashboards and reporting or use built-in SQL query engine for ad-hoc queries. For example, aggregate sensor failures across a fleet to report on fleet performance every week. | No. Suited to perform streaming queries on IoT data, such as generating alerts when a sensor fails. |
| Predictive Fleet Maintenance | Yes. Collect, process, and store IoT data and use pre-built templates to build and deploy predictive models. For example, predicting when HVAC systems will fail on connected vehicles so the vehicle can be rerouted and docking expedited to prevent shipment damage. | No. Predictive maintenance requires a historical analysis on long-term data to build models. |
| Real-Time Device Monitoring | No. | Yes. Kinesis Analytics can aggregate data over time windows continuously, detect anomalies, and take actions such as sending alerts. For example, Kinesis Analytics can calculate rolling 10-second averages of valve temperatures every 5 minutes in industrial equipment, and detect when the temperature exceeds certain preset thresholds. It can then alert control systems to automatically shut off machinery, avoiding accidents. |
Q. When do I use Amazon IoT Analytics and Amazon Kinesis together?
Use Amazon IoT Analytics and Amazon Kinesis together when you need both historical and real-time analytics. For example, use Kinesis Analytics to calculate 10-second rolling averages of valve temperatures in industrial equipment to detect when the temperature exceeds certain thresholds. Kinesis Analytics can then alert control systems to automatically shut off machinery, avoiding accidents. Then, use Kinesis Streams to send data to IoT Analytics. You use IoT Analytics to understand trends and also predict when valves should be replaced or serviced.
Q. When working with IoT data, when should I use Amazon IoT Analytics vs. Amazon Kinesis Streams, Amazon Kinesis Analytics and Amazon Kinesis Firehose?
Customers can use Amazon Lambda to send data from Amazon Kinesis Streams to an Amazon IoT data channel and then to Amazon IoT Analytics.
Amazon Kinesis Analytics is designed for streaming analytics, while IoT Analytics is designed for analytics on data at rest. Customers who need both real-time and IoT analytics can use a combination of Kinesis Analytics and IoT Analytics.
Amazon Kinesis Firehose is the easiest way to load streaming data into Amazon Web Services data stores Amazon S3, Amazon Redshift, and Amazon Elasticsearch Service, enabling near real-time analytics with existing business intelligence tools. IoT Analytics does not support Kinesis Firehose as a data source.
Discover more Amazon IoT Analytics pricing
