Creating a near-realtime dashboard on Amazon CloudWatch for a Migration usecase

Monitoring performance metrics of Amazon Web Services resources is crucial for any business use case running in cloud. Amazon Web Services Well-Architected Framework best practices recommend customers to setup observability and monitoring for their infrastructure on Amazon Web Services. But, as Amazon Web Services usage for customer increases, it gets harder to observe and visualize the data and metrics at scale and identify any performance blockers. Amazon Web Services provides readily available Amazon CloudWatch metrics for Amazon Web Services resources that are system-defined . Customers can also create custom metrics on Amazon CloudWatch that are user-defined . But over the years, customers have found it hard to visualize all these metrics together at one place without the right visualization tool.

In this blogpost, you will learn how to visualize metrics for several resources on a single Amazon CloudWatch dashboard and make it scalable. We will create a near-realtime dashboard on Amazon CloudWatch with Migration as the use case. The user-defined metrics will be derived from the Amazon Web Services service used for Migration. In this blog, we discuss how to automatically put and update the system-defined and user-defined CloudWatch metrics on the Amazon CloudWatch dashboard.

Architecture

Figure 1. Architecture diagram for creating a near-realtime Amazon CloudWatch Dashboard

This is the architectural diagram for creating a near-realtime dashboard on Amazon CloudWatch for Migration use case. In this blog, we will be monitoring the Amazon Web Services Application Migration Service (Amazon Web Services MGN). Amazon Web Services MGN is a highly automated move and improve migration service that can migrate physical, virtual, or any cloud servers “as is” to the Amazon Web Services cloud.

1. The solution above starts with an Amazon EventBridge cron rule that runs every 1 minute to invoke the Amazon Web Services Lambda function.

2. The Amazon Web Services Lambda function will run and gather Amazon CloudWatch metrics from Amazon Web Services MGN for monitoring.

3. The Amazon Web Services Lambda function will then publish these metrics on to a Amazon CloudWatch dashboard. This dashboard will be used by the viewer to see performance metrics or other statistics. The red dotted box in the architecture diagram above can be generalized for any use case.

Solution Overview

1. The Lambda code here in python is customized for the Migration use case. You will be using the following code in this lambda function to push system-defined and user-defined metrics to Amazon CloudWatch dashboard.

Note – The lambda function is compatible with python 3.0 or above. This function is designed to receive metrics from Amazon Web Services MGN and create custom CloudWatch metrics from it. It is designed to create test, cutover and replication state custom metrics and push them on to Amazon CloudWatch. Then, the lambda function is creating an Amazon CloudWatch dashboard and including system-defined metrics provided by Amazon Web Services MGN service and user-defined custom metrics generated by the Lambda function itself. If you are using the above code as is, then please make sure to replace <region> with a specific Amazon Web Services region such as us-east-1 , or us-west-2 etc.

2. Zip the above Lambda function or the custom Lambda function for your use case and upload it to an S3 bucket of your choice. Name the zip file as “my-deployment-package.zip”. Use this link to learn how to zip a python file.

3. Use the YAML template here to launch the infrastructure through Amazon Web Services Cloudformation in the Amazon Web Services region selected in the step 1. In this YAML template, replace <s3-bucket> with the s3 bucket name derived from step 2. Deploy this YAML template through Amazon Web Services Cloudformation, and it will launch the Lambda function, Amazon Web Services Event Bridge rule and necessary Lambda roles and permissions.

Make sure that the Amazon Web Services region you choose to deploy the Cloudformation YAML template matches the region in your Lambda function code and the S3 bucket from step 2.

Note that the YAML template creates a Lambda role with the policy that is required to access the Amazon Web Services MGN service.

4. Once the environment is deployed by Amazon Web Services CloudFormation, go to Find Services and search for CloudWatch.

5. On the left navigation pane choose Dashboards , then choose MGN-Dashboard . If you have source machines added in the Amazon Web Services MGN console, then you will see 5 widgets in total. We will add one more in the following steps.

6. In the top right corner, choose the + (plus) button to add a new widget.

7. Choose Number .

8. Search for mgn in the search bar and choose Amazon Web Services MGN under Custom namespaces.

9. Choose Across all Source Servers.

10. Select the first two metrics: MachinesFullySynced and MachinesSyncing . Then select the Graphed metrics tab. Change the statistic for both the metrics to Maximum and Period to 1 second as shown below.  Select Create Widget .

11. Similarly, you can add other custom metrics (generated by the lambda function) as widgets to the Amazon CloudWatch dashboard. You can add more metrics to this dashboard after it is created which makes it scalable. The Eventbridge rule runs every 1 min to update this dashboard which also makes it near-realtime.

12. Set the duration for events displayed on dashboard to Custom (5m) interval instead of default (3h) so that dashboard only shows data for all metrics during the last 5 minutes.

13. Choose Save dashboard.

Result

The CloudWatch dashboard is near-realtime that has all the metrics that you added for visualizing the status of your Migration. This CloudWatch dashboard is also scalable as you can continue to add more widgets/metrics without making any changes to the existing
settings.

Conclusion

We saw how easy it is to create a near-realtime dashboard on Amazon CloudWatch and make it scalable for adding more metrics to it. Most system-defined CloudWatch metrics are updated automatically but the solution discussed in this blogpost can also be used for periodically updating the user-defined metrics on the Amazon CloudWatch dashboard. Moreover, you can reuse this solution and some of the existing code for other non-migration use cases as well.

The Cloudwatch charges will be minimal for this scenario but please refer to the CloudWatch pricing page for custom use cases.

About the author: