Posted On: Dec 6, 2021
Amazon FSx for Lustre, a service that provides cost-effective, high-performance, scalable file systems for compute workloads, is making it even easier to process data residing in Amazon S3 by enabling your FSx for Lustre file system’s contents to be updated automatically as data is deleted or moved in S3.
Integrated with Amazon S3, FSx for Lustre enables you to easily process S3 datasets with a high-performance file system. Before today, when you linked your file system to an S3 bucket, FSx for Lustre transparently presented S3 objects as files and updated its contents automatically on an ongoing basis as objects were added to or changed in your S3 bucket. With today’s launch, FSx for Lustre provides an additional option designed to automatically update the file system when objects in linked S3 buckets are deleted or moved. This new option allows you to keep the file system contents synchronized as you perform many kinds of updates (adds, changes, deletes, and moves) on the linked S3 buckets, enabling you to run parallel workflows that manipulate data on S3 and on the file system at the same time.
The new functionality is available at no additional cost on all Amazon FSx for Lustre file systems created after July 23, 2020 in Amazon Web Services China (Beijing) Region, operated by Sinnet and Amazon Web Services China (Ningxia) Region, operated by NWCD. You can configure your file system to automatically import S3 updates by using the Amazon Web Services Management Console, CLI, and SDKs. Learn more about using automatic import with FSx for Lustre file systems in our user guide.