# Amazon FSx for Lustre: High-Performance Storage for HPC and ML Workloads

Key Takeaways

  • Amazon FSx for Lustre is a fully managed version of the open-source Lustre file system, built for workloads needing hundreds of GB/s throughput and sub-millisecond latency.
  • It integrates natively with Amazon S3, letting you link a Lustre file system directly to an S3 bucket for seamless data movement.
  • Common use cases: machine learning training, genomics, financial risk modeling, media rendering, and seismic processing.
  • Two deployment types exist: Scratch (for temporary, high-performance workloads) and Persistent (for long-running workloads needing durability).
  • Pricing scales with provisioned throughput and storage capacity, not data transferred.

What Is Amazon FSx for Lustre?

Amazon FSx for Lustre answers the common question "what is fsx for lustre" directly: it's AWS's managed offering of the Lustre file system — an open-source, parallel file system originally built for supercomputing clusters, now running as a fully managed AWS service.

Lustre is designed to serve thousands of compute clients simultaneously with aggregate throughput that scales linearly as you add storage. This makes amazon fsx for lustre the storage backbone for compute-intensive workloads like deep learning training jobs that need to stream terabytes of training data to hundreds of GPU instances in parallel.

Deployment Types

Scratch File Systems

Designed for temporary, high-performance processing where data doesn't need to survive a hardware failure. Lower cost per GB, higher burst throughput, but no built-in data replication.

Persistent File Systems

Designed for longer-running workloads requiring high availability. Data is replicated within an Availability Zone, and failed file servers are automatically replaced.

aws fsx create-file-system \
  --file-system-type LUSTRE \
  --storage-capacity 1200 \
  --storage-type SSD \
  --subnet-ids subnet-0123456789abcdef0 \
  --security-group-ids sg-0123456789abcdef0 \
  --lustre-configuration '{
    "DeploymentType": "PERSISTENT_2",
    "PerUnitStorageThroughput": 250,
    "DataCompressionType": "LZ4"
  }'

Linking FSx for Lustre to S3

The most powerful feature of fsx lustre is its native S3 integration — you can create a file system that lazily loads objects from an S3 bucket on first access, and write results back to S3 automatically.

aws fsx create-data-repository-association \
  --file-system-id fs-0123456789abcdef0 \
  --file-system-path /training-data \
  --data-repository-path s3://f9-ml-datasets/imagenet/ \
  --s3 '{
    "AutoImportPolicy": {"Events": ["NEW", "CHANGED", "DELETED"]},
    "AutoExportPolicy": {"Events": ["NEW", "CHANGED", "DELETED"]}
  }'

This means your ML training job reads directly from a POSIX-compliant, high-throughput file system that transparently mirrors your S3 dataset — no manual sync scripts required.

Architecture Overview

<svg viewBox="0 0 740 260" xmlns="http://www.w3.org/2000/svg" font-family="Arial, sans-serif" font-size="13" fill="currentColor">
  <defs>
    <marker id="a3" markerWidth="10" markerHeight="7" refX="9" refY="3.5" orient="auto">
      <polygon points="0 0, 10 3.5, 0 7" fill="currentColor"/>
    </marker>
  </defs>
  <style>
    .box { fill: none; stroke: currentColor; stroke-width: 1.6; }
    .arrow { stroke: currentColor; stroke-width: 1.6; fill: none; marker-end: url(#a3); }
  </style>

  <rect class="box" x="30" y="30" width="160" height="60" rx="8"/>
  <text x="110" y="65" text-anchor="middle">Amazon S3</text>
  <text x="110" y="80" text-anchor="middle" font-size="11">(dataset bucket)</text>

  <line class="arrow" x1="190" y1="60" x2="290" y2="60"/>
  <text x="240" y="48" text-anchor="middle" font-size="11">auto import/export</text>

  <rect class="box" x="290" y="20" width="200" height="90" rx="8"/>
  <text x="390" y="55" text-anchor="middle">FSx for Lustre</text>
  <text x="390" y="75" text-anchor="middle" font-size="11">(high-throughput file system)</text>
  <text x="390" y="95" text-anchor="middle" font-size="11">100s GB/s aggregate</text>

  <line class="arrow" x1="390" y1="110" x2="390" y2="160"/>

  <rect class="box" x="230" y="160" width="100" height="60" rx="8"/>
  <text x="280" y="195" text-anchor="middle" font-size="12">GPU Node 1</text>

  <rect class="box" x="350" y="160" width="100" height="60" rx="8"/>
  <text x="400" y="195" text-anchor="middle" font-size="12">GPU Node 2</text>

  <rect class="box" x="470" y="160" width="100" height="60" rx="8"/>
  <text x="520" y="195" text-anchor="middle" font-size="12">GPU Node N</text>
</svg>

Mounting on EC2

sudo amazon-linux-extras install -y lustre
sudo mkdir -p /mnt/fsx
sudo mount -t lustre -o noatime,flock fs-0123456789abcdef0.fsx.us-east-1.amazonaws.com@tcp:/fsxlustre /mnt/fsx

Once mounted, your training scripts read from /mnt/fsx exactly as they would a local POSIX filesystem, with the file system transparently handling object fetches from S3 behind the scenes.

Choosing Throughput and Storage Capacity

FSx for Lustre throughput scales with provisioned capacity — you select a "per-unit storage throughput" value (e.g., 125, 250, 500, or 1000 MB/s per TiB) at creation time. Higher throughput tiers cost more per GB but are essential for GPU-bound training jobs where storage I/O is the bottleneck.

Per-Unit ThroughputUse Case
125 MB/s/TiBLight batch processing
250 MB/s/TiBStandard ML training
500-1000 MB/s/TiBMulti-GPU distributed training, HPC simulations

FAQ

Is fsx for lustre the same as Amazon EFS? No. EFS is a general-purpose NFS file system for shared access across many instances at moderate throughput. FSx for Lustre is purpose-built for extreme parallel throughput and is the better fit for HPC and ML training workloads.

Can I use FSx for Lustre without S3 integration? Yes — you can use it as a standalone high-performance file system without linking a data repository, though most production use cases benefit from S3 integration for durability.

What happens to data in a Scratch deployment if the underlying hardware fails? Data is lost — Scratch deployments don't replicate data, which is why they're recommended only for workloads where source data lives durably elsewhere (like S3) and results are exported before teardown.

How is fsx lustre priced? You pay per GB of provisioned storage per month plus a throughput capacity charge; there's no separate data transfer fee for reads/writes within the file system.

Need a High-Performance Storage Architecture?

f9infotech designs FSx for Lustre architectures for ML training pipelines, HPC clusters, and media rendering farms — sized correctly for your throughput needs from day one. Talk to our AWS infrastructure team about your high-performance computing storage requirements.