AWS DataSync: Accelerating Data Transfer for Software and Hardware Teams

# AWS DataSync: Accelerating Data Transfer for Software and Hardware Teams

Key Takeaways

• AWS DataSync moves large datasets between on-premises storage (NFS, SMB, HDFS, object storage) and AWS services like S3, EFS, and FSx, at up to 10x the speed of standard copy tools.
• It handles the hard parts automatically: encryption in transit, data integrity verification, retry logic, and bandwidth throttling.
• Pricing is per-GB transferred, with no agent licensing cost — the DataSync agent itself is free to deploy.
• It's built for one-time migrations and recurring sync jobs alike, which makes it useful for both software teams migrating storage and hardware teams syncing sensor/test data.

What Is AWS DataSync?

AWS DataSync is a managed data transfer service that moves files and objects between on-premises storage systems and AWS storage services — or between AWS storage services themselves — without you having to write custom transfer scripts, manage SSH tunnels, or babysit rsync jobs over a flaky VPN.

It solves a problem every team with legacy file storage eventually hits: moving terabytes of data to the cloud over the internet is slow, error-prone, and painful to automate reliably. DataSync's agent reads data in parallel, verifies every byte transferred, retries automatically on failure, and reports metrics to CloudWatch, so you're not guessing whether a 40TB transfer actually completed intact.

How AWS DataSync Works

<svg viewBox="0 0 780 240" xmlns="http://www.w3.org/2000/svg" style="width:100%;max-width:780px;font-family:sans-serif;color:inherit">
  <style>
    .box { fill: none; stroke: currentColor; stroke-width:1.5; }
    .lbl { fill: currentColor; font-size:13px; }
    .arrow { stroke: currentColor; stroke-width:1.5; marker-end:url(#arrow2); }
  </style>
  <defs>
    <marker id="arrow2" markerWidth="10" markerHeight="10" refX="8" refY="3" orient="auto">
      <path d="M0,0 L8,3 L0,6 Z" fill="currentColor"/>
    </marker>
  </defs>
  <rect class="box" x="10" y="90" width="160" height="70" rx="8"/>
  <text class="lbl" x="30" y="118">On-Prem NFS/SMB</text>
  <text class="lbl" x="30" y="136">or HDFS/Object store</text>

  <rect class="box" x="220" y="90" width="150" height="70" rx="8"/>
  <text class="lbl" x="245" y="118">DataSync Agent</text>
  <text class="lbl" x="245" y="136">(VM in your network)</text>

  <rect class="box" x="420" y="90" width="150" height="70" rx="8"/>
  <text class="lbl" x="440" y="118">DataSync Service</text>
  <text class="lbl" x="440" y="136">(encryption, retries)</text>

  <rect class="box" x="620" y="30" width="150" height="50" rx="8"/>
  <text class="lbl" x="650" y="60">Amazon S3</text>

  <rect class="box" x="620" y="100" width="150" height="50" rx="8"/>
  <text class="lbl" x="650" y="130">Amazon EFS</text>

  <rect class="box" x="620" y="170" width="150" height="50" rx="8"/>
  <text class="lbl" x="650" y="200">Amazon FSx</text>

  <line class="arrow" x1="170" y1="125" x2="218" y2="125"/>
  <line class="arrow" x1="370" y1="125" x2="418" y2="125"/>
  <line class="arrow" x1="570" y1="110" x2="618" y2="60"/>
  <line class="arrow" x1="570" y1="125" x2="618" y2="125"/>
  <line class="arrow" x1="570" y1="140" x2="618" y2="190"/>
</svg>

The flow is simple: deploy a DataSync agent (a VM image for VMware, Hyper-V, KVM, or a physical hardware appliance) inside your network, point it at your source storage, and tell DataSync where the data should land. The agent handles the read; the managed service handles transport, retries, and the write to AWS.

Setting Up a DataSync Task

Step 1: Deploy the agent

Download the agent VM image from the AWS console and deploy it on your hypervisor. Once it's running, activate it:

aws datasync create-agent \
  --agent-name "onprem-nfs-agent" \
  --activation-key "AAAA-BBBB-CCCC-DDDD" \
  --subnet-arns "arn:aws:ec2:us-east-1:111122223333:subnet/subnet-0abcd1234"

Step 2: Create source and destination locations

# NFS source on-premises
aws datasync create-location-nfs \
  --server-hostname "nfs.internal.example.com" \
  --subdirectory "/exports/data" \
  --on-prem-config AgentArns="arn:aws:datasync:us-east-1:111122223333:agent/agent-0abc123"

# S3 destination
aws datasync create-location-s3 \
  --s3-bucket-arn "arn:aws:s3:::f9infotech-migrated-data" \
  --s3-config BucketAccessRoleArn="arn:aws:iam::111122223333:role/DataSyncS3Role"

Step 3: Create and start the task

aws datasync create-task \
  --source-location-arn "arn:aws:datasync:us-east-1:111122223333:location/loc-nfs-source" \
  --destination-location-arn "arn:aws:datasync:us-east-1:111122223333:location/loc-s3-dest" \
  --name "nfs-to-s3-migration" \
  --options VerifyMode=POINT_IN_TIME_CONSISTENT,TransferMode=CHANGED

aws datasync start-task-execution \
  --task-arn "arn:aws:datasync:us-east-1:111122223333:task/task-0def456"

The TransferMode=CHANGED option means re-running the task only syncs files that changed since the last run — ideal for recurring sync jobs rather than one-time migrations.

DataSync vs. Other Transfer Options

DataSync and Storage Gateway solve different problems: DataSync moves data; Storage Gateway provides live, ongoing access to on-prem storage from AWS without fully migrating it.

Use Cases for Software and Hardware Teams

• Software teams migrating a legacy NFS-backed application's file storage to S3 or EFS as part of a modernization project, with minimal downtime via incremental syncs.
• Hardware/test engineering teams syncing large volumes of sensor logs, firmware test artifacts, or lab equipment output from local NAS devices into S3 for centralized analysis, without writing custom upload scripts.
• Media teams moving raw video/audio assets from on-prem editing storage into S3 for archival or further processing pipelines.
• Compliance-driven migrations where data integrity verification (checksums on every file) needs to be provable, not just assumed.

Monitoring and Cost

DataSync publishes metrics to CloudWatch — bytes transferred, files transferred, files skipped, and transfer duration. Set an alarm on task failures:

aws cloudwatch put-metric-alarm \
  --alarm-name "datasync-task-failures" \
  --metric-name "TaskExecutionResultStatus" \
  --namespace "AWS/DataSync" \
  --statistic Sum \
  --period 300 \
  --threshold 1 \
  --comparison-operator GreaterThanOrEqualToThreshold \
  --evaluation-periods 1

Pricing is per-GB of data copied (no charge for unchanged files on incremental runs), plus whatever AWS storage you write into. There's no license fee for the agent itself.

FAQ

Does AWS DataSync require a permanent agent running on-premises? Only during active transfers if you're doing periodic syncs; for a one-time migration, you can decommission the agent after the final run completes. For recurring sync jobs, keep the agent running and schedule tasks via EventBridge.

Can DataSync transfer data between two AWS regions or accounts, not just from on-prem? Yes — DataSync supports S3-to-S3, EFS-to-EFS, and FSx transfers across regions and accounts, without needing an on-prem agent in that case.

How does AWS DataSync differ from AWS Storage Gateway? DataSync performs a transfer and finishes; Storage Gateway provides ongoing, live access to on-prem data from AWS (or vice versa) without a full migration. Use DataSync to move data; use Storage Gateway to keep using it from both sides.

Is data encrypted during a DataSync transfer? Yes, all data in transit is encrypted using TLS, and DataSync verifies data integrity with checksums after every transfer.

---

Planning a large-scale storage migration or recurring data sync pipeline into AWS? f9infotech's AWS migration and managed services team can design and run the DataSync architecture for you — talk to us about your data transfer project.