# AWS Device Farm: Testing Software on Real Devices

Key Takeaways

  • AWS Device Farm runs your mobile and web app tests on physical Android, iOS, and desktop browser devices hosted by AWS — no device lab required.
  • It supports Appium, Espresso, XCUITest, Calabash, and built-in "Explorer" fuzz testing, plus remote manual access sessions.
  • Pricing is pay-per-device-minute or an unlimited monthly plan, which matters a lot for CI pipelines running tests frequently.
  • Device Farm plugs directly into CodePipeline/CodeBuild for automated test gates on every build.

What AWS Device Farm Actually Does

AWS Device Farm is a managed testing service that lets you run automated test suites, or interact manually, with apps on real physical devices sitting in AWS data centers. The core problem it solves: emulators and simulators don't catch every bug — screen size quirks, OEM-specific Android skins, carrier network behavior, and hardware sensor interactions only show up on actual hardware. Maintaining an in-house device lab across dozens of Android models and iOS versions is expensive and a maintenance burden; Device Farm rents that infrastructure by the minute.

Supported Test Types

Device Farm supports several test frameworks out of the box:

  • Appium (Java, Python, Node, Ruby) — cross-platform UI automation
  • Espresso and UI Automator — native Android
  • XCUITest — native iOS
  • Calabash — legacy cross-platform
  • Built-in Fuzz / Explorer — automatically interacts with the app to surface crashes without writing test scripts
  • Remote Access — a live, interactive session on a real device through your browser, useful for manual exploratory testing

Device Farm Test Flow

<svg viewBox="0 0 640 220" xmlns="http://www.w3.org/2000/svg" style="width:100%;max-width:640px;font-family:sans-serif;font-size:13px">
  <style>
    .box{fill:none;stroke:currentColor;stroke-width:1.5}
    .lbl{fill:currentColor;text-anchor:middle}
    .arrow{stroke:currentColor;stroke-width:1.5;marker-end:url(#arr2)}
  </style>
  <defs>
    <marker id="arr2" markerWidth="8" markerHeight="8" refX="6" refY="3" orient="auto">
      <path d="M0,0 L6,3 L0,6 Z" fill="currentColor"/>
    </marker>
  </defs>
  <rect class="box" x="10" y="80" width="130" height="50" rx="6"/>
  <text class="lbl" x="75" y="102">App Binary</text>
  <text class="lbl" x="75" y="118">(.apk / .ipa)</text>
  <line class="arrow" x1="140" y1="105" x2="190" y2="105"/>
  <rect class="box" x="195" y="80" width="130" height="50" rx="6"/>
  <text class="lbl" x="260" y="102">Device Farm</text>
  <text class="lbl" x="260" y="118">Upload + Schedule</text>
  <line class="arrow" x1="325" y1="105" x2="375" y2="105"/>
  <rect class="box" x="380" y="20" width="130" height="50" rx="6"/>
  <text class="lbl" x="445" y="42">Real Android</text>
  <text class="lbl" x="445" y="58">Devices</text>
  <line class="arrow" x1="325" y1="105" x2="375" y2="105"/>
  <rect class="box" x="380" y="140" width="130" height="50" rx="6"/>
  <text class="lbl" x="445" y="162">Real iOS</text>
  <text class="lbl" x="445" y="178">Devices</text>
  <line class="arrow" x1="380" y1="45" x2="325" y2="95"/>
  <line class="arrow" x1="380" y1="165" x2="325" y2="115"/>
  <line class="arrow" x1="510" y1="45" x2="560" y2="90"/>
  <line class="arrow" x1="510" y1="165" x2="560" y2="120"/>
  <rect class="box" x="560" y="80" width="70" height="50" rx="6"/>
  <text class="lbl" x="595" y="102">Results</text>
  <text class="lbl" x="595" y="118">+ Logs</text>
</svg>

Running a Test via CLI

Here's a minimal workflow using the AWS CLI to upload an app and schedule a run.

# 1. Create a project
aws devicefarm create-project --name "MyApp-QA" --region us-west-2

# 2. Create an upload for your app binary
aws devicefarm create-upload \
  --project-arn arn:aws:devicefarm:us-west-2:123456789012:project:abc-123 \
  --name app-release.apk \
  --type ANDROID_APP

# 3. Upload the actual file (presigned URL returned by step 2)
curl -T app-release.apk "<presigned-url-from-step-2>"

# 4. Schedule a run against a device pool
aws devicefarm schedule-run \
  --project-arn arn:aws:devicefarm:us-west-2:123456789012:project:abc-123 \
  --app-arn arn:aws:devicefarm:us-west-2:123456789012:upload:abc-123/app \
  --device-pool-arn arn:aws:devicefarm:us-west-2:123456789012:devicepool:abc-123/pool \
  --name "release-1.2.0-regression" \
  --test type=APPIUM_JAVA_TESTNG,testPackageArn=arn:aws:devicefarm:us-west-2:123456789012:upload:abc-123/tests

Integrating into a CI/CD Pipeline

Most teams wire Device Farm into CodePipeline so that every merge to main triggers a device test run automatically.

# buildspec.yml (AWS CodeBuild)
version: 0.2
phases:
  build:
    commands:
      - ./gradlew assembleRelease
      - aws devicefarm create-upload --project-arn $PROJECT_ARN --name app-release.apk --type ANDROID_APP > upload.json
      - UPLOAD_URL=$(jq -r '.upload.url' upload.json)
      - curl -T app/build/outputs/apk/release/app-release.apk "$UPLOAD_URL"
      - aws devicefarm schedule-run --project-arn $PROJECT_ARN --app-arn $(jq -r '.upload.arn' upload.json) --device-pool-arn $POOL_ARN --name "ci-run-$CODEBUILD_BUILD_NUMBER" --test type=INSTRUMENTATION,testPackageArn=$TEST_PACKAGE_ARN
artifacts:
  files:
    - '**/*'

A failed Device Farm run can be wired to fail the CodeBuild stage, blocking a bad build from reaching production.

Device Pools: Curated vs. Custom

AWS provides curated device pools (e.g., "Top Devices", "Android 13 only"), but you can also build a custom pool targeting specific models and OS versions that match your actual user base — pulled from your analytics, not a generic "top 10 phones" list.

aws devicefarm create-device-pool \
  --project-arn arn:aws:devicefarm:us-west-2:123456789012:project:abc-123 \
  --name "TargetMarketDevices" \
  --rules '[{"attribute":"OS_VERSION","operator":"GREATER_THAN_OR_EQUALS","value":"\"12\""}]'

Pricing Considerations

Device Farm bills in two modes:

  • Pay-as-you-go: per device-minute, good for occasional or exploratory testing.
  • Unmetered plan: a flat monthly rate per device slot, which becomes cheaper once you're running frequent CI test suites across multiple devices in parallel.

Teams running nightly regression suites across 10+ device/OS combinations almost always come out ahead on the unmetered plan.

FAQ

Q: Does Device Farm support iOS testing? A: Yes — XCUITest and Appium against real iPhones/iPads are both supported, alongside Android.

Q: Can I use Device Farm without writing automated tests? A: Yes, via Remote Access sessions — you get an interactive live session on a real device for manual exploratory testing.

Q: How does Device Farm differ from Firebase Test Lab? A: Both offer real-device cloud testing; Device Farm integrates natively with AWS CI/CD (CodePipeline, CodeBuild) and IAM, which matters if your stack is already AWS-centric.

Q: What happens if a test crashes a device? A: Device Farm automatically resets device state between runs, so a crash in one test doesn't contaminate the next.

Get Reliable Mobile QA with F9infotech

Setting up a repeatable, CI-integrated AWS Device Farm pipeline takes real DevOps investment — device pool strategy, artifact handling, and failure reporting all need to be designed correctly. F9infotech's AWS DevOps team can build this pipeline for you end to end. Reach out to get started.