# What Is AWS Glue? A Quick Read on Serverless ETL

Key Takeaways

  • AWS Glue is a serverless data integration service for discovering, preparing, and combining data for analytics, ML, and application development.
  • Its core components are the Data Catalog, Crawlers, Jobs (ETL scripts), and Glue Studio (visual job builder).
  • Glue supports Python (PySpark) and Scala jobs, auto-generates ETL code, and scales workers automatically.
  • It integrates tightly with S3, Redshift, RDS, DynamoDB, and Athena.
  • Pricing is based on Data Processing Units (DPUs) billed per second, with a 1-minute minimum per job run.

What Is AWS Glue?

To directly answer "what is aws glue": it's AWS's fully managed, serverless ETL (Extract, Transform, Load) service. It discovers data sitting in various stores, catalogs its schema, and runs transformation jobs — all without you provisioning or managing servers.

Glue is the backbone of many AWS analytics pipelines: raw data lands in S3, a Glue Crawler infers its schema, a Glue Job transforms it, and the result becomes queryable via Athena or loadable into Redshift.

Core Components

Data Catalog

A persistent, centralized metadata repository — effectively a managed Hive Metastore. It stores table definitions (schema, location, format) that other AWS services like Athena, Redshift Spectrum, and EMR can query directly.

Crawlers

Crawlers scan your data stores (S3, JDBC databases, DynamoDB) and automatically populate the Data Catalog with inferred schemas.

aws glue create-crawler \
  --name f9-sales-data-crawler \
  --role arn:aws:iam::123456789012:role/GlueCrawlerRole \
  --database-name sales_db \
  --targets '{"S3Targets": [{"Path": "s3://f9-data-lake/raw/sales/"}]}'

aws glue start-crawler --name f9-sales-data-crawler

Jobs

A Glue Job is a script (PySpark or Scala) that reads from a source, transforms the data, and writes to a target. Glue can auto-generate a starting script, or you write your own.

import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job

args = getResolvedOptions(sys.argv, ["JOB_NAME"])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args["JOB_NAME"], args)

# Read raw sales data from the Data Catalog
datasource = glueContext.create_dynamic_frame.from_catalog(
    database="sales_db", table_name="raw_sales"
)

# Transform: drop nulls and rename a column
mapped = ApplyMapping.apply(
    frame=datasource,
    mappings=[
        ("order_id", "string", "order_id", "string"),
        ("amount", "double", "order_amount_usd", "double"),
        ("order_date", "string", "order_date", "date"),
    ],
)

# Write transformed data back to S3 in Parquet format
glueContext.write_dynamic_frame.from_options(
    frame=mapped,
    connection_type="s3",
    connection_options={"path": "s3://f9-data-lake/processed/sales/"},
    format="parquet",
)

job.commit()

Submitting the Job

aws glue create-job \
  --name f9-sales-etl-job \
  --role arn:aws:iam::123456789012:role/GlueJobRole \
  --command '{
    "Name": "glueetl",
    "ScriptLocation": "s3://f9-scripts/sales_etl.py"
  }' \
  --default-arguments '{"--TempDir": "s3://f9-scripts/temp/"}' \
  --glue-version "4.0" \
  --worker-type G.1X \
  --number-of-workers 5

aws glue start-job-run --job-name f9-sales-etl-job

How Glue Fits into a Data Pipeline

<svg viewBox="0 0 740 240" xmlns="http://www.w3.org/2000/svg" font-family="Arial, sans-serif" font-size="13" fill="currentColor">
  <defs>
    <marker id="a4" markerWidth="10" markerHeight="7" refX="9" refY="3.5" orient="auto">
      <polygon points="0 0, 10 3.5, 0 7" fill="currentColor"/>
    </marker>
  </defs>
  <style>
    .box { fill: none; stroke: currentColor; stroke-width: 1.6; }
    .arrow { stroke: currentColor; stroke-width: 1.6; fill: none; marker-end: url(#a4); }
  </style>

  <rect class="box" x="20" y="90" width="120" height="60" rx="8"/>
  <text x="80" y="125" text-anchor="middle">Raw Data (S3)</text>

  <line class="arrow" x1="140" y1="120" x2="210" y2="120"/>

  <rect class="box" x="210" y="40" width="130" height="60" rx="8"/>
  <text x="275" y="75" text-anchor="middle">Glue Crawler</text>

  <line class="arrow" x1="275" y1="100" x2="275" y2="150"/>

  <rect class="box" x="210" y="150" width="130" height="60" rx="8"/>
  <text x="275" y="185" text-anchor="middle">Data Catalog</text>

  <line class="arrow" x1="340" y1="120" x2="420" y2="120"/>

  <rect class="box" x="420" y="90" width="120" height="60" rx="8"/>
  <text x="480" y="125" text-anchor="middle">Glue ETL Job</text>

  <line class="arrow" x1="540" y1="120" x2="610" y2="120"/>

  <rect class="box" x="610" y="90" width="110" height="60" rx="8"/>
  <text x="665" y="118" text-anchor="middle">Redshift /</text>
  <text x="665" y="136" text-anchor="middle">Athena</text>
</svg>

Glue Studio: The Visual Option

For teams that prefer a no-code approach, Glue Studio provides a drag-and-drop canvas to build ETL jobs, auto-generating the underlying PySpark script. It's a good starting point for simple transformations before hand-tuning performance-critical jobs.

Pricing

AWS Glue bills by Data Processing Unit (DPU) hours, in 1-second increments with a 1-minute minimum per job run. A standard worker (G.1X) is roughly 4 vCPU/16GB and costs about $0.44/DPU-hour. Crawler runs are billed the same way. There's no charge for the Data Catalog itself up to the first million objects stored and accessed per month.

FAQ

Is AWS Glue the same as Apache Spark? Glue ETL jobs run on a managed Apache Spark (or Python Shell) environment — so under the hood, yes, it's Spark, but AWS handles provisioning, scaling, and patching the cluster for you.

Do I need to write code to use AWS Glue? Not necessarily. Glue Studio offers a visual job builder, and Crawlers require no code at all. Hand-written PySpark is typically reserved for complex transformations needing fine control.

Can AWS Glue connect to on-premises databases? Yes, via JDBC connections through a Glue connection object, typically routed through a VPC with network connectivity (VPN or Direct Connect) to your on-prem database.

What file formats does AWS Glue support? CSV, JSON, Parquet, ORC, Avro, and XML are natively supported for both reading and writing.

Need an AWS Data Pipeline Built Right?

f9infotech designs and implements AWS Glue-based ETL pipelines — from crawler setup to optimized PySpark jobs feeding Redshift or Athena. Talk to our data engineering team about modernizing your data pipeline on AWS.