AWS Glue + PySpark.
WhyMove from fragile script-driven processing toward managed, repeatable ETL workflows.
ContextReliability and maintenance.
A layered refinery — raw feed clarified into curated serving.
01 — The problem
Legacy shell-script ETL, fragmented siloed data, constrained dashboard performance, and limited self-service discovery for internal users.
02 — The system
Select a layer to see its role.
Data flow
03 — The decision
Move from fragile script-driven processing toward managed, repeatable ETL workflows.
ContextReliability and maintenance.
Separate scalable storage from query-optimized consumption.
ContextStorage and analytics had competing concerns.
Establish clearer boundaries between raw, standardized, and curated data.
ContextData processing lacked clear separation.
Reduce friction between the data platform and business users.
ContextUsers depended heavily on existing workflows for data access.
04 — The trade-off
Optimized for
Sacrificed
Technology — why it was used
S3
Durable, scalable object storage for the centralized data lake.AWS Glue
Managed ETL platform suited to replacing legacy script-driven workflows.PySpark
Distributed transformation capability for larger ETL workloads.Redshift
Analytical serving layer for structured reporting and downstream consumption.Power BI
Business-facing analytics and reporting layer.05 — Evidence

Takeaway
Boundaries between lake, warehouse, and BI matter more than moving ETL onto AWS alone.
What landed
What I would do next