Mansi
HOMEWORKASKABOUTCONTACT
HOMEWORKASKABOUTCONTACT
Resume

Home/Work

Project 01

Legacy Data Modernization

A layered refinery — raw feed clarified into curated serving.

Lead Data Engineer · Data Platforms · Cloud Architecture · ETL ModernizationData Engineering · Cloud · Analytics

AWS S3AWS GlueRedshiftRedshift SpectrumLambdaDMS
Legacy data platformSources land in Bronze, Glue refines Silver, Gold serves Redshift, then reporting.SourcesBronzeSilverGoldReportingDevicesfiles / loadsMySQLincrementalLambdaingestS3 rawBronze lakeDMSCDC / loadDirect ConnectEventBridgeruleStep FunctionsGlue / PySparkclean + filterS3 cleanSilverOrchestrationEventBridge + SFNGlue jobsmarts + catalogRedshiftservingSpectrumPower BIQuickSightNL modelStreamingCloudWatch · IAM · KMS · Secrets
Sources land in Bronze, Glue refines Silver, Gold serves Redshift, then reporting.

01 — The problem

Legacy shell-script ETL, fragmented siloed data, constrained dashboard performance, and limited self-service discovery for internal users.

02 — The system

Select a layer to see its role.

Data flow

  1. 01IngestData from partner databases and files enters the platform.
  2. 02ProcessAWS Glue / PySpark workflows clean and transform the data.
  3. 03CurateData is organized into analytical models and data marts.
  4. 04ServeRedshift and BI tools provide consumption and reporting.

03 — The decision

Decision

AWS Glue + PySpark.

Why

Move from fragile script-driven processing toward managed, repeatable ETL workflows.

Context

Reliability and maintenance.

Decision

S3 for durable data lake storage + Redshift for analytical serving.

Why

Separate scalable storage from query-optimized consumption.

Context

Storage and analytics had competing concerns.

Decision

Layered data architecture.

Why

Establish clearer boundaries between raw, standardized, and curated data.

Context

Data processing lacked clear separation.

Decision

Refactor self-service tooling and improve Power BI consumption.

Why

Reduce friction between the data platform and business users.

Context

Users depended heavily on existing workflows for data access.

04 — The trade-off

Optimized for

  • Scalable centralized data
  • Modern ETL operability
  • Layered analytics consumption

Sacrificed

  • Minimal moving parts
  • Fastest possible lift-and-shift

Technology — why it was used

S3

Durable, scalable object storage for the centralized data lake.

AWS Glue

Managed ETL platform suited to replacing legacy script-driven workflows.

PySpark

Distributed transformation capability for larger ETL workloads.

Redshift

Analytical serving layer for structured reporting and downstream consumption.

Power BI

Business-facing analytics and reporting layer.

05 — Evidence

A layered architecture separated ingestion, transformation, analytical storage, and consumption.
A layered architecture separated ingestion, transformation, analytical storage, and consumption.

Takeaway

Boundaries between lake, warehouse, and BI matter more than moving ETL onto AWS alone.

  • Bronze → Silver → Gold creates clearer reuse than ad-hoc transforms.
  • Self-service tooling is product work alongside core pipelines.

What landed

  • Achieved centralized and scalable data management through an S3 data lake.
  • Migrated legacy shell-script ETL processes to AWS Glue jobs.
  • Improved dashboard performance through an optimized database backend.
  • Delivered internal self-service data workflows for operational updates.

What I would do next

  • Formalize consumer contracts earlier between lake, warehouse, and BI layers.
  • Expand automated lineage and replay documentation across Glue workflows.
BackAll workAll workNextML-Driven Allocation Engine
Mansi

© 2026 Mansi Dhruv. All rights reserved.

Created with curiosity