AI-Enabled Data Engineering Career Accelerator

Module 1: Data and Coding Foundations

• Data engineering fundamentals and data lifecycle
• Linux commands, Git and GitHub workflows
• SQL fundamentals and advanced SQL
• Python programming and data structures
• API and file ingestion pipelines
• Python logging, error handling and configuration
• Maintainable, modular and testable Python pipelines

Module 2: Data Modelling and Storage

• Relational database modelling and normalisation
• Entity Relationship diagrams and database design
• Primary keys, foreign keys and data constraints
• Dimensional modelling and data warehouse concepts
• Star schema and Snowflake schema
• Slowly Changing Dimensions (SCD Type 1 and Type 2)
• Data lakes, lakehouses and storage architecture
• CSV, JSON and Parquet file formats
• Data partitioning and schema evolution
• Cloud storage and data platform foundations

Module 3: Distributed Processing and Orchestration

• Apache Spark architecture and PySpark fundamentals
• Spark DataFrames, transformations and actions
• Data processing, joins and distributed transformations
• Performance optimisation and data partitioning
• Apache Airflow and workflow orchestration
• DAGs, tasks, dependencies and scheduling
• Pipeline retries, timeouts and failure handling
• Incremental data processing and loading strategies
• Idempotency, backfills and data reconciliation

Module 4: Warehouse, dbt and Data Quality

• Enterprise data warehouse architecture
• Snowflake or equivalent warehouse platform exposure
• dbt Core and SQL-based transformations
• Staging, intermediate and mart data models
• Incremental models and data testing
• Data documentation and lineage
• Data quality rules and validation frameworks
• Source-to-target reconciliation
• Docker, containers and environment configuration
• GitHub Actions and basic CI workflows

Module 5: Production Support and AI Productivity

• Production pipeline monitoring and logging
• Data freshness, volume and schema checks
• Failure investigation and root-cause analysis
• Incident management and recovery practices
• Pipeline performance and cost optimisation basics
• Prompt engineering and responsible GenAI usage
• AI-assisted SQL and Python development
• AI-assisted debugging, testing and documentation
• Validation of AI-generated code and recommendations

Module 6: Enterprise Capstone and Career Readiness

• Business requirements and technical architecture design
• Source-to-target mapping and data modelling
• End-to-end data ingestion and transformation pipelines
• PySpark, Airflow and dbt integration
• Data quality, testing and pipeline monitoring
• Enterprise capstone development and deployment
• Technical documentation and project defence
• GitHub portfolio and project storytelling
• Resume, LinkedIn and interview preparation