[Udemy] Data Engineering 101: The Beginner's Guide - Undercurrents

DataOps

• DevOps for data
  • DevOps : deploy software in a more iterative & robust manner
    • build, manage cloud infra
    • observability of cloud infra
    • build automated CI(Continuous Integration)/CD(Continuous Deployment) Pipeline
  • DataOps : data product deployments more iterative and robust
    • build, manage cloud infra for data tools
    • observability of data systems(incident reporting and notifications of problems)
    • automation including CI/CD
• observability and monitoring : critical
  • Why? data problems often go unnoticed!
• Automation
  • Cron jobs : scheduling task
    • if cron jobs failed, it requires manual fixing → orchestration
  • Orchestration : any of the jobs fail → retry automatically
    • more robust data pipeline and saves resource
• CI/CD process
  • ingestion of python code
  • sql statements for transformation
  • orchestration code
  • small data engineering team
    • real : transformation, orchestration code changes are done in production
    • problem : CI/CD is not possible, error-prone, manual
    • improve : learn from software engineering(UI based workflow → code base workflow)
      • code base workflow : separate development environment and production environment

 

Orchestration

• orchestration : coordinating multiple data engineering jobs
  • coordination : many jobs running either in sequence or in parallel
    • form: DAG. Directed Acyclical Graph
      

      

      
      
      • Direct : only one direction
      • Acyclical : not infinite loops
  • crucial for complex data pipelines
    • improve efficiency : parallel execution → improve performance
    • complex dependency management → reliable and simpler
    • improve reliability : retries and timeouts → notify error
    • automation
      • small data engineering team : run cron jobs instead of orchestration system despite of importance of orchestration
  • end-to-end visibility : command & control for data engineering jobs
    • effective monitoring and debugging
    • understanding how data flows
    • how datasets relate to each other
    • which systems they touch
  • observe data lineage : where data came from
    • keeping metadata → easily tracking down which information passes

 

Security, Privacy, Data Quality

• security
  • principle of least privilege : only provide enouth to get their job done and no more
  • type
    • authentication : confirm the user
    • authorization : give permission to access
  • layers of defenses = there isn’t any bulletproof security
    • more layer = more security
• privacy
  • build a culture of data privacy
    • masking PII(personally identifiable information)
    • setting up data stack to be able to remove individual users data when requested
• data quality
  • more nuanced(slightly different) than it seems at first
  • all about TRUST
  • data fails silently
  • robust Validation, Testing, observability
  • software vs data

	Software	Data
When	only when deploy software	independently from code → change out of control
Test	bug break or simple thresholds	need statistical test

 

Development Workflow

• workflow : development → production
  • development : build & test before release
  • benefits
    • fast iteration : testing in local laptop is faster than testing cloud
    • isolation : production system is robust to change of development
    • cost reduction
  • challenges
    • not always get good sample of data
      • sampling a large dataset is not always easy
    • parity of dev/prod environment is not always possible
      • both env must be similar : cloud tools not always available locally

 

Reference

https://www.hivelocity.net/kb/what-is-cron-job/