Data engineering: the data layer, the build choices and what it costs to run

What a data engineer does, and what the role pays

Google Cloud's Professional Data Engineer exam tests five areas: designing data processing systems, ingesting and processing data, storing it, preparing it for analysis, and maintaining and automating data workloads.¹⁰

O*NET lists Data Engineer among the reported titles for Database Architects, an occupation that designs strategies for enterprise databases and data warehouse systems. Data Warehousing Specialists, who design and implement corporate data warehousing, are listed apart.^11,12,13

For its 2025 State of Analytics Engineering Report, a vendor-run survey, dbt Labs collected answers from 459 data practitioners and leaders between October and December 2024. Data engineers were 36% of the individual contributors who answered.^18,19

In that survey, 57% of respondents said they spend most of their workdays maintaining or organizing data sets. In North America, over 80% of individual contributors in data roles earned more than $100K, up from 69% the year before.^20,21

The data layer: ingestion, storage, transformation, semantic layer and serving

Databricks calls the medallion pattern a recommended best practice but not a requirement. AWS describes ELT as the reverse of ETL: data is loaded into the target before it is processed, using the target database's own processing power.^23,28,29

Data engineering on CIOPages

• Data Lakehouse PlatformsBuyer Guide
• ETL & ELT Data Integration ToolsBuyer Guide
• Data Streaming Platforms & Event Stream ProcessingBuyer Guide
• Enterprise Cloud Data Warehouse PlatformsBuyer Guide
• Data Warehouse AutomationBuyer Guide
• Data Quality & ObservabilityBuyer Guide
• Data Catalog Tools & Metadata ManagementBuyer Guide
• Databricks vs SnowflakeComparison
• Microsoft Fabric vs DatabricksComparison
• Microsoft Fabric vs SnowflakeComparison
• Amazon Redshift vs SnowflakeComparison
• Cloudera vs DatabricksComparison
• RFP questions: Enterprise Cloud Data Warehouse PlatformsRFP questions
• RFP questions: integrationRFP questions
• State of Play: ETL & ELT Data Integration ToolsState of Play
• Modern data stack selection guideArticle
• Introduction to data management and analyticsArticle
• Data pipelines: ETL, ELT and streamingArticle
• DataOps and data observabilityArticle
• Data quality management checklistFree tool
• Data storage cost calculatorFree tool

Data warehouse vs. data lakehouse, and Apache Iceberg vs. Delta Lake

Databricks defines a data lakehouse as a data management system that combines the benefits of data lakes and data warehouses.³⁰

Apache Iceberg is a table format for huge analytic tables that brings SQL table behavior to big data. Its project site says engines including Spark, Trino, Flink, Presto, Hive and Impala can work with the same tables at the same time, and that schema changes never require rewriting a table.^3,31,32

Delta Lake is the storage layer of the Databricks lakehouse and supports ACID transactions. It unifies streaming and batch processing on existing data lakes, so a Delta table is both a batch table and a streaming source and sink. Its UniForm feature lets Iceberg and Hudi clients read Delta tables.^4,33,34,35

How three warehouses handle open formats, per their documentation:

• Snowflake's Iceberg tables combine Snowflake query semantics with external cloud storage that the customer manages.⁵
• BigQuery's Iceberg managed tables reside in the customer's own buckets, and open-source engines can query them without moving the data.³⁶
• Redshift Serverless bills queries that read open file formats in Amazon S3 in the same RPU-hours as its other work.⁹

Apache Kafka publishes, stores and processes streams of events. Kafka 4.0, released in March 2025, was its first major release to run entirely without Apache ZooKeeper, so operators no longer maintain a separate ZooKeeper ensemble.^{37,38,39,40,41}

What a data platform costs: published warehouse and pipeline prices

An X-Large Snowflake warehouse uses 16 credits an hour. At the $3.00 Enterprise on-demand rate in AWS US East (Northern Virginia), that is $48 for each hour it runs.^6,42

For staff, the BLS median for database architects, the occupation O*NET maps data engineers to, was $139,500 in May 2025.^11,14

Data quality and data lineage: tests, observability and OpenLineage

Poor data quality was the challenge data teams reported most often in dbt Labs' 2025 survey, cited by over 56% of respondents. Data quality and observability was the second-largest area for increased investment, with 38% planning to increase it over the next 12 months.^52,53

Delta Lake's schema enforcement blocks bad records during ingestion. In dbt, data tests are assertions about models and sources. The built-in tests check that a column is not null, is unique, or matches a value in another model.^54,55,56

OpenLineage is an open framework and specification for collecting data lineage, built on dataset, job and run entities that can be extended with facets. Its documentation lists integrations with Apache Airflow, Flink, Hive and Spark, dbt, Great Expectations and Trino.^57,58,59

OpenLineage is a Graduate project of the LF AI & Data Foundation. Marquez is its reference implementation. Platforms also track lineage themselves: Unity Catalog records lineage as data is transformed in a Databricks lakehouse.^60,61,62

Where data engineering goes wrong

• Paying for idle or thrashing compute. Snowflake bills a 60-second minimum each time a warehouse resumes. With regular 2- or 3-minute gaps between queries, an auto-suspend of 1 or 2 minutes keeps the warehouse suspending and resuming.^63,64
• Buying size instead of speed. Snowflake says a larger warehouse consumes more credits and may not result in any performance increase.⁶⁵
• Assuming LIMIT saves money. BigQuery on demand charges for the columns a query selects even with an explicit LIMIT. Partitioning and clustering reduce the data processed.⁶⁶
• Duplicating connections. Fivetran counts each connection toward MAR separately, even when two sync the same data from the same source.⁶⁷
• Skipping table maintenance. Iceberg snapshots accumulate until expired, small files add metadata and slow queries, and failed jobs can leave orphan files that table metadata does not reference.^68,69,70
• Writing around the platform. Google warns that modifying the data files of a BigQuery Iceberg managed table outside BigQuery can cause query failure or data loss.⁷¹
• Defining metrics in each BI tool. The dbt Semantic Layer moves definitions out of the BI layer and into the modeling layer, so a changed metric is refreshed everywhere it is used.^72,73
• Building lineage integration by integration. Without a shared standard, OpenLineage's documentation says, each project has to instrument its own metadata collection, duplicating effort.⁷⁴