Apache Iceberg is an open-source table format designed for large analytic datasets stored in data lakes. It adds ACID transactions, schema evolution, hidden partitioning, and time travel to files sitting in object storage like S3 or GCS. Unlike older Hive-style tables, Iceberg tracks changes at the file level, so you can safely update, evolve, and query tables without rewriting entire datasets.<\/p>\n\n\n\n
Quick question: have you ever changed a column type in a data lake table and watched a “simple” migration turn into a multi-hour rewrite job?<\/p>\n\n\n\n
That’s the exact problem Iceberg was built to solve.<\/p>\n\n\n\n
Apache Iceberg is a table format \u2014 not a storage system, not a query engine. It sits as a metadata layer on top of files (usually Parquet) in object storage, tracking exactly which files make up a table at any point in time.<\/p>\n\n\n\n
Pro Tip:<\/strong> If you’re coming from a Hive background, the mental shift to make is this: Hive tracks tables at the directory<\/em> level, while Iceberg tracks them at the individual file<\/em> level. That one difference is what unlocks most of Iceberg’s other features.<\/p>\n\n\n\n Before Iceberg, most data lakes used the Hive table format, which has a few well-known pain points:<\/p>\n\n\n\n When we worked through migrating a mid-sized Parquet\/Hive table to Iceberg during a Q1 2026 internal proof-of-concept, the partition evolution feature alone eliminated what used to be a multi-hour backfill job every time the partitioning strategy needed to change \u2014 it became a metadata-only operation instead. [HUMAN EDITOR: Replace with an actual sourced case study or verified internal benchmark if available; this is a placeholder illustrative example, not a published statistic.]<\/p>\n\n\n\n The catalog tracks which tables exist and points to their current metadata. Common catalog implementations include Hive Metastore, AWS Glue, Nessie, and REST-based catalogs.<\/p>\n\n\n\n Each table has a metadata file describing its schema, partition spec, and snapshot history \u2014 essentially the table’s “table of contents.”<\/p>\n\n\n\n These track which data files belong to which snapshot, enabling Iceberg to know exactly what to read without scanning the whole directory.<\/p>\n\n\n\n Every write creates a new snapshot. This is what makes time travel possible \u2014 you’re not overwriting history, you’re adding to it.<\/p>\n\n\n\n Data Point:<\/strong> Iceberg’s snapshot-based design is also what enables safe concurrent writes without table locks, since readers always see a consistent snapshot rather than a partially-written state.<\/p>\n\n\n\n Before you start, you’ll need:<\/p>\n\n\n\n For this tutorial, examples use Apache Spark with a local Hadoop catalog<\/strong>, since it requires the least setup for learning purposes.<\/p>\n\n\n\n # Add the Iceberg Spark runtime when launching Spark<\/p>\n\n\n\n spark-shell \\<\/p>\n\n\n\n –packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.5.0 \\<\/p>\n\n\n\n –conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \\<\/p>\n\n\n\n –conf spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog \\<\/p>\n\n\n\n –conf spark.sql.catalog.local.type=hadoop \\<\/p>\n\n\n\n –conf spark.sql.catalog.local.warehouse=\/tmp\/iceberg-warehouse<\/p>\n\n\n\n Warning:<\/strong> Version-match your Iceberg runtime package to your Spark version. A mismatch here is the single most common setup error beginners run into.<\/p>\n\n\n\n Step 1: Create the table<\/strong><\/p>\n\n\n\n CREATE TABLE local.db.orders (<\/p>\n\n\n\n order_id BIGINT,<\/p>\n\n\n\n customer_id BIGINT,<\/p>\n\n\n\n order_date DATE,<\/p>\n\n\n\n amount DECIMAL(10,2)<\/p>\n\n\n\n )<\/p>\n\n\n\n USING iceberg<\/p>\n\n\n\n PARTITIONED BY (months(order_date));<\/p>\n\n\n\n Step 2: Insert data<\/strong><\/p>\n\n\n\n INSERT INTO local.db.orders VALUES<\/p>\n\n\n\n (1, 101, DATE ‘2026-01-15’, 250.00),<\/p>\n\n\n\n (2, 102, DATE ‘2026-01-20’, 89.99);<\/p>\n\n\n\n Step 3: Query the table<\/strong><\/p>\n\n\n\n SELECT * FROM local.db.orders WHERE order_date >= DATE ‘2026-01-01’;<\/p>\n\n\n\n Step 4: Inspect metadata (this is where Iceberg gets interesting)<\/strong><\/p>\n\n\n\n SELECT * FROM local.db.orders.snapshots;<\/p>\n\n\n\n SELECT * FROM local.db.orders.history;<\/p>\n\n\n\n These metadata tables are unique to Iceberg \u2014 you’re querying the table’s own change history as if it were regular data.<\/p>\n\n\n\n This is the section that usually convinces engineers to migrate.<\/p>\n\n\n\n Adding a column<\/strong> \u2014 no rewrite required:<\/p>\n\n\n\n ALTER TABLE local.db.orders ADD COLUMN region STRING;<\/p>\n\n\n\n Changing partitioning<\/strong> \u2014 also no rewrite required:<\/p>\n\n\n\n ALTER TABLE local.db.orders REPLACE PARTITION FIELD months(order_date) WITH days(order_date);<\/p>\n\n\n\n Best Practice:<\/strong> Existing data files stay under the old partition scheme, while new writes use the new scheme. Iceberg handles this transparently at query time \u2014 you don’t need to backfill unless you specifically want uniform partitioning across old data.<\/p>\n\n\n\n Because every write creates a snapshot, you can query the table as it existed at a previous point in time.<\/p>\n\n\n\n — Query as of a specific snapshot ID<\/p>\n\n\n\n SELECT * FROM local.db.orders VERSION AS OF 1234567890123;<\/p>\n\n\n\n — Query as of a specific timestamp<\/p>\n\n\n\n SELECT * FROM local.db.orders TIMESTAMP AS OF ‘2026-06-01 00:00:00’;<\/p>\n\n\n\n This is genuinely useful for debugging (“what did this table look like before yesterday’s job ran?”) and for audit or compliance scenarios where you need to reproduce a report exactly as it existed on a past date.<\/p>\n\n\n\n Pros of Iceberg:<\/strong><\/p>\n\n\n\n Cons of Iceberg:<\/strong><\/p>\n\n\n\n
\n\n\n\nWhy It Exists: The Problem With Hive Tables<\/h2>\n\n\n\n
\n
\n\n\n\nCore Concepts You Need to Understand<\/h2>\n\n\n\n
Catalog<\/h3>\n\n\n\n
Metadata Files<\/h3>\n\n\n\n
Manifest Lists and Manifest Files<\/h3>\n\n\n\n
Snapshots<\/h3>\n\n\n\n
Setting Up: Prerequisites<\/h2>\n\n\n\n
\n
\n\n\n\nStep-by-Step: Creating Your First Iceberg Table<\/h2>\n\n\n\n
\n\n\n\nSchema and Partition Evolution in Practice<\/h2>\n\n\n\n
\n\n\n\nTime Travel Queries<\/h2>\n\n\n\n
\n\n\n\nApache Iceberg vs. Delta Lake vs. Apache Hudi<\/h2>\n\n\n\n
Factor<\/strong><\/th> Apache Iceberg<\/strong><\/th> Delta Lake<\/strong><\/th> Apache Hudi<\/strong><\/th><\/tr><\/thead> Origin<\/td> Netflix<\/td> Databricks<\/td> Uber<\/td><\/tr> Engine Support<\/td> Broad (Spark, Trino, Flink, Snowflake, etc.)<\/td> Strongest in Databricks\/Spark ecosystem<\/td> Strong for streaming\/CDC use cases<\/td><\/tr> Partition Evolution<\/td> Yes, without rewrite<\/td> Limited<\/td> Limited<\/td><\/tr> Best Fit<\/td> Multi-engine environments<\/td> Databricks-centric stacks<\/td> High-frequency upsert\/streaming workloads<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n \n