Big data platforms in 2026: the lakehouse, Spark and open table formats
Big data no longer means running a Hadoop cluster. It means cheap object storage, an open table format, and whichever engines you want to query it with.
Ten years ago, a big data project usually meant building a Hadoop cluster. Today most companies keep large datasets in cloud object storage and query them with managed engines. The pieces have changed, and so has the cost of entry.
The building blocks
| Layer | What it does | Examples |
|---|---|---|
| Storage | Holds data cheaply at any scale | Amazon S3, Azure Data Lake Storage, Google Cloud Storage |
| File format | Stores data in columns for fast scans | Parquet |
| Table format | Adds database features to files in storage | Apache Iceberg, Delta Lake, Apache Hudi |
| Processing engine | Runs queries and transformations | Spark, Trino, Flink, cloud warehouses |
| Catalog | Tracks tables, schemas and permissions | Unity Catalog, AWS Glue, Polaris |
Why table formats matter
The table format is what turns a pile of files into a lakehouse. Apache Iceberg describes itself as a high-performance format for huge analytic tables, and says engines including Spark, Trino, Flink, Presto, Hive and Impala can safely work with the same tables at the same time. It supports schema changes without rewriting the table, time travel queries against an exact past snapshot, and rollback to a known-good state.
The business benefit is independence. Data stored in an open format can be read by several engines and vendors, so changing your query engine doesn't mean migrating your data. Major warehouse vendors, including Snowflake, Databricks, Google and AWS, now support Iceberg tables.
What it costs
Serverless engines charge by data scanned or compute used. Google's BigQuery, for example, charges $6.25 per TiB processed for on-demand queries, with the first 1 TiB each month free, and the first 10 GiB of storage free. At those rates a mid-size company can analyze large datasets without a cluster, but a badly written query over a huge table can also cost real money. Partitioning, clustering and query limits keep that in check.
Choosing a platform
If you already run a cloud data warehouse, start there: most now read and write open table formats. If you have several teams using different engines, an open lakehouse with a shared catalog avoids copying data between systems. Hadoop clusters on-premises are now mainly a migration project.
What IT teams should do
- Store new large datasets in an open table format such as Iceberg or Delta Lake.
- Set per-user or per-project query cost limits on serverless engines.
- Pick one catalog to manage permissions across engines.
- If you still run Hadoop, plan its migration to object storage and a managed engine.