Explore Generative AI’s impact on data forecasting with our benchmarking study, comparing LLMs to traditional methods like ARIMA.
Unified Access
Query Delta Lake from Athena and Snowflake via Iceberg and Unity Catalog
- By Saket Suman
Databricks Delta Lake, Databricks Unity Catalog, and open table formats like Apache Iceberg promise seamless, governed data access across analytics engines. However, in practice, integrating these technologies with AWS Athena and Snowflake introduces frustrating compatibility challenges. From protocol version errors to multi-engine data duplication, teams are often forced to choose between governance and flexibility. This article provides a comprehensive, step-by-step walkthrough for building a single-source, interoperable solution that connects Databricks Unity Catalog-managed Databricks Delta tables to Athena and Snowflake using Iceberg metadata. All of this is achieved without duplicating data or sacrificing performance. Whether you need to query Databricks Delta Lake from Athena or integrate Databricks-managed Iceberg tables with Snowflake, this guide ensures smooth interoperability across platforms.
Table of Contents
Add a header to begin generating the table of contents
Table of Contents
Add a header to begin generating the table of contents
Introduction
Delta Lake is an open-source storage layer that brings ACID transactions, schema enforcement, time travel, and unified batch and streaming capabilities to data lakes. It forms the backbone of the Lakehouse architecture, enabling scalable, reliable, and performant analytics directly on cloud object storage.
However, integrating Delta Lake with query engines like Amazon Athena and Snowflake often introduces compatibility challenges—particularly around Delta protocol versions. In this blog, we will focus only on EXTERNAL tables in Databricks Unity Catalog.
One of the most common and frustrating issues users face is this error:
“Delta protocol version is too new for Athena DDL engine.”
At the same time, customers want to avoid duplicating their data (stored on External locations like S3 or ADLS, in this case) just to serve different downstream tools. Managing multiple copies adds unnecessary complexity and cost.
What you will learn
This blog post walks through a step-by-step solution that enables you to:
- Maintain Delta table compatibility with Athena
- Enable Iceberg interoperability for Snowflake and Athena
- Avoid Delta protocol version mismatches with Unity Catalog
- Seamlessly integrate Delta/Iceberg tables with Unity Catalog for centralized governance and discoverability
- Use a single copy of data to support multiple query engines—no replication required
Why a single copy matters:
It serves as a golden source of truth, reduces ELT complexity, eliminates the need for data synchronization across engines, simplifies structure management, and cuts costs.
Technologies used
- Databricks workspace running on AWS (Azure Databricks would only read read-only access to S3 & cannot write)
- AWS Athena
- Snowflake query engine
- AWS IAM
- AWS S3
- Apache Iceberg
What is Unity Catalog?
Databricks Unity Catalog
Databricks’ unified governance solution for all data and AI assets.
It provides:
- Centralized access control across all Databricks workspaces
- Fine-grained permissions at the catalog, schema, table, and column levels
- Automated lineage tracking for data and AI assets.
- Built-in auditing and data discovery tools
- Support for open formats like Delta Lake, Apache Iceberg, Hudi, and Parquet
Unity Catalog uses a three-level namespace (catalog.schema.table) and supports federated governance across multiple clouds and external systems — including Snowflake, Redshift, BigQuery, and Athena — without requiring data migration.
By combining Unity Catalog with Delta Lake and Iceberg, you can build a governed, interoperable lakehouse that supports:
- Open data sharing
- Cross-platform querying
- Secure collaboration
The problem
Compatibility challenges between Delta Lake and Athena
Delta Lake on Unity Catalog relies on protocol versions to manage feature compatibility. When advanced features are enabled—such as:
- Deletion Vectors
- Column Mapping
- Iceberg Compatibility
…the Delta table’s protocol is automatically upgraded (often to minWriterVersion = 7 or higher).
Unfortunately, AWS Athena does not currently support these higher protocol versions. As a result, when attempting to create a Glue schema over an existing Delta table in Unity Catalog, Athena throws a version-related error.
This issue blocks Athena from reading Unity Catalog–managed Delta tables that use modern features, disrupting interoperability across platforms.
You can find additional details in the official AWS documentation: Query Linux Foundation Delta Lake tables – Amazon Athena.
The solution: Step by step guide
In the following set of steps, we will work around the above stated issue to make this integration work with just one copy of data.
Note: You may have to add the following spark setting at the cluster level.
spark.databricks.delta.deletionVectors.allowAutoEnablefalse
Create a Databricks Delta Table with Safe Protocol Versions
Data used for this blog (Please read disclaimer and public data use policy guidance before downloading any data):
TLC Trip Record Data – TLC (NYC Open Data – FAQ)
%sql
-- Drop the table if it already exists;
The table name and the Catalog/Schema names are based on the goals of this blog
and the fact that we are using the NYC Taxi dataset downloaded from NYC Yellow Taxi
Trip Data & TLC Trip Record Data - TLC
DROP TABLE IF EXISTS athenasnowint.nyctaxidemo.yellow_taxi_200910_iceberg_delvectorfalse;
-- Create a Delta table with minimal protocol versions and deletion vectors disabled;
-- NOTE: The table schema was pre-created for this blog and may be very different in your scenario.
CREATE TABLE athenasnowint.nyctaxidemo.yellow_taxi_200910_iceberg_delvectorfalse (
vendor_name STRING,
Trip_Pickup_DateTime TIMESTAMP,
Trip_Dropoff_DateTime TIMESTAMP,
Passenger_Count BIGINT,
Trip_Distance DOUBLE,
Start_Lon DOUBLE,
Start_Lat DOUBLE,
Rate_Code DOUBLE,
store_and_forward STRING,
End_Lon DOUBLE,
End_Lat DOUBLE,
Payment_Type STRING,
Fare_Amt DOUBLE,
surcharge DOUBLE,
mta_tax DOUBLE,
Tip_Amt DOUBLE,
Tolls_Amt DOUBLE,
Total_Amt DOUBLE,
Trip_Pickup_Date DATE,
Trip_Pickup_Hour INT
)
USING DELTA
PARTITIONED BY (Trip_Pickup_Hour)
TBLPROPERTIES (
'delta.enableDeletionVectors' = 'false',
'delta.minReaderVersion' = '1',
'delta.minWriterVersion' = '2'
)
LOCATION 's3://<your-bucket-name>/<your-path>/yellow_taxi_200910_iceberg_delvectorfalse/';
Confirm protocol versions in Databricks
%sql
DESC TABLE EXTENDED athenasnowint.nyctaxidemo.yellow_taxi_200910_iceberg_delvectorfalse;
minReaderVersion = 1minWriterVersion = 2
Register the Databricks Delta Table in Glue database
(from Athena query editor)
%sql
CREATE EXTERNAL TABLE yellow_taxi_200910_iceberg
LOCATION 's3://dbxathenabucket/dbxexpriments/yellow_taxi_200910_iceberg_delvectorfalse/'
TBLPROPERTIES ('table_type' = 'DELTA',
'delta.universalFormats.enabledFormats'='iceberg'); --Both these options are required since there is no actual
DELTA support available in ATHENA but it only supports DELTA via ICEBERG
Purge and clean up Delta metadata
(In Databricks notebook)
alt_qry1 = """
REORG TABLE athenasnowint.nyctaxidemo.yellow_taxi_200910_iceberg_delvectorfalse APPLY (PURGE);
"""
spark.sql(alt_qry1)
Enable Iceberg compatibility
(In Databricks notebook)
alt_qry2 = """
ALTER TABLE athenasnowint.nyctaxidemo.yellow_taxi_200910_iceberg_delvectorfalse SET TBLPROPERTIES(
'delta.columnMapping.mode' = 'name',
'delta.enableIcebergCompatV2' = 'true',
'delta.universalFormat.enabledFormats' = 'iceberg'
);
"""
spark.sql(alt_qry2)
minReaderVersion = 2minWriterVersion = 7
Insert Data from any source
(Another Databricks delta table has been used here as the source which was pre-populated but was not Iceberg enabled)
%sql
INSERT INTO athenasnowint.nyctaxidemo.yellow_taxi_200910_iceberg_delvectorfalse
SELECT * FROM athenasnowint.nyctaxidemo.yellow_taxi_200910;
Validate in Athena and Databricks Unity Catalog
(In Databricks notebook)
%sql
SELECT COUNT(*) FROM athenasnowint.nyctaxidemo.yellow_taxi_200910_iceberg_delvectorfalse;
(In AWS Athena query editor)
…in both Athena and Databricks/Unity Catalog.
Now, let’s delete all the records and it is expected to instantly refresh on the Athena side since it is based on Iceberg/Delta metadata.
You’ll see the same row count (0) & it shows up instantly just like Databricks Delta Lake — no MSCK REPAIR needed in Databricks! Yay!
Query the Same Databricks UC table from Snowflake using Iceberg
There are 3+ ways we can integrate Snowflake with Databricks Unity Catalog based delta tables for read only queries.
NOTE: In this blog, we won’t cover the older method of using Storage Integration with Stage to create external tables. We will only cover the newer ways where we will use OBJECT_STORE and ICEBERG_REST as CATALOG_SOURCE.
The trust policy may have the Trust Relationship setup like this. If you want to get the placeholder values used below, use DESC EXTERNAL VOLUME <external_volume_name>;
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<snowflake_account_id>:user/<snowflake_user_path>"
},
"Action": "sts:AssumeRole",
"Condition": {
"StringEquals": {
"sts:ExternalId": "<external_id_1>"
}
}
},
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<snowflake_account_id>:user/<snowflake_user_path>"
},
"Action": "sts:AssumeRole",
"Condition": {
"StringEquals": {
"sts:ExternalId": "<external_id_2>"
}
}
},
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<snowflake_account_id>:user/<snowflake_user_path>"
},
"Action": "sts:AssumeRole",
"Condition": {
"StringEquals": {
"sts:ExternalId": "<external_id_3>"
}
}
}
]
}
-- Snowflake SQL
-- Snowflake OBJECT_STORE-based Catalog Integration for Delta Table
-- 📦 Create an external volume pointing to the S3 location
CREATE OR REPLACE EXTERNAL VOLUME <external_volume_name>
STORAGE_LOCATIONS = (
(
NAME = '<external_volume_name>',
STORAGE_PROVIDER = 'S3',
STORAGE_BASE_URL = '<s3_bucket_path>',
STORAGE_AWS_ROLE_ARN = '<aws_role_arn>'
)
)
ALLOW_WRITES = FALSE;
DESC EXTERNAL VOLUME <external_volume_name>;
-- 🔗 Create a catalog integration using OBJECT_STORE and TABLE_FORMAT = DELTA
CREATE OR REPLACE CATALOG INTEGRATION <catalog_integration_name>
CATALOG_SOURCE = OBJECT_STORE
TABLE_FORMAT = DELTA
ENABLED = TRUE;
DESC CATALOG INTEGRATION <catalog_integration_name>;
-- 📄 Register a Delta table using BASE_LOCATION (relative to external volume)
CREATE OR REPLACE ICEBERG TABLE <snowflake_table_name>
EXTERNAL_VOLUME = <external_volume_name>
CATALOG = <catalog_integration_name>
BASE_LOCATION = '<relative_table_path>/'; --RELATIVE TO THE S3 LOCATION USED FOR EXTERNAL VOLUME
-- ✅ Query the table
SELECT COUNT(*) FROM <snowflake_table_name>;
-- Snowflake SQL
-- Snowflake OBJECT_STORE-based Catalog Integration for Iceberg Table
-- 📦 Create an external volume pointing to the S3 location
CREATE OR REPLACE EXTERNAL VOLUME <external_volume_name>
STORAGE_LOCATIONS = (
(
NAME = '<external_volume_name>',
STORAGE_PROVIDER = 'S3',
STORAGE_BASE_URL = '<s3_bucket_path>',
STORAGE_AWS_ROLE_ARN = '<aws_role_arn>'
)
)
ALLOW_WRITES = FALSE;
DESC EXTERNAL VOLUME <external_volume_name>;
-- 🔗 Create a catalog integration using OBJECT_STORE and TABLE_FORMAT = ICEBERG
CREATE OR REPLACE CATALOG INTEGRATION <catalog_integration_name>
CATALOG_SOURCE = OBJECT_STORE
TABLE_FORMAT = ICEBERG
ENABLED = TRUE;
DESC CATALOG INTEGRATION <catalog_integration_name>;
-- 📄 Register an Iceberg table using METADATA_FILE_PATH (relative to external volume)
CREATE OR REPLACE ICEBERG TABLE <snowflake_table_name>
EXTERNAL_VOLUME = <external_volume_name>
CATALOG = <catalog_integration_name>
METADATA_FILE_PATH = '<relative_metadata_path>'; --PATH OF THE LATEST METADATA JSON FILE
-- ✅ Query the table
SELECT COUNT(*) FROM <snowflake_table_name>;
Use ICEBERG_REST + ICEBERG type Catalog integration (Optional)
-- Snowflake SQL
-- 📦 EXTERNAL VOLUME
CREATE OR REPLACE EXTERNAL VOLUME <external_volume_name>
STORAGE_LOCATIONS = (
(
NAME = '<external_volume_name>',
STORAGE_PROVIDER = 'S3',
STORAGE_BASE_URL = '<s3_bucket_path>',
STORAGE_AWS_ROLE_ARN = '<aws_role_arn>'
)
)
ALLOW_WRITES = FALSE;
DESC EXTERNAL VOLUME <external_volume_name>;
-- 🔗 CATALOG INTEGRATION
CREATE OR REPLACE CATALOG INTEGRATION <catalog_integration_name>
CATALOG_SOURCE = ICEBERG_REST
TABLE_FORMAT = ICEBERG
--CATALOG_NAMESPACE = '<optional_namespace>'
REST_CONFIG = (
CATALOG_URI = ' https://<Databricks Workspace URL>/api/2.1/unity-catalog/iceberg',
CATALOG_NAME = '<unity_catalog_name>'
)
REST_AUTHENTICATION = (
TYPE = BEARER,
BEARER_TOKEN = '<databricks_PAT>'
)
ENABLED = TRUE;
DESC CATALOG INTEGRATION <catalog_integration_name>;
-- 🔍 DISCOVERY
SELECT SYSTEM$LIST_NAMESPACES_FROM_CATALOG('<catalog_integration_name>');
-- 📄 REGISTER ICEBERG TABLE
CREATE ICEBERG TABLE <snowflake_table_name>
EXTERNAL_VOLUME = <external_volume_name>
CATALOG = <catalog_integration_name>
CATALOG_NAMESPACE = '<schema name>'
CATALOG_TABLE_NAME = '<table_name_in_unity_catalog>';
-- 🔄 REFRESH METADATA IF NEEDED
ALTER ICEBERG TABLE <snowflake_table_name> REFRESH;
-- ✅ QUERY
SELECT COUNT(*) FROM <snowflake_table_name>;
Performance comparison
| Method | Setup | Metadata Sync | Performance |
|---|---|---|---|
OBJECT_STORE + DELTA | Simple | No | Fast |
OBJECT_STORE + ICEBERG | Simple | No | Fast |
ICEBERG_REST + ICEBERG | Complex | Managed | Fastest |
Lessons learned
- Always check Delta features early to avoid protocol upgrades.
- Deletion Vectors silently bump protocol to v7 irrespective of spark configurations — avoid unless needed.
- Iceberg compatibility can be added after Athena setup. With that Athena can read the table as an external table without issues.
- Snowflake reads both Delta and Iceberg table metadata with ease.
- We can make all this work with just one copy of the data without duplicating it
What if I want an Athena/Glue based catalog instead of having it centered on Unity Catalog in Databricks?
Alternatively, if someone has a priority to create tables in Glue via Athena & access it via Databricks Unity catalog, here is how they can do it.
NOTE: We can create ICEBERG tables directly from Athena and Glue ETL/Catalog as well.
However, in this case, we chose Databricks to be the central platform where Unity Catalog and ICEBERG REST based tables will be created from. To create ICEBERG Tables in Athena using S3, here is the syntax:
CREATE TABLE IF NOT EXISTS yellow_taxi_200910_iceberg_ice
WITH (table_type = 'ICEBERG', format='parquet', location='s3://dbxathenabucket/dbxexpriments/yellow_taxi_200910_iceberg_delvectorfalse_ath_ice/', is_external=false) AS
SELECT * FROM yellow_taxi_200910_iceberg;
SELECT * FROM dbxathenainttest.yellow_taxi_200910_iceberg_ice LIMIT 10;
Valid values for FORMAT: [ORC, PARQUET, AVRO, RCBINARY, RCTEXT, SEQUENCEFILE, JSON, OPENX_JSON, TEXTFILE, ION, CSV, REGEX] No support for DELTA & only Managed tables are allowed. So, is_external has to be ‘false’.
Unity Catalog does not currently support registering external Glue Iceberg tables as native Unity Catalog tables. So, we cannot access this table from Databricks.
How can you access this from Databricks via Unity Catalog Glue federation?
This is an option for customers that do not want to create tables using Databricks but want to read from Databricks via Unity Catalog.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "lakeformation.amazonaws.com"
},
"Action": "sts:AssumeRole"
},
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<ACCOUNT_ID_1>:root"
},
"Action": "sts:AssumeRole",
"Condition": {
"StringEquals": {
"sts:ExternalId": "<EXTERNAL_ID>"
}
}
},
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<ACCOUNT_ID_2>:role/<ROLE_NAME>"
},
"Action": "sts:AssumeRole"
}
]
}
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"glue:GetDatabase",
"glue:GetDatabases",
"glue:GetTable",
"glue:GetTables",
"glue:GetPartition",
"glue:GetPartitions"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::<Your S3 bucket where Athena ICEBERG table was created>",
"arn:aws:s3::: <Your S3 bucket where Athena ICEBERG table was created>/*"
]
}
]
}
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Resource": "arn:aws:iam::<ACCOUNT_ID_2>:role/<ROLE_NAME>"
}
]
}
%sql
CREATE CONNECTION glue_connection
TYPE GLUE
OPTIONS (
aws_account_id = '<Account ID>',
aws_region = '<Region>',
credential = '<The credential name we just created above>'
);
How can you access this from Snowflake?
-- Snowflake OBJECT_STORE-based Catalog Integration for Iceberg Table
-- 📦 Create an external volume pointing to the S3 location
CREATE OR REPLACE EXTERNAL VOLUME <external_volume_name>
STORAGE_LOCATIONS = (
(
NAME = '<external_volume_name>',
STORAGE_PROVIDER = 'S3',
STORAGE_BASE_URL = '<s3_bucket_path>',
STORAGE_AWS_ROLE_ARN = '<aws_role_arn>'
)
)
ALLOW_WRITES = FALSE;
DESC EXTERNAL VOLUME <external_volume_name>;
-- 🔗 Create a catalog integration using OBJECT_STORE and TABLE_FORMAT = ICEBERG
CREATE OR REPLACE CATALOG INTEGRATION <catalog_integration_name>
CATALOG_SOURCE = OBJECT_STORE
TABLE_FORMAT = ICEBERG
ENABLED = TRUE;
DESC CATALOG INTEGRATION <catalog_integration_name>;
-- 📄 Register an Iceberg table using METADATA_FILE_PATH (relative to external volume)
CREATE OR REPLACE ICEBERG TABLE <snowflake_table_name>
EXTERNAL_VOLUME = <external_volume_name>
CATALOG = <catalog_integration_name>
METADATA_FILE_PATH = '<relative_metadata_path>'; --PATH OF THE LATEST METADATA JSON FILE
-- ✅ Query the table
SELECT COUNT(*) FROM <snowflake_table_name>;
Conclusion
While using Databricks as the central data processing tool & Unity Catalog as the central metadata, with the right setup:
- And with just one copy of data in Databricks Unity Catalog enabled Delta Lake with assured ACID compliance, you can,
- Query Databricks UC Delta tables from AWS Athena
- Query Databricks UC Delta tables from Snowflake
- You can Avoid metadata headaches and enable auto sync
- You can build a true multi-engine lakehouse
Share with your network
You may also enjoy
Optimize your Databricks workflows by setting up monitoring, using realistic test datasets, and fine-tuning cluster configurations through iterative testing. Regularly review job performance to reduce costs and improve efficiency.
