AWS Glue Data Catalog
Hudi tables can sync to AWS Glue Data Catalog directly via AWS SDK. Piggyback on HiveSyncTool
, org.apache.hudi.aws.sync.AwsGlueCatalogSyncTool
makes use of all the configurations that are taken by HiveSyncTool
and send them to AWS Glue.
Configurations
Most of the configurations for AwsGlueCatalogSyncTool
are shared with HiveSyncTool
. The example showed in
Sync to Hive Metastore can be used as is for sync with Glue Data Catalog, provided that the hive metastore
URL (either JDBC or thrift URI) can proxied to Glue Data Catalog, which is usually done within AWS EMR or Glue job environment.
For Hudi streamer, users can set
--sync-tool-classes org.apache.hudi.aws.sync.AwsGlueCatalogSyncTool
For Spark data source writers, users can set
hoodie.meta.sync.client.tool.class=org.apache.hudi.aws.sync.AwsGlueCatalogSyncTool
Avoid creating excessive versions
Tables stored in Glue Data Catalog are versioned. And by default, every Hudi commit triggers a sync operation if enabled, regardless of having relevant metadata changes. This can lead to too many versions kept in the catalog and eventually failing the sync operation.
Meta-sync can be set to conditional - only sync when there are schema change or partition change. This can avoid creating excessive versions in the catalog. Users can enable it by setting
hoodie.datasource.meta_sync.condition.sync=true
Glue Data Catalog specific configs
Sync to Glue Data Catalog can be optimized with other configs like
hoodie.datasource.meta.sync.glue.all_partitions_read_parallelism
hoodie.datasource.meta.sync.glue.changed_partitions_read_parallelism
hoodie.datasource.meta.sync.glue.partition_change_parallelism
Partition indexes can also be used by setting
hoodie.datasource.meta.sync.glue.partition_index_fields.enable
hoodie.datasource.meta.sync.glue.partition_index_fields
Other references
Running AWS Glue Catalog Sync for Spark DataSource
To write a Hudi table to Amazon S3 and catalog it in AWS Glue Data Catalog, you can use the options mentioned in the AWS documentation
Running AWS Glue Catalog Sync from EMR
If you're running HiveSyncTool on an EMR cluster backed by Glue Data Catalog as external metastore, you can simply run the sync from command line like below:
cd /usr/lib/hudi/bin
./run_sync_tool.sh --base-path s3://<bucket_name>/<prefix>/<table_name> --database <database_name> --table <table_name> --partitioned-by <column_name>