Overview

Welcome to Apache Hudi! This overview will provide a high level summary of what Apache Hudi is and will orient you on how to learn more to get started.

What is Apache Hudi

Apache Hudi (pronounced “hoodie”) is the next generation streaming data lake platform. Hudi brings core warehouse and database functionality directly to a data lake. Hudi provides tables, transactions, efficient upserts/deletes, advanced indexes, ingestion services, data clustering/compaction optimizations, and concurrency all while keeping your data in open source file formats.

Not only is Apache Hudi great for streaming workloads, but it also allows you to create efficient incremental batch pipelines. Read the docs for more use case descriptions and check out who's using Hudi, to see how some of the largest data lakes in the world including Uber, Amazon, ByteDance, Robinhood and more are transforming their production data lakes with Hudi.

Apache Hudi can easily be used on any cloud storage platform. Hudi’s advanced performance optimizations, make analytical workloads faster with any of the popular query engines including, Apache Spark, Flink, Presto, Trino, Hive, etc.

Hudi-rs is the native Rust implementation for Apache Hudi, which also provides bindings to Python. It expands the use of Apache Hudi for a diverse range of use cases in the non-JVM ecosystems.

Core Concepts to Learn

If you are relatively new to Apache Hudi, it is important to be familiar with a few core concepts:

• Hudi Timeline – How Hudi manages transactions and other table services
• Hudi File Layout - How the files are laid out on storage
• Hudi Table Types – COPY_ON_WRITE and MERGE_ON_READ
• Hudi Query Types – Snapshot Queries, Incremental Queries, Read-Optimized Queries

See more in the "Concepts" section of the docs.

Take a look at recent blog posts that go in depth on certain topics or use cases.

Getting Started

Sometimes the fastest way to learn is by doing. Try out these Quick Start resources to get up and running in minutes:

• Spark Quick Start Guide – if you primarily use Apache Spark
• Flink Quick Start Guide – if you primarily use Apache Flink
• Python/Rust Quick Start Guide (Hudi-rs) - if you primarily use Python or Rust

If you want to experience Apache Hudi integrated into an end to end demo with Kafka, Spark, Hive, Presto, etc, try out the Docker Demo:

• Docker Demo

Connect With The Community

Apache Hudi is community focused and community led and welcomes new-comers with open arms. Leverage the following resources to learn more, engage, and get help as you get started.

Join in on discussions

See all the ways to engage with the community here. Two most popular methods include:

• Hudi Slack Channel
• Hudi mailing list - (send any msg to subscribe)

Come to Office Hours for help

Weekly office hours are posted here

Community Calls

Attend monthly community calls to learn best practices and see what others are building.

Contribute

Apache Hudi welcomes you to join in on the fun and make a lasting impact on the industry as a whole. See our contributor guide to learn more, and don’t hesitate to directly reach out to any of the current committers to learn more.

Have an idea, an ask, or feedback about a pain-point, but don’t have time to contribute? Join the Hudi Slack Channel and share!