Nessie

Nessie is to Data Lakes what Git is to source code repositories. Therefore, Nessie uses many terms from both Git and data lakes.

This page explains how Nessie makes working with data in data lakes much easier without requiring much prior knowledge of either Git or data lakes.

Nessie is designed to give users an always-consistent view of their data across all involved data sets (tables). Changes to data, for example from batch jobs, happen independently and are completely isolated. Users will not see any incomplete changes. Once all the changes are done, all the changes can be atomically and consistently applied and become visible to your users.

Nessie eliminates the hard and often manual work required to keep track of the individual data files. Nessie knows which data files are being used and which data files can safely be deleted.

Production, staging and development environments can use the same data lake without risking the consistent state of production data.

Nessie does not copy data, instead it references the existing data, which works fine, because data files are immutable.