Dolt - Version Control for Data

GitHub - dolthub/dolt: Dolt – It’s Git for Data

A SQL database with Git-style version control built into the core:

Core Concept:

  • Git + SQL: Combines familiar Git workflows with SQL database operations
  • Data Versioning: Track changes to data like you track changes to code
  • Collaboration: Multiple people can work on the same dataset simultaneously
  • Audit Trail: Complete history of who changed what and when

Key Features:

Git-Style Operations:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# Clone a database
dolt clone dolthub/ip-to-country

# Make changes to data
dolt sql -q "INSERT INTO countries VALUES ('XX', 'Test Country')"

# Stage changes
dolt add .

# Commit changes
dolt commit -m "Added test country"

# Push to remote
dolt push origin main

SQL Database Functionality:

  • Standard SQL: Full MySQL-compatible SQL interface
  • ACID Transactions: Complete transaction support
  • Indexes: Performance optimization with standard database indexes
  • Constraints: Primary keys, foreign keys, unique constraints

Unique Capabilities:

Branch and Merge Data:

1
2
3
4
5
6
7
8
9

# Create feature branch
dolt checkout -b feature/cleanup-data

# Make data changes
dolt sql -q "DELETE FROM users WHERE last_login < '2020-01-01'"

# Merge back to main
dolt checkout main
dolt merge feature/cleanup-data

Time Travel Queries:

1
2
3
4
5
6
7
8

-- Query data as of specific commit
SELECT * FROM users AS OF 'abc123';

-- Compare data between commits
SELECT * FROM diff('main', 'feature/cleanup', 'users');

-- Show history of specific row
SELECT * FROM dolt_history_users WHERE id = 123;

Use Cases:

Data Analytics:

  • Experiment Tracking: Different feature engineering approaches
  • Model Versioning: Track training data versions with model performance
  • Reproducible Research: Exact data state for research papers
  • A/B Testing: Compare dataset variants and results

Data Engineering:

  • ETL Pipeline Versioning: Track data transformation steps
  • Data Quality: Rollback corrupted data changes
  • Collaboration: Multiple analysts working on same dataset
  • Audit Compliance: Complete change history for regulations

Application Development:

  • Schema Evolution: Version database schema alongside data
  • Feature Flags: Different data configurations for different features
  • Testing: Isolated test data environments
  • Rollback Safety: Safe deployment with easy rollback

Architecture Benefits:

Storage Efficiency:

  • Content Addressable: Deduplication of identical data blocks
  • Incremental Changes: Only store what actually changed
  • Compression: Efficient storage of large datasets
  • Remote Sync: Only transfer changed data

Concurrent Access:

  • MVCC: Multiple version concurrency control
  • Branch Isolation: Changes don’t interfere until merge
  • Conflict Resolution: Merge conflict handling for data
  • Distributed: Clone and work offline

Comparison with Traditional Approaches:

vs Database Backups:

  • Granular Changes: See individual row changes, not just snapshots
  • Efficient Storage: Don’t duplicate unchanged data
  • Branch Support: Multiple parallel data versions
  • Merge Capability: Combine changes intelligently

vs Data Lakes:

  • Structured Data: SQL interface with schema enforcement
  • ACID Properties: Transactional consistency
  • Version Control: Built-in change tracking
  • Query Performance: Optimized for relational queries

Getting Started:

Installation:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# Install Dolt
curl -L https://github.com/dolthub/dolt/releases/latest/download/install.sh | bash

# Initialize new database
mkdir my-database && cd my-database
dolt init

# Create table and add data
dolt sql -q "CREATE TABLE users (id INT PRIMARY KEY, name VARCHAR(100))"
dolt sql -q "INSERT INTO users VALUES (1, 'Alice'), (2, 'Bob')"

# Track changes
dolt add .
dolt commit -m "Initial users"

Connect Existing Tools:

1
2
3
4
5

# Start SQL server
dolt sql-server

# Connect with any MySQL client
mysql -h 127.0.0.1 -P 3306 -u root my-database

Enterprise Features:

  • DoltHub: GitHub-like hosting for Dolt databases
  • Access Control: User permissions and authentication
  • API Access: REST and GraphQL APIs
  • Integration: Works with existing BI and analytics tools

Limitations and Considerations:

  • Performance: Not optimized for high-throughput OLTP
  • Ecosystem: Newer tool with growing ecosystem
  • Learning Curve: New concepts for traditional database users
  • Storage: Version history can grow large over time

Dolt represents a paradigm shift in how we think about data management, bringing software engineering best practices to database operations and making data collaboration as natural as code collaboration.