datacompy

A Python library that facilitates the comparison of two DataFrames in Pandas, Polars, Spark and more. The library goes beyond basic equality checks by providing detailed insights into discrepancies at both row and column levels.

dvt

Data Validation Tool compares data from source and target tables to ensure that they match. It provides column validation, row validation, schema validation, custom query validation, and ad hoc SQL exploration.

koala-diff

A high-performance Python library for comparing large datasets (CSV, Parquet) locally using Rust and Polars. It features zero-copy streaming to prevent OOM errors and generates interactive HTML data quality reports.

FutureSearch SDK

Python SDK that dispatches parallel web-research agents across

DataSpoc Pipe

Data ingestion engine that connects 400+ Singer taps to Parquet files in cloud buckets (S3, GCS, Azure). Streaming, incremental, with auto-catalog.

Enrich.sh

Managed event ingestion service that converts JSON sent to a REST API into Hive-partitioned Parquet on Cloudflare R2, queryable from DuckDB, ClickHouse, BigQuery, Snowflake, and Python.

enrich-companies

CLI tool to enrich CSV files with company data (financials, contacts, metadata) from 250M+ company records. Available on [npm](https://www.npmjs.com/package/enrich-companies).

ingestr

CLI tool to copy data between databases with a single command. Supports 50+ sources including PostgreSQL, MySQL, MongoDB, Salesforce, Shopify to any data warehouse.

Kafka

Publish-subscribe messaging rethought as a distributed commit log.

AWS Kinesis

A fully managed, cloud-based service for real-time data processing over large, distributed data streams.

RabbitMQ

Robust messaging for applications.

dlt

A fast&simple pipeline building library for Python data devs, runs in notebooks, cloud functions, airflow, etc.

drt

OSS Reverse ETL CLI. Sync data from warehouses to business tools via YAML.

FluentD

An open source data collector for unified logging layer.

Embulk

An open source bulk data loader that helps data transfer between various databases, storages, file formats, and cloud services.

Apache Sqoop

A tool designed for efficiently transferring bulk data between Apache Hadoop and structured datastores such as relational databases.

Heka

Data Acquisition and Processing Made Easy. Deprecated.

Gobblin

Universal data ingestion framework for Hadoop from LinkedIn.

Nakadi

An open source event messaging platform that provides a REST API on top of Kafka-like queues.

Pravega

Provides a new storage abstraction - a stream - for continuous and unbounded data.

Apache Pulsar

An open-source distributed pub-sub messaging system.

AWS Data Wrangler

Utility belt to handle data on AWS.

Airbyte

Open-source data integration for modern data teams.

DBConvert Streams

self-hosted database migration and change data capture (CDC) tool with built-in SQL IDE.

Artie

Real-time data ingestion tool leveraging change data capture.

Sling

CLI data integration tool specialized in moving data between databases, as well as storage systems.

Meltano

CLI & code-first ELT.

Google Sheets ETL

Live import all your Google Sheets to your data warehouse.

CsvPath Framework

A delimited data preboarding framework that fills the gap between MFT and the data lake.

Estuary Flow

No/low-code data pipeline platform that handles both batch and real-time data ingestion.

db2lake

Lightweight Node.js ETL framework for databases → data lakes/warehouses.

data-genie

High-performance, streaming-first ETL engine for Node.js and TypeScript with constant memory footprint.

Kreuzberg

Polyglot document intelligence library with a Rust core and bindings for Python, TypeScript, Go, and more. Extracts text, tables, and metadata from 62+ document formats for data pipeline ingestion.

pdfmux

Python PDF-to-Markdown orchestrator. Classifies each page and routes to the optimal backend (PyMuPDF, Docling, RapidOCR, Gemini Flash), emitting Markdown plus a per-page confidence score so ingestion pipelines can quarantine low-trust pages before feeding LLMs or retrieval.

DataRaven

Managed cloud object storage transfers for ingestion workflows.

Xquik

Real-time X (Twitter) data extraction platform with REST API (76 endpoints), 20 bulk extraction tools, account monitoring, HMAC-signed webhooks, and MCP server for AI agent integration.

Arpe.io

High-speed CLI tools for database export, import, replication and migration with parallel streaming to CSV, Parquet, JSON and cloud storage, supporting PostgreSQL, MySQL, Oracle, SQL Server and 80+ sources.

Crustdata

A real-time B2B data API for company and people intelligence, providing firmographics, headcount signals, job listings, web traffic, and funding events via REST API and webhooks for data enrichment pipelines.

crdt-merge

Conflict-free merge for DataFrames, JSON, ML models & distributed agents — powered by CRDTs.

LinkedIn Jobs Scraper

Crawlee-based actor extracting structured LinkedIn job listings at scale without API keys.

CARQ

Context-Aware RAG Processing Queue for high availability and adaptive rate-limiting.

Duckle

Local-first, open-source desktop ETL/ELT studio: drag a pipeline onto a canvas (or describe it to a built-in on-device AI assistant) and run it at native speed through DuckDB. 290+ connectors, a scheduler, and an MCP server for driving pipelines from an LLM. No cloud, no servers.

Rawbbit

Open-source self-hosted analytics pipeline that lands raw events as Parquet in your own object storage. Uses NATS JetStream for durable buffering and BigQuery external tables for querying. Designed for teams that want to own their raw event data.

faucet-stream

Config-driven data-movement platform for Rust with pluggable source and sink connectors, running ETL, CDC, and streaming pipelines declaratively from YAML or embedded as a library.

HDFS

A distributed file system designed to run on commodity hardware.

AWS S3

Object storage built to retrieve any amount of data from anywhere.

Alluxio

A memory-centric distributed storage system enabling reliable data sharing at memory-speed across cluster frameworks, such as Spark and MapReduce.

CEPH

A unified, distributed storage system designed for excellent performance, reliability, and scalability.

JuiceFS

A high-performance Cloud-Native file system driven by object storage for large-scale data storage.

OrangeFS

Orange File System is a branch of the Parallel Virtual File System.

SnackFS

A bite-sized, lightweight HDFS compatible file system built over Cassandra.

GlusterFS

Gluster Filesystem.

XtreemFS

Fault-tolerant distributed file system for all storage needs.

SeaweedFS

Seaweed-FS is a simple and highly scalable distributed file system. There are two objectives: to store billions of files! to serve the files fast! Instead of supporting full POSIX file system semantics, Seaweed-FS choose to implement only a key~file mapping. Similar to the word "NoSQL", you can call it as "NoFS".

S3QL

A file system that stores all its data online using storage services like Google Storage, Amazon S3, or OpenStack.

LizardFS

Software Defined Storage is a distributed, parallel, scalable, fault-tolerant, Geo-Redundant and highly available file system.

AKF

The AI native file format. Trust scores, source provenance, and compliance metadata that embed into 20+ formats (DOCX, PDF, images, code). EXIF for AI.

Apache Avro

Apache Avro™ is a data serialization system.

Apache Parquet

A columnar storage format available to any project in the Hadoop ecosystem, regardless of the choice of data processing framework, data model or programming language.

Apache ORC

The smallest, fastest columnar storage for Hadoop workloads.

Apache Thrift

The Apache Thrift software framework, for scalable cross-language services development.

ProtoBuf

Protocol Buffers - Google's data interchange format.

SequenceFile

A flat file consisting of binary key/value pairs. It is extensively used in MapReduce as input/output formats.

Kryo

A fast and efficient object graph serialization framework for Java.

PFC-JSONL

Specialized JSONL log compressor with block-level timestamp indexing and DuckDB integration. Achieves ~9% compression ratio (better than gzip) with time-range random access queries.

ParquetKit

Browser-based viewer, SQL workbench and converter for Parquet files powered by DuckDB-WASM. Fully client-side, no upload.

Apache Beam

A unified programming model that implements both batch and streaming data processing jobs that run on many execution engines.

Spark Streaming

Makes it easy to build scalable fault-tolerant streaming applications.

Apache Flink

A streaming dataflow engine that provides data distribution, communication, and fault tolerance for distributed computations over data streams.

Apache Storm

A free and open source distributed realtime computation system.

Apache Samza

A distributed stream processing framework.

Apache NiFi

An easy to use, powerful, and reliable system to process and distribute data.

Apache Hudi

An open source framework for managing storage for real time processing, one of the most interesting feature is the Upsert.

CocoIndex

An open source ETL framework to build fresh index for AI.

VoltDB

An ACID-compliant RDBMS which uses a [shared nothing architecture](https://en.wikipedia.org/wiki/Shared-nothing_architecture).

PipelineDB

The Streaming SQL Database.

Spring Cloud Dataflow

Streaming and tasks execution between Spring Boot apps.

Bonobo

A data-processing toolkit for python 3.5+.

Robinhood's Faust

Forever scalable event processing & in-memory durable K/V store as a library with asyncio & static typing.

HStreamDB

The streaming database built for IoT data storage and real-time processing.

Kuiper

An edge lightweight IoT data analytics/streaming software implemented by Golang, and it can be run at all kinds of resource-constrained edge devices.

Zilla

- An API gateway built for event-driven architectures and streaming that supports standard protocols such as HTTP, SSE, gRPC, MQTT, and the native Kafka protocol.

SwimOS

A framework for building real-time streaming data processing applications that supports a wide range of ingestion sources.

Pathway

Performant open-source Python ETL framework with Rust runtime, supporting 300+ data sources.

Hadoop MapReduce

A software framework for easily writing applications which process vast amounts of data (multi-terabyte data-sets) - in-parallel on large clusters (thousands of nodes) - of commodity hardware in a reliable, fault-tolerant manner.

Spark

A multi-language engine for executing data engineering, data science, and machine learning on single-node machines or clusters.

AWS EMR

A web service that makes it easy to quickly and cost-effectively process vast amounts of data.

Data Mechanics

A cloud-based platform deployed on Kubernetes making Apache Spark more developer-friendly and cost-effective.

Tez

An application framework which allows for a complex directed-acyclic-graph of tasks for processing data.

Bistro

A light-weight engine for general-purpose data processing including both batch and stream analytics. It is based on a novel unique data model, which represents data via _functions_ and processes data via _columns operations_ as opposed to having only set operations in conventional approaches like MapReduce or SQL.

Substation

A cloud native data pipeline and transformation toolkit written in Go.

dna-claude-analysis

Personal genome analysis toolkit with Python scripts analyzing raw DNA data across 17 categories (health risks, ancestry, pharmacogenomics, nutrition, psychology, etc.) and generating a terminal-style single-page HTML visualization.

Highcharts

A charting library written in pure JavaScript, offering an easy way of adding interactive charts to your web site or web application.

ZingChart

Fast JavaScript charts for any data set.

C3.js

D3-based reusable chart library.

D3.js

A JavaScript library for manipulating documents based on data.

SmoothieCharts

A JavaScript Charting Library for Streaming Data.

PyXley

Python helpers for building dashboards using Flask and React.

Plotly

Flask, JS, and CSS boilerplate for interactive, web-based visualization apps in Python.

Apache Superset

A modern, enterprise-ready business intelligence web application.

Redash

Make Your Company Data Driven. Connect to any data source, easily visualize and share your data.

Metabase

The easy, open source way for everyone in your company to ask questions and learn from data.

stratif.io

Open-source, self-hosted, warehouse-native product analytics. Runs funnels, retention, and paths directly on DuckDB, Postgres, Snowflake, or ClickHouse.

PyQtGraph

A pure-python graphics and GUI library built on PyQt4 / PySide and numpy. It is intended for use in mathematics / scientific / engineering applications.

Seaborn

A Python visualization library based on matplotlib. It provides a high-level interface for drawing attractive statistical graphics.

QueryGPT

Natural language database query interface with automatic chart generation, supporting Chinese and English queries.

AI for Database

Agentic AI platform to connect any database (PostgreSQL, MySQL, MongoDB, etc.) and query in plain English; includes self-refreshing intelligent dashboards and action workflows triggered by data changes.

Dekart

Open-source SQL to map platform for BigQuery, Snowflake, and PostGIS.

LunaPad

Open-source analytics notebook for reusable SQL workflows, interactive reports, and AI-assisted data exploration.

Bonnard

Governed, multi-tenant MCP access to your customers' data. Turn your warehouse, dbt, or semantic layer into a secure, per-customer MCP for AI agents.

Nika

Intent-as-code workflow engine for AI data pipelines: reviewable YAML DAGs statically checked (schema, permits, cost floor) before execution, with tamper-evident run traces.

OrionBelt Semantic Layer

Open-source semantic sidecar that compiles YAML-defined dimensions, measures, and metrics into optimized SQL across 8 engines (BigQuery, ClickHouse, Databricks, Dremio, DuckDB, MySQL, PostgreSQL, Snowflake). Unified REST, MCP, and Postgres wire protocol; one model powers AI agents, analytics, DQ rules, and KPIs.

Bruin

End-to-end data pipeline tool that combines ingestion, transformation (SQL + Python), and data quality in a single CLI. Connects to BigQuery, Snowflake, PostgreSQL, Redshift, and more. Includes VS Code extension with live previews.

DataFlow

Open-source platform for data preparation, synthetic data generation, and AI/data pipelines. Includes reusable skills for automating workflow steps across data and AI tasks.

Luigi

A Python module that helps you build complex pipelines of batch jobs.

CronQ

An application cron-like system. [Used](https://chairnerd.seatgeek.com/building-out-the-seatgeek-data-pipeline/) w/Luigi. Deprecated.

Cascading

Java based application development platform.

Airflow

A system to programmatically author, schedule, and monitor data pipelines.

Azkaban

A batch workflow job scheduler created at LinkedIn to run Hadoop jobs. Azkaban resolves the ordering through job dependencies and provides an easy-to-use web user interface to maintain and track your workflows.

Oozie

A workflow scheduler system to manage Apache Hadoop jobs.

Pinball

DAG based workflow manager. Job flows are defined programmatically in Python. Support output passing between jobs.

Dagster

An open-source Python library for building data applications.

Hamilton

A lightweight library to define data transformations as a directed-acyclic graph (DAG). If you like dbt for SQL transforms, you will like Hamilton for Python processing.

Kedro

A framework that makes it easy to build robust and scalable data pipelines by providing uniform project templates, data abstraction, configuration and pipeline assembly.

Dataform

An open-source framework and web based IDE to manage datasets and their dependencies. SQLX extends your existing SQL warehouse dialect to add features that support dependency management, testing, documentation and more.

Dotflow

A lightweight Python library for building execution pipelines with retry, parallel execution, cron scheduling, and async support.

Census

A reverse-ETL tool that let you sync data from your cloud data warehouse to SaaS applications like Salesforce, Marketo, HubSpot, Zendesk, etc. No engineering favors required—just SQL.

dbt

A command line tool that enables data analysts and engineers to transform data in their warehouses more effectively.

Kestra

Scalable, event-driven, language-agnostic orchestration and scheduling platform to manage millions of workflows declaratively in code.

RudderStack

A warehouse-first Customer Data Platform that enables you to collect data from every application, website and SaaS platform, and then activate it in your warehouse and business tools.

PACE

An open source framework that allows you to enforce agreements on how data should be accessed, used, and transformed, regardless of the data platform (Snowflake, BigQuery, DataBricks, etc.)

OneQuery

Self-hosted gateway for safe, auditable queries for agents across approved data sources.

Prefect

An orchestration and observability platform. With it, developers can rapidly build and scale resilient code, and triage disruptions effortlessly.

Multiwoven

The open-source reverse ETL, data activation platform for modern data teams.

SuprSend

Create automated workflows and logic using API's for your notification service. Add templates, batching, preferences, inapp inbox with workflows to trigger notifications directly from your data warehouse.

Mage

Open-source data pipeline tool for transforming and integrating data.

SQLMesh

An open-source data transformation framework for managing, testing, and deploying SQL and Python-based data pipelines with version control, environment isolation, and automatic dependency resolution.

lakeFS

An open source platform that delivers resilience and manageability to object-storage based data lakes.

Project Nessie

A Transactional Catalog for Data Lakes with Git-like semantics. Works with Apache Iceberg tables.

Ilum

A modular Data Lakehouse platform that simplifies the management and monitoring of Apache Spark clusters across Kubernetes and Hadoop environments.

Gravitino

An open-source, unified metadata management for data lakes, data warehouses, and external catalogs.

FlightPath Data

FlightPath is a gateway to a data lake's bronze layer, protecting it from invalid external data file feeds as a trusted publisher.

rawquery

Managed lakehouse platform on Apache Iceberg with DuckDB query compute, S3 storage, Postgres wire protocol, and SQL transforms.

docker-logstash

A highly configurable Logstash (1.4.4) - Docker image running Elasticsearch (1.7.0) - and Kibana (3.1.2).

elasticsearch-jdbc

JDBC importer for Elasticsearch.

ZomboDB

PostgreSQL Extension that allows creating an index backed by Elasticsearch.

Gockerize

Package golang service into minimal Docker containers.

Flocker

Easily manage Docker containers & their data.

Rancher

RancherOS is a 20mb Linux distro that runs the entire OS as Docker containers.

Kontena

Application Containers for Masses.

Weave

Weaving Docker containers into applications.

Zodiac

A lightweight tool for easy deployment and rollback of dockerized applications.

cAdvisor

Analyzes resource usage and performance characteristics of running containers.

Micro S3 persistence

Docker microservice for saving/restoring volume data to S3.

Rocker-compose

Docker composition tool with idempotency features for deploying apps composed of multiple containers. Deprecated.

Nomad

A cluster manager, designed for both long-lived services and short-lived batch processing workloads.

ImageLayers

Visualize Docker images and the layers that compose them.

DexPaprika

Free real-time DEX data via SSE streaming across 34 blockchains. 30M+ pools, 27M+ tokens, ~1 second price updates. No API key, no rate limits. [Docs](https://docs.dexpaprika.com)

Helium MCP

Remote MCP server for real-time financial data, 3.2M+ news articles, ML options pricing, and news bias analysis. Free, no API key. [MCP](https://heliumtrades.com/mcp)

Twitter Realtime

The Streaming APIs give developers low latency access to Twitter's global stream of Tweet data.

Sorsa API

Real-time X (Twitter) data API providing tweets, profiles, search, communities and engagement metrics. Up to 50x cheaper than the official X API with 20 req/sec rate limit, JSON output.

Eventsim

Event data simulator. Generates a stream of pseudo-random events from a set of users, designed to simulate web traffic.

Eventum

Data generation platform for producing synthetic event streams with complex correlations.

Reddit

Real-time data is available including comments, submissions and links posted to reddit.

GitHub Archive

GitHub's public timeline since 2011, updated every hour.

Common Crawl

Open source repository of web crawl data.

Wikipedia

Wikipedia's complete copy of all wikis, in the form of Wikitext source and metadata embedded in XML. A number of raw database tables in SQL form are also available.

The Quiet-Broke Index

A 30-metro composite of US household cost burdens (housing, taxes, childcare, healthcare, transport) aggregated from Census ACS, BLS Consumer Expenditure Survey, and HUD Fair Market Rents. Open methodology, free, no email gate.

FirstData

The world's most comprehensive authoritative data source knowledge base. 160+ curated sources from governments, international organizations, and research institutions with MCP integration.

Mindweave Synthetic Business Data

42-table synthetic SME dataset with double-entry accounting, tax compliance (AU/US/UK), and temporal realism. CSV, SQL, Parquet, SQLite. Ideal for ETL pipeline testing.

LatAm Synth

Synthetic financial savings behavior generator for Latin America: users, savings goals, and transactions calibrated on 506K real records (2015–2024). Reproducible by seed, 100% synthetic.

Prometheus.io

An open-source service monitoring system and time series database.

HAProxy Exporter

Simple server that scrapes HAProxy stats and exports them via HTTP for Prometheus consumption.

Signals CLI

Intent signal monitoring CLI. Track LinkedIn engagers, keyword posters, job changers, funding events. JSON output for data pipelines.

Data Profiler

The DataProfiler is a Python library designed to make data analysis, monitoring, and sensitive data detection easy.

YData Profiling

A general-purpose open-source data profiler for high-level analysis of a dataset.

Desbordante

An open-source data profiler specifically focused on discovery and validation of complex patterns in data.

SchemaCrawler

Open-source and free relational database schema discovery and comprehension tool. Documents and diagrams relational database schemas from your Java programs, build tools and the command line. Find database design issues with lint, and write scripts against the database. Includes an MCP Server for use by AI agents.

Aegis DQ

Open-source agentic data quality framework with LLM-powered diagnosis, root-cause analysis, SQL auto-fix proposals, and 31 rule types — DuckDB, Postgres, BigQuery, Databricks, Athena, Snowflake.

Grai

A data catalog tool that integrates into your CI system exposing downstream impact testing of data changes. These tests prevent data changes which might break data pipelines or BI dashboards from making it to production.

DQOps

An open-source data quality platform for the whole data platform lifecycle from profiling new data sources to applying full automation of data quality monitoring.

DataKitchen

Open Source Data Observability for end-to-end Data Journey Observability, data profiling, anomaly detection, and auto-created data quality validation tests.

GreatExpectation

Open Source data validation framework to manage data quality. Users can define and document “expectations” rules about how data should look and behave.

Provero

A vendor-neutral, declarative data quality engine. Define checks in YAML, run anywhere. Includes 16 built-in check types, SQL batch optimizer, anomaly detection, and data contracts.

Scherlok

Zero-config data quality CLI. Profiles every table on first run, then auto-detects anomalies (volume drops, schema drift, freshness misses, distribution shifts) on subsequent runs. No YAML, no rules to write. Works with Postgres, BigQuery, Snowflake, and dbt.

RunSQL

Free online SQL playground for MySQL, PostgreSQL, and SQL Server. Create database structures, run queries, and share results instantly.

Spark Playground

Write, run, and test PySpark code on Spark Playground's online compiler. Access real-world sample datasets & solve interview questions to enhance your PySpark skills for data engineering roles.

daffy

Decorator-first DataFrame contracts/validation (columns/dtypes/constraints) at function boundaries. Supports Pandas/Polars/PyArrow/Modin.

Snowflake Emulator

A Snowflake-compatible emulator for local development and testing.

DataScreenIQ

Real-time data quality firewall for pipelines and APIs. Screens rows in milliseconds for schema drift, null spikes, type mismatches, and data anomalies with PASS / WARN / BLOCK decisions.

DataDriven

Interview practice with SQL query execution, Python, and data modeling exercises.

Fixzi

JSON/XML validation and API contract monitoring tool for debugging and testing structured data.

/r/dataengineering

News, tips, and background on Data Engineering.

/r/etl

Subreddit focused on ETL.

AI Dev Jobs

Job board focused on AI, ML, and data engineering roles with 7,400+ listings, salary data, and a free REST API.

Data Council

The first technical conference that bridges the gap between data scientists, data engineers and data analysts.

Chain of Thought

Interviews with AI and data infrastructure leaders on building production systems.

Data Engineering Podcast

The show about modern data infrastructure.

Latent Space

Technical deep dives on AI engineering, from model training to deployment.

Practical AI

Making AI practical, productive, and accessible to everyone.

Software Engineering Daily

Daily interviews about technical software topics, including data infrastructure.

The Analytics Engineering Podcast

How analytics engineers build and maintain data pipelines at scale.

The Data Stack Show

A show where they talk to data engineers, analysts, and data scientists about their experience around building and maintaining data infrastructure, delivering data and data products, and driving better outcomes across their businesses with data.

Snowflake Data Engineering

A practical introduction to data engineering on the Snowflake cloud data platform.

Best Data Science Books

This blog offers a curated list of top data science books, categorized by topics and learning stages, to aid readers in building foundational knowledge and staying updated with industry trends.

Architecting an Apache Iceberg Lakehouse

A guide to designing an Apache Iceberg lakehouse from scratch.

Learn AI Data Engineering in a Month of Lunches

A fast, friendly guide to integrating large language models into your data workflows.