A qualitative compilation of production-ready edge computing projects with a focus on Edge-2-Cloud Data Engineering
- Major IoT Clouds
- Major Edge OSS Projects
- Edge Messaging
- Message Serialization
- Messaging & Data Streaming
- Stream Processing
- Databases
- Batch Processing / Analytics
- Industrial IoT with Kubernetes
- High Quality
- Production Ready
- Well Documented
- Well Maintained
- Cloud Ready
- Scalable
- AWS IoT - Unlock your IoT data and accelerate business growth
- Azure IoT Operations - Build interoperable IoT solutions that transform physical operations at scale.
- Eclipse Ditto - the open-source project of Eclipse IoT that provides a ready-to-use functionality to manage the state of Digital Twins
- Eclipse Hono - provides uniform (remote) service interfaces for connecting large numbers of IoT devices to a (cloud) back end.
- Akri - Akri is a Kubernetes Resource Interface that lets you easily expose heterogeneous leaf devices
- Akraino - LF Edge is an umbrella organization that establishes an open, interoperable framework for edge computing
- Shifu - Shifu provides users with a transparent framework for full-scene device hosting and integrated software development
- HiveMQ - Enterprise MQTT Platform
- Nats.io - Connective Technology for Adaptive Edge & Distributed Systems
- Mosquitto - An Open Source MQTT v3.1/v3.1.1 Broker.
- VerneMQ - A distributed MQTT message broker.
- Waterstream - MQTT broker leveraging Apache Kafka as its own storage and distribution engine.
- NanoMQ - A light-weight and Blazing-fast MQTT Broker for IoT Edge platform.
- Node CoAP - A client and server library for CoAP modeled after the http module.
- Rust CoAP - A Constrained Application Protocol (CoAP) library implemented in Rust.
- OPC-UA - Interoperability standard for the secure and reliable exchange of data
- ModBus - A data communications protocol for use with Modicon PLC's
- ZeroMQ Overview - An open-source universal messaging library
- NanoMSG - a simple high-performance implementation of several "scalability protocols"
- NNG - nanomsg-next-generation -- light-weight brokerless messaging
- uws - Tiny WebSockets (access to the C++ library, µWebSockets, via Node.js)
- uWebsockets - is a lightweight and efficient MQTT broker designed to raise the bar for pub/sub performance.
- Tungstenite WS - Lightweight stream-based WebSocket implementation for Rust.
- Python Websockets - A library for building WebSocket servers and clients in Python
- Autobahn.ws - Open-source real-time framework for Web, Mobile & Internet of Things.
- socket.io - Bidirectional and low-latency communication for every platform
- Apache Avro - Apache Avro™ is a data serialization system
- Apache Thrift - scalable cross-language services development
- CapNProto - Apache Avro™ is a data serialization system.
- JSON - is a lightweight data-interchange format.
- MessagePack - MessagePack is an efficient binary serialization format.
- CBOR - RFC 8949 Concise Binary Object Representation
- FlatBuffers - an efficient cross platform serialization library
- Protocol Buffer - a language-neutral, platform-neutral extensible mechanism for serializing structured data
- Azure Event Hub - a fully managed, real-time data ingestion service that’s simple, trusted, and scalable
- Azure Managed Kafka - Azure HDInsight, a customizable, enterprise-grade service for open-source analytics.
- Azure Queue Storage - a service for storing large numbers of messages
- Azure Event Grid - a highly scalable, serverless event broker that you can use to integrate applications using events.
- Confluent Cloud - Apache Kafka as a Service
- AWS MSK - Securely stream data with a fully managed, highly available Apache Kafka service
- AWS SQS - Fully managed message queuing for microservices, distributed systems, and serverless applications
- AWS Kinesis - Easily stream data at any scale
- Strimzi - Cloud Native Apache Kafka
- Apache Pulsar - a distributed pub-sub messaging platform with a very flexible messaging model and an intuitive client API.
- Apache Kafka - distributed publish-subscribe messaging system.
- Apache NiFi - Apache NiFi is an integrated data logistics platform for automating the movement of data between disparate systems.
- redpanda - A Kafka® replacement for mission critical systems; 10x faster. Written in C++.
- Fluentd - tool to collect events and logs.
- Kestrel - distributed message queue system.
- Apache ActiveMQ Artemis - The Next Generation Message Broker by ActiveMQ, AMQP 1.0 is a natively supported protocol.
- Apache Qpid Java Broker - A pure-Java AMQP message broker.
- Solace - Messaging Platform that supports AMQP 1.0
- RabbitMQ with AMQP 1.0 plugin.
- AWS Kinesis Data Analytics - Gain actionable insights from streaming data with serverless, fully managed Apache Flink
- Azure Stream Analytics - Serverless real-time analytics, from the cloud to the edge
- KSQL DB - KSQL DB on Confluent Cloud
- Faust - stream processing and event processing in Python
- Siddhi - stream Processing and Complex Event Processing Engine
- KSQL - The database purpose-built for stream processing applications.
- Benthos - Declarative stream processing for mundane tasks and data engineering
- Go Streams - A lightweight stream processing library for Go
- Flogo - Elegant stream processing pipeline written entirely in Golang
- Apache Heron - A realtime, distributed, fault-tolerant stream processing engine
- Apache Beam - Implement batch and streaming data processing jobs that run on any execution engine.
- Apache Flink - Stateful Computations over Data Streams
- Apache Samza - A distributed stream processing framework
- Databricks - All your data, analytics and AI on one Lakehouse platform
- Apache Mahout - For Creating Scalable Performant Machine Learning Applications
- H2O - The #1 open-source machine learning platform for the enterprise
- Apache Drill - Schema-free SQL Query Engine for Hadoop, NoSQL and Cloud Storage.
- Apache Spark - Apache Spark™ is a unified analytics engine for large-scale data processing.
- Azure SQL - Build apps that scale with managed and intelligent SQL in the cloud
- Azure CosmosDB - Fast NoSQL / Graph / Relational database with open APIs for planet scale
- Azure PostgreSQL - Fully managed, intelligent, and scalable PostgreSQL
- Azure Data Explorer - Fast and highly scalable data exploration service.
- AWS Aurora - AWS Relational DB
- AWS DynamoDB - fast, flexible, and serverless NoSQL database for any scale
- AWS DocumentDB - Scale JSON workloads with ease using a MongoDB-compatible document database
- AWS Keyspaces - Run your Apache Cassandra workloads on a scalable, highly available, and managed wide column database service.
- AWS Neptune - Build applications that work with highly connected datasets using a fast, reliable graph database service.
- AWS Ledger - Provide transparent, immutable, cryptographically verifiable transaction logs with a fully managed ledger database service.
- AWS Timestream - Provide transparent, immutable, cryptographically verifiable transaction logs with a fully managed ledger database service.
- InfluxDB - Scalable datastore for metrics, events, and real-time analytics
- TimescaleDB - An open-source time-series database optimized for fast ingest and complex queries
- VictoriaMetrics - fast, cost-effective monitoring solution and time series database
- Cortex - A horizontally scalable, highly available, multi-tenant, long term Prometheus.
- Crate Data - is an open source massively scalable data store. It requires zero administration.
- MongoDB - Document-oriented database system.
- RavenDB - A transactional, open-source Document Database.
- RethinkDB - document database that supports queries like table joins and group by.
- Couchbase - The Modern Database for Enterprise Applications
- Apache Cassandra - Manage massive amounts of data, fast, without losing sleep
- TiKV - a highly scalable, low latency, and easy to use key-value database.
- BadgerDB - an embeddable, persistent and fast key-value (KV) database written in pure Go
- FoundationDB - scalable distributed database
- RocksDB - embeddable persistent key-value store for fast storage based on LevelDB.
- Citus - Distributed PostgreSQL
- Cockroach - Scalable, Geo-Replicated, Transactional Datastore.
- TiDB - TiDB is a distributed SQL database
- yugabyteDB - open source, high-performance, distributed SQL database compatible with PostgreSQL
- MariaDB - enhanced, drop-in replacement for MySQL
- Apache Calcite - the foundation for your next high-performance database
- Vitess - A database clustering system for horizontal scaling of MySQL
- VoltDB - claims to be fastest in-memory database.
- BerkeleyDB - a software library that provides a high-performance embedded database for key/value data.
- LevelDB - a fast key-value storage library written at Google that provides an ordered mapping from string keys to string values.
- Memcached - high-performance, distributed memory object caching system
- Azure Arc - Azure Arc simplifies governance and management by delivering a consistent multicloud and on-premises management platform.
- EKS Anywhere - Create and operate Kubernetes clusters on your own infrastructure
- AKS Edge - AKS Edge Essentials is an on-premises Kubernetes implementation running containers at scale
- K3s - The certified Kubernetes distribution built for IoT & Edge computing
- microk8s - The best Kubernetes experience for developers, DevOps, cloud and edge
- Cilium - eBPF-based Networking, Observability, and Security
- Calico - most widely adopted solution for Kubernetes networking and security
- Flannel - Flannel is a simple and easy way to configure a layer 3 network fabric
- Antrea - Kubernetes networking based on Open vSwitch
- KubeEdge - An open platform to enable Edge computing
- Kube Router - networking solution built for operational simplicity and performance.
- Liqo - P2P, Seamless, Dynamic and Secure Resource Sharing
- Admiralty - The simplest way to deploy applications to multiple Kubernetes clusters.
- Submariner - direct networking between Pods and Services in different Kubernetes clusters
- Skupper - Skupper Multicloud communication for Kubernetes
- Istio - the leading multi-cluster service mesh
- Linkerd - multi-cluster services with security, observability, and reliability
- Consul - Service Mesh for any runtime or cloud
- Traefik Mesh - Traefik Mesh is a straight-forward, easy to configure, and non-invasive service mesh
- Kuma - The universal Envoy service mesh for distributed service connectivity
- Istio - the leading service mesh
- Serf - decentralized solution for service discovery and orchestration
- etcd - a distributed, reliable key-value store for the most critical data of a distributed system
- Lightbend Akka - runtime for distributed, and fault tolerant event-driven applications on the JVM.
- Lightbend Lagom - opinionated microservices framework
- GoKit - A toolkit for microservices
- Goa - Goa provides a holistic approach for developing remote APIs and microservices in Go.
- Micro - Micro is a cloud platform for API development
- Dapr - Dapr is a portable, event-driven, runtime for building distributed applications across cloud and edge.
- Rook - Open-Source, Cloud-Native Storage for Kubernetes
- Minio - Object Storage for the Era of the Hybrid Cloud
- Longhorn - Cloud native distributed block storage for Kubernetes
- OpenEBS - OpenEBS builds on Kubernetes to enable Stateful applications to easily access Persistent Volumes
- Serverless - zero-friction serverless development
- Apache Camel - easily integrate various systems consuming or producing data.
- Apache Openwhisk - Open Source Serverless Cloud Platform
- Fission - Open source, Kubernetes-native Serverless Framework
- KNative - Serverless Kubernetes functions
- Kubeless - The Kubernetes Native Serverless Framework
- Kyma - Extend your applications with Kubernetes
- Nuclio - High-Performance Serverless event and data processing platform