Design of an IoT data pipeline
Don’t tell me what to do
The requirements
- Small low-powered IoT devices going to send measurement data.
- Scalable data flow and processing, something that handles backpressure.
Design Decisions
- Accepting MQTT message from the IoT devices because making HTTP request would lead to higher power consumption for a low-powered IoT device.
- I could’ve picked OPC UA or some other protocol. However, the idea is to fire events from Message Broker (mosquitto) to Kafka.
- Kafka is the perfect choice because of high throughput, scaling, availability, storage, buffering and reprocessing of events.
- Picked InfluxDB because it’s time series database. Could’ve used Prometheus.
Data Flow
How to display data
- APIs can be built to fetch data from InfluxDB.
- Grafana dashboard can be built on top of InfluxDB data.
Stop talking and show me the code
Live long and prosper!