What are event-driven APIs? How do they differ from REST APIs? How to use the Webhooks, WebSockets, and Server-Sent Events to build them? — In this article, you’ll learn the foundations of event-driven APIs, how they interact with consumers, the technology choices to build them, and how to document them with AsyncAPI specification. Polling is dead. We must move on We, as information consumers, have a craving desire to know things as they happen.

Why Dapr excels at building distributed, loosely-coupled, event-driven Microservices — Microservices architectures are inherently distributed. Building Microservices always bring in the most challenging problems, such as resilient service invocation, distributed transactions, on-demand scaling, and exactly-once processing of messages. Putting Microservices on Kubernetes doesn’t always solve these problems as Kubernetes doesn’t have an application logic. Frameworks like Spring Boot, Akka, and…

How to detect, capture, and propagate changes in source databases to target systems in real-time, event-driven manner — This post serves as an introduction to the Change Data Capture (CDC) practice, rather than a deep-dive on a particular tool. First, I will explore the motivation behind CDC and illustrate the components of a real-time event-driven CDC system. …

How WebAssembly powered data transformations are changing the data scrubbing story for streaming data platforms? — Data scrubbing or massaging is a critical aspect of data engineering, especially in streaming data pipelines where the data must be cleaned, filtered, and scrubbed in real time with the help of stream processing engines or some Python jobs. However, that is about to change. Streaming data platforms have started…

This is the first post of the series that shows building operational use cases with Apache Kafka and Apache Flink. — Apache Kafka is a distributed streaming data platform. It is designed for high throughput, low-latency streaming workloads, where scalable real-time data ingestion and fault-tolerant storage are critical. Apache Flink is an open-source, unified stream-processing and batch-processing framework capable of executing arbitrary dataflow programs on data streams.

How do these two technologies work? how do they differ, and when is the right time to use them? — Choosing between a stateful stream processor and a streaming database has been a debatable question for a long time. I’ve been combing through the Internet to find a few rationales, failed, and decided to write this post to share my experience and knowledge with you. After reading this, you should…

How to build an incrementally updated materialized view that serves queries in a faster and scalable manner? — Microservices architecture promotes a decentralized data management practice, where each service keeps its data private and exposes it only via well-defined API interfaces. Although that is for the greater good, developers find it challenging to implement queries that span across multiple service boundaries.

How to build an incrementally updated materialized view that serves queries in a faster and scalable manner? — This post is the second and final installment of the article series that explores building a read-optimized view with Debezium, Kafka, Materialize, and Apache Pinot. Part 1 discussed the problem space of building an online pizza order tracker and a possible solution architecture. Part 2 will walk you through the…

How to build a streaming data pipeline to capture MySQL database changes and stream them to Apache Pinot via Debezium and Kafka — Upserting means inserting a record into a database if it does not already exist or updating it if it does exist. Analytics database at the end of a streaming data pipeline can benefit from upserts to maintain the data consistency with the source database. This article explores a minimal viable…