What are event-driven APIs? How do they differ from REST APIs? How to use the Webhooks, WebSockets, and Server-Sent Events to build them? — In this article, you’ll learn the foundations of event-driven APIs, how they interact with consumers, the technology choices to build them, and how to document them with AsyncAPI specification. Polling is dead. We must move on We, as information consumers, have a craving desire to know things as they happen.

Why Dapr excels at building distributed, loosely-coupled, event-driven Microservices — Microservices architectures are inherently distributed. Building Microservices always bring in the most challenging problems, such as resilient service invocation, distributed transactions, on-demand scaling, and exactly-once processing of messages. Putting Microservices on Kubernetes doesn’t always solve these problems as Kubernetes doesn’t have an application logic. Frameworks like Spring Boot, Akka, and…

How to detect, capture, and propagate changes in source databases to target systems in real-time, event-driven manner — This post serves as an introduction to the Change Data Capture (CDC) practice, rather than a deep-dive on a particular tool. First, I will explore the motivation behind CDC and illustrate the components of a real-time event-driven CDC system. The latter parts discuss some potential use cases where CDC is…

A technology-agnostic explanation of stateless operators in stream processing. — If you read my recent post on stream processing, you should already understand how a basic stream processing program works. We learned that a stream processing program can be modeled, understood, and executed as a dataflow graph. This graph comprises operators collectively producing the desired output when data elements flow…

Let’s understand the basics of stream processing by exploring the principles of Dataflow programming. — Are you new to stream processing? If yes, then you are reading the right post. This post will walk you through the foundational concepts to understand stream processing, especially introducing you to dataflow programming, which helps you build a mental model to understand stream processing. I can’t cover everything about…

Not all real-time analytics systems are made equal by design. We can classify them into four groups based on five dimensions. — Real-time analytics systems derive actionable insights from continuous data streams, enabling organizations to make informed decisions and act fast. Today, we see different variations of real-time analytics systems trying to deliver the same objective from different perspectives, ranging from stream processors to real-time data warehouses, streaming databases, and much more…

How does the Bring Your Own Cloud (BYOC) model achieve data privacy and sovereignty of self-hosting with the ease and scalability of fully managed services? — In today’s fast-paced digital landscape, maintaining data privacy within an organization is paramount. Data breaches have far-reaching consequences, both financially and reputationally. According to statistics from the Ponemon Institute, the average cost of a data breach in 2020 was $3.86 million, and the average number of records exposed per breach…

How WebAssembly powered data transformations are changing the data scrubbing story for streaming data platforms? — Data scrubbing or massaging is a critical aspect of data engineering, especially in streaming data pipelines where the data must be cleaned, filtered, and scrubbed in real time with the help of stream processing engines or some Python jobs. However, that is about to change. Streaming data platforms have started…

This is the first post of the series that shows building operational use cases with Apache Kafka and Apache Flink. — Apache Kafka is a distributed streaming data platform. It is designed for high throughput, low-latency streaming workloads, where scalable real-time data ingestion and fault-tolerant storage are critical. Apache Flink is an open-source, unified stream-processing and batch-processing framework capable of executing arbitrary dataflow programs on data streams.