This post serves as an introduction to the Change Data Capture (CDC) practice, rather than a deep-dive on a particular tool. First, I will explore the motivation behind CDC and illustrate the components of a real-time event-driven CDC system. The latter parts discuss some potential use cases where CDC is applicable and conclude with some open-source tools available in the market.

P.S I made a YouTube videos for this post to explain this visually as well. If you are interested, you can check it below.

The motivation behind CDC

Applications start with a small data footprint. Initially, a single database fulfils every data…

In the previous post, I talked about an Event Streaming Platform (ESP) concept, why you should have an ESP in your organization, and the high-level architecture of a typical ESP.

In this post, I will explain the storage layer of an ESP in detail. We’ll discuss the architecture and fundamental concepts of it while taking examples from popular ESPs out there.

Responsibilities of the storage layer

The storage layer is responsible for storing the ingested events in a highly scalable and durable manner. Simply put, it ensures that not a single event ingested is lost due to the failures of the entire ESP.

The storage…

News

Debezium 1.5.0.Final Released

Debezium 1.5.0 is out with transactional metadata for the MySQL connector, LogMiner-based CDC implementation for Oracle promoted from incubating to stable, support for RedisStreams in Debezium Server, and much more.

Apache Pulsar 2.7.1 is out

Apache Pulsar is a cloud-native, distributed messaging and streaming platform created at Yahoo! and now a top-level Apache Software Foundation project.

New Lambda Operator Guide in the AWS documentation

That provides opinionated guidance on key topics such as Event-driven architectures, application design, debugging, and optimization.

General event-driven architecture.

The Design of an Event Store by Emil Koutanov

Why Threat Modelling Should Be Part of Your Serverless Development by Sheen Brisals.

Anatomy of an Event Streaming Platform — Part 1…

Businesses today are dealing with a massive influx of real-time data. Financial transactions, application logs, website clickstreams, and IoT telemetry data are few examples.

As a business, you can’t simply ignore the importance of real-time data. Even if you do, your competitors won’t. Modern businesses use them to get meaningful insights in a timely fashion to stay ahead of the competition.

Crude oil is the best analogy we can apply to real-time data as well. They have to be extracted, transported, and refined to get a meaningful value. …

Wilfried,

Thanks for the question. I too had that confusion when I began to go through the specification.

However, this is how I understood that.

We create an API definition to tell how it will behave to the consumers of that API. That includes specifying message formats and operation names. In my opinion, this is like ordering our consumers what to do next.

If you are creating an API that publishes messages to the consumer, then the consumer should subscribe to it. That's the order I'm giving to the consumer. Conversely, if your API consumers messages from the outside, then…

Welcome to the very first issue of our weekly review. Starting from this week, we will be bringing you a curated set of content and news related to designing, building, and managing applications in the event-driven application space.

Let’s start by exploring Event Meshes.

Event Meshes

What is an Event Mesh?

This article gives an overview of Event Mesh, the business and technical value, and its core features.

Can we use a Service Mesh for Event-Driven Messaging?

In this article, Kasun Indrasiri explores the potential of using Service Meshes for Event-Driven Messaging and the key capabilities such a Service Mesh should have.

Event-driven Microservices

…

Event-driven applications are getting increasingly popular these days compared to their request-reply counterpart. When you want to build asynchronous, loosely coupled, highly scalable, and resilient applications, the Event-driven Architecture (EDA) is the best option to consider.

But designing, building, and managing event-driven applications are somewhat difficult as there are many principles, frameworks, and technologies to choose from. Today, we can see tons of learning materials around the Internet related to various event-driven technologies and how to get things done with them.

Whether you are a novice or an expert in the EDA space, it is always better to have a…

P.S I did some edits to reflect the feedback I received from the audience. Thanks for your valuable contribution. I expect more! :)

Everything in Kafka revolves around partitions. They play a crucial role in structuring Kafka’s storage and the production and consumption of messages.

Understanding partitions helps you learn Kafka faster. This article walks through the concepts, structure, and behavior of Kafka’s partitions.

Events, Streams, and Kafka Topics

Before diving into partitions, we need to set the stage here. So let’s look at some high-level concepts and how they relate to partitions.

Events

An event represents a fact that happened in the past. Events are…