In this article, you’ll learn the foundations of event-driven APIs, how they interact with consumers, the technology choices to build them, and how to document them with AsyncAPI specification.
We, as information consumers, have a craving desire to know things as they happen.
Where is my package right now? What’s the score of the game? How is Dogecoin performing today? Likewise, the list goes on. A majority of Internet users today would like information to be pushed towards them rather than pulling.
Microservices architectures are inherently distributed. Building Microservices always bring in the most challenging problems, such as resilient service invocation, distributed transactions, on-demand scaling, and exactly-once processing of messages.
Putting Microservices on Kubernetes doesn’t always solve these problems as Kubernetes doesn’t have an application logic. Frameworks like Spring Boot, Akka, and Quarkus offered great help when building distributed Microservices architectures. But they are tightly coupled to specific programming languages and often opinionated.
So developers remain searching for a portable runtime to build distributed, scalable, and event-driven Microservices.
Dapr is a portable, event-driven runtime that makes it easy for any developer to…
This post serves as an introduction to the Change Data Capture (CDC) practice, rather than a deep-dive on a particular tool. First, I will explore the motivation behind CDC and illustrate the components of a real-time event-driven CDC system. The latter parts discuss some potential use cases where CDC is applicable and conclude with some open-source tools available in the market.
P.S I made a YouTube videos for this post to explain this visually as well. If you are interested, you can check it below.
Applications start with a small data footprint. Initially, a single database fulfils every data…
Back in the day, analytics and BI has been restricted for the power users of an organization. Only a handful of analysts, managers, and data scientists had the chance of playing around with expensive analytics and BI toolsets.
Today, we see many businesses have started to expose a rich set of analytics directly to their end-users. For example, Internet-scale companies like LinkedIn does that with the “Who viewed your profile” feature. Twitter provides analysis for your tweets, Writing platforms like Medium provides analytics on your content.
Message consumers deal with state manipulations very often. Message brokers can send duplicate messages due to various reasons. So the consumers should master the art of ‘Idempotent message handling’ to
overcome severe flows in the application state.
In this post, I discuss a simple mechanism to detect and discard duplicate messages in message processing.
The execution cycle of a typical message consumer would look like this.
An event is an immutable record of a fact that happened in the past. Essentially, an event has a key, value, and timestamp. For example:
Although the Microservices maintain their state private to them, they hardly operate in isolation. Some business use cases require them to change their state first, then notify that change to a broader audience.
The Transactional Outbox pattern describes an approach for services to execute these two tasks atomically.
In this post, I discuss why dual writes in distributed systems are bad, how you can fix that with the outbox pattern, and some inner workings of the pattern. This post doesn’t stand up as an implementation guide but instead brings in the fundamentals you should use when implementing in production.
In typical API-based communication, the processing of some API requests goes beyond the scope of an HTTP request. Processing a large file, performing a calculation, or executing a workflow exceeds the regular HTTP request timeout.
To keep the UI responsive, we often carry out these tasks asynchronously. But in some cases, the UI demands a confirmation about the task’s execution status. Based on that, the UI shows a notification to the user about how the task execution went.
Usually, the backend updates a database once the task is completed. The front end keeps polling the database until it sees the…
The Microservices architecture mandates keeping service’s data private, promoting the use of database per service pattern.
That brings up challenges when one service tries to access the data owned by another service. There are two patterns to implement that; the API composition pattern and the CQRS based materialized view pattern.
This post digs into the details of using materialized views to scale out the inter-service query operations.
Before we dive deep, let me set the stage for our discussion.
Imagine we have the following customer loyalty platform built on top of Kafka.
Operational data accumulated in OLTP databases often need to be taken out to perform useful tasks other than transaction processing. That includes moving data out to data warehouses, updating caches, dashboards, etc.
Change Data Capture (CDC) is the process of observing all data changes written to a database and extracting them in a form in which they can be replicated to downstream systems for other purposes.
In this article, I discuss some real-world use cases where CDC is applicable. Most of the examples are related to Debezium, on open-source CDC engine built on top of Kafka.
If you are new…