What are event-driven APIs? How do they differ from REST APIs? How to use the Webhooks, WebSockets, and Server-Sent Events to build them? — In this article, you’ll learn the foundations of event-driven APIs, how they interact with consumers, the technology choices to build them, and how to document them with AsyncAPI specification. Polling is dead. We must move on We, as information consumers, have a craving desire to know things as they happen.

Why Dapr excels at building distributed, loosely-coupled, event-driven Microservices — Microservices architectures are inherently distributed. Building Microservices always bring in the most challenging problems, such as resilient service invocation, distributed transactions, on-demand scaling, and exactly-once processing of messages. Putting Microservices on Kubernetes doesn’t always solve these problems as Kubernetes doesn’t have an application logic. Frameworks like Spring Boot, Akka, and…

How to detect, capture, and propagate changes in source databases to target systems in real-time, event-driven manner — This post serves as an introduction to the Change Data Capture (CDC) practice, rather than a deep-dive on a particular tool. First, I will explore the motivation behind CDC and illustrate the components of a real-time event-driven CDC system. …

How do these two technologies work? how do they differ, and when is the right time to use them? — Choosing between a stateful stream processor and a streaming database has been a debatable question for a long time. I’ve been combing through the Internet to find a few rationales, failed, and decided to write this post to share my experience and knowledge with you. After reading this, you should…

How to build an incrementally updated materialized view that serves queries in a faster and scalable manner? — Microservices architecture promotes a decentralized data management practice, where each service keeps its data private and exposes it only via well-defined API interfaces. Although that is for the greater good, developers find it challenging to implement queries that span across multiple service boundaries.

How to build an incrementally updated materialized view that serves queries in a faster and scalable manner? — This post is the second and final installment of the article series that explores building a read-optimized view with Debezium, Kafka, Materialize, and Apache Pinot. Part 1 discussed the problem space of building an online pizza order tracker and a possible solution architecture. …

How to build a streaming data pipeline to capture MySQL database changes and stream them to Apache Pinot via Debezium and Kafka — Upserting means inserting a record into a database if it does not already exist or updating it if it does exist. Analytics database at the end of a streaming data pipeline can benefit from upserts to maintain the data consistency with the source database. This article explores a minimal viable…

How do speed, accuracy, and cost-effectiveness influence the choice between batch and streaming analytics? — We use data analytics to make better and improved business decisions. Data analytics systems can be broadly categorized into batch and streaming (real-time) analytics systems based on their event time and processing time semantics. Completeness, speed, and cost are crucial for selecting batch or streaming analytics systems to implement a…

Why modern streaming stack is replacing the classic streaming architecture? What’s the composition and what values it brings? — Lately, I’ve heard buzzes around “unbundling” on data Twitter. Their focus was primarily on data engineering, spurring many conversations and leading to many open-ended questions. I’ve been living in the data space for a while, focusing on event streaming and data analytics. So I decided to “unbundle” the modern real-time…