What are event-driven APIs? How do they differ from REST APIs? How to use the Webhooks, WebSockets, and Server-Sent Events to build them? — In this article, you’ll learn the foundations of event-driven APIs, how they interact with consumers, the technology choices to build them, and how to document them with AsyncAPI specification. Polling is dead. We must move on We, as information consumers, have a craving desire to know things as they happen.

Why Dapr excels at building distributed, loosely-coupled, event-driven Microservices — Microservices architectures are inherently distributed. Building Microservices always bring in the most challenging problems, such as resilient service invocation, distributed transactions, on-demand scaling, and exactly-once processing of messages. Putting Microservices on Kubernetes doesn’t always solve these problems as Kubernetes doesn’t have an application logic. Frameworks like Spring Boot, Akka, and…

How to detect, capture, and propagate changes in source databases to target systems in real-time, event-driven manner — This post serves as an introduction to the Change Data Capture (CDC) practice, rather than a deep-dive on a particular tool. First, I will explore the motivation behind CDC and illustrate the components of a real-time event-driven CDC system. …

How do speed, accuracy, and cost-effectiveness influence the choice between batch and streaming analytics? — We use data analytics to make better and improved business decisions. Data analytics systems can be broadly categorized into batch and streaming (real-time) analytics systems based on their event time and processing time semantics. Completeness, speed, and cost are crucial for selecting batch or streaming analytics systems to implement a…

Why modern streaming stack is replacing the classic streaming architecture? What’s the composition and what values it brings? — Lately, I’ve heard buzzes around “unbundling” on data Twitter. Their focus was primarily on data engineering, spurring many conversations and leading to many open-ended questions. I’ve been living in the data space for a while, focusing on event streaming and data analytics. So I decided to “unbundle” the modern real-time…

Learn how to use Airbyte, Kafka, and Pinot to build a data pipeline for a user-facing analytics dashboard. — This post was originally published at Airbyte. Typically, a real-time analytics pipeline consists of several components, including ELT pipelines, an event streaming platform, and an analytical database that can answer queries at scale. However, building and maintaining such a platform is expensive and demands significant engineering effort and time.

Reverse ETL promotes operational analytics, avoids data silos, and enables operational teams to make informed decisions without leaving their trusted tools. — Reverse ETL has become a standard pattern in the Modern Data Stack. Today, organizations and practitioners leverage reverse ETL tools like Census and Hightouch to sync analytics back to the operational systems. That helps bring the analytics back to where it matters most; operational decision making.

Building scalable analytics infrastructure to deliver low-latency user-facing analytics with Apache Pinot, Kafka, and Debezium — This article is going to be a long one. So let me summarise the things that matter. What is user-facing analytics? What’s the business value of user-facing analytics Why achieving real-time user-facing analytics is hard? What are the options for building an analytics infrastructure to deliver user-facing analytics?

An attempt to capture the entire OLAP and data warehousing landscape into a single post, compare different technologies and predict what will come next in this space. — In this post, I will go back to the roots of OLAP systems and attempt to summarise different OLAP and data warehousing strategies from their early days to modern data space. At the end of the post, you will be able to capture the entire OLAP landscape.