Redpanda Data

Big news!!! 🎊 Redpanda has acquired the popular stream processing and connectivity framework Benthos. That means with our new Redpanda Connect we're now the most comprehensive end-to-end streaming data platform! 🔗 Gone are the days of suffering through multiple projects that manually stitch data from your streams to your data warehouse or operational database. Read what our founder Alexander Gallego has to say about all that's new at Redpanda. 👇 https://lnkd.in/gpUfGYgs

🎥 TUNE IN LIVE: Emergence of the Live Data Stack Join us for a live virtual event hosted by Joe Reis, co-author of The Fundamentals of Data Engineering, and Alex Gallego, founder and CEO of streaming data pioneer Redpanda.

One hour to go until 🚀 Alexander Gallego and I host Redpanda Data’s Emergence of the Live Data Stack! We will discuss the next generation of data intensive applications and how to move from batch to real-time. You also get to hear from special guests from ShareChat and The Hotels Network. Data doesn’t sleep. Join today. Register here: https://lnkd.in/gZJQVPgN

We are excited to announce that Shubham Dhal, one of our tech engineers, will speak at Redpanda’s events on the Emergence of Live Data Stack on July 17th at 9:30 PM IST / 9 AM PST. It is a perfect opportunity to learn how leading tech companies like ShareChat leverage streaming systems at scale to power their ML Pipelines and Infra. Also, learn how real-time processing replaces batch with SLAs moving from days/hours to seconds! Be part of the event: https://bit.ly/3Y6XEnH. Redpanda Data #Redpanda #ML #AI #Data #streaming

In Redpanda 24.1 we introduced write caching to turbocharge performance and enhance data safety while lowering storage costs. But here at Redpanda, we really love #data. So, we ran a benchmark to back our claims with hard numbers. In this post, we take you behind the scenes and show you exactly how write caching helps you shift your #streamingdata into gear 🏁 https://lnkd.in/ejdieKQM

🛋 Want to build a #streamprocessing app in 45 mins without leaving your sofa? Then today is your lucky day. Join Dunith Danushka at the #BigData Demystified #London virtual event to build a stream processing app with Redpanda and Apache #Flink! 🗓 Tuesday, July 16, 2024 🕓 4 PM to 4:45 PM BST If you're a #dataengineer or #developer just starting out with #streamingdata, this event is for you 👌 Sign up to join👇 https://lnkd.in/eQQ6EgBQ

🎥 LIVESTREAM JULY 17: Emergence of the Live Data Stack Want to learn how leading companies are implementing a “live data stack” for one-click responsiveness? What about how #realtime processing is replacing batch, real-time storage is replacing the #data warehouse, or how #AI is taking over #analytics? It's mind-boggling stuff that you definitely won't want to miss. So sign up to make sure you don't!👇 🗓 Wednesday, July 17th 🕘 9 am PDT | 12 pm EDT | 5 pm BST https://lnkd.in/gEE-N6jK

How to build read-optimized denormalized views to feed the UIs that are read-heavy and latency-sensitive? Special thanks to Gunnar Morling for providing feedback on this post 🙏 🎯 The problem Consider a customer profile UI you would see on an online store. This UI could display essential customer information along with recent orders. Populating this customer profile UI requires the UI Microfrontend to fetch necessary data from two domain services, Customers, and Orders, and join by the customerId. The UI Microfrontend does this on demand, every time the UI is loaded. It will invoke Customer and Order services on the fly, wait for both services to respond, transform the data as necessary, and respond to the UI with the data it expects. The problem with this approach is the added latency due to network calls and joins. Not to mention the complexity of error handling. Moreover, this becomes difficult to scale when the load on the UI increases. ✅ A solution with Change Data Capture (CDC) Instead of fetching and joining data on demand, what if we maintain a denormalized view in a read-optimized location, such as in a cache, keyed by the customer ID? That way we could avoid costly joins and transformations to reduce the read latency. But how do we keep this denormalized view fresh and up to date with upstream data sources? We will use CDC to keep this cache updated whenever Customer or Order databases change. Here’s how we’d do that. Firstly, we need to configure customers and order databases with a CDC system, such as Debezium. That way, the CDC system can detect the database changes in real-time and stream them into two Redpanda topics as change events. Once the change event streams landed in Redpanda, we have two options: 1️⃣ Option 1 - Use a Stream Processor, like Apache Flink, to join two change streams and build a denormalized view. The stream processor will update the cache as it receives change events from either of the upstream sources. This view can be sinked to a cache or key-value store, like Redis, via Redpanda Connect’s sink connector. For example, the value can be the denormalized view formatted as JSON (customer information + orders) while the customerID serves as the key. If there’s an upstream change, the value could be updated with the latest. 2️⃣ Option 2 - Use a streaming database to ingest the change data feeds, maintain the materialized view for UI, and also serve it directly. Streaming databases are good at building and maintaining denormalized views as incrementally updated materialized views. The benefit here is that most streaming databases are also optimized for reads. So you can serve the view directly to the UI service, without using an additional component. Both options avoid costly ad-hoc API calls, in-memory joins, and the burden of error handling from Microservices. The read latency on the UI will be reduced significantly as it reads from a denormalized view, where, the join has already been done.

Please join us for our upcoming [In-Person + Online] Stream Processing Meetup on Wednesday, July 24th (5:30 - 8:00pm US/Pacific)! We will have 3 exciting presentations: Kafka @ GitHub: Challenges and future direction - Eric Sun, GitHub Revenue alerting at scale - Arsh Khandelwal & Kshitij Grover, Orb Streaming Data Enrichment and Analysis with Redpanda, CDC, and Python - Bryan Wood, Redpanda Data Hope to see you there! #linkedin #streamprocessing #apachekafka #apachesamza #apacheflink #apachebeam

📣 LAST CALL: Getting started with Redpanda Today is the day! Join our friendly pandas for a free online session all about Redpanda's architecture, deployment options, practical use cases, and best practices. If you're just entering the #streamingdata scene, then this is the best welcome pack to our...pack 🐾 🗓 Wednesday, July 10th 🌎 Americas: 9 am PT | 12 pm ET 🌍 EMEA: 8 am ET | 1 pm BST You're still in time to join! Sign up here👇 https://lnkd.in/gjd75ZgJ