Redpanda Data

🚀 WHAT'S NEW: Redpanda 24.2 and Azure integration for Redpanda Cloud Redpanda 24.2 is out and ready to roll! This exciting release is packed with goodies for #developers, #enterprise, #compliance, and much more 👀 Here's a peek at what's new in 24.2: ☁ 𝐀𝐳𝐮𝐫𝐞 𝐥𝐢𝐦𝐢𝐭𝐞𝐝 𝐚𝐯𝐚𝐢𝐥𝐚𝐛𝐢𝐥𝐢𝐭𝐲 for Redpanda Cloud's BYOC with six new regions and Azure Private Link support for extra security. 🔥 𝐑𝐞𝐝𝐩𝐚𝐧𝐝𝐚 𝐂𝐥𝐨𝐮𝐝 𝐒𝐞𝐫𝐯𝐞𝐫𝐥𝐞𝐬𝐬 𝐮𝐩𝐝𝐚𝐭𝐞𝐬, including schema registry support, a 99.5% uptime SLA, and Cloud API. 🙌 𝐀 𝐬𝐥𝐞𝐞𝐤𝐞𝐫 𝐝𝐞𝐯 𝐞𝐱𝐩𝐞𝐫𝐢𝐞𝐧𝐜𝐞 with handy features for our in-broker Data Transforms and built-in Schema Registry. 🔎 𝐄𝐱𝐩𝐚𝐧𝐝𝐞𝐝 𝐂𝐥𝐨𝐮𝐝 𝐜𝐨𝐦𝐩𝐥𝐢𝐚𝐧𝐜𝐞 with a HIPAA self-attestation, a FIPS-compliant binary, and more SOC 2 Type 2 controls. The real treasure is in the details, so read our release blog here👇 https://lnkd.in/gfwAkwX6

Redpanda may be a fully remote company, but our pack still loves the chance to get together in person! This summer, teams in Austin, TX, Bay Area, CA, and London, UK gathered together for social meet-ups with Redpandas in their local area. ☀️ We look forward to more teams connecting this fall! 🍁 #teamredpanda

I will be in Bangalore this weekend, speaking at two events. The first event is ServerlessDays Bengaluru, where I'll discuss building a serverless streaming ETL pipeline with Redpanda. Event details are as follows. 🕰️ Agenda: https://lnkd.in/eAZRHbMk 🗓 Date: August 31st, 2024 🕘 Time: 9:00 AM - 6:00 PM IST 📍 Venue: Radisson Blu, Marathahalli If you haven’t already, secure your spot now by visiting https://konfhub.com/sdb The second event, organized by the Bengaluru Streaming community and hosted by the folks at Platformatory, takes place on the same evening. I'll be presenting on "Building a Real-Time API Usage Analytics Platform with Redpanda (Kafka), Flink, and Pinot." Join us this Saturday, August 31, at WeWork Embassy TechVillage. RSVP: https://lnkd.in/g93T_SfN I'd love to connect with you during my time in Bangalore! If you're passionate about streaming data, analytics, serverless architecture, or GenAI, let's grab a coffee and chat. It's a great opportunity to share insights, discuss the latest trends, and explore potential collaborations in these exciting fields. Feel free to reach out to me on LinkedIn or Twitter to set up a meeting. Looking forward to engaging in conversations and meeting fellow tech enthusiasts! #ServerlessDaysBLR2024 #Bangalore

ICYMI: Redpanda uses #Seastar's heap memory profiler, but it's costly and unusable in production. Since our #engineers are such curious and determined pandas, they developed a few (seriously cool) optimizations 💪 The result? Instant detection of what's guzzling the most memory at runtime to swiftly debug and fix out-of-memory (OOM) issues 🏆 Read how they did it in just 5 instructions👇 https://lnkd.in/eYXcStBM

This week's sketchnote is about data freshness. 🚀 Understanding Data Freshness in Data Engineering 🚀 Data freshness is a critical concept in data engineering. It directly impacts the accuracy, relevance, and overall trustworthiness of the data we use to make decisions. 📊 What is Data Freshness? It’s all about how up-to-date your data is. Fresh data leads to better, more informed decisions. Outdated data? Not so much. ⏱️ How to Measure It? The key metric is Freshness Lag—the time difference between when data is captured and when it’s available for use. 🔧 Why It Matters: • Accuracy: Fresh data = accurate insights. • Relevance: Outdated data can mislead. • User Experience: Think real-time updates in your favorite apps. 📈 Improving Data Freshness: - Optimize Data Ingestion: Streamline the ETL processes to reduce latency. This can include parallel processing, incremental data loading, and real-time data streaming. - Enhance Pipeline Architecture: Design a more efficient data pipeline with fewer stages and optimized processing at each stage. Consider using micro-batch processing or event-driven architectures. - Leverage Modern Technologies: Utilize technologies such as Apache Kafka/Redpanda, Apache Flink, or cloud-based data services that support real-time data processing and low-latency data flows. - Monitor and Alert: Implement monitoring tools to track data freshness metrics and set up alerts for any significant delays or staleness. - Improve Network Infrastructure: Enhance network capabilities to reduce latency and ensure faster data transfer between systems. - Scale Resources: Ensure that the systems involved in the data pipeline have adequate resources (CPU, memory, storage) to handle the data volume and processing requirements efficiently. - Automate Data Quality Checks: Use automated tools for data validation and cleansing to minimize the time spent on manual checks and ensure the timely availability of fresh data. 👇 What strategies do you use to keep your data fresh? Share your thoughts! 👇 #DataEngineering #DataFreshness #DataScience #BigData #MachineLearning #TechEducation #Sketchnote

The rise of interconnected proprietary AI models. 🤖 With data as the most important digital asset across the enterprise, companies aren’t going to just give it up – even for exciting AI use cases. 📈 In #theCUBE’s latest segment from the AI Infrastructure Silicon Valley - Executive Series event, Redpanda Data CEO 🚀 Alexander Gallego discusses companies building their models and creating a network of integrations through APIs. 🗣️ “People think that this is an anomaly, but then you go on and you talk to the largest drug manufacturing company, and they're like, ‘I just spent $6 billion developing this new drug that's never leaving my five layers of network security,’ That's more the case than the exception," Gallego said. The adoption of specifications, such as the HTTP interface from OpenAI, allows for seamless communication between different providers while keeping data secure. 📺 Watch the full interview:  https://lnkd.in/g9A4tfTK #AIHardwareAndInfrastructure #theCUBE #AIHardware #GenAIHype #datastreaming

The Bengaluru Streaming community is meeting this weekend for interesting talks on streaming, particularly, building a Real-Time API Usage Analytics Platform with Redpanda (Kafka), Flink and Pinot by Dunith Danushka, Developer Advocate at Redpanda Data. The Platformatory team will also be talking about building an Unified access layer for Kafka resources as part of the Platform strategy. Join us this Saturday (Aug 31) at the WeWork Embassy TechVillage for an evening filled with interesting conversations on the latest streaming tech. RSVP: https://lnkd.in/g93T_SfN #kafka #flink #pinot #redpanda #meetup

💻 WEBINAR - Sovereign AI: private, traceable, secure apps The world wants to build with #GenAI, but they don't want to share their #data and have serious security concerns. If this is you, then join Tyler Rockwood for an insightful webinar from The Linux Foundation to learn how Redpanda's Sovereign #AI lets you build with confidence thanks to its guaranteed privacy, trust, and security 💪 🗓 Tuesday, August 27 🕘 9 am (PDT) Sign up to join👇 https://lnkd.in/gP2ETnAU

Overly complex architectures, temperamental dependencies, and fluctuating #cloud costs are enough to make anyone sick of #streamingdata 🤒 Our prescription? Try Redpanda Serverless! With instantly available clusters, no infrastructure to manage, and pay-as-you-go pricing — it's proven to help you stream #data faster and easier 🌴 Try Serverless for free👇 (You'll feel better after the first dose. Promise.) https://lnkd.in/ga4j-Y6S #streamingdatadoodles

Game, Set, Match: Transforming Live Sports with AI-Driven Commentary. The recording of the talk that Mark Needham and I delivered at Devoxx UK is now available. There, we built an AI-assisted commentary bot based on a score from a live Tennis match. You may extend the model to suit other real-time use cases, like automated live commentary on motor traffic, weather, and auctions, etc. Feel free to watch our talk and let us know your thoughts.