🔭 Galileo

Builders often ask us "What's the best LLM for RAG?" 🤔 To help answer this question, our latest LLM Hallucination Index evaluates 22 leading models across 3 RAG tasks. We provide exclusive insights on cost, verbosity, and performance to help teams choose the optimal LLM for their use case. Some of our key findings: 🏆 Anthropic outperforms OpenAI – Claude beats GPT. ⭐ Open-source is quickly closing the gap with closed-source – Meta and Alibaba Group models show great promise. 🐜 Larger is not always better – Llama outperformed bigger models. 📏 Context length matters – just because a model is effective with long context, doesn't mean it's the best choice for short. Get the complete LLM Hallucination Index report in the comments! #ML #MachineLearning #RAG #LLM #GenAI #AI #ArtificialIntelligence #DataScience #DataScientist #DataEngineer #DataEngineering #CIO #CTO #CDO

Missed our #LLMOps Summit, but want to catch all the goodness? Now you can! Here's a playlist with all of the talks: https://pbase.ai/47lSneN Last week 150 #AI leaders and ML engineers joined us in Downtown San Francisco to discuss the future of #GenAI with a focus on how to make a big impact with small models (#SLMs). The energy was 🔥 and the talks were packed with insights! Here’s a list of talks that you'll find on the playlist: 👏 Devvret Rishi, Co-founder and CEO at Predibase, Welcome Address 🍎 Piero Molino, Co-founder and CSO at Predibase: Why Apple and Other AI Leaders are Betting Big on Small Language Models ✅ Vlad Bukhin, Staff ML Engineer at Checkr, Inc.: Streamlining Background Checks with Fine-tuned SLMs 🔥 Arnav Garg. ML Eng Lead at Predibase: Next Gen LLM #Inference: Blazing Fast + Cost-Effective 🕵♂️ Atin Sanyal, Co-founder & CTO at 🔭 Galileo: Fine-Tuning SLMs for Enterprise-Grade Evaluation  🛠 Maarten Van Segbroeck, Ph.D., Head of Applied Science at Gretel, Building Better Models Faster with #SyntheticData Big thank you to all of our speakers and attendees for an amazing event! See you at the next one 🎉

👨🔬 9 Tips for Evaluating Enterprise RAG Define Clear Objectives: Understand the specific requirements and objectives of the RAG system within your enterprise. For example, a healthcare organization may prioritize accuracy and up-to-date information in medical queries. Select Appropriate Benchmarks: Choose benchmarks that align with your objectives and cover the relevant domains and task types. For instance, a financial institution may benefit from using benchmarks that include financial reports and numerical reasoning tasks. Comprehensive Testing: Conduct thorough testing across various dimensions, including context length, domain, and performance metrics. Use a combination of benchmarks like ChatRAG-Bench and Hallucination Index to get a holistic view of the model's capabilities. Continuous Monitoring and Updating: Regularly monitor the performance of the RAG system and update it as needed to ensure it remains accurate and reliable. Implement a feedback loop to incorporate new data and improve the system continuously. Evaluation Tooling: Utilize specialized tools and frameworks to automate and streamline the evaluation process. High fidelity metrics like ChainPoll can help detect hallucinations and improve answer accuracy. These tools can save time and provide more consistent and objective evaluations compared to manual assessments. Human Evaluation: Incorporate human evaluations to assess the quality of the generated responses and identify areas for improvement. Human evaluators can provide insights into the nuances of the responses that automated metrics might miss. For instance, in legal or medical domains, experts can evaluate the relevance and accuracy of the information provided by the model. Scenario-Based Testing: Create specific scenarios that reflect real-world use cases within your enterprise. For example, a customer support system might be tested on its ability to handle a variety of customer queries, while a research assistant might be evaluated on its ability to retrieve and summarize the latest research papers. Domain-Specific Customization: Customize the evaluation process to account for the unique challenges and requirements of your domain. This might involve creating custom datasets, developing specialized metrics, or tailoring benchmarks to better reflect the specific tasks and information needs of your enterprise. Error Analysis: Conduct detailed error analysis to understand the types of mistakes the RAG system makes. This involves categorizing errors, identifying common failure points, and analyzing why these errors occur. By understanding the root causes of errors, you can make targeted improvements to the system, enhancing its overall performance and reliability. 🔗 https://lnkd.in/guSvamrp #ML #MachineLearning #AI #ArtificialIntelligence #LLM #DataScience #DataScientist #DataEngineering #DataEngineer

👉 Lessons from launching Enterprise GenAI solutions at Coinbase 🎯 Continuous Accuracy Evaluation Every iteration of LLMs brings improvements. However, the frequent updates and changing leaderboard positions mean companies must continually reassess which LLM to use. Flexibility in choosing and adapting LLMs is a must for maintaining high performance. 🛡 Trust and Safety Ensuring LLMs don’t produce harmful outputs (hallucinations or jailbreaking) is essential. Companies must build guardrails, as trust and safety benchmarks are often insufficient. This demands varying levels of effort in tuning and prompting different LLMs. ⚖ Balancing Cost & Performance LLMs come with diverse capabilities and price tags. For instance, GPT-4 is 10x more expensive than GPT-3.5, and Anthropic's Claude 3 Opus is 60x the price of Claude 3 Haiku. Strategic selection based on task requirements (like using a cheaper LLM for profanity checks) can optimize costs without compromising on necessary accuracy. ⏱ Managing Latency Latency can range broadly. For real-time applications like chatbots, low latency is essential. However, in batch processing scenarios such as web page translations, higher latency is often more acceptable. 🔋 Ensuring Availability With the high demand for LLM models and the current GPU chip shortage, securing adequate capacity is challenging. This involves negotiating with cloud providers like AWS or GCP. For customer-facing products, balancing user experience with latency and quality of service during high loads is critical. Coinbase's experience proves that a holistic approach prioritizing strategic decision-making and flexibility are paramount for leveraging GenAI effectively: https://lnkd.in/dzkjPy_M #GenAI #AI #ArtificialIntelligence #ML #MachineLearning #DataScience #DataEngineering #DataScientist #DataEngineer #LLMOps #LLM

Small, cost-effective models may be all you need. So how can you build the #GenAI architecture of the future? Join Predibase, Gretel, Checkr, Inc. and us for the LLMOps Micro-Summit: https://lnkd.in/g8Huhj2g 📺 Can't make it in-person? Watch the live stream: https://lnkd.in/gv76krEr #LLM #LLMOps #AI #ArtificialIntelligence #ML #MachineLearning #DataScience #DataScientist #DataEngineer #DataEngineering #CTO #CIO #CDO

🕵♂️ Everyone is talking about AI agents, but do you know how to actually build and deploy agentic apps at scale? Thankfully Taranveer Singh from Chegg Inc. is here to show you how! Register now to learn... 🛠 How to build scalable, reliable, highly personalized agentic solutions 📋 Best practices for bringing agentic solutions to production 🔭 Ways to improve quality, unlock new capabilities, and accelerate development 🔗 Don't miss it: https://lnkd.in/gwPrr5qF #LLM #GenAI #AI #Agents #ArtificialIntelligence #ML #MachineLearning #DataEngineer #DataEngineering #DataScience #DataScientist

🗣 Multi-Agent Customer Interaction Systems 🤩 Want to build a sophisticated multi-agent system for customer interactions? Laurie Voss shares how in a great deep-dive post! It features a diverse array of specialized sub-agents, complemented by three key meta-agents: 🛎 Concierge – This agent initiates customer engagement, deciphering complex, multi-step requests that may require the expertise of multiple agents. 🎼 Orchestrator – Tasked with identifying and deploying the most suitable sub-agent for each immediate task at hand. 👷♂️ Continuity Manager – This agent maintains a holistic view, assessing whether the customer's original request has been fully addressed or if additional sub-agent interventions are necessary. This architecture allows each sub-agent to excel in its specific domain without the need for an unwieldy, all-encompassing prompt. The modular and scalable approach offers tremendous potential for a wide breathe of applications! 🔗 https://lnkd.in/dB34RCu6 #ML #MachineLearning #AI #ArtificialIntelligence #DataEngineer #DataEngineering #Agents #GenAI #LLM #Dev

👩🔬 Want to know how to evaluate LLMs for your RAG use case? We published a deep-dive post covering... Dimensions to Evaluate for RAG Challenges in RAG Evaluation - Issues with Vibe-Check Based Evaluation - Issues with LLM-as-a-Judge Approaches - Biases in LLM-as-a-Judge Approach Metrics for Evaluating LLM in RAG - RAGAS - Trulens - ChainPoll - Context Adherence - Luna RAG Metric Comparison RAG Benchmarks Limitations of Benchmarks 🔗 in the comments 👇 #ML #MachineLearning #AI #ArtificialIntelligence #DataScience #DataScientist #DataEngineer #DataEngineering #LLM #LLMOps #GenAI

🍎 Pioneered by Apple, the new GenAI stack is built on small models and cost-effective inference without sacrificing performance. So what does it take to build like Apple? Join 🔭 Galileo, Predibase, Gretel, Checkr, Inc., and Apple to learn how to build next-gen #GenAI architecture: https://lnkd.in/gSFKfw-V 📺 Can't make it in-person? Watch the live stream: https://lnkd.in/gzScEKM4 #LLM #LLMOps #AI #ArtificialIntelligence #ML #MachineLearning #DataScience #DataScientist #DataEngineer #DataEngineering #CTO #CIO #CDO

👉 While many teams have been building #LLM applications for over a year now, there is still much to learn about RAG and all types of hallucinations. Check out our roundup of the top #GenAI and LLM articles for August: https://lnkd.in/eZXhb7RJ #AI #ArtificialIntelligence #ML #MachineLearning #DataScience #DataScientist #DataEngineer #DataEngineering