Sourabh Agrawal’s Post

Pricing can be tricky, especially when you're busy getting your startup off the ground. We usually focus a lot on our products, tech, and sales, and sometimes, we don't give enough thought to how we're going to price our services. I recently attended a really helpful talk by Bill Bedsworth, who leads the pricing team at Stripe and has previously revolutionised pricing at companies like Adobe and Zendesk. Here are a few tips he shared: 1. Keep it simple: Your pricing should be easy to understand. It’s best when customers can quickly get it and feel comfortable with it. 2. Start low: If your platform has lots of features, don’t make it expensive right away. Start with a lower price so it’s easy for customers to give it a try, then adjust as they use more. 3. Know the value: Figure out how much your product helps your customers either make or save money. Price your product based on that value. 4. Check the competition: Make sure your prices are in line with what others are charging. Be competitive, especially in the beginning, and focus on winning over customers. At UpTrain (YC W23), we've made some mistakes with pricing in the past. The advice from Bill was a great reminder of how to approach pricing the right way. We’ve come a long way and iterated our pricing a lot since launching to ensure our pricing model always aligns with our customer needs. Also, a huge shoutout to Intercom and Darragh C. for their unique pricing model for Fin, which is based on the number of resolutions— it's a cool way to pay only for the value you receive!

The only right way to do prompt engineering is to follow a data-driven process. Vibe check can get you started but won't work as your usage scales. With UpTrain, you can now version prompts and seamlessly evaluate them on 20+ preconfigured metrics, spanning across response evals, context evals, conversation evals and many more. Check it out and let us know your feedback!

We have completely revamped the dashboards at UpTrain (YC W23) for a new, interactive and supercharged experience. Try them out!

There have been numerous informal observations about prompt drifts in Large Language Models (LLMs), with the most notable case being GPT-4 showing signs of laziness, especially for coding tasks by the end of the previous year. Discussions on Twitter also hint at a decline in Claude Sonnet’s effectiveness over the past few days. Given the closed-source nature of these models, it's impossible to know what happens behind the scenes, and most often, these drifts go unnoticed until they get flagged by the community. Today, a structured approach to tracking these shifts in the model’s performance is lacking. So we decided to undertake this as a community initiative to monitor prompt drift and identify any regressions systematically. You can check it out and learn more about our methodology here: https://lnkd.in/g93s7Gfv While building this out was fun, performance monitoring presents many challenges—notably how to do this efficiently (from a cost perspective) and yet get good, stable results. Looking ahead, we plan to enlarge our benchmarking dataset as well as include additional models (ex: Claude 3). Happy to chat if you want to enable this for your organization and give trust to your customers that while LLMs are unpredictable, your application is not :) Shikha Mohanty

Great to be a partner of Hippocratic AI since the very early days, and excited to see them reach new heights 👏

We are excited to announce the LlamaIndex <> UpTrain (YC W23) integration! It’s been months in making, but we wanted to deliver something of real value to our community. Evaluations are not just about computing a final score for your RAG pipeline but getting 𝐚𝐜𝐭𝐢𝐨𝐧𝐚𝐛𝐥𝐞 𝐢𝐧𝐬𝐢𝐠𝐡𝐭𝐬 𝐨𝐧 𝐰𝐡𝐞𝐫𝐞 𝐭𝐡𝐢𝐧𝐠𝐬 𝐚𝐫𝐞 𝐠𝐨𝐢𝐧𝐠 𝐰𝐫𝐨𝐧𝐠 𝐚𝐧𝐝 𝐡𝐨𝐰 𝐭𝐨 𝐢𝐦𝐩𝐫𝐨𝐯𝐞 𝐭𝐡𝐚𝐭 𝐬𝐜𝐨𝐫𝐞. With this integration, you can evaluate all individual components of your RAG pipeline, such as retrieval, reranking, sub-query, etc. and get deep insights into where your LlamaIndex pipelines need improvements, all with a single line of code. At UpTrain, we are building the gold standard of LLM evaluations with high-quality scores that learn your preferences. • Evaluate different aspects of your application with 20+ preconfigured checks • A high degree of customisation allows you to modify eval prompts, choose evaluator LLM or create your own checks. • Experiment with prompts, LLMs, embedding models, RAG modules, etc. • Do root cause analysis to find failure modes and hidden patterns. and finally, • Interactive dashboards to visualise results and do side-by-side comparisons [More coming soon] Check out the blog: https://lnkd.in/gnwkG2nw Check out UpTrain: https://lnkd.in/gcbM2kY3 It was great fun collaborating with the LlamaIndex team - Ravi Theja Desetty Laurie Voss Jerry Liu! Shikha Mohanty Dhruv Chawla Shreyansh Jain @Ashish Anand Raghav