Arize AI

🎆 Phoenix 2.0 launch week 🎆 is wrapping up with a vengeance — five days of big releases, all with new cookbooks and demos. ICYMI: ⚙️Day 1, Hosted Phoenix (aka LlamaTrace): the hosted version of Phoenix offers the ability to persist application telemetry data generated during AI development in order to better experiment, iterate, and collaborate in development or production. 📈Day 2, Datasets: a new core feature in Phoenix that lives alongside your projects. They can be imported, exported, created, curated, manipulated, and viewed within the platform, and make fine-tuning and experimentation much easier. 🧪Day 3, Experiments: provides a full iteration workflow in Phoenix; start from a dataset of example cases, define your evaluators, then iterate through as many prompt variations, model combinations, or chain structures as you'd like — Experiments collects all the results side by side in the Phoenix UI so that you can quickly choose the optimal approach or identify problem areas. 🔧Day 4, Function Calling Evals: a new built-in evaluator to Phoenix that measures the performance of function or tool calling within LLMs. 🚧Day 5, Guardrails: integration with Guardrails AI enables you to capture spans and traces on their existing suite of Guards. View the invocation of these guards within Phoenix alongside your LLM calls and create Datasets based on guards triggering. John Gilhuly will be covering everything in this open source town hall and Q&A: https://lnkd.in/gsDgYy2M

We're thrilled to have Cyrus Nouroozi (DSPy key contributor) join us tomorrow to discuss a recent DSPy paper that introduces LM Assertions, a programming construct for expressing computational constraints that LMs should satisfy. Register 👉 https://lnkd.in/dmEY6C8F More about the paper... Chaining language model (LM) calls as composable modules is fueling a new way of programming, but ensuring LMs adhere to important constraints requires heuristic “prompt engineering.” The researchers integrated their constructs into the recent DSPy programming model for LMs and present new strategies that allow DSPy to compile programs with LM Assertions into more reliable and accurate systems. They also propose strategies to use assertions at inference time for automatic self-refinement with LMs. They reported on four diverse case studies for text generation and found that LM Assertions improve not only compliance with imposed rules but also downstream task performance, passing constraints up to 164% more often and generating up to 37% more higher-quality responses. Dat Daryl Ngo and Sally-Ann DeLucia will lead the discussion. Read the paper here: https://lnkd.in/gaP-mH3a

For OSS enthusiasts: Join us this Thursday as John Gilhuly and Jason Lopatecki cover all the new Phoenix features and talk about why open source is so critical to continued AI development. 🚀 Sign up to join us live: https://lnkd.in/gsDgYy2M We've been shipping a major new Phoenix feature each day this week. Here's what's out so far: ✨ Hosted Phoenix aka LlamaTrace ✨ Datasets ✨ Experiments Keep up with the releases here: https://lnkd.in/gG3rKKDC

Hot off the heels of Observe, we're doubling down on our commitment to open source. Following the announcement of our new Hosted Phoenix Option, we’re kicking off Phoenix 2.0 Launch Week! 🚀🐦🔥 5 days, 5 major releases. All open-source, free, and ready to use. Follow along at: https://lnkd.in/gG3rKKDC

Huge thanks to everyone who came out for Observe — we were thrilled to host you today. 🫶 We were incredibly lucky to hear from cutting-edge teams working on generative and researchers pushing the boundaries of what’s possible in improving AI-powered systems. Hat tip to all of our speakers who gave so many thoughtful presentations on all things LLM observability and evaluation. Last but not least, huge thanks to our sponsors for making this day possible: Microsoft Cerebral Valley Battery Ventures Swift Ventures Vectara Modelbit MindsDB PromptLayer Innodata Inc. + Pinecone And also!!! As many of you noted to us, the venue was pretty great. (Cc: SHACK15)

Really nice to meet our customers, announce new features

We just kicked off Observe with a keynote by Aparna Dhinakaran and Jason Lopatecki who introduced and demoed Copilot—the first AI assistant for AI—and so much more. 🚀 Amazing lineup today, we’re just getting started!!

Another Arize:Observe keynote drop 🎁: LlamaTrace, a hosted version of Arize OSS Phoenix. The new LLM tracing and observability platform works natively with the LlamaIndex and Arize ecosystem. With a foundation based on Phoenix OSS, the hosted version of Phoenix offers the ability to persist application telemetry data generated during AI development in order to better experiment, iterate, and collaborate in development or production. The solution features a fully hosted, online, persistent deployment option for teams that do not want to self host. AI engineers can instantly log traces, persist datasets, run experiments, run evaluations – and share those insights with colleagues. The new offering is available today, and can be accessed through either a LlamaIndex or Arize account.

✨Introducing Arize Copilot, the first AI Assistant for AI ✨ Premiering onstage today at Arize:Observe, Copilot offers high-level model insights, data quality analysis, and LLM-specific functionalities like evaluation summarization and retrieval process troubleshooting. It also identifies issues and patterns in your evaluation results (suggesting pre-built or custom evaluations), optimizes your prompts based on specific concerns or evaluation data, and leverages AI Search to curate data with natural language queries combined with traditional filters. Read more in this blog by Sally-Ann DeLucia https://lnkd.in/gFuHDDcp

🚀 Tomorrow's the big day! We're kicking off Arize AI Observe - our conference in SF on all things related to #evals, going from development to production with AI, #agents and more. There are research, builders and innovators tracks with a ton of amazing speakers in the community Mistral AI, OpenAI, AI at Meta, Anthropic, Jerry Liu, Google, Microsoft and so, so many more that I'll just drop the link here: https://lnkd.in/dMgF2XKf Arize product and eng have been hard at work on some pretty amazing new capabilities - I won't spoil it, but message me and I'll share some recaps afterwards! And if you're in the area - I have a few last minute tickets left, DM if you're interested