Anyscale

We’ve recently introduced the Anyscale Unified Log Viewer, which simplifies and streamlines the debugging and optimization of distributed Ray applications! Key features include: • Structured logging with actor, instance, and request attributes for effective searching across the cluster • UI and log persistence • Comprehensive search and filtering capabilities • Scalable to 100s of GB of logs and thousands of nodes Check out the video: https://lnkd.in/gwr5hsTf  🛠️ To get started, log in to Anyscale, run your Ray application, and navigate to the new log viewing section in your dashboard. Start debugging and optimizing with ease! 🌟 Read more in our announcement: https://lnkd.in/gpbEvr2h

Python's default logging library has a lot of advantages, but logs are often very unstructured, which adds complexity for downstream parsing, querying, managing, and exporting. We're excited to announce Structured Logging is now available on Anyscale. Logs are now in machine-readable JSON format, simplifying filtering, searching, and analyzing. No more messy text logs and writing regex to debug. Read more in our announcement: https://lnkd.in/gUt9e7sV Try on Anyscale: https://bit.ly/3Y2BnHJ

Excited to partner with PingCAP on the #TiDB Future App Hackathon!

🚀 Follow along this week as we highlight some of the features we shipped over the past quarter! Up first: Anyscale Replica Compaction. Anyscale’s Replica Compaction optimizes resource scheduling in Ray Serve, solving resource fragmentation and enhancing efficiency. Why it matters:  Resource fragmentation happens when scaling leads to uneven utilization across nodes. As traffic decreases, previously needed nodes become underutilized, leading to increased cost and poor efficiency. Ray Serve scales each model independently, leading to this issue. Solution: Replica Compaction automatically migrates replicas into fewer nodes, optimizing resource use and reducing costs with zero downtime. Results: On an Anyscale Service with multiple LLMs (Mistral, Mixtral, Llama3), efficiency increased by 10.7% (measured by Tokens / GPU Second).  We found instance seconds declined by 3.7% despite an 11.2% traffic increase. Learn more in our blog: https://lnkd.in/e5WjVSFH Get started today: https://bit.ly/3Y6qHrx

It was great to see everyone at the Ray in Financial Services Meetup in New York 🏙 A big thank you to all the community members who made it possible. 👏 We discussed Ray and it's use cases, got a hands on look at running parallel experiments with cpus and gpus, and shared more on the upcoming roadmap. Looking forward to our next meetup - training day in New York this July 31st! Register here:https://lnkd.in/ePbX2KYs

Mira Murati, CTO of OpenAI, will be a keynote speaker for Ray Summit 2024! 🎉 Known for her work on ChatGPT, DALL-E, and GPT-4, Mira advocates for ethical AI aligned with human values. Join us and learn from her groundbreaking work and vision for the future of AI 🚀 Register here: https://t.co/VTdE4cbXB9

We’ve recently contributed FP8 support to vLLM in collaboration with Neural Magic -- with this feature, you can see up to a 1.8x reduction in inter-token latency, with >99% accuracy preservation! A common concern with FP8 is whether users will experience accuracy degradation. To address this, Neural Magic has produced many checkpoints for key models with >99% accuracy preservation across a wide range of benchmarks (https://lnkd.in/gTimN5dZ), including: - Llama3-70b - Mixtral 8x7b - Llama3-8b You can easily try this out on vLLM, and read more about the feature here -- https://lnkd.in/gzKJqerB

🌟 It's not too late! 🌟 Join us tonight, July 9th, from 5:30 - 8:30pm in NYC for the Ray in Financial Services Meetup. Don’t miss out on discovering how Ray powers AI in finance and networking with industry peers Sign up here: https://lu.ma/vrlylh28 We hope to see you there!

🚨Last chance!🚨 Tomorrow, July 9th, 5:30 - 8:30pm in NYC, join the Ray in Financial Services Meetup. Discover how Ray powers AI in finance, explore AI-driven use cases, get insights into the Ray 3.0 roadmap, and network with industry peers🤝 Minimal spots left, sign up here: https://lu.ma/vrlylh28