Yanqi Zhou

Come see this talk, which I view as the culmination of several years of discussions and co-teaching with George Fairbanks. It's a bit of an overview, a bit of an explainer, a bit of a how-to, but mostly a sneaky way to make some important vocabulary from design become load-bearing in how you think about software testing.

86

1 Comment

Its sometimes hard to believe that we live in an era where driverless vehicles aren't just a vision anymore but a reality. Waymo driverless cabs for example, are transforming the San Francisco streets forever! It was a wonderful experience working on an application of autonomous vehicles as part of my final project in the course CS267: Applications of Parallel Computing at the University of California, Berkeley. Our team, comprising Terence N., Zeid Solh, and myself, focused on parallelizing stereo vision for driverless applications. Stereo vision is crucial for self-driving cars as it builds a 3D understanding of the environment by using two cameras. However, real-time processing of these images is computationally expensive, posing a significant challenge for autonomous vehicles. We tackled this by exploring various parallel computing techniques to improve the speed of calculating the Disparity Map algorithm. We used popular techniques such as #SIMD (Single Instruction, Multiple Data), #OpenMP (multi-threading), and CUDA (NVIDIA GPU acceleration), and benchmarked the performance. By distributing the computational load across multiple cores and processors, we achieved substantial performance improvements, enabling real-time processing of high-resolution images and videos. The results were promising! Utilizing CUDA, we saw significant speedups, demonstrating the potential for efficient, real-time depth perception in autonomous systems. This project signified the importance of parallel computing in advancing autonomous driving technologies. I'm incredibly proud of our team's hard work and the insights we gained through this project. Check out our GitHub repo: https://lnkd.in/gYHrEU9z for more details on our implementation! #UCberkeley #CS267 #ParallelComputing #StereoVision #AutonomousDriving #DeepLearning #ComputerVision

50

3 Comments

A recent breakthrough titled "Matrix Multiplication-Free LLMs" demonstrates a huge advancement in the area of Large Language Models (LLMs) by reducing computational costs. The authors have eliminated MatMul operations from LLMs, claiming to 10 times reduction in memory usage and a 25.6% increase in training speed, all while maintaining strong performance at billion-parameter scales. Paper link: https://lnkd.in/ggph8qXc #AI #machinelearning #deeplearning #LLMs

16

Check out our latest AI+EDU paper at OOPSLA 2024! Syntactic bugs and semantic bugs are intrinsically different. A tool that fixes syntactic bugs does not check test case results. A tool that fixes semantic bugs can be started only if the target program has no syntactic bug. But what if a program contains both types of errors, which is happening way too often in intro-level students' programs?  😔 Powered by LLM, PyDex is the first tool to repair both syntactic and semantic bugs in real-world Python programs, with an accuracy of 96.5%. Paper link: https://lnkd.in/euW2SWAF

18

Sparse Autoencoders (SAEs) are one of the most promising and popular methods for LLM interpretability and explainability. I wrote a simple, intuitive introduction to SAEs, complete with diagrams and reference PyTorch code. Among other things, I cover why we use SAEs, how they are used for interventions, and challenges with using SAEs. https://lnkd.in/gzrvi6Kg

13

From the experience of professionally working in compiler engineering and performance engineering role for the last six months, I can say with quite a bit of confidence that ... NOBODY, absolutely NOBODY is taking over NVIDIA's market share for the next decade. It's kind of unbelievable how bad the closest competitor's software stack is. It does not really matter how much compute you offer in your hardware if there is no way to harness that power through your software stack. Even if it was barely usable, the complete lack of documentation makes it impossible to even tweak the most basic of things like which compiler to point at or which runtime driver to use to offload to the device from the host side. So, I guess this is why the closest competitor to NVIDIA sponsors an F1 team instead of fixing their codebase. After all, they have something in common: Neither of them have good drivers.

18

Leverage StarCoder 2, a powerful LLM for code generation, completion, summarization, and documentation. Try the now available NVIDIA-hosted StarCoder 2 API, packaged as NIM inference microservice for free from the NVIDIA API catalog. https://dy.si/Zf8AD

8

Excited to share the work that Chris Glaze Changho Shin Frederic Sala and I did on long context LLM evaluations. While there’s plenty of problems with LLM evals, a recurring one we’ve seen at Snorkel, working with enterprise clients, is having to rely on far too simple/toy benchmarks that do not relate to business problems. It is why we developed SWiM, a framework that lets users test long context model capabilities directly on their own data and tasks. Read the paper here: https://lnkd.in/g3uSuKcW

27

In-Context Learning at Extreme Scales: My Thoughts As large language models (LLMs) continue to expand their context capacities, the domain of in-context learning (ICL) at extreme scales has piqued my interest. A recent paper by researchers at Carnegie Mellon University and Tel Aviv University provides a detailed exploration into this area, unveiling both impressive capabilities and notable challenges. 🔍 Key Insights: Performance Improvement: The study demonstrates that ICL can improve significantly by integrating hundreds to thousands of examples into the context window—far beyond traditional few-shot settings. This is particularly impactful in complex tasks like 77-way banking intent classification, where long-context ICL matches or surpasses traditional finetuning. Input Sensitivity: Long-context ICL shows reduced sensitivity to input shuffling compared to shorter contexts. However, organizing examples by label dramatically reduces performance, emphasizing the need for diverse context configurations. Source of Gains: Performance gains primarily arise from accessing more relevant examples, rather than aggregating knowledge across many examples during encoding. 📉 Challenges & Drawbacks: Sensitivity to Example Order: Despite reduced sensitivity, the performance still varies with input order, indicating potential instability in practical scenarios. Performance Saturation: The models often do not fully leverage their extended context capacities, showing performance saturation before reaching the maximum context length. Computational Cost: The increased computational cost associated with using large context windows could limit practical deployment, particularly in resource-constrained environments. Efficiency Issues: There is a noted inefficiency in how models utilize the encoded information within large datasets, suggesting that current architectures may need refinement to better use available data. 🚀 Future Directions: The paper suggests several exciting avenues for future research, including optimizing the trade-offs between computational costs and performance, exploring model adaptation strategies, and refining ICL approaches for specific applications or datasets. As we push the boundaries of what LLMs can achieve, effectively leveraging their full context could unlock new capabilities. This area is ripe for further exploration and could lead to major advancements in how we deploy AI. For those at the cutting edge of language model development, I highly recommend this paper for its thorough analysis and forward-looking perspective. Check out the full work at https://lnkd.in/d7AmGh4y. What are your thoughts on the future of in-context learning at such scales and the challenges discussed? #AI #MachineLearning #DeepLearning #LanguageModels #InContextLearning

17

Just read Farquhar et al's Nature paper on detecting hallucinations in large language models using semantic entropy. Interesting result on quantifying uncertainty in LLMs. When you ask an LLM a question, its answers may look correct, but be made up - a confabulation. For example, the LLM may answer incorrectly that the STARD10 protein is a "negative regulator of the mTOR pathway" instead of its true role as a lipid transfer protein. When an answer is confabulated, multiple subsequent generations will have different meanings. For example, a second answer of the LLM may incorrectly say that STARD10 is a "negative regulator of the meiotic recombination process". The variability in meaning can be quantified using semantic entropy, which measures variability over multiple generated answers. Higher entropy suggests greater variability and potential confabulation. The LLM can then refuse questions with high semantic entropy. To calculate semantic entropy, you generate 5-10 answers. You then cluster the answers based on semantic meaning using bi-directional entailment. Two answers are part of the same cluster if they entail - or imply - each other (e.g. "Joe buys a red apple" implies "A red apple was bought by Joe" and vice-versa). The entailment is calculated using another, possibly less powerful LLM, so it's an interesting applications of using language models to correct other language models. After clustering, the probabilities of each cluster are calculated from the output probabilities of the LLM, and then the resulting entropy. Thresholding the entropy can classify a question as likely to produce confabulations with an AUROC of around 0.75. Link to paper below.

63

2 Comments

Lightweight large language model (LLM) might be a reality soon Recently I came across the paper “Scalable MatMul-free Language Modeling” where Rui-Jie Zhu et al propose a scalable matrix-multiplication-free model capable of similar performance when compared to state-of-the-art transformers while reducing computational resources. This is achieved, among other things, by modifying the self-attention to use element-wise Hadamard products instead of matrix multiplication. If those finding are confirmed the possibility of fasters, energy efficient models with less GPU requirement become real to the point that these models could reside in the end-user machine. Scalable MatMul-free Language Modeling” https://lnkd.in/gtu55sFF

9

I'm thrilled to share a fascinating advancement in the integration of graph-based approaches with large language models (LLMs) to enhance reasoning abilities. The recent paper [GNN-RAG: Graph Neural Network and Retrieval-Augmented Generation]( https://lnkd.in/gwJyn5NM ) introduces a novel GNN-RAG approach that shows promising results in improving Knowledge Graph Question Answering (KGQA). This approach not only outperforms GPT-4 but also leverages multi-hop information retrieval to facilitate faithful LLM reasoning on complex questions. The potential of GNN-RAG to enhance the depth and accuracy of AI reasoning is truly inspiring. #AI #MachineLearning #GraphNeuralNetworks #LLM #KGQA #Innovation #Research #GNNRAG

17

The Fall Semester 2023 at the University of California, Berkeley, in the UC Berkeley Electrical Engineering & Computer Sciences (EECS) was a challenging curriculum as it pushed me beyond my comfort zone, fostering not just learning but practical implementation. I would like to highlight one of my projects from my Embedded Systems (EE249A) class titled "Robot Environment Mapping using SLAM." Our team demonstrated a robot equipped with advanced sensors, such as #LiDAR and #stereovision cameras, employing the Simultaneous Localization and Mapping (SLAM) algorithm to navigate and map unknown environments in real time. We utilized Hector SLAM to create and refine occupancy grid maps through scan matching, enabling the robot to accurately determine its location relative to its surroundings. This project was supported by the Robotic Operating System (#ROS) on a Linux OS, showcasing the seamless interaction of autonomous systems with complex environments. The potential applications are vast, from autonomous vehicles to robotic assistance in hazardous areas. Check out our project in detail on GitHub: https://lnkd.in/gPsJiAAB A huge thank you to my teammates Anusri Sreenath, Haonan Chen, and Abdalla Eltayeb for their contribution. Feel free to connect if you're interested in learning more about our work! #Robotics #SLAM #AutonomousSystems #ComputerVision

92

2 Comments

Along the same vein, I also want to mention Prof. Rao's (Subbarao Kambhampati) argument that many folks may have misinterpreted LLMs' *approximate retrieval* as reasoning. See https://lnkd.in/e5qeYcdn Also, Prof. Rao's group did rigorous work to show that LLMs cannot plan. See https://lnkd.in/exR8Yb5h https://lnkd.in/eJSS7k-H https://lnkd.in/e62WA7e7 #artificialintelligence #machinelearning #deeplearning 

3

Today, we’re launching Substrate. We’re also announcing our $8M Series Seed led by Lightspeed. We believe the most robust, reliable, and productive uses of AI are when many models are used in coordination with each other, with regular programming mixed in. Most people building with AI already know this. But the available tools are suboptimal, unless you work for Google. For the rest of us, we’re left creating an unwieldy mess of chained API calls to multiple providers, with many roundtrips in between. We took a hard look at this state of affairs, and recognized how much it is stifling progress. Building large multi-step AI workloads requires sophisticated high-performance tooling and infrastructure. Nobody wants to deal with more tooling and infrastructure… but everyone would benefit from simple, intuitive interfaces that abstract away a powerful system underneath. No tooling, no infrastructure – just elegant abstractions.

25

1 Comment

The best part of investing at the earliest stages of company building is every once in a while you meet founders who can see the future. Ben Guo & Rob Cheung are two of those founders and after a year of building in stealth they're launching ꩜Substrate, the first inference API optimized for multi-step AI workloads. 👉 https://substrate.run With ꩜Substrate, you connect nodes from a curated library that includes optimized ML models, built-in file and vector storage, a code interpreter, and logical control flow. By simply connecting nodes, you describe a graph workflow, which Substrate then analyzes and runs as fast as possible. Entire graphs of many nodes will often run on a single machine, with microsecond communication between tasks. We are thrilled to participate in their 💵 $8M Seed led by Lightspeed (Guru Chahal/Nnamdi Iregbulem) alongside our friends at South Park Commons (Arian Agrawal), Craft Ventures, Red Swan Ventures (Eric Rubin/Sandy Cass), joined by amazing 🧠 founders and 👨‍🏭 operators like Guillermo Rauch, Immad Akhund, William Gaybrick, Chris Best, Gokul Rajaram, Shreyas Doshi, Jana Messerschmidt, Michael Manapat, and Matthew Hartman.

23

1 Comment