Ajay Saini

There are three fundamental problems researchers face when building foundational LLM in any language. They are... 👇 1. GPU cost: Training an LLM from scratch is costly. Very costly. 2. Absent Revenue Model: Unless an Indic-language LLMs performance beats top existing models, it is tough to monetize -- and therefore, impossible to recover GPU costs. 3. Data: LLM training requires clean, accurate, high-quality data. This is currently absent for most Indian languages. It is possible to create it, but requires substantial work. Even if one solves the first two problems, the third one is a major challenge. Due to this, even fine-tuned versions of existing LLMs don't perform as well as the original model. Unless one is confident about cracking all three problems - it is not worth getting into. There are numerous (100s or 1000s) of other problem statements in the AI domain that one can take on instead.

17

1 Comment

About 2 months ago Li Yin didn't hold back in critiquing LLM application frameworks. Fast forward to 7/3/24, and her team has just released the alpha version of LiteRAG, with beta on the horizon(7/9/24). First Impressions? It's a nice balance between how I use instructor and DSPy. Instead of Pydantic, it uses the dataclasses, which is a standard library construct. The library also introduces DataClass as a wrapper to simplify generating schemas, adding examples, and loading JSON AND YAML. If you're familiar with Pytorch, or DSPy, then you'll feel at home composing your Components, which is a core abstraction. My favorite component as of now, Sequential, allows you to chain together your components input -> output. Definitely looking forward to seeing how this library grows! But, you don't need me to give you your opinion, Go check it out and make your own. LightRAG -> https://lnkd.in/g4cWDP88 If you want to see how I applied it to a current project, check out the video! Happy MonYay yall!

81

6 Comments

Large Language Models (LLMs) as super lossy compression. I have been thinking a lot about demystifying what LLMs do because lots of y'all's are very mystified. In the linked article, I try to reframe them for you, hoping it will help you understand what they good for and what they are not good for. Side note: I am still looking for my next adventure. If communicating complicated concepts so that regular people can grasp them is something you need, you should message me. I write code and stuff too and am usually fun to be around. #WrittenByMe

6

3 Comments

Hallucinations in LLMs are inevitable, no seriously! Josh Fritzsche and I have worked hard at Voze to find a way to tune, prompt and even plead 🙏 with LLMs to prevent them from the occasional 🍄 hallucination. It seemed like the better we have done the harder the edge cases have been to solve and eventually we realized that they tend to frequently happen when LLMs predict or make anticipations on "nothing". I remember at one point nearly going insane with frustration because when LLMs had a hallucination, they were EPIC! 😱 Then we pushed through it and benefited for the effort! How did we solve this common problem? We didn't (Not completely), we got better at identifying the issues as they happened. Once we identified in the data we were able to start looking for patterns. The same catalysts that cause it to happen once if left unchanged caused it over and over again. Our solution is astoundingly simple. Once you know that it's likely to happen..... let it! Accept it will 🍄, but also be preemptive in not focusing on it and use alternative data sources, generators or providers. Change the conditions that cause it but continue to monitor it as it happens. As Dan Caffee is fond of saying to me on a regular bases. "Dusty we need the system to be adaptive and provide mechanisms to learn from it's past." Well he is right! All we needed was data and the people that contribute to it's quality. Huge props 👏 to strong support from Kathy and Janelles audit teams. We were armed with a knowledge and a desire to learn from past mistakes. Armed with confidence that we could correct these problems we could focus on the data to prevent their impact going forward. When doing millions of LLM (Generative AI) calls regularly Josh and I have learned that hallucinations will happen. We also know exactly how we will handle and prevent it going forward! Failure is not doing anything and accepting defeat.

4

2 Comments

I wrote a blog post on a thorny but underappreciated problem in ML-based marketing personalization: event attribution. It's how you go from the actions you take and customer behavior you observe to creating training experiences for your model. Good attribution makes a huge difference in driving uplift, there are a lot of degrees of freedom in configuring it, and at the same time, it's rarely discussed in ML/bandits/RL literature.

38

4 Comments

In LLMs, long input sequences presents a significant challenge in detecting hallucinations using smaller, cheaper models. One of the key innovations (that blows past LLM-as-judge techniques) introduces a novel windowing mechanism specifically designed to address this issue. Consider this -- conventionally, the standard procedure for models with limited context lengths includes Segmentation, Localized Prediction and Aggregation. This traditional approach proves suboptimal for hallucination detection. e.g. a scenario where the language model generates a statement, "X", which is substantiated by context, "Y". If segments are created such that "X" and "Y" reside in separate chunks, the model remains oblivious to the mutual reinforcement between "X" and "Y", thereby impairing accurate hallucination detection. The approach we take circumvents the limitation by deploying a more refined windowing strategy (RAG context and the model-generated responses getting divided into small overlapping windows). This allows a smaller, auxiliary model to evaluate every possible pair of possible context and response windows. This shift in the windowing methodology offers a more robust framework for hallucination detection, advancing the frontier of reliability in RAG deployments. If you're interested in learning more, read our research here: https://lnkd.in/gFAqr7wB #llmhallucinations #llms #generativeai #llmevaluation #llmevals #llmbenchmarking

32

2 Comments

Stole a few points from Prof Christopher Manning, who talked about LLMs and language modeling in general in the latest TWIML AI Podcast: 📌 Humans acquire language skills in a way very different from LLMs. We need millions of words, compared to billions or even trillions of tokens for LLMs. LLM researchers may want to investigate and learn from how humans acquire language skills. 📌 LLMs cannot reason. However, there are other deep learning models, such as AlphaGo, that can. LLM researchers may want to look into how to integrate that type of reasoning/searching/planning capability into LLMs. 📌 LLMs' world models should enable search and discovery. Although Prof. Manning didn’t call this out explicitly, my understanding is more similar to a knowledge graph type of structure. 📌 Next-gen LLM idea: a soft form of locality and hierarchy. Transformers attend every token to every other token, which is very inefficient, while human language can be modeled by n-grams most of the time. #artificialintelligence #machinelearning #deeplearning https://lnkd.in/euzwMQ6p

84

18 Comments

I have to write outreach emails/messages regularly to potential clients. I wanted to automate the writing process. I could use LLMs to write it for me, but I hate it when it sounds more like a robot and it's even worse when it hallucinates 😞 So I created a RAG-based LLM application (running locally) using LlamaIndex that takes in all my previous emails, proposals, and portfolio projects and generates content that sounds more like me 😎 . To top it off, it often identifies and highlights connections between our previous work and the client's industry (See the demo below for a closer look.) Note: This is just a demo, I haven't provided it with much data yet. I think it can do even better with more personalized content. #LLM #RAGtechnique #AI #outreach #codeclever

26

7 Comments

Some quick Q1 reflections on Knowledge Graphs and RAG, with high level principles to think about when developing Graph RAG systems: 1. The market for RAG is huge because white collar work is largely an information retrieval process 2. RAG is a process, not a system, and should be automated in piecemeal. As each question can almost be regarded as an information retrieval process. Group questions and tackle RAG accordingly. 3. Small Graphs and Minimum Viable Graphs as the future of Graph RAG, not large Graphs.

130

9 Comments

I just published Vol. 58 of "Top Information Retrieval Papers of the Week" on Substack. My Substack newsletter features the 7-10 most notable research papers on information retrieval (including recommender systems, search & ranking, etc.) from each week, with a brief summary, and links to the paper/codebase. This week’s newsletter highlights the following research work: 📚 Learnable Late-Interaction Models for Efficient and Accurate Information Retrieval, from Google 📚 Gaming the AI: How Crafted Content Can Hijack LLM-Powered Search Results, from ETH Zurich 📚 Dynamic Text Segmentation for Enhanced Retrieval in RAG Systems, from Duarte et al. 📚 Improving Collaborative Filtering with Simplified Sampled Softmax Cross-Entropy, from Visa 📚 Automating Quality Assessment in Retrieval-Augmented Generation with GPT-4, from Visa 📚 Efficient Listwise Passage Reranking via Embedding Compression, from Liu et al. 📚 Empowering Smaller LLMs in Passage Re-Ranking through Pre-Filtering, from Polytechnique Montréal 📚 Efficient Online Hyper-Parameter Optimization for RAG Systems, from Microsoft 📚 Decomposed and Distilled Large Language Models for Efficient and Accurate Semantic Search, from Ant Group 📚 Accelerating LLM Reranking with Single-Token Logits, from Reddy et al. #InformationRetrieval #ResearchPapers #CuratedContent #Newsletter  #substack

73

1 Comment

I'm so excited to share that TLM v1.0 has been officially launched! 🎉 TLM is where Data Curation 🤝 LLMs. Thanks to the wonderful efforts of the Cleanlab team, we've created an awesome product that can: - Improve LLM response reliability - Quantify the trustworthiness of responses from any LLM LLMs will always have some hallucinations, but by providing a trustworthiness score with every output, Cleanlab TLM lets you identify when the LLM is hallucinating to enable reliable deployment of LLM-based applications. Read more about it in our research blog: https://lnkd.in/dTZUVTmG Check it out for yourself here: https://tlm.cleanlab.ai/

31

1 Comment

Looks like fine tuning out performs RAG and Large context LLMs for data accuracy with longer contexts Paper: https://lnkd.in/gj9-3jsK Summary of "BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack" - Introduction to BABILong Benchmark: - Designed to test LLMs' reasoning abilities over extremely long documents. - Comprises 20 diverse reasoning tasks, such as fact chaining, induction, deduction, counting, and handling lists/sets. - Uses natural text from the PG19 corpus, extendable to document lengths up to 1 million tokens. - Current LLM Performance: - Popular LLMs utilize only 10-20% of the context effectively. - Performance declines with increased reasoning complexity. - Retrieval-Augmented Generation (RAG) methods achieve 60% accuracy on single-fact QA, independent of context length. - Top Performing Models: - Recurrent Memory Transformers (RMT) show the best performance, processing up to 11 million tokens. - Small models like Mamba (130M) and fine-tuned RMT (137M) achieve high accuracy on long-context tasks. - Evaluation Findings: - Current benchmarks (e.g., Longbench, L-Eval) are insufficient for LLMs with capabilities beyond 40,000 tokens. - BABILong tests models' efficiency in distinguishing relevant facts from irrelevant text over long contexts. - Popular LLMs fail to maintain high performance as context length and complexity increase. - Fine-Tuning Insights: - Fine-tuning improves model performance significantly. - Mamba and RMT demonstrate successful QA with long contexts, outperforming RAG models. - RMT's memory mechanism allows efficient processing of long sequences, with marginal quality degradation up to 11 million tokens. - Benchmark Significance: - BABILong provides a rigorous evaluation framework for LLMs' long-context reasoning abilities. - Highlights the limitations of current LLMs and the need for improved context processing mechanisms. - Key Contributions: - Introduction of BABILong, a scalable generative multi-task benchmark. - Evaluation of over 20 recent LLMs on various context lengths and tasks. - Demonstration of the limitations of current models in long-context utilization. - Setting a new record for sequence size processed by a single model with RMT.

4

5 Comments

I just published Vol. 52 of "Top Information Retrieval Papers of the Week" on Substack. My Substack newsletter features the 7-10 most notable research papers on information retrieval (including recommender systems, search & ranking, etc.) from each week, with a brief summary, and links to the paper/codebase. This week’s newsletter highlights the following research work: 📚 A Comprehensive Survey of Retrieval-Augmented Large Language Models: Architectures, Training, and Applications, from PolyU 📚 An Analysis Framework for Benchmarking Retrieval-Augmented Generation Systems, from Tu et al. 📚 Distilling Cross-Encoders from Large Language Models for Efficient Passage Re-ranking, from Schlatt et al. 📚 A Comprehensive Benchmark for Assessing Retrieval-Augmented Language Models in Biomedicine, from UMTC 📚 Improving Factual Accuracy of Large Language Models for Healthcare using Targeted Knowledge Graph Retrieval, from Huawei 📚 Identifying Shopping Product Questions in Voice Assistants via Graph Attention Networks, from Amazon 📚 A Learnable Tokenizer for Generative Recommendations, from NUS 📚 An Enhanced Retrieval-Augmented Generation Framework for Accurate, Efficient, and Personalized Language Models, from UTS 📚 Constructing Synthetic Test Collections Using Large Language Models for IR Evaluation, from Rahmani et al. 📚 Enhancing Sequential Recommendation with Soft Contrastive Learning, from Renmin University #InformationRetrieval #ResearchPapers #CuratedContent #Newsletter  #substack

70

2 Comments

There is an obvious need for standardized metrics across the decentralized computing marketplace landscape, which is why Sami Kassab wrote a killer blog this week proposing the use of Floating Point Operations per Second (FLOPS) as that standardized metric. In yesterday’s Big Brain Breakdown Outpost Strategies, we expand on Sami’s original post, and add some additional context of our own. Here are the key takeaways from yesterday's breakdown: > The decentralized computing marketplace sector lacks a standardized metric that measures a marketplaces total computing capacity, making it difficult for investors to accurately assess the value of these projects. > Sami proposes using FLOPS (Floating Point Operations per Second) as this standardized metric. FLOPS is widely recognized in high-performance computing and can provide a clear, hardware-agnostic measure of computational capacity. > FLOPS can accurately represent the computational capacity of diverse hardware, including GPUs, CPUs, and specialized accelerators like TPUs and FPGAs. This standard helps users assess the feasibility and cost of tasks like AI and ML workloads in real-time. > 6079’s Proof of Inference Protocol (PoIP) is one solution that can be used to verify and record FLOPS onchain, ensuring transparent and accurate reporting of computational resources through cryptographic proofs and game-theoretic mechanisms. > FLOPS should be complemented with additional metrics such as benchmark scores for AI/ML workloads, memory bandwidth, and energy efficiency to provide a complete picture of computational resources. This combination ensures a robust and sustainable growth of decentralized compute marketplaces. > The adoption of standardized metrics like FLOPS benefits users by providing clear insights into computational capabilities, aids investors in making informed decisions on the value of a project’s marketplace, and supports legitimate projects by highlighting their true value, thereby weeding out low-value, vaporware projects. Of course, we go into much more depth in the full breakdown. If you thought that was interesting, go ahead and give the full piece a look and do not forget to read Sami's original piece. He is an excellent writer! https://t.co/BtVuG82fcB https://t.co/ryQf2e8Zuu

10

Meta just dropped Llama 3 (70b) last week. Here's what you need to know. Trained on 15 trillion tokens of data. 😯 Llama 3 is leading the space in capabilities for math and reasoning. Scoring an 85 on the MMLU benchmark. (Massive Multitask Language Understanding) Compared to Claude 3's impressive 86.9 score. However, with 25 trillion less training tokens. Llama 3 definitely has the potential to exceed Claude 3. The most exciting development comes in the form of Meta's approach to AI. 𝗢𝗽𝗲𝗻 𝗦𝗼𝘂𝗿𝗰𝗲. Llama 3's being released to developers, Along with a "Getting Started" guide, as well as new trust/safety features. - Llama Guard 2 A beefier version of their Llama guard content classification system. - Code Shield A screener for potentially malicious, or improper code outputs. - CyberSec Eval 2 A system for protection against prompt injections and code interpreter abuse. 💡 Benefits of Open Sourcing Llama 3 Llama 3 is comparable to GPT-4 in terms of capabilities. Tech hardware companies like Nvidia and AMD are supporting the model, As well as Cloud services like AWS, and Hugging Face. By it becoming publicly available to whomever wants to design with it. ✅We'll quickly have many GPT-4 models entering the market. The competition is on! - I'm really excited to announce Hank and I's collaboration with Christoph Rosenboom Keep an eye out next Monday! #llama3 #AI #LargeLanguageModel

17

19 Comments

Excited to announce another major upgrade to WhyHow.AI’s Knowledge Graph SDK - tying vector chunks to graph nodes automatically, for a more deterministic and richer context window. Check out how we do it, why we did it, and an example benchmark of the increased completeness of the answer. Tired of just returning single-word triples from your knowledge graph? WhyHow.AI’s latest upgrade with vector chunk linking now lets you use a graph structure to determine which raw vector chunks to return to the context window, combining the best of knowledge graphs and vector search. “While the triples in a Knowledge Graph are useful in providing specific information that semantic similarity was unable to retrieve, we wanted to also allow leeway in the information represented and retrieved from the graph, to include the surrounding words and retrieving the relevant raw vector chunk tied to that graph node as well. By tying vector chunks to a knowledge graph, we get the advantages that lie in both vector and graph search.” - WhyHow.AI Design Partner WhyHow.AI builds workflow tools for data orchestration, and graph creation, and we work on top of any data extraction model you want to bring. In this case, we work on top of OpenAI, Neo4j, and Pinecone, and will be supporting the most popular data extraction models, LLMs, graph and vector databases. https://lnkd.in/eEJdUNPi

68

3 Comments

A recent breakthrough titled "Matrix Multiplication-Free LLMs" demonstrates a huge advancement in the area of Large Language Models (LLMs) by reducing computational costs. The authors have eliminated MatMul operations from LLMs, claiming to 10 times reduction in memory usage and a 25.6% increase in training speed, all while maintaining strong performance at billion-parameter scales. Paper link: https://lnkd.in/ggph8qXc #AI #machinelearning #deeplearning #LLMs

16

We will be seeing the release of #llama3 in the next month. Open Source LLMs paved a huge benchmark over the Natural Language Text Generation and Natural Language Understanding. I would say the stepping stone to the Chain of thoughts, Code Generation, and Code Testing was given by the Microsoft #llama2 model which comes in various flavors. Happy to see the release and awaiting to use the power of the llama3 model. Read the entire source from here: https://lnkd.in/gJS8ykE7 #llama3 #opensourcemodels #foundationalmodels #chainofthoughts

28