Tamirlan Seidakhmetov

LLMs can take gigabytes of memory to store, which limits what can be run on consumer hardware. But quantization can dramatically compress models, making a wider selection of models available to developers. You can often reduce model size by 4x or more while maintaining reasonable performance. In our new short course Quantization Fundamentals taught by Hugging Face's Younes Belkada and Marc Sun, you'll: - Learn how to quantize nearly any open source model - Use int8 and bfloat16 (Brain float 16) data types to load and run LLMs using PyTorch and the Hugging Face Transformers library - Dive into the technical details of linear quantization to map 32-bit floats to 8-bit integers As models get bigger and bigger, quantization becomes more important for making models practical and accessible. Please check out the course here: https://lnkd.in/g66yNW8W

10,608

143 Comments

Large language models (LLMs) are known to generate plausible sounding factually incorrect responses, especially when faced with unfamiliar concepts, queries. Recent research suggests that strategic fine-tuning and in-context learning, retrieval augmentation can significantly reduce these hallucinations. #llms #chatgpt4 #llmops #aiml #gemini #phi #mlops https://lnkd.in/duRrC3yg

4

Pro Tip for Working with LLMs: When you ask a Language Learning Model (LLM) to generate code and it doesn't work, request fixes. If you hear "I apologize for the oversight" three times, it's time to pause—further attempts might make the code messier. At this point, you have two options: 1. Manually correct the code. 2. Start over with the same or a different LLM, though there's no guarantee of better results. Happy coding!

5

This writeup is a goldmine of knowledge on building with LLMs ✨ My most important takeaway being: “Build eval framework. Can be super basic, but have one” https://applied-llms.org/

35

Two new papers this week with my colleagues at Visa: 1. Measuring machine learning model robustness in production using properties of harmonic functions (https://lnkd.in/gnhbseY6) 2. Measuring LLM hallucination based on same, unsupervised and automated! (https://lnkd.in/geMGX7HV)

33

6 Comments

I just published Vol. 55 of "Top Information Retrieval Papers of the Week" on Substack. My Substack newsletter features the 7-10 most notable research papers on information retrieval (including recommender systems, search & ranking, etc.) from each week, with a brief summary, and links to the paper/codebase. This week’s newsletter highlights the following research work: 📚 A Comprehensive Review of State-of-the-Art Universal Text Embeddings, from Cao et al. 📚 A Survey of Generative Retrieval Techniques, from NTU 📚 An Approach to Structurally Aligning Visual and Textual Embeddings, from Alibaba 📚 Leveraging Dynamic Granularity for Improved Retrieval-Augmented Generation, from Zhong et al. 📚 A Framework for Consistent and Reliable LLM-based Passage Ranking, from OSU 📚 Boosting Sample Efficiency in Recommender Systems through Large Language Model Integration, from Huawei 📚 A Task-Specific Loss Function for Improved Multi-Task Learning Performance in Online Advertising, from Huawei 📚 Robust Interaction-Based Semantic Relevance Calculation for Large-Scale E-commerce Search Engines, from Alibaba 📚 Comparing Auto-Encoding and Auto-Regression Models in Sequential Recommendation, from Wang et al. 📚 An Unsupervised Pipeline for Measuring Retrieval Complexity and Classifying Difficult Questions in QA, from Amazon Alexa AI #InformationRetrieval #ResearchPapers #CuratedContent #Newsletter  #substack

31

1 Comment

Sparse Autoencoders (SAEs) are one of the most promising and popular methods for LLM interpretability and explainability. I wrote a simple, intuitive introduction to SAEs, complete with diagrams and reference PyTorch code. Among other things, I cover why we use SAEs, how they are used for interventions, and challenges with using SAEs. https://lnkd.in/gzrvi6Kg

13

🚨 Meta & Georgia Tech: Fighting Climate Change Plus - Firms Build AI-LLMs In-House With Lamini ➡️ Georgia Tech and Meta teamed up to create a massive open-source database aimed at accelerating the development of direct air capture technologies. The database will train AI models to design these systems for specific environments, making them more efficient and faster to deploy. This tech pulls carbon dioxide straight out of the air, a crucial tool for tackling climate change. READ MORE -> https://lnkd.in/efjKsQwP ➡️ Lamini, an enterprise AI platform, has raised $25 million to support businesses to develop their own large language models (LLMs). These AI models can be trained on a company's specific data, unlike public models, allowing them to tackle highly specialized tasks and gain a deeper understanding of a company's domain. This will bring a new wave of innovation within businesses. READ MORE -> https://lnkd.in/e9buaUmh ➡️ Researchers at Ohio State University developed an AI model that analyzes vast datasets of patient information. This AI can predict stroke risk and recommend the most effective treatment options, mimicking the effectiveness of traditional clinical trials. This could help personalized medicine and streamline the often slow and expensive process of clinical research. READ MORE -> https://lnkd.in/eTCAF2uh ➡️ Japanese Prime Minister Fumio Kishida has announced an international framework for the regulation and use of generative AI, emphasizing the need to confront the dark side of AI, such as the risk of disinformation. The framework aims to promote cooperation among countries to ensure that people all over the world can benefit from the use of safe, secure, and trustworthy AI. READ MORE -> https://lnkd.in/ekyanTwU ~~~ ~~~ ~~~ 💡 What do you say? What AI news resonated with you today? Let me know in the comments ✍️ ~~~ ~~~ ~~~ Reshare ♻️ to help others discover more AI news. Thank you! 🚀 Follow Fabio Sparano for daily AI news. ~~~ ~~~ ~~~ #ai #ainews #artificialintelligence #aidevelopment

23

37 Comments

Function calling is a powerful way to extend the capabilities of LLMs and AI agents by letting them use external tools. Our new short course Function calling and Data Extraction with LLMs, created with Nexusflow and taught by Jiantao Jiao and Venkat Srinivasan, demonstrates how to prompt LLMs to form calls to external functions. You'll work with NexusRavenV2-13B, a 13B parameter open-source model that excels in function calling tasks while still being small enough to host locally. Learn to use function calling to extract structured data from unstructured text and access web APIs, and build an end-to-end application that processes customer service transcripts. You'll learn how to build LLM-powered applications that can analyze feedback, automate data entry, and enhance search. Please get started here: https://lnkd.in/g9FpiCuH

5,031

114 Comments

I'm thrilled to share a fascinating advancement in the integration of graph-based approaches with large language models (LLMs) to enhance reasoning abilities. The recent paper [GNN-RAG: Graph Neural Network and Retrieval-Augmented Generation]( https://lnkd.in/gwJyn5NM ) introduces a novel GNN-RAG approach that shows promising results in improving Knowledge Graph Question Answering (KGQA). This approach not only outperforms GPT-4 but also leverages multi-hop information retrieval to facilitate faithful LLM reasoning on complex questions. The potential of GNN-RAG to enhance the depth and accuracy of AI reasoning is truly inspiring. #AI #MachineLearning #GraphNeuralNetworks #LLM #KGQA #Innovation #Research #GNNRAG

17

These findings motivate our design of a novel framework to investigate hidden representations in LLMs with LLMs, which we call Patchscopes. The key idea behind this framework is to use LLMs to provide natural language explanations of their own internal hidden representations. Patchscopes unifies and extends a broad range of existing interpretability techniques, and it enables answering questions that were difficult or impossible before. For example, it offers insights into how an LLM's hidden representations capture nuances of meaning in the model's input, making it easier to fix certain types of reasoning errors. While we initially focus the application of Patchscopes to the natural language domain and the autoregressive Transformer model family, its potential applications are broader. For example, we are excited about its applications to detection and correction of model hallucinations, the exploration of multimodal (image and text) representations, and the investigation of how models build their predictions in more complex scenarios. Google DeepMind

8

I just published Vol. 54 of "Top Information Retrieval Papers of the Week" on Substack. My Substack newsletter features the 7-10 most notable research papers on information retrieval (including recommender systems, search & ranking, etc.) from each week, with a brief summary, and links to the paper/codebase. This week’s newsletter highlights the following research work: 📚 Graph Retrieval-Augmented Generation for Context-Aware Reasoning, from Emory University 📚 Enabling Fine-Grained Ranking at Varying Granularities via Multi-Vector Embeddings, from Reddy et al. 📚 Learning to Ask Clarification Questions for Ambiguous Retrieval Queries, from UC Berkeley 📚 Principled and Practical Multi-Vector Retrieval with Approximation Guarantees, from Google Research 📚 Towards User-Centric Recommendations via LLM-Based Simulation and Tool Learning, from USTC 📚 Building Pseudo-Graph Databases from Raw Texts for Retrieval-Augmented Language Models, from Renmin University 📚 Retrieval-Augmented Early Exiting for Efficient Large Language Model Inference, from Huang et al. 📚 A State-of-the-Art Generalist Embedding Model, from NVIDIA 📚 Enhancing Multilingual Language Model Training via Repeat Ranking, from Devine et al. 📚 A Real-World Evaluation of HNSW for Vector Retrieval, from Marqo #InformationRetrieval #ResearchPapers #CuratedContent #Newsletter  #substack

46

2 Comments

Apple recently unveiled an Open Source Efficient LLM (OpenELM), a ground breaking project aimed at making LLMs more accessible. Led by Sachin Mehta, Mohammad Hossein Sekhavat, and Qingqing Cao, OpenELM offers a family of smaller LLMs. What sets OpenELM apart is its focus on efficiency and accessibility. The model is designed to be lightweight & efficient enough to run on a smartphone, and with parameter counts ranging from 270 million to 3 billion, OpenELM models can process 2,048 tokens of context, providing users with powerful language capabilities on their mobile devices. The authors pretrained OpenELM on 1.8 trillion tokens, drawing from subsets of publicly available text datasets. Results have shown that OpenELM outperforms other open-source models trained solely on publicly available data. For example, a 1.08 billion parameter OpenELM surpassed a similar model (1.18 billion parameter OLMo), achieving superior performance across 5 tasks. Apple’s commitment to user privacy is achieved by allowing the models to run entirely on the device, Apple ensures that user activity remains secure and private. In a time when language technology is rapidly advancing, initiatives like OpenELM pave the way for more accessible and efficient language models. Follow AI Nexus - Club for more… #ai #apple

3

Meta AI has made a significant leap forward in model alignment with human preferences through their innovative approach, "Iterative Reasoning Preference Optimization". They haven't presented any information on generalizability, suggesting possible overfitting. Because computing reward solely based on final answer is limiting, e.g. the model could learn false reasoning and it is not easy to apply to dialogue and open-ended tasks. But by introducing a set of critic models in a jury setting, can effectively eliminate the risk and allow generalizability. Increasing accuracy for Llama-2-70B-Chat: - 55.6% -> 81.6% on GSM8K - 12.5% -> 20.8% on MATH - 77.8% -> 86.7% on ARC-Challenge https://lnkd.in/gGYkxTh7

3

LLM-Select: Feature Selection with Large Language Models D P. Jeong, Z C. Lipton, P Ravikumar Carnegie Mellon (2024) https://lnkd.in/ei5v2myA - LLMs can perform effective feature selection for supervised learning tasks by just using feature names and task descriptions, without seeing any training data. - The authors propose 3 methods for LLM-based feature selection: scoring features by importance (LLM-Score), ranking features (LLM-Rank), and sequentially selecting features via dialogue (LLM-Seq). - Sufficiently large LLMs like GPT-4 can identify predictive features just as well as traditional data-driven methods like LASSO, even with simple prompting strategies like zero-shot scoring features one at a time. - LLM-based feature selection works well across diverse real-world datasets, suggesting LLMs encode rich knowledge about real-world relationships that can inform feature selection. - LLMs may be useful not only for selecting features after data collection, but also for deciding what features to collect, especially in expensive domains like healthcare. abs: In this paper, we demonstrate a surprising capability of large language models (LLMs): given only input feature names and a description of a prediction task, they are capable of selecting the most predictive features, with performance rivaling the standard tools of data science. Remarkably, these models exhibit this capacity across various query mechanisms. For example, we zero-shot prompt an LLM to output a numerical importance score for a feature (e.g., "blood pressure") in predicting an outcome of interest (e.g., "heart failure"), with no additional context. In particular, we find that the latest models, such as GPT-4, can consistently identify the most predictive features regardless of the query mechanism and across various prompting strategies. We illustrate these findings through extensive experiments on real-world data, where we show that LLM-based feature selection consistently achieves strong performance competitive with data-driven methods such as the LASSO, despite never having looked at the downstream training data. Our findings suggest that LLMs may be useful not only for selecting the best features for training but also for deciding which features to collect in the first place. This could potentially benefit practitioners in domains like healthcare, where collecting high-quality data comes at a high cost. #AI #machinelearning #computation #language #LLM

31

I am extremely excited to share the work I am most proud of in my life so far! I believe this work will have a substantial impact, potentially revolutionizing the way inverse problems are solved using diffusion models. I want to extend my heartfelt thanks to all my collaborators Taihui Li, YUXIANG WAN, Tiancong Chen, especially my advisor Ju Sun and my mentor Xu Zhang. For more details, please visit https://lnkd.in/gjpGRmSN; our code is available at https://lnkd.in/g_BJ4_d2. #InverseProblem #DiffusionModel #GenAI #GenerativeAI #UMNProud

27

1 Comment

🚀 I'm happy to announce that our paper "Semi-supervised learning and visual transformers for product attribute extraction from e-commerce images" has been accepted at the CVPR 2024 Workshop on Fine-Grained Visual Categorization. Thanks to co-authors Manuel Lagunas, Víctor Martínez Gómez, Virginia Fernandez Arguedas, Christos Georgakis, Sofia Braun, Felipe Bertrand for the collaboration! 🙂 If you are attending CVPR later in Seattle, let's connect! Extracting product attributes from images involves classifying subtle differences between similar objects, like knowing the difference between a plaid pattern and a gingham pattern. Visual Transformers (ViT) are powerful but usually supervised, while e-commerce labels are noisy or expensive to clean. In this work, we demonstrate the benefit for e-commerce of semi-supervised techniques like Semi-ViT, a ViT model fine-tuned with unlabeled data. We demonstrate that when Semi-ViT is compared to fully supervised ResNets and ViTs, it improves precision by at least +107bps and coverage by+311bps even when trained with 25% less labeled data. #amazonscience #cvpr

128

3 Comments

Enhancing LLMs with Multi-Agent Methods: A Thought Experiment I recently explored the potential of smaller language models, like Llama3 8B, to match the performance of larger models such as ChatGPT using multi-agent methods. By integrating sampling and voting with strategies like CoT prompting and self-reflection, the hypothesis is that Llama3 8B could significantly improve its accuracy across tasks. This approach suggests that with increased ensemble size and collaborative methods, smaller models could approach ChatGPT's performance. This thought experiment indicates that smaller, cost-effective models, when enhanced with multi-agent techniques, could provide competitive performance. Future empirical studies are needed to validate these promising insights. https://lnkd.in/exj4nkMh https://lnkd.in/erFhc7t3

25

1 Comment