Ayoub Kachkach

Facebook just released a very different kind of LLMs. These are focused on code optimization and compiler optimization. This is super important because it can fundamentally change the way code is executed on any machine. Here's how it works right now: Programs written in many languages like C/C++ have intermediate steps before they are run on the computer. For example they are converted into Byte Code and then converted into machine code via LLVM (Low level virtual machine). But Facebook's new model can take in the byte code as input and convert it to into a more efficient representation so that when this new code is converted into machine code, it will be faster. "In theory", it could mean that all existing code might be made faster without any change by developers. Our games can have higher frame rates on the same GPUs. Photoshop might run faster on the same laptop. Our software might complete tasks faster without any changes at all. But that will probably take quite a long time. To be honest I don't expect these models to actually be used in production since unless they were 100% as good as compilers they will probably break the code. And are they 100% good? No, not even close. But the speed at which improvements are happening, they will probably reach that level eventually. And that's when we will see models like these everywhere. 

11

These findings motivate our design of a novel framework to investigate hidden representations in LLMs with LLMs, which we call Patchscopes. The key idea behind this framework is to use LLMs to provide natural language explanations of their own internal hidden representations. Patchscopes unifies and extends a broad range of existing interpretability techniques, and it enables answering questions that were difficult or impossible before. For example, it offers insights into how an LLM's hidden representations capture nuances of meaning in the model's input, making it easier to fix certain types of reasoning errors. While we initially focus the application of Patchscopes to the natural language domain and the autoregressive Transformer model family, its potential applications are broader. For example, we are excited about its applications to detection and correction of model hallucinations, the exploration of multimodal (image and text) representations, and the investigation of how models build their predictions in more complex scenarios. Google DeepMind

8

In-Context Learning at Extreme Scales: My Thoughts As large language models (LLMs) continue to expand their context capacities, the domain of in-context learning (ICL) at extreme scales has piqued my interest. A recent paper by researchers at Carnegie Mellon University and Tel Aviv University provides a detailed exploration into this area, unveiling both impressive capabilities and notable challenges. 🔍 Key Insights: Performance Improvement: The study demonstrates that ICL can improve significantly by integrating hundreds to thousands of examples into the context window—far beyond traditional few-shot settings. This is particularly impactful in complex tasks like 77-way banking intent classification, where long-context ICL matches or surpasses traditional finetuning. Input Sensitivity: Long-context ICL shows reduced sensitivity to input shuffling compared to shorter contexts. However, organizing examples by label dramatically reduces performance, emphasizing the need for diverse context configurations. Source of Gains: Performance gains primarily arise from accessing more relevant examples, rather than aggregating knowledge across many examples during encoding. 📉 Challenges & Drawbacks: Sensitivity to Example Order: Despite reduced sensitivity, the performance still varies with input order, indicating potential instability in practical scenarios. Performance Saturation: The models often do not fully leverage their extended context capacities, showing performance saturation before reaching the maximum context length. Computational Cost: The increased computational cost associated with using large context windows could limit practical deployment, particularly in resource-constrained environments. Efficiency Issues: There is a noted inefficiency in how models utilize the encoded information within large datasets, suggesting that current architectures may need refinement to better use available data. 🚀 Future Directions: The paper suggests several exciting avenues for future research, including optimizing the trade-offs between computational costs and performance, exploring model adaptation strategies, and refining ICL approaches for specific applications or datasets. As we push the boundaries of what LLMs can achieve, effectively leveraging their full context could unlock new capabilities. This area is ripe for further exploration and could lead to major advancements in how we deploy AI. For those at the cutting edge of language model development, I highly recommend this paper for its thorough analysis and forward-looking perspective. Check out the full work at https://lnkd.in/d7AmGh4y. What are your thoughts on the future of in-context learning at such scales and the challenges discussed? #AI #MachineLearning #DeepLearning #LanguageModels #InContextLearning

17

An amazing article on using  Low Rank Adaptation (LoRA) and QLoRA (Quantized Low-Rank Adaptation)  for fine-tuning LLMs. https://lnkd.in/gHu7hfAE #lora #qlora #machinelearning #llms

16

Less than 20 lines of code is all it takes to get started with Llama-3 on Intel Xeon Processors. Get started with the tutorial below. #IntelXeon #LLM #Developer

6

How to optimize Ollama for CPU-only inference? I am wondering if anyone has documentation, blog posts, ideas, ... really anything(!) about optimizing inference for CPU-only. The reason is that most people do not have nice GPU(s) in their notebooks and rely on API providers or remote hardware. Would be great to give those people a good hint on which model they could use locally. We are currently on a good way of doing a massive benchmark about CPU-only inference so i appreciate your input greatly!

It’s day 4 of our 11 Days of Inference Acceleration Techniques. Today, we’re moving on to runtime level optimization best practices. 𝐅𝐨𝐮𝐫𝐭𝐡 𝐭𝐢𝐩: 𝐓𝐚𝐤𝐞 𝐚𝐝𝐯𝐚𝐧𝐭𝐚𝐠𝐞 𝐨𝐟 𝐠𝐫𝐚𝐩𝐡 𝐜𝐨𝐦𝐩𝐢𝐥𝐚𝐭𝐢𝐨𝐧 📈 Graph compilers such as TVM, Tensor-RT, and OpenVino work by getting a computation graph of a specific model and generating an optimized code adjusted for the target hardware. Graph compilation can optimize the graph structure by merging redundant operations, performing kernel auto-tuning, enhancing memory reuse, preventing cache misses, and more. But few things to be aware of: 📝 Not all models compile equally. 📝 The impact of compilation on model performance can vary. Hence, make sure to check the architecture's ability to compile early on in the process to avoid wasting time and resources on training a model that can’t be optimized for fast inference. – What’s the #11DaysofInferenceAccelerationTechniques? The Deci team is posting, for 11 days, a series of inference acceleration techniques for deep learning applications. If you’re looking for practical tips and best practices for improving inference, follow Deci AI so you won’t miss an update. #deeplearning #machinelearning #neuralnetworks #computervision

9

Software Engineering and IO for CPU and GPU: Everybody these days is working with big data. I’m going to speak a bit about Direct Memory Access (DMA) and how it’s used in HPC for both CPUs and GPUs to significantly speed up workflows. Remote DMA (RDMA) is a spec that defines an interface for host software to communicate with Non-Volatile Memory (NVM) subsystems. The NVM express (NVMe)/RDMA transport provides a method of transporting NVMe traffic over a network that supports RDMA such as InfiniBand (via an API). RDMA enables applications to bypass the software stack when processing network traffic, which produces lower latency and higher performance speeds. This is accomplished by the NVMe device sending messages across the Peripheral Component Interconnect (PCI) bus, which contain physical memory addresses. The speedup comes because this does not involve the CPU, saving precious clock cycles for more important stuff. A Memory Management Unit (MMU) is responsible for making these memory accesses correct. Ultimately, NVMe/RDMA allows multiple compute machines networked via a fabric like Infiniband to communicate directly with each other’s memory without involving the CPUs to fetch and load data via a bounce buffer, which enables acceleration of HPC workloads. When writing applications that utilize GPUs, it is important to recognize that the CPU must still be involved. CPUs launch work for the GPU, and may perform compute intensive tasks. In fact, some tasks are still better suited for a CPU than a GPU, so processing power and availability on the CPU is still very important for GPU applications. GPUDirect Storage (GDS) by NVIDIA takes DMA a step further when considering GPUs. GDS allows for storage to communicate directly with GPU memory, skipping any bounce into CPU memory prior to loading to GPU. This distinction is important, because it allows us to skip the host (CPU) memory step which which would normally be required without GDS. GDS ultimately increases available GPU memory bandwidth, and CPU utilization. This was intended to be an introduction to HPC storage. I’d encourage you to dive deeper into any of the topics discussed above to gain an understanding and appreciation of the genius that has gone into designing these high performance systems. If you like my content, feel free to follow or connect! #softwareengineering #hpc #storage

277

22 Kommentare

PMC-LLaMA: Towards Building Open-source Language Models for Medicine An year old paper, but a lot of interesting, practically significant details about how fine-tuning can *significantly* improve models for a specific domain. Many companies do fine-tune models for certain domains representing their business, and here are interesting insights how to achieve greater gains "Compared with baseline 7B LLaMA model, integrating biomedical papers brings a performance gain from 44.54% to 44.70% and 48.51% to 50.54% on MedQA and MedMCQA respectively. While after adding books for training, the performance is improved significantly, i.e., obtaining 1.02%, 2.94%, and 1.2% on MedQA, MedMCQA and PubMedQA respectively. Both observations have shown the importance of injecting fundamental medical knowledge." "Furthermore, integrating conversations with rationale QA for instruction tuning can produce substantial enhancements, with performance boosts from 49.32% to 54.43% on MedQA. This demonstrates the pivotal role played by the diversity of question types during the instruction tuning stage, as all involved questions will be limited on medical choice tests without conversation. In addition, the incorporation of a knowledge graph introduces a further improvement of 1.93% on the MedQA dataset, demonstrating the importance of using explicit instructions to emphasize the key medical concepts." and other details are inside https://lnkd.in/gCvTkJf7

6

The OP is wrong about the count, I think they are counting all models in HF as LLMs. Because there are only 700K+ total models in HF. But it sure begs the question why ~100K text generation models (if not LLMs). For perspective sentence-transformer library based models are only ~7K Related reads ⭕️ Elon’s idiot index for LLMs: https://lnkd.in/eWsbM45J ⭕️ How open is your LLM: https://lnkd.in/eY6JE7Ae

15

3 Kommentare

Leverage StarCoder 2, a powerful LLM for code generation, completion, summarization, and documentation. Try the now available NVIDIA-hosted StarCoder 2 API, packaged as NIM inference microservice for free from the NVIDIA API catalog. https://dy.si/Zf8AD

8

April was an exciting month for AI at Meta! We launched MTIA v2 https://lnkd.in/gvC-UgVN, Llama3 https://lnkd.in/gdbfXa-6, presented a tutorial and paper on the PyTorch2 compiler at ASPLOS https://lnkd.in/gsT9DUr5, released PyTorch 2.3 https://lnkd.in/gYn3Vnzm and, to top it off, we launched the PyTorch ecosystem solution for mobile and edge deployments, ExecuTorch Alpha https://lnkd.in/gWv96jpm optimized for Large Language Models. What better than to combine all of these... running Llama3 on an a mobile phone exported with the PT2 Compiler's torch.export, and optimized for mobile deployment. And you can do all of this in an easy-to-use self-service format starting today, for both iPhone and Android as well as many other mobile/edge devices. The video below shows Llama3 running on an iPhone. (Makers will love how well models run on Raspberry Pi 5!) PyTorch is being developed by a multi-disciplinary team comprising ML engineers, accelerator experts, compiler developers, hardware architects, chip designers, HPC developers, mobile developers, and specialists and generalists that are comfortable across many of the layers involved in building end-to-end solutions. Even better -- if you're excited by the possibilities of AI, and solving the system design challenges of making AI run well across all hardware types, we are looking for YOU! The Pytorch team has openings across PyTorch core, compilers, accelerators and HW/SW co-design https://lnkd.in/gu2Y_KpY https://lnkd.in/gQVXBP6P and a broad range of positions that involve PyTorch from model development all the way to hardware deployments https://lnkd.in/gjDdUh5w #PyTorch #ExecuTorch #Llama3 #AICompilers #MTIA #AcceleratedAI #MetaAI #Meta

64

Kolmogorov–Arnold Networks seem like a paradigm shift in ML architectures. Curious to see how they evolve. "While MLPs have fixed activation functions on nodes ("neurons"), KANs have learnable activation functions on edges ("weights"). KANs have no linear weights at all -- every weight parameter is replaced by a univariate function parametrized as a spline. " https://lnkd.in/gdPk_qCQ

29

At the AWS Summit Berlin this year, I was asking around everyone one question- "Do you have any LLM application running in production?" The most common response involved hesitation followed by a "No- we have experimented a bunch though". Almost 1.5 years after the chatgpt revolution began, this is the reality as far as industrial adoption of LLMs is concerned. In fact, these are the usage trends: - Most companies have done some form of experimentation and maybe have some prototypes they're struggling to trust- so, nothing in production. - Some companies are actively using some LLM-based applications to help improve their internal processes and workflows (as opposed to exposing LLM results to their end customers). - Still fewer companies have an LLM-based app that their customers interacts with in some way (either directly via a chatbot or so OR by consuming the output of an LLM in some form). - An extremely tiny portion of companies have succeeded in using opensource LLMs in production in addition to the enterprise ones. Despite all the cool developments in the area, the adoption rate as far as productionization of these models go, remains low. I'll ask around the same question at next year's AWS summit and hopefully, will get a lot of yeses ;) #awssummit #llm #ai #opensource #chatgpt

32

3 Kommentare

The King Ludwig Railway never reached Heilbronn. To make things worse, it never even got permission to get an extension to Würzburg. 🤣 Of course, you should not really expect good factually true answers from Llama 8B. It is a cool model for fine-tuning with your custom datasets. For more "truthfulness" you should try Llama 70B, which I forgot to download to my MacBook Pro this morning. 🤗 The intention of this post is to suggest to you to give Anything LLM a try. If you do not want to work with Python, Jupyter Notebooks, the command line et cetera, and just want a straight-to-business user interface, start here. Anything LLM allows you to run open-source LLMs on your computer. And since it integrates Ollama chances are high that you will get some decent hardware acceleration on your machine.

6

2 Kommentare

New paper "Better & Faster Large Language Models via Multi-token Prediction" can change direction of LLMs. Traditional LLMs rely heavily on next-token prediction, which requires extensive data and computing power. This method slows down progress and limits the practical use of these powerful models. This paper's new approach, multi-token prediction, forecasts several future tokens at once, using resources more efficiently and improving learning speed. The models learn more effectively, using less data. There's a notable enhancement in tasks like coding and natural language processing. Models run up to three times faster, suitable for real-time applications. This method marks a significant shift in AI training, promising more sustainable and widespread use of LLMs. It opens doors to further innovations across various AI applications. The work by Gloeckle and his colleagues is a major step forward for AI, making LLMs faster and more capable. This could change how we use AI in everyday technology. Check out the full paper https://lnkd.in/gPusEdvQ for more details. Join the conversation on how this innovation could shape the future of AI.

8

2 Kommentare

Is your LLM ready for the real world? Loong benchmark puts it to the test. Evaluating the capabilities of Large Language Models (LLMs) can be quite tricky and with constantly growing context sizes, new benchmarks have been popping up left and right. But are current long-context benchmarks truly reflective of real-world scenarios? Researchers argue that existing benchmarks often rely on irrelevant noise to artificially extend the length of test cases. While this might help assess raw capacity, it doesn't align with how LLMs are actually used. Loong is a new long-context benchmark that aims to change that. It introduces four types of tasks: Spotlight Locating, Comparison, Clustering, and Chain of Reasoning. These tasks cover a range of context lengths and are meant to provide a more realistic and comprehensive evaluation of long-context understanding. In addition, the test cases are designed around extended multi-document question answering, ensuring that each document is relevant to the final answer. So, how do existing models fare on Loong? The results suggest that there's still room for improvement. Even retrieval augmented generation (RAG), a popular approach for long-context modeling, struggled with Loong's challenges. While a lot of long-context applications might not be feasible yet, there's so much room left for growth in this area. ↓ Liked this post? Join my newsletter with 35k+ readers that breaks down all you need to know about the latest LLM research: llmwatch.com 💡

89

1 Kommentar

So many beginners make 1 mistake when finetuning LLMs. They spend months and get nowhere. Recently, I worked with a team on an LLM summarization task. A fairly simple task. The team finetuned Mistral 7B Instruct model. I asked if the model improved significantly, but they didn’t know. They took 20 samples to evaluate the LLM. Visually. I didn’t know what to say. ❌ Visual evaluation could be essential but not scalable. ❌ The first 20 samples couldn't be sufficient. ❌ You need the next 200 samples. The first 20 doesn’t account for the next 200. That’s when I suggested using a simple yet effective metric. Recall-Oriented Understudy for Gisting Evaluation” (Rouge). What is Rouge, and how does it work? Rouge is an evaluation metric that compares reference and generated words. It uses unigrams, bigrams, and n-grams to compare responses. If you are not familiar with these terms, ☑️ unigram: A single word ☑️ bigram: A group of two words ☑️ n-grams: A group of n-words Now, let’s see the 3 categories of Rouge score: ✅ Rouge - 1 Metric Utilizes unigrams to compare the responses generated by LLM with the reference words. Rouge -1 is the simplest and least reliable metric. ✅ Rouge-2 Metric Utilizing bigrams, Rouge-2 is a more robust metric that compares LLM-generated responses with reference. ✅  Rouge-L Metric Rouge-L examines the "Longest Common Subsequence" (LCS) to compare the responses generated by LLM with the reference text in terms of words. Rouge-L is the most reliable evaluation metric. 📚 Bonus If you want to learn more, check the resources below by: → Fabio Chiusano → Edwin (Xue-Yong) Fu → Hafsa Farooq → Harel Wilner 🎖️ Rouge evaluates all your test set, not only the first 20 samples. 📌 If you want to support my work (and want to make my day): → Like → Repost ♻️ Thanks! 📌 P.S. Full code, resources with examples below - for FREE 👇

11

8 Kommentare