Colin Borys

Routing the control flow within an RAG application based on the user’s query intent can help us create more useful and powerful Retrieval Augmented Generation (RAG) based applications. #llms #rag #chatgpt4 #gemini #aiml #llmops https://lnkd.in/d54eVrHZ

8

A collection of RAG techniques to help you develop your RAG app into something robust that will last #genai #llms #aiml #rag #chatgpt4 #llmops https://lnkd.in/dGugWfbz

8

This is guide for any AI Engineer looking to build autonomous AI Agents. #llms #chatgpt4 #llmops #ai #datascience #mlopd #aiagent https://lnkd.in/g4md6Vvn

4

���� "Efficient Multimodal Large Language Models: A Survey" - a comprehensive review of recent progress in efficient MLLMs. Key advances include:   💡 Architecture: optimizations in vision encoder, vision-language projector, language model, vision token compression, and using efficient structures like MoE and Mamba. 💡 Training: pre-training, instruction-tuning, and fine tuning 💡 Data: pre-training data, instruction data and benchmarks   Efficient MLLM efforts aim to make the powerful models more accessible - reducing cost and latency, and enabling deployments on device for privacy protection.   Read the full paper: https://lnkd.in/gFEw_XQ8   #generativeai #MLLM #largelanguagemodels

21

Learn how an image-text multi-modality model can perform image classification, image retrieval, and image captioning #llms #llmops #image #opencv #chatgpt4 #gemini #aiml https://lnkd.in/gJFh9Ruf

1

Spatial Reasoning did not ‘emerge’ spontaneously in Large Language Models (LLMs) the way so many reasoning capabilities did. Humans have specialized, highly capable spatial reasoning capabilities that LLMs have not replicated. But every subsequent release of the major models — GPT, Claude, Gemini- promises better multimedia support, and all will accept and try to use uploaded graphics along with texts. #llms #chatgpt4 #gemini #llmops #aiml #ml https://lnkd.in/dFSWKqKs

2

Nice. Greatly useful to improve diffusion models generalization with multi-modal data!

9

1 Comment

This writeup is a goldmine of knowledge on building with LLMs ✨ My most important takeaway being: “Build eval framework. Can be super basic, but have one” https://applied-llms.org/

35

Anyone working on production grade RAG?  I'm trying to get a sense of the best foundation models suited for RAG that can fit on a 24GB GPU(A10G). Specifically looking for good "needle in a haystack" performance. Currently using Mistral v0.2 Q8 and while it's okay,I think there might be better model in the 13b range.  Tried using command R - which is trained specifically for RAG tasks, but the 2 and 3 bit quants have significant performance degradation and the 4+ quants are too slow to be usable. 

2

8 Comments

📖 "Scaling Synthetic Data Creation with 1,000,000,000 Personas" - A very interesting work in diverse synthetic data generation. In addition to the paper's focus of data generation, this rich and diverse persona hub itself is an exciting source of many applications!   Key highlights: 💡 Persona hub: 1 billion personas curated from the web (that is ~13% of world population!) 💡 Scalability and diversity: prompt web text with LLM to extract personas, and derive under-represented personas via interpersonal relationships 💡 Applications: diverse perspectives from personas enable synthetic data generation for various applications(e.g. math, logical reasoning, user instructions, game NPCs, etc.)   Note: ⚠ Security risk in potential exploitation of LLMs' data and knowledge. ✅ 200k personas are released in https://lnkd.in/gvVB_7m8   Read the full paper: https://lnkd.in/gjT3CVQg #generativeai #llms #ai

8

Prompt Decomposition is the process of taking a complicated prompt and breaking it into multiple smaller parts. This is the same idea that is found in design theory and sometimes called task decomposition. Simply put, when we have a large complicated task, we break it down into multiple steps and each step is individually much easier. #llms #chatgpt4 #llmops #aiml #rag #mlops https://lnkd.in/dR-aci8h

14

Large language models (LLMs) are known to generate plausible sounding factually incorrect responses, especially when faced with unfamiliar concepts, queries. Recent research suggests that strategic fine-tuning and in-context learning, retrieval augmentation can significantly reduce these hallucinations. #llms #chatgpt4 #llmops #aiml #gemini #phi #mlops https://lnkd.in/duRrC3yg

4

🥁 Llama3 is out 🥁 8B and 70B models available today. 8k context length. Trained with 15 trillion tokens on a custom-built 24k GPU cluster. Great performance on various benchmarks, with Llam3-8B doing better than Llama2-70B in some cases. More versions are coming over the next few months. https://lnkd.in/dNZx72FJ

19,208

292 Comments

Intuitive Example, Principles, Code Explanation and Insights about CRAG Take your LLM performance to the next level. #llm #llmops #chatgpt4 #mlops #ai #datascience #ml #gemini tps://https://lnkd.in/dDkZiZ5m

13

Great work complementing benchmarks which focus on output quality. Speed and cost effectiveness are equally crucial in productization to deliver customer value and business value!

7

This article will explore how leveraging lower-precision formats can enhance training and inference speeds up to 3x without compromising model accuracy. Mixed-precision training is a powerful technique that significantly enhances the speed and efficiency of training deep neural networks on modern GPUs. Traditionally, 64-bit precision has been the standard in scientific computing due to its high accuracy, but it's computationally expensive and not optimized for GPUs. In deep learning, 32-bit precision, known as single-precision, strikes a balance by reducing memory usage and speeding up computation without greatly impacting model accuracy. Mixed-precision training takes this a step further by strategically alternating between 32-bit and 16-bit precision during training. Just using 16-bit for training might mean the model's loss will not converge. What does mixed-precision means? Mixed-precision training involves converting neural network weights to 16-bit precision for faster computation, then computing gradients in 16-bit before converting them back to 32-bit for numerical stability. Finally, these gradients are used to update the original 32-bit weights. This method not only cuts down training time but also reduces memory consumption, allowing larger models and batch sizes to be processed efficiently. Interestingly, the slight imprecision introduced by lower precision can act as a regularizer, potentially improving the model's generalization and accuracy. New floating-point formats like bfloat16 offer extended dynamic ranges, making them particularly suitable for deep learning tasks on modern hardware, such as NVIDIA's A100 Tensor Core GPUs. For even greater performance improvements, quantization can be employed (during inference, or even while training). This process converts model weights to low-bit integer representations, like 8-bit or 4-bit integers, offering substantial gains in efficiency and speed during inference. The author,  Sebastian Raschka, explains all this using experiments on a model. https://lnkd.in/eEu3Rfg6

7

The idea of agents has been around for a while. What’s new here? In his talk, Andrew Ng explains agentic systems that use design patterns such as reflection, tool use, planning, and multi-agent collaboration to outperform the results of zero-shot prompting. #llms #aiagent #ml #rlhf #mlops #datascience #llmops https://lnkd.in/d2Q8Yj7X

6

After reading Anthropic and OpenAI success stories with sparse autoencoders for interpretability of text, I was curious about its application to images... After a quick search, I found out that Hugo Fry wrote a blogpost in LessWrong: https://lnkd.in/dA33fWqE which also provides some playground! Of course part of the success is due to the ImageNet structure itself (as many papers out there) but still is an interesting exercise. It enables lots of experiments, and I already can think about practical usage for improving image search in marketplaces. #interpretability #image #CLIP

7

3 Comments