Zachary Kratochvil

Heard the term research parasite? It might sound like a bad thing - but it just refers to researchers who make use of data previously generated (i.e. secondary use). It's actually a great thing IMO. It saves time, money, and resources compared to generating the same datasets multiple times. It also democratizes research by enabling researchers to use data they might not have the resource to generate themselves (e.g. small-medium biotechs). However, it isn't always an easy thing to accomplish. Finding, accessing, downloading, processing, and analyzing large datasets is challenging. See this case study from Kai Lawson-McDowall at Sonrai on how we can streamline this process. Want help getting access and analyzing public data for your research? Don't currently have the bandwidth/resources/technical know how? Send me a DM! https://lnkd.in/exB6fAqv #research #translationalresearch #researchparasite #drugdevelopment #bioinformatics #datascience

23

This tiny article all started with a simple question: if we can use masked modeling for pre-training, why not use autoregression? It turns out - yes! With a gentle overview on MIM, I'll introduce a latest (to-be) ICML publication, using auto-regressive model for foundation model pre-training, which shows great potentials for scaling.

3

I worked on a project called "MTLLM" for #jaclang at Jaseci Labs. This project gives 💪 superpowers to regular functions and classes using LLMs without additional code modifications. I hope it would be a direct competitor to #dspy & #lmql. You can check out the repository here: https://lnkd.in/g5GrN9uJ. The documentation is still in progress. Paper is on the way. #research #mtllm #jaclang #llm #ai

31

🚀 Weekend Read: Delving into Transformer Reasoning Capabilities with Graph Algorithms 🚀 Looking for a thought-provoking read this weekend? I’m excited to dive into "Understanding Transformer Reasoning Capabilities via Graph Algorithms" by the talented team at Google Research and Columbia University. 🔍 What’s Inside: Transformer Scaling Regimes: Discover how transformers handle different classes of algorithmic problems, especially graph reasoning tasks. Theoretical Foundations: Gain insights into the theoretical underpinnings of transformers’ representational capabilities based on their depth, width, and additional tokens. Empirical Validation: See how transformers perform on graph reasoning tasks using the GraphQA benchmark compared to specialized graph neural networks (GNNs). 📊 Why It’s a Good Read: Innovative Framework: The paper introduces a novel representational hierarchy categorizing graph reasoning tasks by complexity and transformer capabilities. Performance Insights: Learn about the superior performance of transformers in tasks requiring long-range dependency analysis. Practical Impact: Explore the potential real-world applications of these insights in AI, from language modeling to computer vision. Check out the paper in the first comment. Let’s make this weekend insightful reading an excellent research! #AI #MachineLearning #Transformers #GraphAlgorithms #WeekendRead #ResearchDiscussion

11

1 Comment

On why you should use Jupyter Notebooks: "Notebooks are a great way to tell a story, and telling stories is what all fields should be about. Especially computer science." This is from an interview with Doug Blank, Head of Research at @cometml. He continues: "To me, a Jupyter notebook is a blank sheet of paper. You can write a story in it. And if you change the name of one of the characters in paragraph one, you have to change the name of the character throughout the whole story." People often tell me they don't like notebooks because people write bad code in them. Nonsense. Bad developers write bad code. Notebooks have nothing to do with that. "Some educators feel that it's too open-ended and too flexible, but I disagree — it's a new way of doing computing (...)." I always recommend that developers learn how to use notebooks. Not as their primary way of writing software but as an alternative tool they can use to experiment, troubleshoot, and become more effective at their work. The math in my head is simple: A developer who knows how to use notebooks effectively is better than a developer who doesn't. Notebooks aren't a replacement for what you do. They are a boost. The rest of the interview is pretty good: https://lnkd.in/e7ezJsx7

83

8 Comments

🔥🚀 Exciting News from MIT: Introducing FlowMap for Deep Learning-based Structure from Motion! 🔥🚀 MIT unveils FlowMap, a groundbreaking self-supervised, end-to-end differentiable SfM method delivering COLMAP-level accuracy for 360° scenes. FlowMap revolutionizes the process by precisely solving camera poses, intrinsics, and dense depth of video sequences without relying on unique image features or manual matching. This innovation outperforms prior gradient-descent based methods and rivals COLMAP, marking a significant advancement in SfM technology. 👉 Explore the project page, paper, and code: https://lnkd.in/gyyinuTu #Reconstruction #SfM #AI #DeepLearning #Innovation

7

Thanks to a new cron job running on the HACDC server, these Hugging Face datasets will be updated nightly with fresh data (until something breaks :) ) usc-billstatus (metadata about each bill | one row per bill) https://lnkd.in/ePtcUa-C usc-textversion (different text versions for each bill | one row per version) https://lnkd.in/eAMiit6q usc-unified (combined metadata and text versions for a bill | one row per bill) https://lnkd.in/eZyaMbMe

11

The reasons why AI-powered search stinks at sourcing are both technical and simple, Shah explained. The technical explanation involves something called retrieval-augmented generation (RAG), which works a bit like a professor recruiting research assistants to go find out more information about a specific topic when the professor’s personal library isn’t enough. RAG does solve a couple of problems with how the current generation of large language models generate content, including the frequency of hallucinations, but it also creates a new problem: It can’t distinguish good sources from bad. In its current state, AI lacks good judgment......

2

If you are benchmarking Computer Vision models and you use random inputs because you don’t have input data yet, be careful! If your model includes preprocessing, you may be getting higher times. Why? Usually the most expensive operation in preprocessing is the resize, and there’s a catch: It takes more time to resize a completely random image (noise) than a natural image! For example, here it is an example with the Pillow Python library: https://lnkd.in/d3RHYxEe In the notebook, you see that it takes almost twice the time for the random image when compared to a natural image or a monochromatic one. The solution is always use images that follow the expected distribution in production. For example in Nvidia Model Analyzer there is the “input-data” option (https://lnkd.in/dum_eRXF) where you can set the directory of input images or a JSON with the base64 string of the images. The exception is if you are analyzing just the model and you are happy with providing random images at the exact size the model is expecting (i.e. 224x224), so there is no resize involved.

8

anthropics/anthropic-cookbook: A collection of notebooks/recipes showcasing some fun and effective ways of using Claude. https://bit.ly/3QpOnSV #AI #MachineLearning #DeepLearning #LLMs #DataScience

36

In the current landscape, we've witnessed significant advancements in Large Language Models (LLMs). Precise assessment of these models is crucial for model selection, performance evaluation, and identifying and addressing biases. Let's talk about BLEU (Bilingual Evaluation Understudy), a widely used method for performance evaluation.  This metric, an extrinsic evaluation tool, compares model-generated outputs with one or more reference texts to gauge similarity. Scores range from 0 to 1, with higher scores indicating superior performance. This approach proves invaluable for evaluating the performance of translation models. Scores below 10 are negligible, while those exceeding 50 denote good translation quality. The BLEU formula comprises two key components:  1. Brevity Penalty: Penalizes overly short-generated translations compared to the reference length using an exponential decay 2. N-Gram Overlap: Assesses the match between unigrams, bigrams, trigrams, and four-grams in the generated output and the reference translations. To prevent overcounting, the n-gram counts are capped at the maximum occurring in the reference. This overlap is calculated as a precision-based metric, with the geometric mean of these four precision values being taken. Important considerations when utilizing BLEU for evaluation: 1. BLEU is corpus-based: It's not suitable for assessing individual sentences. 2. Lacks differentiation between content and function words: For instance, removing a function word like 'the' incurs the same penalty as replacing a significant term like 'Earth' with 'Mars'. 3. Limited in capturing sentence meaning: It only considers up to four-grams and overlooks long-range dependencies.

14

🤖 Staying on top of ML research Machine learning research is a fire hydrant. The field moves so fast, the course you took 5 years ago is probably outdated. To help myself and others stay updated, I thought I could share a few resources that have helped me stay on top of the latest machine learning research papers and trends. 🔍 Emergent Mind (https://lnkd.in/dJK4GbYe) Trending computer science papers from arXiv by topic curated by AI. Provides helpful GPT-4 generated summaries broken down by each section. I find it to be a great way to quickly identify impactful new research to dig into further. 🧠 Bytez (https://bytez.com) AI agent trained on the corpus of AI papers. Summarizes papers and can answer questions on specific topics. I use Bytez to keep browse keep tabs on papers and use it as my first reading step before diving deeper into the actual paper. 📧 Newsletters - 💡 Data Machina (https://lnkd.in/d-mPpbzB) Weekly roundup summaries and papers in MLOps, datasets, deep learning, and AI research - 💡 Deep (Learning) Focus (https://lnkd.in/dsdM2Uhe) In-depth dive into a single topic at a time. (Also a blessing to your LinkedIn feed if you follow the curator 😁) - 💡 AI Tidbits (https://www.aitidbits.ai/) Weekly roundup from 10 other newsletters broken down by subject (language models, vision, audio, industry, open-source packages, etc.) - 💡 AlphaSignal (https://alphasignal.ai/) Short and sweet summaries of top news, papers, and repos. Sometimes it comes with a nice tip that surprises my team (e.g., cutting GPU memory usage with sequential backward calls when using multiple losses in PyTorch).

26

Put simply, the three libraries— #LILO (library induction from language observations), #Ada (action domain acquisition), and #LGA (language-guided abstraction)—all work to provide human-like reasoning across certain functions, such as computer programming, task planning, and robotic tasks. Using the neurosymbolic method baked into LILO, MIT uses its Stitch ( get it? ) algorithm to identify abstractions. This allows #LLMs to apply commonsense knowledge with sophistication that...

2

Day 5 of learning in public 🚀 Traditional fine-tuning methods for large language models often face challenges like high computational requirements, catastrophic forgetting, and single-task specialization. Enter LoRA (Low-Rank Adaptation), a game-changing technique that addresses these limitations, enabling efficient and practical fine-tuning of large language models. LoRA introduces a novel approach to fine-tuning by adding a small number of task-specific weights to the pre-trained model instead of updating all the weights. Here's how it works: 1. Low-Rank Matrices: For each layer in the pre-trained model, LoRA introduces a pair of low-rank matrices: one for the weights and one for the biases. 2. Adaptation: During fine-tuning, LoRA updates these low-rank matrices instead of modifying the pre-trained weights directly. 3. Composition: At inference time, the low-rank matrices are composed with the pre-trained weights to produce the final, adapted weights for the fine-tuned model. The low-rank nature of these matrices is the key to LoRA's efficiency, dramatically reducing the memory and computational requirements for fine-tuning. This makes it possible to adapt large language models even on consumer-grade hardware like CPUs or edge devices. But LoRA's advantages don't stop there. It can match or even outperform the performance of full fine-tuning while using far fewer updatable parameters. Additionally, the LoRA updates are reversible, allowing the model to revert to its original state, and composable, enabling a single pre-trained model to be fine-tuned for multiple tasks simultaneously. To illustrate the power of LoRA, let's dive into a practical example. Here, I'll demonstrate how to implement LoRA from scratch and fine-tune the DistilBERT model using LoRA on the IMDB dataset for sentiment analysis. https://lnkd.in/g-Tz2NU7 As you can see, LoRA is a remarkable technique that addresses the limitations of traditional fine-tuning methods, making it more efficient, practical, and versatile. By introducing low-rank, task-specific updates, LoRA enables efficient adaptation while preserving the model's original knowledge and allowing for multi-task specialization. As the demand for deploying large language models on resource-constrained devices continues to grow, techniques like LoRA will make these powerful models more accessible and practical.

4

LLMs can output tokens 200% faster using multi-token prediction. And it’s now been open sourced by Meta. Here’s how it works — Multi-token prediction extends standard language modeling -- it predicts several future tokens at once instead of just the next one. - First, the model uses a shared core to make a context representation from its past tokens. - Then, it predicts multiple future tokens in parallel using separate prediction layers. This approach is much more memory-efficient, so during training, it can handle large vocabulary sizes without overloading the GPU. And at inference, it can decode much faster. Why does it work so well? According to the paper, it’s because multi-token prediction is able to reduce the mismatch between training and inference. This makes the model better at handling long-term dependencies. Paper: https://lnkd.in/ge_quFaS HuggingFace models: https://lnkd.in/gUgzsbtZ #AI #LLMs

7

Do you want to build and deploy custom LLM agents from scratch? Check out my blog post with Databricks Mosaic Research to read about how JetBlue is shipping robust GenAI solutions with DSPy and Databricks! https://lnkd.in/gKcVKQ_B

82

4 Comments

AI isn't just about creating an impressive demo. True value lies in an AI that can work in predictive manner and generalize well on large-scale production data. At a recent AI hackathon, I noticed people getting excited over demos running on a few samples, but there was no benchmarking involved. How can we be sure it will perform well on large datasets in a real production environment? Machine learning and AI are ineffective without proper benchmarking on datasets that mimic the production environment. #ML #AI #LLM

7

3 Comments

📖 "Efficient Multimodal Large Language Models: A Survey" - a comprehensive review of recent progress in efficient MLLMs. Key advances include:   💡 Architecture: optimizations in vision encoder, vision-language projector, language model, vision token compression, and using efficient structures like MoE and Mamba. 💡 Training: pre-training, instruction-tuning, and fine tuning 💡 Data: pre-training data, instruction data and benchmarks   Efficient MLLM efforts aim to make the powerful models more accessible - reducing cost and latency, and enabling deployments on device for privacy protection.   Read the full paper: https://lnkd.in/gFEw_XQ8   #generativeai #MLLM #largelanguagemodels

21

Pydantic is sponsoring Frank Hoffmann because inline-snapshots is transformative to how we write unit tests in Python! https://lnkd.in/en6HtXCF If you're using pytest, and you're not using inline-snapshots, you're wasting your time; simple as that.

336

19 Comments