Edward Chan

🎉 PyTorch just made available the alpha release of ExecuTorch library. 🦋 What it does: 🎈 Helps deploy Large Language Models (LLMs) on edge devices (including mobiles) 👉 Since, mobile devices are highly constrained for compute, memory, and power, this introduces a considerable amount of challenge in bring the LLMs to such devices. To pack these models appropriately, the PyTorch team heavily leveraged quantization and other techniques. 🎁 What it offers/supports: 👉 supports 4-bit post-training quantization using GPTQ 👉 production tested, ExecuTorch is used by Meta for hand tracking on Meta Quest 3 and a variety of models on Ray-Ban Meta Smart Glasses 👉 supports Llama 2 7B efficiently on iPhone 15 Pro, iPhone 15 Pro Max, Samsung Galaxy S22, S23, and S24 phones and other edge devices 👉 early support for Llama 3 8B 👉 broad device support on CPU via dynamic shape support and new dtypes in XNNPack 🚀 Blog post: https://lnkd.in/gTFxdDhP 👉 GitHub: https://lnkd.in/gN-Yjj4A 👉 Docs: https://lnkd.in/gVD-2Tzt #LLMs #edgedevice #deploment #ml #pylib #executorch #pytorch

8

Earn royalties for automating troubleshooting steps? See an AI Assistant use your steps to respond to alerts and questions? If you can be in San Francisco on Sat June 22nd, sign up for SRE Day and use discount code KYLE20 so we can provision you a dev environment for our workshop. Link in the comments. I'm also taking volunteers for anyone who wants to race Eager Edgar, one of our AI Assistants, troubleshooting popular Kubernetes-based apps... #sre #devops #platformengineering #whatdoirunwhen

76

10 Comments

OpenPipe Introduces a New Family of ‘Mixture of Agents’ MoA Models Optimized for Generating Synthetic Training Data: Outperform GPT-4 at 1/25th the Cost Quick read: https://lnkd.in/gfxdiDT5 OpenPipe’s MoA models have excelled in rigorous benchmarking tests, achieving notable scores on LMSYS’s Arena Hard Auto and AlpacaEval 2.0. The MoA model scored 84.8 on Arena Hard Auto and 68.4 on AlpacaEval 2.0, indicating its superior performance in generating high-quality synthetic data. These benchmarks are critical as they represent challenging user queries that test the robustness and adaptability of AI models. The MoA model has been benchmarked against various GPT-4 variants in real-world scenarios. Results showed that OpenPipe’s MoA model was preferred over GPT-4 in 59.5% of the tasks evaluated by Claude 3 Opus. This is a significant achievement, highlighting the model’s effectiveness and practical applicability in diverse tasks encountered by OpenPipe’s customers...... OpenPipe

43

3 Comments

Kiddos are cranking EdgeIn If there’s a hot new tag or topic with companies behind it > we surface the data for you in <20 minutes. Presenting the Eigenlayer and AVS list of organizations, investors, people, and soon news/events. https://lnkd.in/gabF6tjy Your Data, Your Network, Your Keys, Your Earnings!!!! #web3 #AI

5

I want to mention an important update in zk proof innovation for those of you who were interested in my last talk on this topic at LABITCONF: This VM recently launched by a16z crypto may be a breaking point in the history of zk proof machines: https://lnkd.in/e_JNiCxY What are the main advantages of JOLT? - Jolt allows developers to use zero-knowledge (ZK) proofs without the need to deeply understand snarks, - significant improvement in the speed of the testing process (prover) compared to other ZKVMs such as RISC-Zero and SP1, which makes it more practical for real applications. - reduced risk of errors and increased reliability thanks to simplified formal verification. If you are interested in scaling your blockchain application, let's talk!

13

Nearly 2% of jobs have roughly half their tasks overlapping with AI. Estimating future impact that number grows to nearly half of all jobs. Great read on the current and future impact of AI. It reinforces my understanding that jobs are made of tasks. Many of those tasks will be automated. Some won’t. Those that are not automated will be increasingly valuable.

These stories of non-meteoric growth are so important.

1

Major ChatGPT outage. If you're experiencing issues and need access to an LLM, come try out Datasaur: llm.datasaur.ai - you can try out GPT 4o, Claude 3, Gemini Flash Pro, Mistral and more 🌈

83

I've seen some cool stuff at Google I/O. Context caching is a game-changer. Here’s why context caching in Gemini 1.5 Pro is crucial. The latest LLMs offer increasingly bigger context windows. While Anthrophic, Google, and OpenAI are battling it out, here is what matters to your AI strategy & transformation: 1) Cost Efficiency: 👉 Reusing cached context cuts costs per prompt. 👉 Great for apps with high traffic and repetitive queries. 2) Performance Improvement: 👉 Faster response times. 👉 Less need to recompute activations. 3) Scalability: 👉 Handles larger context windows efficiently. 👉 Supports extensive documents, large codebases, or long video content. So in a nutshell: 💡 Enjoy Enhanced Application Performance Applications can handle more complex and larger datasets without needing more computational resources. This makes them more responsive and efficient, especially in customer service, content creation, and data analysis. 💡 Effective Cost Management Cost savings are significant in high-traffic environments where the same context is used repeatedly. This makes advanced AI capabilities more accessible and affordable for various applications. Context caching could change everything. ✅ Make sure you have this on your radar for AI strategy and technology decisions.

545

39 Comments

Inexpensive token generation and agentic workflows introduce fresh prospects for training Large Language Models (LLMs) with synthetic data, potentially elevating output quality. Direct pretraining of LLMs on their generated responses proves ineffective, but employing agentic workflows could yield superior results, akin to humans learning from their own thinking processes. LLM training typically comprises pretraining on vast amounts of unlabeled text data, followed by fine-tuning and alignment to human values. However, sourcing adequate training data remains a hurdle, prompting exploration of innovative methods like training smaller models on the output of larger ones. Despite the substantial cost of token generation, agentic workflows present promising avenues for creating high-quality synthetic data, ultimately enhancing LLM performance.

11

Machine Learning Engineer - Founding Team - London £200k If you answer yes to 4/5 of these questions, then you want to check out this job (link in comments) 1️⃣ Do you have experience with quantization techniques (e.g., PEFT, DeepSpeed, TensorRT) to optimize ML models? 2️⃣  Have you ever managed or worked with a multi-node HPC cluster? 3️⃣ Are you familiar with Deep Learning using PyTorch Library? 4️⃣ Do you have experience with Kubernetes container orchestration? 5️⃣  Have you worked with some of the recent Large Language Models (e.g., OpenAI, LLaMA)? #LLM #ArtificialIntelligence #Quantization

21

3 Comments

Went overboard building agents at the AGI-House hackathon this weekend in SF. The idea was to build a personal AI agent which acts as your executive assistant, has access to your calendar, can chat with your contacts on Telegram & Slack, and can book an Uber pickup for you. Didn't they say, "the next big thing will start out looking like a toy"? Hacked away a crewAI + Open Interpreter demo in a few hours with Rahul Chhabra, re-realized how broken the agent build/flow engineering toolkit is. Still! Played around with OpenInterpreter 01 and experimented with a lot of verbal prompting to build/train new skillsets. Built out a tool for the agent where natural language instructions to OpenInterpreter were fed, and those got converted to AppleScript commands on the fly. Cherry on top, bagged a podium finish at the hackathon! 🥉 Agents are the future. 🚀 Github in comments.

47

7 Comments

Key Takeaways from Retool’s "The State of AI 2024" Report Retool's "State of AI 2024" report offers a comprehensive look at the current AI landscape, based on insights from ~750 tech professionals across various roles and industries. The survey explores AI sentiment, adoption rates, use cases, development challenges, and the evolving AI tech stack, providing a pragmatic view of how businesses are really building with AI and what's delivering ROI. This summary highlights the most important findings and trends from the report: 1. AI sentiment remains measured, with most considering it slightly overrated. 2. 55% of companies have built AI-powered chatbots, but there's some fatigue around this use case. 3. Top internal AI use cases: writing code/queries (42%), knowledge base Q&A (36%), and support chatbots (34%). 4. OpenAI models dominate, used by 77% of AI users. GPT-4 leads at 45%, followed by GPT-3.5 at 25%. 5. Vector databases are on the rise, with 64% of respondents using them (up from 20% in 2023). 6. Pain points in AI development: model output accuracy (39%), lack of technical expertise (38%), and data access/security (34%). 7. 60% of respondents believe AGI is likely within the next decade, with 2.7% thinking it already exists. 8. Developers need more than just better AI tech – they require executive buy-in, adequate staffing/funding, and robust data governance for successful AI projects. The report suggests a maturing AI landscape, with businesses refining their approach to AI implementation and seeking real ROI from their investments. Check full report here: https://lnkd.in/eehX5xkx #AITrends #GenerativeAI #Innovation

3

"Ai is two markets, training, and inference. Inference is going to be 100 times bigger than training." On his All In Podcast, major investor Chamath Palihapitiya, known for his investments in AMD and Groq, discussed several key topics related to Ai development. One of which was Meta's "overspending" on the training of its Llama 3 and the growing impact of open-source developments in the LLM marketplace- which was stated as "catching up to that of OpenAI" Any thoughts on the potential impact of open-source advancements in the AI industry?

1

Hard Truth that many people in tech and specifically Venture Capitalists wont like to hear: 1. GPT models are basically stuck and having stunted growth since ~oct 2022, any groundbreaking development is either by coupling some hardcore backend engineering (related to clever non AGI algorithms or engineering of scale) OR by bluffing the actual demo like gemini/devin/video generator of chatgpt etc. 2. Backend and systems engineering continue to be the CORE of innovation for exponential impact. Hardware engineering contiues to grow at the compounded growth rate that it has been last 20 odd years. If something seems too good to be true it probably is. 3. There has been 0 development in GPT models being able to solve tough problems. Best it can do is scrape the data to check for problems solution and then plagiarize it. Most developed nations are going to ban this.

9

4 Comments

I'll share some thoughts on this as there is some confusion. "The federal government has spent $22 billion in recent years on technology from the top 100 national-security startups, a paltry portion of overall contract spending and less than half of what venture capitalists have invested in those same companies." While VCs are obviously concerned about revenue, the reporter ignores the most important issue. Valuations in young private companies are something like one part guesswork, one part aspiration, two parts salesmanship, and three parts milestones, with a fair amount of fraud mixed in particularly in bubbles, but even in more mature growth companies a 10x multiple is conservative. That $22 billion in DOD contracts probably represents something like $300-400 billion in valuations claimed by the ventures and VCs, if not more. Even if the realized valuation is less than that for DOD contractor ventures, it would still represent an attractive ROI over a decade. If total invested is $50 billion and the realized value for VCs is $150 billion over 10 years, it's still 3x, and within that there would be likely be several 10X ++. That's not much different than the rest of the markets VCs invest in. My guess is it's more on track to be 5x or more for the entire portfolio, which is much better than most investments, though needs to be due to the risk involved. The more troubling statistic is that only 1% of total DOD contracts are going to VC-backed startups. That is a terrible failing grade for the DOD with regard to favoritism for incumbents, and explains most of the problem about unhealthy relationships and insufficient innovation. Given the size of the DOD budget, it may not be quite as bad as it seems as they can and should act as their own investors as a proactive customer, and they do some of that, but the reality is that the bureaucracy is so expensive to navigate that it usually requires significant capital. In fact, the most successful DOD startups in recent years were founded or co-founded by billionaires, which pretty well tells the entire story. Quote: “Startups are either purposely being kept out, or the DOD has just not figured out how to take advantage of them,” said Tara Murphy Dougherty, chief executive of defense software company Govini. Venture-backed companies have consistently received around 1% of total Defense Department award spending annually going back through 2018, despite throngs of new companies joining the defense-tech field, according to Govini data. https://lnkd.in/gK7emEqT

2

3 Comments

"It will NEVER happen ❌" Yann LeCun, chief AI scientist at the social media giant Meta, has explained that the Large Language Models (LLMs) that power generative AI products will never be able to reach the ability to reason and plan like humans can. 🤖 Instead of focusing on forming the AI abilities of tools such as ChatGPT to surpass the reasoning capabilities of humans, he is focusing on the radical idea of creating “superintelligence” in machines. 🧠 He argued against relying on advancing ‘intrinsically unsafe’ LLMs as they can only answer prompts accurately if they have been fed the right data. It begs the importance of data cleansing before building your LLMs. 📊 If you would like to find out more about how I and SR2 are supporting businesses through the minefield of information, please feel free to reach out. 📩 We offer expert guidance on navigating the complexities of AI, data analysis, and digital transformation, ensuring your business stays ahead in the ever evolving technology landscape. You can check out the full article here: https://archive.ph/kYMDt #AI #ArtificialIntelligence #MachineLearning #AIGeneration #TechInnovation #FutureOfAI #Superintelligence #TechNews #YannLeCun #Meta #LLMs #AIResearch #AIFuture #TechTrends #Innovation #ChatGPT #TechInsights #AIRevolution #DigitalTransformation #NextGenTech

11

OpenAI just released an updated model, GPTo. There is a lot of meat here to chew through, my hot take: 💰 Most of what is announced amounts to cost savings in inference, first of all for OpenAI. The speedups in all kinds of use cases are to an extent a side product of this. 📉 The race to the bottom on price is well and truly on: much of the pricing announced seems aimed at not losing customers to other models which easily offer factors of up to 10x improvement in cost. 📈 Scaling remains very real: raw power trumps carefully chosen niche models. This hits many startups and scaleups such as pi.ai and hume.ai to name a few. 🐣 All of this still feels like evolution, although at the same breakneck pace we have seen for the last 1.5 years. 🧓 Alexa and Siri are in serious trouble - they feel positively ancient now! What do you think? The announcement: https://lnkd.in/emucqsNT #AI #GPTo #GPT #ChatGPT #Impact #AIEvolution

25

4 Comments