Jesse Robbins

I like this idea of "technical taste". It gets into where software engineering is as much creative art as it is science. Four great takeaways (IMHO) for software devs in this era in particular: 1. Aspiring engineers should cultivate a sense of curiosity, experiment with different tools and technologies, and embrace a mindset of continuous learning. 2. AI has the potential to streamline processes and enhance productivity for engineers, but it may also lead to disruptions in traditional software development workflows. 3. Developing technical taste and judgment is essential for making informed decisions about which technologies and approaches to pursue. 4. Collaboration and open-mindedness are key to leveraging the full potential of AI and staying ahead of technological advancements. I also really liked these two quotes from Sam Schillace: 1. "The right time to do something is when you have that feeling in the pit of your stomach that's like, 'oh, this is a great idea and it's going to suck to build because nothing's ready yet.'" 2. "Technical taste is like, 'how well have you consolidated that set of experiences and heuristics into judgment that you can apply accurately when you see new things?'" 3. "It may be the case that very small teams can do very large projects, or like we were talking about before, it may be the case we're just going to get really ambitious about what we try to do with the same size of teams, which is kind of where I would put my money." Great job on continuing to put out great episodes, Brett Berson and team First Round Capital! #ai #engineering #softwaredevelopment

5

💡 Why do we need so many evaluation tools for LLMs? As engineers, we build 'production-ready' LLM products using these metrics. But what happens next? How do we maintain control and ensure reliability? At Feedback Intelligence, we’ve crafted a cookbook to keep your LLMs reliable and aligned with user expectations. 🍲 📖 Give it a read and let’s chat!

16

There is an obvious need for standardized metrics across the decentralized computing marketplace landscape, which is why Sami Kassab wrote a killer blog this week proposing the use of Floating Point Operations per Second (FLOPS) as that standardized metric. In yesterday’s Big Brain Breakdown Outpost Strategies, we expand on Sami’s original post, and add some additional context of our own. Here are the key takeaways from yesterday's breakdown: > The decentralized computing marketplace sector lacks a standardized metric that measures a marketplaces total computing capacity, making it difficult for investors to accurately assess the value of these projects. > Sami proposes using FLOPS (Floating Point Operations per Second) as this standardized metric. FLOPS is widely recognized in high-performance computing and can provide a clear, hardware-agnostic measure of computational capacity. > FLOPS can accurately represent the computational capacity of diverse hardware, including GPUs, CPUs, and specialized accelerators like TPUs and FPGAs. This standard helps users assess the feasibility and cost of tasks like AI and ML workloads in real-time. > 6079’s Proof of Inference Protocol (PoIP) is one solution that can be used to verify and record FLOPS onchain, ensuring transparent and accurate reporting of computational resources through cryptographic proofs and game-theoretic mechanisms. > FLOPS should be complemented with additional metrics such as benchmark scores for AI/ML workloads, memory bandwidth, and energy efficiency to provide a complete picture of computational resources. This combination ensures a robust and sustainable growth of decentralized compute marketplaces. > The adoption of standardized metrics like FLOPS benefits users by providing clear insights into computational capabilities, aids investors in making informed decisions on the value of a project’s marketplace, and supports legitimate projects by highlighting their true value, thereby weeding out low-value, vaporware projects. Of course, we go into much more depth in the full breakdown. If you thought that was interesting, go ahead and give the full piece a look and do not forget to read Sami's original piece. He is an excellent writer! https://t.co/BtVuG82fcB https://t.co/ryQf2e8Zuu

10

*Developers*...JACoB ("Just Another Coding Bot") is officially open-source and ready for you! Kevin Leneway and Chris Pirich have been incubating this project at PSL for months and we're super excited to formally launch and see what the world thinks. JACoB is a coding agent that automates repetitive programming tasks to boost developer productivity. JACoB is especially good at handling tasks like turning designs to code via Figma integration, refining and improving GitHub issues, and conducting code reviews. JACoB stands apart from other solutions by offering the freedom of an open source tool and the quality of closed platforms like GitHub Copilot. (Check out JACoB's Elo score relative to other tools in the Product Hunt link below!) Please give it a try, share it with friends, and vote or drop us a comment in Product Hunt! https://lnkd.in/gXAEewh3 Pioneer Square Labs

40

1 Comment

#ObsessivelyEnthusiasticInvestor #UnreasonablyEnthusiasticInvestor #IrrationallyEnthusiasticInvestor at REFASHIOND Ventures: The Industrial Transformation Fund. We look for the people we believe are doing the best work in Industrial Transformation & Supply Chain Technology; Attempting the impossible. Then we go all-in on them. With obsessive and unreasonable enthusiasm. Join us. #IndustrialTransformation #SupplyChain #Innovation #Technology #Startups #VentureCapital #DemandTheImpossible #UnreasonableEnthusiasm #ObsessiveEnthusiasm #IrrationalEnthusiasm 

6

This is a great list, and a good example of fractal complexity inherent in seemingly "solved" problems. But: most frameworks handle the majority of these out of the box for you — except for SSO / SAML, which should probably not be a pre-launch concern. More broadly: software is about making decisions and trade-offs with varying degrees of confidence. Sometimes those decisions are very clearly "build vs. buy"-shaped; sometimes they're more esoteric. You owe it to yourself as an engineer and operator to make those kinds of decisions not because you think "well, this is what everyone says you gotta do", but because you've weighed the pros and cons and come to a conclusion, even if you don't know everything about the problem space.

1

Taking LLMs to production is hard and interoperability is becoming critical. Portkey is an open-source project that acts as a gateway to connect 100+ open & closed-source models. GitHub: https://lnkd.in/dTZP4iFa Project Owners: Ayush Garg & Vrushank Vyas Share Your Open-Source Project: if you want us to highlight it, please fill out this quick form: https://lnkd.in/dRYM-Av8 Generative AI World MLOps World #llms #interoperability

25

2 Comments

Thanks for sharing these insights. What I am seeing is that orgs test an idea using a proprietary #LLM like #OpenAI, then put it into production with an open source LLM using a development platform like #Llamaindex. That way, private data remains private.

1

If you're using LLMs in production you need to be running systematic evals. Here's a quick tutorial on how to get started using DeepEval, an open-source eval framework, to run evals in just minutes. https://lnkd.in/gvEifRxT

3

Hallucinations in LLMs are inevitable, no seriously! Josh Fritzsche and I have worked hard at Voze to find a way to tune, prompt and even plead 🙏 with LLMs to prevent them from the occasional 🍄 hallucination. It seemed like the better we have done the harder the edge cases have been to solve and eventually we realized that they tend to frequently happen when LLMs predict or make anticipations on "nothing". I remember at one point nearly going insane with frustration because when LLMs had a hallucination, they were EPIC! 😱 Then we pushed through it and benefited for the effort! How did we solve this common problem? We didn't (Not completely), we got better at identifying the issues as they happened. Once we identified in the data we were able to start looking for patterns. The same catalysts that cause it to happen once if left unchanged caused it over and over again. Our solution is astoundingly simple. Once you know that it's likely to happen..... let it! Accept it will 🍄, but also be preemptive in not focusing on it and use alternative data sources, generators or providers. Change the conditions that cause it but continue to monitor it as it happens. As Dan Caffee is fond of saying to me on a regular bases. "Dusty we need the system to be adaptive and provide mechanisms to learn from it's past." Well he is right! All we needed was data and the people that contribute to it's quality. Huge props 👏 to strong support from Kathy and Janelles audit teams. We were armed with a knowledge and a desire to learn from past mistakes. Armed with confidence that we could correct these problems we could focus on the data to prevent their impact going forward. When doing millions of LLM (Generative AI) calls regularly Josh and I have learned that hallucinations will happen. We also know exactly how we will handle and prevent it going forward! Failure is not doing anything and accepting defeat.

4

2 Comments

Very practical advice for Financial Services professionals thinking of embracing AI 1. Start small. Identify opportunities to deploy GenAI where the bar to entry is lower. 2. Find a champion. Tap an influential colleague who is positive about GenAI capabilities, and can advocate for the effort across the organization. 3. Define the scope. Clearly state the project’s range and goals, and provide well-defined, measurable business outcomes. 4. Make it transparent. Ensure everyone understands how the solution should work - how it receives inputs, combines them, and develops results. (Sufficient transparency is essential in the event outputs need to be explained to regulators.) 5. Keep security paramount. Implement solutions with proper safeguards to ease leadership concerns and demonstrate to regulatory authorities a secure, responsible approach. 6 Make results clear. Root design in explainable results, and ensure full transparency of how models are used and how data is accessed. Bring the team along. Train internal partners as the project progresses to accelerate adoption and reduce costs over the long run. Allan Francis Beechinor Christopher Harrison Roman R. Bahjat Safadi Rami Albatal Manish Khadka Pete Rivett

3

As we look ahead to 2025, it's becoming clear that the future of enterprise AI lies in compound systems, not monolithic models. Compound AI systems combine multiple models that work together iteratively, coordinating and leveraging external tools to evaluate, refine, and improve their own results. 🔄 The benefits are clear: ☑ Better performance than individual models ☑ Faster and less expensive iteration  ☑ Granular access controls on sensitive data  ☑ More reliably ensure that models behave as intended ☑ Flexible performance-cost optimization Early examples like AlphaCode 2, LLM Debate, and CoT-SC are already showing impressive results. 📈 As we move towards 2025, compound AI systems will become increasingly critical for enterprises looking to deploy AI at scale reliably, securely, and cost-effectively. 🎯 Check out my latest B2BaCEO newsletter for a detailed look at where AI is headed and what it means for founders: https://lnkd.in/gRrzX9nq h/t to Berkeley Artificial Intelligence Research and my colleagues Jaya G. Andrew Han Courtney Fiske for their research help

65

6 Comments

OpenAI announced GPT-not-5 yesterday. This release shows amazing capabilities in speed and performance, which have huge potential in systems that need low latency (i.e. a caller doesn't want to wait 30 seconds for a virtual agent to respond). But it does seem to indicate that the advancement of GPT models have reached an asymptote or a flattening of the curve of advancement in capabilities. More than ever, this shows that the real opportunity is to use this functionality to build better products on top of this platform. The most natural example: making Siri and Alexa actually useful. But, it doesn't end there. The advances in speed and voice capability open up a world of possibilities to use voice as a part (not all) of your day to day interface to compute.

16

3 Comments

101 real-world gen AI use cases featured at Google Cloud Next ’24 by @google snapshot of how 101 of these industry leaders are putting AI into production today, creating real-world profitable use cases! #rafAIgual #GenAI #usecases #enterprise #adoption

2

Earn royalties for automating troubleshooting steps? See an AI Assistant use your steps to respond to alerts and questions? If you can be in San Francisco on Sat June 22nd, sign up for SRE Day and use discount code KYLE20 so we can provision you a dev environment for our workshop. Link in the comments. I'm also taking volunteers for anyone who wants to race Eager Edgar, one of our AI Assistants, troubleshooting popular Kubernetes-based apps... #sre #devops #platformengineering #whatdoirunwhen

76

10 Comments

There's no surprise that these three key attributes can help you get more from LLMs. It's also ironic that these same three tips are helpful in leader's ability to get the most from their team. For example... 1. Be polite. This reflects a broader truth in leadership: respectful communication fosters a positive work environment and enhances team performance. Leaders should strive to be courteous and considerate, recognizing that tone and approach significantly impact team dynamics and outcomes. Creating a respectful and supportive atmosphere encourages team members to share creative ideas without fear of criticism. 2. Set Context. Leaders should equip their teams with the necessary context and information to perform tasks effectively. Contextual clarity minimizes misunderstandings and empowers teams to deliver more accurate and meaningful results. By clearly defining the problem space and sharing relevant background information, leaders can inspire innovative solutions from their teams. 3. Clearly outline a role. In leadership, clearly defined roles and responsibilities are crucial. By clarifying each team member's role, leaders can ensure that everyone understands their contribution, fostering accountability and enhancing overall team performance. Encouraging team members to take ownership of specific aspects of a project can drive focused and innovative contributions. Just as LLMs require thoughtful interaction to yield the best results, so too do our teams benefit from mindful, strategic guidance. Let’s lead with clarity, context, and respect, and watch our teams—and their innovations—thrive. #Leadership #Teamwork #AI #LLMs #Communication #Innovation #ContextualClarity https://lnkd.in/eUNKMG8P.

Good effort by Richard Waters in cutting through the Big Tech cloud/GenAI hype-storm. He makes some critical points, including that very little of the Q1 revenue had anything to do with GenAI, and that all the Big Techs have committed to spending obscene amounts of capital. Another point most forget is the change in depreciation -- extending the life of chips for accounting purposes added $billions in profits. Let's take Microsoft, for example, as they reported the most revenue from GenAI products. How much of that revenue was sourced from MSFT strategic investments like OpenAI whose CEO has said he's willing to burn $50 billion a year chasing AGI? OpenAI alone might account for a quarter of Azure's AI revenue -- all of LLM chatbot investments perhaps half. Should cloud services spend from LLM investments be recorded as revenue for Microsoft? Probably not. It's MSFT's capital investment being recirculated back as cloud revenue. What's the margin on these LLM investments? Probably negative for everyone but chip makers and cloud vendors, unless factoring in capex, in which case I suspect it's still a loss for cloud vendors. The underlying problem is consumer LLMs are not only very costly--probably a deep loss even before copyright compensation, and use an insane amount of power and water, but are also inherently insecure and inaccurate due to the nature of the technology. Moreover, they are mostly unnecessary. Precision data management systems like the patented core of our KOS are far more efficient, and frankly much more intelligent from an enterprise perspective. 1) No copyright infringement or related liability or repetitional risk, not to mention risking the entire product investment if SCOTUS rules against LLMs. 2) Much less risk and much greater accuracy. We don't use language models except for GenAI (compartmentalized) -- not for most of the personalized learning, not for intel, not for risk management and prevention, and not for most of the productivity. 3) We reduce financial and environmental costs over LLMs by about 90% off the top due mostly to focus on high quality data vs boiling the entire human knowledge base. A glaring omission from the discussion is the assumption (still) that Big Techs will dominate AI, which is not necessarily the case at all. That wrongly presumes LLMs are the best method for customers when in fact LLMs are the best for Big Techs (in the near term, but could be disastrous in the mid term). https://lnkd.in/gyAag5_U

2