🤷 Nico Martin’s Post

My talk "From ML to LLM: on-Device AI in the Browser" from UphillConf is now on youtube. If you want to see an emotional rollercoaster, just skip to 6:30: 😊😳😥🤔😥☹️🤷💡😄🎉👏 #WebAI

My very first conference talk was a short lightning talk at Front Conference Zurich in 2018. And this year I am super happy to be back on that stage again. You want to join me? the call for paper is open until June 16th 👇

Hey everyone, get ready for another cool #WebAI sideproject 🎉 In the last weeks I spent a lot of time building a #RAG solution directly in the browser without dependencies to any AI cloud provider. And today I am proud to share it with you: https://pdf.nico.dev/ RAG stands for Retrieval Augmented Generation. And it solves a crutial problem in working with Large Language Models: Enriching an LLM with your own data. LLMs have a lot of "built-in" knowledge. This means that because there was a lot of information about Albert Einstein in the training data, the LLM can answer a question about Albert Einstein quite well. But as soon as we ask for information on internal company knowledge or personal matters, the LLM is at a loss. Die idea behind RAG is that we first search a large data set for specific information that answer the question and then send it along with the propmpt as context for the question. With #AskMyPDF I built a #Webapplication that does exactly that with PDFs. So the idea is that you can process any PDF locally, the content is then parsed line by line, converted into a so-called *embedding* and saved in a #vectorDB. If a question is now asked about the document, the question is also converted into an embedding and the lines from the vectorDB whose vectors are most similar are sent to the LLM together with the question. The LLM can then answer the question based on the sources we provide. All of that is happening directly in the browser. It uses the all-MiniLM-L6-v2 model from Hugging Face with #TransformersJS, a vectorDB that I wrote myself and then the #Gemma2B from Google DeepMind to generate the answer. No content will leave your device at any point. Ask my PDF is publicly available on GitHub and under an open source license. Just like most of my projects 😊 https://lnkd.in/eTFmUkKK What do you think? Will WebAI be the future?

I have some exciting news 🎉! Back in 2021 I was extremely happy to be able to join the #GoogleDeveloperExperts program for #WebTechnologies. In the last months I spent a lot of time playing around with usecases for #AI in the browser.. ..And now I am proud to announce that in addition to the web, I am also part of the developer experts program in the AI/ML category. More precisely, in the new #onDevice AI/ML subcategory. The perfect overlap between #PWA and AI! This means I am now part of a group of extremely talented people that try to push the boundaries of what is possible with AI directly on our devices. Which means we can build apps that respect #privacy by default, run #offline and are considerably cheaper to operate. At the moment we are just at the beginning and I look forward to seeing what the future holds!

A few days ago, the first AI edition of the UphillConf took place on Bern's local "mountain", the Gurten. And I was extremely happy to be a part of it! Happy, but also a bit insecure due to the Imposter Syndrome. In addition to familiar faces from the AI world such as Elle O'Brien and Joshua Starmer PhD, who have been doing valuable work in AI teaching for a long time, the Hugging Face'rs Leandro von Werra and Lewis Tunstall who were able to talk a lot about the current development of models, and many other well-known names from the AI world, there was me - A self-thought front-end developer. And what is my super power? I build stuff. A lot of stuff 😄! Therefore I wanted to speak about the things I built and the learnings I had. From simple #TensorflowJS face-recognition models that are extremely performant using #WebGPU to my own "Im Endeffeckt" speech recognition model (you will get the reference going through the slides 😉 - https://lnkd.in/ehWh_Pue ) to LLMs like Google Gemma 2B that run directly in the Browser. Which also allowed me to build a small #RAG tool that uses sentence transformer models for semantic similaritiy search with #TransformersJS completely in the Browser! But more on that in the near future 😊. #WebAI is here to stay! My talk was certainly not the one with the most real world insights and the most complex technical background, but I received a lot of positive feedback. As in all my talks, I have always tried to awaken in developers the joy of playing around with new technologies. And according to the feedback I've received, I've succeeded in doing that. Which makes me super happy! Finally, I would like to emphasize the great organization. Organizing a developer conference in the middle of Switzerland is not easy. And Franziska Ackermann-Rubin, Samira Bakari, Matthias Junker, Edi Spring, Wayne Maurer and the whole organization team exceeded all my expectations. We had an incredible speakers trip on wednesday, it has been the most beautiful conference venue I have ever seen and the conference day itself went exactly as planned. Of course also thanks to Chanel G., who did an amazing job as MC! All in all a pretty amazing conference experience. 10/10. Will go again 😊

#WebNN becomes reality 🎉 Last friday at UphillConf I only briefly touched on the WebNN API, as it still seemed to be a thing of the future. Today at #MSBuild, Microsoft announced that it is already in the current developer preview of the Edge browser. This means that frameworks like TensorflowJS or TransformersJS no longer have to write adapters for CPU, wasm or #WebGPU and we developers no longer have to decide which technology our Webapp should use to calculate Neural Networks. WebNN will decide which is the most performant way for the device in use. This is particularly important because more and more devices are being shipped with #NPU, a processor that has been specially developed for processing AI tasks and is therefore even faster than the GPU! Having a central API that will take the most performant way to process our AI operations is a huge step for #WebAI 🌟

Guten Morgen UphillConf. Ich freue mich auf viel AI-Insights und natürlich auch on-device #WebAI😊

Schon mal was von #RAG gehört? Eines der grössten Probleme von #LargeLanguageModels ist, dass sie sehr schlecht darin sind zuzugeben, dass sie etwas nicht wissen. Und dann einfach halluzinieren. Die einfachste Art, dem entgegen zu wirken, ist, dass man die Informationen direkt mit in den Prompt gibt: "Hier ist der Wikipedia-Eintrag von Albert Einstein. [...] In welchen Städten hat Albert Einstein gewohnt?" Das führt zu sehr grossen (und teuren) Prompts. Ausserdem wird die Antwort unter Umständen durch unwichtige Informationen verwässert. Die Lösung? Retrieval Augmented Generation (RAG) Anstatt dass wir den gesammten Kontext in das Prompt übergeben, werden erst Ausschnitte gesucht, die zur Frage "In welchen Städten hat Albert Einstein gewohnt?" passen. Der beste Weg für diese Suche ist eine Vektor-Datenbank, wo die einzelnen EInträge als mehrdimansionale Vektoren gespeichert werden. Diese Vektoren werden dann mit dem Vektor des Such-Queries verglichen. Da ich keine JavaScript-Lösung gefunden habe, die komplett im Browser läuft und meinen Anforderungen entspricht, habe ich meine eigene Implementierung gebaut, welche #TransformersJS und das "all-MiniLM-L6-v2" model verwendet um die Texte in Vektoren zu verwandeln: https://lnkd.in/eyRduWW2 Ausserdem ist es eine super Möglichkeit mit der experimentellen Implementierung der #WebGPU API in TransformersJS herumzuspielen. Mehrere 1'000 Einträge können damit innerhalb von Sekunden eingelesen werden. Für die gesammten AGBs von Twitter komme ich zum Beispiel auf gerade mal 0.8 Sekunden😳. Da kommen mir direkt ein paar sehr spannende Usecases für #WebAI und #WebLLM in den Sinn, welche ich noch weiterverfolgen werde😊