Abhinav Ramana

I'm thrilled to be speaking about "Apache XTable (Incubating): Interoperability across Open Table Formats" at The Fifth Elephant 2024 annual conference tomorrow at BIC, Bangalore organized by Hasgeek In this talk you will learn about: 1. What are Table Formats? Why do we need Inter-Operability ? 2. A deep dive on XTable and problems that can be solved using XTable. 3. A demo of XTable in action. 4. The roadmap for the OSS project. You can check out and give a star to the OSS repository for the project here: https://lnkd.in/g53VWyU4 🎟️ Get your tickets and join us in-person: http://has.gy/sSCX P.S First 5 people to buy a ticket to the conference using the above link get a 10% discount! So go grab your ticket fast! #Hasgeek #AI #TheFifthElephant #MachineLearning #DataScience #BigData

84

I have always tried to approach nearly everything, personal and professional, with a mindset of assuming good intentions. A starting point of believing that the other person is doing what they think is the best thing, the best they know how. This applies everywhere, whether it's with leadership or a peer. I ended up in a state this past winter where I lost this mindset. I found myself with a negative outlook in many areas, and it led to a few incidents at work where I was not my best self, and some poor choices in my personal life as well. I've learned a few things from that time. The first is that the longer you are in this headspace, the harder it is to pull yourself out. I'm fortunate in that it was my default state before, and I only spent a couple months in the weeds. Talking, both with my therapist and posting here, helped me a lot to remember my own values. The second is that the company you keep matters a lot. The alternative viewpoint is that you need to be suspicious of everyone's intentions. This is poisonous - it spreads to everyone around you because our brains are tuned for threat assessment. You start seeing threats in every action, and it prevents you from understanding and moving forwards. I've started being very cautious about getting into conversations where the tone is negative and suspicious. The third is that the damage to your reputation that happens from being in this place because of how your interactions go can be devastating and lasts far longer than the actual mental state does. You can have a negative interaction with someone and usually apologize and move on. If that spills over to a more public setting, say exiting a meeting with no explanation, people connect the dots and talk. Your clouded judgement prevents you from protecting your mental health in a productive and understandable way. Now, because I'm back where I need to be, I can easily be compassionate about people talking. I know that they are concerned, both for me and the role I take on at work. I know that when HR reaches out with resources, it's from a place of compassion and wanting to make sure I have the support I need. And I can look at policy decisions and understand better what drives them and how I can work with them, rather than assuming that it's adversarial. I can honestly say that I know I have done damage to my professional reputation. I'm dedicated to fixing that, even though I know it will take time to demonstrate the consistent behaviour. I can also honestly say to anyone concerned or curious, you can reach out directly to me for a 1 on 1 discussion (whereas earlier, this would not have been good for either of us).

81

7 Comments

Hot take: Building a business around open-source software is a lost battle. In last few months, we have seen how companies such as HashiCorp, Redis labs have faced the heat when they adopted business source licensing model. It resulted in their alternatives popping up within few days like OpenTofu, Dragonfly etc. There is no moat for such businesses because the underlying software is open-sourced which means new entrants can easily replicate what they are doing. They can sell it at a cheaper price or with a better ecosystem(if they are cloud vendors) or both. These businesses also face the dilemma of whether to merge a major improvement in the open-source or put it behind the SaaS offering. An alternative to this model is to build the business around consulting & training. The business keeps the software open sourced & provide consulting services to clients who want to use it in the most efficient manner. Pivotal has done a great job with Spring framework using this model. This way your revenue stream is not marginalized by new players easily & the core team can focus on improving the software without worrying whether it makes their core revenue model i.e. SaaS offering less attractive. Especially in the non-ZIRP era, companies are bound to reconsider where they are spending. Hence they will choose the option that burns the smallest hole in their pocket. #opensource

8

2 Comments

I met a founder who quit his cozy Google job & founded not 1 but 2 unicorns. He ignored nearly all lean startup advice. Then grew to $3M in 1 year: Arvind Jain was a Distinguished Engineer at Google. He made Google “faster than the blink of an eye”. Then he got bored— so he quit. He co-founded Rubrik, which grew to $500M in revenue and just filed to go public. It wasn’t enough. 5 years ago, he noticed a problem at Rubrik. As they hired more engineers, productivity slowed down. New hires couldn’t find the information they needed. So Arvind quit Rubrik and founded Glean to solve enterprise search. He did everything lean startup tells you not to do. - He didn’t focus on launching an MVP. - He ignored early market feedback. - He didn’t charge beta users anything— for 2 years. And it worked. He went from $0 to $3M in revenue the year he launched publicly. He tripled to about $9M the year after. And tripled every year since. Two months ago, Glean raised $200M at a $2B valuation. Most founders can’t do exactly what Arvind Jain did. He’s a proven founder who raised $15M with just a deck. But after spending 10 years building 2 unicorns, here’s his number 1 advice for early-stage founders: “Conviction and persistence." If you see a problem…don't think too much about who else is solving it or how big the market is. Just go and solve the problem and persist and believe in it. There's going to be ups and downs. There's going to be people telling you they wouldn’t pay for it. Don't worry about all of that. If you think that it's a large enough problem, and a lot of people face it, having that conviction and belief and staying on course is, I think, the most important thing.” #startups #venturecapital #founders Listen to the full podcast on The Product Market Fit Show

8

2 Comments

Observation: a lot of domains put up with ad hoc / limited programming models just to get ease of deployment and 0 serialization boilerplate. (Think: KSQL vs Kafka + normal code) You can have the best of both. Ease of deployment, etc, but with a rich, general-purpose language.

4

If you are creating an LLM based agent/system and you find yourself creating many small rules to align the model behavior with your expectations, you quite possibly need to question the fundamental context you have given the model. Defining a clear role and task are 90% of the battle. This is because they essentially set the baseline behavior profile for the LLM prompt. Always be ready to question whether your role or task definition are mis-directing the model. "The LLM, a polymorphic chorus, harmonizes beautifully with the right notes but falters on the wrong ones. — ChatGPT"

19

1 Comment

I was hot headed, typically obnoxious, and overtly arrogant when I started working at my first job. Chirag and Pravin dai will attest to that 😂.   I remember this one incident very clearly. Mahesh was my QA guy and we were building #MONyog, an agentless #MySQL monitoring tool. We missed a customer meeting. Just plain forgot. Obviously the boss' "blood (was) boiling" and his "head exploding" -- phrases we heard a lot those days! He rescheduled the meeting and stood over us while we took the call.   The problem the customer was facing was that every time MONyog started up his network usage shot up for the first few minutes and used up most of the available bandwidth. He showed us #Nagios graphs. Without thinking I asked him what the size of his error log was. He said it was around 1TB.   Every time MONyog booted up, it was reading 1TB of data using SFTP. How did I know? I had no idea. Boss looked at me, "How did you know?". All I said was: "I felt it". Sufficed to say that my boss' blood was no longer boiling over, and his ire was redirected at Sandhya who built the feature 🤣.   It's been over a decade since that incident, and I was reminded of it as I was reading Daniel Kahneman's #ThinkingFastAndSlow. His work, it shines a light on the dark recesses of my mind. It's incredible how the brain works, and we owe it to ourselves to take care of it, to nurture it. I had no idea, when I picked up the book, that Daniel would pass away just a day or two later. God rest his soul. Share your story. Show up.

23

9 Comments

Some relevant research here: The Paradox of Meritocracy in Organizations https://lnkd.in/gfF_tN-u Promoting meritocracy sounds great, but the demonstrable and easily explained impact is that it provides cover for bias.

7

Next week, Prakul Agarwal will cover how to set up and configure MongoDB as the foundation for your RAG application. Register: https://lnkd.in/giR3KnCj We'll also go over... ✔ How to implement best practices for data storage, retrieval, and augmentation. ✔ Integrating Phoenix to analyze and debug your RAG application’s performance. ✔ Conducting comprehensive evaluations and benchmarking using Arize’s LLM evaluation suite. ✔ Identifying and addressing common challenges in RAG application development and deployment A must-attend for those of you building RAG or wanting to improve performance!

10

Hot off the press just minutes ago, Confluent just announced Freight clusters in #KafkaSummit Bangalore. They’re a new innovative cluster type that lowers prices by up to 90% (!) 🔥 What is it and how does it work? It’s all about latency. Some workloads like: 🔷telemetry 🔷metrics 🔷logs don’t care whether the latency is 50ms, 500ms or 2s. But the same workloads typically have a TON of data. And when you need to replicate that data 3x and pipe it through the network… the network costs can eat you alive 💀 So what did #CFLT do? They decided to write the data straight to S3! Instead of having Kafka replicate it across brokers, they just write it directly to S3. S3 itself does the replication under the hood (for much cheaper) 💸 The catch? S3 is much slower. But as we mentioned, those workloads don’t care. Similar to a cargo ship, you’re fine letting the cargo (data) take longer to ship as long as the price is good. For the critical use cases - you have planes. ✈️ For everything else - you have Freight. 🛳️

601

36 Comments

"An API is an abstraction (not a program code – class or a function) that acts like a “gate” setup by a provider for a resource (sometimes more than one resource) behind the gate. Through this gate a consumer extracts information and manipulates the resource as per provider’s rules. Provider makes money per call to this gate. Set of rules agreed between provider and consumer OR published rules by consumer (anyone who is using the gate shall comply) is called "Contract" " Let me know what you think about this definition that I have come up with. Bas Dijkstra Wayne Roseberry Pramod Yadav ?

25

24 Comments

Celery is a distributed task queue for Python, managing async job execution across multiple workers Celery is ideal for handling vast amounts of background tasks efficiently. Advantages: - Distributed Nature: Easily scales across multiple nodes. - Support for Multiple Brokers: Works with RabbitMQ, Redis, etc., for managing task queues. - Robust Monitoring: Integrates with tools like Flower for real-time monitoring of tasks. Disadvantages: - Setup Complexity: Requires additional setup for message brokers and workers. - Overhead for Small Projects: May introduce unnecessary complexity for smaller, less-demanding applications. This snippet demonstrates defining a simple task to add two numbers using Celery, highlighting its ability to offload tasks asynchronously for improved application responsiveness and scalability.

5

Memcached Vs Reddis: Here is a radar chart comparing Memcached and Redis across several key categories: **Data Types and Structures:** - Redis (5): Supports a wide range of data structures. - Memcached (3): Primarily a simple key-value store. **Persistence:** - Redis (5): Offers data persistence options. - Memcached: Does not provide persistence. **Performance and Scalability:** - Redis (4) and Memcached (4): Both offer high performance. - Redis: More versatile. - Memcached: Simpler. **Advanced Features:** - Redis (5): Provides replication, transactions, and Lua scripting. - Memcached (2): Focuses on straightforward caching. **Use Cases:** - Redis (5): Suitable for complex use cases. - Memcached (3): Best for simple caching scenarios. **Community and Ecosystem:** - Redis (5): Vibrant community and broad ecosystem. - Memcached (3): Mature but less extensive community. --- Feel free to customize and expand on each section as needed! #Cloud #Memcached #Rediss #Caching

16

Hiring Manager rounds are the most undervalued ones. Many people take them lightly and give interviews without preparing for them. What do they end up getting up as a feedback: "Your examples were not strong and complex enough for the role, so we cannot proceed with this role but if you want we can proceed with <some lower> role". Hiring managers have their own expectations and goals they are evaluating you on. If you don't want to end up with some feedback like above, then give enough time for its preparation. Preparing for this round requires you to think about stories from your past experience and map them to the situations you navigated in those stories. You need to think about the stories that are complex enough for the role you are applying. If you need help preparing for such rounds or need a mock interview for this, connect with me via enginEBogie here: https://lnkd.in/dTqdUjwd Enginebogie has many other top-quality experts too, you can connect with them here: https://lnkd.in/g2hYi4UY

16

1 Comment

I have a hypothesis: US-India pair programming can ~2x the speed of execution of engineering teams. Imagine a high quality engineer in US who passes the baton to his/her counterpart in India - at the end of the day. The Indian counterpart then runs at the same developmental speed and takes over for the next 8-10 hours. They then pass the baton to their US counterpart. It could even be a senior-junior relationship - making sure instructions are clear and only one person is in the driving seat. Of course, I am using pair programming a bit loosely here but I could not think of a better framing. Diving into this thesis for the next few days. If you have worked in such a setup or have insights, I would love to chat and learn what you know.

15

5 Comments

The Godfather culture .. do you know what is it ?? that X person commented on your ADR / arch design .. now your course of design work will change .. that Y person reviewing your code / PR / MR .. now it will take a lot of time to merge your code .. Few X and Y have been Glamourised by others and then godfathers were created. Such culture will lead to a couple of challenges over a period of time .. Like, their (godfather) opinion/feedback is not evaluated properly as assumed it will be always better .. fresher will feel pressure when they get comments from them .. it will reduce the confidence of average employees when they get comments from them. So let's not make anyone godfather and let's not be the one. #culture #softwarecompany #architecturedesign #codereview #bestpractice #softwareengineer

18

🔥 PaliGemma - The new opensource multi modal model, which can be used out of the box for a few usecases, even for Indic Languages. 🔥 While everyone is talking about the multi modal capabilities of propitiatory models of Google & OpenAI, I feel, the largely unnoticed open source PaliGemma model by Google - built on PaLI-3 & Gemma-2b is going to be a game changer. Have run some initial tests on Telugu and Hindi, and it does a decent job out of the box. Gives us a good baseline to build upon atleast. Excited about how we can leverage the multi modality and what we can build for Indic Languages, especially because it is equipped with the extensive Gemma tokenizer. #GoogleIO #llms #multimodal #ai #gemma #huggingface

48

2 Comments

INDstocks by INDmoney saw almost 3x scale of concurrent users using the application through the last 3 days. As #India was getting prepared for its much awaited #generalElectionResults, the Engineering team at INDmoney embraced this challenge of getting ready for humungous scale. While many brokers had issues scaling for this traffic surge, primarily on June 3 (Monday), #INDstocks was quite resilient and gave all it's users highly reliable trading experience on all the three days (June 3 - 5). We have achieved this through a lot of best practices in #Engineering. All our Backends are independently scalable micro services, which gives us greater flexibility. We decoupled core components to a large extent, which allows seamless experience to our users on both Apps and website, even in the events of unprecedented scale. On days when volatility hits through the roof, an ultra-efficient square-off system which protects users from going into dangerous circuits and thereby losing money, is of paramount importance. Through our #INDprotect feature, our users' square-off orders were sent to exchanges in under 5 ms, which helped users reduce their losses significantly! At the market open, we saw close to 50% more price ticks from exchanges on June 4 and INDstocks users didn't have to face any latency in viewing the live prices on their web / mobile devices. Our #superFast #Pricer delivered all the ticks to users in under 1ms. #INDmoney #INDstocks #Election #Buzz #IndianEquities #Stock #Market #Economy Ashish Kashyap Nikhil Behl Mourya Venkat vasu choudhary Ashwin Sreekumar Sonu Singh Tejas Chhotani Deepak Gupta Kartikay Aggarwal Tushar Kumar Singh Deepak Pandey Nikhil Choudhary Sivajee Battina Pooja MD Harsh S. Aniket Pathak

217

8 Comments