Here's a concise representation of your ML platform, courtesy of Laszlo Sragner.
I wrote over the weekend that any notebooks can be rewritten in a day to be production-ready (this went down as well as you'd expect); I also had an article regarding a DS using Angular and Terraform and why this is a bad idea. Lastly, my comment about "Market for Lemons" on the quality of ML consultancies was so successful Maria Vechtomova picked it up into a standalone article. Let's tie all this up with a review of DS skills and my favourite psychological concept, "Self Determination Theory". The idea is to stay motivated to do something; you need three things to be present at the same time: Mastery, Autonomy and Relatedness. For a DS, this translates into: Use skills that you frequently use to get better (Mastery) to resolve problems the way you want to (Autonomy) in long-term projects that you can feel yours, be proud of, and work with others (Relatedness). How does this relate to DSes doing Terraform? The DS/ML/AI space is incredibly complex, so much so that participants suffer from "Bounded Rationality". This means that no matter how much you try, you won't be able to comprehend it in its entirety. No surprise, given the top of the stack (most abstract), is essentially business analysis and business strategy (cue BI and "data stories"), and the bottom of the stack (most concrete) is pretty much an inch about hardware (Kubernetes, low-level YAML hacking and Terraform). How do you expect the same person to resolve this at the same time? If you are fighting BR, you need to forget something in order to remember the details of something else. "Shuffle it out of the cache" with computer speak. This instantly hurts Mastery, given you are context-switching all the time. The other failure mode is being responsible for rarely used technologies, which again counters Mastery. This is why trying to do Terraform on top of your BI and ML/AI (and some DE) work is a bad idea. You are just making your BR situation worse. Low Mastery also hurts Autonomy, as you will feel that you are unable to do what you want as you are not good enough, but you can't get better as you constantly need to shift out of these skills or use it so rarely that you don't have a chance to improve. This was recognised at the beginning of the MLOps era, and MLOps engineers were born to help out DSes. This was a good idea, but now the homogenous DS cohort is split into two different types of mindsets. And you still need to allocate each skill into the two roles. The decision was to make the DSes stick to high-level business issues and MLOps Engs to low-level technical ones. Immediately, running into the problem of how to coordinate between these two. There are two main concepts: Assembly Line and Abstraction. In "Assembly Line", DSes write POCs in notebooks and "throw them over the wall" to MLEs, who will essentially re-implement them, rediscovering all the decisions the DSes made during the POC phase. [continues on my blog, don't forget to subscribe]
