Presentation: Machine Learning Engineering - A New Yet Not So New Paradigm

ML Engineering is a relatively newly defined role at organizations which often refers to a specific subset within a broad spectrum of skills in the goal of either building ML driven products or enabling ML capabilities across products.   It is interesting that although it is a new role, we still leverage expertise from various parallel domains. For example: principles and tools in distributed and high performance computing are used in optimizing training and inference pipelines. Ideas and tools which are quite common place in scientific high performance computing domain like using vectors instructions, Cudann, profiling of GPUs, prefetching, compiler optimizations are finding its way into the industry. Similarly, best practices from the data intensive distributed computing space like leveraging spark for preprocessing. Micro services architectures and design principles are being leveraged to deploy atomic ML capabilities in a scalable and robust way.   Unlike traditional software engineering where a system/product is designed based on user requirements and heuristics in a deterministic way, experimentation and non deterministic nature of building Machine learning capabilities brings a very new set of challenges and opportunities. Infrastructure for rapid experimentation and friction free iteration between experimentation and deployment are two of the critical aspects. On the deployment front, divergence between tools/tech stack used in experimentation and deployment is a challenge. This is especially keen in terms of ondevice deployments of ML models. Compute, memory, space, binary size requirements for running ML models on device, coupled with difference in pace of iteration in ML frameworks like Tf/Pytorch as compared to device software makes things even more interesting. Lastly, testing of ML pipelines/models and validation of ML models also requires a new way of thinking around bringing A/B testing closer to experimentation cycle.