OctoML turns machine learning models into software functions

We’re excited to bring Transform 2022 back in person on July 19 and pretty much July 20-28. Join AI and data leaders for insightful conversations and exciting networking opportunities. Register today


When it comes to betting machine learning (ML) models in the enterprise, optimization and acceleration – that is, deploying better and faster – are the keys to cost and time savings.

But according to the Seattle-based OctoML — Founded in 2019 to help enterprises optimize the deployment of machine learning models — there are many bottlenecks — including the dependencies between ML training frameworks, model types, and required hardware at each stage of the model lifecycle.

Today, OctoML announced a new model deployment platform that it claims is a “huge milestone for the AI ​​developer community” – enabling app developers and IT operations to “turn trained ML models into flexible, portable, reliable software features that integrate easily with their existing application stack and devops workflows.”

OctoML is built on the open source Apache TVM, a machine learning compiler framework for central processing units (CPUs), graphics processing units (GPUs), and machine learning accelerators. The makers of Apache TVM founded OctoML, including CEO Luis Ceze.

Machine learning alignment with devops

According to Ceze, the solution to the problem of bottlenecks in the implementation of ML models is that ML must align with standard software devops practices rather than the popular MLops† Organizations need a way to abstract complexity, remove dependencies, and automatically generate and maintain trained models that can be delivered as production-ready software features.

Models-as-function can run with high performance anywhere from cloud to edge, remaining stable and consistent even as hardware infrastructure changes, he explained. This devops-inclusive approach eliminates redundancy by unifying two parallel implementation flows: one for AI and the other for traditional software. It also maximizes the success of the investments already made in model creation and model operations.

Evolving relationship between data science and devops

Developers and IT operations haven’t had much of a chance to participate in the AI/ML implementation, despite being responsible for building and maintaining virtually every other element of the enterprise app stack, Ceze said.

“The secret to getting devops into the fold is the ability to treat models as flexible, portable, and reliable software features,” he said. “Today, the rigid interdependence between models, libraries, framework and hardware creates a complex and highly specialized implementation path that is closed to all highly skilled ML engineers.”

The proliferation of MLops platforms has led to a parallel development flow specific to AI deployment, he added: “Devops is a proven discipline for agile development and software deployment and there is an abundance of talent there, so devops models being able to deploy them as intelligent software functions unites these disconnected development paths and unlocks a lot of value.”

Constant shifts in the MLops space

Ceze explained that based on customers and the wider industry, he anticipates an ongoing shift from large, monolithic MLops platforms to more “best-of-breed” solutions that meet the needs of ML engineers, IT ops and app developers at every stage of the development cycle.

“AI is now mainstream and more companies want to build intelligence into apps and services, but the needs are very diverse,” he said. “Users will want to control their AI/ML implementations as they control the rest of their applications – their models, their infrastructure, their application stack – while guaranteeing SLAs for performance, cost and user experience while integrating effectively with their own applications. workflows.

As the company deploys more AI services, having granular control over deployment choices and the ability to accelerate across hardware targets will help them control costs and address inherent governance questions through the data and models deployed under keep their control.

“This reflects broader business technology adoption patterns that indicate a shift toward tools and technology that can be fed into the flow of users and provide greater control over their own APIs and workflows,” he said.

The mission of VentureBeat is a digital city square for technical decision makers to gain knowledge about transformative business technology and transactions. Learn more about membership.

Leave a Comment

Your email address will not be published.