News

AWS Open Sources Neo-AI for Deploying Machine Learning Models

• By David Ramel
• 02/06/2019

Amazon Web Services Inc. (AWS) has open sourced its project for optimizing machine learning models to run better in the cloud and on resource-constrained network edge devices.

The project donated to the community stems from new functionality called Neo that AWS introduced last fall as part of its Amazon SageMaker service, used to train machine learning models once and then deploy them to various locations.

In announcing Neo for SageMaker, AWS described it as a "deep learning model compiler [that] lets customers train models once, and run them anywhere with up to 2X improvement in performance."

AWS said the new Neo-AI offering helps developers and users tune machine learning models to optimize them for different platforms, hardware and software configurations, a manual process further complicated by the lack of compute and storage resources typically found on edge devices to which models are often deployed.

"Neo-AI eliminates the time and effort needed to tune machine learning models for deployment on multiple platforms by automatically optimizing TensorFlow, MXNet, PyTorch, ONNX, and XGBoost models to perform at up to twice the speed of the original model with no loss in accuracy," AWS said in a Jan. 23 blog post.

"Additionally, it converts models into an efficient common format to eliminate software compatibility problems," AWS said. "On the target platform, a compact runtime uses a small fraction of the resources that a framework would typically consume. By making optimization easier, Neo-AI allows sophisticated models to run on resource-constrained devices, where they can unlock innovation in areas such as autonomous vehicles, home security and anomaly detection."

Currently Neo-AI supports platforms from Intel, NVIDIA, and ARM, while support for Xilinx, Cadence and Qualcomm is said to be coming soon. AWS said its own dev team and those from the above organizations will provide community contributions to help guide the project. One main component of the Neo-AI, a runtime, is reportedly currently deployed on devices from many vendors including ADLINK, Lenovo, Leopard Imaging, Panasonic and others.

The source code for Neo-AI is available on GitHub.