Avoid Silent FP16 Conversion in ONNX Runtime & CoreML

Have you ever spent hours fine‑tuning a neural network, only to find that the inference speed on your iPhone or laptop is a bit slower than you expected? You run a quick test, and the numbers look great on a desktop, but on a mobile device the model feels sluggish. What if the culprit isn’t the code, but a silent conversion happening behind the scenes? That’s the story of how ONNX Runtime and CoreML can quietly shift your model from 32‑bit floating point (FP32) to 16‑bit (FP16) precision, and why you should know about it.

What’s the Deal with FP16?

In the world of machine learning, precision matters. FP32 offers high numerical accuracy, but it also consumes more memory and computational power. FP16, on the other hand, halves the memory footprint and can double throughput on GPUs that support it. However, this speed‑boost comes with a trade‑off: some models lose a bit of accuracy when their weights and activations are rounded to 16 bits.

Both ONNX Runtime and Apple’s CoreML love FP16 because it makes models leaner and faster. The problem? They sometimes apply the conversion without you even noticing. That’s why it’s essential to understand when and how it happens.

Why Does the Silent Conversion Happen?

• Performance Optimizations: The runtimes automatically detect that your device has a fast FP16 path and switch to it to squeeze out every ounce of speed.
• Memory Constraints: On devices with limited RAM, converting to FP16 reduces the memory pressure, allowing larger models to fit.
• Default Settings: In many SDKs, FP16 is the default precision for inference because most modern hardware can handle it efficiently.

While this is great for speed, it can be a problem if you’re measuring accuracy or debugging numerical issues. The conversion is often silent – no warning, no log, just a subtle shift that can change the output slightly.

How to Spot the Silent Conversion

Don’t let your model hide in plain sight. Here are some quick ways to check if your ONNX or CoreML model is running in FP16:

ONNX Runtime

1. Enable profiling in the session options. Look for the “precision” flag in the output logs.
2. Use the SessionOptions.optimized_model_filepath to export a debug version that preserves FP32.
3. Run a small test where you compare outputs from float32 and float16 sessions manually.

CoreML

• When converting a model with coremltools, the convert function has a minimum_deployment_target and compute_units parameter. Setting compute_units=ComputeUnit.ALL forces the conversion to stay FP32.
• Inspect the generated .mlmodel file in Xcode’s model inspector; it will display the precision of each layer.
• Use the mlmodelc command line tool to query the model’s metadata for the precision field.

How to Control the Conversion

Want to keep your model in FP32 for accuracy tests? Or do you want to lock in FP16 for a production release? Here’s how to take the reins:

ONNX Runtime

• session_options.graph_optimization_level = GraphOptimizationLevel.ORT_ENABLE_ALL – This will keep FP32 unless you explicitly enable FP16.
• Use the providers=[('CPUExecutionProvider', {'precision': 'float32'})] setting to force CPU to use FP32.
• Wrap your model in a onnxruntime.InferenceSession with the enable_fp16 flag set to false.

CoreML

• When converting, specify float32=True in the convert call to avoid automatic FP16 conversion.
• Use coremltools.models.utils.convert_onnx_model with minimum_deployment_target=10.0 and compute_units=ComputeUnit.CPU_ONLY to force FP32.
• After conversion, use the mlmodelc tool to confirm the precision setting.

When Is FP16 the Right Choice?

Consider the following scenarios:

• Edge Devices: If you’re deploying to a mobile phone or embedded system, FP16 often gives you the performance boost you need.
• Large Models: Reducing memory usage can allow a model that would otherwise exceed device limits to run.
• Batch Inference: On GPUs, FP16 can double throughput for inference workloads.

But if you’re validating accuracy, training, or dealing with numerically sensitive tasks, sticking with FP32 is safer.

Wrap-Up: Keep an Eye on the Numbers

So next time you notice a sudden speed bump—or a tiny dip in accuracy—remember that ONNX Runtime or CoreML might have quietly nudged your model into FP16. By checking the logs, inspecting the model metadata, and using the right flags, you can control precision like a pro. Stay curious, test diligently, and keep those models running just the way you want them to.

Got questions about managing precision in your own projects? Drop a comment below, and let’s dive into the details together!