Allowing the Neural Engine slowed it

laya-coreml runs the Laya typed decision models on Apple hardware through Core ML. Its BENCHMARKS.md has a small table that most people would skip, and it is the most useful part.

The test is the multilingual model answering one short question on an M3 Max. With Core ML limited to CPU+GPU, the median is 11.28 ms. Allow ALL compute units and the median is 78.04 ms. CPU+NE gives 81.34 ms and CPU only gives 82.54 ms.

The cause is in the compute plan. In the CPU+GPU plan, 1345 operations preferred the GPU. In the ALL and CPU+NE plans, 1318 operations were attributed to the CPU. The doc says it plainly: No operation in these inspected plans preferred the Neural Engine.

So letting Core ML choose freely did not unlock the Neural Engine here. It dropped the work onto the CPU and cost about 7 times the latency. The doc also holds back on the conclusion: these are anticipated MLComputePlan assignments, not a runtime hardware trace, and allowing the Neural Engine does not show that it accelerated the model.

The practical rule is to measure each compute unit setting on your own model before trusting the default. The repo's recommended setting is CPU+GPU.

Source: BENCHMARKS.md in mizorewww/laya-coreml.