Tract + Coprocessors?

[FL33TW00D]

Hi @kali,
First of all thank you for creating Tract, it is one of the most impressive single man software crusades out there.

I understand that currently you're using it internally on ARM64 CPU only. Have you given any thought to how we could add additional backends such that tract could utilise heterogeneous coprocessors with the CPU?

Tons of ARM64 boards are now shipping with either an integrated GPU or a custom NPU.

Many thanks,
Chris

[kali]

Hello! Thanks for the kind words!

I am aware of the GPU and NPU trend. We have some in-house development at Sonos for some specific boards, but we have no plans to opensource them as they are... well, specific.

But it does show on tract design: tract-core operator set (where your model lands after decluttering) has been reduced to be as small and simple as possible. So a couple of teams around are "forking" away from tract at that tract-core stage to translate the network into a non-tract runtime. That way you get the versatility of tract as a model loader and model cooker, and use a custom runtime.

I do not make formal commitment to keep this API and operator set stable, but I do pay attention to not break it un-necessarily. Some teams are using to target SGX enclaves, some others do stuff way beyond my understanding :) https://hackmd.io/mGwARMgvSeq2nGvQWLL2Ww

I think it make sense to do a partial mapping to run specific class of models on specific hardware (that what we are doing internally). Beyond this, there could be opportunities for general target API (android NN API ?).

[FL33TW00D]

Thanks for the insights!