TGIS build requires PyTorch nightly

[Xaenalt]

So, as of the most recent code drop, the TGIS build requires PyTorch nightly
https://github.com/IBM/text-generation-inference/blob/main/Dockerfile#L145

This is likely not going to work with security, how tightly integrated is torth._dynamo?

[njhill]

@Xaenalt we currently require the nightly builds to be able to use PyTorch 2 compile. However, we could probably change the code so that it will work with earlier PT versions provided you don't enable PT2_COMPILE.

P.S. I think you pasted the wrong link

[Xaenalt]

Oops, yeah updated that. So torch._dynamo is needed for PyTorch 2 compile? Without it, it's not possible to use any of PyTorch 2?

[njhill]

@Xaenalt pytorch 2 can be used but just not pytorch 2 compile.

[njhill]

@Xaenalt in the latest update, the nightly build is not required if you aren't using PyTorch 2 compile (it will still fail to start if PT2_COMPILE=true is set though).

[njhill]

@Xaenalt apologies, it turns out compile does work with the latest stable release of pytorch 2.0.1, we'll revert to that. I think it's 2.0.0 that we had a problem with and had switched to using nightlies prior to 2.0.1 being released.

[Xaenalt]

Excellent, we'll pull that in then!

[njhill]

Just to close the loop on this, we do rely on fixes/capabilities in the latest pytorch nightlies after all, for example compile only works in a very limited way in 2.0.1 and has had lots of updates since. We may be able to switch to 2.1.0 when it's released next week, but I expect that we may need to move to nighties again at some point after that given how fast it is changing.