[v2.3.1] Release Tracker

[atalman]

🐛 Describe the bug

This issue is for tracking cherry-picks to the release branch. Following is release branch for the 2.3.1 release.

Our plan from this point is roughly the following:

• Phase 1 (until 5/27): Cherry-pick post deadline (End of day 5PM PST)
• Phase 2 (after 5/27): Perform extended integration/stability/performance testing based on Release Candidate builds.

Only issues that have ‘cherry-picks’ in this tracker will be considered for the release.

Cherry-Pick Criteria

Phase 1 (until 5/27):

The Releng team relies on the cherry pick process to manage risk to release quality, i.e. by porting a small set of commit from trunk that are "must-have" into the release branch, we limit the change to the minimal to address pressing issues. Thus, not everything a developer land into the trunk will make it into the release. So, please consider the criteria below and follow the cherry picking process. Only low-risk changes may be cherry-picked from master:

1. No feature work allowed
2. Fixes to regressions against the most recent release (e.g. 2.3.0 for 2.3.1 release; see module: regression issue list)
3. Low risk critical fixes for: silent correctness, backwards compatibility, crashes, deadlocks, (large) memory leaks
4. Critical Fixes to new features being introduced in 2.3.0 release
5. Documentation improvements
6. Release branch specific changes (e.g. blocking ci fixes, change version identifiers)

Any other change requires special dispensation from the release managers (currently @atalman, @huydhn, @PaliC, @malfet). If this applies to your change please write "Special Dispensation" in the "Criteria Category:" template below and explain.

Phase 2 (after 5/27):

Note that changes here require us to rebuild a Release Candidate and restart extended testing (likely delaying the release). Therefore, the only accepted changes are Release-blocking critical fixes for: silent correctness, backwards compatibility, crashes, deadlocks, (large) memory leaks

Changes will likely require a discussion with the larger release team over VC or Slack.

Cherry-Pick Process

1. Ensure your PR has landed in master. This does not apply for release-branch specific changes (see Phase 1 criteria).

2. Create (but do not land) a PR against the release branch.

   # Find the hash of the commit you want to cherry pick
   # (for example, abcdef12345)
   git log
   
   git fetch origin release/2.3
   git checkout release/2.3
   git cherry-pick abcdef12345
   
   # Submit a PR based against 'release/2.3' either:
   # via the GitHub UI
   git push my-fork
   
   # via the GitHub CLI
   gh pr create --base release/2.3

3. Make a request below with the following format:

Link to landed trunk PR (if applicable):
* 

Link to release branch PR:
* 

Criteria Category:
* 

1. Someone from the release team will reply with approved / denied or ask for more information.
2. If approved, someone from the release team will merge your PR once the tests pass. Do not land the release branch PR yourself.

NOTE: Our normal tools (ghstack / ghimport, etc.) do not work on the release branch.

See HUD 2.3

Versions

2.3.1

[snadampal]

Link to main PR (if applicable):

• aarch64: cd: switch from libomp to libgomp builder#1803

Link to 2.3.1 release branch PR:

• aarch64: cd: switch from libomp to libgomp builder#1805

Criteria Category:
Fixes performance regression issues pytorch/builder#1774 and #124922

---

@atalman merged, pending validation with test described here: pytorch/builder#1774 (comment)

[saitcakmak]

Link to landed trunk PR (if applicable):

• [Doc] Update docstrings for torch/random.py #125265

Link to release branch PR:

• [Doc] Update docstrings for torch/random.py #125443

Criteria Category:

• Documentation improvements

---

@atalman merged

[soulitzer]

Link to landed trunk PR (if applicable):

• Do not import transformers when import torch._dynamo #124634

Link to release branch PR:

• Do not import transformers when import torch._dynamo (#124634) #125755

Criteria Category:

• low risk critical fix

---

@atalman merged

[eqy]

Link to landed trunk PR (if applicable):

• This is not a PR cherrypick but a revert commit: 51cf57c

Link to release branch PR:

• Revert "Include torch warn in each error in cudnn/Conv_v8.cpp (#120719)" #125790

Criteria Category:

• Low risk critical fix

---

@atalman merged

[atalman]

Link to landed trunk PR (if applicable):

• Restore DILL_AVAILABLE for backwards compat with torchdata #122616

Link to release branch PR:

• Restore DILL_AVAILABLE for backwards compat with torchdata #126094

Criteria Category:

• Critical Fix - torchdata compatibility

---

@huydhn merged

[atalman]

Link to landed trunk PR (if applicable):

• Add userbase library dir to windows dll search path #125684

Link to release branch PR:

• Add userbase library dir to windows dll search path #126095

Criteria Category:

• Critical fix for Windows: 2.3.0 wheel can not be imported if installed for a single user only #125109

---

@huydhn merged

[atalman]

Link to landed trunk PR (if applicable):

• [MPS] Fix abs for complex types #125662

Link to release branch PR:

• [MPS] Fix abs for complex types #126096

Criteria Category:

• Critical fix for silent correctness issue: Tensor.abs() gives incorrect results on Complex64 when using MPS #125135

---

@huydhn merged

[atalman]

Link to landed trunk PR (if applicable):

• Separate arm64 and amd64 docker builds #125617

Link to release branch PR:

• Separate arm64 and amd64 docker builds #126099

Criteria Category:

• Critical fix for: Broken Docker Image on dockerhub #125094

---

@huydhn merged

[atalman]

Link to landed trunk PR (if applicable):

• Publish PyTorch docs to pytorch/cpp repo #122895

Link to release branch PR:

• Publish PyTorch docs to pytorch/cpp repo #126102

Criteria Category:

• Fixes doc failure on Release branch

---

@huydhn merged

[atalman]

Link to landed trunk PR (if applicable):

• [MPS] Native nonzero implementation #125355

Link to release branch PR:

• [MPS] Native nonzero implementation #126100

Criteria Category:

• Critical Fixes aten::nonzero calls taking a huge amount of time when using MPS backend vs CPU #124850

---

@huydhn merged

[atalman]

Link to landed trunk PR (if applicable):

• Fix ref leak in dtype.to_complex()/to_real() #125154

Link to release branch PR:

• Fix ref leak in dtype.to_complex()/to_real() #126101

Criteria Category:

• Fixes Crash tensor.dtype.to_complex() crashes kernel after ~100 calls in ipython kernel #124868

---

@huydhn I update the cherry pick manually to remove changes from #124479, let's see if that works.
@atalman merged

[atalman]

Link to landed trunk PR (if applicable):

• [MPS] Fix large copy #124635

Link to release branch PR:

• [MPS] Fix large copy #126104

Criteria Category:

• Critical fix Data corrupted when transferring over 4GiB to MPS #124335

---

@huydhn merged

[atalman]

Link to landed trunk PR (if applicable):

• [dynamo] fix 3.11+ refleak #124238

Link to release branch PR:

• [dynamo] fix 3.11+ refleak for 2.3.1 release #126235

Criteria Category:

• Critical fix CUDA Memory leak w/ torch.compile in both stable and trunk #119607

---

@huydhn This cherry pick is complex from what I see because cherry picking it blindly won't work. It depends on #122146, that enables dynamo for 3.12. And that stack is not a small one. So, I think we need to rework the cherry pick #126107 manually if we want to go ahead with this (cc @williamwen42 @atalman @malfet @albanD)

@atalman Closed - #126107 . Merged - #126235

[atalman]

Link to landed trunk PR (if applicable):

• Fixes format utils executable #123407

Link to release branch PR:

• Fixes format utils executable #123482

Criteria Category:

• Fixes new feature

---

@huydhn merged

[atalman]

Link to landed trunk PR (if applicable):

• Lint: Update older-python test to 3.6 #125843

Link to release branch PR:

• Lint: Update older-python test to 3.6 #126108

Criteria Category:

• Required to fix Lint errors on release

---

@huydhn merged

[wanchaol]

Link to landed trunk PR (if applicable):

• [device_mesh] add back the private init backend option #124780

Link to release branch PR:

• [cherry-pick][device_mesh] add back the private init backend option (#124780) #126147

Criteria Category:

• Fixes to regressions against the 2.3.0 release

---

@atalman merged

[atalman]

Link to landed trunk PR (if applicable):
*

Link to release branch PR:

• [Release only] Advance triton version to 2.3.1 #126204
• Revert "[Release only] Release 2.3 start using triton package from pypi" #126202
• Pin triton-rocm to latest 2.3.1 commit #126309

Criteria Category:

• Release only - use triton 2.3.1 version rather then current 2.3.0

---

@atalman merged

[antoinebrl]

Hello 👋!
I am facing some issue regarding none-persistent buffers in distributed scenarios. A solution was submitted and merged into main (#125337). I was doing the cherry picking process to submit this fix for it to be part of the next patch release. However, the solution was developed after a refactoring which tackled another problem introduced in 2.3 (a891779). This create a conflict preventing me to submit the cherry picked commit as is. Should I submit both modifications in the corresponding order for them to be included in 2.3.1?
Tagging @fegin who authored both contributions.

[atalman]

Link to landed trunk PR (if applicable):

• [EZ] Get rid of utf-8 quotes #124932

Link to release branch PR:

• [EZ] Get rid of utf-8 quotes #126301

Criteria Category:

• Documentation

---

@atalman merged

[williamwen42]

Link to landed trunk PR (if applicable):

• [dynamo] use proxies to nn.Module in dynamo generated GraphModules #120756

Link to release branch PR:

• [dynamo] use proxies to nn.Module in dynamo generated GraphModules #126332

Criteria Category:

• Critical fix CUDA Memory leak w/ torch.compile in both stable and trunk #119607 torch.compile refleak

---

@atalman merged

[jithunnair-amd]

Link to landed trunk PR (if applicable):

• [ROCm] Add cublasGemmAlgo_t -> hipblasGemmAlgo_t #121030

Link to release branch PR:

• [release/2.3] Added cublasGemmAlgo_t -> hipblasGemmAlgo_t  #126448

Criteria Category:

• Low risk addition of hipify mappings to enable DeepSpeed transformer extensions on ROCm

[weifengpy]

Link to landed trunk PR (if applicable):

• [FSDP1] fix _same_storage check for DTensor #123617

Link to release branch PR:

• [FSDP1] fix _same_storage check for DTensor (#123617) #126464

Criteria Category:

• Fixes to regressions against the 2.3.0 release

[atalman]

Link to landed trunk PR (if applicable):

• NA

Link to release branch PR:

• [Release only] Pin rocm docker images #126452
• Revert "[Release only] Pin rocm docker images" #126890

Criteria Category:

• Release only changes - pin docker image for rocm CI. Temporary PR. ROCm test jobs were failing with the MIOpen error because a subtle difference crept in the MIOpen kdb files when the docker images were rebuilt. Hence pin to make CI jobs green. Will unpin once kdb issue is resolved

---

@atalman merged #126452

[mvpatel2000]

Link to main PR (if applicable):

• [DCP][state_dict] Remove the check of FSDP has root #121544

Link to 2.3.1 release branch PR:

• [DCP][state_dict] Remove the check of FSDP has root (#121544) #126557

Criteria Category:

• Low risk critical fix for checkpointing. This PR removes an additional check introduced in torch 2.3 which is actually incorrect and causes checkpointing to fail if at least 1 forward/backward pass has not been run.

[mvpatel2000]

Link to main PR (if applicable):

• Remove activation checkpointing tag to get correct FQNs #124698

Link to 2.3.1 release branch PR:

• Remove activation checkpointing tag to get correct FQNs (#124698) #126559

Criteria Category:

• Low risk critical fix for checkpointing. When checkpointing with activation checkpointing, the names of the variables (FQNs) are changed with the tag _checkpoint_wrapped_module. This breaks checkpointing if you resume without activation checkpointing.

[mvpatel2000]

Link to main PR:

• [DSD] Correctly handle _extra_state #125336

Link to 2.3.1 release branch PR:

• [DSD] Correctly handle _extra_state (#125336) #126567

Criteria Category:

• Low risk critical fix for checkpointing. Torch 2.3 ignores _extra_state (whereas prior PyTorch versions correctly handle it), breaking checkpoint loading from prior checkpoints. This also breaks integration with Nvidia's TransformerEngine.