Scalafix sbt causing OOM errors

[tgodzik]

This started happening after the most recent release and always breaks on a stack trace:

java.lang.OutOfMemoryError: Java heap space
	at org.eclipse.jgit.internal.storage.file.PackIndexV2.<init>(PackIndexV2.java:95)
	at org.eclipse.jgit.internal.storage.file.PackIndex.read(PackIndex.java:101)
	at org.eclipse.jgit.internal.storage.file.PackIndex.open(PackIndex.java:67)
	at org.eclipse.jgit.internal.storage.file.Pack.idx(Pack.java:161)
	at org.eclipse.jgit.internal.storage.file.Pack.get(Pack.java:273)
	at org.eclipse.jgit.internal.storage.file.PackDirectory.open(PackDirectory.java:216)
	at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedObject(ObjectDirectory.java:393)
	at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedFromSelfOrAlternate(ObjectDirectory.java:356)
	at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openObjectWithoutRestoring(ObjectDirectory.java:346)
	at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openObject(ObjectDirectory.java:331)
	at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:132)
	at org.eclipse.jgit.revwalk.RevWalk.getCachedBytes(RevWalk.java:1119)
	at org.eclipse.jgit.revwalk.RevCommit.parseHeaders(RevCommit.java:126)
	at org.eclipse.jgit.revwalk.RevWalk.markStart(RevWalk.java:308)
	at org.eclipse.jgit.api.LogCommand.add(LogCommand.java:343)
	at org.eclipse.jgit.api.LogCommand.add(LogCommand.java:180)
	at org.eclipse.jgit.api.LogCommand.call(LogCommand.java:133)
	at scalafix.internal.sbt.JGitCompletion.$anonfun$x$1$1(JGitCompletions.scala:29)
	at scalafix.internal.sbt.JGitCompletion$$Lambda$9375/0x00007ec7[2180](https://github.com/scalameta/metals/actions/runs/8298008580/job/22710386419#step:4:2181)8600.apply(Unknown Source)
	at scala.util.Try$.apply(Try.scala:213)
	at scalafix.internal.sbt.JGitCompletion.<init>(JGitCompletions.scala:29)
	at scalafix.internal.sbt.ScalafixCompletions.gitDiffParser$lzycompute(ScalafixCompletions.scala:119)
	at scalafix.internal.sbt.ScalafixCompletions.gitDiffParser(ScalafixCompletions.scala:118)
	at scalafix.internal.sbt.ScalafixCompletions.parser(ScalafixCompletions.scala:186)
	at scalafix.sbt.ScalafixPlugin$.$anonfun$scalafixInputTask$4(ScalafixPlugin.scala:357)
	at scalafix.sbt.ScalafixPlugin$$$Lambda$8642/0x00007ec7215f5600.apply(Unknown Source)
	at scala.Function1.$anonfun$compose$1(Function1.scala:49)
	at scala.Function1$$Lambda$1172/0x00007ec7204dee78.apply(Unknown Source)

even if no scalafix task is actually run.

Were there any changes recently that could cause it?

[tgodzik]

Maybe the jgit update broke things?

[tgodzik]

Or more likely https://github.com/scalacenter/sbt-scalafix/pull/378/files

[tgodzik]

Or it might be scalameta/scalameta#3650 though it's curious the stack trace show ScalafixCompletions and scalafix is not even being run

[xuwei-k]

I think jgit use too many memory.

here is profile memory result just launch sbt in my private large repository.

just launch sbt

if remove .git directory before launch sbt. for avoid isGitRepository == true

[xuwei-k]

Or more likely https://github.com/scalacenter/sbt-scalafix/pull/378/files

Yes scalacenter/sbt-scalafix#378 remove scalafixCompletions from globalSettings. I think it generate many JGitCompletion instances

• add scalafixJGitCompletion key. reduce memory usage sbt-scalafix#405

[bjaglin]

Thanks for the report @tgodzik and for the fix @xuwei-k!

[xuwei-k]

also jgit use many memory if repo is too many commits and/or many branch, tag.

I think JGitCompletions.scala should not retain jgit objects for avoid memory leak.

https://github.com/scalacenter/sbt-scalafix/blob/0ab8c2ce40b7021e0cc421020a48e00cba9aa66f/src/main/scala/scalafix/internal/sbt/JGitCompletions.scala#L21

• refList is Seq[org.eclipse.jgit.lib.Ref]
• refs is List[org.eclipse.jgit.revwalk.RevCommit]

see following reproduce repo

• xuwei-k/scalafix-jgit-issue@8e49b17
• xuwei-k/scalafix-jgit-issue@518d8dd

[bjaglin]

also jgit use many memory if repo is too many commits and/or many branch, tag.

I think JGitCompletions.scala should not retain jgit objects for avoid memory leak.

Indeed, there is an inherent leak there, which predates sbt-scalafix 0.12.0.

I am on my phone right now so I cannot check, but I am unsure how the upper bound set by setMaxCount(20) causes so much memory consumption after the initial lookup.

In any case, feel free to open a follow-up PR 👍

[xuwei-k]

workaround

-addSbtPlugin("ch.epfl.scala" % "sbt-scalafix" % "0.12.0")
+// TODO remove explicit "scalafix-interfaces" dependency if sbt-scalafix 0.12.1 released
+addSbtPlugin("ch.epfl.scala" % "sbt-scalafix" % "0.11.1")
+libraryDependencies += "ch.epfl.scala" % "scalafix-interfaces" % "0.12.0"

[xuwei-k]

Any plans to release new version? 👀

[bjaglin]

Yes! 2.13.14 is around the corner so I am holding the release a bit (to provide 2.13.14 compatibility in the testkit). Based on the current ETA, it should be last days of April.