Improve performance #2982
Comments
|
I'd love to help contribute on number 2. I've developed some decent familiarity with the section management functions. I'll keep an eye out for the new issue. |
|
Just want to comment that this would be really cool, especially number 2: I regularly do source-rewriting on a fairly large project with rewriters which are a bit hit-or-miss (they'll false-positive and rewrite stuff they should not and the serialisation will rewrite slightly more than it should) so there needs to be a post-op filtering to create the actual commits (remove stuff or split changes across multiple commits) add -p/checkout -p kinda work but the lack of context and the keyboard answers means it's easy to "go automatic" and end up e.g. reverting a change which should not have or staging one which shouldn't be. Magit's diff-committed and diff-uncommitted are absolute game-changers there, they're absolutely stellar in UI and UX. Their issue however is that staging changes are slow, and the slowness becomes huge as the diff size grows, with a ~15kloc diff (wasn't kidding about the bigness) it takes about 1mn30 between me hitting |
|
@xmo-odoo Have you tried |
|
@braham-snyder I'm really not fond of ediff (though that may be because I've almost never used it so I find it odd), and the complete overview of the working copy is probably the thing I love most about magit's diff views. |
|
One more observation. |
|
I am planning to do that too. Added it to the description above. |
|
There may be some good pointers over at gitstatus that can help also make |
|
It seems that everybody here is aware of these issues, but just for the record I must mention that:
I installed magit yesterday and I was excited about it, but for now I switch back to git at command line because |
Looks like this one is on an expanding front :). |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
OK, thank you for the feedback: I digged deeper: This happens, when you have gettext- / *.po files in your tree. If you dismiss all changes on these kind of files (we have about 20 in our repo), magit responses quite fast. I tried also to start the emacs without init.el, so no special hooks are activated to open gettext files. Is this a new bug or should I only add it here as hint for other users, which have the same issue. |
|
Actually lets look at this a bit. 20 untracked (or ignored?) files isn't much at all. So if you have those ~20 files, then it takes 20-40 seconds to do a simple refresh (C-g)? Please time it. And without them it takes less than a second? Are these files being ignore by Git (i.e. is there such a rule in Does the Magit status buffer show any information about these files? (Is there a |
|
More input to this issue: $> git checkout cs.po da.po de.po el.po en.po es.po fr.po hu.po it.po ja.po $> git checkout da.po... $> git checkout *.po
How large are our files: About 300 translations per language. A lot of translations are sentences. |
|
At the moment I have in my magit-status.el When I remove Then I can reduce the time from 38 to 10 seconds |
|
I just remembered we recently added something to make this easier: set Here's what those numbers normally look like: |
|
FYI I don't see FWIW, refresh takes about 400ms in my repo and I don't have forge enabled. Though I just disabled magit-todos and it brought it down to 200ms... yuck. Edit: Is there a way to limit what gets refreshed when staging and unstaging? Ideally it'd only refresh the staged and unstaged sections and it'd be great to disable things like magit-todos during that refresh. |
I think that was meant to be |
|
Ah, thanks. Looks like forge still takes a little time even if the repo isn't forge enabled yet, and magit-todos (specifically |
|
Repositories with large amount of both files and changes per commit, e.g. produced by generated code (e.g. when the source code is regenerated after an update to the code generator), make Long story short, not sure if the following idea is implied by the comprehensive list of improvements at the top but I'd still mention it. I believe a drastic performance improvement for
However, doing only that would spoil usability too indeed. The idea is to remap the key that is currently expanding diffs (which are already inserted in the buffer) to a function that would instead create a separate buffer where sections of diff hunks are inserted for that particular file only and immediately focus it; upon quit, focus back the parent This does not sound like a lot of work and can be introduced as an optional mode to use for performance reasons. What do you guys think? NOTE: #2985 looks interesting indeed but it's far too complex and ambitious to implement in the nearest future. One has to be reasonable with the ratio of outcome to delivery time/effort. Solving the outstanding performance problem can be done in stages, where the above approach seems appealing as the very first stage since it offers high ratio. |
|
it’s not a big rebase either, I’m rebasing maybe 10 commits. |
|
My initial profiling has pointed to Lines 978 to 1038 in 2145477 |
|
If you want to do more profiling, you may find https://github.com/aspiers/etrace useful. |
|
I also think it's good to use gitstatus. |
+1 Every time I rename a directory, I have to use some other Git client to commit the change, as Magit creates a huge, 500MB or even 1GB, status buffer with diffs nobody asked for, blocking the entire Emacs. It would help if we could at least hit |
|
Regarding refreshing .. Is this an example of something that is known for magit's performance issues? Here I show staging and unstaging a file. First I do it with
I have noticed similar delay and delay workarounds for other commands like committing |
Lately I have noticed similar failures to redisplay for many things unrelated to Magit. I suspect there is a regression in Emacs' |
I'm actually running emacs-27 version 27.2 on OSX as I come across this. I haven't tested whether I also exerience this on emacs 28 |
|
The described symptoms are very similar, but it appears this is a different issue. I have no idea what could be wrong here. |
See magit/magit#2982 Essentially, symlinks to git somehow perform slower.
|
Let me share a change in magit which made (defun magit-rev-format (format &optional rev args)
(let ((str (magit-git-string "log" "-1" "--no-patch"
(concat "--format=" format) args
(if rev (concat rev "^{commit}") "HEAD") "--")))
(unless (string-equal str "")
str)))In other words, use |
|
Not sure if useful, but here is a case where |
|
If we use libgit2, can magit still work over tramp? I wonder if there's value in some kind of git server that magit can connect to, and perhaps the git server can watch key files indicating git's state (and update accordingly) |
|
|
|
Interesting, that was the case for me aswell. Can you instruct git to schedule gcs more often somehow? |
|
I'm doing quite a large rebase, where I've removed it from |
|
I was recently struggling with a slightly sluggish Just wanted to mention the git fsmonitor here since I didn't find any other references to it in the various performance threads in this repo. Here is the github blog post I followed to setup the daemon. |
|
Thanks for the hint. Yeah, I should look into that and add a note alongside the other performance hints. (I've added that to my todo list, but it might be a while until I get around to it.) |
|
@SteVwonder thank you! Refresh on my monorepo went from 0.740s to 0.468s. Untracked time went from 0.235s to 0.028s. |
|
I find that Partially writing this here for its relevance – partially just writing it down so I can also investigate what can be done here once I'm back from running errands. |
That surprises me. How many commits does that normally list in that repository? |
It's still up to the value of Worth noting that times improved with caching that I expect either Git or the OS is doing. I've included performance numbers in the commits for #5075. Edit: oh, to answer your question directly, still just ten commits. |
|
Dear all,
Working with a merge of a month's work I had a typical Every change (staging, unstaging, trying to select "ours" or "theirs" but only being told I could trash the file and refusing...) triggered it. The redisplay also constantly returned the cursor to the top of the buffer, which, it seemed, added insult to injury, if you follow my metaphor. I found the idea of a lesser version of the status-buffer seductive, it would only show new/deleted/unmerged/modified + file path. I would deal with all of the files that I didn't need to look into, or use ediff. Perhaps commit the bulk of quick decisions and use a classic magit-status buffer for the rest. All this now appears unnecessary. Looking into code to try to bypass functions according to a flag, I found that there is already one: Thus, after allowing the display to settle a first time, I think I might put toggling this variable into a transient, and maybe add a I suggest this as an effective work-around for those whose workflow can deal with each file separately, or who are content to call updates manually. |
and others
Magit's performance isn't so good to say the least. I would go as far as to say that mediocre and in certain situations outright bad performance currently is Magit's biggest problem.
We have made some improvements in the past, like e.g. caching calls to
gitduring a buffer refresh, but these improvements are to a large extend outweighed by the addition of new features. Already there exist useful features that are disabled by default because they can be to costly in some cases.One major difference between Git and Magit is that the latter shows all kinds of useful information up-front. The major disadvantage of Magit's approach is that this information has to be always up-to-date (else it couldn't be relied on), and that doing so costs time.
To achieve good performance the cost of keeping (primarily, but not only) the status buffer up-to-date has to be reduced. And in order to do so, some fundamental changes have to be made. Doing that will take a while because it makes it necessary to replace some core abstractions. The good news is that improving or even replacing those abstractions will have other benefits beside improved performance.
A These are the primary causes of bad performance:
After the user has performed some action, the complete status buffer is refreshed. And that is done by recreating its content from scratch. Every time. All of it.
The required information is collected by calling
gitmany times. Until recently, there was no alternative, but now Emacs has a foreign function interface and so we can start usinglibgit2now.Expensive parsing (primarily diffs and logs) is done synchronously, even when we don't need the complete result immediately (because most of it is not visible until the user scrolls).
The results of expensive parsing are not being cached. (The result of calls to
gitare cached, but that mitigates the effect of (2), not (1).)Many sections are created by inserting the output directly into the user-visible buffer, where it is then manipulated to look and feel the way we want to. Without changing that, it would be very hard to address (3) and (4).
B There are three primary paths for improving performance:
Implement an Elisp binding for libgit2 #2959 Start using
libgit2instead of callinggitover and over again.Rethink parsing, caching, and refreshing #2985 Completely change how sections are created (using separate parsing-buffers, which likely will be preserved as a caching mechanism) and make it possible to update sections asynchronously and independently of the containing buffer.
Closely related to (2), only update those parts of a buffer that actually need updating.
Delay updating buffers that are not visible.
Insert diffs in two steps. Get a list of the modified files. For files that are expanded get the actual diff and insert it.
Luckily these two paths can be tackled in parallel, because in most cases a given section is either created by inserting and then "washing"
gitoutput (A5) or by processing a list of items (or a single value) using Elisp and then inserting that as the sections content (A2).So aside from this overview, these changes will also be discussed independently, in dedicated issues.
The text was updated successfully, but these errors were encountered: