Improve gmail

[mtalexander]

This change adds a new option: "skipallmailduplicates" which works much like skipcrossduplicates but applies only to the GMail "All Mail" folder. This is useful for copying gmail to other if most messages have very few GMail labels and you want them to go into a folder for each label, but don't want messages that are labeled to also go into the All Mail folder.

It also changes the gmail to gmail default options to process only the All Mail folder. It already implied sync and resync labels so this is enough to do a full gmail to gmail sync. Everything is in All Mail and this way all messages will be copied and will have all the correct labels.

[gilleslamiral]

This change adds a new option: "skipallmailduplicates" which works much like skipcrossduplicates but applies only to the GMail "All Mail" folder. This is useful for copying gmail to other if most messages have very few GMail labels and you want them to go into a folder for each label, but don't want messages that are labeled to also go into the All Mail folder.

I don't get the exact idea so we have to discuss. Let's start with the --gmail1 context: the source is Gmail and the destination is not Gmail, ie, the source has labels/folders but the destination has only folders. The current --gmail1 option makes a message with several labels going only into the first folder/label encountered alphabetically; plus, the "All Mail" folder won't get duplicates since it is treated the last. If I understand your need, you want cross-duplicates, different labels generating duplicates across folders, except for the "All Mail" folder. I don't want to change the --gmail1 behavior with that, duplicates when duplicates were avoided, folders sizes matter, do they? This behavior, different labels generating duplicates across folders except for the "All Mail" folder, is obtained with a sync with no --gmail1 but instead --exclude "All Mail" followed by another sync with --gmail1 but without --exclude "All Mail", this second sync is done to get the messages with no labels at all from "All Mail", if they are some of them. It can be a new option, --skipallmailduplicates is a good name, but I can't put it as a default in --gmail1

It also changes the gmail to gmail default options to process only the All Mail folder. It already implied sync and resync labels so this is enough to do a full gmail to gmail sync. Everything is in All Mail and this way all messages will be copied and will have all the correct labels.

This one could be a very good one but I'm not sure that every message is in "All Mail". I remember having your idea when developing --synclabels --resynclabels but I entered a headache nightmare when noticing Gmail labels behavior across folders.
Maybe Gmail has changed and that currently "All Mail" really contains all emails with all the labels well set to do a good Gmail to Gmail sync but I have to investigate on that.

Anyway thanks for your proposal and your time on it Mike, I appreciate.

[mtalexander]

On 26 Jan 2021, at 15:56, Gilles LAMIRAL wrote: I don't get the exact idea so we have to discuss. Let's start with the --gmail1 context: the source is Gmail and the destination is not Gmail, ie, the source has labels/folders but the destination has only folders. The current --gmail1 option makes a message with several labels going only into the first folder/label encountered alphabetically; plus, the "All Mail" folder won't get duplicates since it is treated the last. If I understand your need, you want cross-duplicates, different labels generating duplicates across folders, except for the "All Mail" folder. I don't want to change the --gmail1 behavior with that, duplicates when duplicates were avoided, folders sizes matter, do they? This behavior, different labels generating duplicates across folders except for the "All Mail" folder, is obtained with a sync with no --gmail1 but instead --exclude "All Mail" followed by another sync with --gmail1 but without --exclude "All Mail", this second sync is done to get the messages with no labels at all from "All Mail", if they are some of them. It can be a new option, --skipallmailduplicates is a good name, but I can't put it as a default in --gmail1

Yes, that's what I want. If someone treats GMail sort of like any other IMap server then they will tend to put only one or two labels on any one message and this behavior makes perfect sense. Besides storage is free today for all practical purposes. I helped two friends rescue large GMail archives for relatives who had passed away and both fit this description. In both cases they wanted to preserve the folder/label structure, but not put everything in the archive folder too. I wonder how your suggestion works. Won't the second sync for All Mail get *all* of All Mail, even messages with labels? Even if you add the skipcrossduplicates option, I didn't think it would actually skip anything since it hasn't scanned any other folders. How does it know what other folders messages are in without scanning those folders? I must be missing something. If this two pass mode works the same as my one pass with skipallmailduplicates then you can replace my code with that if you want. However it will certainly take longer since it scans all labeled messages twice. Changing the default for gmail to non-gmail copies is ok. That's really your call. I though the change made sense for most people, but if you think otherwise, that's ok.

This one could be a very good one but I'm not sure that every message is in "All Mail". I remember having your idea when developing --synclabels --resynclabels but I entered a headache nightmare when noticing Gmail labels behavior across folders. Maybe Gmail has changed and that currently "All Mail" really contains all emails with all the labels well set to do a good Gmail to Gmail sync but I have to investigate on that.

I had the same concern, but I don't think it was justified (at least in the cases I looked at). The first person I helped decided in the end to archive the mail to another GMail account and I used the technique I described to do this. I made several checks to ensure that I got all the mail in the new account and convinced myself that I had. I also tried it with some other, smaller GMail accounts and it seemed to work ok. On the other hand if you're concerned about this, then don't change it. I can always get the same effect by explicitly specifying the parameters I want. Mike

[gilleslamiral]

Yes, that's what I want.

Goint point for us!

I wonder how your suggestion works. Won't the second sync for All Mail get all of All Mail, even messages with labels?

No. The second sync performs the "All Mail" after all the other folders. It's --gmail1, it is not --gmail1 --folder "All Mail"

[mtalexander]

On 27 Jan 2021, at 17:31, Gilles LAMIRAL wrote: I wonder how your suggestion works. Won't the second sync for All Mail get all of All Mail, even messages with labels? No. The second sync performs the "All Mail" after all the other folders. It's --gmail1, it is not --gmail1 --folder "All Mail"

OK, I missed that detail, sorry about that. However this means that the total time for the two runs will be even larger since it scans all mailboxes twice, not just All Mail. It should, however, do what I wanted. I think the new code is equivalent and should take roughly half the time. Mike

[gilleslamiral]

However this means that the total time for the two runs will be even larger since it scans all mailboxes twice, not just All Mail. It should, however, do what I wanted. I think the new code is equivalent and should take roughly half the time. Mike

I don't think so. Most of the time is spent copying the messages, not getting the folder listings. It will take half the time after the first complete sync, for update runs, but update runs are usually fast.

Thanks anyway, you've seen how imapsync code is a mess and you could deal a little with it :-)

[mtalexander]

On 28 Jan 2021, at 23:38, Gilles LAMIRAL wrote: I don't think so. Most of the time is spent copying the messages, not getting the folder listings. It will take half the time after the first complete sync, for update runs, but update runs are usually fast.

Perhaps. It depends to some extent how many messages we're talking about. The account that started all this had about 141,000 messages and scanning them wasn't exactly instantaneous. However, I agree that this probably isn't reason enough to add yet another option and more special case code. Like many of us at UMich Barbara started using EMail in the mid 70s and had accumulated a lot of it. Her husband wrote the first network EMail software at Michigan in 1982 back before the Internet as we know it existed. His software sent mail around by various commercial networks and would even dial up a modem sometimes to deliver mail.

Thanks anyway, you've seen how imapsync code is a mess and you could deal a little with it :-)

I've seen a lot worse. It's actually better than I first expected when I saw that it is one big file. Except for that little detail it's pretty good. Fortunately I'm not intimidated by large source files. :-) Mike

[gilleslamiral]

Hi Mike!

I won't apply your pull request but I wrote a FAQ item about this need in the FAQ.Gmail.txt document.
https://imapsync.lamiral.info/FAQ.d/FAQ.Gmail.txt

Feel free to criticize it.

=======================================================================
Q. The context is a Gmail account at the source, but not Gmail at the
destination. I want cross-duplicates, different Gmail folders/labels
generating duplicate messages across folders on the destination,
except for the "All Mail" folder.

R. This behavior, different Gmail labels generating duplicates across
folders except for the "All Mail" folder, is obtained with a
synchronization with no --gmail1 but instead --exclude "All Mail"
followed by another synchronization with --gmail1 but without
--exclude "All Mail".

  imapsync ... --exclude "All Mail"
  imapsync ... --gmail1

This second sync is done to get the messages with no labels at all
from the folder "All Mail", if they are some of them.

Technically speaking, the first synchronization has not the option
--skipcrossduplicates but the second one has the option
--skipcrossduplicates and it also performs the Gmail "All Mail"
folder/label at the end so only the messages that have no labels will
end up in the "All Mail" destination folder (which is not Gmail,
remember the context).

Thanks to Mike Alexander for this suggestion. 
Mike also provided a patch to implement that directly in imapsync.
See https://github.com/imapsync/imapsync/pull/266
(I refused it because well maintaining it will be very hard for me
and I have other priorities).

=======================================================================