How to configure Fetchmail with a remote IMAP server? #3994

Doni49 · 2024-04-29T02:52:49Z

Doni49
Apr 29, 2024

For the past couple decades, I have had a hosted cPanel web account that includes an email server. This has been working well -- but I have been dealing with storage capacity issues. My goal is to have docker-mailserve & fetchmail connect to the remote server, download messages and present them to clients that connect locally to dms. DMS will never be exposed to the public.

I have docker-mailserve installed and fetchmail is enabled. I have a user created in docker-mailserve and have successfully sent and received a test message to and from this account.

But I don't see how to specify how to connect to the remote server.

I've looked through the fetchmail documentation.

But I'm not sure where to put any of the fetchmail settings I need to use.
I'm also unsure how to tie the remote user messages to the local user inbox. i.e. download messages from RemoteUserInbox and place them in LocalUserInbox? As of right now, I'm the only user. But at some point in the future, I may need to add additional users so knowing this will be even more of an issue.
The remote server is IMAP compatible. I like using IMAP because it results in near instant arrival of messages. I'd like to connect to the remote server via IMAP, download the messages and delete them from the server. Is that feasible?

Answered by polarathene

May 2, 2024

Local test between two DMS with fetchmail

Here you go, this will let you try the feature locally with two instances of DMS.

# compose.yaml
services:
  dms-fetch:
    image: ghcr.io/docker-mailserver/docker-mailserver:latest
    hostname: mail.example.test
    environment:
      ENABLE_FETCHMAIL: 1
      # We change this setting to 10 for quicker testing:
      FETCHMAIL_POLL: 10
    # Link the DNS lookup `remote.test` to resolve to the `dms-remote` container IP (for `@remote.test` address):
    # This is only for this example, since no real DNS service is configured, this is a Docker internal DNS feature:
    links:
      - "dms-remote:remote.test"
    # NOTE: Optional, You only need to p…

View full answer

Doni49 · 2024-04-29T02:53:22Z

glime-ai[bot]
bot Apr 29, 2024

Please note that this response was created using generative AI, leveraging previously available information from the repository. This information may be helpful. However, it is our strong recommendation that you carefully consider the information before blindly applying it.

To configure Fetchmail with Docker Mailserver (DMS) to connect to a remote server, download messages, and manage them locally, you need to follow these steps:

Configure Fetchmail Settings:
- You need to create a fetchmail.cf configuration file. This file should be placed in the docker-data/dms/config/ directory of your Docker Mailserver setup.
- An example configuration for an IMAP server (like your remote server) would look like this:
```
poll imap.remote-server.com proto imap
  user 'remoteuser@example.com'
  pass 'password'
  is 'localuser@yourdomain.com'
  ssl
  keep
```
- In this configuration:
  - poll imap.remote-server.com specifies the IMAP server of your remote email.
  - user 'remoteuser@example.com' and pass 'password' are the credentials for the remote email account.
  - is 'localuser@yourdomain.com' specifies the local DMS user where the emails will be delivered.
  - ssl ensures that the connection to the IMAP server is encrypted.
  - keep is optional and if included, Fetchmail will leave a copy of the emails on the remote server. If you want to delete them after fetching, remove this line.
Tie Remote User Messages to Local User Inbox:
- The is 'localuser@yourdomain.com' part of the Fetchmail configuration ensures that emails fetched from the remote server are delivered to the specified local user's inbox in DMS.
Setting Up IMAP for Instant Message Arrival and Deletion:
- Fetchmail supports IMAP with options to delete messages after fetching if you do not include the keep command in your configuration. This setup will download the messages and then delete them from the remote server, as per your requirement.
Adding Additional Users in the Future:
- To add more users, you would replicate the user-specific block in the fetchmail.cf file for each new remote-local user pair, adjusting the user, pass, and is fields accordingly.

After setting up your fetchmail.cf, ensure that the Fetchmail service is enabled in your Docker Mailserver by setting the environment variable ENABLE_FETCHMAIL to 1 in your compose.yaml file.

This setup should allow you to achieve your goal of connecting to a remote IMAP server, downloading messages, and managing them locally without exposing DMS to the public.

This comment was generated by Glime.

2 replies

Doni49 Apr 29, 2024
Author

Unfortunately, I did that but none of the remote messages are getting downloaded. I thought I misread how to do it so I asked here.

Doni49 Apr 29, 2024
Author

I've got the polling time set to 120 seconds. But even when I send a test message to the remote server (which I can see the message is there if login via webmail -- but don't click is to avoid having it marked as read) and wait an hour or more, it doesn't show up in the local server.

Here is my fetchmail.cf file (remote domain name and remote username/password have been changed for posting here):

poll 'mail.example.com' proto imap
  user 'user@example.com'
  pass 'pass'
  is 'don@home.arpa'
  ssl
  --nokeep

Here is my compose.yaml file for docker-mailserver:

services:
  mailserver:
    image: ghcr.io/docker-mailserver/docker-mailserver:latest
    container_name: mailserver
    # Provide the FQDN of your mail server here (Your DNS MX record should point to this value)
    hostname: docker.home.arpa
    env_file: mailserver.env
    # More information about the mail-server ports:
    # https://docker-mailserver.github.io/docker-mailserver/latest/config/security/understanding-the-ports/
    ports:
      - "25:25"    # SMTP  (explicit TLS => STARTTLS, Authentication is DISABLED => use port 465/587 instead)
      - "143:143"  # IMAP4 (explicit TLS => STARTTLS)
      - "465:465"  # ESMTP (implicit TLS)
      - "587:587"  # ESMTP (explicit TLS => STARTTLS)
      - "993:993"  # IMAP4 (implicit TLS)
    volumes:
      - /mnt/docker/mailserver/data/:/var/mail/
      - /mnt/docker/mailserver/state/:/var/mail-state/
      - /mnt/docker/mailserver/logs/:/var/log/mail/
      - /mnt/docker/mailserver/config/:/tmp/docker-mailserver/
      - /etc/localtime:/etc/localtime:ro
    restart: always
    stop_grace_period: 1m
    # Uncomment if using `ENABLE_FAIL2BAN=1`:
    # cap_add:
    #   - NET_ADMIN
    healthcheck:
      test: "ss --listening --tcp | grep -P 'LISTEN.+:smtp' || exit 1"
      timeout: 3s
      retries: 0

And Here's a screenshot taken from portainer showing the environment variables that are relevant to fetchmail:

polarathene · 2024-05-02T02:32:48Z

polarathene
May 2, 2024
Maintainer

Local test between two DMS with fetchmail

Here you go, this will let you try the feature locally with two instances of DMS.

# compose.yaml
services:
  dms-fetch:
    image: ghcr.io/docker-mailserver/docker-mailserver:latest
    hostname: mail.example.test
    environment:
      ENABLE_FETCHMAIL: 1
      # We change this setting to 10 for quicker testing:
      FETCHMAIL_POLL: 10
    # Link the DNS lookup `remote.test` to resolve to the `dms-remote` container IP (for `@remote.test` address):
    # This is only for this example, since no real DNS service is configured, this is a Docker internal DNS feature:
    links:
      - "dms-remote:remote.test"
    # NOTE: Optional, You only need to publish ports if you want to verify via your own mail client.
    #ports:
    #  - "465:465"  # ESMTP (implicit TLS)
    #  - "993:993"  # IMAP4 (implicit TLS)
    # You'd normally use volumes here, for simplicity of the example, all config is contained within `compose.yaml`:
    configs:
      - source: dms-accounts-fetch
        target: /tmp/docker-mailserver/postfix-accounts.cf
      - source: fetchmail
        target: /tmp/docker-mailserver/fetchmail.cf

  dms-remote:
    image: ghcr.io/docker-mailserver/docker-mailserver:latest
    hostname: mail.remote.test
    environment:
      # Allows for us send a test mail easily by trusting any mail client run within this container (`swaks`):
      PERMIT_DOCKER: container
      # Alternatively, trust and accept any mail received from clients in same subnet of dms-fetch:
      #PERMIT_DOCKER: connected-networks
    configs:
      - source: dms-accounts-remote
        target: /tmp/docker-mailserver/postfix-accounts.cf

# Using the Docker Compose `configs.content` feature instead of volume mounting separate files.
# NOTE: This feature requires Docker Compose v2.23.1 (Nov 2023) or newer:
# https://github.com/compose-spec/compose-spec/pull/446
configs:
  fetchmail:
    content: |
      poll 'mail.remote.test' proto imap
        user 'jane.doe@remote.test'
        pass 'secret'
        is 'john.doe@example.test'
        no sslcertck

  # DMS requires an account to complete setup, configure one for each instance:
  # NOTE: Both accounts are configured with the same password (SHA512-CRYPT hashed), `secret`.
  dms-accounts-fetch:
    content: |
      john.doe@example.test|{SHA512-CRYPT}$$6$$sbgFRCmQ.KWS5ryb$$EsWrlYosiadgdUOxCBHY0DQ3qFbeudDhNMqHs6jZt.8gmxUwiLVy738knqkHD4zj4amkb296HFqQ3yDq4UXt8.

  dms-accounts-remote:
    content: |
      jane.doe@remote.test|{SHA512-CRYPT}$$6$$sbgFRCmQ.KWS5ryb$$EsWrlYosiadgdUOxCBHY0DQ3qFbeudDhNMqHs6jZt.8gmxUwiLVy738knqkHD4zj4amkb296HFqQ3yDq4UXt8.

$ docker compose up

# John has no mail in his inbox (Your local DMS instance with fetchmail)
$ docker compose exec -it dms-fetch doveadm mailbox status -u john.doe@example.test messages INBOX
INBOX messages=0

# Jane has no mail in her inbox (Your remote DMS instance)
$ docker compose exec -it dms-remote doveadm mailbox status -u jane.doe@remote.test messages INBOX
INBOX messages=0

# Send the remote DMS a mail internally (PERMIT_DOCKER ENV allows to send this without authentication):
$ docker compose exec -it dms-remote swaks --server localhost --from someone@example.test --to jane.doe@remote.test

# Jane has mail now, until fetchmail takes it:
$ docker compose exec -it dms-remote doveadm mailbox status -u jane.doe@remote.test messages INBOX
INBOX messages=1

# Wait for fetchmail poll to pass, and check inboxes again,
# Fetchmail has retrieved Janes mail, deleting from her inbox, sending the copy to John's inbox:
$ docker compose exec -it dms-remote doveadm mailbox status -u jane.doe@remote.test messages INBOX
INBOX messages=0
$ docker compose exec -it dms-fetch doveadm mailbox status -u john.doe@example.test messages INBOX
INBOX messages=1

Fetchmail config

--nokeep is not what you want for the fetchmail.cf file, that is for the CLI daemon options, the equivalent config keyword is no keep (default).
Likewise ssl should be the default AFAIK. As my example above was testing locally, extra work is required to setup TLS certs that can be verified, so to skip that additional complexity I used no sslcertck. Careful not to have this line in production config, as you will likely want to verify trust is correct.

You can see that it works as intended. So please refer to this example and adapt it to your remote mail server. It should successfully be able to retrieve the mail. Then adapt the example to your actual mail server setup, and it should help you troubleshoot where the problem is 👍

Looking at our docs for fetchmail config, the settings are effectively the same in the example above as they are in the docs example. I don't see anything that I can update there, beyond mentioning this more complete example for local testing/verification?

Docs also provide some debug advice: https://docker-mailserver.github.io/docker-mailserver/v13.3/config/advanced/mail-fetchmail/#debugging

For any additional trobleshooting, be sure to check the logs from DMS. When Fetchmail has problems connecting, it should be logging messages like this:

dms-fetch   | May  2 13:50:25 mail fetchmail[653]: remote.test: upgrade to TLS failed.
dms-fetch   | May  2 13:50:25 mail fetchmail[653]: Unknown login or authentication error on jane.doe@remote.test
dms-fetch   | May  2 13:50:25 mail fetchmail[653]: socket error while fetching from jane.doe@remote.test
dms-fetch   | May  2 13:50:25 mail fetchmail[653]: Query status=2 (SOCKET)
dms-remote  | May  2 13:50:25 mail dovecot: imap-login: Disconnected: Connection closed (no auth attempts in 0 secs): user=<>, rip=172.19.0.3, lip=172.19.0.2, session=<rmao024XPqKsEwAD>

As you can see, the connection to mail.remote.test was partially successful, but the TLS connection was not, which caused the actual login/auth error for jane.doe@remote.test. This was when I tried with some self-signed certs locally, but SSL_TYPE=self-signed ENV or SSL_TYPE=manual wasn't happy verifying certs as the root CA used wasn't trusted within the container 😓

15 replies

polarathene May 5, 2024
Maintainer

I think I get what you were trying to explain now.

Yes that's right! 😁

Two containers are configured, both running DMS. One is meant to represent your local DMS container with fetchmail, the other the other mail server you want fetchmail to collect from.

I ran your code and it didn't present any errors.

Fantastic! 😎 🎉

Now you can continue with the earlier guidance.

Adjust the example fetchmail.cf to connect to your real remote mail server (you may want to add the line keep for testing if you don't want it to remove the mail from your real remote just yet. Alternatively, skip to the next step if it's easier for you to inspect mail.
Then try that fetchmail.cf on your real local DMS (either with the fake or real remote server)
- If it's with the fake, you'll need the dms-remote service added to your real compose.yaml and the link from dms-fetch setting added to your real DMS container.
- If you have the fetchmail.cf adjusted for your real remote mail server, then hopefully everything is working now 👍 (this may be the easiest for you to try first?) Don't forget to remove the no sslcertck line from the config once you're successful.
If you're not having success, then you'll need to experiment a bit with the information / guidance I've provided to troubleshoot what is misconfigured.

Doni49 May 5, 2024
Author

I modified fetchmail.cf to download from my real remote mail server. The test message that I sent is no longer there. So it appears that it should be waiting for me on the local server. I don't have an email client configured to check the test account that was created locally. But since I was previously able to connect both my android phone and thunderbird on my laptop to the local server, I'm assuming that should be a simple process.

Thanks for all your help! It has been greatly appreciated!

polarathene May 5, 2024
Maintainer

So it appears that it should be waiting for me on the local server.

Yes, although in my compose.yaml example there is no /var/mail volume mount like you'd usually have. This means as soon as the container is removed so is that data. Hence why I mentioned adding keep to fetchmail.cf when testing, which would prevent deleting the original mail from the remote server, once you've got it working as you want you can remove that line and the default behaviour to delete the original mail when fetched will occur going forward.

I don't have an email client configured to check the test account that was created locally

Thunderbird can do this, but it's probably not something I have time to walk you through (especially with my offline compose.yaml example with two mailservers) for local container connections.

This is why my example commands showed you how to check a mail had arrived:

# This is checking `john.doe@example.test` account as defined in `postfix-accounts.cf`, for the example dms-fetch container:
$ docker compose exec -it dms-fetch doveadm mailbox status -u john.doe@example.test messages INBOX
# The output from the command should look something like this:
INBOX messages=1

That's an easy way to confirm the email is received locally at least, since you can see the count increase each time.

There's other commands to access the actual email contents, but I don't recall what those are. The mail is stored with filenames that aren't very human friendly to identify/type 😓

If you have /var/mail mounted, you can possibly browse and inspect file contents to find the raw mail data, which a mail client will parse to display.

But since I was previously able to connect both my android phone and thunderbird on my laptop to the local server, I'm assuming that should be a simple process.

Oh, if you've got separate client devices that's probably easier yes 👍 My compose.yaml example omitted publishing any ports, which you'd normally do for them to be available from the Docker host or any devices that can connect to that server. Then you'd be able to connect with Thunderbird or any other mail client.

Thanks for all your help! It has been greatly appreciated!

You're welcome!

Thanks for giving DMS a go and helping to improve our documentation on fetchmail 😎 (the next release of DMS has our fetchmail docs page link to this discussion to help others troubleshoot config)

Doni49 May 5, 2024
Author

I added my own user account to the local server. Then I changed the fetchmail.cf configuration so that mail in the remote inbox goes to the new account. I've got thunderbird and my phone configured to get messages from the local server. The test messages are arriving! :)

Oh -- and I added the necessary ports to allow email clients to connect to the local server.

polarathene May 5, 2024
Maintainer

EDIT: Seems like you deleted what I was replying to before I sent this, I take it you figured it out 😎

Can I safely have fetchmail.cf check TWO remote servers and download messages from BOTH inboxes into ONE local account?

AFAIK that should be possible. View the current edge release docs (these will be v14 when released): https://docker-mailserver.github.io/docker-mailserver/edge/config/advanced/mail-fetchmail/

So something like this should work as your fetchmail.cf based on what we've used in the above discussion:

poll 'mail.remote.test' proto imap
  user 'jane.doe@remote.test' pass 'secret' is 'john.doe@example.test'

poll 'mail.another-remote.test' proto imap
  user 'alice@another-remote.test' pass 'secret' is 'john.doe@example.test'

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Docker Mailserver

How to configure Fetchmail with a remote IMAP server? #3994

{{title}}

Replies: 2 comments 17 replies

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Docker Mailserver

How to configure Fetchmail with a remote IMAP server? #3994

Doni49 Apr 29, 2024

Local test between two DMS with fetchmail

Replies: 2 comments · 17 replies

glime-ai[bot] bot Apr 29, 2024

Doni49 Apr 29, 2024 Author

Doni49 Apr 29, 2024 Author

polarathene May 2, 2024 Maintainer

Local test between two DMS with fetchmail

Fetchmail config

polarathene May 5, 2024 Maintainer

Doni49 May 5, 2024 Author

polarathene May 5, 2024 Maintainer

Doni49 May 5, 2024 Author

polarathene May 5, 2024 Maintainer

Doni49
Apr 29, 2024

Replies: 2 comments 17 replies

glime-ai[bot]
bot Apr 29, 2024

Doni49 Apr 29, 2024
Author

Doni49 Apr 29, 2024
Author

polarathene
May 2, 2024
Maintainer

polarathene May 5, 2024
Maintainer

Doni49 May 5, 2024
Author

polarathene May 5, 2024
Maintainer

Doni49 May 5, 2024
Author

polarathene May 5, 2024
Maintainer