Problems reading inbound email using IMAP

[trainor1]

Hi All,

I apologize for creating another discussion. Nobody was answering the previous one. So, I am hoping to catch more attention here.

I am running Django-helpdesk using PyCharm to figure out a configuration that works. The email provider is Microsoft Office 365. I did succeed in getting the outbound email connection to work using SMTP.

I am still failing to get an inbound connection to work using IMAP. I have configured a queue for the inbound email account. I have added a HELPDESK_OAUTH setting. When I run a test using manage.py get_email, I get an error saying that I am not using HTTPS. I have included a PDF of the error message. I have included a PDF that shows the general form of the values that I am using for HELPDESK_OAUTH settings.

The folks at Microsoft have been having a difficult time understanding the various values that I am trying to populate in the settings file. So, I am worried that I have them wrong. Also, I am wondering if my HTTPS issue has to do with trying to test this on the development server.

So, I am pretty well lost. I would greatly appreciate any advice that might help move this configuration and testing process forward.

Thanks in advance for your help.

Best Regards,

Kevin

[trainor1]

helpdesk_oauth_settings.pdf
get_email_failure.pdf

[brucegibbins]

Sorry Kevin. Had a number of day job commitments that have extended into my no-work day. I will attempt to provide some extra feedback in the other discussion.

…

On Tue, 11 July 2023, 8:37 am Kevin Trainor, ***@***.***> wrote: Hi All, I apologize for creating another discussion. Nobody was answering the previous one. So, I am hoping to catch more attention here. I am running Django-helpdesk using PyCharm to figure out a configuration that works. The email provider is Microsoft Office 365. I did succeed in getting the outbound email connection to work using SMTP. I am still failing to get an inbound connection to work using IMAP. I have configured a queue for the inbound email account. I have added a HELPDESK_OAUTH setting. When I run a test using manage.py get_email, I get an error saying that I am not using HTTPS. I have included a PDF of the error message. I have included a PDF that shows the general form of the values that I am using for HELPDESK_OAUTH settings. The folks at Microsoft have been having a difficult time understanding the various values that I am trying to populate in the settings file. So, I am worried that I have them wrong. Also, I am wondering if my HTTPS issue has to do with trying to test this on the development server. So, I am pretty well lost. I would greatly appreciate any advice that might help move this configuration and testing process forward. Thanks in advance for your help. Best Regards, Kevin — Reply to this email directly, view it on GitHub <#1103>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEAXOGHYN6E3MYHBRHKVTGTXPR73RANCNFSM6AAAAAA2FFA6WY> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[trainor1]

After several months of reading my Microsoft email accounts without problem, the get_email command has stopped retrieving email on 5 of my 6 email accounts (queues). I am attaching a file with the console output from the get_mail command with logging turned up to level 4:

get_mail_output_from_manage.py.txt

I have confirmed that IMAP is still enabled on all 6 email accounts. I also went through the procedure to Register Service Principals on all 6 email accounts again. I got back messages for all 6 accounts that look like they are still configured so that the user can be connected.

There is one other possible cause of this problem. I received an email back on October 12 that I thought was a phishing email. I am reluctant to post the body of the message here due to potential security issues. But, here is the Subject line:

• 3|6c9a5310-f391-43aa-f84e-08dbc9a8beb7|60.36.166.41|joe-oota@zpost.plala.or.jp|(External:Ligent 2FA Important Crediential Notification)

I would appreciate any ideas on how to recover from this failure. I have had 5 of my 6 HelpDesk queues down for several days.

Thanks in advance for your help.

Best Regards,

Kevin

[brucegibbins]

Hi Nothing obvious in log (to me at least) and it appears to be successfully connecting, searching - finding no messages and the logging off. But may I ask an obvious question? Are there any messages in the inbox folders being monitored?

…

On Tue, 17 Oct 2023, 6:05 am Kevin Trainor, ***@***.***> wrote: After several months of reading my Microsoft email accounts without problem, the get_email command has stopped retrieving email on 5 of my 6 email accounts (queues). I am attaching a file with the console output from the get_mail command with logging turned up to level 4: get_mail_output_from_manage.py.txt <https://github.com/django-helpdesk/django-helpdesk/files/12921200/get_mail_output_from_manage.py.txt> I have confirmed that IMAP is still enabled on all 6 email accounts. I also went through the procedure to Register Service Principals on all 6 email accounts again. I got back messages for all 6 accounts that look like they are still configured so that the user can be connected. There is one other possible cause of this problem. I received an email back on October 12 that I thought was a phishing email. I am reluctant to post the body of the message here due to potential security issues. But, here is the Subject line: - 3|6c9a5310-f391-43aa-f84e-08dbc9a8beb7|60.36.166.41| ***@***.***|(External:Ligent 2FA Important Crediential Notification) I would appreciate any ideas on how to recover from this failure. I have had 5 of my 6 HelpDesk queues down for several days. Thanks in advance for your help. Best Regards, Kevin — Reply to this email directly, view it on GitHub <#1103 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEAXOGCNEQXGFNQECLNV6DDX7WHSDAVCNFSM6AAAAAA2FFA6W2VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TEOJWHEYDA> . You are receiving this because you commented.Message ID: <django-helpdesk/django-helpdesk/repo-discussions/1103/comments/7296900@ github.com>

[trainor1]

Hi Bruce,

Yes. I have sent test emails to all 6 of the queues. One queue is still working, so that doesn’t have any mail stuck in the inbox. The other 5 queues all have at least 1 email stuck in the inbox. I logged into the Office 365 portal and confirmed that the emails were still there.

Is it possible that whatever generated the email that I believed to be phishing has somehow gummed up the works?

Best,

Kevin

[brucegibbins]

I'd agree with @uhurusurfa. The log does seem to infer that only one queue is actually being read and the logging option is where you should go. Also, the log you sent is only showing the actual IMAP exchange with the mail server. There is logging performed by the email.py script itself and this is what you need to see

[brucegibbins]

Based on what I saw and without further forensics. I doubt it. If it was me I would clear out the messages that are there and send a basic no frills message with no attachments. Start simple and work up from there. I would also consider upgrading to later release of helpdesk. I know that there has been significant rework on the mail parser.

…

On Tue, 17 Oct 2023, 8:33 am Kevin Trainor, ***@***.***> wrote: Hi Bruce, Yes. I have sent test emails to all 6 of the queues. One queue is still working, so that doesn’t have any mail stuck in the inbox. The other 5 queues all have at least 1 email stuck in the inbox. I logged into the Office 365 portal and confirmed that the emails were still there. Is it possible that whatever generated the email that I believed to be phishing has somehow gummed up the works? Best, Kevin — Reply to this email directly, view it on GitHub <#1103 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEAXOGEOZD7UT7CUYQHEXOLX7WY4XAVCNFSM6AAAAAA2FFA6W2VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TEOJXHAZTM> . You are receiving this because you commented.Message ID: <django-helpdesk/django-helpdesk/repo-discussions/1103/comments/7297836@ github.com>

[trainor1]

Hi Bruce,

I am open to both of your ideas and will begin working on them.

Can you explain what might have changed in my installation that would cause Django HelpDesk to start behaving differently. I am using virtual environments in both my development and production setups. I definitely have not changed anything on the development side. It is remotely possible that on the production side that a new version of Python has been thrown into the mix. Yet, I seem to be getting the same behavior in both environments.

Best,

Kevin

[trainor1]

Hi Bruce:

I am including DEBUG output for get_email that is based on a more realistic database configuration. The last version that I posted here had mistakenly been run with the test database. The test database had only 1 queue. The production database has 7 queues. This run definitely has an error in it. I would be interested in your ideas about what might be causing the error. As before, I am eager to understand how this might be triggered in one email account/queue and not in another. I am also eager to understand how this could begin happening suddenly when I have not changed any software on my end.

Kevin

manage_py_get_email_debug_output_on_2023_10_19.txt

[uhurusurfa]

Set you debug level for Django logging to TRACE and verify that alll expected email queues are actually being processed.
Your log file appears to only show one queue.

[trainor1]

Can you point me to a page that will explain how to "Set you debug level for Django logging to TRACE"?

I am not finding that setting.

[uhurusurfa]

https://docs.djangoproject.com/en/4.2/topics/logging/#loggers

[uhurusurfa]

Apologies - I have been working on Java recently which has TRACE level - set yours to DEBUG level as the most verbose level.

[uhurusurfa]

Each Queue has a "Logging Type" field and a "Logging Dir" field. Edit each queue and set the "Logging Type" field to Debug and the "Logging Dir" field as a path to a valid directory on the file system.
You should then see a log file created for each queue named in the form:
[logging_dir]/ [Q identifier]_get_email.log
That may shed more light on what is going on.

[trainor1]

Thanks.

[uhurusurfa]

There are characters in the received email in one of your mailboxes that is causing the software to crash.

Because of the way the get_email management command is written, if one queue crashes then all subsequent queues do not get processed (the queues will always be processed in the same order so in your case the first queue is successful every time and the 2nd crashes so all the others never get processed either.
So you will never get the rest of the queues processed until the offending email is removed.
There are 2 options right now:

1. You manually remove the offending email if you know which one it is
2. I will generate a fix that will both handle the offending email and ensure that one email in a queue does not stop the rest of the emails in the queue being processed and ensure that one queue does not affect the p[rocessing of another. It may take a few days to get the fix in place as I have other commitments right now.

[trainor1]

Hi Christopher,

Thanks for having explained the cause of this problem. Bruce had been telling me to clean out all of the old mess. I cleaned out all of my tickets. But, I didn't drain my email inboxes. Having read your explanation, I can now understand what Bruce meant for me to do.

I am in the process of moving to code version 1.0 in production. So, this will help.

Your idea #2 above is a good one. We definitely would benefit by making this process better able to recover from errors. Since I believe that I am now beyond my crisis, I guess that this change would need to get on the list of all possible changes that are competing for your attention.

Thanks again for your help.

Best,

Kevin

[trainor1]

This problem appears to have resolved by using version 1.0 of django-helpdesk. I did find one other problem while testing that I am opening as a new discussion. Thanks for your help on this.

[uhurusurfa]

Actually there is 1 other solution that will proibably work.
Version 1.0.0 which was released a few days ago has a completely reqoworked algorithm for processing emails and no longer has the same mechanism that is causing a failure in your case.
It is likely that if you swithc to the 1.0.0 code base it will work for you.

[trainor1]

Thanks. That agrees with the advice that I am getting from Bruce. So, I will definitely give it a try.

[trainor1]

I am in the process of solving the problem by switching to the version 1.0 code. I switched the code version in my development environment and it was able to resolve the problem. I am now working on getting the new code version installed into production.