Wrong sorting of emails (revisited)

[bfittes1]

I had this issue previously, maybe 2 years ago, then it got fixed, I don't remember how.  Did a factory data reset on my phone a couple weeks ago, issue is back and I can't figure out how to fix it.  Issue is that my inbox is sorted on message sent date/time rather than message received date/time. 

I checked the setting on IMAP options (as noted in another message thread), and it was already set to date/time received.  But it isn't working that way. 

I'm on version 1.5.9.14, running Android 5.0.1

[Kostya Vasilyev]

Did you upload messages from an old account to a new one?

Or perhaps moved them all between folders?

Since you did a reset and won't be able to check how the settings were -- how about trying "Date/time sent" (the default is indeed "Date/time received")?

You will need to re-create the account for this, making the change before it receives any messages. The FAQ has more detailed info on this.

[bfittes1]

When I reset my phone, I installed the app and created the account from scratch.

I did 4 things today, all with the same result, which is sorted by sent date/time:

1.  changed the setting to date/time sent, backed up my settings, deleted the account, restored the settings, confirmed the change was still there, synced my messages.

2.  changed the setting back to date/time received, backed up THOSE settings, deleted the account, restored the settings, confirmed the change was still there, synced my messages.

3.  Deleted the account, created it new, confirmed the setting was date/time received, synced the messages.

4.  Deleted the account, created it new, changed setting to date/time sent, synced the messages.

All yielded the same result.

I have one message that I checked to confirm result.  Aqua Mail shows it as 10:02AM, it actually showed up at 2:12PM.   

I have attached the message header from my "test" message.

[mikeone]

...
I have one message that I checked to confirm result.  Aqua Mail shows it as 10:02AM, it actually showed up at 2:12PM.   

I have attached the message header from my "test" message.

Yes, showing 10:02AM is correct, since the receiving time for this message is:

      "Date: Sat, 31 Oct 2015 10:01:30 -0400 (EDT)"

You will find this time stamp a few lines below the yellow marked time:

Received: from W-SMTP-03.local (w-smtp-03.signalgroup.local [192.168.10.26])
                 by email6.solidsignal.com (Postfix) with ESMTPS id CEE84503F51
                 for <bfittes@cinci.rr.com>; Sat, 31 Oct 2015 10:01:30 -0400 (EDT)

 "EDT" is your local time zone:
(Eastern Daylight Time) = UTC/GMT -4 hours

Which means that the "EDT" time will be shown as the "time received" for this message > "10:01:30" or even "10:02AM".

[bfittes1]

I respectfully disagree.  The initial mail server to receive that message did indeed receive it as you noted below.  But clearly, it was relayed to two other mail servers, with times of Sat, 31 Oct 2015 18:12:26 +0000, the final one being a roadrunner server, and in fact being the time that it was received in my inbox.  I have attached the header again with those times highlighted.

I am also attaching a screenprint from my Microsoft Outlook inbox that shows both the sent and received times highlighted for that message and one other (Sears) that had distinctly different sent and received times.  In both cases, Outlook appears to have accurately captured those times, but I don't seem to be able to get AquaMail to process the actual received time, as it shows up on MY mail server.

Also in both cases, the initial mail server to receive those messages reflect receive times about the same as sent times.  But then they are relayed to other servers before finally being received in my inbox at the time noted in the outlook screen print as received time, with big gaps in between.

Could it be that you aren't processing multiple receive times accurately, and it only shows up when there is a big gap between the initial mail server receipt and the final one?

[mikeone]

Thanks for your feedback and explanations.

Could you please be so kind to attach also the header from the other message (from Sears)... just to get  another comparison.

[bfittes1]

Attached. 

Note one thing that should make your head hurt.  It shows a sent time of 08:32:44, and an initial received time of 08:30:23.

As I am fond of saying, the speed of light ain't what it used to be.

[StR]

@bfittes1:
Yes, your reading of the "Received:" headers is correct.

Note one thing that should make your head hurt.  It shows a sent time of 08:32:44, and an initial received time of 08:30:23.

.
That's very simple: some of the computers do not have their time synchronized. Based on the combination of the facts, I'd conclude it is likely the computer that generated the e-mail. Note, that whatever is in the "Date:" header is essentially written by the program that created that message (e-mail client for e-mails sent by a person, an automated script/program for some mass-mailings like that). And in principle, they can write there whatever time they want.

As for your sorting/viewing of the time/date:
There are essentially (at least) three different times associated with the message:
1. "Date:" header formed by the mailer
2. The last (topmost) "Received:" header that is formed by the last SMTP server in the chain.
3. The time of when the IMAP server noticed the message in the Inbox (or any other mailbox, if on-server filtering/sorting is enabled). This can be IMAP-server dependent.

I am not 100% sure, but based on Kostya's comment and what is written in the FAQ under the item "Messages out of order, or wrong message dates" (Did you read it, as Kostya suggested?) http://www.aqua-mail.com/?page_id=227 , -  I'd say Aquamail is using #3 for the "received" time.
And as that FAQ answer describes, a number of things can affect that "received" time.

Additionally, from Kostya's remark, I suspect that Aquamail looks at that "received" time only once, when it downloads the message. (I.e. the setting for the time doesn't have retroactive force.)
From your description, I am not completely sure if you restore the messages from the backup, or they get downloaded each time. If you are restoring the entire database with the messages that have been downloaded by Aquamail, then the originally "recorded" time for the message (per Aquamail) is probably used in the sorting regardless of your setting.

HTH.

[bfittes1]

I am not sure how #2 and #3 are different.  I'd appreciate Kostya's perspective on that.  But from other stuff that I have read from Kostya, it seems to me that he is only pulling dates from the message header.  #3 seems to suggest that there is a date/time somewhere NOT in the header, and that doesn't make sense to me.

To clarify, each time I have re-synced my account, I have pulled from the IMAP server.  Not re-loading from a backup or something like that.

I have some other ideas for a test, but this still just doesn't look right to me.

[Kostya Vasilyev]

Re: The time of when the IMAP server noticed the message in the Inbox

Yes, and it's actually part of the IMAP protocol -- called INTERNALDATE. Not the SMTP backtrace ("Received:"), I use that one in POP3 accounts.

This is the one used by default.

The Date header is only used if you set the app up that way, in the account's options and folders.

Perhaps it would be best to do this:

Long press the account -> Diag: message dates -> choose the right folder -> run the test and send me the log.

[bfittes1]

ding ding ding ding.  I figured it out.

This line from your last message is key:

Yes, and it's actually part of the IMAP protocol -- called INTERNALDATE. Not the SMTP backtrace ("Received:"), I use that one in POP3 accounts.

I have my Outlook account (on my laptop) set up as a POP3 server.  When I set up my mail account on my phone AFTER the reset, I took the default setup for my account as AquaMail set it up, and that was IMAP.  I just created a new AquaMail account for the same email, but set it up as POP3. 

And what I NOW have in my Inbox is exactly what I have been looking for.

There may still be, I suppose, an issue with sent versus received in the IMAP implementation in AquaMail.

I understand POP is bad, IMAP is good.  But unless I can get my new mails at the top of my inbox, regardless of delays in delivery, then I think I am back to POP.

And I did send the log.

 

[Kostya Vasilyev]

Sounds like something's messing up INTERNALDATE values.

Not seeing the log for some reason, maybe I missed it, or didn't recognize it was about this issue...