Author Topic: "Date:" displayed by Aquamail  (Read 7018 times)

StR

  • Hero Member
  • *****
  • Posts: 1558
"Date:" displayed by Aquamail
« on: November 10, 2015, 02:52:32 am »
Kostya,

We have recently discussed which dates Aquamail can use for the purpose of message sorting. To my surprise, I discovered today that Aquamail actually displays that date as well.
So, in case the sorting is done by the time the IMAP server sees the message (INTERNALDATE), it seems like Aquamail does not display what is in the "Date:" header, but rather displays INTERNALDATE.

I think, this is the first e-mail client that I've seen that does this. Some programs can be set to show both in the "message-list" view. But I haven't seen any that shows it in the header form (thus effectively modifying that header). -- But then I am not using Outpook and many other programs.

But, I think this is confusing.
"Date:" header is a shared reference between the sender and recipient.  When the sender tells the recipient, in the message I sent to you on November 1 (or at 10 am), the recipient should be able to see THAT time. (The only exception could be the translation into the local time zone, but even that is arguable.)
So, I think displaying "Date: $INTERNALDATE" is wrong.
I recently had a situation when the mail server was down for some time. So, the messages were accumulated at an MX server or waiting at the original SMTP server and were delivered 1-2 days later. So, the message sent, say, on November 1, shows as Date: November 3.

I understand that the rational behind this decision was to avoid users being confused why the date shows out of order.
In fact, for those people who are used to correspond with people from various time zones, seeing a messages sent from "Tomorrow" is not a surprise.
But even when programs translate the date for the time zone, they never change the date.

PS. I know that I can see the original "Date:" in the View -> Headers, but it is not how you'd be looking for a particular e-mail from your correspondent.

Kostya Vasilyev

  • Hero Member
  • *****
  • Posts: 12740
Re: "Date:" displayed by Aquamail
« Reply #1 on: November 11, 2015, 08:13:38 pm »
"Date" used to be the default, and it's bad because Date is:

- Set by the sending app
- Can be wrong, on the sending side
- Cannot be changed on the receiving side

What this means in practice -- I'd get support email like "why is there a message from January 1, 1970 (or July 15, 2017...) in my mailbox, and how do I fix it".

INTERNALDATE is assigned by the server, and is less likely to be grossly wrong like that.

Most messages are delivered within seconds to minutes (with some rare exceptions), so I really don't think there anything lost here.

There is one exception -- people migrating between email servers and uploading old messages -- which usually sets INTERNALDATE to when a message was uploaded (not all servers do this, some are smarter).

For this exception, there is a setting to use the Date header and it's described in the FAQ.
Creating debug logs for diagnostics: https://www.aqua-mail.com/troubleshooting/

The official FAQ: https://www.aqua-mail.com/faq/

Лог-файлы для диагностики: https://www.aqua-mail.com/ru/troubleshooting/

Вопросы и ответы: https://www.aqua-mail.com/ru/faq/

StR

  • Hero Member
  • *****
  • Posts: 1558
Re: "Date:" displayed by Aquamail
« Reply #2 on: November 12, 2015, 12:28:11 am »
"Date" used to be the default, and it's bad because Date is:

- Set by the sending app
- Can be wrong, on the sending side
- Cannot be changed on the receiving side
The same 3 arguments are applicable to "From:" field, but you are not rewriting that?

Quote
What this means in practice -- I'd get support email like "why is there a message from January 1, 1970 (or July 15, 2017...) in my mailbox, and how do I fix it".
And often, I hear people saying:
"Why does this mail (spam) tell me that it was sent by me? I didn't send it!..."

There is bad (or malicious) software on the sending side. I know that you are doing a lot in various areas to compensate for some shortcomings of other programs, servers, etc.
But I think this is not correct here.
As I wrote above, Date: is one of the very few headers shown by default by all e-mail programs that allows to identify the message (from a particular sender) almost uniquely.
And it frequently used in the business world: "In your message of November 17, 10:15am (EST), ..."


Quote
INTERNALDATE is assigned by the server, and is less likely to be grossly wrong like that.

Most messages are delivered within seconds to minutes (with some rare exceptions), so I really don't think there anything lost here.

There are many exceptions.
Let me name a few:
1. When greylisting is enabled on the receiving servers (as an anti-spam measure), messages can be arriving half an hour later.
(I am not a proponent of this solution, but I've had it enabled by my previous employer.)
2. When the messages are sent by the sender's device at a later point (e.g. a person writes messages in the absence of connectivity, and then they are queued for sending). This happens all the time: when people are working during flights, or even during a ground trip with the device that does not have mobile connectivity (laptop, tablet, ..).
The discrepancy can be several hours.
3. There could be some maintenance window (scheduled or emergency) when the mail could be queued for hours at an MX server.
4. Mail quarantine at the dedicated spam-scanning servers at a large organization periphery. After you release a message from the quarantine, it will have essentially the date of when you released it. Since in some cases you can only do that from the internal network, if you are traveling, you don't get to look through the quarantined messages until you are back. (Yes, there is a VPN, but that's a different issue. And sometimes you cannot use VPN, especially from some public (NATted) networks or during international travel.)

...

All those things happen on a regular basis.

I think it is great that you enabled sorting based on INTERNALDATE! That makes perfect sense, at least for any folder with the incoming messages. (Sorting incoming messages based on "Date:" header can result in missed messages. I had had that struggle with other mail clients until I fixed the settings for sorting.)

What I am talking about is what date to display.
And I think that it should be the "Date:" header even when the sorting is done by INTERNALDATE.

Sorting and displaying date should be decoupled.
Sorting: optional (Sent Date, INTERNALDATE (received))
Display: Sent Date (as defined in RFC5256 https://tools.ietf.org/html/rfc5256#section-2.2 )


I had had some problems with Aquamail earlier, while I was trying to find a message from a sequence of several messages. In one case there were some 10 messages or so. The sender was referring to a particular message by the Date, and I couldn't understand why it was not there. I had not had time to investigate, and a few other cases it was somewhat easier to locate the message. So, I hadn't figured it out until this week, when I was looking at the messages that we discussed via e-mail.
In this, most recent case the Date: and INTERNALDATE were different by 1-2 days, and it became apparent what date is displayed.

Additional, in many (most?) cases INTERNALDATE does not exist in the message headers.  This makes matching the date displayed to the "raw" message very difficult.

« Last Edit: November 12, 2015, 12:39:58 am by StR »

Kostya Vasilyev

  • Hero Member
  • *****
  • Posts: 12740
Re: "Date:" displayed by Aquamail
« Reply #3 on: November 12, 2015, 12:47:16 am »
Re: "Why does this mail (spam) tell me that it was sent by me? I didn't send it!..."

I thought we were discussing date/time values, not From headers?

Re: Sent Date in RFC 5256

( note this is not the baseline IMAP RFC )

This RFC defines "sent date" based on the Date header. RFC 3501, "the" IMAP RFC uses same approach.

---

This mail app has a per-account option for which one to use.

It's "one" not "both, one here and the other there" because of some fairly obscure technical reasons which I won't bore you with.
Creating debug logs for diagnostics: https://www.aqua-mail.com/troubleshooting/

The official FAQ: https://www.aqua-mail.com/faq/

Лог-файлы для диагностики: https://www.aqua-mail.com/ru/troubleshooting/

Вопросы и ответы: https://www.aqua-mail.com/ru/faq/

StR

  • Hero Member
  • *****
  • Posts: 1558
Re: "Date:" displayed by Aquamail
« Reply #4 on: November 12, 2015, 01:26:21 am »
Re: "Why does this mail (spam) tell me that it was sent by me? I didn't send it!..."

I thought we were discussing date/time values, not From headers?

I am just drawing parallels. "From:" and "Date:" headers are two of just about 5 headers that any and all mail programs, including Aquamail, display. They are (almost unique) practical references to the messages for most users.


Quote
Re: Sent Date in RFC 5256

( note this is not the baseline IMAP RFC )

This RFC defines "sent date" based on the Date header. RFC 3501, "the" IMAP RFC uses same approach.

I used that only to refer to the exception when the displayed "Date:" could be different from the original header. That's just a side note, not worthy the digression.

Quote
This mail app has a per-account option for which one to use.
That's useful, as the sorting may need to be different. But it doesn't replace the need for the correct date displayed.

Quote
It's "one" not "both, one here and the other there" because of some fairly obscure technical reasons which I won't bore you with.
So, do I understand correctly that the reason for displaying INTERNALDATE instead of "Date:" is purely (or at least primarily) technical?
(I assume it is to avoid an extra request to the IMAP server, and thus speed-up the sync process...)

Ghm... Let me think about that...

Kostya Vasilyev

  • Hero Member
  • *****
  • Posts: 12740
Re: "Date:" displayed by Aquamail
« Reply #5 on: November 15, 2015, 01:27:28 am »
Re: do I understand correctly that the reason for displaying INTERNALDATE instead of "Date:" is purely (or at least primarily) technical

Yes, there is a technical reason why I am unable to keep both dates per message.

Re: doesn't replace the need for the correct date displayed

Correct date?

Displaying "January 1, 1970" or "July 15, 2025" would be displaying the correct date, and at the same time, useless.
Creating debug logs for diagnostics: https://www.aqua-mail.com/troubleshooting/

The official FAQ: https://www.aqua-mail.com/faq/

Лог-файлы для диагностики: https://www.aqua-mail.com/ru/troubleshooting/

Вопросы и ответы: https://www.aqua-mail.com/ru/faq/

StR

  • Hero Member
  • *****
  • Posts: 1558
Re: "Date:" displayed by Aquamail
« Reply #6 on: November 15, 2015, 07:43:53 am »
Displaying "January 1, 1970" or "July 15, 2025" would be displaying the correct date, and at the same time, useless.

I understand and appreciate the technical difficulty (and I suspect I know where in IMAP API it stems from).
But I respectfully disagree with the premise used for the justification.

So, we have two things happening:
1. In pathological messages (a very small minority), the correct date is useless. "Correction" (displaying INTERNALDATE) eliminates questions from users "Why do I have this weird date?"

2. In all normal messages (which are the majority), this "correction" removes the practical reference to the message for people who need it (which is a standard business practice). Moreover, in some cases, it may screw up somebody's legal procedure (where a wrong date of communication is referenced in the response). (Although, I don't know how much of such communication is done on the phone or even a tablet.)

So, what's more important? To help someone's confusion (in rear cases most of which for the message that is likely a spam anyway) at the cost of messing up somebody else's standard business procedure?

PS. BTW, because of this artificial date, matching the messages between Aquamail and a different e-mail client (alpine, Thunderbird) was rather complicated and very non-ergonomic (required viewing all headers for each message in Aquamail) in the exercise we just went through.

Kostya Vasilyev

  • Hero Member
  • *****
  • Posts: 12740
Re: "Date:" displayed by Aquamail
« Reply #7 on: November 15, 2015, 03:18:04 pm »
Re: In pathological messages (a very small minority), the correct date is useless

You keep calling the "Date:" header "correct" even when it's off by a few decades?

Re: So, what's more important

More important to me: not having users think (and write, and comment in Play):

"oh no, a horrible bug in this shitty app"

or

"how do I fix 1970 to when I actually received this message"

Re: because of this artificial date

Artificial?

Artificial would be something like System.currentTime() + Math.random() * 100000.

INTERNALDATE is not "artificial", it's when a message was received.

And btw, it also makes any just received messages pop up at the top of the list (again, user expectations).

Re: legal procedures

Legal procedures with a mobile mail app on a tiny screen screen? Are you joking?

A Real Corporate Lawyer, Esq. (tm), would not touch anything but Outlook (r)...

... or Special Certified Software (tm) that Connects Directly To Mail Servers in order to be Compliant With Established Legal Procedures (chapter 8.12 sections 6-18, appendix II, part 4).

But should the aboveforementioned Lawyer wish to do his work in AquaMail -- I'm sure he'd be Knowledgeable Enough to use Menu -> View -> Message headers, which is a Special Menu Command provided Just For The Purpose of Viewing Message Headers.

---

And to address your earlier "but all other mail apps..."

- Fastmail web mail: shows internaldate, not Date: header

- Gmail web mail: shows internaldate, not Date: header

- Gmail for Android (Gmail accounts): shows internaldate, not Date: header

On the other hand:

- Gmail for Android (non-Gmail accounts): shows Date: header, shows internaldate

OH NO, IT'S INCONSISTENT, SOMEONE REPORT THIS TO GOOGLE QUICK!!!! 1111 ALERT ALERT!!!!

AND MY JUST RECEIVED MESSAGE POPPED OUT SOMEWHERE DOWN THE LIST!!!! WHERE I COULDN'T SEE IT AND HAD TO SCROLL!!!!!!

THAT'S IT GOOGLE I'M SWITCHING TO THE IPHONE!!!!!!!!!1111!!!!11!!!!!

---

I understand that you have strong feelings about it.

Just change the damn option, it's there so it can be changed.

« Last Edit: November 15, 2015, 03:26:40 pm by Kostya Vasilyev, Aqua Mail »
Creating debug logs for diagnostics: https://www.aqua-mail.com/troubleshooting/

The official FAQ: https://www.aqua-mail.com/faq/

Лог-файлы для диагностики: https://www.aqua-mail.com/ru/troubleshooting/

Вопросы и ответы: https://www.aqua-mail.com/ru/faq/

StR

  • Hero Member
  • *****
  • Posts: 1558
Re: "Date:" displayed by Aquamail
« Reply #8 on: November 16, 2015, 10:18:53 pm »
Kostya, I am sorry, my arguments annoyed you.


Re: Just change the damn option, it's there so it can be changed.

As you know, there is no option for just displaying, it is linked to sorting.
In all the situations when well formed, bona fide messages get delayed with transmission, sorting by the header "Date:" can result in missed messages, as we both know. And that's why you created that option (I saw those old threads). 
That's why I will have to live with the smaller of the two problems in otherwise great Aquamail.  "Nobody's perfect". https://goo.gl/5B6ZDB  ;)

Anyway, thanks for listening!

PS.
... and there are plenty of reasons more serious than just INTERNALDATE why I don't use Gmail much.

Kostya Vasilyev

  • Hero Member
  • *****
  • Posts: 12740
Re: "Date:" displayed by Aquamail
« Reply #9 on: November 16, 2015, 11:21:43 pm »
Re: As you know, there is no option for just displaying, it is linked to sorting.

Yes. And there is a technical reason there, it's very difficult to go back and fix.

Sorry, my apologies, sorry again.
Creating debug logs for diagnostics: https://www.aqua-mail.com/troubleshooting/

The official FAQ: https://www.aqua-mail.com/faq/

Лог-файлы для диагностики: https://www.aqua-mail.com/ru/troubleshooting/

Вопросы и ответы: https://www.aqua-mail.com/ru/faq/