1.6.0-14-dev1.6 - conversation view / группировка сообщений

[StR]

Languages. I love them, but as a developer, wish everyone used English.

Here in Russia everyone is used to Re and Fwd -- but Hotmail uses localized prefixes.

So when I saw "Отв: test 1" last night ("Отв" - "Ответ" - response, answer as a noun), it almost made me fall under my desk.

I am subscribed to an international mailing list, and every so often, I see some national analogues of "Re:". E.g. from one author from .dk, it comes as "SV:".
Then what happens frequently is that there is a chain looking like:
Subject: Re: SV: RE: Fwd: Nonsense (fwd)
Some programs are trying to avoid this long chain (Re: Re: Re: Re:...), but
when it is many languages they can fail..

And what I meant to say yesterday, some e-mail programs, consider that a subject like
"Subject: OT: viral video"  ("OT" for "Off Topic")
contains a response prefix from a different locale, and in response to that writes not "Subject: Re: OT: viral video", but rather "Subject: Re: viral video".
And they even do that with some longer, 3-4 letter words with a colon at the end
(PESO:, FSF: ... )

And then, when it goes through a mailing list, if the mailing list adds [ListName] to the subject, and different mail clients work differently with that which ends in something like that:
Subject: [ListName] Re: [ListName] Re: Whatever

I am also getting some messages from one Russian source that use Russian "НА:" as a prefix. It is either from some Outpook or possibly web-based interface interface to a corporate Exchange mail.
I am looking at one of the actual e-mail subjects:
Subject: Re: HA: HA: Re: HA: HA: HA: photo

Additionally, the "message read" confirmations (I hate those), - add the prefix "Read:" to the subject. And once, when the conversation was about those confirmations, my correspondent was replying to the confirmation message, to show how it looks... It was a weird chain of "Read: Re: ..."


Now, one more realistic business situation.
There is a discussion/conversation with a Subject: Threading error.
Then someone adds: "Subject: Resolved? Was: Threading error.", continuing the the same discussion.


Finally, once in a while, a person accidentally types an extra character in the Subject field or accidentally erases the last character or two (I've done that myself a few times).

So, basically, all these cases are indicating that if you wanted to do subject-matching, you need to implement some fuzzy matching.

Re: you are not going to make the judgement based on the irrelevant content, are you

Not sure I got that, "irrelevant content"?

Well it was an attempt at a semi-joke...
How exactly you match all those weird subject is a question about figuring out what is a change in the subject and what is not.
So, I was suggesting to extend the same approach to the message body, - in case some {ab}users respond to the old thread without changing the subject, while writing about totally new (thus, irrelevant) matter.

There are some heuristic algorithms that are trying to deal with matching the e-mail content in combination with the Subject (I believe those are implemented by mail-archive.com engine.) mail-archive.com seems to recognize conversations and make decisions on the conversation tree arrangement based on: (a) "message-id:" references, (b) "In-reply-to:", and, if those are missing, (c) combination of "Subject:" with some matching of the quoted text...
If there is matching subject, but no quoted text, it doesn't add that message to the existing thread.

But I agree, let's NOT venture in that area!

Subject is now a relevant factor in message linking logic.

I don't have plans for any additional heuristics based on recipients or dates or anything else.

Partly because of performance, partly because I want to be "done" with it, and finally, the logic needs to be simple enough for me to be able to explain it to users (if they ask).

Agree, and I think what several people here are suggesting to drop even the Subject, because that has too many "edge" cases.

The priority right now is to avoid *horribly messing up* the user's inbox to the point where he/she *is unable to find a message*.

I agree, but ...
I didn't care much for the conversation view prior to starting using it. But I got hooked on it with dev1 through dev1.4.
Last night dev1.6 broke 2 out of 4 or 5 current threads in the most active account. And I understand you are trying to avoid that too:

Finally, has anyone actually run into some horrible confusing situation because some messages, with different subjects, didn't get grouped?

I had this 7-messages-long thread. It was exchange with the support of box.com (which is handled via zendesk.com engine).
It got all separated under -dev1.6. And some of those seem to share the subject..

Yet another thread that got separated is less important, but still, it is not what I'd expect...

Kostya I will try to e-mail you the messages for analysis...

[StR]

I think I just caught a potentially missed scenario (not related to the "Subject" issue -- it's present in dev1.4).

I understand that Aquamail is using "In-Reply-To:" and "References:" (together with "Message-ID:") headers for assembling the conversation threads.
It looks to me that it doesn't check the cross-reference between those two themselves, but relies on the "parent" Message-ID being present.
Here is what I mean

Message 1:
Message-ID:  MM1

Message 2:
In-Reply-To: MM1

Message 3:
In-Reply-to: MM1

Now, if Message 1 is missing, Messages 2 and 3 wouldn't get grouped into one conversation. I don't think that should be the case. I think Aquamail should be cross-matching In-Reply-To: (as well as "References:", and probably even across the two types of headers).

It seems obvious why, but if it is not, - I can provide real-life examples.

[mikeone]

Kostya:
In addition to StR's (and Paris Geek's) explanations and examples I would also prefer grouping messages by referenced ID's rather than by subject.

Just today I saw a mail conversation with modified subject lines:,  but all related together with send > reply > reply > reply > reply >...

1. Company's Insurance program
2. AW: Company's Insurance program 1/1/2016
3  RE: AW: Company's Insurance program 1/1/2016
4. RE: Project #162072 | RE: AW: Company's Insurance program 1/1/2016 - General Liability

Reworded:
So, I expect that these messages should be grouped into one conversation. But with the "new" dev1.6 logic this wouldn't be the case.. .  :-(

Another scenario could be that security programs will add a tag in subject lines like;
[POSSIBLE VIRUS], [VIRUS ERROR], [Suspected Spam],... whatever...

Therefore I  recommend to leave the choice by the user with an option in Settings > Message view > :

Conversation view
Combine related messages by:
  ( )  subject line
  ( )  referenced message ID
        > subject changes will be ignored <

Yes, I changed my mind...

Updated post:
http://www.aqua-mail.com/forum/index.php?topic=4100.msg21789#msg21789

Kindest regards
Mikeone

[mikeone]

> So, I expect that these messages wouldn't be grouped into one conversation with the "new" dev1.6 logic.

Mikeone, I suppose that you are asking that those messages be grouped, aren't you?

Yes, indeed.
Sorry,  it seems that this wasn't "self-explanatory" enough ... 

[mikeone]

Yes, the good choice here is to have a user option to select the Conversation grouping method:

- break conversation by message subject
- do not break by message subject

With a reminder that full reindexation (possibly lasting minutes) will be processed following the choice/change.

Personally, I will definitely choose not to break by subject, leaving the other option to those who like to take risks with broken conversations

+1
Exactly. That would really be the best solution to make "all" users happy.

[Iceman_jkh]

@Iceman_jkh

Thank you very much for your post. You are a real supporter of AquaMail, and you proved it by spending valuable time gathering the information and packaging it

So you're in Smart Folder (not in Inbox as you previously stated), with a recipient (and yourself) changing the subject of a message from a previous conversation to launch a new conversation -- instead of using "create a new message" (the pen icon). And you're not using desktop Gmail (windows) which would create a new conversation when the user changes the subject, as explained in a previous post.

Indeed, in that case, AquaMail dev1.4 would consider all those messages (sent/replies) as part of one conversation (because it's one conversation from a technical point of view), unless the subject is taken into account (that's what dev1.6 does). In dev1.4 you'll have all messages in one thread, and in dev1.6 you'll get 3 different threads (in Smart Folder).

In dev1.4, the messages of September had not disappeared, they are simply grouped in the conversation. I understand that you recipient has changed the subject of a reply to create a new conversation, but what I am seeing is that you also did the same for the third conversation. So you really need that subject be considered in order to get your 3 conversations separated in 3 threads, because not using  "create a new message" to create a new message in a new conversation.

Edit: k9, TypeMail don't break conversations by subject. Gmail and Hotmail do.

I think I had the same results when  using inbox (just that in that pic I'd used smart folder).

The trouble is, sometimes you can't control whether the recipient/respondent creates a 'new' email by changing the subject or via the pencil icon.

When Kostya and I debugged we found a link between group TOP and group MIDDLE, but I honestly don't know how /why it's there (could have been one of the other users of that mailbox sending a 'new' email by only changing the subject )

I partly agree with your logic behind grouping, but still can't explain why group MIDDLE and BOTTOM were linked (unless they had a similar issue, but Kostya didn't mention it to me because finding the problem once [TOP vs. MIDDLE] gave him enough info to figure out the problem).

Cheers.

Ps. I agree with your (and others) logic for how grouping should work, but IMHO, k9 and TypeMail are using flawed logic too. On one hand, where everyone uses email perfectly the argument for ignoring subject change makes sense, but on the other, email is used by humans, so there will always be (many) exceptions, and therefore compensating/accounting for these could be argued to be a more relevant logic (because it yields a better result/experience for the AM user at the other end). Clearly however, we're all AM users, and have different ideas of what our perfect/better experience should be - and we are all entitled to (within reason )


Sent from my SM-G920F using Tapatalk

[Kostya Vasilyev]

@Iceman_jkh:

Thank you, very nice and logical, but I'm trying to (having to) look at this from a more practical point of view.

And this practical point of view is -- you actually ran into a real issue where you thought the message was missing, and it took you a lot of effort to finally find it, and in a place you didn't expect.

The "new subject -> new conversation" rule solves this, and there won't be other cases like this with other users (who might not be as nice as you were, posting the issue, helping me understand...)

On the other hand, the "let's not break on subject change" so far has not, to my knowledge, resulted in any serious consequences.

A setting for this is possible, but right now I'm not convinced that it's really necessary, or that it should be a priority.

[StR]

Just today I saw a mail conversation with modified subject lines:,  but all related together with send > reply > reply > reply > reply >...

1. Company's Insurance program
2. AW: Company's Insurance program 1/1/2016
3  RE: AW: Company's Insurance program 1/1/2016
4. RE: Project #162072 | RE: AW: Company's Insurance program 1/1/2016 - General Liability
...

Another scenario could be that security programs will add a tag in subject lines like;
[POSSIBLE VIRUS], [VIRUS ERROR], [Suspected Spam],... whatever...

Those are two great examples, Mikeone!

On the other hand, the "let's not break on subject change" so far has not, to my knowledge, resulted in any serious consequences.

A setting for this is possible, but right now I'm not convinced that it's really necessary, or that it should be a priority.

It depends what you call "serious consequences". If I am expecting a response inside the conversation, and it is not there, despite the correspondent claiming he responded a day ago, - that's as serious, as not being able to find a message that is inside the collapsed conversation. (After all, one situation is achieved from the other by the inversion of space.  )

I perfectly understand the frustration of a person who cannot find a message that is hidden inside a collapsed conversation. But to a large extent, that is similar to the situation when one writes a new e-mail as a response and doesn't even change the subject (that was my earlier point). And the same people would likely send both types of mis-responses.  {*} 
I have a colleague who is notorious about picking some old e-mail and replying to it with a totally new content, often without changing the subject. In that case, with a conversation view enabled, I am risking missing a message from him inside the conversation in which I am not immediately interested at the moment.

So, being potentially affected by both situation, - how would I resolve this?
If the "reply-abusers" are frequent for the particular account, I would disable the threaded view altogether. But then, for the accounts that have many/several mailing list subscriptions or project-based spontaneous conversations, - I would enable the conversation view. There are two primary reasons for that: (1) to be able to view the entire conversation together, and (2) to avoid filling up the message list view during the "bursts" on one of those lists/conversations.

Actually, this thinking convinced me about the need for per-account configuration for enabling/disabling conversation view.

I've been also thinking about a "toggle view" button (easily reachable, not buried in the settings). For a desk-/laptop based program, I 'd want to have that one. But I am not convinced if I want to have it on the phone... - primarily due to the real estate limitation. (Or maybe, it should be enabled with a two-finger-spreading (one -up, second-down) gesture?)

However, for myself, I would not consider "break thread for new subject" solution because it is not effective against {*} above.

[Kostya Vasilyev]

If I am expecting a response inside the conversation, and it is not there, despite the correspondent claiming he responded a day ago, - that's as serious, as not being able to find a message that is inside the collapsed conversation

But "expecting a response inside the conversation" will not keep you from seeing the folder's message list -- because luckily, the granularity unit for "looking at messages" is a folder, not a conversation.

And the response will sit there at the top (sorted by date/time), right there.

In the second scenario, the new message is pretty easy to miss.

[StR]

If I am expecting a response inside the conversation, and it is not there, despite the correspondent claiming he responded a day ago, - that's as serious, as not being able to find a message that is inside the collapsed conversation

But "expecting a response inside the conversation" will not keep you from seeing the folder's message list -- because luckily, the granularity unit for "looking at messages" is a folder, not a conversation.

And the response will sit there at the top (sorted by date/time), right there.

Not quite. You are coming from a different situation.
That's only true if you are looking at freshly loaded messages. If you are looking at the conversation spanning over several days (and that's where the conversation view is especially helpful!), getting outside of it wouldn't show the message that had arrived 2 or even 8 days ago: to get to those you'd have to scroll and scroll... And if you didn't know about the very existence of that message, assuming everything is linked together, would you actually try looking for it?

[Kostya Vasilyev]

"the message that had arrived 2 or even 8 days ago"

You're right, I didn't think of that (personal habit -- catch up on email every night, even if it's 2am )

[papete]

I think clearly the best solution would be to give the user the option to select the method. The default method could be by subject if you think that it would give you the least troubles, but AquaMail has always been about options. The app has the biggest customization features and options I've seen in an email app and that is its strength.

I remember that a couple of years ago I bought the app because it was the only one that gave me a proper tasker integration (options), and adding the option to select the conversation sorting method would also make it the only one that gives the user the power to select this. So I think it is in the very spirit of the app to make this available.

[mikeone]

I think clearly the best solution would be to give the user the option to select the method. The default method could be by subject if you think that it would give you the least troubles, but AquaMail has always been about options. The app has the biggest customization features and options I've seen in an email app and that is its strength.

I remember that a couple of years ago I bought the app because it was the only one that gave me a proper tasker integration (options), and adding the option to select the conversation sorting method would also make it the only one that gives the user the power to select this. So I think it is in the very spirit of the app to make this available.

+1

Thanks for your thoughts, Papete.

[Kostya Vasilyev]

Re: - show in Inbox conversations received messages only (like k9)

Oh, K9 Mail can't show Sent and Incoming messages in a combined list?

Anybody know about "the other apps" -- TypeMail, Apple iOS Mail, Boxer? What about non-Gmail accounts in Gmail app?

[StR]

I haven't tried this yet, but I am curious if Aquamail can combine conversations between "other" folder and Sent, if the option "combine with Sent" is enabled.