Post by: Kostya Vasilyev on November 08, 2015, 08:38:41 pm
---
+ Added settings -> message list -> tap to open first message.
+ Conversations now considers subject: new subject means new conversation.
+ Fixes for conversation grouping in EWS accounts.
+ Please reindex conversations, settings -> message list -> conversation view, turn off and back on.
---
+ Добавил настройки - список сообщений - открывать первое сообщение при нажатии.
+ Группировка сообщений учитывает тему: новая тема, значит новая группа.
+ Исправления в коде группировки для учеток Exchange.
+ Необходимо пересчитать группировку, настройки - список сообщений, выключить группировку и включить обратно.
Post by: Kostya Vasilyev on November 08, 2015, 09:18:15 pm
I think -- separately from the visuals and interaction patterns -- this is what people are used to.
Not breaking conversations by subject change can be confusing, and there already is evidence of that:
http://www.aqua-mail.com/forum/index.php?topic=4086.msg21642#msg21642
Post by: Kostya Vasilyev on November 08, 2015, 09:39:49 pm
"
I now think that people do it all the time without knowing.
"can be confusing also (many examples can illustrate that"
But not so confusing -- because the broken out messages will still be there, and not having been linked, they'll be easy to see immediately.
As long as we use the same word -- "confusing" -- for both cases, I think we can agree that:
"hey, I see this message, but why is it not linked up" is less confusing than "where did that message go, I can't find it"
or putting it differently:
"this app sucks, it can't even properly group messages" vs. "this app sucks, I can't even find that message"
Post by: StR on November 08, 2015, 10:45:56 pm
There are examples of frequently used software that doesn't break threads based on the new subject.
Moreover, it might be hard I to handle all possible variations of "Re:" and "Fwd " in various languages.
Even worse, I've seen some mail clients striping a portion at the beginning that ends with a column.
Post by: Kostya Vasilyev on November 08, 2015, 11:41:04 pm
Hard but not impossible :)
Re: some mail clients striping a portion at the beginning that ends with a column
colon? ":"?
Don't think it's necessary to be that extreme.
Re: just tested IOS Mail (iPhone): it doesn't break conversation by subject
Great! So the score so far is:
Gmail, Hotmail, Mail.ru: new subject -> new conversation
iPhone Mail: new subject -> same conversation
3 : 1
Oh, and iPhone Mail doesn't show contact images. Should I now remove this too?
---
I raised a point about different kinds of "confusing", and haven't seen anyone respond to that.
Does this mean agreement -- that "confusing -> where the heck is my message" is worse than "confusing -> why is this message not linked up"?
That's my rationale, choosing the "less perfect for some but also less horrible when it runs counter to user expectations".
---
And yes, it's nice to have a setting for everything.
This -- consider subjects or not -- would mean full reindexing if it's changed. Not very nice.
Post by: mikeone on November 09, 2015, 12:19:20 am
Quote
Does this mean agreement -- that "confusing -> where the heck is my message" is worse than "confusing -> why is this message not linked up"?Kostya:
I agree with your "ranking", of course 🙋
Quote
And yes, it's nice to have a setting for everything.
This -- consider subjects or not -- would mean full reindexing if it's changed. Not very nice.
... too bad :'(
UPDATE:
... not too bad :D
Post by: Kostya Vasilyev on November 09, 2015, 12:42:13 am
Yes, but in -dev6 this is a full reindex only because I had to change some data structures.
Eventually, in "official release" version, I'll switch back to incremental -- which is much faster, only recomputes the messages "missed" while conversation was off.
A change in subject would need full reindex.
Post by: StR on November 09, 2015, 12:56:37 am
Re: some mail clients striping a portion at the beginning that ends with a column
colon? ":"?
Don't think it's necessary to be that extreme.
Err. colon and stripping.
My argument above meant to give an example what wouldn't be recognized by AM as a thread, because a portion of the subject was stripped away.
Re: confusion.
Yes I see some people "reuse" irrelevant messages for a new message. But similarly, people do that without changing subject. Both could be annoying, but you are not going to make the judgement based on the irrelevant content, are you?
IMHO, no matter what you do, people will be confused. ... sometimes even about being confused...
But the app behavior should be consistent: if you determine threads by the message-id, and do not use the subject, do not use the subject for some conditional decisions, especially since you are bound to have misses with various languages, combinations of pREfixes with mailing-list-based labels, etc...
E.g., Re: {list} Av: {list} НА: subject (fwd)
(Square brackets instead of the curly ones. Had to do it because of the forum limitation. )
And the some of those are removed here and there...
because the conversation participants are with different default languages...
Post by: Kostya Vasilyev on November 09, 2015, 01:19:27 am
Here in Russia everyone is used to Re and Fwd -- but Hotmail uses localized prefixes.
So when I saw "Отв: test 1" last night ("Отв" - "Ответ" - response, answer as a noun), it almost made me fall under my desk.
---
Re: you are not going to make the judgement based on the irrelevant content, are you
Not sure I got that, "irrelevant content"?
Subject is now a relevant factor in message linking logic.
I don't have plans for any additional heuristics based on recipients or dates or anything else.
Partly because of performance, partly because I want to be "done" with it, and finally, the logic needs to be simple enough for me to be able to explain it to users (if they ask).
---
And yes, people do "things", and sometimes don't even know it.
Which brings us back to:
The priority right now is to avoid *horribly messing up* the user's inbox to the point where he/she *is unable to find a message*.
StR's comment is excellent: "no matter what you do, people will be confused".
And so, taking this as a given, that it's not going to be "perfect" for everyone's expectations (especially expectations that one isn't even consciously aware of!) -- this is the right thing to do.
---
Finally, has anyone actually run into some horrible confusing situation because some messages, with different subjects, didn't get grouped?
Post by: StR on November 09, 2015, 07:18:35 am
Back to -4, the problems are gone.
I will describe the problems tomorrow, from the laptop.
Post by: Iceman_jkh on November 09, 2015, 12:55:27 pm
I'm not suggesting the dissenting opinions/views on grouping aren't valid however.
I would agree that an option would be optimal, but that's much easier for me to say than for Kostya to implement effectively. Keeping clear logic, control and consistency over the msg grouping display while also protecting AM and some users from themselves (eg: Google Play Review: 'This dumb app can't sort...my msgs are gone! 0 stars.') might be tough. :/
Re: (re)indexing for "Include SUBJECT in Conversation Grouping Criteria: YES/NO". I would expect that once the user decides how they want their grouping, they would leave AM it set that way and not touch it again - let's presume the default is set to ON. Therefore, there sh/would hardly ever be any reindexing beyond the very first time the choice is made. On a side note, is there any chance of including the option as a 'Developer Mode/Advanced Mode' setting? Something that requires a little 'digging' to find - you might even have other options you have been saving up and want somewhere to place them all?
That way, subject grouping can be configured if the user knows what they're doing (and looks hard enough/searches the web), but 90% of the users (eg: average joe) will simply have no problem with it being set to default of ON. Any thoughts?
Post by: papete on November 09, 2015, 01:36:40 pm
Post by: pyler on November 09, 2015, 03:22:39 pm
Post by: Iceman_jkh on November 09, 2015, 05:26:27 pm
Dear user,
Are you able to post screenshots (dev1.4 compared to dev1.6) to let us understand the improvement you get with dev1.6? I am not understanding the issue you have faced, since I was not able to reproduce it.
Hopefully this works.
The first message group (6 msgs) (at the bottom) was me sending (CC myself) someone a request (4 msgs total) to quote and them replying (2 msgs total).
The second message group (2 msgs) (in the middle) was them sending to me (1 msg) and me acknowledging receipt (1 msg).
The last message group (7 msgs) (at the top) me sending (CC myself) (2 msgs) and them acknowledging (1 msg) and then me sending (CC myself) more (4 msgs).
A few things to note:
- I have had to redact a large amount of information in these images, so hopefully you can still follow what's going on.
- Sender@gmail represents my email address (modified for privacy).
- The white boxes with numbers are redacted message body content. Where numbers are identical it represents identical content. Why identical content?....Because I have an auto CC/BCC back to the sender@g for all mails sent via this (sender@g) email address. Please note that the numbers are NOT chronological. They simply started at 1 (at the top) and I kept incrementing as necessary as I worked my way down.
- As you can see, previously (under dev 1.4) the messages were all incorrectly grouped under the wrong header (Drainage...) and wrong date (21 Oct 15)
- AM dev-1.6 correctly groups these emails into 3 separate conversations (as they should be, and which is also consistent with gmail's grouping style).I can't show an image as it's nearly 1:30am (bedtime!)
- I use smart folder, slim padding, dark theme, no chips :)
Hopefully that shows you the problem I had.
Post by: stevealb on November 09, 2015, 05:46:21 pm
Post by: StR on November 09, 2015, 09:02:05 pm
Languages. I love them, but as a developer, wish everyone used English.
Here in Russia everyone is used to Re and Fwd -- but Hotmail uses localized prefixes.
So when I saw "Отв: test 1" last night ("Отв" - "Ответ" - response, answer as a noun), it almost made me fall under my desk.
I am subscribed to an international mailing list, and every so often, I see some national analogues of "Re:". E.g. from one author from .dk, it comes as "SV:".
Then what happens frequently is that there is a chain looking like:
Subject: Re: SV: RE: Fwd: Nonsense (fwd)
Some programs are trying to avoid this long chain (Re: Re: Re: Re:...), but
when it is many languages they can fail..
And what I meant to say yesterday, some e-mail programs, consider that a subject like
"Subject: OT: viral video" ("OT" for "Off Topic")
contains a response prefix from a different locale, and in response to that writes not "Subject: Re: OT: viral video", but rather "Subject: Re: viral video".
And they even do that with some longer, 3-4 letter words with a colon at the end
(PESO:, FSF: ... )
And then, when it goes through a mailing list, if the mailing list adds [ListName] to the subject, and different mail clients work differently with that which ends in something like that:
Subject: [ListName] Re: [ListName] Re: Whatever
I am also getting some messages from one Russian source that use Russian "НА:" as a prefix. It is either from some Outpook or possibly web-based interface interface to a corporate Exchange mail.
I am looking at one of the actual e-mail subjects:
Subject: Re: HA: HA: Re: HA: HA: HA: photo
Additionally, the "message read" confirmations (I hate those), - add the prefix "Read:" to the subject. And once, when the conversation was about those confirmations, my correspondent was replying to the confirmation message, to show how it looks... It was a weird chain of "Read: Re: ..."
Now, one more realistic business situation.
There is a discussion/conversation with a Subject: Threading error.
Then someone adds: "Subject: Resolved? Was: Threading error.", continuing the the same discussion.
Finally, once in a while, a person accidentally types an extra character in the Subject field or accidentally erases the last character or two (I've done that myself a few times).
So, basically, all these cases are indicating that if you wanted to do subject-matching, you need to implement some fuzzy matching.
Quote
Re: you are not going to make the judgement based on the irrelevant content, are you
Not sure I got that, "irrelevant content"?
Well it was an attempt at a semi-joke...
How exactly you match all those weird subject is a question about figuring out what is a change in the subject and what is not.
So, I was suggesting to extend the same approach to the message body, - in case some {ab}users respond to the old thread without changing the subject, while writing about totally new (thus, irrelevant) matter.
There are some heuristic algorithms that are trying to deal with matching the e-mail content in combination with the Subject (I believe those are implemented by mail-archive.com engine.) mail-archive.com seems to recognize conversations and make decisions on the conversation tree arrangement based on: (a) "message-id:" references, (b) "In-reply-to:", and, if those are missing, (c) combination of "Subject:" with some matching of the quoted text...
If there is matching subject, but no quoted text, it doesn't add that message to the existing thread.
But I agree, let's NOT venture in that area!
Quote
Subject is now a relevant factor in message linking logic.
I don't have plans for any additional heuristics based on recipients or dates or anything else.
Partly because of performance, partly because I want to be "done" with it, and finally, the logic needs to be simple enough for me to be able to explain it to users (if they ask).
Agree, and I think what several people here are suggesting to drop even the Subject, because that has too many "edge" cases.
Quote
The priority right now is to avoid *horribly messing up* the user's inbox to the point where he/she *is unable to find a message*.
I agree, but ...
I didn't care much for the conversation view prior to starting using it. But I got hooked on it with dev1 through dev1.4.
Last night dev1.6 broke 2 out of 4 or 5 current threads in the most active account. And I understand you are trying to avoid that too:
Quote
Finally, has anyone actually run into some horrible confusing situation because some messages, with different subjects, didn't get grouped?
I had this 7-messages-long thread. It was exchange with the support of box.com (which is handled via zendesk.com engine).
It got all separated under -dev1.6. And some of those seem to share the subject..
Yet another thread that got separated is less important, but still, it is not what I'd expect...
Kostya I will try to e-mail you the messages for analysis...
Post by: StR on November 09, 2015, 09:30:42 pm
I understand that Aquamail is using "In-Reply-To:" and "References:" (together with "Message-ID:") headers for assembling the conversation threads.
It looks to me that it doesn't check the cross-reference between those two themselves, but relies on the "parent" Message-ID being present.
Here is what I mean
Message 1:
Message-ID: MM1
Message 2:
In-Reply-To: MM1
Message 3:
In-Reply-to: MM1
Now, if Message 1 is missing, Messages 2 and 3 wouldn't get grouped into one conversation. I don't think that should be the case. I think Aquamail should be cross-matching In-Reply-To: (as well as "References:", and probably even across the two types of headers).
It seems obvious why, but if it is not, - I can provide real-life examples.
Post by: mikeone on November 09, 2015, 10:39:11 pm
In addition to StR's (and Paris Geek's) explanations and examples I would also prefer grouping messages by referenced ID's rather than by subject.
Just today I saw a mail conversation with modified subject lines:, but all related together with send > reply > reply > reply > reply >...
1. Company's Insurance program
2. AW: Company's Insurance program 1/1/2016
3 RE: AW: Company's Insurance program 1/1/2016
4. RE: Project #162072 | RE: AW: Company's Insurance program 1/1/2016 - General Liability
Reworded:
So, I expect that these messages should be grouped into one conversation. But with the "new" dev1.6 logic this wouldn't be the case.. . :-(
Another scenario could be that security programs will add a tag in subject lines like;
[POSSIBLE VIRUS], [VIRUS ERROR], [Suspected Spam],... whatever...
Therefore I recommend to leave the choice by the user with an option in Settings > Message view > :
Conversation view
Combine related messages by:
( ) subject line
( ) referenced message ID
> subject changes will be ignored <
Yes, I changed my mind...
Updated post:
http://www.aqua-mail.com/forum/index.php?topic=4100.msg21789#msg21789
Kindest regards
Mikeone
Post by: mikeone on November 09, 2015, 10:56:49 pm
> So, I expect that these messages wouldn't be grouped into one conversation with the "new" dev1.6 logic.Yes, indeed.
Mikeone, I suppose that you are asking that those messages be grouped, aren't you?
Sorry, it seems that this wasn't "self-explanatory" enough ... 8)
Post by: mikeone on November 09, 2015, 11:18:15 pm
Yes, the good choice here is to have a user option to select the Conversation grouping method:+1
- break conversation by message subject
- do not break by message subject
With a reminder that full reindexation (possibly lasting minutes) will be processed following the choice/change.
Personally, I will definitely choose not to break by subject, leaving the other option to those who like to take risks with broken conversations ;)
Exactly. That would really be the best solution to make "all" users happy.
Post by: Iceman_jkh on November 09, 2015, 11:41:37 pm
@Iceman_jkh
Thank you very much for your post. You are a real supporter of AquaMail, and you proved it by spending valuable time gathering the information and packaging it :)
So you're in Smart Folder (not in Inbox as you previously stated), with a recipient (and yourself) changing the subject of a message from a previous conversation to launch a new conversation -- instead of using "create a new message" (the pen icon). And you're not using desktop Gmail (windows) which would create a new conversation when the user changes the subject, as explained in a previous post.
Indeed, in that case, AquaMail dev1.4 would consider all those messages (sent/replies) as part of one conversation (because it's one conversation from a technical point of view), unless the subject is taken into account (that's what dev1.6 does). In dev1.4 you'll have all messages in one thread, and in dev1.6 you'll get 3 different threads (in Smart Folder).
In dev1.4, the messages of September had not disappeared, they are simply grouped in the conversation. I understand that you recipient has changed the subject of a reply to create a new conversation, but what I am seeing is that you also did the same for the third conversation. So you really need that subject be considered in order to get your 3 conversations separated in 3 threads, because not using "create a new message" to create a new message in a new conversation.
Edit: k9, TypeMail don't break conversations by subject. Gmail and Hotmail do.
I think I had the same results when using inbox (just that in that pic I'd used smart folder).
The trouble is, sometimes you can't control whether the recipient/respondent creates a 'new' email by changing the subject or via the pencil icon.
When Kostya and I debugged we found a link between group TOP and group MIDDLE, but I honestly don't know how /why it's there (could have been one of the other users of that mailbox sending a 'new' email by only changing the subject
) I partly agree with your logic behind grouping, but still can't explain why group MIDDLE and BOTTOM were linked (unless they had a similar issue, but Kostya didn't mention it to me because finding the problem once [TOP vs. MIDDLE] gave him enough info to figure out the problem).
Cheers.
Ps. I agree with your (and others) logic for how grouping should work, but IMHO, k9 and TypeMail are using flawed logic too. On one hand, where everyone uses email perfectly the argument for ignoring subject change makes sense, but on the other, email is used by humans, so there will always be (many) exceptions, and therefore compensating/accounting for these could be argued to be a more relevant logic (because it yields a better result/experience for the AM user at the other end). Clearly however, we're all AM users, and have different ideas of what our perfect/better experience should be
- and we are all entitled to (within reason 
) Sent from my SM-G920F using Tapatalk
Post by: Kostya Vasilyev on November 10, 2015, 12:11:27 am
Thank you, very nice and logical, but I'm trying to (having to) look at this from a more practical point of view.
And this practical point of view is -- you actually ran into a real issue where you thought the message was missing, and it took you a lot of effort to finally find it, and in a place you didn't expect.
The "new subject -> new conversation" rule solves this, and there won't be other cases like this with other users (who might not be as nice as you were, posting the issue, helping me understand...)
On the other hand, the "let's not break on subject change" so far has not, to my knowledge, resulted in any serious consequences.
A setting for this is possible, but right now I'm not convinced that it's really necessary, or that it should be a priority.
Post by: StR on November 10, 2015, 05:24:03 am
Just today I saw a mail conversation with modified subject lines:, but all related together with send > reply > reply > reply > reply >...
1. Company's Insurance program
2. AW: Company's Insurance program 1/1/2016
3 RE: AW: Company's Insurance program 1/1/2016
4. RE: Project #162072 | RE: AW: Company's Insurance program 1/1/2016 - General Liability
...
Another scenario could be that security programs will add a tag in subject lines like;
[POSSIBLE VIRUS], [VIRUS ERROR], [Suspected Spam],... whatever...
Those are two great examples, Mikeone!
On the other hand, the "let's not break on subject change" so far has not, to my knowledge, resulted in any serious consequences.
A setting for this is possible, but right now I'm not convinced that it's really necessary, or that it should be a priority.
It depends what you call "serious consequences". If I am expecting a response inside the conversation, and it is not there, despite the correspondent claiming he responded a day ago, - that's as serious, as not being able to find a message that is inside the collapsed conversation. (After all, one situation is achieved from the other by the inversion of space. :) )
I perfectly understand the frustration of a person who cannot find a message that is hidden inside a collapsed conversation. But to a large extent, that is similar to the situation when one writes a new e-mail as a response and doesn't even change the subject (that was my earlier point). And the same people would likely send both types of mis-responses. {*}
I have a colleague who is notorious about picking some old e-mail and replying to it with a totally new content, often without changing the subject. In that case, with a conversation view enabled, I am risking missing a message from him inside the conversation in which I am not immediately interested at the moment.
So, being potentially affected by both situation, - how would I resolve this?
If the "reply-abusers" are frequent for the particular account, I would disable the threaded view altogether. But then, for the accounts that have many/several mailing list subscriptions or project-based spontaneous conversations, - I would enable the conversation view. There are two primary reasons for that: (1) to be able to view the entire conversation together, and (2) to avoid filling up the message list view during the "bursts" on one of those lists/conversations.
Actually, this thinking convinced me about the need for per-account configuration for enabling/disabling conversation view.
I've been also thinking about a "toggle view" button (easily reachable, not buried in the settings). For a desk-/laptop based program, I 'd want to have that one. But I am not convinced if I want to have it on the phone... - primarily due to the real estate limitation. (Or maybe, it should be enabled with a two-finger-spreading (one -up, second-down) gesture?)
However, for myself, I would not consider "break thread for new subject" solution because it is not effective against {*} above.
Post by: Kostya Vasilyev on November 10, 2015, 10:06:39 pm
Quote
If I am expecting a response inside the conversation, and it is not there, despite the correspondent claiming he responded a day ago, - that's as serious, as not being able to find a message that is inside the collapsed conversation
But "expecting a response inside the conversation" will not keep you from seeing the folder's message list -- because luckily, the granularity unit for "looking at messages" is a folder, not a conversation.
And the response will sit there at the top (sorted by date/time), right there.
In the second scenario, the new message is pretty easy to miss.
Post by: StR on November 10, 2015, 11:03:50 pm
Not quite. You are coming from a different situation.QuoteIf I am expecting a response inside the conversation, and it is not there, despite the correspondent claiming he responded a day ago, - that's as serious, as not being able to find a message that is inside the collapsed conversation
But "expecting a response inside the conversation" will not keep you from seeing the folder's message list -- because luckily, the granularity unit for "looking at messages" is a folder, not a conversation.
And the response will sit there at the top (sorted by date/time), right there.
That's only true if you are looking at freshly loaded messages. If you are looking at the conversation spanning over several days (and that's where the conversation view is especially helpful!), getting outside of it wouldn't show the message that had arrived 2 or even 8 days ago: to get to those you'd have to scroll and scroll... And if you didn't know about the very existence of that message, assuming everything is linked together, would you actually try looking for it?
Post by: Kostya Vasilyev on November 10, 2015, 11:58:40 pm
You're right, I didn't think of that (personal habit -- catch up on email every night, even if it's 2am :) )
Post by: papete on November 11, 2015, 09:38:25 am
I remember that a couple of years ago I bought the app because it was the only one that gave me a proper tasker integration (options), and adding the option to select the conversation sorting method would also make it the only one that gives the user the power to select this. So I think it is in the very spirit of the app to make this available.
Post by: mikeone on November 11, 2015, 10:46:36 am
I think clearly the best solution would be to give the user the option to select the method. The default method could be by subject if you think that it would give you the least troubles, but AquaMail has always been about options. The app has the biggest customization features and options I've seen in an email app and that is its strength.+1
I remember that a couple of years ago I bought the app because it was the only one that gave me a proper tasker integration (options), and adding the option to select the conversation sorting method would also make it the only one that gives the user the power to select this. So I think it is in the very spirit of the app to make this available.
Thanks for your thoughts, Papete.
Post by: Kostya Vasilyev on November 11, 2015, 08:26:53 pm
Oh, K9 Mail can't show Sent and Incoming messages in a combined list?
Anybody know about "the other apps" -- TypeMail, Apple iOS Mail, Boxer? What about non-Gmail accounts in Gmail app?
Post by: StR on November 11, 2015, 09:52:49 pm
Post by: Kostya Vasilyev on November 11, 2015, 10:33:39 pm
Incoming type folders (the main Inbox, and others) will pull in messages from the account's Sent folder.
Sent folder will pull in messages from all the incoming type folders of same account.
Post by: Kostya Vasilyev on November 11, 2015, 10:37:37 pm
Quote
Apps breaking conversation by subject:
Gmail, Cloudmagic, Mailwise, Mailbox, Outlook
Not breaking conversation by subject:
Apple IOS Mail, boxer, Typemail, K9, KatMail
Not combining Sent and Received:
K9, Katmail
Thank you, very interesting...
Post by: Kostya Vasilyev on November 12, 2015, 11:39:51 pm