Bogus unread message count and total message count

[Beachbum]

Inbox message count in shows unread messages when there are none, and total message count is too large by the same amount. PC email program, and webmail confirm AM message counts wrong.

For a couple of months this over count has been 1, now is 2.  IMAP provider is Network Solutions. I tried recreating the account in AM but no change.

The only strange messages I have seen are like 'message not found' which I think occurs when AM is open and displaying an email, another client deletes the email, and AM refreshes. (or maybe not! hard to tell sometimes).

Seems others have similar issues, attributed to IMAP servers.

Log of one IMAP poll attached (account and server names REDACTED), 8 actual messages in inbox (all read), summary displays 2/10, Thunderbird on PC shows 8 emails all read.
The log file lists 10 messages on server, but 2 messages say "BODY: NO Cannot open" - which accounts for the discrepancy.

Why the server has become confused we can only guess. Not in our control.

Enhancement request: Display message counts (read & total) adjusted by 'Cannot open' in place of counts provided by server.

[Kostya Vasilyev]

The "BODY: NO Cannot open" sounds like something's pretty seriously wrong with the server.

I can see in your log that when AquaMail asks the server to count unread messages, it gets back data that amounts to 2 unread messages.

Unfortunately, I can't tell what the server actually returns, since your log was made without "raw data".

I can understand why, since you were going to post the log on the forum -- but it made it only half-useful.

Please either re-capture the issue with "raw session data" enabled and email it to me directly rather than posting here -- or post the result of "UID SEARCH UNSEEN UNDELETED" that comes after "SELECT INBOX".

About your request:

Are you serious? It's like asking a car manufacturer to make their engine deal with gasoline that has sand and water in it.

We have a mail server that's reporting inconsistent data, and is unable to access its own messages (the "NO Cannot open" issue).

The mail server should be fixed, not AquaMail. I don't necessarily mean the mail server's software -- sometimes it's the environment, a storage volume that's full, a file that's become corrupted, etc.

[Beachbum]

Log with "raw data" attached (server and user redacted). Today 6 messages, 1 unread. Reported as 3/8. Seems messages 1 and 2 cannot be opened (lines 207 - 210), there are no flags on these messages (lines 165, 166).

> it gets back data that amounts to 2 unread messages.
yes, but the state is 'currently corrupted', we don't know if we can rely on the flags as the message is corrupted; the messages could for example have been corrupted after being read. The inbox does not list corrupted messages, my suggestion is the counters should behave in the same way.

>Are you serious? It's like asking a car manufacturer to make their engine deal with gasoline that has sand and water in it.
Funny. But yes I'm serious, communications protocols check for errors all the time (AM does it when parsing messages from a server). The trick is graceful degradation on an error. In this case there can be zero apparent degradation by counting the messages actually loaded (easier than my original suggestion). The server is actually saying it is corrupt, which using the analogy is like the pump telling the car it needs a better fuel filter today.

>The mail server should be fixed, not AquaMail.
I understand why you say this, and you may even be "right". But finger pointing is not helpful. What if the server said rightly or wrongly the the client(s) put me in this corrupted state? The issue is not whose fault, but how to recover gracefully.

I point this one out to you because my other email clients Thunderbird, and webmail both display the correct number of unread messages. Change this and AM looks better.

-----------------

As to the cause, who knows?! When I first installed AM the error was zero, after a couple of weeks 1, the 2 after another month. Correlation is not causation. But since there was no errors at install, I wonder if it is not some interaction between multiple clients, server and perhaps Android activity states.

But it really doesn't matter just count the messages loaded, and the corrupted state is hidden; which means, to a user, AM works the same as other email programs.

[Kostya Vasilyev]

Thanks for the updated log. I only see one mail check in it, and this is what I see:

2014.07.25 16:28:26.553 -1000   AquaMail   [IMAP.206] Sending: kman21 UID SEARCH UNSEEN UNDELETED
2014.07.25 16:28:26.700 -1000   AquaMail   [IMAP_RAW.206] Data is <49>:
* SEARCH 8974 9690 9770
kman21 OK SEARCH done.

2014.07.25 16:28:26.700 -1000   AquaMail   [IMAP_RAW.206] Line: * SEARCH 8974 9690 9770
2014.07.25 16:28:26.701 -1000   AquaMail   [IMAP_RAW.206] Line: kman21 OK SEARCH done.
2014.07.25 16:28:26.701 -1000   AquaMail   [IMAP.206] Result for kman21: 0  SEARCH done., traffic: 49 read, 36 write
2014.07.25 16:28:26.702 -1000   AquaMail   [IMAP.206] Unread message count: 3
2014.07.25 16:28:26.702 -1000   AquaMail   [SYNC.206] Adjusted command batch size: 25
2014.07.25 16:28:26.703 -1000   ImapSyncByCountStrategy   Loading at most 25 messages from INBOX
2014.07.25 16:28:26.703 -1000   AquaMail   [SYNC.206] The account's sync deleted is true, delete plan is 0
2014.07.25 16:28:26.704 -1000   ImapSyncByCountStrategy   Creating a fetch check command for 1 to 8
2014.07.25 16:28:26.704 -1000   AquaMail   [IMAP.206] Sending: kman22 FETCH 1:8 (UID FLAGS)
2014.07.25 16:28:26.852 -1000   AquaMail   [IMAP_RAW.206] Data is <311>:
* 1 FETCH (UID 8974 FLAGS ())
* 2 FETCH (UID 9690 FLAGS ())
* 3 FETCH (UID 9692 FLAGS (\Seen))
* 4 FETCH (UID 9694 FLAGS (\Answered \Seen))
* 5 FETCH (UID 9712 FLAGS (\Seen))
* 6 FETCH (UID 9754 FLAGS (\Seen))
* 7 FETCH (UID 9766 FLAGS (\Seen))
* 8 FETCH (UID 9770 FLAGS ())
kman22 OK FETCH completed.


The server is reporting three unread ("unseen") messages.

The data returned by "SEARCH UNSEEN UNDELETED" and actual per-message flags are consistent with each other: check the UIDs, they match.

There is no reason for Aqua to "fix" anything, the data is consistent.

To reiterate: there are no inconsistencies wrt. unread message state in this log. The server reports three UIDs of unread messages, and their  individual state is unread too.

There are errors with the server being unable to return message data, so this is some sort of corruption, and it has to be dealt with not by posting here, but by calling the mail service's support and presenting evidence from the log file:

1 - messages marked read in web mail being reported unread over IMAP
2 - server not being able to return data for two of your messages

I will also address these:

>>> "finger pointing is not helpful"

Did you have to use a loaded expression, "finger pointing"?

Working with a debug log, and discovering something wrong with a mail server / service is not "finger pointing".

It's debugging (in the broad sense) to figure out what's wrong.

When I see that the issue is with the mail server, well, am I supposed to take the blame just to be polite?

>>> "The server is actually saying it is corrupt"

Yes, there is something wrong with your account / messages / server's database / cache / index file / whatever else may be causing it.

Again, nothing to do with Aqua. The server is unable to return message data for two messages.

The IMAP commands are valid -- they may be different from those used by Thunderbird, or Outlook, or K9, or CloudMagic -- but IMAP is a rich protocol.

The server has no problem responding to those commands and returning message data for all messages but two.

And so it's black and white. Something's wrong with the server / service / cache / index file /... and should be fixed there.

>>> "graceful degradation on an error"

The IMAP protocol does not have error correction. There is no redundancy, extra information, as with error correction coding (and "graceful degradation" is something else entirely, actually).

You know that the two messages for which the server can't return message data are marked read in web mail.

But what if they were marked unread, and the server was returning them as read?

The exact opposite of the current scenario?

It's easy to see that there would be absolutely no basis for Aqua to assume that those "NO Cannot open" messages are either read or unread.

>>> "the client(s) put me in this corrupted state"

The client did not put anyone, anything, anywhere in a corrupted state. You're doing the finger pointing now.

There is absolutely no evidence that Aqua somehow caused this "corrupted state", and no reason to think that it did.

Further, since a mail app does not access a mail server's database / index / cache files directly -- and works by issuing IMAP commands -- then, by definition, any corruption is the mail server's fault.

Note that I'm not agreeing that this corruption is somehow caused by Aqua. There is absolutely no reason to think so, and no evidence to support this fantasy.

>>> "Correlation is not causation"

Exactly.

So when Yahoo let their SSL certificate for SMTP expire...

.. and when Apple broke SASL PLAIN authentication for IMAP

.. and when Office 365 broke all kinds of IMAP authentication

.. and when Gmail stopped sending those five-minute keep-alives for push mail

.. and when O2.pl was not returning any unread messages after a server "upgrade"

... ..... ..... ....

Since there were people using Aqua with those mail services at the time, was all of the above also caused by AquaMail?

While we're at it, let's not forget the hurricanes in Florida and the earthquakes in Indonesia.

-----

PS - Not trying to tell you what you should and should not do... but you might have had the issue resolved by now, if you contacted the mail service's support.

[Beachbum]

I'll take that as a 'no'.      Issue closed.

[Kostya Vasilyev]

What I was trying to say -- things should be fixed where they're broken.

And yes, I wrote too much there (a large cappuccino late in the evening)