Data usage management (and feature request)

[maaltan]

I'm sorry.  This will probably considered a repost but I've searched the forum and I've seen my question addressed several times, but a lot of the assumptions in the answers are now stale.

My question is:
Why is Aquamail consuming about 500mb of data a week?

I am currently trying to transition to Aquamail from Enhanced Email because it better supports the pebble watch's notification forward (it lists the email in notification instead of saying "X unread emails")

In EnhancedEmail, i had 2 google accounts (polling every 10 min) and an exchange account (server push) running.   With these settings EE uses about 120MB a week.

With AquaMail, I've moved the exchange and one google account.   The exchange account is poling ever 5 minutes.  The gmail account is using IMAP Push.  Data usage is running abut 400mb a week.

I have read that IMAP push (aka IMAP-IDLE apparently) is very wasteful and is probably the cause of my issues. I have disabled the account entirely it to see what affect it has.   My exchange account probably gets 200+ emails a day.  the google account i transferred only probably gets 5-10.

Also, as i mentioned before, I am trying to get my Pebble Time watch optimized.  (note all the following is conjecture and probably abusing some jargon) The way the pebble app seems to handle notifications is it links the notification object/id/whatever to the message it sends to the watch.  When you do a quick reply/dismiss on the watch, it does that action to the notification object.  Unfortunately, pretty much every app in existence now seems to delete and rebuild a collected notification for similar events.  Aquamail is no exception.

Therefore i would like to request a feature (or have it pointed out to me if ive missed it) so that Aquamail:
1.  Does not combine notifications (IE create one for each new email)
2.  Allow for alterations of the notification template to be more usable on watch (don't put entire to/from/cc/etc list in notification,etc, truncate accountname, etc)

THanks

[Kostya Vasilyev]

EWS -- Exchange -- is very verbose. Much more so than ActiveSync (which is binary, and EWS is XML based).

Second, a lot depends on how many messages you're asking the app to sync / keep fresh. The default is 25-50 per folders, but the setting goes to thousands, definitely should not be combined with frequent mail checks.

Three, same goes for IMAP, push or not (the "messages to sync"), and yes, it can be somewhat chatty too. But I suspect that the main culprit here is the Exchange.

Re: Pebble.

AquaMail does not directly support this watch (its software), it supports Android Wear.

It is my understanding that Pebble fairly recently made it so that it "emulates" Android Wear -- an app will "think" it's talking to an Android Wear device, and there is a Pebble watch there instead.

This "emulation" (if I'm right about its existence) is provided by Pebble. On "real" Android Wear devices, AquaMail does post the proper "stream of cards", one per message.

Therefore, if it's different on the Pebble, it has to be the fault of Pebble's "Android Wear emulation" and would need to be fixed there, not in AquaMail. I'm sure Pebble provides great support.

[speri]

Hi
I have the same problem since 2 months on my Samsung with Android 5.0.02. I consume almost 3 Gb per month, it's absurd. I don't download any attachment or image, I've limited the number of emails to synch. I have 7 accounts 2 of them exhange. what is strange that I never had this problem before. Anyone can help me ?
Sandro

[StR]

I am not sure how accurate it is, - but when I was curious about data usage by apps, I've used:
Bytes Insight: https://play.google.com/store/apps/details?id=com.oasisfeng.bytesinsight
I've also used (and I am still using it, albeit an older version) Onavo Count: https://play.google.com/store/apps/details?id=com.onavo.android.onavoid - but presumably it is not very accurate with some versions (more recent ones?).

[Kostya Vasilyev]

Re: I consume almost 3 Gb per month, it's absurd.

Yes it is.

How often do you have the app check mail? How many folders are there total to check (in 7 accounts, that's at least 28 for Inbox / Sent / Deleted / Drafts )?

But then let's see:

3,000 megabytes (let's use a round number) per 30 days (ditto).

100 megabytes per 24 hours (let's use 25).

4 megabytes per hour.

Assuming mail checks every 15 minutes:

1 megabyte per mail check.

Divide by 28 (ok, 25) folders:

40 kilobytes per folder per mail check.

Seems high:

I just tested an IMAP account, when there are no server changes, and "messages to sync" is 100, it takes ~5 kilobytes per folder.

An Exchange account's folder took much more, about 40 kilobytes: EWS is very verbose (being XML based).

However, Aqua's Exchange code tries to use data compression, but not all servers support it. Office 365 does, don't have info about others.

-----

Given that most of your accounts are IMAP, I'd say it should not use this much, maybe a few times less.

Feel free to capture a debug log (please see my signature below). With logging enabled, please "refresh" all accounts and wait for it to complete. Then send me the log file.

-----

Oh, just remembered something else which can use more traffic in an IMAP account. If you've got *a lot* of unread messages (old messages), say over a hundred, their IDs are transmitted too. So in this case, please mark your old messages as "read", perhaps using web mail.

[StR]

Oh, just remembered something else which can use more traffic in an IMAP account. If you've got *a lot* of unread messages (old messages), say over a hundred, their IDs are transmitted too. So in this case, please mark your old messages as "read", perhaps using web mail.

Kostya, is that fact a general case or only specific to some IMAP servers?
I.e. if I have almost 8000 of unread messages in Inbox, Aquamail would get 8000 IDs for them?
What about Sent folder? In general, not that many people regularly "read" their own messages...
And Deleted? And Drafts?

[Kostya Vasilyev]

Re: I.e. if I have almost 8000 of unread messages in Inbox, Aquamail would get 8000 IDs for them?

It will ask for the UID's, every time it checks mail.

IMAP server response here is fairly efficient (a space separated list, each UID is a number) -- and if the server supports ESEARCH then Aqua will use that, allowing the server to return "ranges" (instead of "1 2 3 4 5" -> "1-5").

Checking them against the local database can take a bit of time too (I'd say hundreds of milliseconds to a second or two), part of the sync algorithm.

Re: sent

But sent messages are "read" everywhere I've seen (??)

Re: deleted / drafts

Drafts - yes

Deleted and spam - no, the app doesn't even ask the server.

[StR]

I am puzzled. So, let me step back and clarify my question(s), as I suspect we might be talking about different things.

Kostya, my questions in the posting above were in regard to the unreadmessages that are beyond the sync horizon.
So, when you answered "Yes", - did you talk about those?
I.e. (explicitly),  Aquamail still receives information about those messages that are unread and beyond the sync horizon at each sync. Is that correct?

Or, rather were you talking about the "overhead" that comes from the unread messages within the sync range?


And a minor followup question (in either case):
Why would Aquamail be checking which messages in Drafts are unread?

Re: sent

But sent messages are "read" everywhere I've seen (??)

Who would set the "read" status in them? The sending e-mail client while storing the message via IMAP?
At least with some IMAP clients it is not the case. I cannot tell for sure at the moment (as for different accounts I am using a mixture of Thunderbird, Alpine, Aquamail (and in one case, - stock e-mail app), so I am not sure which of them might be setting the read flag, if any.
Is that a requirement of the IMAP protocol?

.. well, I see that Gmail (server) sets the "read" flag. But in that case the IMAP client doesn't have anything to do with that.


Update: I did a few tests:
alpine does not set that "read", while the rest of the clients listed above did.
Ghm...
Still, is that required by the IMAP protocol? If yes, - I would say it is a fault of alpine. (Unless the RFC was updated after alpine's development stopped.) If it is not required, - then Aquamail should not rely on that fact.

[Kostya Vasilyev]

Kostya, my questions in the posting above were in regard to the unreadmessages that are beyond the sync horizon.
So, when you answered "Yes", - did you talk about those?
I.e. (explicitly),  Aquamail still receives information about those messages that are unread and beyond the sync horizon at each sync. Is that correct?

Yes, the app asks for UIDs of ALL unread messages in the folder.

Server response might look like this:

* ESEARCH (TAG "kman34") UID ALL 583:585,591:592,594,900:903

Or this (if the server does not support ESEARCH):

* SEARCH 1717 1722 1724 1726 1727

Why would Aquamail be checking which messages in Drafts are unread?

It's valid for a draft to be unread, and the Draft folder's unread count is shown in the app (although not added to the account's).

Who would set the "read" status in them? The sending e-mail client while storing the message via IMAP?
At least with some IMAP clients it is not the case. I cannot tell for sure at the moment (as for different accounts I am using a mixture of Thunderbird, Alpine, Aquamail (and in one case, - stock e-mail app), so I am not sure which of them might be setting the read flag, if any.
Is that a requirement of the IMAP protocol?

No it's not required, IMAP does not deal with sending.

It's just my personal experience (a variety of web mail services).

Does not seem useful to mark a sent message "unread" -- presumably, the human doing the sending did read what he/she wrote before hitting Send.

But -- have you got thousands of unread's in drafts (probably not?) or sent?

[StR]

Yes, the app asks for UIDs of ALL unread messages in the folder.

There must be some logic why that is done...

I am looking at the account I was going to move from the stock app (where it is for historical reasons, - to separate that one account from the rest) to Aquamail.
Unread messages: Inbox:7643, Drafts: 431, sent-mail: 3634.
I'll have some substantial waste on those, I suspect...

Why would Aquamail be checking which messages in Drafts are unread?

It's valid for a draft to be unread, and the Draft folder's unread count is shown in the app (although not added to the account's).

But why someone would be "reading" Drafts? (as opposed to be writing), and better yet, why someone would care if messages in Drafts are read or unread?

Does not seem useful to mark a sent message "unread" -- presumably, the human doing the sending did read what he/she wrote before hitting Send.

Well, as you know, just because there is no real marking "unread", but only "read", it requires the program to do an additional action to mark "read".
And you'd be surprised how many people send messages without reading them (after writing).

But -- have you got thousands of unread's in drafts (probably not?) or sent?

Close to that.... See above.

My most active account that is in Aquamail has some 364 unread messages in Sent and 69 in Drafts (and ~1200 in Ibox).


The high count of unread messages in Drafts is probably due to the fact that it needs to be cleaned: sometimes intermediates copies of the message in progress (especially those created by Thunderbird) do not get deleted.

[Kostya Vasilyev]

Unread messages: Inbox:7643, Drafts: 431, sent-mail: 3634.

WOW

But why someone would be "reading" Drafts? (as opposed to be writing), and better yet, why someone would care if messages in Drafts are read or unread?

I don't know!

It's just that there is no special case for "do not maintain the unread count" for Drafts, as there is for Deleted and Spam.

And to have *hundreds* of *draft* messages, I mean at all, not just unread -- I thought I'd seen it all... It just never occurred to me.

sometimes intermediates copies of the message in progress (especially those created by Thunderbird) do not get deleted

Are they (Thunderbird) going to fix this?

Well, as you know, just because there is no real marking "unread", but only "read", it requires the program to do an additional action to mark "read".

Not true. Both IMAP and EWS allow a mail app to specify a message's read/unread state when uploading said message to the server.

And I've never seen Sent messages being marked (or left) Unread (in any of: Gmail, Fastmail, Yandex, Mail.ru, Yahoo, Hotmail, Office 365).

---

To summarize:

Yes, it's going to be a waste in your specific case.

The UIDs of 7643 messages is probably about 40K of data traffic, for sent, maybe about 20K.

( if the server supports ESEARCH, it's going to be less, depending on how well those UIDs might coalesce into ranges )

[StR]

Thank you, Kostya, for the detailed explanation. It helps a lot.

Unread messages: Inbox:7643, Drafts: 431, sent-mail: 3634.

WOW

I was surprised myself when I looked at it. I don't look at those frequently, especially not at Drafts.

I just cleaned Drafts in one of the accounts.
It is a mixture of several things in that account (and the picture is similar in the other one):
1. Thunderbird sometimes leaves behind without cleaning previous versions even after the message is sent.
2. Sometimes I write a message (or sketch a preliminary version of it), and let it sit there, while I am thinking about it. Then, either the circumstances make it obsolete, or I just decide not to send it, but let it sit there just in case the issue arises again.
3. At least with the stock (Motorola?) e-mail app, - I have barely started versions of replies: either when I touch "reply" accidentally and exit from there, or when I indeed start a reply, but need to switch to a different app before I actually wrote anything. The app saves that copy in the Drafts. Then, sometimes, I start a new reply (forgetting that one existed, or just because its faster - just 1 click, as compared to going to the Drafts folder), or, I am suspecting, even if I continue, the previous draft might be preserved.
And I suspect I might be getting some similar "orphaned" Drafts from Aquamail.
I am not blaming any app for this, - most likely it's just my laziness and sloppiness.

But why someone would be "reading" Drafts? (as opposed to be writing), and better yet, why someone would care if messages in Drafts are read or unread?

I don't know!

It's just that there is no special case for "do not maintain the unread count" for Drafts, as there is for Deleted and Spam.

I rest my case, Your Honor!
 

And to have *hundreds* of *draft* messages, I mean at all, not just unread -- I thought I'd seen it all... It just never occurred to me.

I am surprised too.

sometimes intermediates copies of the message in progress (especially those created by Thunderbird) do not get deleted

Are they (Thunderbird) going to fix this?

I've stopped reporting bugs to Mozilla projects a while ago, - because they are not that responsive.

Well, as you know, just because there is no real marking "unread", but only "read", it requires the program to do an additional action to mark "read".

Not true. Both IMAP and EWS allow a mail app to specify a message's read/unread state when uploading said message to the server.

And I've never seen Sent messages being marked (or left) Unread (in any of: Gmail, Fastmail, Yandex, Mail.ru, Yahoo, Hotmail, Office 365).

I am sure you are right about that.
I was talking from the point of view of the e-mail message, not the IMAP server, - the "Status: R" (or "RO").
Long time ago, I've started using e-mail messages from the Unix shell: mail, elm, pine (alpine).

Elm and pine would  be saving a local copy of the sent message, and that wouldn't go through any IMAP server. And they didn't add any additional status line.
It's been several years since I've stopped using elm and mail (and one of the reasons was that they were unable to work well (or at all) with IMAP servers.
Yesterday, I double-checked that alpine doesn't provide "read" status even when it saves "sent" message via IMAP server (dovecot).

To summarize:
Yes, it's going to be a waste in your specific case.
...

Many thanks for that summary, it helps in adjusting my accounts.

[Kostya Vasilyev]

At least with the stock (Motorola?) e-mail app, - I have barely started versions of replies: either when I touch "reply" accidentally and exit from there, or when I indeed start a reply, but need to switch to a different app before I actually wrote anything. The app saves that copy in the Drafts.

Gmail app is also too happy to save drafts like that...