Author Topic: Gmail IMAP Implementation  (Read 15527 times)

ggggg

  • Newbie
  • *
  • Posts: 23
Re: Gmail IMAP Implementation
« Reply #15 on: August 27, 2015, 08:36:00 am »
THX. no offence ... I like your app !!! Just a technican who wants to save juice and understand tings  ;-)

Quote
1) Under androids account settings I had one Aquamail account (gmail, google apps). After changing the language setting it said appliing settings and now I have a 2nd account for IMAP,POP3. I double checked ... no IMAP in Aqua general settings, no IMAP in Aqua account setting and no IMAP in accounts folder settings !

3) What happens if data is enabled (delay=0) and data not yet stable ... does aqua wake lock and for how long does it try ?

4) It looks if one sync event fails (time written in red) Aqua doesnt do any further sync ? How can I overcome that ?

5) When does it sync if enable the sync event "Auto-Sync setting"

6) What is the best setting to have no wake locks and minimum battery drain (please keep my data strategy in mind)
0 - One thing you can try is Menu -> Compact the database in Aqua's main window, think I'm seeing some database slowness too in a couple of places.

1 - "AquaMail (IMAP, POP3)" is a category for both IMAP and POP3 accounts.

2 - Right, AquaMail doesn't use the Android "Sync Adapter" framework to do its work.

3 - The attempt to check mail without a network connection won't work and will have network errors

4 - It will check mail next time based on the mail check schedule (by default, 15 minutes).

5 - Toggling auto-sync, the actual setting name and location varies by Android version, but in terms of code, it's this:
com.android.sync.SYNC_CONN_STATUS_CHANGED

ad 0) did not find "compact data base" ?

ad 3) would it say "network error" in log or what would the message be ?

3a) What is the effect of "network check" ON/OFF ?

ad 5) So this is the androids general Autosync ON/OFF button ... not Aquas individual one

5a) is there any way/event to sync when googles framework is syncing

ad 6) which sync would be least on bat...  ?
« Last Edit: August 27, 2015, 03:03:28 pm by ggggg »

mikeone

  • Hero Member
  • *****
  • Posts: 2748
Re: Gmail IMAP Implementation
« Reply #16 on: August 27, 2015, 08:56:44 am »
point 1:
"...and now I have a 2nd account for IMAP,POP3.... "

> ahm, is this 2nd account (generated "automatically"?) an IMAP or a POP3 account?

point 4:
if you exactly know what the app will do, why did you asked the developer about this?

Kostya Vasilyev

  • Hero Member
  • *****
  • Posts: 12740
Re: Gmail IMAP Implementation
« Reply #17 on: August 27, 2015, 04:48:04 pm »
ad 0) did not find "compact data base" ?

--- Menu in the app's main window (account list)

ad 3) would it say "network error" in log or what would the message be ?

-- Yes, network error (briefly) and then the last "good" sync time would turn red below in the account

3a) What is the effect of "network check" ON/OFF ?

-- In app settings -> network? Some devices don't properly report that they're connected, this setting lets you bypass the check.

ad 5) So this is the androids general Autosync ON/OFF button ... not Aquas individual one

-- Yes, system setting

5a) is there any way/event to sync when googles framework is syncing

-- Don't think so.

ad 6) which sync would be least on bat...  ?

-- My phones are all on the default 15 minute sync schedule, no push, no battery saving / memory optimizing apps of any kind.
Creating debug logs for diagnostics: https://www.aqua-mail.com/troubleshooting/

The official FAQ: https://www.aqua-mail.com/faq/

Лог-файлы для диагностики: https://www.aqua-mail.com/ru/troubleshooting/

Вопросы и ответы: https://www.aqua-mail.com/ru/faq/

ggggg

  • Newbie
  • *
  • Posts: 23
Re: Gmail IMAP Implementation
« Reply #18 on: August 27, 2015, 05:40:28 pm »
ad 0) did not find "compact data base" ?
...
THX, I found it.

7.) mobile connection event: How would I see in log, if I have to set any "settle time" ?

8.) I think I still have not understood the difference between "number mails synced" and "number mails local stored"
"number stored": I think it means the storage on device per synced folder
"number synced": ??
a) Please explain the difference
b) and how it influences the duration of sync if the gmail account has not got any new mails
« Last Edit: August 27, 2015, 05:42:15 pm by ggggg »

Kostya Vasilyev

  • Hero Member
  • *****
  • Posts: 12740
Re: Gmail IMAP Implementation
« Reply #19 on: August 27, 2015, 10:09:39 pm »
Re: 7.) mobile connection event: How would I see in log, if I have to set any "settle time" ?


When Android enables a particular network connection (mobile or WiFi) it sends the "network connection available" event before the connection is actually usable (fully brought up and connected).

The "settle in" time is to make AquaMail not rush with checking mail right away, a delay, from the time Android tells Aqua "WiFi is available" or "mobile data is available" and when the app starts the sync process.

Re: "number stored": I think it means the storage on device per synced folder
"number synced": ??

Messages to sync: how many server messages are checked every time, monitored for changes (e.g. disappeared, or their read/unread/starred state changed).

Those are the messages you see when you open an account's folder (message list).

The larger the setting, the more work the app has to do, even if there are no new messages -- since "old" messages can change their state or be deleted at any time (on the server).

Messages to cache: previously loaded messages, not necessarily up to date (wrt. their state or existence), kept around in case you scroll down in a message list.

If you do, the app will be able to update their state (read/unread/starred) without reloading from server from scratch.
Creating debug logs for diagnostics: https://www.aqua-mail.com/troubleshooting/

The official FAQ: https://www.aqua-mail.com/faq/

Лог-файлы для диагностики: https://www.aqua-mail.com/ru/troubleshooting/

Вопросы и ответы: https://www.aqua-mail.com/ru/faq/

ggggg

  • Newbie
  • *
  • Posts: 23
Re: Gmail IMAP Implementation
« Reply #20 on: August 28, 2015, 08:44:25 am »
Re: 7.) When Android enables a particular network connection (mobile or WiFi) it sends the "network connection available" event before the connection is actually usable (fully brought up and connected).
The "settle in" time is to make AquaMail not rush with checking mail right away, a delay, from the time Android tells Aqua "WiFi is available" or "mobile data is available" and when the app starts the sync process.
1 ad  network settle: But how could I see a not yet stable conn. in log ?
- what about the attached log-AquaMail_79s.txt, it could be an example ?

2 and why is log-AquaMail_why_just_2.8s.txt so fast - is it skipping all ? why ?
Code: [Select]
2015.08.27 13:46:45.832 +0200 [SYNC.1346] Will skip INBOX
2015.08.27 13:46:45.833 +0200 [SYNC.1346] Will skip [Gmail]/Entwürfe
2015.08.27 13:46:45.833 +0200 [SYNC.1346] Will skip [Gmail]/Gesendet
2015.08.27 13:46:45.834 +0200 [SYNC.1346] Will skip [Gmail]/Papierkorb

3 Last night aqua synced about 15 times (1 skipped, rest went OK) ... but why did it start 45 times instead of e.g. 15 times. PLease see attached log-AquaMail_nodelay_45_starts.log and screenshot


Quote
Re: "number stored":
Messages to sync: how many server messages are checked every time, monitored for changes (e.g. disappeared, or their read/unread/starred state changed). Those are the messages you see when you open an account's folder (message list).
The larger the setting, the more work the app has to do, even if there are no new messages -- since "old" messages can change their state or be deleted at any time (on the server).

Messages to cache: previously loaded messages, not necessarily up to date (wrt. their state or existence), kept around in case you scroll down in a message list.
If you do, the app will be able to update their state (read/unread/starred) without reloading from server from scratch.
4 so if the no. synced= 50 and stored=250 (I think thats default)
- it checks the LAST 50 of each synced folder every time
- the rest 200 are untouched even if the state (e.g. read/unread) changes ... I guess I am still wrong, because if one of the 200 is deleted it musr be checked too
- what happens if one has received 100 new mails in its account and sync is on 50 ?
« Last Edit: August 28, 2015, 01:32:07 pm by ggggg »

Kostya Vasilyev

  • Hero Member
  • *****
  • Posts: 12740
Re: Gmail IMAP Implementation
« Reply #21 on: August 30, 2015, 01:21:35 am »
Re: network settle: But how could I see a not yet stable conn. in log ?

If the device's connectivity isn't ready -> network (connectivity) errors. In the log, you'll see them as Java exceptions.

Re: and why is log-AquaMail_why_just_2.8s.txt so fast - is it skipping all ? why ?

Yes. AquaMail can (with the "better IMAP servers") very quickly detect if there are no changes since last sync.

And since we're talking about Gmail IMAP here, a change to a single folder (even if it's not synced!) will mark all folders as "changed".

Re:  I guess I am still wrong, because if one of the 200 is deleted it musr be checked too

This won't be detected immediately -- but since the app will only show last 50, you won't see any "ghost" messages.

Until you scroll down to "see more messages" -- then it will check 50 more.


I guess this has to do with how Gmail doesn't separate messages into folders -- rather, there is an account-level message database + each message may have "labels".

Other IMAP servers (Dovecot, Cyrus) are able to properly track changes per-folder.

Re: what happens if one has received 100 new mails in its account and sync is on 50

You will see 50 new messages and the unread count will have 100, so you'll know to scroll down.

And when you do scroll down, the app will load the next 50, so you'll have your 100.
Creating debug logs for diagnostics: https://www.aqua-mail.com/troubleshooting/

The official FAQ: https://www.aqua-mail.com/faq/

Лог-файлы для диагностики: https://www.aqua-mail.com/ru/troubleshooting/

Вопросы и ответы: https://www.aqua-mail.com/ru/faq/

ggggg

  • Newbie
  • *
  • Posts: 23
Re: Gmail IMAP Implementation
« Reply #22 on: August 30, 2015, 09:27:13 am »
If the device's connectivity isn't ready -> network (connectivity) errors. In the log, you'll see them as Java exceptions.
THX Kostya,
- would the conn error look like this ?
Code: [Select]
2015.08.29 09:27:47.744 +0200 StartSyncService onStartCommand 1
2015.08.29 09:27:47.747 +0200 GenericWorkerThread Executing org.kman.AquaMail.core.StartSyncService$1@4296bcc0
2015.08.29 09:27:47.748 +0200 [*** IDLE ***.729] checkWatchers 0x0
2015.08.29 09:27:47.748 +0200 [*** IDLE ***.729] About to check network type
2015.08.29 09:27:47.753 +0200 [*** IDLE ***.729] Active network info: NetworkInfo: type: mobile[HSPA+], state: CONNECTED/CONNECTED, reason: connected, roaming: false, failover: false, isAvailable: true, isConnectedToProvisioningNetwork: false
2015.08.29 09:27:47.756 +0200 [*** IDLE ***.729] Checking system settings: 0
2015.08.29 09:27:47.757 +0200 StartSyncService ***** ACTION_SYNC, planned list is -1 long, grabAll is false, runNow is false *****
2015.08.29 09:27:47.758 +0200 StartSyncService ***** handleStartSyncAccounts() *****
2015.08.29 09:27:47.759 +0200 MailAccountManager Constructor, loadDb = true, loadErrors = true
2015.08.29 09:27:47.762 +0200 AquaMail ***** STACK TRACE FOR : GenericDbOpenHelper: Messages.sqldb
java.lang.Throwable: GenericDbOpenHelper: Messages.sqldb
at org.kman.Compat.util.MyLog.logStackTrace(MyLog.java:159)
at org.kman.AquaMail.data.GenericDbOpenHelper.<init>(GenericDbOpenHelper.java:35)
at org.kman.AquaMail.data.MailDbOpenHelper.<init>(MailDbOpenHelper.java:972)
at org.kman.AquaMail.data.MailDbOpenHelper.get(MailDbOpenHelper.java:644)
at org.kman.AquaMail.data.MailDbHelpers.getDatabase(MailDbHelpers.java:57)
at org.kman.AquaMail.mail.MailAccountManager.<init>(MailAccountManager.java:1740)
at org.kman.AquaMail.mail.MailAccountManager.get(MailAccountManager.java:103)
at org.kman.AquaMail.core.StartSyncService.handleStartSyncAccounts(StartSyncService.java:277)
at org.kman.AquaMail.core.StartSyncService.onStartCommand(StartSyncService.java:224)
at android.app.ActivityThread.handleServiceArgs(ActivityThread.java:3110)
at android.app.ActivityThread.access$2200(ActivityThread.java:174)
at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1422)
at android.os.Handler.dispatchMessage(Handler.java:102)
at android.os.Looper.loop(Looper.java:146)
at android.app.ActivityThread.main(ActivityThread.java:5756)
at java.lang.reflect.Method.invokeNative(Native Method)
at java.lang.reflect.Method.invoke(Method.java:515)
at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:1291)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1107)
at de.robv.android.xposed.XposedBridge.main(XposedBridge.java:132)
at dalvik.system.NativeStart.main(Native Method)

- and could you please comment on #3 of my last post too
« Last Edit: August 30, 2015, 07:59:51 pm by ggggg »

Kostya Vasilyev

  • Hero Member
  • *****
  • Posts: 12740
Re: Gmail IMAP Implementation
« Reply #23 on: August 31, 2015, 08:28:44 pm »
- would the conn error look like this ?

No, that's an "exception stack trace" but not a network error. That's the stack trace for opening the database (for my use, really).

Creating debug logs for diagnostics: https://www.aqua-mail.com/troubleshooting/

The official FAQ: https://www.aqua-mail.com/faq/

Лог-файлы для диагностики: https://www.aqua-mail.com/ru/troubleshooting/

Вопросы и ответы: https://www.aqua-mail.com/ru/faq/

Kostya Vasilyev

  • Hero Member
  • *****
  • Posts: 12740
Re: Gmail IMAP Implementation
« Reply #24 on: August 31, 2015, 08:32:48 pm »
Re: #3

45 of "what"?

Mail checks driven by connectivity changes:

$ grep "Will kick off sync" log-AquaMail_nodelay_45_starts.txt
2015.08.27 18:28:05.352 +0200   KickSyncReceiver   Will kick off sync in 0 seconds
2015.08.27 19:28:05.564 +0200   KickSyncReceiver   Will kick off sync in 0 seconds
2015.08.27 20:28:05.392 +0200   KickSyncReceiver   Will kick off sync in 0 seconds
2015.08.27 21:28:05.878 +0200   KickSyncReceiver   Will kick off sync in 0 seconds
2015.08.27 22:28:04.598 +0200   KickSyncReceiver   Will kick off sync in 0 seconds
2015.08.27 23:28:05.323 +0200   KickSyncReceiver   Will kick off sync in 0 seconds
2015.08.28 00:28:05.518 +0200   KickSyncReceiver   Will kick off sync in 0 seconds
2015.08.28 01:28:06.466 +0200   KickSyncReceiver   Will kick off sync in 0 seconds
2015.08.28 02:28:04.661 +0200   KickSyncReceiver   Will kick off sync in 0 seconds
2015.08.28 03:28:04.809 +0200   KickSyncReceiver   Will kick off sync in 0 seconds
2015.08.28 04:28:05.856 +0200   KickSyncReceiver   Will kick off sync in 0 seconds
2015.08.28 05:28:05.744 +0200   KickSyncReceiver   Will kick off sync in 0 seconds
2015.08.28 06:28:04.584 +0200   KickSyncReceiver   Will kick off sync in 0 seconds
2015.08.28 07:28:05.640 +0200   KickSyncReceiver   Will kick off sync in 0 seconds
2015.08.28 08:28:05.697 +0200   KickSyncReceiver   Will kick off sync in 0 seconds

Count: 15


Wake lock releases (paired to acquires, of course):

$ grep "Releasing the Wake lock, time held" log-AquaMail_nodelay_45_starts.txt
2015.08.27 18:28:08.248 +0200   LockManager   Releasing the Wake lock, time held = 2.85 sec
2015.08.27 19:28:14.490 +0200   LockManager   Releasing the Wake lock, time held = 8.91 sec
2015.08.27 20:28:12.564 +0200   LockManager   Releasing the Wake lock, time held = 7.16 sec
2015.08.27 21:28:14.028 +0200   LockManager   Releasing the Wake lock, time held = 8.14 sec
2015.08.27 22:28:11.808 +0200   LockManager   Releasing the Wake lock, time held = 7.19 sec
2015.08.27 23:28:15.350 +0200   LockManager   Releasing the Wake lock, time held = 10.00 sec
2015.08.28 00:28:13.461 +0200   LockManager   Releasing the Wake lock, time held = 7.92 sec
2015.08.28 01:28:14.428 +0200   LockManager   Releasing the Wake lock, time held = 7.95 sec
2015.08.28 02:28:11.898 +0200   LockManager   Releasing the Wake lock, time held = 7.21 sec
2015.08.28 03:28:12.906 +0200   LockManager   Releasing the Wake lock, time held = 8.08 sec
2015.08.28 04:28:13.584 +0200   LockManager   Releasing the Wake lock, time held = 7.71 sec
2015.08.28 05:28:13.825 +0200   LockManager   Releasing the Wake lock, time held = 8.05 sec
2015.08.28 06:28:11.787 +0200   LockManager   Releasing the Wake lock, time held = 7.19 sec
2015.08.28 07:28:13.517 +0200   LockManager   Releasing the Wake lock, time held = 7.86 sec
2015.08.28 08:28:13.579 +0200   LockManager   Releasing the Wake lock, time held = 7.87 sec

Count: 15
Creating debug logs for diagnostics: https://www.aqua-mail.com/troubleshooting/

The official FAQ: https://www.aqua-mail.com/faq/

Лог-файлы для диагностики: https://www.aqua-mail.com/ru/troubleshooting/

Вопросы и ответы: https://www.aqua-mail.com/ru/faq/

ggggg

  • Newbie
  • *
  • Posts: 23
Re: Gmail IMAP Implementation
« Reply #25 on: September 01, 2015, 09:18:12 am »
Re: #3   45 of "what"?
It was about the number of wakelocks over night. Aqua syncs once per hour triggered by data ON/OFF for 1 minute. So over night thats about 12-15 syncs, but it produces about 45 wakelocks ... see screenshot.

- Sorry, but I still have no idea what the conn error then would look like ...?

- if I set sync to 50 and local stored to 250 in general settings and I am offline ... I just see the last 50 synced  ... I expected to have 250 ?
« Last Edit: September 01, 2015, 04:26:12 pm by ggggg »

Kostya Vasilyev

  • Hero Member
  • *****
  • Posts: 12740
Re: Gmail IMAP Implementation
« Reply #26 on: September 01, 2015, 11:26:02 pm »
Quote
It was about the number of wakelocks over night. Aqua syncs once per hour triggered by data ON/OFF for 1 minute. So over night thats about 12-15 syncs, but it produces about 45 wakelocks ... see screenshot.

In the log, the wake lock was released 15 times (and so acquired 15 times too).

What is the app's actual battery usage %% value? In system settings -> battery?

Quote
- if I set sync to 50 and local stored to 250 in general settings and I am offline ... I just see the last 50 synced  ... I expected to have 250 ?

Maybe, but it will show 50 because (as explained above), "how many to sync" means "how many messages to check for existence / flags changes".

The other 200 are "beyond the sync window" and will come into play if you scroll down to "see more". But first the app will check the server for the current state of these messages.

If you need to see the cached messages, even if their state (or existence) may not be up to date, do Menu -> View -> Offline list.
Creating debug logs for diagnostics: https://www.aqua-mail.com/troubleshooting/

The official FAQ: https://www.aqua-mail.com/faq/

Лог-файлы для диагностики: https://www.aqua-mail.com/ru/troubleshooting/

Вопросы и ответы: https://www.aqua-mail.com/ru/faq/

Kostya Vasilyev

  • Hero Member
  • *****
  • Posts: 12740
Re: Gmail IMAP Implementation
« Reply #27 on: September 01, 2015, 11:28:34 pm »
Wake lock *acquires* from the log:

2015.08.27 18:28:05.393 +0200   LockManager   Acquiring the Wake lock
2015.08.27 19:28:05.578 +0200   LockManager   Acquiring the Wake lock
2015.08.27 20:28:05.404 +0200   LockManager   Acquiring the Wake lock
2015.08.27 21:28:05.890 +0200   LockManager   Acquiring the Wake lock
2015.08.27 22:28:04.614 +0200   LockManager   Acquiring the Wake lock
2015.08.27 23:28:05.344 +0200   LockManager   Acquiring the Wake lock
2015.08.28 00:28:05.535 +0200   LockManager   Acquiring the Wake lock
2015.08.28 01:28:06.478 +0200   LockManager   Acquiring the Wake lock
2015.08.28 02:28:04.677 +0200   LockManager   Acquiring the Wake lock
2015.08.28 03:28:04.821 +0200   LockManager   Acquiring the Wake lock
2015.08.28 04:28:05.868 +0200   LockManager   Acquiring the Wake lock
2015.08.28 05:28:05.773 +0200   LockManager   Acquiring the Wake lock
2015.08.28 06:28:04.597 +0200   LockManager   Acquiring the Wake lock
2015.08.28 07:28:05.655 +0200   LockManager   Acquiring the Wake lock
2015.08.28 08:28:05.711 +0200   LockManager   Acquiring the Wake lock


15 of them.

And "all together now":

$ egrep "Acquiring the Wake lock|Releasing the Wake lock, time held" log-AquaMail_nodelay_45_starts.txt

2015.08.27 18:28:05.393 +0200   LockManager   Acquiring the Wake lock
2015.08.27 18:28:08.248 +0200   LockManager   Releasing the Wake lock, time held = 2.85 sec
2015.08.27 19:28:05.578 +0200   LockManager   Acquiring the Wake lock
2015.08.27 19:28:14.490 +0200   LockManager   Releasing the Wake lock, time held = 8.91 sec
2015.08.27 20:28:05.404 +0200   LockManager   Acquiring the Wake lock
2015.08.27 20:28:12.564 +0200   LockManager   Releasing the Wake lock, time held = 7.16 sec
2015.08.27 21:28:05.890 +0200   LockManager   Acquiring the Wake lock
2015.08.27 21:28:14.028 +0200   LockManager   Releasing the Wake lock, time held = 8.14 sec
2015.08.27 22:28:04.614 +0200   LockManager   Acquiring the Wake lock
2015.08.27 22:28:11.808 +0200   LockManager   Releasing the Wake lock, time held = 7.19 sec
2015.08.27 23:28:05.344 +0200   LockManager   Acquiring the Wake lock
2015.08.27 23:28:15.350 +0200   LockManager   Releasing the Wake lock, time held = 10.00 sec
2015.08.28 00:28:05.535 +0200   LockManager   Acquiring the Wake lock
2015.08.28 00:28:13.461 +0200   LockManager   Releasing the Wake lock, time held = 7.92 sec
2015.08.28 01:28:06.478 +0200   LockManager   Acquiring the Wake lock
2015.08.28 01:28:14.428 +0200   LockManager   Releasing the Wake lock, time held = 7.95 sec
2015.08.28 02:28:04.677 +0200   LockManager   Acquiring the Wake lock
2015.08.28 02:28:11.898 +0200   LockManager   Releasing the Wake lock, time held = 7.21 sec
2015.08.28 03:28:04.821 +0200   LockManager   Acquiring the Wake lock
2015.08.28 03:28:12.906 +0200   LockManager   Releasing the Wake lock, time held = 8.08 sec
2015.08.28 04:28:05.868 +0200   LockManager   Acquiring the Wake lock
2015.08.28 04:28:13.584 +0200   LockManager   Releasing the Wake lock, time held = 7.71 sec
2015.08.28 05:28:05.773 +0200   LockManager   Acquiring the Wake lock
2015.08.28 05:28:13.825 +0200   LockManager   Releasing the Wake lock, time held = 8.05 sec
2015.08.28 06:28:04.597 +0200   LockManager   Acquiring the Wake lock
2015.08.28 06:28:11.787 +0200   LockManager   Releasing the Wake lock, time held = 7.19 sec
2015.08.28 07:28:05.655 +0200   LockManager   Acquiring the Wake lock
2015.08.28 07:28:13.517 +0200   LockManager   Releasing the Wake lock, time held = 7.86 sec
2015.08.28 08:28:05.711 +0200   LockManager   Acquiring the Wake lock
2015.08.28 08:28:13.579 +0200   LockManager   Releasing the Wake lock, time held = 7.87 sec


15
Creating debug logs for diagnostics: https://www.aqua-mail.com/troubleshooting/

The official FAQ: https://www.aqua-mail.com/faq/

Лог-файлы для диагностики: https://www.aqua-mail.com/ru/troubleshooting/

Вопросы и ответы: https://www.aqua-mail.com/ru/faq/