AquaMail - A couple of questions..

[jmccabe]

Hi

(Thought I'd add "AquaMail" at the front to avoid this looking like some of the spam in here!)

I have a couple of general questions.

First of all, as someone who is looking to use AquaMail as a replacement for K9, I'd like to know how AquaMail deals with deleted messages on a POP3 account. I'm not too interested in the detail, but I'd like to be sure that I'm not going to see the problem I see in K9. Basically in K9 for some reason (I was told, but can't remember exactly what the answer was) a record is maintained in the database of pretty much every message deleted. It eventually gets to the point that, when I "Empty Trash", K9 just crashes because of issues with the number of these records. The solution is to use something like aSQLiteManager to go in to the database and manually delete all these records using a delete query. Not ideal, especially if you're not someone who knows about databases (I'm a software engineer so it's not so bad, but...). If anyone can explain how that's avoided in AquaMail it would be appreciated.

I also would like to talk about future development. It sounds like AquaMail is a one-man team development like MailDroid so what happens when you get bored with it? Do you envisage POP3 support continuing for ever? The reason I ask that last question is that although IMAP seems to be quite common at the minute I can see a time when there is just too much data for ISPs/Email providers to store on their servers and they'll be looking to users to download and keep their own email (in other words, use POP3!). I'm with Demon in the UK and, although they've just installed a new email system that allows IMAP access, they only keep your messages for 30 days and you're expected to download them by then if you want to keep them. There are also big security issues with IMAP servers, especially when coupled with the prevalence of web-based mail system, where many yahoo customers' mail accounts have been hacked into recently. In my opinion IMAP is a disaster waiting to happen!

Finally, is there a means of exporting mail to a format that can be easily imported into other mail clients, in particular, Thunderbird? I'm particularly interested in this since, as I mentioned, I use Demon in the UK so I'm basically on a POP3/SMTP system where sent messages will be saved only on my phone. I'd like to be able to regularly export the ones I want to keep them on my 'main' email system at home.

Anyway - just thought I'd ask...
John

[Rachel Ambler]

Demon are still going? Wow, I remember using them way back in the mid 90's!

If you have Thunderbird already set up then can't you set that up with POP3 to not delete retrieved emails so that will maintain your long term email store? Then use your phone for the 30 messages?

Rachel

Sent from my SCH-I535 using Tapatalk 2

[jmccabe]

Thanks for your reply Rachel. I've been using Demon since the mid 90s :-) They're not particularly great but my email address is used in so many places. ..

I'll think about your suggestion but I think I considered that once and it didn't quite fit my needs.  Worth thinking again though as I've got a new phone.

John

Sent from my GT-I9300 using Tapatalk 2

[jmccabe]

Rachel or Kostya,

I was wondering if you might have answers to these questions; I haven't used IMAP so I'm not sure.

1) If I used IMAP on my phone, would it synchronise with the server such that older messages I'd already copied to the phone would be deleted or would they stay on my phone?

2) If I set up my phone to use IMAP, could I create both an IMAP version and a POP3 version of my Demon email account on Thunderbird so that the IMAP version synchronised messages sent from my phone and copied them into the POP3 versions sent mail folder?

I'll probably try this out but if you know the answers that would be great!

Thanks

[Rachel Ambler]

John,

I'm sure Kostya knows about IMAP than I'd ever even dare to guess, but my understanding of the protocol is that with an IMAP connection you are simply looking at what's on the server for each folder you interrogate (after applying filters I assume such as # of messages etc.).

Not having used Demon in like forever (I left them and the UK behind in 2001) I'm not sure on how they handle specific folders such as Sent items but I'd hazard a guess and say that they have them, just like any other folder.

So I think on the IMAP side that you'd be fine. I'm not so sure about the POP3 one though. I last used POP3 about 7 years ago and so am unsure how it handles folders - If memory serves though I seem to feel it doesn't - If so this leads you to a quandary on how to ensure emails sent from your phone are also kept on Thunderbird.

There's two fudge's that I can think of here to get round this, one's ugly the other's fugly.

Ugly option is to bcc yourself each email you send from your phone, that way the sent email will make it's way to your Inbox and, as a result, to your POP3 connected Thunderbird.

Fugly is to manually move each sent mail to the Inbox after you've sent it. Again this will ensure the POP3 fetch picks it up.

Rules on Thunderbird might enable you to move these emails from the Inbox to your Sent email folders.

This all said though, I'm rather In the opposing camp viz-a-viz the future between IMAP and POP3 and personally I can see POP3 being the protocol that will first be abandoned. Disk space is amazingly cheap these days and emails are not big items to store. I have a Gmail account going back 7 odd years and have only used 2GB or so - Google give you about 10 these days just for asking. Whilst POP3 has certain advantages, the move to multiple small clients such as smartphones or tablets makes the idea of having a single POP3 store seem rather antiquated. The cloud is pretty much here and the question is, you going to have a public or private one?

So, going to your questions...

1) The IMAP email store on AquaMail would reflect what's currently on the server. They would stay on your phone until they either get deleted by Demons most ridiculous 30 day policy or if you delete them yourself (caveat emptor: filters on AquaMail may prevent older emails being retrieved, however AquaMail will not on of itself delete them.

2) Yes, but with an understanding that the sent mails are a nasty proposition as I mentioned earlier.

As a question, have you considered other email systems? I use Google Apps myself along with 1&1 which gives me a huge amount of flexibility regarding addresses, aliases etc.

R

Sent from my SCH-I535 using Tapatalk 2

[jmccabe]

Hi Rachel

Thanks for your reply. Sorry this is a very long response as I've quoted your message but there were a lot of places I felt I could comment.

... but my understanding of the protocol is that with an IMAP connection you are simply looking at what's on the server for each folder you interrogate (after applying filters I assume such as # of messages etc.).

Yes, that's how I envisaged it, which is a bit of a nuisance if you want to keep hold of messages locally that are going to be deleted remotely.

Not having used Demon in like forever (I left them and the UK behind in 2001) I'm not sure on how they handle specific folders such as Sent items but I'd hazard a guess and say that they have them, just like any other folder.

It's a Microsoft Exchange based system, so they have "Sent Items", "Deleted Items" etc. Which was a bit of a bitch when I set up the experiment using Thunderbird last night as it likes "Sent" and "Trash" and has a roundabout way of getting access to the "Sent Items" and "Deleted Items"; I got there in the end though :-)

I'm not so sure about the POP3 one though. I last used POP3 about 7 years ago and so am unsure how it handles folders - If memory serves though I seem to feel it doesn't

Yeah, there's no concept of folders in POP3; it's simply a mail retrieval mechanism and you're expected to maintain your own folder structure locally.

Ugly option is to bcc yourself each email you send from your phone, that way the sent email will make it's way to your Inbox and, as a result, to your POP3 connected Thunderbird.

That would be a devastatingly bad option :-)

Fugly is to manually move each sent mail to the Inbox after you've sent it. Again this will ensure the POP3 fetch picks it up.

That would also be quite bad - both of them would end up just not looking right (if you know what I mean) as they'd be received messages not raw "sent" ones.

Rules on Thunderbird might enable you to move these emails from the Inbox to your Sent email folders.

Well, last night I did part of the experiment I mentioned. I already have POP3 email on Thunderbird so created an IMAP version of the same account. As I mentioned, I struggled with the folders because Thunderbird makes it a little awkward; you have to use the "Subscribe" context menu to get the right ones if you're not using default "Sent" and "Trash" folders (Aqua-Mail handled that beautifully with no intervention from me at all, so h/t to Kostya :-). I was able to copy sent messages from my POP3 account into the IMAP account's "Sent Items" folder and saw, through the Outlook Web Access interface to my mailbox, that they had been synchronised with the Demon server. That means, on my other computer (the one that holds my complete email repository), I would be able to set up the same IMAP version and copy the sent ones into the equivalent POP3 version of the account there, which is pretty much what I want to achieve in the long run!

At the moment I've set up Aqua-Mail to have both POP3 and IMAP versions, but I have the POP3 one effectively disabled so I'm working from the IMAP version for the moment. I'll see how that goes, but it would be nice if I could get it to keep old messages. I'm not sure whether the new Demon system will let you move messages from the Inbox into a separate folder which doesn't have the 30 day limit. I think I need to check that out.

This all said though, I'm rather In the opposing camp viz-a-viz the future between IMAP and POP3 and personally I can see POP3 being the protocol that will first be abandoned. Disk space is amazingly cheap these days and emails are not big items to store. I have a Gmail account going back 7 odd years and have only used 2GB or so - Google give you about 10 these days just for asking. Whilst POP3 has certain advantages, the move to multiple small clients such as smartphones or tablets makes the idea of having a single POP3 store seem rather antiquated. The cloud is pretty much here and the question is, you going to have a public or private one?

Personally I would rather not rely on someone else to make sure my historical email is safe. I think the Yahoo issue recently strengthened that belief; I have friends who were hacked, probably due to the exploit that was publicised recently, so I'd rather my mail was in my own storage, not Demon's or Google's or Yahoo's etc. Other than that, who knows what those organisations are doing with the information in your mailbox? I know there's no guarantees with Demon either, but it gives me a warmer feeling knowing that my mail is only hanging around on their system for a short time :-)

I agree that POP3 is not ideal for the multiple client scenario, which is why I've got this issue :-) I just don't think IMAP is particularly suitable for long term storage either. Some half-way house would be nice, which is one of the reasons I bought a NAS - with the intention of setting it up to collect my email via POP3 then serve it to me through IMAP. Unfortunately I haven't got that working for two reasons. Firstly when I activate the Dovecote/XMail combination on my QNAP TS-212 it means the disks never spin down, which is great for response on my Sonos system etc, but not great for power consumption! Secondly I can't connect to the IMAP server from the wireless network using my external IP address - my router doesn't seem to make this possible.

So, going to your questions...

1) The IMAP email store on AquaMail would reflect what's currently on the server. They would stay on your phone until they either get deleted by Demons most ridiculous 30 day policy or if you delete them yourself (caveat emptor: filters on AquaMail may prevent older emails being retrieved, however AquaMail will not on of itself delete them.

2) Yes, but with an understanding that the sent mails are a nasty proposition as I mentioned earlier.

Thanks

As a question, have you considered other email systems? I use Google Apps myself along with 1&1 which gives me a huge amount of flexibility regarding addresses, aliases etc.

I've been considering it for a long time, mainly due to the fact I can get ISP services quite a bit cheaper if I use a different provider. The email address is the thing though. I already maintain my own domain name with hosting that provides mail facilities so I could use that, but I could also presumably set my MX record up for that domain to send directly to my home address and have an SMTP server on my NAS then do the IMAP thing from there, if it wasn't for the IP address through wireless issue, ... so I haven't got very far with any of that yet :-)

Nightmare :-)

Thanks again
John

[Rachel Ambler]

Quick reply - have you set up port forwarding on your router to enable external connectivity to the NAT'd IMAP server?

Sent from my SCH-I535 using Tapatalk 2

[jmccabe]

I had done; getting at it from the outside wasn't the problem; it was getting at it from inside using the external IP address. I.e. if I set my client up to point at my IMAP server at <xxx>.demon.co.uk it was fine through the mobile network but just hung when trying through wireless. I had to set it directly to the address of the NAS to get at it through wireless. It's basically the problem described here, http://superuser.com/questions/208710/public-ip-address-answered-by-router-not-internal-web-server-with-port-forwardi, except with IMAP and not web.

I found this one, http://stackoverflow.com/questions/450894/accessing-internal-network-resource-using-external-ip-address, which suggests more local DNS management, but that's only really feasible if you've only got one machine serving on the inside. If I wanted to have separate machines doing different things it wouldn't work. Saying that, I'd only have one server at the moment so might be worth re-reviewing. Still doesn't get round the "no disk spin-down" issue though :-)

John

[Kostya Vasilyev]

Wow, I surely missed a lot

Will try to address some of the questions.

1 - AquaMail (henceforth referred to as "the app"... ) never deletes messages from mail servers, either using IMAP or POP3, when checking mail. There is simply no code there to do that.

2 - The app retrieves / updates a limited number of most recent messages during each mail check. This is the "Messages per page" setting. The mechanism of course varies between IMAP and POP3.

3 - The app keeps a limited number of recent messages in its store. This is the "Messages to keep" setting. The excess is trimmed from the app's data storage automatically, about once  every 24 hours.

4 - The above sometimes causes issues with people "hiding" a lot of messages in a folder -- then after the app cleans out old stuff and checks mail, sometimes those messages are treated as new. Not good, but the alternative, keeping an eternal, untrimmed history of all hidden / deleted messages seems worse to me (we're talking about very limited devices).

5 - There is no offline access to messages past the current page for IMAP -- the app only shows what it knows is up to date. I'd like to remove this limitation, allowing offline access to the message cache past the ones known to be up to date (as there are practical reasons to want to do that).

6 - With regards to security, I feel there is nothing different about IMAP vs. POP3. Both protocols use plain text authentication (even if it's wrapped in a fancy format, like SASL PLAIN), and then SSL on top is possible for both. For SMTP, the app supports CRAM-MD5, and it works most of the time, but for POP3, there is no support for APOP or MD5 since in practice, there are too many servers where this is misconfigured, causing login errors.

7 - I wouldn't trust my own mail app to keep my personal mail archive (if it had any value). This sounds like a bad thing coming from the developer, but:

* I regularly see the app's database get corrupted, about 1-2 users per 100,000, per app upgrade. Very infrequent, but still. All of this seem to happen on Android 4.0+. Not sure if it's a SQLite bug (I know it's the best unit tested piece of software in the world, but I've seen a message where its developer admits to a data-corrupting bug, specifically in the context of Android), or an issue with ext4 specifically on Android, but...

( FWIW, I've seen K-9 lose all its data, and I've seen reports on the web about SQLite databases getting corrupted in other apps on other platforms )

* Desktop mail apps just seem (to me) to have better UIs -- hey, a 10" tablet is "large", a 10" notebook is "small". My monitors are 24"

* The ability to make backups -- you can easily copy a Thunderbird profile folder, or a Windows Live Mail message directory... Try doing that on Android (without root, I mean).

8 - POP3 support will continue, it's already there, there is absolutely no reason to take it out

9 - Boredom and stuff -- so far, it's the opposite, there is so much more work to do before the app looks and works half decent...

10 - IMAP storage -- well, Google certainly wants your whole mail archive... I've seen other services where I have accounts also offer more and more storage over the last few years. Not familiar with Daemon at all.

[jmccabe]

Kostya

Many thanks for taking the time to respond to my questions. That certainly gives me a warmer feeling than the one I got on the K9 forum last week when the response to my question "is there a way to move the database to a new phone and use it there" was, err, nothing!

FWIW - See http://en.wikipedia.org/wiki/Demon_Internet for more details about Demon. Basically they were pretty much the first ISP in the UK to offer afforable personal internet service in the early 1990s. They had a number of advantages in the early days compared to other ISPs. Firstly, they supplied you with a static IP address and a fixed host name (mine's "assen"). Also they supplied email in a different way to most; it was delivered to your PC via SMTP so you had to run an SMTP server, although the MX record would point to Demon's machine and they would store it until you dialled in and then they would forward it to your server. This meant that basically you had full control over what email addresses you used before the @ so totally unlimited mailboxes.

A guy called Ewan Kirk actually wrote an app called WinDIS which had a simple SMTP server in it and a built in gateway to Pegasus Mail so that was really quote convenient.

Over the years things changed and when ADSL got rolled out I understand British Telecom prevented delivery of email in that way so we had to move to Demon's POP3 service. That, again, was pretty flexible though; their servers just accepted mail for <anyone>@<yourhost>.demon.co.uk and you used a special login format to get only the mail you were interested in through POP3. It's only since the mid of last year that they've offered an IMAP option but, as I mentioned earlier, there are restrictions in that they only retain your messages for up to 30 days before deleting it. They also had probably the best USENET feed of certainly any provider in the UK and probably one of the best worldwide. They still provide that but USENET's mostly SPAM these days like a lot of things!

So, ultimately, it's not as competitive as it was in the early days, but I do have basically unlimited downloads (unlike a lot of the cheaper options) and they don't really block much.

So there you have it - Demon, a brief guide :-)

[Kostya Vasilyev]

This is a test response. Testing IMAP BODYSTRUCTURE.