Based on your description, this should be fixed in one of the recent development builds. Feel free to try the latest one (see the Development Builds section of this forum).
Long tech story: it has to do with how some IMAP servers return zero sized text in messages, differently from others. Both are valid.
The released version only handles the more common representation, and falls back to fetching such messages completely to extract the text.
This can comes up because some other IMAP servers (e.g. Exchange) can return 0 for the size of text in message with attachments, even when there actually is text there, and the code tries to work around that as well.
The recent development versions handle both representations of zero sized text, and will stop there without fetching the attachments.