Yes, the app has to do some work when it checks mail, and it takes work just to determine that there are no changes.
And on the other hand, modern Android devices typically have multi-core CPUs running at crazy gigahertz speeds and all that, and Android is a modern multi-tasking operating system.
Is it perhaps the use of the (limited perhaps) network connection? With video playback taking up all the connectivity it can get?
Is Android or your router capable of QoS (quality of service) to give higher priority to video traffic? Myself, I don't know, sorry.