I had a similar problem so I created a plugin that is able to stream zip-files and doesn't need build them before sending.
It might help you, too.
I have to do some more testing though before I'll make it available to the public. But chances are good I'll release it later this weekend. There are some limitations but I'll explain them on release.
But as you say there maybe only albums affected that are uploaded via ftp. It could be that php has no write access to these folders, which it needs if i remember correctly, as a temporary .zip file is created inside the related album folder.
But maybe acrylian can confirm this as I'm not sure if those files were created anyways or only with caching enabled.