Zenphoto does do a lot of filesystem work, to enable it to detect new images and albums on disk without having to resort to a manual upload process. It's part of its design that it mirrors the albums on disk. Because of this, it can be slightly slower per page, but it is generally fast even for very large albums.
I haven't looked at the backend PHP scripts in a while, but it could be time for some optimization. However, 320ms for an album of over a thousand images does appear reasonable.
Is there a particular case for which you need the speed? Would page caching be sufficient to handle extra load or decrease load times?