Zenphoto integrations with wordpress are going to be slow for a good reason - every single page load, zenphoto has to load Wordpress' entire program, and then its own program logic. I don't mean to knock Wordpress, but it barely slides by on its own, so compounding its load time with Zenphoto's load time is even less bearable. I am willing to bet that most of the page load time is Wordpress' part (since I know Zenphoto page loads are usually below 0.05 seconds on average).
Especially having a very dynamic UI (lots of page loads while browsing images, etc.) a photo gallery like Zenphoto has to be fast, and loading Wordpress on each page just doesn't sit well with that.
I do not understand the obsession my colleagues have with pre-caching ;-) The image generation only happens once, photos will be generated as needed. Especially on a fast server like yours you'd hardly notice the difference. But you can try that too.
Okay, finally my main recommendation. Integrating Wordpress and Zenphoto is not usually a good idea, for all the reasons above. It's pretty easy to take a Wordpress theme or original design and "port" it to a Zenphoto theme. Zenphoto themes are very simple, you can look at the existing ones to see where to put the code blocks and other than that it's just HTML and CSS as usual. A pure zenphoto theme would be tens of times faster than loading it within wordpress.
Good luck, just thought I'd explain that more clearly.