I had a similar issue with thumbnailing some large (4000x3000) images. The server error log indicated a memory problem, so I added this line to the beginning of the lib-GD.php file in /zp-core:
ini_set("memory_limit", "128M");
This was just before the line:
// force UTF-8
Of course, I have 8GB memory on this machine, so that was no big deal to just bump it up. I'm not sure if that value would be appropriate for your situation.
Once ImageMagick is implemented, this shouldn't be a problem anyway, if I understand correctly.