Home› General support

Cache Manager performance issues.

tplowe56 Member
edited December 2018 in General support

I'm having problems getting Cache Manager to process images, in vers. 1.5.1. I had been testing cache manager for another issue, which has been solve with a work around. The cache manager hangs after processing 60 - 150 cached image sizes. I am testing on an album with 120 images with 7 cached image sizes. So cache manager should create 840 cached images. It rarely generates more than 60 before stopping/hanging. I can't find any error mentioned in logs. The cache manager shipped with earlier versions would process 1000's of images with no problems

I increased the PHP memory limit tp 256m which had no effect. I have tested this on my local server and on my live server. Both hang after a short time.

I should mention that the reason I must use the Cache Manager is that if I don't create cache files locally and upload them to my server, GoDaddy throttles my account as soon as a user loads a bunch of thumbnail files. Running the Cache Manager on my live server also will throttle the account. If I generate the cache files locally myself, GoDaddy does not throttle my account, and everything works smoothly.

When ZP 1.5 was installed this introduced a thumbnail naming issue which had me researching if I had to recreate my entire Cache Folder. The gallery holds 14,000 images, so this would have entailed regenerating at least 14,000 new thumbnails.

Tags:

Comments

  • acrylian Administrator, Developer
    edited December 2018

    Do you have cURL enabled/available (setup will mention)? If not please see if you can and try again. Using cURL is new in 1.5.1 and helps issues with image generation the old way had.

    If you have cURL and something does not work there should be entries in the debug log of ZP.

    In any case regenerating all images versions for 14,000 images will be quite a task and in any case time consuming. Not to speak that it is quite some work on the server (thus probably why you host throttles your account…).

  • acrylian Administrator, Developer

    I don't have any test site with 14,000 images right now. But i just tried wiht 391 on a local test site with PHP 7.2.10 (currently quite high with 512 mb memory because of some other testing). Here are the numbers using cURL (five image sizes were to be created per image):

    • Image cache sizes generated: 1955/1955
    • Images processed: 391/391
    • Albums processed: 29/29
    • Processing time: 14.85 minutes
  • tplowe56 Member

    Thanks Acrylian, I have cURL enabled on both my local installation and my live installation. I generated thousands of cached image files back in 2016, and everything went perfect. With an Intel i7.

    I get throttled by Godaddy if I just upload a new album with 100 images in it, because the cache generation triggers a CPU limit on the shared server.

    To me it would be an xampp setting, but its happening on Godaddy also.

  • tplowe56 Member

    I just re-ran setup to double check errors. The only error I have locally PHP tidy support [is not present], which is enabled on my live server.

  • tplowe56 Member

    Acrylian, I have run cache manager just on the stock themes, with the same result. I also tried using another browser just in case that was an issue with same result.

  • tplowe56 Member

    Some more info:

    I have done clean install of Xampp, ZP and new database. Cachemanager still hangs when processing images. I have also double checked for file names that may be non-standard, and found none.

    This is the only thing that shows up in debug log:

    {11880:Fri, 28 Dec 2018 23:22:41 GMT} Zenphoto v1.5.1

    WARNING: "continue" targeting switch is equivalent to "break". Did you mean to use "continue 2"? in C:\xampp\htdocs\gallery\zp-core\admin-functions.php on line 863 require_once called from require_once (admin-globals.php [15]) from admin-edit.php [11]

  • acrylian Administrator, Developer

    Without any actual error it is hard to say or do something right now. How large are your images? Processing imges is still a heavy task for any server especially if they have high resolutions (dimensions).
    Can you perphas try locally with one album with just a few images just in case it is something specific to your images.

    I will try to raise the number of images locally to try with such a huge amount. In case this overloads something else…

    That debug log entry I have never seen before. I will take a look at the lines what this might be about.

  • tplowe56 Member
    edited December 2018

    Acrylian,

    I have done further research.

    I did a clean install of Zenphoto version 1.4.14 [f5b47da52f] (Official build).

    Using that version I can generate cache images at lightning speed on multiple albums.

    Cache manager in 1.4.14 is capable of the following:

    On 1 album, 120 images, 8 cache sizes, 960 images generated, less than 60 seconds total time.

    Using mass-edit function: 7 subalbums, 297 images, 5 caches sizes, 1485 cache images generated, less than 60 seconds total time.

    So using vers. 1.4.14 locally I could generate all required cache sizes for my entire gallery in a relatively short time, compress the Cache file and upload to server and everything is fine.

    Vers. 1.5 & 1.5.1 Cache manager seems to be incapable of any of this. For one Cache manager no longer works and the mass-edit capabilities also seem to have been degraded (or not available at all, not sure about that bc cache manager hangs during processing of first album.)

    So after introducing a code change that rendered 1000's of thumbnails useless, now you have degraded all capabilities to solve this problem that were previously available in 1.4.14. I've let it run for hours and it never starts to process again. PHP memory is set at 2048m.

    Sorry to be negative. I am trying to help diagnose this problem, but I can only test, I don't read or write code.

  • acrylian Administrator, Developer
    edited December 2018

    Thanks for the feedback. Sorry for your problems. Interestingly we introduced the change because the "old way" did caused sizes constantly not being rendered properly. The new way indeed takes longer because the images are renderde one by one as the other way started multiple tasks that overloaded some servers so the new way was more reliable in our tests. Of course we didn't test a huge site with 14,000 images with multiple sizes to be generated. I will try to reproduce the issue and maybe be able to improve it (or we add an option to still use the old way).

    But it should be possible to re-generate these using 1.4.14 for now.

    the mass-edit capabilities also seem to have been degraded

    We didn't work on the mass-edit functionality at all. So I am not sure what you are exactly referring to?

    One wish for the future: It would be great if you from now on could test the support build frequently so we can test such things before releasing a new version at all.

  • acrylian Administrator, Developer

    I am doing some investigation and found a few things. Have to do some tests, probably have nothing before next year.

  • tplowe56 Member

    Thanks Acrylian,

    Just to be clear, I would never try to generate all cached images at once. In 2016 I was quite successful in locally generating all sizes for my gallery going in batches. It didn't really take too long. Upon further research my gallery is 9,600 published images. I have 14,000 local originals. Also, I think the Mass-edit works fine, it's just hanging after a certain amount of time and goes no further, which is caused by cachemanager.

    Thanks!

  • acrylian Administrator, Developer

    Thanks, I will keep you posted about any progress.

  • acrylian Administrator, Developer

    Please try the master from GitHub (not just the plugin because there are a few new dependencies!). Unavoidable it takes quite some time to generate but I again tried with hundreds of images and albums successfully now.

  • tplowe56 Member

    Thanks so much Acrylian, I will be testing sometime this weekend.

  • tplowe56 Member

    Acrylian,

    I ran some tests and had the same results. We must have different testing environments. So to help eliminate unneeded anomalies, I started from new with still no improvement.

    My latest test:

    W10 - latest Update
    Xaamp latest build

    I uninstalled & reinstalled Xampp. Deleted htdocs directory.
    Started with fresh database for ZP.
    Installed latest support build of ZP.

    Live view of gallery generates proper cache sizes.
    Clear image cache.
    Generate cache sizes for Admin & Basic

    Cache-manager processing stalls every time. Album contained 120 images, cache sizes requested were for Basic & Admin.
    No new entries in Debug log. If album page running Cache manager is refreshed multiple times the process will complete. I takes 3 refreshes to complete 120 images.

    Here are screen captures of processing on screen when the process stopped. The process stops every time. Since there are no errors in Debug. I'm not sure where that leaves me going forward. At this point my only option is to return to 1.4.14 and set some workarounds in zpArdoise cache settings. The 1.4.14 cache-manager still runs fine and quite a bit faster than new version.

    I have bumped my PHP memory up to 10,240mb, which has no effect.

    Alt text

    Alt text

  • acrylian Administrator, Developer
    edited January 2019

    That is really weird but sadly I have no idea if there is really no error log entry anywhere. DId you view the PHP error log, too? The PHP memory should not have to do anything with this.

    The old cacheManager may work for you bit for me it always didn't process a lot of images and was quite unreliable. The new one goes through for me. This is how it should look like (processing one album with some sub albums in this example):
    https://www.zenphoto.org/test/cachemanager.jpg.html

    Please mail me directly so we can discuss this a bit more direct to solve thisand to find out what is different on your install. I just use a plain as is MAMP install for developing.

  • tplowe56 Member

    I will run more tests this week and see if I can find some errors in the PHP log.

  • tplowe56 Member

    Thanks Acrylian,

    I found the problem in the PHP error log:

    [Sun Jan 06 18:09:42.033999 2019] [php7:error] [pid 9676:tid 2096] [client 127.0.0.1:50612] PHP Fatal error: Maximum execution time of 30 seconds exceeded in C:\xampp\htdocs\gallery\zp-core\functions-basic.php on line 1756, referer: http://127.0.0.1/gallery/zp-core/zp-extensions/cacheManager/cacheImages.php?album=2018/Wagon Days - Sun Valley&XSRFToken=41c498e87cc15517e0497cfc1b166af501ca7b99

    So I increased the "max_execution_time" from 30 up to 360 seconds. Now it runs fine. So that issue is solved. Don't know why that is an issue now, but wasn't on the previous releases.

  • acrylian Administrator, Developer

    Thanks, as mentioned the cachemanager has been reworked to use cURL requests to create the images instead of loading all images via <img> tags (which still exists as a fallback if cURL is not available). That old way in my tests always resulted in images not being generated so it had to be repeated especially on lots of sizes and images to process. And it also loaded lots of MB on that page, too. This new way worked much more reliable.

    I do have the same limit on even my local server but didn't encounter it (I did within a Imagick library function which has been changed to avoid this in the master). But of course not all servers are the same and behave the same.

    If you can please experiment with some cURL settings. In line 1747 and 1748 of functions-basic you find definitions. Try setting that to false and raise the number on the time out (with your orginal execution time of 30). We will later also run further tests on our own server (which is shared hosting so nothing that special),

  • tplowe56 Member

    Acrylian,

    I ran the test as you suggested, the process still times out at 30 seconds. And generated the same php error.

    CURLOPT_RETURNTRANSFER => false,
    CURLOPT_TIMEOUT => 5000,

    This had no effect. Raising "max_execution_time" in php.ini was the only setting that allowed the process to complete.

    Let me know if you want me to try any other tests.

    Thanks.

  • acrylian Administrator, Developer

    Thanks so far. We need to do some more investigation but will take some time. We did tests on our own server which is just fairly standard shared hosting actually and all worked so far. Have to look what timeout setting we have here actually. We can actually try to raise the timeout in the function itself but that is last resort…

  • tplowe56 Member

    Acrylian, I used the Cachemanager locally for my entire gallery for 8 cache sizes. Worked fine with the max_execution_time set very high. Uploaded 76,000 cache files in about 7 zip files. Only took 1 day of work. I will test cachemanager on my live site, which has the 30 second limit, and see if it generates an error in the php log. I get throttled very quickly so I haven't tested yet.

    Thanks

  • acrylian Administrator, Developer

    As I posted on the other topic, it seems not that nothing is happening - at least what Fretzl reported from his server - but for some reason no output is shown. But it should frequently do that one by one (we are testing some modifications). Still investigation why this behaves so different and even without errors.

  • acrylian Administrator, Developer

    @tplowe56 The support build now pragmatically adds option to choose precaching using the classic image ouput way or the cURL way. Sadly we couldn't figure out yet why the cURL way behaves so different to weird on some servers. Anyway, now anyone can choose what works best and classic way is default for that reason…

  • tplowe56 Member

    Thanks Malte, that sounds like a good solution.

Sign In or Register to comment.