Zenphoto deleting and recreating cache by album on page load

J_C

I've fully cached my live site using cacheManager. I've noticed when I go to test it that thumbnails in certain albums are loading super slow, with some albums taking multiple attempts to fully load. When I look into the individual album cache folders, I noticed that the contents are being deleted and then recreated. So what once had all the images cached, will, after usually a few attempts to get the page to fully load, have only the newly created thumbnails.

This doesn't seem to be affecting albums with only a few images, I've been noticing it on those that contain 25 images or more.

When I look as a page loads in albums with problems, as the thumbnails go through i.php they are all taking a long time to load, and many of those then start getting "HTTP/2 500 Internal Server Error" in the console.

As some of the thumbnails start to show 500 errors, they will also show the following in Firefox-

NS_BINDING_ABORTED

or

NS_ERROR_NET_INTERRUPT

or

NS_BINDING_REDIRECTED

When I look at the actual server logs, I'm getting hundreds of-
SoftException in Application.cpp:690: Could not execute script "/host/zenphoto/zp-core/i.php", referer: https://domain.com/

I've checked permissions on the files and folders, and they seem to be okay. I checked .htaccess and it is the default 1.4.12 version installed by Zenphoto. I'm not running Obscure Cache Filename, Protect Image Cache, Secure Image Processing, or IPTC Imbedding.

The Debug log shows nothing. This is on 1.5.7 and I'm using Imagick.

I've set the memory_limit to 256M in php.ini (which is the max for my host).

I've tested this in Safari (14.0.3), Chrome (89.0.4389.114), and Firefox (87.0)

Running &debug on the thumbnails with problems loading seems to show nothing special-

Image filesize: 1791921

Debug i.php | Arguments:

    size = 250
    width =
    height =
    cw = 0
    ch = 0
    cx =
    cy =
    quality = 85
    thumb = 1
    crop = 

Any ideas on why the image processor is going through and deleting the cache by album and reloading everything? If it has an issue loading something, it seems it shouldn't delete all the cache files. That is only making a slow problem slower.

acrylian

The image processor itself does either take existing files or creates them. It should not delete them by itself on normal theme usage.

As already mentioned in other places the image procesing is a heavy taks and if you have huge images (remember: dimensions not file size) it may consume quite some server processing power. WHich is independent on the memory size. It may just be too slow to be ready in time. Just a guess.

It may also be theme issue. Which are you using?
It also would really help to see this issue live as I didn't encounter this anywhere so far.

acrylian

Please also review your PHP error log. Also check if you have some server security active in case that complain on the i.php name.

J_C

I also tested this on my local MAMP v6.3 test server with 1.5.8RC, where I wouldn't be running into the same limitations as my live server, and the issues were there as well (pages just eventually loaded faster). Thumbnails were still not loading, showing the NS_BINDING_REDIRECTED error, and album cache folders were being completely reset when it happened.

On MAMP I have the PHP memory limit set to 1024M, so I don't think that should be it.

I'm using a slightly modified version of the Basic theme, with only some small cosmetic changes. The only real difference is that the options for it are 50 thumbnails for albums and 100 for images. I was getting this on pages with 26 thumbnails though, so not a lot.

I'm not getting anything new in the PHP error log, I am getting in mysql_error_log-
[Warning] InnoDB: Table mysql/innodb_table_stats has length mismatch in the column name table_name. Please run mysql_upgrade

J_C

To try to eliminate any issues, I set the theme to Basic, reverted all the theme options back to default. The only thing I kept the same was thumbnail size at 250 pixels longest side, not cropped. The issue with deleting the cached album contents remained. Some of the albums I was experiencing it on had only 9 or 15 pictures.

acrylian

Sorry, I cannot reproduce this issue of cache files being deleted. If they are there they are servered. If resized images are not loading or show a broken image it is normally a sign that they are not yet cached or too large to be created in time. Normally after reload this disappears.

All browsers nowadays do very aggressvie caching so sometimes they try to serve things from cache still although the site itself already changed. Always clear the browser cache manually.

Without seeing the actual site it is hard to help.

J_C

I found out what’s common amongst all the folders that are having their caches reset! They are all at least three folders within the main albums folder. It doesn’t matter how many files are in there, I saw it happening to folders with only one picture, it’s not the folder name, or the contents, it’s where they are in the folder chain. It’s not happening to the first folder, or folders within that, but third folders down.

So-
albums/Test1/Test2/Test3/Test4

When Zenphoto opens or accesses the Test3 folder after cacheManager has created the entire cache for it (this can be even for loading a random thumbnail from it for the main gallery page), it will delete the contents of the cache folder and start new. Same with folder Test4. For folders Test1 and Test2, it will use the existing cache for them fine.

This was tested in 1.5.8RC.

J_C

I should add that after it deletes the album cache and starts to recreate it, Zenphoto seems to keep it after that. You just have to manually create the cache by visiting all of the pages for it.

The result is if you reset or create the cache with cacheManager for a large site with a deep folder structure, instead of now being fast to load, it will be extra slow as it tries to create the cache for many folders at once, especially if you have thumbnails for the gallery set to random.

acrylian

I am sorry, I really cannot reproduce this and the image processor does not delete any cache files. It checks if the requested size is there and serves it or creates it if not.

The cache is not even ever cleared automatically, you have to do this manually via the button on the overview (only available if the cacheManager is active). Even running the cacheManager does not clear the image cache.

I can only assume something on your server is doing something it should not do for unknown reasons.

acrylian

I really don't understand how anything could be deleted automatically.

There might be a delay of creating files when running the cacheManager via cURL mode as those are basically separate processes that are started so it keeps going in the background. And the larger the images the longer it takes to create them especially if there are a lot of size registerted.

Pre-caching ireally a power consuming task which a server may even be not capable of.

But all this does not explain "self deleting" anything…

especially if you have thumbnails for the gallery set to random.

That setting is really not recommende if having lots of images and lots of subalbum levels. Even with cache images. Better try to set a fixed one instead.

acrylian

I really need to see this myself to help. Send me a mail via our contact page if you can't or don't want to link the page in public.

J_C

I figured it out, it's cacheManager. cacheManager is only creating the cache for the first two folders down.

I got it wrong by looking at all the folders after opening the gallery page once. Since my thumbnails were set to random, it was creating the folders for everything with three folders down and more, and only putting the new thumbnails in those folders. I thought it had deleted the contents, but you were right, nothing was being deleted.

It won't even create the cache folder for a second level folder if it contains folders past three or more down, that added to the confusion of trying to figure it out.

Since it was taking cacheManager so long to try to cache my site, it wasn't something I could easily test. It was only by starting from scratch, and comparing a screenshot from before when it finished, that I noticed it processed the same amount of images. It was only about 1/3 of the total images, and 2/3 of the folders. I thought the folder number was folders that only contained other folders, and that the image count might be off because I tried to run cacheManager before.

This was with cURL, but I tried to go back after and use both cURL and Classic, but it did not add anything to the cache.

acrylian

Okay, good. It may be a limitation of your server/host plus having to many images and too many sizes at once to work on. I can only recommend to not upload too many stuff at once and perhaps pre-cache by album if you really need to. And try to limit the sizes really needed (like sizedimages and thumbs) since if missing they will be done when they are anyway.

Or lower the size (=dimensions) of the original full images so processing is less power consuming maybe.

J_C

I tried it in MAMP on my local computer with very few limitations, and on my live server, both were only scanning and creating caches for the first and second level folders.

I even just tried it in MAMP again with a new gallery with only nine folders and less than 200 images. It failed on every folder except for the only one with images two levels down. The other folders contained images three and four levels down.

Is it possible there is a restriction in the cacheManager code as to what is being scanned?

acrylian

Not that I am aware of (except dynamic album naturally) and I am sure it all worked when the we tested this with thousands of images and a number of sublevels.

But I just tested again locally and indeed on an album with three sublevels it only covered the toplevel and first sublevel. So there is indeed a bug here. Strange as nothing was changed here to my memory for quite some time. Firs thought that it accidenttally ignored them because being unpublished but that wasn't it. Thanks for being persistent on this. I will look into this.

acrylian

Found it already, a simple typo in a function missng just a single "s". Please try 1.5.8RC update.

J_C

It worked! I did have to restart it a few times, but it eventually cached most of my files.

I did get a couple timeout errors using cURL, which is odd, since I had both max_execution_time and max_input_time in php.ini set to 200 and it failed on 30. If it happens again, I'll look into it some more.

[10-Apr-2021 16:50:41 America/Los_Angeles] PHP Fatal error:  Maximum execution time of 30 seconds exceeded in /Applications/MAMP/htdocs/zenphoto158test/zp-core/functions-basic.php on line 1782

[10-Apr-2021 22:25:05 America/Los_Angeles] PHP Fatal error:  Maximum execution time of 30 seconds exceeded in /Applications/MAMP/htdocs/zenphoto158test/zp-core/functions-basic.php on line 1779

Thanks for fixing it!

acrylian

Somehow your server timeouts on the curl request. Probably it is just not capable to do the huge task.

Try to lower the value on the Caching concurrency: option (Options > Image) if that perhaps helps.