http://www.zenphoto.org/support/topic.php?id=10466Wow, so finally I concluded that regardless of when it's done ImageMagick in zenphoto crashes my server seemingly due to abundant memory usage by parallel processes where image magick by hand (convert) works fine and also in G3. and I get... "go to G3... thread closed" I was at least expecting "so resize by hand"
I wasn't rude or demanding and even said it's one of those things where ok, maybe for now that's how it is. But I would think the dev would say, "gee if our multi-threading crashes someone, that's a bummer. It's corner case for userbase(maybe maybe not) though and probably not in our time and scope to sort it out now though." It seems the devs here don't want any critical feedback and take it as insulting. Actually, I spent far more time(not that it was much) trying to udnerstand the problem-- downloaded the beta from svn, investigated code differences with how G3 does conversion, and would have investigated more-- than the time I was trying to save by using what seemed like a more user friendly program. Why? Because I'm trying to be insulting? I read a review about ZenPhoto where someone ditched it for this very reason. I guess now I get it. Feedback isn't insults. If you don't want feedback don't open source your code with a public forum.
Maybe this will get me banned. If not, the funny thing, I may keep looking at it as I get a chance and if I find a solution I'll post it back. I respect the work done here in spite of the odd responses.
Zenphoto
Comments
Zenphoto is a volunteer project, so the customer is not always right.
We really do not care how G3 (or any other gallery "does it".) If those are better for your needs, be gone. If they are not then be straight to the point.
As to Imagick crashing your server. Neither (yes, there are only two of us!) of the developers has access to a server that supports Imagick so cannot even test. If your endevors find somehting that can be corrected please submit the change.
"We really do not care how G3 (or any other gallery "does it".)"
Ok, that says a lot when theirs works and yours breaks. It could turn out it's something about my server; that's always a possibility though.
You're volunteer I get that. If you aren't interested in developing ZP, don't. If you just can't address this issue, even possibly with input, say so. No need to get snippy.
Case in point about snippiness, maybe you should actually look at my judgement before you tell me what maybe I should do. I did go ahead and use the product and the judgement that it doesn't automatically detect what theme is installed and thus what images it needs (at pre-cache time) held up just fine (maybe I overgeneralized guessing that there was NO interface to the theme :P ). Sorry for trusting the description (which was correct).
Before this ends in a unnecessary flame war. We appreciate any feedback and of course critism and we don't just ban someone... If you look around you will see that we help a lot if someone asks and often more than on other forums...
But please understand that your quite long postings sound a little too demanding, sarcastic and insulting a la "there it works and with our 'crap' software it doesn't". Note, that we are a much smaller team than G3 (which you used as a reference). Try to be a little more objective. Common issuse that written text is easily missunderstood. Maybe you just wanted to provide feedback but it reached us on the wrong foot.
Of course we cannot really care about any other tools and how they may work. IMHO Zenphoto is focussed more on personal portfolio websites than huge pure image galleries as G3 might be.
Once again: Zenphoto does not know about images sizes needed on a theme (unless ituses the default option values) until the page is requested. You have the flexibility to create a layout with any size and changing on every image if needed. The only way to make those precachable is to register the sizes as mentioned on the other topic.
Also, the imagick support was done by a co-developer that currently lacks the time to work further on that. So there are some thing that might not be right or optimal. That is the nature of software. There are a few tickets open and postponed for that reason). I do have a server (well, shared hosting) with Imagick but I don't use that much and big images on there.
If you want to help improving the code, you are invited to work on that and/or open a ticket to make suggestions. Open also a ticket about the crashing with all technical details. Maybye Our co-developer can look into those later. As said we are smaller so these things might take a while. That is the way to do it.
If it's crashing for memory issues, though, then that's not something that Zenphoto or Imagick actually handles beyond the very basics. Regardless, there is a memory limit option for Imagick (which is exposed as a Zenphoto option) that could potentially alleviate your problems. It will cap the memory usage of any given Imagick process, the default being unlimited, meaning that it will leave it up to the server to manage the memory. (see http://us3.php.net/manual/en/imagick.setresourcelimit.php)
Please be aware as well that G3's implementation of ImageMagick differs from ours. G3 directly invokes `convert` via PHP's `exec`, while Zenphoto uses the Imagick PHP extension which exposes ImageMagick's functionality directly to PHP. (I saw that you already know this from your other post, but it's good to put here in case someone else stumbles upon this in the future.)
The reason for the difference is that ImageMagick itself is fairly ubiquitous whereas Imagick, while being available on many hosts, is not as readily available to everyone. It's a trade-off between code "correctness" and a more hackish, yet absolute way of handling the images.
In regards to `httpd` processes using both memory and disk I/O, processing images will necessitate loading the entire image into memory, which can be very resource intensive. During processing, Imagick will use the temp directory set by ImageMagick's `MAGICK_TEMPORARY_PATH`, memory mapping images to the filesystem, thus the I/O.
One more thing: the way that the Zenphoto image processor works is that each image will generate a separate HTTP request. So, based on server setup, each image will spawn a separate process.
"One more thing: the way that the Zenphoto image processor works is that each image will generate a separate HTTP request. So, based on server setup, each image will spawn a separate process."
Thanks for the reply. You hit/reviewed many relevant points, but I think this one is probably the key. I know about the imagemagick memory limit, but it doesn't help (in the end converting large images takes large memory). I doubt the convert vs php interface is the problem either, but I think we both recognize it's a difference so it is something to be ruled out possibly. As far as I can tell both interfaces behave pretty much the same though.
For whatever reason G3 seems to serialize more and that's the difference.
For me I think what I would need for ZP to work is a precache mechanism that serializes, limiting processing to one file at a time without needing to globally limit the number of httpd or process on my server at all other times.
In the other thread I made some other comments regarding ideas on how themes could register all their caching needs (even as the theme is used and configured in different ways, for anything but truly on-demand image sizes) so the cacher can be aware of what images to produce. This is a more general idea about making a precacher better integrated and work more easily (probably for developers and users in the long run), an idea which evolves naturally once a specific use case like mine really requires a precacher.
However the core point is that a use case like mine does actually require a precacher, and specifically requires a strictly serialized precacher. As it turns out, the parallelism problem exists in the precacher too.
http://www.zenphoto.org/trac/ticket/2260
Performance is always a trade-off between memory, CPU, and disk. In an environment as you have described, memory and CPU are the first resources to be exhausted and the parallelism exacerbates the issue. It's typically not a bad thing to max out the RAM and CPU as long as it isn't causing problems.
http://www.imagemagick.org/script/resources.php#environment
In a threaded `httpd` environment, it might be possible to use `MAGICK_THREAD_LIMIT` to enforce the max number of concurrent threads. In a standard `httpd` environment (that spawns processes, not threads), limiting threads alone will not work. A better solution might be to use `MAGICK_THROTTLE`. Doing this, you can be sure that any given caching process will, at convenient points in the processing, yield to other processes that are vying for CPU time.
As for the precaching mechanism, the best solution would be to have the user select how many images are precached at a time (with a sane default). In your case, you could choose to have one image cached at a time. So, at any given time, you would have one image being cached plus however many are currently being cached on-the-fly. Once all images are cached, it sounds as though the server can handle concurrent requests.
To me, this is really the issue here: the precaching mechanism can be too resource intensive for some servers. As last I recall, precaching is considered unsupported, so it's really up to the devs what to do from here.
Given that fact, and the fact from your ticket that you are using uncompressed bitmaps, I think the most prudent advice at the moment would be to use processed images on your site instead. That will help alleviate resource issues and you can provide the uncompressed version of the images outside of Zenphoto.
As last I recall, precaching is considered unsupported, so it's really up to the devs what to do from here.
Since Zenphoto is normally used for "normal sized sites/galleries" so it is considered normally not necessary. Bitmaps file types are also no "standard web format".
Caching is actually supported because we got a lot of requests to it because similar reports. We have a cache_manager plugin now.
It is actually quite intregrated but it requires the sizes set manually if you set custom sizes on the theme directly which always overriding backend settings and are only "known" to the theme otherwise (a great flexibilty feature IMHO if you need fixed sizes or maxspace images for whatever layout reasons), You can even let your theme (or plugin) register sizes to the cache manager. Examples are found on the themeoptions.php of standard themes like Zenpage.
Sbillard is the better person to respond on the technical site but unavailable until around the weekend.
I'm not using uncompressed bitmaps. I'm just using full size jpegs. However when imagemagick resizes, it must uncompress teh image to a bitmap (at least one copy) in memory and it seems that's exactly what happens because a 5MB jpeg using soemthing like 100MB of memory as I recall during resizing. That's fine for one or two at a time. For me, 10 or even 5 at a time is not fine.
That's the crux of the problem. I can serve out 5MB images on my bandwidth very well and that doesn't use that much memory. I cannot have imagemagick uncompress a bunch of them though.
Yes figuring out how to make the precacher do one image at a time would solve the problem
...other than the "and whatever images are also being cached on the fly" issue. Once things are precached, and lets say we add one more image that's not, I'm sure I can disable on the fly caching by commenting out the call to imagemagick in one line. I don't know if some page will cause errors, but that's even ok for my case.
However when kagutsuchi you say on the fly is so integral to things, I think for this purpose what this really means is that even the precacher is really using the on-the-fly mechanism and is just asking for a url for each image or page and letting the on-the-fly thing do it's work. So what it really means to say it's very integral and cannot be disabled is that someone would have to actually make the precacher build the resize command for each image on its own, without quite as much code reuse, but still some. Once all the iamges are there, commenting out the on-the-fly call to the resize will certainly not break anything since that code branch won't even be reached, so it's just a matter of getting them there in the first place.
You should be able to configure your server to limit the threads that a web application can use. That is the simplest and most effective solution.
Of course you don't actually comment it out to do it right, you wrap it in a conditional and return one of those broken image icons.
I've actually done quite a lot of programming at the under the hood level, not user interface level, so I'm not just talking from the perspective of how I think the world should be. I just don't know php or uncommented code. I have looked through the code enough to have a little clue though even regarding this specific code.
There is already as I recall a conditional that says if the image is not there, then "precache" it. You add one more conditional inside that says and if disable_precaching is true then return a default blank image or broken icon or page error or whatever level of elegant failure you want (even do nothing and rely on sanity checks/errors/failures elsewhere).
The only issue then is satisfying the first conditional.. ie making sure the all the cached images are already there. And so again it comes down to making a precacher that doesn't just leach off the on-the-fly mechanism. It can still call the same code snippet to do the resizing though once it gets the information it needs about the image name and size requirements. It just can't do it by simply requesting to GET that image.
I get that on the fly caching is beloved by the developers but here are 4 facts I hope we can all agree on:
a) if an image is already in the cache directory (and we're just talking about a file in a directory) it won't be cached on the fly when it's next requested and the Imagick scale call will not be made.
b) It is possible to put an image in the cache without using your on the fly code. In fact it can even be done by hand or by bash script so long as the person or script has access to information about filenames and sizes needed as the precacher does.
c) If the on the fly code (or me personally) can get that information at precache time then another code also can.
d) Even if an image is not there when requested, failure is an option. And well that's obvious: if you use precaching only and you don't precache something, then a request for the image fails. That's true in any pre-cache only system but that's like a "der" kind of thing that is obvious to the user when they select the option to disable it.
Please tell me which of these 4 statements is not true.
global bool on_the_fly_enabled
Show_image(image scaled_image, bool precache){
If scaled_image exists then
display scaled_image
elif on_the_fly_enabled==true or precache==true then
resize (image_source, scaled_image)
display scaled_image
else
burn in magnificant flames of destruction! (or fail gracefully)
fi
}
Precahcer(){
For i in all_scaled_images_list {
show_image(i,true) > /dev/null &
c note the metaphorical ampersand above should be removed
c to serialize this
}
}
Notice that I even make a precacher that DOES use the on the existing on-the-fly code(I'm sure it's not necessary but it sounds like that's the case presently), but with minor modifications to add the on-the-fly and precache booleans in that on-the-fly code.
Maybe the precacher works on a page by page basis or whatever, fine.. it still just needs to set a flag with some scope limited to it's own process to enable the "on-the-fly" code. The philosophy doesn't change. I would think it could just call the bit inside the conditional directly but I'm trying to be a team player and work within the philosophy of always using the on-the-fly code.
Second, an image's size in memory is dependent on the dimensions of the image, not the filesize. These links give an overview of what goes into loading an image into memory:
http://imagemagick.sourceforge.net/http/www/FAQ.html#C9
http://www.imagemagick.org/script/architecture.php#overview
Third, here's a quick synopsis of how Zenphoto's image processor works. When a Zenphoto page attempts to display an image, it will use something like:
``
For each `` tag, the browser will make a separate HTTP request. `i.php`, the image processor, will return the appropriately sized image either from the cache or as soon as it has been processed.
This is not a process that can be switched on/off with a boolean as the entire architecture of Zenphoto has been built around this philosophy. It takes advantage of how browsers and web pages work by offloading the crux of processing to a separate process to be displayed once it is ready (also known as lazy loading).
Gracefulness may not be an issue to you, but it certainly is to the average user. There should never be a use case in which an image which could be displayed is simply not. It's disingenuous to call it an error and it's negligent to say "too bad".
The real issue here, in my opinion, is that the full-sized images are simply too large for your server to process. It's like trying to run the latest and greatest game on a 90s Compaq - it might run, but it will never be fully playable. You can massage as much performance out of it as you want, but it will never be optimal.
If you can find a way to make it work for you, then that's wonderful and we would love to hear the solution so that other users might benefit from it! You could even write your own precacher plugin from existing code. It's entirely possible, but it does not sound as though this is a route that the devs will put into development.
These responses are getting a little strange.
Every image that is displayed is decompressed. Monitors do not know about discrete fourier transforms. They know about RGB pixels and that's bitmaps. All image maipulation tools short of a few things like jpegtran that can work directly with DCT coeeficients.. decompress images. Most web servers don't manipulate images though, so most don't decompress them.
But never mind that, regardless, the fact is imagick uses over 100MB of memory to manage a 5MB jpeg and that's an observable fact. So if we disagree about decompression lets just leave it that we do and I'll get back to the important point and my results
Whooh.. on to the important stuff, ok I made a partial solution so far as you suggest and am sharing as you suggest. I just did this to your code:
function cacheImage($newfilename, $imgfile, $args, $allow_watermark=false, $theme, $album) {
}
Guess what? It works fine. Nothing dies catastrophically. All pages work just fine but show the usual broken image icon when images are not available, exactly as expected. Even the slide show page operates but with broken image icons. The web page loads, the layout is fine, nothing burns up in flames at all. Wow.
So... yes you CAN disable on-the-fly caching.
Three things:
1) Defaults
2) Errors when errors are
3) Admin choice
The devs like a content providing system that also has ABILITY for automatic content creation and that's probably great for some (obviously not for many), but one VERY simple change (that I just proved) can allow other users to enjoy your wonderful content providing system without using your content creation system. It doesn't have to be the default, but it can be an option (admin choice). But content providing takes FAR less resources than content creation and not everyone needs both or at least not on-the-fly.
Almost every content providing server on the planet gives an error in some form when linked content has not been made available. It's weird to act like it's the end of the world to your high expectation user base that if they CHOOSE to turn off automatic content creation and do not pre-create the content, that they would then somehow be massively turned off by a very expected and appropriate error, the same error response that most every web server in the world would give when content does not exist.
So long as all images exist your content providing work beautifily with on-the-fly creation disabled with a one line code change. With on the fly creation disabled, it behaves exactly as one would expect when content is not available too. I can't see why you keep acting like this one very well modularized routine is critical to the working of every line of code in your project. IF that was really true you would be the worst programmers in the world. But it is not true, as I knew, and as I have now proven.
I have not tested every theme and every plugin. Maybe one will give an error in a slightly less graceful way, but an error is still expected and an error is what it will return. Most will probably just produce broken image icons though as such sane behavior is I think built deeply into the html/php tools used. Other functions, IF ANY, that produce slightly less desireable error forms can be worked around probably with a three line sanity check somewhere.
The full size images are not too large for my server to "process". I can serve 40 at time. I cannot resize 3 at time.
As said in the ticket response.. disabling multi-threading on my server is dumb becasue multithreading is used in many ways. the argument was used as a reason NOT to serialize content creation, but it's exactly the reason why serializing instead of reducing server threads IS the right solution.
There is no reason to require parallel content creationg level of resources to allow people to use your great content serving software. They are two separate needs with different resource needs and it's trivial to actually separate them.
I meant that there are also many for whom on the fly creation is not great. It came out wrong.
Okay, some further reading reveals that, yes, ImageMagick does decompress JPEGs in memory. This is necessary as image manipulation requires the raw image data. Quality is what is irreversible.
You did not read my links on images in memory. Let me pull the most relevant part here: Image processing necessarily uses a great deal of memory no matter how you go about it. This is why machines dedicated to high resolution image generation/manipulation are typically loaded with RAM. You simply can't expect the same performance from an outdated machine; it is ill-equipped for image processing. I'm not sure what the complaint is here.
The idea behind on-the-fly caching is that it is used only when needed. If the cached image exists, that will be used, and the cached image only needs to be generated once. Your use case is what the precacher is for. What happens for the user who isn't the sole provider of his site's content? What if someone who is not technologically savvy is meant to upload images? His images will not appear until they are explicitly precached. This is not an optimal use case. For your usage, on-the-fly caching IS failing gracefully. It's not that the content doesn't exist, it's that it hasn't been generated in the correct form yet. Your idea that throwing an error in this case seems strange to me. Very few sites serve solely static content these days. The majority of it is dynamically-generated. And yes, it is extremely off-putting to a user when a piece of software that is meant to display images does not display an image.
In the end, if you get a solution working, feel free to post it and we can discuss specifics at that point. Until then, we are just arguing in circles. Everything said above is my own personal opinion and the devs may disagree. I will be interested to see what comes of this. Please refrain from ad hominem attacks. Let's keep the discussion civil. Thanks.
I have never said it is impossible to change Zenphoto's architecture. What I'm saying is that, since Zenphoto's inception, this has been a distinguishing feature compared to other gallery software and our paradigm seems to work quite well for many users. If a solution to your problem is implemented, it will need to cater to both sides of the fence.
Personally, I feel that the ideal solution can be reached by reducing the size of your uploaded photos and/or creating a sequenced precacher. Yet again, just my opinion.
As one last aside, the mechanism we're talking about involves processes, not threads. They are quite different.
"I have never said it is impossible to change Zenphoto's architecture. What I'm saying is that, since Zenphoto's inception, this has been a distinguishing feature compared to other gallery software and our paradigm seems to work quite well for many users."
really? here's your last quote about it:
"It is impossible to turn off on-the-fly caching without either major architecture change to Zenphoto or implementing some manner of IPC. "
(ok not ACTUALLY impossible, but are we playing word games here or what?... what we know is it is possible WITHOUT changing any architecture.)
I turned it off with one character of coding and got exactly the same behavior as other systems that don't have this "distinguishing feature" without any architecture change or IPC. One character of change (plus the option and conditional) can give your users the choice to use a traditional style server (with its traditional errors IF AND ONLY IF THE ADMIN MESSES UP) or your on-the-fly server.
It's a great distinguishing feature but also one that is easily turned off. Providing the option would give people who like everything else, but not this, to still use your code. I get why the feature is nice, but having the mentality that people either enjoy your code the way you think they should, or don't use it is kind of silly. I get that too if we're talking about a bunch of work, but we're not. We're just talking about a single conditional with a null action that gives people choice and expand your userbase to people with different usage ideas than your own.
"Personally, I feel that the ideal solution can be reached by reducing the size of your uploaded photos and/or creating a sequenced precacher. Yet again, just my opinion."
Giving a solution to a need to by suggesting one changes their need is disingenuous, especially when the real solution is simple.
Yes, a sequenced precacher is the point, but we go in circles on this: sbillard says why should he sequence the precaher when I should reduce my threads to 2. In the same post he says, reducing threads on a server is bad ... so go explain that. Discussing something with you guys is like playing basketball with a moving net. If you're going to make a sequenced precacher you don't also want the ability for one browser request to bypass that.
Content creation takes TWENTY TIMES more resources than content serving (as you now seem to get) and they CAN BE OPTIONALLY SEPARATE NEEDS. Forcing them to be linked when they don't need to be is odd.
Most recently (in the ticket) sbillard says he does think this is possible, which also goes against what you are saying (this time) and what I've proven with an extremely simple code change.
What could your massive architecture and IPC code overahaul that is just way too complicated for my small brain to get possibly change to make a static content option behave better than the one I made with one character? How could all that supposed required work to do this do better than giving a broken image icon?
Ok.. I'll give an actual answer(even though one is not needed). If you really want to make the option cool you COULD make it remove the missing images from the pages. That is NOT a massive architecture change either. Somewhere all themes plugins, pages determine what images are available. One simply needs to improve that decision to add "has all needed resized images" as oposed to just "source file exists" to the definition of "available" when creating the list. I would say that improvement is something you could do before you take the "beta function" qualifier off the option check box, but it's nothing like a requirement to roll out the option.
Other static content servers including ones far more popular than your dynamic one, just give broken icons though, and many people are very happy with that. It's true using their usual precache(import) tools that error won't happen, but again, you can also get to that as a future enhancement without a massive overhaul and without requiring use of an import tool.
You have the ability to go two or three better than them. You can give the default option for dynamic creation AND you can have an option for their static method but easier (because you can just copy the images in before hitting the precacher). So far this is all with almost zero work... AND if you want you can do a little more work and get rid of the broken image icons (in case the admin didn't do his job or something gets deleted).
And hey, I'm not facebook guy. It's your code. You don't owe me anything. But if you're willing to discuss it, I'm going to point out where your points don't make any sense. And while you don't owe anyone anything.. it's just a shame to let your ideals get in the way of spreading the usefulness of your work and optionally reducing resource requirements by about TWENTY TIMES over such a simple option. It's weird to have so many thousands of lines of amazing code and to pin your pride in the whole thing on a fetish with one small feature coded in relatively few lines.
But generally the caching is an quite integral part of the image processor and Zenphoto's feature
that all from sized to cropped ones is created automatically is quite a good feature. Especially for the "normal website owners" I do setup sites for.
It is the behaviour of Zenphoto since its creation actually and we don't get that much complains about it at all. Which is always a guide about this core behaviour.
We/I do appreciate your ideas and discussing them. It would be sbillard's decision since he would be the one who would do these changes.
I also understand the affection with this good feature. I think I've also shown though by actual trial, that while the feature is central to the thinking, the nice structured coding used does in fact make it a piece of cake to safely remove, even if nobody ever thought to make it so easy. Clearly it was a core idea and I think even the programmers just assumed that it is inseparable since everything was built with this idea in mind and nobody ever put any thought into how to remove it. It's weird to me that sbillard still thinks it's impossible to remove after I did it though.
I might make a brief (unlike me) new summary thread explaining my findings in a shorter way. Maybe someone who does know php will carry it on some day. I might still try to hack the other half of it (the serialization). But I won't be programming the option buttons.
We're not saying that what you're asking can't be implemented, we're saying that it won't be by us. If you come up with a solution, we will take a look at it then, but do be aware that it's not guaranteed to be included in the core code either.
Suggesting that someone does not upload huge images is our typical advice, so much so that we have an article dedicated to such issues: http://www.zenphoto.org/news/why-are-some-of-my-thumbnails-sized-images-not-generated-why-does-zenphoto-make-my-server-crash-while-processing-images-
Your need can still be fulfilled by uploading smaller images and offering the full-sized versions outside of Zenphoto. Content creation would be automatic and the full-sized images would still be available.
Or the precacher should do its thing. Images that are not cached (or fail to cache) could be set as unpublished. Images that are newly uploaded could start as unpublished and the new precacher could set them to published as they are successfully cached. There will need to be warnings when caching an image fails, though, otherwise users will become confused when their image is missing without warning, thinking that the precacher worked correctly. Some ideas for your implementation.
But I still personally feel it is a better option to use the precacher and fall back on on-the-fly loading. It's the best of both worlds and, IMO, a server should be appropriately equipped to handle such a situation.
One other note is that "has all needed image sizes" is not always a static requirement for all users. It can change over time and code needs to account for that.