Problem with "umlaut" in photo filenames

Opunte

Hi,
I'm currently exploring zenphoto for a family intranet with arround 20.000 photos. Performance and look&feel is great, but I noticed some problems with filenames:

If a file or a folder contains a german "umlaut" like ä,ö,ü I cannot view the original file (option unsecure is set in image settings). If I set it to download the photo, this works.

I've seen this error in the httpd logs when viewing such a file:
[Mon May 21 21:26:24 2012] [error] PHP Warning: htmlspecialchars() [function.htmlspecialchars]: Invalid multibyte sequence in argument in /usr/local/www/zenphoto/zp-core/functions-basic.php on line 241, referer: http://www.intranet.local/fotoalbum/test/

And for videos with preview image (video.mp4 and video.jpg in a folder) the display of the preview image does not work if it contains umlauts. Instead the default image is shown.

I have tested this with Zenphoto 1.4.2.4 and the default theme on FreeBSD/Apache 2.2/PHP 5.3.

Can someone help debugging this?
thx

sbillard

When you installed Zenphoto did you make the character set test file as suggested?

Your problem is that the character set of the file name is not what Zenphoto has been told to expect. If you did to the test file and paid attention to setup warnings these things should be properly discovered.

Opunte

The charset is set to what the setup script suggested: ISO?8859?1.

I've just done a test with a new installation of Zenphoto 1.4.2.4, created "zp-data/charset-tést" and the setup recognized it. The setup (german) message was "Der Zenphoto-Dateisystem-Zeichensatz ist als ISO?8859?1 definiert [bestätigt]".

Hm, I'm a bit clueless about what to test next.

acrylian

I can only recommend to avoid any special characters in file and folder names. (I am German as well). This makes your life really easier... If uploading via the backend, Zenphoto has a tool to "correct" these.

Opunte

My name has a "umlaut" in it...

Since Zenphoto is able to create thumbnails from photos, it can actually read them. Only some functions have problems.

acrylian

My name as well... Still it makes a lot of things easier as this encoding stuff is a really complicated part...

sbillard

Be sure your site character set is set to UTF-8. If it is then I do not have any ideas on this. The actual failure is trying to html encode a string (presumably conatining the umlaut diacritical.) The error message is saying that the character is not coded correctly for UTF-8 which is the internal character set of Zenphoto.

Opunte

UTF-8 is set as site character set. What I just noticed: from another client the image URL works. Both are Firefox, but maybe it's something with language selection? I'll do some more tests.

sbillard

There is a third possibility, but unusual, that the URI will need to be in UTF-8. There is an option for that as well.

Opunte

> There is a third possibility, but unusual, that the URI will need to be
> in UTF-8. There is an option for that as well.
Tried that, but then nothing works

I just switched to protected images and those links work. So this is ok.

But this still doesn't:
> And for videos with preview image (video.mp4 and video.jpg in a
> folder) the display of the preview image does not work if it contains
> umlauts. Instead the default image is shown.

I've seen that some URLs are generated from the template-functions.php, so I'll try changing those first...

acrylian

Note that videos and especially video players may have their own issues with umlauts. We have no hand in those as they are third party.

You should of course never modify core files.

Opunte

>Note that videos and especially video players may have their own issues
>with umlauts. We have no hand in those as they are third party.
I use the flowplayer plugin, never had problems so far.

>You should of course never modify core files.
Why not? If I can merge my changes with new releases?

acrylian

Why not? If I can merge my changes with new releases?
Well, you are free to do that but it will make our support hard and might also have unexpected side effects.

Opunte

Ok, I understand.

Sooo, back to the preview image of videos.

I've searched my way from zp-core/zp-extensions/class-video.php, to zp-core/functions.php to zp-core/functions-basic.php and found safe_glob() where I changed line 1445 from

return $glob;
to
return utf8_encode($glob);

which fixed the problem for videothumbnails with umlauts.

I've tested this with 4 files in an album:
"video mit ö.jpg"
"video mit ö.mp4"
"video.jpg"
"video.mp4"

Before changing the multimedia file "video mit ö.mp4" was shown with the standard thumbnail mp4Default.png, after changig I got the correct thumbnail "video mit ö.jpg".

Bad news: the change breaks various things in the admin backend (plugin listings, plugin options, ...)

Suggestion: maybe image files need another globbing treatment like backend files which also takes care of the character-set that was configured via setup.php?

sbillard

This change will certainly break other parts of Zenphoto. Generally speaking zenphoto will convert file names to and from the filesystem character set as needed. Perhaps there is some place that is not being done, but it is not to be done in the general glob function.

If you know the calling sequence leading up to this function then perhaps we can tell where something is missing.

Opunte

I traced it like this:

zp-core/zp-extensions/class-video.php, line 82:
$this->objectsThumb = checkObjectsThumb($album->localpath, $filename);

checkObjectsThumb is in zp-core/functions.php and is called with the multimedia filenames.
There the file with umlaut is passed, just add
echo $image;
after line 199 to see it.

On line 202 safe_glob is called with the filenames, which is in zp-core/functions-basic.php.

Add an "echo $file;" in functions-basic.php after line 1434 and you see that "readdir" gets the filenames with a different charset not UTF8. My browser cannot display them and shows a "?".
Use "echo utf8_encode($file);" to show the filenames correctly.

So the problem is that safe_glob does not return the filenames with umlauts in it. They are not returned to checkObjectsThumb. So for those files class-video.php asumes the default image.

I hope this is what you meant with "calling sequence"

sbillard

Thanks for your efforts. Actually, safe_glob returns file names as stored in the file system, which it should. Zenphoto should be converting them as needed by the useage. Seems that this was not happening correctly for these "stand-in" thumbnails.

There is a correction for this in the nightly build of Zenphoto 1.4.3 Beta. If you can we would appreciate your testing the fix.

Opunte

I'll test the Beta and report back here.

Opunte

Tried it with version 1.4.3-BETA[10213]. Strange thing is: sometimes it works, sometimes not.
I've checked that the thumb images are generated for all testfiles in the cache folder, but the preview of the thumbs works exactly one time, after clearing the cache. Refreshing the browser shows only photos without umlauts. Strange...

In addition, there is a httpd referrer error, when trying to view any full sized image in protected mode:
[Wed May 30 10:09:21 2012] [error] [client 192.168.97.2] script '/usr/local/www/apache22/data/zenphoto-test\\index.php' not found or unable to stat, referer: http://localhost:8080/zenphoto-test/test/icon.png.php

sbillard

Please make a trouble ticket for this. Try to put as much information as possible in the description. For instance what the links are when the image shows and what they are when it fails. Any other thing that seems relevant.

Not at all sure what might be the issue with that second error message. certainlyh index.php must normally be findable or nothing would work.

Opunte

Here's the ticket for the thumbnail issue:
http://www.zenphoto.org/trac/ticket/2179