Sorting by Change Date

Tobias

Hi!

I know there already are some discussions (like this or this) concerning change dates of images, but I haven't found a good answer to this topic yet.
My goal is to get a list of all images changed since a certain timestamp to sync these with a remote CMS. Those changes should include editing metadata, refreshing metadata from file (EXIF, IPTC, …) and filesystem operations like moving images to another album.

Looking at the database table images I noticed the fields date and mtime, but these seem to correspond to the date/time an image was taken and the modification timestamp of the file respectively. Obviously metadata fields like EXIFDateTime aren't useable for this purpose either. The field publishdate sounded most promising, but is only used to schedule publishing an image in the future.
I also thought about using custom_data, but then it couldn't be used for anything else and couldn't even store serialized data because that would prohibit sorting.

Given that this seems to be a common question and that the topic comes up every now and then: Is there a best practice approach you guys would recommend? Is there anything planned for a future release?

Tobias

acrylian

mtime should actually be the image file date if was uploaded. Not sure right now what happens if you upload an image with the exact same name to replace it (via ftp for example). Then Zenphoto would not treat it as new and might not change the mtime value.

The question is also what is the change date here. Change to the image file or changes to the titlte, description on the Zenphoot backend. For the latter there are plans to add a last changedate as Zenpage items have already.

You can very well use the custom data field as it is an all purpose field. You can also use one of the other standard fields if you know you don't use them otherwise.

There is also a very inconvenient way to add further extra custom fields using the the fieldextender plugin. It requires some coding, instructioms are in zp-extensions/common/fieldextender. Handling of such extra custom fields will be largely reworked in the future (and may not be directly compatible - remember our talk about moving image meta data to a separate table).

Tobias

I have to revisit this topic as I have finally found some time to move forward with the sync plugin I mentioned.
I decided to use the field publishdate and update it to a current timestamp whenever an image is changed (metadata & filesystem operations). I might change that to changedate once that field is available, but for now publishdate seems to be the best approach for me.

I'm going to use the Root_class object filter save_object for this purpose, but the user guide does not explain the first (boolean) parameter of this filter. When invoked by PersistentObject->save() the parameter is always supplied with true, but I'd like to understand it's meaning in case it will be used more "dynamically" in the future.

Another thing I noticed is that this filter is only invoked right at the end of PersistentObject->save() so in order to change the data written to the database I have to update it and call save() again. For this to work I need to un-register my filter plugin first of course to prevent an endless loop. After saving I have to re-register it again. Any idea on how to improve that?

acrylian

Can't answer about save_object filter parameters as I don't know offhand.

You can remove a filter using zp_remove_filter(). It will be temprary since on page reload the normal process will re-register. Not sure I understand the loop issue but I am not into your problem. W

Just to note publishdate is a standard colum of the images table and is setup for datetime (YYYY-MM-DD HH:MM:SS) actually and not (UNIX) time stamps. Just to avoid confusion.

If you are really into have a changedate you could also use the general purpose plugin_storage table without needing to re-use "publishdate" for something else or even create a new column. (Not sure why I forgot to mention that apparently).

Tobias

The loop issues will come up if you try something like this:

function myFilter($bool, $obj) { $obj->set(...); $obj->save(); }

zp_register_filter('save_object', 'myFilter');
$mediaObject->save();

This will cause save() to call myFilter() which will call save() which will call myFilter() which will call save() and so on. So you have to un-register the filter inside myFilter() before calling save() in order to prevent the callback. Not the prettiest of solutions, but it should work.

Thanks for mentioning plugin_storage. I will have a look at it, but it might not deliver best performance if I add an entry for every image and filter over data (text).
(Is there an index for plugin_storage.data or images.publishdate?)

acrylian

Hm, I haven't looked at the code yet. But you should not need to call $obj->save() within your filter call. It should only modify the object before saving it by the normal process.

If it does not do that the filter might be implemented wrong for ages and no one noticed…

(Is there an index for plugin_storage.data or images.publishdate?)

No,actually no primariy columns. plugin_storage has indices on type and aux.

And the images table only on id, albumid and filename as those are the primary fields.

acrylian

Just have take a quick look as the save() method. It seems the filter is called after any db action is performed. So the data modified indeed will never be updated without $obj->save(). It's not even added to the cache as the filter does not return anything. At first glance it look like this is somehow not right at all. Have to search if this filter is actually used by anything in the core and if how…

Tobias

[…] you should not need to call $obj->save() within your filter call. It should only modify the object before saving it by the normal process.

That's what I expected as well.

Thank you for looking into this!

acrylian

I also have quickly looked at the new/update_* filters for Zenpage items and to new_image/album. These all do pass the object through so functions hooked can modify it but all are called after it is already saved. So also need to do an additional save if needed.

Therefore thinking of introducing additional filter hooks before saving objects and also to not break any existing usages.

but the user guide does not explain the first (boolean) parameter of this filter.

It is just useless. Probably meant as a check to be false on some occasion. Generally zp_apply_filter returns the first parameter value (unless the function hook does different things).

Btw, did you try to use the new_image or Image_instaniate filter instead of the save_object one in the root class? That should not create a loop.

Tobias

So zp_apply_filter() cannot even pass the object to a filter function because only the first parameter is passed, correct?

zp_apply_filter('save_object', true, $this);

function zp_apply_filter($hook, $value = '') {
    […]
    $args[1] = $value;
    $new_value = call_user_func_array($the_['function'], array_slice($args, 1));
    […]
}

Introducing additional filter hooks sounds like solid idea. Existing ones can still be useful to notify plugins of events like "object (successfully) saved".

I cannot use the filter hooks new_image and image_instantiate for my plugin because it needs to know whenever an image object is changed. At a glance it looks like new_image is only called for new discovered images. So all changes that are made later on like editing the title will not invoke my filter to update the change date. image_instantiate on the other hand seems to be invoked whenever a runtime [image] object is created. So this will be called way too often and also although no change was made to the image.

acrylian

So zp_apply_filter() cannot even pass the object to a filter function because only the first parameter is passed, correct?

No, the first is actually just default and passed through if nothing is hooked. Actually it returns what ever the attached function/method returns if it returns anything. So it could return the modified object even as the 2nd parameter if it was returned by doing $this = zp_apply_filter('save_object', true, $this); for example.

Thanks for the explaination why you have to use that filter. Makes absolutely sense. Since the root save() method does differ between "new" and "update" two new hooks could be added. Right before the foreach loops for preparing the db data occurs would probably be the best place. What do you think? Then I would the next days add them to to the 1.5.2a and you could test them.

(Sidenote: just tried the Zenpage ones and although they don't return the object either, they can be used to modify).

acrylian

Just took another look. Zenpage items have said "new/update" filter if articles, pages or categories are saved on the backend.

Images and albums don't have such named filters but actually the two existing save_image_utilities_data and 'save_album_utilities_data despite their name can do the same. They don't return the object but they are called right before calling save() on the objects.

Unless you really need to do this on class level those probably could help you, too? Or did you try those already?

Also would a new column lastchange (and matching lastchangeowner) help as Zenpage articles and pages have it? We thought of adding those for images and albums anyway to match the other items.

acrylian

I was wrong, seems a filter call like zp_apply_filter('somefilter', $value, $obj); would indeed update $obj if the function attached modifies it. No need to return or save here. Somehow this detail escaped me right now although lots of filters are setup… You could try that with the filters I mentioned above. The save_object filter still is not placed correctly for the actual saving IMHO.

Tobias

I'm sorry for the late reply! Had a busy week at work.
I had to debug zp_apply_filter() to get an idea of how exactly it is doing what it's doing.
Setting $value as $args[1] kinda threw me off as well as array_slice($args, 1).
(The first one is only required if $value was not supplied to zp_apply_filter(). The second one is used to truncate $hook from the list of arguments passed to the filter function.)

Since the root save() method does differ between "new" and "update" two new hooks could be added. Right before the foreach loops for preparing the db data occurs would probably be the best place. What do you think?

That's exactly what I thought.

Then I would the next days add them to to the 1.5.2a and you could test them.

That would be awesome!

Images and albums don't have such named filters but actually the two existing save_image_utilities_data and save_album_utilities_data despite their name can do the same. […]
Unless you really need to do this on class level those probably could help you, too? Or did you try those already?

I haven't tried already, but I'm not sure using them would be good idea. The description in the user guide sounds like they are intended for completely different tasks. So although they might work now, I think it would be better to use a hook that is intended to modify an object right before saving.

Also would a new column lastchange (and matching lastchangeowner) help as Zenpage articles and pages have it? We thought of adding those for images and albums anyway to match the other items.

Yes, that would be exactly the thing I was looking for. I can live with publishdate for the time being and migrate my data once those columns will be added (if they are to be).

I was wrong, seems a filter call like zp_apply_filter('somefilter', $value, $obj); would indeed update $obj if the function attached modifies it.

Are you sure this isn't depending on whether $obj is an Object or a variable? Maybe that would influence if it is passed as call-by-reference or call-by-value and hence whether you are modifing the original section of memory or just a copy of that.

acrylian

… but actually the two existing save_image_utilities_data and save_album_utilities_data

Well, they are in the same place as individual "save" ones would be like:

zp_apply_filter('save_image_utilities_data', $image, $index); $image->save();

Despite their name they are not limited to "utilities data" (which refers to plugins adding extras to the backend right sidebar). Please try them nevertheless. No need to add things if these work for your purpose

Are you sure this isn't depending on whether $obj is an Object or a variable? Maybe that would influence if it is passed as call-by-reference or call-by-value and hence whether you are modifing the original section of memory or just a copy of that.

I thought too but I didn't find any passing by reference directly yet. At least with the ZEnpage filter it works as I see saved changes without doing an extra save within the filter hook.

I can live with publishdate for the time being

Okay, I will consider to add these in 1.5.2a soon. Publishdate is really meant for scheduled publishing.

You see ZP is a complex tool that even we as devs don't always remember everything ;-)

Tobias

I tried all three hooks and while they work as expected when editing the image data (like title, description etc.), they don't seem to be invoked when refreshing metadata or moving an image to a different folder.
I was expecting PersistentObject->save() to be called on these occasions too because both operations will result in changes on the images database table.
Any idea why it doesn't work or what I could try instead?

acrylian

Yes, save_image_utilities_data is a backend only so onl applies when saving changes there.

The updateMetaData() method of the image class does have its own filter image_metadata. The meta refresh is actually handled by the gallery class method garbageCollect() which matches the database with the file system and it does call that method and does call save() on the image object actually. So the save_object filter should actually be used.

Tobias

I created Image->save() and added a debugLog() command to it. Then I activated DEBUG_FILTERS in global-definitions.php. This is what happens with my filter registered to hook save_object and move an image from one folder to another:

[start of request]
{12839:Sun, 17 Feb 2019 16:04:26 GMT}
[Zenphoto starting to register all the filters]
[…]
{12839:Sun, 17 Feb 2019 16:04:26 GMT}
media_change_date.php=>media_change_date::updateChangeDate registered to save_object at priority 5
[…]
{12839:Sun, 17 Feb 2019 16:04:26 GMT}
Image::save() was called from:
#0 zp-core/admin-functions.php(2557): Image->save()
#1 zp-core/admin-edit.php(278): processImageEdit(Object(Image), 0)
#2 {main}
{12839:Sun, 17 Feb 2019 16:04:26 GMT}
Apply filters for save_image_utilities_data
exiftool::putEXIF
[This another plugin of mine. Proofs filters get applied.]
{12839:Sun, 17 Feb 2019 16:04:26 GMT}
Image::save() was called from:
#0 zp-core/admin-functions.php(2557): Image->save()
#1 zp-core/admin-edit.php(278): processImageEdit(Object(Image), 1)
#2 {main}
{12839:Sun, 17 Feb 2019 16:04:26 GMT}
Image::save() was called from:
#0 zp-core/class-image.php(857): Image->save()
#1 zp-core/admin-functions.php(2564): Image->move(Object(Album))
#2 zp-core/admin-edit.php(278): processImageEdit(Object(Image), 1)
#3 {main}
{31647:Sun, 17 Feb 2019 16:04:26 GMT}
[Zenphoto starting to register all the filters again after roundtrip]
[…]

So although the filter is registered and Image->save() gets called multiple times the filter is not applied.

acrylian

Thanks for the tests. The image class move() method actually does call save() if there was something to move actually. So this will need some more investigation I fear.

If we add a new column for last change I would implement that they are automatically set with the change date automatically. But nevertheless the filter probably should work as at least we expect it to do.

acrylian

lastchange and lastchangeuser are now implemented in the support build. lastchange is set to the current whenever an object is saved automatically. Regardless if by code or if it is from the backend pages via direct admin request. In that case the lastchangeuser is set additionally, otherwise it is empty (backend reads "Code request" for now).