This has happened several times after upgrading 1.2.9 to 1.5.7. All files in /zp-core/setup/ have .xxx extension added to file name (class-setup.php.xxx). Requires uploading clean files and re-running upgrade to restore admin access. What's causing this?

This is setup protection and occurs after setup has run. It does not require uploading of files: Please see here:
https://www.zenphoto.org/news/installation-and-upgrading/#--re-running-and-re-uploading-setup-files

How to login as admin if setup is disabled?

Any way to disable this feature?

How to login as admin if setup is disabled?
Please see: https://www.zenphoto.org/news/login-and-password-problems/#----what-is-the-link-to-login-on-the-backend--
Any way to disable this feature?
There is no way to disable this feature

How to login as admin if setup is disabled?

Not sure what you mean exactly but logging in has nothing to do with the setup files being disabled actually.

path to your installation/admin
path to your installation/zp-core/admin.php

Both links go to gallery front page, not login. Running Basic theme.

path to your installation/admin

If that doesn't work you either don't have the htaccess file in place or modrewrite is not working

path to your installation/zp-core/admin.php

This should lead to a login form. please review your error logs.

There is no .htaccess in upgrade package /zenphoto-1.5.7/zp-core/. Found 'htaccess file version 1.2.8.0' I had saved a month ago, no idea from where. Uploaded to /zp-core, now /zp-core/admin.php works.

There should be a htaccess template file within zp-core which setup would use to create the actual root .htaccess file from. And it is for sure there:
https://github.com/zenphoto/zenphoto/blob/master/zp-core/htaccess

If this for some reason didn't happen - sometimes setup failes to recognice that modrewrite is active as there is not really secure way to check that - you can just copy it to root and rename it. Change the rewrite base to your install's and/or re-run setup.

Thte htacces from 1.2.8.0 is outdated, current version is 1.4.12.

Ok, I've copied /zp-core/htaccess from upgrade package to /zp-core/.htaccess. There already was a /zp-core/htaccess, looks like upgrader copied it without renaming it. I upgraded directly from 1.2.9 to 1.5.7, there were a lot of errors but gallery seems to be working ok. Are there any problems I should look out for?

Ok, I've copied /zp-core/htaccess from upgrade package to /zp-core/.htaccess.

You misunderstood. The .htaccess file must be in the root of your install. The backend does not use rewritten urls.
https://www.zenphoto.org/news/installation-and-upgrading/#--content-of-the-zenphoto-package

It was already in /photo root, it's now in /photo and /photo/zp-core. I get 404 at /photo/admin, login at /photo/zp-core/admin.php. Login is successful, so we're good. Thanks.

There should be no .htaccess in photo/zp-core as the backend does not use modrewrite itself. Except the /admin URL which is handled internally and not directly via a htaccess file.