How can I adjust the size of the flowplayer playlist, not the actual flowplayer player that you can set in the admin page?

The song titles are too long for the playlist box and gets cut off.

The flowplayer playlist is pure CSS styled HTML (with JS of course). The css file responsible is within zp-core/zp-extensions/flowplayer3_playlist. But you don't need to change that ifself, just copy the file to your theme's folder and change as needed. It will then be used instead of the original plugin one.

Btw, this is all written on the plugin doc:
http://www.zenphoto.org/documentation/plugins/_zp-extensions---flowplayer3_playlist.php.html

For infos how playlist itself work please refer to the documentation on flowplayer.org.