@sbillard A ha! I finally understand. Thanks for taking the extra time. I'll summarize this for other people who are confused.
Having a album published means that it will show up for all users (unless the site is password protected). If they don't have access permission on the album it will show up with a lock picture, but the user will still know it's there.
If you have a user with 'view all' permissions they won't see unpublished albums unless they have been specifically given access in the "Managed Albums" area. I found this pretty counter intuitive.
So to give you a step by step:
[list=1]
[*]Unpublish all you albums.
[*]Create a view all user that has 'View all', and 'Overview', rights.
[*]In the Managed Albums area for that user click each one of the albums.
[*]Create a limited subset user that has 'Overview' rights.
[*]In the Managed Albums area for that user click only the Albums you wish them to see.
[/list]
I might create a feature request thread for the permission 'view unpublished'. Thanks again.