I am sure this is a dumb question but how do I delete a photo?

From the back-end:
Login first > [i]Albums[/i] tab > pick an album > [i]Images[/i] tab > click the radio button [i]Delete image[/i] in the [i]Utilities box[/i] on he right side of the page > click [i]Apply[/i]

or

From the front-end (if you are logged in) :
Browse to the image (single image page) you want to delete > [i]Admin[/i] (in the top menu) > [i]Edit Image[/i] > you are directed to the back-end > click the radio button [i]Delete image[/i] in the [i]Utilities[/i] box on he right side of the page > click [i]Apply[/i]

or

from the backend on the "album sort" tab each album has using the bulk actions.

or

directly via FTP within the album within the main albums folder.

Just curious, does deleting via FTP in the albums directory also delete all the cache variants of the image?

No, FTP bypasses almost everything. On the next album visit it is only cleared from the database. You would have to clear the cache manually.