In a letter back sent by Dr. Mudd while imprisoned at Fort Jefferson, he wrote, "Owing to the peculiarity of my skin, and not much exposed to the sun, I am paler or fairer than when I left home."
So perhaps my memory was off when I thought he was in the sun, but it's still odd that a man imprisoned on a tropical island didn't tan.