Kudos, Gene. Indeed it was Frederick Douglass. Later that day he went to the White House, and he had difficulties entering until Abraham Lincoln came upon the scene.
I came across the quote while reading Every Drop of Blood: The Momentous Second Inauguration of Abraham Lincoln by Edward Achorn.