I think a cane (or walking stick--contemporaries seem to have used the terms interchangeably) was considered part of a man's formal dress.
At his first inauguration, Lincoln carried a gold-headed cane (possibly the one pictured in the second link), which he used to keep the manuscript of his inaugural address in place.