Nope, Michael, not Fessenden. The correct answer was never a Cabinet member.
Hint #5: This man's support for Lincoln declined after the issuance of the Emancipation Proclamation. He was upset to the point that he considered voting for the Democrat in the 1864 Presidential election, but in the end, he backed off that position.