The persons who turned down the invitation were sort of "colleagues" for whom meeting the president was not unusual but (daily) business, not a unique opportunity (at all, unlike today, the president was during his office hours accessible for everyone). Plus Abraham Lincoln AFAIK frequently, also at last minute, asked for company. Plus then like nowadays, I guess theater wasn't everyone's cup of tea.