Please keep in mind that it was a reply to Stephens "If slavery were abolished, what are we to do? I know that negroes will not work, unless forced to it, and I tell you that we shall all starve together."
(It seems Stephens either didn't know of those "honest, hard-working Southern people who never owned a slave and tilled the soil", or he didn't have much faith in them. It was just a question that left IMO no other reply. Should Lincoln have expressed pity instead?)