How many sympathetic free people were there willing to actively help versus the "benefits" and interests on the other side? Enough? Quite a lot of people are too convenient to care much about others' fate as long as not directly concerned, then as now. And also in the north there were those fearing the slaves to be freed.