1865 Southern Maryland road and town map.

[mike86002000]

I doubt Parson Wilmer knew anything about Booth and Herold. He was a staunch Union supporter and very well known for it. I think he and his descendants would be outraged at the implication that he had anything to do with them.
The possible reason for Booth and Herold wanting to go to St. Paul's was that direct road to Port Tobacco from there.
I don't know where Wilmer actually lived. At one time, I think he was pastor at the church in Port Tobacco. Perhaps at the same time he held services at St. Paul's. That would have made him a frequent traveler on the road between.
Mike