After 30 years working with horses and mules, a few observations. It is not unusual for those suffering from some mental deficiency to be very well adept in handling horses. I think that they appear less threatening to the animal.
I am ducking now to avoid the rocks thrown at me, but women often give off the same vibes. I took my wife shoeing whenever I had a recalcitrant animal to handle because she could calm him down better than a man. A female generally is less threatening than a male--although I have known a lot of woman horse owners who were hell on wheels with their animal.
A man with a hat is sure to panic a flighty horse. I always took mine off when approaching a flighty horse or mule. If you are real careful, put the horse's nose in your armpit, or blow very gently up his nostrils. Careful, a quick shot of air will be perceived as a challenge.
If you put your hand on a horse, he will read you like a book in seconds and know if you mean what you say or can be manipulated. Some will do this even before you touch them. As a horse shoer, you have about 10 seconds to get the horse comfortable with your presence, especially if you are new to him. Otherwise, it is rodeo time!