Sometimes very young children are afraid of clowns, along with people with masks on, and other things that obscure a person's face.

For anyone older than that, I have to assume it's influenced by media.