Could this be a result of little kids being viewed as closer to "babies" then not, and as such closer to a mothers / women's responsibility? Or could this be a fear driven pattern of behavior, where men are viewed as predators and thus are avoided?

I think it's both. I think people both see women as more capable of raising and caring for young children and are also wary of predatory men.