You're not crazy. Guys are much larger than girls, and are quite capable of hurting us, but you have to think - women are capable of hurting men (physically and emotionally) too. I agree with BananaPancakes though. I'm nervous around guys too, but only those that I don't know. A lot of my friends are guys so I've seen the weaker side of them.
This is most definitely not crazy. Guys (most) are disgusting pigs. You have every right to be nervous around them. Just don't hide away from them, because there are a lot of amazing ones out there.