I have a crush on my ex

As the title states I may have a crush on my ex-boyfriend. Well... we dated in middle school and I broke up with him. We dated for like two or three weeks. We didn't get very far either. He was so shy that he barely held my hand. We never went out on an actual date so that's why I'm not even sure you can call him an ex because I'm not sure you can really say we dated.
Sorry, back to the problem. I may have a crush on him. His name is James. He is really nice, and pretty cute, and so sweet. Even though his friends totally resented me when I broke it off he didn't.
I have been thinking about him a lot lately and I'm afraid I'm falling for him!!! The problem with him is he is so quiet and shy. I am like that so two of us is just not going to work. He isn't outgoing enough so I'm afraid it will just turn out to be another dead end relationship.
What should I do. Should I take the plunge and go for it or finally let it go and just move on. I need to figure it out because I'm starting to flirt with him and I don't want to lead him on.

HELP!!!!!