Sounds like he's using you, and you said yourself you don't want him all the time so I would say just break it off. If he comes back after you grovelling then make sure he treats you properly even if it is just a on the side thing you've got going on :)
Honestly, I disagree with Jai. Forget about doing what makes you happy and pray about it. Chances are God'll give you a solution that sucks right now, but it'll also make you a lot happier in the long run. Any guy that uses you as the back-up plan in ANY way isn't worth your time or affection. And that sounds so cliché, but it's true. If you're not the most beautiful amazing woman in the world to him, he's not worth your energy. And if he feels that way about you he'll let you know.