If you don't want to be with him, then saying no is perfectly okay. You shouldn't feel pressured to be with someone if you really don't want to be with them. It's fine to tell someone no if you don't wish to go out with them.
It was absolutely fine to say "no". Especially since you don't want to get back together,
Hanging out would give him hope, in his mind.
And that would create a bigger problem, because he would keep trying.
He might not see it now, but you're doing him a great kindness. because no matter how much he might hurt now, letting yourself hang on to hope that isn't there, that "maybe someday" thought is a dangerous one.
I've been the person in his situation, wanting to hang on.
And though it would have hurt initially, I would have been better off in the long-run if my ex would have done what you're doing and just try to make a clean break.
I think you have chosen right. :) Especially since you don't want him back. (It would be different if you did.)
:) I hope everything works out smoothly in the end. :)
Don't be afraid to say no.