The Comment Swap function is buggy, so it isn't that reliable. It would be better to host comment swaps in your blogs.
Some people are silent readers, it can be frustrating, but there's really nothing to do. Sometimes they really just don't know what to say. Keep doing what you're doing.
You're in the same boat as a lot of people I think ... I get the same people comment which I appreciate but it'd be nice for other people to comment as well and SNAP!!!!! with the blooming comment swap I do it for them and don't get nothing back so I gave up doing them :/