Am I right to just ignore him?

Yes your right if you don't like him then ignore him but tell him that you have enough friends and you don't need more so he won't think your mean or rude.