I am the same :)
The thing is not everyone thinks the same way and therein lies the prob - for example, when you're in a relationship and the other person belongs to a big friend circle or has a big family then there are only so many invitations one can decline...without being judged
I've made peace with my decision to enjoy my company - it would be great if those close to me to understand and respect my decision