take on the google chat list

One thing I cannot stand about Gmail which I can't stand is that whenever you send a message to anybody with an Gmail address they get automatically added to your Google chat thing..  I get that this might be convenient for people who you are planning on communicating with on a regular basis but I cannot stand how adding them is basically a default

I'm looking over at that area right now and out of the 9 people who have a green dot next to them: my sister in-law, my brother's girlfriend, some dude I went to college with, a buddy's wife and five people who I cannot place for the life of me.. I scrolled over one of them and I honestly thought it might be Hacksaw Jim Duggan..   Not only do I not need to see this person in real life, I prefer to not accidentally have to see him topless on my Gmail if I happen to scroll over his name.

The worst thing is not just that adding people is the default action that GMail does but that I have absolutely no idea how to delete them from this list..