People, precisely because they're people with their imperfections, will get you (me) down at certain points in life. And since we live in an imperfect society with a million different social and historical baggage, it's hardly surprising how people can easily make you (me) feel bad about your (my) own life.
The point is: words can hurt if you let them. Also, don't forget that your biggest critic is probably yourself.
