Is this a toxic situation? Why does he do this? Why do I let him?

He obviously doesn't like you and you obviously don't like yourself. If you had any self respect, you would tell this guy to kick rocks and find someone who likes and respects you for you.