hold it against someone

(verb) to blame someone for doing something bad

Example Sentences:

A: How do you feel about your father not teaching you to play sports?
B: I hold it against him. I wish he spent more time with me when I was a child.

Thomas held it against Beth because she had an affair with his best friend.

A: My roommate ate all of my food!
B: When?
A: Last month?
B: Why are you still mad?
A: I’m holding it against him because he still hasn’t apologized.