The definition of fly in the ointment is:
(idiom) a minor problem that spoils something
A: Did you have a good day at the beach?
B: It was all right. There was a fly in the ointment, though. Some teenagers there were being noisy.
A: I hate teenagers!
The only fly in the ointment during our whole vacation was when our flight was delayed.