Yes, quite often I get quite agitated when trying to compose a picture free from (unwanted) human contamination.
The worst example was when taking a picture of the change of the (Norwegian Royal) Guard (I used to one myself), when a person came wandering into the frame with a huge orange folder under one arm. He looked at me, but just didn't give an f.... I looked up from my WLF and saw it was Morten Krogvold, Norway's most known portrait photographer, carrying a carton of Agfa copying paper, no doubt to show some copies to his buddy HRH the King. I should have triggered the shutter. But I could not have used the picture for anything but personal use, as publishing in any form or manner, without the subject's permission, is forbidden in this backwater of the world.