It seems appropriate, in some odd way, that people seem to talk about Diane Arbus more than her photographs. Personally, I have never enjoyed looking at her photos (as good as they are technically) as they unnerve me. Maybe that's what she wanted. I don't know. To me, the sitters all appear odd or damaged and, for whatever reason, I always visualise them as hospital / institutional mugshots rather than having an aesthetic.
Few other photographers I can think of seem to fit into the same niche.