Lou may not have been everyone's cuppa tea, but he sure was mine. Brian Eno once said that the first pressing of the first Velvet album only sold 30,000 copies but everyone that bought a copy started a band. A bit of an exaggeration, but not by much. I did not get one of first copies of that album (I was only 8 when it came out) so that is why I did not start a band! ;-(
Regarding being "put off" by artists that do stuff outside their established domain: I am put off when an artist (or their fanbois) imply that everything the artist does is genius. But I see no evidence that Lou was saying he was god's gift to photography. He could put out a book of photography so he put out a book of photography.
I also do not have a problem when an artist generates a disproportionate amount of interest when they produce art outside their normal domain. It does not have to be rabbid fanboyism. If you relate to the work of an artist you may want to see how they see the rest of the world. It is like reading a biography about them. I see this as legitimate. I have bought books by musicians (eg nick cave, party smith), seen art exhibits by directors (Tim burton), etc. Not all of it has been great if it stands on its own, but most of it has been worth experiencing as it had helped me understand their vision.
So I will at least have a look at his book. And more importantly, My wife and I will raise a glass to him at dinner and bore our kids with why he matters.