I think it's high time we stop feeling guilty about judging books by the outside. With people it's a different matter, but a designer has the opportunity (and obligation, I'd say) to make the cover of a book reflect what you can expect in terms of a reading experience. Even brand covers should give you a heads-up, imho.
Covers are a large part of the reason why people interested in animal husbandry don't pick up Silence of the Lambs as a veterinary handbook, for example. You immediately get an idea of who the audience is (age group, gender, sometimes even race), a ballpark of genre, and a sense of what the tone of the book may be (note that chick flick is almost never brown or black). If the inside doesn't match the expectation I formed from the cover, I'm not ashamed to say I'll be annoyed. I don't see anything wrong with that kind of judging.
I feel like this might be why I hate movie book covers so much; it always seems to me that the designer grabbed a still or two at random. The whole movie's about this text; where can you go wrong, right? The result always looks to me more like someone wanted to slap a popular actor's face on a product than establish a meaningful relationship between cover and text. Movie covers are just another version of celebrity-endorsed cereal to me.
