Let's break this down, because it needs to be made clear that no matter what parallels exist between the two, they are inherently different things.
Point #1 -- Consumption
When you listen to music, you are engaging in a particular form of auditory consumption that requires very little in the way of thought processes. This is not to say that music cannot foster thought, just that the vast majority of people listen to music primarily for effect. The necessity for anything else does not typically exist.
This is not true with books. When you read a book, you are engaging several different sections of your mind. You are using visual thought processes on top of a string of cognitive processes that take in the words and translate them so your brain can make the appropriate visual or non-visual stimuli that denotes understanding. You cannot read a book without also thinking. It's impossible. To put it simply: we read books and listen to music. This is irrefutable. To say that this is not true is to essentially claim that anything we know about human culture and biology is 100% incorrect.
Now, obviously audiobooks change the equation a bit, but only slightly. All an audiobook does is change the visual process to an auditory one; everything else, generally speaking, remains the same, with exception to poor audio quality or annoying voice acting that can ruin the listening process. This leads us into point #2.
Point #2 -- Determining Quality
One of the primary problems with the self-publishing argument for the book/music analogy is that it intentionally ignores the process through which consumers determine quality. As with the modes of consumption, determinations of quality for books and music differ greatly, and this is linked directly to how we consume these two things.
With music, determining quality is typically immediate, with little time on the part of the consumer to create an opinion. Most people have particular listening tastes (such as only liking certain genres) and have different reactions to different forms of music. The result of this is that usually a consumer can tell if something will be enjoyable (of any degree) within the first few seconds (this also varies somewhat depending on the music. I hate country music, so when I hear two seconds of a country song, I tune out; but I don't hate all rock music, and sometimes it can take ten to twenty seconds to decide if I want to listen to any more of a song).
Books, however, require of consumers a considerable amount of time. One cannot, for example, multitask while reading a book (with minor exceptions), and so when a consumer reads a book, they have dedicated themselves to the process. Unlike music, determinations of quality in books are not immediate, and neither are they quick or smooth processes. Bad books are not always determined by the first sentence or even the first twenty pages. Sometimes a bad book doesn't show itself until the end, and getting there understandably takes time. Even if it takes you until the end of a song, chances are it will have taken you only a few minutes, as opposed to several hours.
The only way we currently have of determining quality in books is through editors or reviews; neither are perfect, and usually the latter is useless primarily because personal taste always enters into it--tastes are different from person to person.
Point #3 -- Indie Problematics
Self-published authors often try to claim that because independent music took off, so too must independent writers. The problem is that a lot of the times, these same authors have no idea what they are talking about.
The indie music scene is not a new thing. It wasn't even new when mp3.com and the various other indie music sites appeared. In fact, the independent music industry has been around since the early 1900s, and it has never been quite as non-traditional as people think. The creation of indie labels was not an attempt to allow artists to do whatever they wanted with their music, but simply a way of escaping a system of enormous record labels who wanted too much control; the big labels still exist, and so do many of the indie labels, who have since become rather large themselves. Additionally, true indie music is not nearly as glamorous as people think, and often the instances people cling to as great examples of how "self-publishing" can work are actually of bands/singers who already had enormous followings before going true indie. Some good examples of artists starting indie and being successful do exist, but they succeed primarily because of the first two points in this post.
The book industry, by the way, already has its own indie industry. They're called small presses, and these places publish all sorts of niche literature all across the world. They have editors and marketing teams too, but obviously are not as powerful as the big boys. But where everything falls apart in the self-publishing argument is when they make the assumption that if indie worked for music, it must work for them too. Well, that would be true if the first two points of this post were incorrect. Since they are not, the reality has to be acknowledged: all success in indie music is because of points #1 and #2. Consumers simply do not view music the same as books, and, thus, are much more willing to accept music as a self-published form. After all, a consumer can listen to samples of music and spend only a few minutes of their day doing so; they cannot do the same with books.
What all of these points come to is this: books are not like music, and the only way the indie music scene can function in an indie author scene is if the first two points are acknowledged and addressed appropriately. I'm not sure that is possible, considering that there are not quick and easy ways to make #2 work for books.
Did I miss any points? If so, let me know in the comments. I tried to gravitate towards the most obvious, but perhaps I missed one that should be mentioned here.