To me it seems like this is a difficult question to answer. To a certain extent the genre tags are important. They help book stores figure out where to put things and are great for categorizing in libraries and online. I think the question applies, however, to the growing number of works that don't really sit into any one genre. Take Zoran Zivkovic's work. His work is highly literary, in my opinion, but there is a lot of that "fantastic" element within each story, some more so than others. It would be somewhat difficult to pin Zivkovic's work into any particular genre. I would call his work "magical realism", but at the same time it is a bunch of other things. You just can't pin his work down. However, there are other writers who are easier to pin down (and I don't mean this in a negative manner). Tobias S. Buckell's novels Crystal Rain and Ragamuffin are rather easy to define. Well, at least the latter is. The first reads very much like a steampunk/fantasy novel, with SF stuff towards the end, and the latter is very much an SF work.
I will concede that there is obviously little need for genre tags for novels that are just too difficult to pin down. After all, I think it is something unfair to stick something into the fantasy section that is also literary, since people who read literary fiction might not read fantasy and therefore wouldn't find that novel.
But I think that the genre tag is more important than Ford and Chabon seem to think. I think writers do have a certain level of interest in how their books are categorized. If you wrote a science fiction novel and they put it in the westerns section, you might be a little miffed about it. After all, you worked hard to write a science fiction novel and now all the people who would be interested in your work won't find it. Folks who write very specifically within certain genres probably care a lot more than people who don't necessarily write in a genre.
Then what about the reader? I don't read a lot of literary fiction, except what I have to read for school. I just don't like literary fiction. It bores me and quite honestly I read to be entertained. Reading shouldn't feel like a chore to me. So, I don't go into the regular fiction section or the literary fiction section when I'm at the book store. I have no reason to. I spend my time wandering the scifi/fantasy section, and occasionally I'll go to the literary criticism section to see if they have anything interesting in relation to SF/F, or into the science section, or something similar. I'm not the only one that does this. There are a hell of a lot of readers who do the same thing. Why? Because many of us simply don't like anything else. We like space ships and swords and the like. All fiction is escapist. For me, escaping means actually escaping, not pretending that real life is the same thing as an escape. I live real life and quite honestly, I don't want to read about it. That's me.
If you took away the genre tag I would be extremely irritated. It already takes me an hour at least to go through the science fiction and fantasy sections at Borders or at Logo's (this awesome used bookstore downtown). What would happen if you suddenly got rid of all the tags? Well, a lot of new SF/F authors might find themselves in a bit of a pickle. The great thing about the SF/F section is that I can find new authors whose work is of interest. If you take away the tag all the authors will simply be put into a giant fiction section. There's a good chance I'd never find a new author into the field because of this. There is a huge market for SF/F and if there are works that clearly belong in those genres that aren't labeled that way it can be a big problem for that industry. I think there is a lot of value in having genre tags, particularly for the fantasy market, which has sales through the roof and is showing no signs of stopping. If everything that was fantasy suddenly became nothing, people would actually have to pull every book off the shelf to see the cover to make sure it is what they'd like to read.
Keep the tags. They're useful.