My experience has been that it's what size fish & what body of water they came out of, that determines how they taste. But, in any event, I'd shave off the red meat portion of the fillets, as that's where the strong (fishy/muddy) taste comes from.
I've eaten them, back in my youth ... and the larger ones were not all that good. But, they weren't filleted, and the red meat wasn't removed (on any size) ... so that may have been the problem.
I think a lot of people don't eat them, for the simple reason that they're just downright butt ugly ... and people associate their looks with what they perceive the taste will be.
It's also noteworthy to mention that Freshwater Drum are mentioned several times in the Consumption Advisory (PCB's & Mercury) for certain bodies of water. (see the Ky Fishing/Boating Guide for details)
... pappy