I don't think it's a taste issue as much as a contaminated one. Large fish generally have more mercury/pcbs etc than small fish and are therefore more "dangerous" to eat.