Try downsizing a couple of sizes when that happens, I have the same problem with catfish and it is usually smaller fish biting. The smaller hook improve my catch ratio on the smaller fish. Good luck.

Joe