Might need to try a 5" worm instead of a 7. But when you feel the tap tap tap of the fish on the other end, you usually wind up any slack line you might have out and rear back and set the hook.... Should work, maybe try a 1/0 hook to see if it's just smaller fish messing around with the tail of the worm also. Might even just try a inline Mepps Spinner or Rooster Tail.