I think what you've been told is correct, in my opinion. If it doesn't have some kind of growth (algae, muck, etc.) on it, I think fish think its a foreign object an move on. I would try to find "established" cover, meaning something that has been there awhile that the fish are familiar with.
On the other hand......think of when the water levels rise in a body of water. Fish move to all the freshly flooded willows, cypress trees, etc. when the water is way up.
Now I'm dumbfounded. I guess, if I had the time and wasn't fishing a tournament, I would fish it all!