You have a problem with either the wheel's bearings or the bearing buddy. They should not fall off. They should be hammered into place and you really need to work hard at getting them back off. In other words, they should fit tightly to the wheel bearing hole in your wheel.
Perhaps you have the wrong sized bearing buddies?
I've had the same set of bearing buddies on my boat's trailer tires for 30 years now. The only time one of them came off in all those years was when my right wheel feel apart one night on my way back home from fishing. The steel part of the wheel came off and the entire rubber wheel went out into a farm field for about 200 yards as I was going around a turn. I found the wheel out in the field the next day when it was daylight. And I found my bearing buddy in the ditch along the side of the road. The very inside part of the steel part of the wheel was still bolted onto the wheel hub after the outside part of the steel wheel rusted though and fell off on the curve. I went out and bought three new Steel Wheels and three new tires for the boat after this.