This is a common misconception when learning Netwon's law. You aren't alone!
The trick to understanding how to apply Newton's law in situations involving walls and sledgehammers is to break free of our intuition regarding how much force a sledgehammer actually exerts. Intuitively, we tend to associate the force of a strike (such as that of a sledgehammer) with its maximum force -- the force it would exert against an immovable object. A sledgehammer can exert a lot of force like this. However, our intuition leads us astray here.
What actually happens is that the sledgehammer while it is impacting the wall, is exerting less than it's maximum force. In fact, it's exerting exactly the same force as the wall is exerting on it (Newton's Third Law in action). The result of those forces is that the hammer is decelerated by the force the wall applies on it, and the wall is driven backwards by the force of the hammer applied on it. This force accelerates the wall away from the sledgehammer.
When the wall breaks, that decreases how much force the wall can apply on the hammer, and thus by the 3rd law, how much force the hammer can apply on the section of wall directly in front of the head of the hammer. This lesser force is still used to accelerate the wall and its reaction is still decelerating the hammer. Given that there is likely very little material in front of the hammer and that it has been accelerated by the hammer already (before the wall fully broke), the amount of force applied here is very small. Almost the same as if the hammer was just flying through the air.
What this does mean is that if you tossed a feather in the air and then struck it with a sledgehammer, the sledgehammer would apply a very small force to the feather, and the feather would apply an equally small force to the hammer. The result would rapidly accelerate the feather up to the speed of the hammer, and would decelerate the hammer ever so slightly.
Beyond this, another thing that may help is to remember that striking events like these are not actually instantaneous. They can take a few milliseconds to occur, and the forces can change dramatically during that time. As an example, you can see how golf balls deform during impact when hit with a golf club. I find this helpful because a lot of the confusion that can happen learning Newton's laws comes from the idea of a strike happening instantaneously. In slow motion, you can see a lot more is going on!
