Light is wave in the electromagnetic field and obeys a wave equation (3D) similar to a wave in a taut string (1D) or on a drum membrane (2D). You may imagine a "tension" whenever the field varies from place to place. This is simply a fundamental property of the EM field that occurs in the absence of any matter or charge. $$\frac{d^2u}{dt^2}=\nabla^2u,$$ where $u$ is any of the 6 components of the EM field (note that there are further constraints between the components). $\nabla^2u$ measures how much $u$ at a given point is bigger/smaller than $u$ at nearby points. $\frac{d^2u}{dt^2}$ relates that to how $u$ changes in time.
In the interior of a light beam, the EM field only varies in the direction of propagation, so the motion only occurs in that direction. At the edges of a finite beam, the field is strong towards the inside and weak on the outside, so the light leaks out of the beam. This is diffraction. A hole in a wall is an easy way to create a finite beam, but in the end the spreading happens afterwards and is a property of the light itself. E.g. if you could somehow create a finite segment of a plane EM wave just in empty space, it would still diffract as if it had just passed through an aperture.
You can also consider disturbances in the EM field as always "flowing" in all directions. In the interior of a light beam, the flows perpendicular to the propagation direction cancel out, since in each such plane the field is constant, while the flows along the propagation actually do something. But at the edges, the natural flow of light from the inside to the outside is not balanced, so we have diffraction. (This point of view is the Huygens-Fresnel principle.)
So no, the atoms/material of the wall have nothing to do with diffraction (no "pulling") outside of the act of destroying some of the light. Light knows how to diffract all on its own.