Fermat's principle is one way of rationalizing the bending of a light ray (or other waves like sound or seismic waves that have well-defined wave fronts) travelling from point A to point B when traversing across a boundary between regions of two different indices of refraction. As you correctly summarized in the first part of your post, the light ray bends towards the normal when moving from a region of faster speed to a region of slower speed.
In response to the specifics of your question, if you imagine the light ray moving in the reverse direction (from point B to point A) with the light moving from a region of slower speed to faster speed, you would see the light bending the reverse way as it retraces the path it took , that is away the normal. This symmetry is evident in Snell's Law
$$n_{1} \sin (\theta _{\rm incidence }) = n_{2} \sin (\theta _{\rm refraction })$$
as the equation applies for both path directions.
It is important to recognize that there are two different types of minimum paths (minimum distance and minimum transit time). The shortest path by distance is a straight line, but the shortest path by time is the bent path, which is what waves follow. As an equation:
$$time_{transit} = \frac{distance{_{1}}}{velocity _{1}} + \frac{distance{_{2}}}{velocity _{2}}$$
has a minimum value along a bent path.
This has also been explored by a math professor who would walk his dog along the beach and throw a tennis ball into the water. The dog ran at a faster speed on sand than in the water. The professor observed that his dog took a bent path that was close to the calculated minimum path predicted by Snell's Law, rather than chasing after the ball along the path of minimum distance.
I worked out this problem some time ago and am including a diagram of the main analysis. The first image shows the calculation of the time of transit for the straight path, with speed of the dog on sand (5 m/s) > speed of dog in water (0.5 m/s). The dog is 4 m from the water initially, and the ball is 12 m away from the sand. I made these numbers up.

The second images shows the calculation of how far the dog should run on the sand before moving into the water to take the path of minimum transit time.

The third image shows the calculation of minimum travel time from the geometry calculated in the second figure.

I also include a screen capture of a spreadsheet analysis where I verified that the path calculated has the minimum transit time among many path alternatives that vary among distance d along the sand before entering the water:

There is a completely independent way of deriving Snell's Law from wave fronts that I have illustrated in a diagram for another question:
Can a light wave front narrow or widen in refraction, and what does this really mean for light?