The electromagnetic potential $A^\mu$ is a four-vector, and hence transforms in the fundamental representation of $\mathrm{SO}(1,3)$, i.e. $A^\mu\mapsto \Lambda^\mu_\nu A^\nu$ where $\Lambda$ is the usual 4x4 matrix associated to a Lorentz transformation.
Your question seems fundamentally confused about the difference between the field and the particle. The field transforms as an ordinary four-vector. A particle does not. This is because you have to differentiate between two different kinds of group representations here:
Each (classical) field $\phi: \mathbb{R}^4\to V$ has some target space $V$, where $V$ is a finite-dimensional representation of the relevant symmetry groups. In the case of the electromagnetic potential, $V$ is the fundamental representation - the "vector representation" - of the Lorentz group $\mathrm{SO}(1,3)$.
Upon the Hilbert space of states of the quantum theory, there is another, unitary representation. The unitary representations of the Poincare group are necessarily infinite-dimensional (the Lorentz group has no finite-dimensional unitary representations), and the possible infinite-dimensional representations correspond to particles by Wigner's classification.
The finite-dimensional representation $\rho_\text{fin}$ and the unitary representation $\rho_\text{U}$ are related by one of the Wightman axioms:
$$ \rho_\text{fin}(\Lambda)\phi = \rho_\text{U}(\Lambda)\phi\rho_\text{U}(\Lambda)^\dagger\tag{1}$$
where on the l.h.s. the $\phi$ is acted on as the classical vector it was, and on the r.h.s. it is acted on as the operator-valued field upon an infinite-dimensional Hilbert space it is in the quantum theory. This relation essentially is there to ensure that the transformation of the field as an operator upon the space of states of the quantum theory is consistent with its classical transformation behaviour.
The masslessness of the photon does not reflect in any change of the commutation relations of the Lorentz algebra (and I don't know why you would think it would). What the masslessness does is change the little group (see surface of transitivity or e.g. these notes), which is the group under which the four-momentum of a particle is invariant. For massive particles, the little group is $\mathrm{SU}(2)$, but for massless ones, it is the two-dimensional Euclidean group $\mathrm{E}(2)$ (or $\mathrm{ISO}(2)$).
Different little groups classify different infinte-dimensional unitary representations, and indeed, the rather unusual little group $\mathrm{E}(2)$ of the photon leads to the standard generators of $\mathfrak{so}(1,3)$ acting rather unusually upon a photon state. In particular, since $\mathrm{SU}(2)$ is not the little group, you cannot expect that rotations act in the same manner as for massive particles.
If you are interested in a detailed derivation of the structure of the little group and its allowed projective representations, take a look at e.g. the second half of chapter 2 of The quantum theory of fields, Vol. I by Weinberg.
There is a subtle issue with the relation $(1)$ for massless fields. As can be shown (cf. again Weinberg), a vector field constructed out of creation/annihilation operators for a massless particle can only obey a modified version
$$ \rho_\text{U}(\Lambda)\phi\rho_\text{U}(\Lambda)^\dagger = \rho_\text{fin}(\Lambda)\phi + \mathrm{d}\Omega\tag{2}$$
for a function of spacetime $\Omega$. Therefore, to make the relation $(1)$ hold on the quantum space of states, we must demand that $A\mapsto A + \mathrm{d}\Omega$ is a gauge symmetry of the theory, which must be quotiented out of the naive space of state upon which $(2)$ holds to obtain the actual space of physical states. Taking the quotient by identifying gauge-related states (and operators), $(2)$ becomes $(1)$ on the actual space of states (since $A+\mathrm{d}\Omega = A$ after taking the quotient). This shows that every massless vector field must be a gauge field in quantum field theory.