Set up a state $|k_{0},\lambda\rangle$ for a photon of fiducial momentum $k_{0}$ moving along the z axis with helicity $\lambda$.
Now, instead of working with states of sharp momentum and helicity, we can consider states of sharp total angular momentum by using the projection operator onto the states of a group irrep.
Suppose we have a group $G$. The group has a unitary representaion $\hat{U}(g)$ which acts on the states of a Hilbert space. The Hilbert space can be decomposed into irreps with states $|a,i,u\rangle$ where $a$ labels the irreps, $i$ labels the states in irrep $a$ and $u$ labels copies of irrep $a$. The projection operator onto the states of the irreps is,
\begin{equation}
\hat{P}_{aij}=\frac{n_{a}}{N}\sum_{g\in G}(U^{(a)}(g))^{j}_{\ \ i}\hat{U}(g^{-1})
=\sum_{u}|a,i,u\rangle\langle a,j,u|
\end{equation}
Here, $n_{a}$ is the dimension of irrep $a$, $N$ is the size of the group $G$, $(U^{(a)}(g))^{j}_{\ \ i}$ is the unitary matrix rep of the group on the irrep $a$. The definition of the projection operator used here is from equation (1) in section 3 of chapter 7 (page 177) of "Theory of Group Representations and Applications", Barut and Raczka, World Scientific, 1986.
In order to get a photon with sharp total angular momentum $j$, the group is $G=SO(3)$ and the projection operator becomes,
\begin{equation}
\hat{P}_{jm_{1}m_{2}}=\frac{2j+1}{N}\sum_{g\in SO(3)} (U^{(j)}(g))^{m_{2}}_{\ \ m_{1}}\hat{U}(g^{-1})
\end{equation}
It helps if the operator is $\hat{U}(g)$ instead of $\hat{U}(g^{-1})$ so change variable $g\rightarrow g^{-1}$.
\begin{equation}
\hat{P}_{jm_{1}m_{2}}=\frac{2j+1}{N}\sum_{g\in SO(3)} ((U^{(j)}(g))^{m_{1}}_{\ \ m_{2}})^{*}\hat{U}(g)
\end{equation}
The projection operator is defined for a finite group, but SO(3) is compact so,
\begin{equation}
\frac{1}{N}\sum_{g\in SO(3)}\rightarrow \int_{g\in SO(3)}d\mu(g)
\end{equation}
where the Haar measure is,
\begin{equation}
d\mu(g)=\frac{\sin{\beta}d\alpha d\beta d\gamma}{8\pi^{2}}
\end{equation}
and $\alpha,\beta,\gamma$ are Euler angles (rotate by $\alpha$ about z, then $\beta$ about resultant y, then $\gamma$ about resultant z). The ranges are $\alpha \in [0,2\pi]$, $\beta\in [0,\pi]$, $\gamma\in [0,2\pi]$.
The projection operator onto a state of sharp total angular momentum $j$ is now,
\begin{equation}
\hat{P}_{jm_{1}m_{2}}=\frac{2j+1}{8\pi^{2}}\int \sin{\beta}\ d\alpha\ d\beta\ d\gamma \{(U^{(j)}(g))^{m_{1}}_{\ \ m_{2}}\}^{*}\hat{U}(g)
\end{equation}
The matrices $(U^{(j)}(g))^{m_{1}}_{\ \ m_{2}}$ are Wigner's D-functions.
Now let's apply the projection operator to the photon state $|k_{0},\lambda\rangle$. A photon of sharp total angular momentum $j$ is,
\begin{equation}
\hat{P}_{jm_{1}m_{2}}|k_{0},\lambda\rangle=\frac{2j+1}{8\pi^{2}}\int \sin{\beta}\ d\alpha\ d\beta\ d\gamma \{(U^{(j)}(g))^{m_{1}}_{\ \ m_{2}}\}^{*}\hat{U}(g)|k_{0}\lambda\rangle
\end{equation}
Now, a general rotation $g\in SO(3)$ is the matrix $R_{z}(\alpha)R_{y}(\beta)R_{z}(\gamma)$. Hence,
\begin{equation}
\hat{U}(g)|k_{0},\lambda\rangle=\hat{U}(R_{z}(\alpha)R_{y}(\beta)R_{z}(\gamma))|k_{0},\lambda\rangle=\hat{U}(\alpha,\beta,0)\hat{U}(0,0,\gamma)|k_{0},\lambda\rangle=\hat{U}(\alpha,\beta,0)e^{-i\gamma\lambda}|k_{0},\lambda\rangle
\end{equation}
Furthermore, Wigner's D-functions are,
\begin{equation}
(U^{(j)}(g))^{m_{1}}_{\ \ m_{2}}=\langle j m_{1}|\hat{U}(\alpha,\beta,\gamma)|j m_{2}\rangle=e^{-im_{1}\alpha}\langle j m_{1}|\hat{U}(\theta=\beta/2)|jm_{2}\rangle e^{-im_{2}\gamma}
\end{equation}
where $\langle j m_{1}|\hat{U}(\theta=\beta/2)|jm_{2}\rangle $ are the reduced d-functions. We can substitute the last two equations in the integral for the state of sharp total angular momentum and carry out the integral over the Euler angle $\gamma$. The result is,
\begin{equation}
\hat{P}_{jm_{1}\lambda}|k_{0},\lambda\rangle=\frac{2j+1}{4\pi}\int \sin{\beta}\ d\alpha\ d\beta\ \{(U^{(j)}(\alpha,\beta,0))^{m_{1}}_{\ \ \lambda}\}^{*}\hat{U}(\alpha,\beta,0)|k_{0},\lambda\rangle
\end{equation}
The state $\hat{U}(\alpha,\beta,0)|k_{0},\lambda\rangle$ is just a photon of momentum $k$ travelling in the direction given by the polar angles $\alpha,\beta$. In other words, a plane wave state $|k,\lambda\rangle$. So, a photon of sharp total angular momentum $j$ is,
\begin{equation}
\hat{P}_{jm_{1}\lambda}|k_{0},\lambda\rangle=\frac{2j+1}{4\pi}\int \sin{\beta}\ d\alpha\ d\beta\ \{(U^{(j)}(\alpha,\beta,0))^{m_{1}}_{\ \ \lambda}\}^{*}|k,\lambda\rangle
\end{equation}
The momentum $k=R(\alpha,\beta,0)k_{0}$ and the element of solid angle is $d\Omega=\sin{\beta}\ d\alpha\ d\beta$ . Finally, a photon of sharp total angular momentum $j$ is the state,
\begin{equation}
|jmk\lambda\rangle=\hat{P}_{jm\lambda}|k_{0},\lambda\rangle=
\frac{2j+1}{4\pi}\int d\Omega\ \{(U^{(j)}(\alpha,\beta,0))^{m}_{\ \ \lambda}\}^{*}|k,\lambda\rangle
\end{equation}
This formula agrees with (8.7-2) on page 147 of Wu-Ki Tung "Group Theory in Physics" and (28.35) on page 218 of Werle, "Relativistic Theory of Reactions", North Holland, 1966.
