Reading about $SU(5)$ unification, texts says that they use the normalization factor $\sqrt{3/5}$ for weak hypercharges in order to embed SM into a $SU(5)$ group. This implies a new $U(1)_Y$ coupling constant given by
$$ \alpha_1 = \frac{5}{3}\alpha_Y. $$
$\alpha_Y$ is the one you obtain without altered normalization.
Where did this extra factor come out? Why do you need it to unification?
I illustrate @Qmechanic's complete answer for the 5 of SU(5), as a mnemonic, if you are caught in a bus trip out of internet reach.
Unification means that the 8 generators of SU(3), the 3 of SU(2) and the hypercharge are all embedded in the 24 of SU(5), and represented by hermitean traceless matrices, normalized uniformly, so, in the fundamental, $\operatorname {Tr} T_a T_b= 2\delta_{ab} $, a peculiar historical vestige of the Pauli matrices sitting in the Gell-Mann ones, which never fails to puzzle mathematicians.
Consider the covariant derivative of the SM fundamental fermions, e.g. quarks, $$ D_\mu -\partial_\mu = i g_{_{QCD}} \frac{\lambda}{2} \cdot G_\mu + i g \frac{\vec \tau}{2} \cdot \vec W_\mu +i g ' \frac{ Y}{2} B_\mu . $$ Since the Hypercharge U(1) commutes with everything, its denominator 2 is immaterial but superfluous, and could be readily incorporated into its definition, $Y'\equiv Y/2 $, as most modern texts do, but not Cheng & Li — which, I understand, you are using. It is just there, here, to expedite trigonometry w.r.t. the SU(2) in the Weinberg angle. So, here, the hypercharge is twice the average electric charge of the weak multiplet involved, $Y=2(Q-T_3)$. (You could see why the modernists make faces at this sort of thing.)
Consider the 5 of SU(5), however, $$ \psi = (d^{red},d^{green},d^{blue}, e^+, -\nu_e^c)_R, $$ comprised out of the colored down quark SU(2) singlet, and the conjugate doublet of leptons in this generation, so the R positron and its weak isopartner R neutrino. Since you have $$ Y \psi= \operatorname {diag} (-2/3,-2/3,-2/3,1,1 ) \psi ~~~~~\Longrightarrow \\ Q \psi= \operatorname {diag} (-1/3,-1/3,-1/3,1,0 ) \psi, $$ it is evident Y is not normalized like, for instance, the color matrices above.
Both Y and Q are traceless, of course.
So, normalize Y so that $$ Y'' \equiv -\sqrt{3/5}\operatorname {diag} (-2/3,-2/3,-2/3,1,1) =-\sqrt{3/5} Y, $$ and hence Tr$Y'' ~^2= 2$, just like the color or SU(2) generators.
It is manifest from the covariant derivative above that $$ g' Y = g_1 Y'', \Longrightarrow (g')^2= \frac {3}{5} g_1^2, $$ leading to the celebrated SU(5) "initial constraint"; that is, at the UV point where $g_1=g_2\equiv g$, the weak mixing angle is constrained to $$ \sin ^2 \theta_W= \frac{g'^2}{g^2+g'^2}= \frac{3/5}{1+3/5}=3/8, $$ which put that gleam in the eyes of its inceptors in the very first place!
In YM theory a coupling constant $g\sim\sqrt{\alpha}$ sits next to the Lie algebra generator $T_a$, so the normalization of the coupling constant $g$ is directly correlated to the normalization of the Lie algebra generator $T_a$. The $3/5$ factor comes from the non-trivially embedding of the weak hypercharge $u(1)$ generator into the $su(5)$ GUT Lie algebra, cf. e.g. Ref. 1 and this Phys.SE post.
References:
