On page 517, the author tells you that the calculation is carried out in Feynman-'t Hooft gauge. The problem with your last equation is that the propagator of the photon is singular before imposing gauge fixing conditions. i.e it has zero modes, and the path-integral is not well-defined.
On page 513, there's another hint for the calculation in the bosonic sector. Equation (16.29) says $$\langle A^{a}_{\mu}(x)A^{b}_{\nu}(y)\rangle=\int\frac{d^{d}k}{(2\pi)^{d}}\frac{-i}{k^{2}+i\epsilon}\left(g_{\mu\nu}-(1-\xi)\frac{k_{\mu}k_{\nu}}{k^{2}}\right)\delta^{ab}e^{-ik\cdot(x-y)}.$$
The Feynman-'t Hooft gauge is when $\xi=1$. i.e $$\langle A^{a}_{\mu}(x)A^{b}_{\nu}(y)\rangle=\int\frac{d^{d}k}{(2\pi)^{d}}\frac{-i}{k^{2}+i\epsilon}g_{\mu\nu}\delta^{ab}e^{-ik\cdot(x-y)}.$$
This time, the photon propagator is non-singular and has an inverse. You only need to calculate the following functional integral of the partition function in the bosonic sector:
$$\int\mathcal{D}[A]\exp\left(-\frac{1}{2}\int d^{d}x A_{\mu}^{a}(x)g_{\mu\nu}^{E}\delta^{ab}\Box A^{b}_{\nu}(x)\right),$$
where $g^{E}_{\mu\nu}$ is the Euclidean metric after a wick rotation. What Peskin and Schroeder didn't say in their book is that equation (16.43) is actually in Euclidean signature. Later, they mentioned in the next paragraph that "this physical effect was illustrated, using the language of Section 9.4, in Problem 9.2."!!! There, in Problem 9.2, readers are asked to perform the same path-integral in Euclidean signature. Please check page 312, 313, and 314.
Now, taking the coupling $g=0$ is essentially going back to the Abelian case, i.e. QED where photons and the ghost fermions decouple, and you can ignore the Killing form $\delta^{ab}$. Then, in such a case, turning to the Euclidean signature, the full partition function should be written as $$\mathcal{Z}=\int\mathcal{D}[A_{1}^{E}]\ldots\mathcal{D}[A_{d}^{E}]\int\mathcal{D}[c]\mathcal{D}[\bar{c}]\exp\left(-\frac{1}{2}\int d^{d-1}xdx^{d}(A_{\mu}^{E}\Delta\delta^{\mu\nu}A_{\nu}^{E}-\bar{c}\Delta c)\right),$$
where $d^{d-1}xdx^{d}=dx^{1}\wedge\cdots\wedge dx^{d-1}\wedge dx^{d}$ is the Euclidean volume form, and $x^{0}_{E}=x^{d}$, and $\Delta$ is the Laplacian operator (the Euclidean counterpart of the d'Alembert operator in Lorentzian spacetime) in the Eulidean space, and $\left\{\delta_{\mu\nu}\right\}=\mathrm{diag}(1,\cdots,1)$.
Then, using the Gaussian integral formula and the Berezin integral formula you should find $$\mathcal{Z}=\left(\mathrm{Det}\left(-\Delta\right)\right)^{-d/2}\mathrm{Det}(-\Delta).$$