Whenever you are applying Gauss' law on any arbitrary Gaussian surface i.e.-
$$\displaystyle \oint \textbf{E} \cdot \textbf{dS}=\dfrac{q_i}{\epsilon_o}$$
Here, the electric field on the gaussian surface is not just the electric field due to the charges inside the surface, it is also the electric field due to the charges outside. Gauss Law by itself, does not really care much about the field, all it cares about is the flux.
So coming to your question-
can I, hypothetically, "remove" those charges that I am not interested in and then apply Gauss Law to find the electric field due to the other charges?
Yes you can, but first just make sure that there are no charges outside the surface, else Gauss law is of no use.
Moreover, to find the electric field due to only some charges inside the cavity, you have to simply use the principle of superposition. Suppose you want to remove a set of charges $\{q_i\}$. What you have to do is, simply, hypothetically add the set of charges $\{-q_i\}$ to those exact places. This way, you will get a net zero contribution of electric field due to those charges. But do realize the fact that this happens only because electrostatics is linear in nature, the explanation of which we will save for another day.
Of course this means the same as actually, physically removing those charges away from that place and keeping them at a large distance from the remaining charges. Now Gauss law will say-
$$\displaystyle \oint \textbf{E}_{new} \cdot \textbf{dS}=\dfrac{q_{new}}{\epsilon_o}$$
in which the field is only due to the remaining charges. But it's still generally not possible to figure out the field at a point due to the charges left inside. Why? Because Gauss law cares only about flux, not about field. If your new configuration displays some symmetry that you can exploit, well and good, go ahead and find the field you desired. But if not, whatever we have done up until this point is pretty much useless.