I) In general, for a given choice of boundary conditions, it is important to adjust the action with compatible boundary terms/total divergence terms in order to ensure the existence of the variational/functional derivative. As OP observes, the problem is (when deriving the Euler-Lagrange expression) that the usual integration by parts argument fails if the boundary conditions (BCs) and the boundary terms (BTs) are not compatible.
II) Concretely, for the mixed BCs
$$\tag{1} q(t_i) ~=~ q_i \qquad\text{and}\qquad p(t_f) ~=~ p_f, $$
which OP considers, we need to prepare the standard Hamiltonian action
$$\tag{2} S_0[p,q] ~=~ \int_{t_i}^{t_f} \! dt ~ \left\{ p\dot{q} -H \right\} $$
with a total divergence term $-\frac{d}{dt}(p_f q)$. The new action becomes
$$\tag{3} S[p,q] ~=~ \int_{t_i}^{t_f} \! dt ~ \left\{ (p-p_f)\dot{q} -H \right\} ,$$
or what amounts to the same,
$$\tag{4} S[p,q] ~=~ \int_{t_i}^{t_f} \! dt ~ \left\{ \dot{p}(q_i-q) -H \right\}. $$
It is straightforward to use the BCs (1) to show that the actions (3) and (4) are equal.
III) Now when we vary the action (3)
$$ \tag{5} \delta S[p,q] ~=~ \left[ (p-p_f)\delta q\right]_{t_i}^{t_f}+
\int_{t_i}^{t_f} \! dt ~ \left\{ (\dot{q} -\frac{\partial H}{\partial p})\delta p -(\dot{p}+\frac{\partial H}{\partial q})\delta q\right\}, $$
the BCs (1) cancels the total derivative term
$$ \tag{6} \left[ (p-p_f)\delta q\right]_{t_i}^{t_f}~\stackrel{(1)}{=}~0, $$
so that the variation (5) only contains bulk terms. The corresponding Euler-Lagrange equations become Hamilton's equations.
IV) The above example can be generalized to other BCs. We leave it to the reader to work out the compatible BTs.