Problem: Given Newton's second law
$$\begin{align} m\ddot{q}^j~=~&-\beta\dot{q}^j-\frac{\partial V(q,t)}{\partial q^j}, \cr j~\in~&\{1,\ldots, n\}, \end{align}\tag{1} $$
for a non-relativistic point particle in $n$ dimensions, subjected to a friction force, and also subjected to various forces that have a total potential $V(q,t)$, which may depend explicitly on time.
I) Conventional approach: There is a non-variational formulation of Lagrange equations
$$\begin{align} \frac{d}{dt} \left(\frac{\partial L}{\partial \dot{q}^j}\right)-\frac{\partial L}{\partial q^j}~=~&Q_j, \cr j~\in~&\{1,\ldots, n\},\end{align}\tag{2} $$
where $Q_j$ are the generalized forces that do not have generalized potentials.
In our case (1), the Lagrangian in eq. (2) is $L=T-V$, with $T=\frac{1}{2}m\dot{q}^2$; and the force
$$ Q_j~=~-\beta\dot{q}^j\tag{3} $$
is the friction force. It is shown in e.g. this Phys.SE post that the friction force (3) does not have a potential. As OP mentions, one may introduce the Rayleigh dissipative function, but this is not a genuine potential.
Conventionally, we additionally demand that the Lagrangian is of the form $L=T-U$, where $T=\frac{1}{2}m\dot{q}^2$ is related to the LHS of EOMs (1) (i.e. the kinematic side), while the potential $U$ is related to the RHS of EOMs (1) (i.e. the dynamical side).
With these additional requirements, the EOM (1) does not have a variational formulation of Lagrange equations
$$\begin{align} \frac{d}{dt} \left(\frac{\partial L}{\partial \dot{q}^j}\right)-\frac{\partial L}{\partial q^j}~=~&0,\cr j~\in~&\{1,\ldots, n\},\end{align}\tag{4} $$
i.e. Euler-Lagrange equations. The Legendre transformation to the Hamiltonian formulation is traditionally only defined for a variational formulation (4). So there is no conventional Hamiltonian formulation of the EOM (1).
II) Unconventional approaches:
Trick with exponential factor$^1$: Define for later convenience the function
$$ e(t)~:=~\exp(\frac{\beta t}{m}). \tag{5}$$
A possible variational formulation (4) of Lagrange equations is then given by the Lagrangian
$$\begin{align} L(q,\dot{q},t)~:=~&e(t)L_0(q,\dot{q},t), \cr L_0(q,\dot{q},t)~:=~&\frac{m}{2}\dot{q}^2-V(q,t).\end{align}\tag{6}$$
The corresponding Hamiltonian is
$$ H(q,p,t)~:=~\frac{p^2}{2me(t)}+e(t)V(q,t).\tag{7}$$
One caveat is that the Hamiltonian (7) does not represent the traditional notion of total energy. Another caveat is that this unconventional approach cannot be generalized to the case where two coupled sectors of the theory require different factors (5), e.g. where each coordinate $q^j$ has individual friction-over-mass-ratios $\frac{\beta_j}{m_j}$, $j\in\{1, \ldots, n\}$. For this unconventional approach to work, it is crucial that the factor (5) is an overall common multiplicative factor for the Lagrangian (6). This is an unnatural requirement from a physics perspective.
Imposing EOMs via Lagrange multipliers $\lambda^j$: A variational principle for the EOMs (1) is
$$\begin{align}L ~=~& m\sum_{j=1}^n\dot{q}^j\dot{\lambda}^j\cr
&-\sum_{j=1}^n\left(\beta\dot{q}^j+\frac{\partial V(q,t)}{\partial q^j}\right)\lambda^j.\end{align}\tag{8}$$
(Here we have for convenience "integrated the kinetic term by parts" to avoid higher time derivatives.)
Classical Schwinger/Keldysh "in-in" formalism: The variables are doubled up. See e.g. eq. (20) in C.R. Galley, arXiv:1210.2745. Ignoring boundary terms$^2$ the Lagrangian reads
$$\begin{align}
\widetilde{L}(q_{\pm},\dot{q}_{\pm},t)
~=~&\left. L(q_1,\dot{q}_1,t)\right|_{q_1=q_+ + q_-/2}\cr
~-~&\left. L(q_2,\dot{q}_2,t)\right|_{q_2=q_+ - q_-/2}\cr
~+~&Q_j(q_+,\dot{q}_+,t)q^j_-\end{align}\tag{9}. $$
The initial conditions
$$\left\{\begin{array}{rcl} q^j_+(t_i)&=&q^j_i,\cr\dot{q}^j_+(t_i)&=&\dot{q}^j_i\end{array}\right.\tag{10} $$
implement the system's underlying initial values.
The final conditions
$$\begin{align}\left\{\begin{array}{rcl} q^j_-(t_f)&=&0\cr \dot{q}^j_-(t_f)&=&0 \end{array}\right. &
\cr\cr\qquad\Downarrow&\qquad\cr\cr
\left.\frac{\partial \widetilde{L}}{\partial \dot{q}^j_+}\right|_{t=t_f}~=~&0 \end{align}\tag{11} $$
implement the physical limit solution $q_-^j= 0$.
The doubling trick (9) is often effectively the same as introducing Lagrange multipliers (8).
Gurtin-Tonti bi-local method: See e.g. this Phys.SE post.
--
$^1$ Hat tip: Valter Moretti.
$^2$ The variational problem (9)+(10)+(11) needs an appropriate initial term, which might not always exist! In particular, since we already imposed $4n$ boundary conditions (10)+(11), it would be too much to also impose the initial condition
$$ q^j_-(t_i)~=~0. \qquad (\leftarrow\text{Wrong!})$$
Example: If $L=\frac{1}{2}m\dot{q}^2$, then $\widetilde{L}=m\dot{q}_+\dot{q}_-$, and one should add an initial term $m\dot{q}_+(t_i)q_-(t_i)$ to the action $\widetilde{S}$.
