If you apply no rules to the possible interactions then what you describe is possible. But the negative bound state mass can be avoided if we assume two things:
- A force field that has always positive energy.
- The definition of mass for one free particle includes the energy in the field around that particle. (And the starting value without that field cannot be chosen arbitrary negative, see below).
We describe the force by a force field $\bf F$, and the total energy in that field is:
$$\begin{align}
E_\text{field} =& \int d^3{\bf x}\ \lambda |{\bf F}|^2 \tag{1}
\end{align}$$
with $\lambda$ some positive constant. If the force is attractive this just means that we can lower the energy in the field by bringing the masses together. For instance if the particles have opposite charge, their fields will have opposite sign, and will partially cancel when they start overlapping if the particles are brought together.
So to compute $\Delta U$ we just have to compute the field energy for the initial and final field configuration:
$$\begin{align}
\Delta U =& \int d^3x\ \lambda |{\bf F}_\text{final}|^2 -\int d^3x\ \lambda |{\bf F}_\text{initial}|^2 \\[6pt]
\Rightarrow \Delta U =& \int d^3x\ \lambda |{\bf F}_\text{bound}|^2 -\int d^3x\ \lambda |{\bf F}_\text{1,free}|^2
-\int d^3x\ \lambda |{\bf F}_\text{2,free}|^2 \tag{2}
\end{align}$$
where the initial configuration just has two terms with the energy in the field ${\bf F}_\text{i,free}$ around each particle (since they initially are far apart and the fields don't overlap).
For one free particle we include the field energy in the mass:
$$\begin{align}
m_1 =&\ m_{1,\text{bare}} + \frac1{c^2} \int d^3x\ \lambda |{\bf F}_\text{1part}|^2 \tag{3}
\end{align}$$
Where the "bare mass" $m_{1,\text{bare}}$ is the mass a particle would have without its field. Now our bound state mass $M$ is:
$$\begin{align}
M =&\ m_1 + m_2 + \frac1{c^2}\Delta U =
m_{1,\text{bare}} + \frac1{c^2} \int d^3x\ \lambda |{\bf F}_\text{1,free}|^2
+ m_{2,\text{bare}} + \frac1{c^2} \int d^3x\ \lambda |{\bf F}_\text{2,free}|^2
\\[6pt]
& \qquad + \frac1{c^2} \Big( \int d^3x\ \lambda |{\bf F}_\text{bound}|^2 - \int d^3x\ \lambda |{\bf F}_\text{1,free}|^2
-\int d^3x\ \lambda |{\bf F}_\text{2,free}|^2 \Big)
\\[6pt]
=&\ m_{1,\text{bare}} +m_{2,\text{bare}} + \frac1{c^2} \int d^3x\ \lambda |{\bf F}_\text{bound}|^2 \\[6pt]
\end{align}$$
Clearly, if both particles have a positive bare mass then the bound sate $M$ will always be positive. If, on the other hand, you allow arbitrary negative values for $m_{1,\text{bare}}$ and $m_{1,\text{bare}}$ then you can always make $M$ end up negative, just by taking the combined bare masses more negative than what the energy in ${\bf F}_\text{bound}$ gives you.
It should be noted that with a negative bare mass we can also make the mass of one free particle negative and we can create other unphysical behavior. It is also related to the [classical electron radius] problem and the unphysical runaway solutions of the Abraham-Lorentz force, for which we also need the radius of the charge distribution [Jackson. p. 787].
So it is definitely safer to keep the bare masses positive.
