There are more quantum numbers to be conserve than charge and spin. There is also lepton number and baryon number and strangeness and charmness and color for strong interactions.
A muon hitting a positron can interact exchanging a photon and keep going, or a W and turn into a muon-neutrino and an electron-antineutrino through the week interaction .
Quarks are worse because they also have color, three extra quantum numbers plus charge number (1/3 or 2/3) plus baryon number(1/3).
Take the pion: pi+ is a linear combination of up anti-down and will eventually turn into a (muon+ anti-muneutrino) via the weak decay.
You may call it annihilation, as you may call all interactions where the incoming particles and the outgoing differ. The customary use of the term is for particle antiparticle annihilation.
It is more complicated than you think.