For a general rotation $R(t_1, t_2, t_3)$ where the $t_i$'s are the components of the rotation vector in the axis-angle representation. Is there closed formula for the derivative of $dR/dt_i$?
I only find derivative with respect to the magnitude of the angle, but not to the components of $t_i$
$$ \frac {dR(\theta)}{d\theta}=L_\theta R(\theta) $$ where $\theta=norm([t_1, t_2, t_3])$ is the rotation angle, and $L_\theta$ is the generator
Not straightforward. You can use the CBH formulae $$ e^{-A} e^{A+\delta A} = 1+ \int_0^1 e^{-sA}(\delta A) e^{sA} ds+ O[(\delta A)^2] $$ and $$ e^{-sA}(\delta A) e^{sA}= \delta A- s[A, \delta A]+ \frac {s^2}{2}[[A,[A, \delta A]]+\ldots $$ It's easier with Euler angles where (using the 2 dim spin representation for convenience) we have \begin{align} R&=& \exp\{-i\phi\sigma_3/2\}\exp\{-i\theta\sigma_2/2\}\exp\{-i\psi\sigma_3/2\}, \nonumber\\ &=& \left(\matrix{ e^{-i\phi/2} &0\cr 0 & e^{i\phi/2}}\right) \left(\matrix{ \cos \theta/2 &-\sin\theta/2\cr \sin\theta/2 &\cos\theta/2 }\right) \left(\matrix{ e^{-i\psi/2} &0\cr 0 & e^{i\psi/2}}\right), \nonumber\\ &=& \left(\matrix{e^{-i(\phi+\psi)/2} \cos\theta/2& -e^{i(\psi-\phi)/2} \sin\theta/2\cr e^{i(\phi-\psi)/2} \sin\theta/2& e^{+i(\psi+\phi)/2} \cos\theta/2}\right). \nonumber \end{align} You get $$ R^{-1}dR = -\frac i2 \sigma_i \,\Omega_{\rm L}^i, $$ where \begin{align} \Omega_{\rm L}^1 &=& \sin\psi\, d\theta -\sin\theta \cos\psi\, d\phi,\nonumber\\ \Omega_{\rm L}^2 &=& \cos\psi\, d\theta +\sin\theta \sin\psi\, d\phi,\nonumber\\ \Omega_{\rm L}^3 &=& d\psi +\cos\theta\, d\phi.\nonumber \end{align}
This answer does not depened on spin-1/2. Its the same for the 3d rotation matrices with the same Euler parameterization.
Given the usual generators $J_i$ for $i=1,2,3$ of the group $SO(3)$, a rotation matrix $R \in SO(3)$ can be written as $$ R = e^{ i \, t_k J_k} \qquad (R^{-1} = e^{ -i \, t_k J_k}) $$ where the repeated index $k$ is summed. Therefore (the limit for small $a$ is understood and $T = t_k J_k $ is an Hermitean matrix), $$ \frac{d}{d t_1}R = a^{-1} \big( e^{ i(t_k+a\delta_{k1}) J_k}-e^{ i t_k J_k} \big) = a^{-1} \big( e^{i T + i a J_1 }-e^{ i T} \big) \\ = a^{-1} \big( e^{ i T + a i J_1 }e^{ -i T} - 1 \big)R $$ Now, could try to use the BCH formula but would still give an infinite series: the "naive" approach would give $$ e^{ iT + ia J_1 }e^{ -i T} \approx e^{i T + ia J_1 - iT +[T + a J_1,T]/2} = e^{ a (i J_1 + [ J_1,T]/2)} \approx 1+ a (i J_1 + [ J_1,T]/2) $$ $$ \frac{d}{d t_1}R = i J_1 + [J_1,T]/2 = i J_1 + n_k [J_1,J_k ]/2 $$ You can easily calculate the last commutator with the usual commutation relation $[J_a,J_b]= i \epsilon_{abc}J_c$. Similarly for $t_2$ or $t_3$. However, this is NOT correct: many other commutators of order $a$ are missing in the truncated BCH formula used above!
Therefore, you may try the derivative of the exponential map, $$ \frac{d}{d t_1}R = \frac{d}{d t_1} e^{i T} = \int_0^1 e^{i (1-s) T} i\frac{d}{d t_1}T e^{i s T} ds= i \int_0^1 e^{i (1-s) T} J_1 e^{i s T} ds = \\ =i\int_0^1 J_1 ds+ i \int_0^1 e^{i (1-s) T} [J_1, e^{i s T}] ds \, =i J_1 + i \int_0^1 e^{i (1-s) T} [J_1, e^{i s T}] ds \, . $$ Again, we have found the correct first term $iJ_1$, but the second integral term is a series (the full series that you would find by using the full BCH formula).
