Update: I removed the links to the Sage code of the complicated explicit examples, because very much easier examples exist. See below for an example, and this preprint for more details concerning this answer and the computation of explicit examples.
Answer: The answer is no. In the following I'll describe how to find two rational functions $f,g\in\mathbb C(X)$, both of odd prime degree $\ell$, such that the following holds:
- $f(g(X))\in\mathbb R(X)$.
- $g(\mathbb R\cup\{\infty\})$ is a smooth closed Jordan curve in the complex plane.
Let $E$ be an elliptic curve, defined over the reals. For $p\in E(\mathbb C)$ we let $\bar p$ be the complex conjugate of $p$. Choose $E$ such that the following holds:
- There is a point $w\in E(\mathbb R)$ with no $y\in E(\mathbb R)$ with $w=2y$.
- There is a point $z\in E(\mathbb C)$ of order $\ell$ with $\bar z\notin\langle z\rangle$. (Such a point always exists.)
Then $C=\langle z\rangle$ is a subgroup of order $\ell$ of $E$, and $E'=E/C$ is an elliptic curve over $\mathbb C$. Let $\phi:E\to E'$ be the associated isogeny, and $\phi':E'\to E$ be the dual isogeny. Then $\phi'\circ\phi:E\to E$ is the multiplication by $\ell$ map.
Let $\beta$ be the automorphism of order $2$ sending $p\in E(\mathbb C)$ to $w-p$. Similarly, define the involutory automorphisms $\beta'$ of $E'$ and $\beta''$ of $E$ by $\beta'(p')=\phi(w)-p'$ and $\beta''(p)=\ell w-p=\phi'(\phi(w))-p$.
Let $\psi$ be the degree $2$ covering map $E\to E/\langle\beta\rangle=P^1(\mathbb C)$, and define likewise $\psi':E'\to E'/\langle\beta'\rangle=P^1(\mathbb C)$ and $\psi'':E\to P^1(\mathbb C)$.
Let $f(X)$ and $g(X)$ be the rational functions defined implicitly by $\psi'\circ\phi=g\circ\psi$ and $\psi''\circ\phi'=f\circ\psi'$. Note that $\psi$, and $\psi''$ are defined over $\mathbb R$, while $\psi'$ is not.
As the multiplication by $\ell$ map $\phi'\circ\phi$ is defined over the reals, we obtain $f(g(X))\in\mathbb R(X)$.
We next claim that $g(X)$ is injective on $\mathbb R$. Suppose there are distinct real $u,v$ with $g(u)=g(v)$. Pick $p,q\in E(\mathbb C)$ with $\psi(p)=u$, $\psi(q)=v$. Then $\psi'(\phi(p))=g(u)=g(v)=\psi'(\phi(q))$, so $\phi(p)=\phi(q)$ or $\phi(p)=\phi(w)-\phi(q)$. Upon possibly replacing $q$ with $w-q$ we may assume $\phi(p)=\phi(q)$, so $p-q\in C$.
Next we study the effect of complex conjugation. As $\psi$ is defined over the reals, and $\psi(p)=u$ is real, we have $\psi(\bar p)=\psi(p)$, so $\bar p=p$ or $\bar p=w-p$. Likewise, $\bar q=q$ or $\bar q=w-q$. Recall that $p-q\in C$, and $\bar C\cap C=\{0\}$ by condition 2. So we can't have $(\bar p,\bar q)=(p,q)$, nor $(\bar p,\bar q)=(w-p,w-q)$.
Thus without loss of generality $\bar p=p$, $\bar q=w-q$. As $p-q$ and $\bar p-\bar q=p-w+q$ have order $\ell$, we see that $2p-w=r$ with $\ell r=0$ and $r\in E(\mathbb R)$. So $w=2(p+\frac{\ell-1}{2}r)$, contrary to condition 1. Furthermore, the function $g(X)$ behaves well at infinity by this geometric interpretation.
An explicit example, with $\omega$ a primitive third root of unity, is
\begin{align*}
f(X) &= \frac{X^3 - 6(\omega + 1)X}{3X^2 + 1}\\
g(X) &= \frac{2X^3 + (\omega + 1)X}{X^2 - \omega}\\
f(g(X)) &= \frac{8X^9 - 24X^5 - 13X^3 - 6X}{12X^8 + 13X^6 + 12X^4
- 1}\in\mathbb Q(X).
\end{align*}
First note that $g(\mathbb R\cup\{\infty\})$ is not contained in a circle, for instance because the points $g(0)=0$, $g(\infty)=\infty$, $g(1/2)=(1+2\omega)/3$, and $g(1)=(5+4\omega)/3$ do not lie on a circle.
Secondly, $g(\mathbb R\cup\{\infty\})$ is a Jordan curve, because $g$ is injective on $\mathbb R$: Suppose that $g(x)=g(x+\delta)$ for real $x,\delta$. A short calculation yields $\delta(8\delta^4 + 14\delta^2 + 49)=0$, so $\delta=0$.
Speaking of "maps into" I do not insist that the map is a covering; let it be any map. Of course, an example with a covering will be even more appreciated:-)
– Alexandre Eremenko Aug 12 '12 at 08:40