TL;DR: The point is that the experimentally obtained mean values $\bar\sigma_j$ are in general not equal to the theoretical expectation values $\langle \sigma_j\rangle_\rho$ and hence your estimate of $\rho$ is in general not equal to it; in particular, it can happen that it is not a density operator at all.
Answer: Let's say we identically prepare $3N$ qubits each in a state described by $\rho$ and on each such system we measure one of the observable corresponding to $\sigma_1,\sigma_2,\sigma_3$; suppose further that each of these observables is measured $N$ times. From these measurements we can extract the average values
$$\bar \sigma_j(N) = \frac{1}{N}\sum\limits_{n=1}^N \sigma_j(n) \quad , \tag{1}$$
where $\sigma_j(n)$ denotes the $n$-th outcome of the measurement of $\sigma_j$, for $j=1,2,3$. The relation to the expectation value, determined by $\rho$, is given by
$$\langle \sigma_j\rangle_\rho:=\mathrm{Tr}\rho \,\sigma_j = \lim\limits_{N\to\infty} \bar\sigma_j(N) \tag{2} \quad .$$
As you can see, for a finite $N$, which is inherent to every experiment, we can only estimate the expected value by the average, i.e. for sufficiently large $N$ we can hope that $\bar\sigma_j(N) \approx \langle \sigma_j\rangle_\rho$.
Now the important point is that the operator $\tilde\rho$, estimated from the average values, might fail to be a density matrix. It is easy to show that it is a density matrix if and only if
$$\sum\limits_{j=1}^3\bar\sigma_j(N)^2 \leq 1 \tag 3 \quad .$$
If, however, $\tilde \rho$ happens to be a density matrix but $\tilde \rho \neq \rho$, then there exists at least one observable corresponding to a hermitian operator $A$ such that $\mathrm{Tr}\rho A\neq \mathrm{Tr}\tilde \rho A$, cf. here. So the estimated density matrix $\tilde \rho$ yields wrong predictions. Whether or not this is experimentally verifiable or important depends on the exact situation, tho.
Finally, note that $\rho=\tilde\rho$ if and only if $\bar\sigma_j = \langle \sigma_j\rangle_\rho$, which holds in general only for $N\to \infty$, as can be seen by equation $(2)$.