The key to this question is: A tensor, what is it that is not a vector nor a scalar (but, we care about it)? What it is, is in a object that transforms as a representation of $SO(3)$ (the group of rotations in $\mathbb{R}^3$... so I'm talking 3-vectors. 4-vectors are similar).
The reason we use scalars, vectors, and tensors is because they are objects that respect certain symmetries of space in a mathematically precise way. They can be grouped into sub-sets (irreducible subspaces) that do not inter-mix under rotations. Moreover, they can put into a basis that are eigenvalues of rotations: they transform into themselves-scaled by a factor. That is a very special property.
At any given order $l\in(0,1,2,\ldots)$, there are $2l+1$ basis tensors, $T^{(l,m)}$, that are eigentensors of rotations($R_{\theta}$), about an arbitrary axis (chosen to be $z$):
$$ R_{\theta}(T^{(l,m)})= e^{im\theta}T^{(l,m)}$$
where $m\in(-l,-l+1,\ldots,l-1,l)$.
Scalars are the trivial representation ($m=0 \implies e^{im\theta}=1$). They are unchanged by rotations. Physics examples of scalars are pressure, density, energy density. (There area also pseudo scalars like $\vec E \cdot \vec B$, which are rotationally invariant, but change sign under reflection). Since there is only one basis scalar, this representation or anything that transforms like it is called a "singlet", and is written as ${\bf 1}$.
Vectors in the fundamental representation are:
$${\bf \hat{e}}^{\pm} = \mp({\bf \hat{x}} \pm i {\bf \hat{y}})/\sqrt{2}$$
$${\bf \hat{e}}^0={\bf \hat{z}}$$
You can verify that they are closed under rotations, and they are eigenvectors of $z$-rotations. This representation is labeled ${\bf 3}$.
At this point, if you encounter a physical object with different transformation properties, scalars and vectors will be insufficient to described it. The 1st step is to create a tensor product space, whose basis are 9 dyads: $({\bf\hat{x}}{\bf\hat{x}},{\bf\hat{x}}{\bf\hat{y}}, \cdots, {\bf\hat{z}}{\bf\hat{z}})$. Alternatively, one could take dyads of the various ${\bf\hat{e}}^m$, such as:
$$ {\bf\hat{e}}^{(2,\pm 2)}\equiv {\bf\hat{e}}^{\pm}{\bf\hat{e}}^{\pm}
= {\bf\hat{x}}{\bf\hat{x}}-{\bf\hat{y}}{\bf\hat{y}}+i
({\bf\hat{x}}{\bf\hat{y}}+{\bf\hat{y}}{\bf\hat{x}})$$
which are also eigentensors of $z$-rotations (with eigenvalue $e^{\pm 2i\theta}$). The trouble is that, in general, the tensor product of two (or more) eigenvectors is not an eigentensor. Nevertheless, that can be solved with Clebsch-Gordan coefficients, giving:
$$ {\bf\hat{e}}^{(2,\pm 1)}\equiv ({\bf\hat{e}}^{\pm}{\bf\hat{e}}^{0}+{\bf\hat{e}}^{0}{\bf\hat{e}}^{\pm})/\sqrt 2
= \frac 1 2[\mp({\bf\hat{x}}{\bf\hat{z}}+{\bf\hat{z}}{\bf\hat{x}})
-i({\bf\hat{y}}{\bf\hat{z}}+{\bf\hat{z}}{\bf\hat{y}})
]$$
$$ {\bf\hat{e}}^{(2,0)}\equiv
({\bf\hat{e}}^{+}{\bf\hat{e}}^{-}+{\bf\hat{e}}^{-}{\bf\hat{e}}^{+}
+2{\bf\hat{e}}^{0}{\bf\hat{e}}^{0})/\sqrt 6
= (-{\bf\hat{x}}{\bf\hat{x}}-{\bf\hat{y}}{\bf\hat{y}}+2{\bf\hat{z}}{\bf\hat{z}})/\sqrt 6$$
The ${\bf \hat{e}}^{(2,m)}$ are the 5 basis for pure rank 2 tensors. Under an arbitrary rotation, they transform among themselves.
You've probably noticed that $5<9$...we're missing some degrees-of-freedom. This is solved by representation theory, via:
$${\bf 3}\otimes{\bf 3}={\bf 5}\oplus{\bf 3}\oplus{\bf 1}$$
which says that a tensor product of triplets can be decomposed into a tensor sum of the quintet, another triplet, and another singlet.
The singlet is the isotropic part of a rank-2 tensor. It is invariant under rotations:
$$ {\bf\hat{e}}^{(0,0)}\equiv
({\bf\hat{e}}^{+}{\bf\hat{e}}^{-}+{\bf\hat{e}}^{-}{\bf\hat{e}}^{+})/\sqrt 3
-{\bf\hat{e}}^{0}{\bf\hat{e}}^{0}
\propto ({\bf\hat{x}}{\bf\hat{x}}+{\bf\hat{y}}{\bf\hat{y}}+{\bf\hat{z}}{\bf\hat{z}})$$
It is proportional to:
$$ \delta_{ij}=\left(\begin{array}{ccc} 1 & 0 & 0\\
0 &1&0\\ 0&0&1 \end{array}\right)$$
(note that $ \delta_{ij}$ is not a matrix, even though I was able to represent
it as one an a Cartesian basis). As part of the Cauchy stress-tensor, it represents the sum of pressure and velocity divergence. The remaining part it is called the deviator:
$$s_{ij}=\sigma_{ij}- \frac 1 2 {\rm Tr}(\sigma)\delta_{ij}$$
which has the same 5 independent components as ${\bf{\hat e}}^{(2,m)}$ , which in a fluid can be associated with quadrupole shape changes of a fiducial volume. Basically, these are the only allowed geometric shapes for a rank-2 process, just as vector can only be associated with 1 direction.
The remaining triplet are the antisymmetric part of the tensor:
$$ {\bf\hat{e}}^{(1,\pm 1)}\equiv ({\bf\hat{e}}^{\pm}{\bf\hat{e}}^{0}-{\bf\hat{e}}^{0}{\bf\hat{e}}^{\pm})/\sqrt 2
= \mp\frac 1 2[({\bf\hat{y}}{\bf\hat{z}}-{\bf\hat{z}}{\bf\hat{y}})
\pm i({\bf\hat{z}}{\bf\hat{z}}-{\bf\hat{x}}{\bf\hat{z}})
]$$
$$ {\bf\hat{e}}^{(1,0)}\equiv ({\bf\hat{x}}{\bf\hat{y}}-{\bf\hat{y}}{\bf\hat{x}})/\sqrt 2$$
The last formula is equivalent to the Cartesian form: it is the cross-product, and describes axial-vector objects like torque, angular momentum, magnetic field, etc.
At higher ranks, the algebra becomes more tedious (c.f., combing three spin-one particles in quantum mechanics). You will find that the 27-components break down as follows:
$${\bf 3}\otimes{\bf 3}\otimes{\bf 3}={\bf 7}\oplus
{\bf 5}\oplus{\bf 5}\oplus{\bf 3}\oplus{\bf 3}\oplus{\bf 3}\oplus{\bf 1}$$
For example, you can verify the following rank-3 tensors (part of ${\bf 7}$) are eigentensors of $z$-rotations:
$${\bf\hat{e}}^{(3,0)}\propto\left(\begin{array}{ccc} 0 & 0 & -1\\
0 &0&0\\ -1&0&0 \end{array}\right)
\left(\begin{array}{ccc} 0 & 0 & 0\\
0 &0& -1\\ 0&-1&0 \end{array}\right)
\left(\begin{array}{ccc} -1 & 0 & 0\\
0 &-1&0\\ 0&0&2 \end{array}\right)
$$
$${\bf \hat{e}}^{(3, -3)}\propto\left(\begin{array}{ccc} 1 & i & 0\\
i &-1&0\\ 0&0&0 \end{array}\right)
\left(\begin{array}{ccc} i & -1 & 0\\
-1 &-i& 0\\ 0&0&0 \end{array}\right)
\left(\begin{array}{ccc} 0 & 0 & 0\\
0 &0&0\\ 0&0&0 \end{array}\right)
$$
At rank-4, one encounters the elasticity tensor relating stress and strain (both rank-2 tensors):
$$ \epsilon_{ij}=c_{ijkl}\sigma_{ij}$$
which is the most general linear form of Hooke's Law. Because of the symmetries of stress and strain, only 21 of the 81 entries in $c_{ijkl}$ are independent. A break down into spherical components would be involved, but it would further elucidate the geometric nature of tensors as the irreducible objects that are representations of the rotation group.
In general reality, the Riemann curvature tensor is also rank-4. There it is clearly a "geometric object"; moreover, the curved nature of the metric mandates consideration of covariance and contra-variance.
Meanwhile, in special relativity, it is the fundamental representations of the Poincare group that give us the allowed particle/field types: right and left spinors, 4-scalars, 4-vectors, and so on.
Finally, if you change your rotation group from $SO(3)$ to $SU(2)$, the irreducible reps are parameterized by half-integers, and the fundamental representation becomes two 2-component spinor, which can be considered the most basic "rotatable" object. Moreover, their tensor product space becomes the sum of a vector and scalar space, per:
$${\bf 2}\otimes{\bf 2}={\bf 3}\oplus{\bf 1}$$
