Probably what you really want is the fusion graph $\Gamma_X$ with respect to your simple object $X$. It is a directed graph with vertices labelled by the simple objects of your category. Between the vertices labelled by simples $Y$ and $Z$, there are $N^{Y,X}_Z=\operatorname{dim}(\operatorname{Hom}(Y\otimes X, Z))$ directed edges from $Y$ to $Z$. The fusion graph with respect to $X^*$ is obtained by reversing the orientation of all the edges, since by semisimplicity and Frobenius reciprocity,
$$
N^{Y,X}_Z
=
\operatorname{dim}(\operatorname{Hom}(Y\otimes X, Z))
=
\operatorname{dim}(\operatorname{Hom}(Y, Z\otimes X^*))
=
\operatorname{dim}(\operatorname{Hom}(Z\otimes X^*, Y))
=
N^{Y,X^*}_Z.
$$
This means that if $X$ is self-dual, then you can view this graph as unoriented.
In your question you've referred to a pair of unoriented graphs. These would arise if you were looking at the principal graphs of a subfactor. Given a finite index subfactor $A\subset B$, we have the semisimple tensor category of $A-A$ bimodules generated by $B$. The simples are obtained by decomposing the tensor powers $\bigotimes_A^n B$ into irreducibles. We can also view $\bigotimes_A^n B$ as an $A-B$ bimodule and decompose into irreducibles. This gives the semisimple module category of $A-B$ bimodules. Similarly, we have a tensor category of $B-B$ bimodules and a module category of $B-A$ bimodules.
We get the principal graphs $(\Gamma_+,\Gamma_-)$ as follows. The graph $\Gamma_+$ has even vertices the simple $A-A$ bimodules and odd vertices the simple $A-B$ bimodules. There are then $\operatorname{dim}(\operatorname{Hom}_{A-B}(Y\otimes_A B, Z))$ edges between the simple $A-A$ bimodule $Y$ and the simple $A-B$ bimodule $Z$. The dual principal graph $\Gamma_-$ is defined analogously with $B-B$ and $B-A$ bimodules.
Edit after the comments:
Given a unitary fusion category $\mathcal{C}$ and an object $X\in\mathcal{C}$, you can construct a subfactor standard invariant $\mathcal{P}_\bullet$ from which you can get a pair of principal graphs $(\Gamma_+,\Gamma_-)$. First, you let
\begin{align*}
\mathcal{P}_{n,+} &= \operatorname{End}(\underbrace{X\otimes X^*\otimes \cdots \otimes X^\pm}_{n \text{ copies}})
\\
\mathcal{P}_{n,-} &= \operatorname{End}(\underbrace{X^*\otimes X\otimes \cdots \otimes X^\mp}_{n \text{ copies}})
\end{align*}
where $X^\pm=X$ if $n$ is odd and $X^*$ if $n$ is even. Now the graph $\Gamma_+$ has even vertices the simple summands of objects of the form $(X\otimes X^*)^{\otimes n}$, and odd vertices the simple summands of objects of the form $(X\otimes X^*)^{\otimes n}\otimes X$. You get the edges similar to before, taking an even $Y$, tensoring with $X$, and decomposing into simples $Z$. The graph $\Gamma_-$ is defined similarly, but swapping the roles of $X$ and $X^*$.
So that's the formal explanation of how you get the principal graphs, but there is a simpler way to compute $(\Gamma_+,\Gamma_-)$ based on the fusion coefficients $N^{Y,X}_Z$.
As @Xiao-Gang points out in the comments, we can compute $\Gamma_+$ as the connected component containing $X$ in the bipartite graph with adjacency matrix
$$
\begin{pmatrix}
0 & N^X
\\
N^{X^*} & 0
\end{pmatrix}
$$
where $N^X$ is the matrix of fusion coefficients for tensoring on the right by $X$, and similarly for $X^*$.
The graph $\Gamma_-$ is then obtained by swapping the roles of $X$ and $X^*$.
Interestingly, if $X\in \mathcal{C}$ is self-dual and the fusion graph $\Gamma_X$ in $\mathcal{C}$ is already bipartite (here, $\mathcal{C}$ is $\mathbb{Z}/2\mathbb{Z}$-graded and $X$ is odd), then $(\Gamma_+,\Gamma_-) = (\Gamma_X, \Gamma_X)$. This occurs for many interesting examples.