Soft question: Deep learning and higher categories

Question

Recently, I have stumbled upon certain articles and lecture videos that use category theory to explain certain aspects of machine learning or deep learning (e.g. Cats for AI and the paper An enriched category theory of language: from syntax to semantics). On the other hand, people have looked into how higher categories and homotopy type theory can be used to talk about computer science (e.g. The Quantum Monadology.) My question is this:

Are there potential applications of higher category theory (or in particular $\infty$-categories) to machine and deep learning? If so, what would be the merit of the homotopy coherent formalization that $\infty$-categories bring along with them in this computer science picture?

Answer 1

Sure, there are plenty of potential applications of higher category theory to machine learning and deep learning. Such applications are still in their infancy, so don't expect a new algorithm, getting better results in faster time, in the immediate future. If that's your goal, then I think Mirco's answer is great (simplicial neural networks) and it's the one I would have given. Since he's already done so, let me instead sketch some higher level, conceptual connections between higher category and machine learning.

To understand such connections, it's helpful to zoom way out and focus on the forest instead of the trees. What is machine learning really about?

1. Data, since the model needs to train on data.
2. Iterating towards a good model.
3. Building up complexity from simple pieces.

Towards (1), category theory has been proposed as an engine for the study of Databases. I learned this from David Spivak's work, and wrote about it previously in this MO answer (which has tons of applications of category theory in the sciences). The idea is that a category can be a model for a database, and you can use category theory to enforce constraints, to make sure your database doesn't have bad data that could break the machine learning algorithm. In my book on Data Systems, we had three chapters devoted to databases and three chapters devoted to constraints. Similarly, you can use category theory to enforce constraints on the types of solutions your machine learning algorithm will come up with. Spivak has also written papers about the uses of category theory in experimental design (also summarized in the MO answer I linked to above), which matters if you want to be sure you get good data on which to build your model. You probably know that Google is running experiments on users all the time. Experimental design still matters. You might also be interested in Spivak's current work, with the Topos Institute, which uses (higher) category theory on all sorts of modern, real-world issues, including Artificial Intelligence.

Towards (2), iterating towards success makes a machine learning algorithm an evolutive system, and Spivak has studied those, too, and I wrote about them in that linked MO answer. This also brings in an area where $\infty$-categories can play a role. In any dynamical system, we care about the time axis. Often, we want to study aspects of the system that are invariant under reparameterizing time, e.g., the long-term tendancy towards equilibrium. When you start considering all possible ways to reparameterize time, you're doing homotopy theory, and $\infty$-categories can appear. There have been many papers studying the homotopy theory of dynamical systems: check out work of Bubenik, Jardine, Gaucher, and Sanjeevi Krishnan, among others.

Towards (3), this is a fundamental use of category theory, via colimits. The mindset of category theory could potentially be useful to one of the fundamental issues of deep learning, which is to understand the inner workings of a neural network and why it comes up with the answer it does. You could even imagine viewing the neural network as a functor, from data to models, and break it down using functor calculus. There are SO MANY ways to apply category theory to these questions, and I expect to see plenty of such papers in the years to come.

Here are some other, more concrete, connections between category theory and deep learning:

• Uniform Manifold Approximation and Projection (UMAP)
• Causal-net condensation
• Topological Deep Learning, which uses topological data analysis. The papers that put this on firm theoretical foundation often use model categories and $\infty$-categories. Again, see work of Bubenik, Krishnan, and others. See also the following papers: A Survey of Topological Machine Learning Methods, Topological data analysis and machine learning, Topological Data Analysis and Machine Learning Theory. And, for the $\infty$-category connection, see: Abstract homotopy theory for topological data analysis, Homotopy Theory and TDA with a View Towards Category Theory, and work of Jardine.

I'll add more if I think of any. The take-away is that there are tons of potential connections and I encourage you to investigate any that seem interesting to you!

Answer 2

After the success of Graph Deep Learning, folks are moving up the chain: a few articles, see for instance this one, have begun considering Simplicial Neural Architectures (SSN) (the main motivation is that one can study complex data structures where the vertices interact in a non dyadic relationship).

Now, all of the above is still in tis infancy, but I would venture to think that in a few years hyper-data structures and hyper-neural networks will be part and parcel of the ML arsenal. At this point, higher category theory may jump in.

How: for instance by describing operations on hyper-architectures, such as morphing one into another one.

Here are just some freewheeling suggestions:

Morphisms as Feature Maps: In an ∞-category framework, morphisms between simplices can be interpreted as feature maps or transformation functions in SNNs. This includes not only the direct relationships between simplices but also the relationships between their faces, edges, etc.

Higher Morphisms for Aggregation and Propagation: Higher morphisms can model the aggregation and propagation of features across different dimensions of the simplicial complex. For instance, a 2-morphism could represent a transformation that involves not just a direct feature map from one simplex to another but also incorporates information from their mutual relationships and boundaries.

Homotopy Coherence for Regularization: The concept of homotopy coherence from ∞-categories can be applied to ensure that operations in SNNs preserve certain topological invariants or properties. This could be seen as a form of regularization that maintains the integrity of data's topological structure, ensuring that learned features are robust and meaningful with respect to the data's inherent geometry.

PS There are a few basic implementations of simplicial NNs which you can play with: one is TopoX