Multi-solution is a prominant feature of ANNs (DNNs/RNNs) when training to perform certain tasks. Is there any common feature between different solutions remains an open questions. This works found that the topology of fixed points of trained network is the universally shared between different network architectures and realizations when those networks are trained for the same task. Further, they demonstrated the topological structure of fixed points of networks indeed interprets computation mechanism of trained networks.
-
Networks with different architectures and nonlinearities do differ in the sense of conventional representational similarity analysis (RSA).
-
Fixed point topology can be invariant across networks performing same tasks.
-
Linearized dynamics showed the computational ability in the dynamical regime around the fixed point points, which further proved the common dynamical feature across different network realizations.
-
This RSA protocol on fixed point topology might be useful to quantify the similarity between RNN and BNNs (biological neuronal networks). (Also it is the critical issue when we use DL to study neurophysiological data.)
-
Understanding fixed point topology and linear behavior around “generalized fixed points” can be a good perspective to study the computation through dynamics in RNNs, with related topics as memory capacity, learning dynamics, online learning (continual learning), and etc.