Collected Works (volume 1): Published Papers

Book excerpt: The motivation for the research that is described in these volumes is the wish to explain things in terms of their underlying causes, rather than merely being satisfied with phenomenological descriptions. When this reductionist approach is applied to information processing it allows the internal structure of information to be analysed, so information processing algorithms can then be derived from first principles. One of the simplest examples of this approach is the diagonalisation of a data covariance matrix – there are many variants of this basic approach, such as singular value decomposition – in which the assumed independent components of high-dimensional data are identified and extracted. The main limitation of this type of information analysis approach is that it is based on linear algebra applied globally to the data space, so it is unable to preserve information about any local data structure in the data space. For instance, if the data lives on a low-dimensional curved manifold embedded in the data space, then only the global properties of this manifold would be preserved by global linear algebra methods. In practice, data whose high-dimensional structure is non-trivial typically lives on a noisy version of a curved manifold, so techniques for analysing such data must automatically handle this type of structure. For instance, a blurred image of a point source is described by its underlying degrees of freedom – i.e. the position of the source – and as the source moves about it generates a curved manifold that lives in the high-dimensional space of pixel values of the sampled image. The basic problem is then to deduce the internal properties of this manifold by analysing examples of such images. A more challenging problem would be to extend this analysis to images that contain several overlapping blurred images of point sources, and so on. There is no limit to the complexity of the types of high-dimensional data that one might want to analyse. These methods then need to be automated so that they do not rely on human intervention, which would then allow them to be inserted as “components” into information processing networks. The purpose of the research that is described in these volumes is to develop principled information processing methods that can be used for such analysis. Self-organising information processing networks arise naturally in this context, in which ways of cutting up the original manifold into simpler pieces emerge automatically.