Summary of the project

We can use language to talk about the world we live in. This involves converting rich sensory/motor representations of the world into high-level symbolic representations. How this happens is largely an open question in cognitive science, because language and sensorimotor processing are currently studied in separate disciplines: linguistics and sensorimotor neuroscience.

Our novel claim is that these disciplines are deeply connected. We propose a linking hypothesis, positing detailed connections between influential theories of language syntax (Chomsky, 1995) and sensorimotor cognition (Ballard et al., 1997). In our hypothesis, the syntax of a sentence reporting a concrete episode in the world directly describes the sensorimotor routine through which the episode was perceived.

Our hypothesis makes detailed predictions about the episode-perception routine associated with any concrete sentence. The project will test these predictions, for four selected sentence types. In each case, we will build a computational model of the actual sensorimotor routine through which the episode type in question is perceived, motivated from experimental evidence. We will then compare this routine to that predicted from syntax. If the predicted routines match those motivated experimentally, this will furnish evidence that syntactic analyses directly deliver information about neural processing: an exciting finding for neuroscience.

The project also involves developing a computational model of the brain mechanism that converts sensorimotor representations of episodes into language. This is expressed as a neural network model of infant language acquisition. The model learns to convert sensorimotor representations into sentences: it can be trained on data from several different languages. Our current focus is on English, Maori and Slovak.

References
Ballard, D., Hayhoe, M., Pook, P., and Rao, R. (1997). Deictic codes for the embodiment of cognition. Behavioral and Brain Sciences, 20(4), 723–767. Chomsky, N. (1995). The Minimalist Program. MIT Press, Cambridge, MA.

Project investigators

Principal Investigator

Assoc. Prof. Alistair Knott

Ali is a linguist and cognitive scientist. He has extensive experience in developing computational models of natural language and its relation to the sensorimotor system, and has published in the top journals in Artificial Intelligence, Computational Linguistics and Cognitive Science. His recent book Sensorimotor Cognition and Natural Language Syntax (MIT Press, 2012) was described in a review by Viviane Deprez as 'a ground-breaking and foundation-building look at the underpinnings of human language syntax', that opens up 'untravelled avenues for interdisciplinary research', and constitutes 'a major advance'.

Associate Investigator

Assoc. Prof. Lubica Benuskova

Luba is an expert in the development of neural network models of cognitive mechanisms. Her research focus is on neurobiologically inspired computing, and the dynamics of biological and artificial neural networks. She has published extensively on synaptic learning rules, neural models of classification and prediction, and computational neurogenetic modelling (a new area investigating the influence of gene regulatory networks on neural dynamics). She has particular expertise in neural network models of sequence learning and sequence representation, which play an important role in the current project.

Postdoctoral Researcher

Dr Martin Takac

Martin has expertise in cognitive semantics, adaptive knowledge representation and computational modelling of infant language acquisition. He has participated in 14 research grants, and is actively involved in the joint Middle-European Interdisciplinary Programme in Cognitive Science. In 2008 he was awarded a prestigious BuildIT postdoctoral scholarship, which allowed him to begin collaborating on language modelling with Drs Knott and Benuskova. He has designed and implemented a number of models of language acquisition and working memory.

Postdoctoral Researcher

Dr Lech Szymanski

Lech is an expert in machine learning and neural networks. His background is in Computer Engineering and Digital Signal Processing, which he studied to MSc level at the University of Ottawa. After a PhD at the Department of Computer Science at the University of Otago, he has worked on several projects relating to machine vision and machine learning. His role in the current project is to develop a model of visual attention to properties of objects.

Publications

Here are some publications relevant to the current project.

Our project in the media

• An interview with Ruth Beran on 'Our Changing World' (NZ National Radio)
• Talking Sense: University of Otago magazine Issue 37 (October 2014)
• The building blocks of language and perception: talk by Alistair Knott at TEDx Athens (December 2013)
• Teaching computers to talk: talk by Alistair Knott at TEDx Auckland (December 2012)