computer science

OPENS DOORS

Computational Linguistics Research Group

Language and Sensorimotor Cognition

The main hypothesis that we are exploring in this project is that the meaning of a sentence is not a direct characterisation of the world, but rather a characterisation of a cognitive process executing in an agent interacting with the world. This interaction can either be via perception, or motor control, or both. We are investigating this hypothesis by looking at sentences describing events which an agent could perceive or bring about using relatively simple sensorimotor mechanisms: existential sentences (e.g. "There is a cat in the garden") and transitive action sentences (e.g. "John grabbed a cup"), and seeing whether models of the syntactic structure of these sentences have anything in common with models of the sensorimotor mechanisms themselves.

We are implementing a simple computational model of several components of sensorimotor processing, including early vision, visual attention, Kahneman et al's object files, object categorisation and modular motor control. Our aim is to build a model of natural language which is parasitic on this sensorimotor model.

There are several points of contact between this work and work going on in other groups. In particular:

  • The NTL (Neural Theory of Language) group at UC Berkeley have been pursuing a very similar hypothesis for several years.
  • Several researchers interested in the evolution of language have also touched on similar topics. See Jim Hurford's home page for some useful links.



    • Sample papers

  • A Knott: Grounding syntactic representations in an architecture for sensorimotor control. Otago University technical report OUCS-2003-04, 2003.
  • A Knott: Argument linking and spatial cognition. Proceedings of SEMPRO 2001: Cognitively Plausible Models of Semantic Processing. University of Edinburgh, July 2001. (PDF version)