GAIPS Talk by Francisco Pereira (National Institute of Mental Health)

GAIPS Talk by Francisco Pereira (National Institute of Mental Health)

On November 24th at 3pm, INESC ID GAIPS Lab (Group of AI for People and Society) will host the talk “Modelling Human Mental Representations of Objects from Behavior and Text” by Francisco Pereira (National Institute of Mental Health).

Francisco Pereira is the leader of the Machine Learning Team at the National Institute of Mental Health and in this talk he will delve into the THINGS Initiative, an ongoing project collecting a vast behavioural dataset on object similarity judgments. Discover a novel machine learning approach unveiling the “core dimensions” of object representation and their impact on human interactions

The event will take place at Técnico – Taguspark (Room A3) and online via zoom.


Modelling human mental representations of objects from behaviour and text:

Psychological research about mental representations of objects describes them at different levels of granularity, from coarse (e.g. animacy, shape, colour, function, taxonomical category) to fine (e.g. thousands of properties in norming studies). However, this does not necessarily tell us which of those characteristics are most important in our interactions with objects, across tasks and purposes. In the first part of this talk, I will describe the THINGS Initiative, an ongoing effort by our collaborators to collect a behavioral dataset of millions of odd-one-out similarity judgements on thousands of objects, in tandem with many complementary datasets (fMRI, MEG, EEG, norming studies). In the context of this project, we developed a novel machine learning approach to identifying the “core dimensions” of object representation used in those judgements. Our approach models each object as a sparse, non-negative embedding, and judgements as a function of the similarity of those embeddings. The resulting model predicts subject behaviour on test data, as well as the fine-grained structure of object similarity. The dimensions of the embedding space are coherently interpretable by test subjects, and reflect degrees of taxonomic membership, functionality, and perceptual or structural attributes, among other characteristics. Furthermore, naive subjects can accurately rate new objects along these dimensions, without training. In the second part of this talk, I will describe a parallel research effort to develop models of an object interaction space. In this project, we developed an approach to represent objects in a space where each dimension corresponds to a broad mode of interaction, based on action verb selectional preferences collected in text corpora. This embedding makes it possible to predict human judgments of verb applicability in new objects. The dimensions in this space can also be used to predict categorical and functional dimensions from the object representation derived from behaviour. Together, these results suggest that interaction knowledge accounts for much of the mental representations of objects.

Francisco Pereira leads the Machine Learning Team at the National Institute of Mental Health ( The mission of the group is to assist NIMH and NIDA researchers who want to address research problems in psychology, psychiatry, and neuroscience, using statistics and machine learning approaches. In parallel, the group develops new methods and analysis approaches, motivated by the needs of researchers. Prior to that, he was a staff scientist at Siemens Healthcare, where he managed the Computational Neuroscience program. He did his postdoc at the Princeton Neuroscience Institute, and has a Ph.D. in Computer Science and Neural Basis of Cognition from Carnegie Mellon University. He promises not to talk about himself in the third person during this talk.”

Papers about his work:
“Revealing the multidimensional mental representations of natural objects underlying human similarity judgements”
“VICE: Variational Interpretable Concept Embeddings”
Mental representations of objects reflect the ways in which we interact with them”

The event is finished.


INESC-ID, “Instituto de Engenharia de Sistemas e Computadores: Investigação e Desenvolvimento em Lisboa” is a Research and Development and Innovation Organization (R&D+i) in the fields of Computer Science and Electrical and Computer Engineering. INESC-ID mission is to produce added value to people and society, supporting the response of public policies to scientific, health, environmental, cultural, social, economic and political challenges. INESC-ID promotes cooperation between academia and industry by addressing research on daily life issues, such as healthcare, space, mobility, agri-food, industry 4.0, and smart grids. This high level of knowledge transfer is achieved through both competitive research projects and direct contracted research. Public and private entities have therefore access to a pool of knowledge, resources and services provided through the unique competencies available at the institution.


INESC-ID is supported by:

Join our newsletter

* indicates required

Subscriber consent

The data submitted through this form will be used exclusively for the sending of INESC-ID Newsletter, NEWS-ID, and will not, under any circumstances, be shared with third parties. If you choose to, you can easily unsubscribe from the newsletter by following the link presented in the footer. In that case, your data will be automatically deleted from our information system. If you need to update your contact information or clarify any questions related to the newsletter, please contact By submitting this form, you give permission to the use of your personal data according to the conditions above.

We use Mailchimp as our marketing platform. By clicking below to subscribe, you acknowledge that your information will be transferred to Mailchimp for processing. Learn more about Mailchimp's privacy practices here.