<!-- to add rest of team as authors w/ CRediT -->


```{r}
#| label: setup
#| include: false

library(psych) # ICC, alpha, omega
library(furrr) # run code parallel
library(tidyverse)
library(flextable)
#devtools::install_github("debruine/webmorphR@dev")
#library(webmorphR)

set_flextable_defaults(digits = 2)
theme_set(theme_bw(base_size = 14))

source("code/custom_functions.R")
source("code/1_dataprep.R")
source("code/2_agreement.R")
```

```{r}
# prep variables to be used later

exp_levels_neutral <- c("attractive", "trustworthy", "dominant", 
                        "memorable", "gender-typical")

exp_levels_neutral_unstd <- c("attractive (unstd)", "trustworthy (unstd)",
                              "dominant (unstd)")

exp_levels_emo_cat <- c("emotion 1", "emotion 2", "emotion 3", 
                        "emotion 4", "emotion 5", "emotion 6")

exp_levels_emo_int <- c("anger", "disgust", "fear",
                        "happiness", "sadness", "surprise")

rainbow <- c("firebrick", "darkorange", "goldenrod",
             "darkgreen", "dodgerblue3", "darkorchid4")
```

```{r eval=FALSE}
# OLD CHUNK -- KEEP FOR NOW WHILE I'M UPDATING, DELETE ONCE FINISHED
# read in main data used

# Model data
ct <- cols(
  age = col_integer(),
  height = col_integer(),
  weight = col_integer(),
  ethnicity_rec = col_factor(levels = c("White", "Black", "Asian", "Indigenous", "MENA", "Latine", "Mixed", "Ambiguous label"), include_na = TRUE)
)
data_models <- read_csv("data/manyfaces-pilot-models_cleaned.csv",
                        col_types = ct)

# Rating data (post exclusions)
exp_levels <- c("attractive", 
                "dominant", 
                "trustworthy", 
                "attractive (unstd)", 
                "dominant (unstd)", 
                "trustworthy (unstd)",
                "memorable",
                "gender-typical",
                "age",
                "anger",
                "disgust", 
                "fear", 
                "happiness", 
                "sadness", 
                "surprise",
                "emotion 1", 
                "emotion 2", 
                "emotion 3", 
                "emotion 4", 
                "emotion 5", 
                "emotion 6"
              )
ct <- cols(
  exp = col_factor(levels = exp_levels)
)
data_exp <- read_csv("data/manyfaces-pilot-exp_cleaned.csv",
                     col_types = ct)

# Questionnaire data (post exclusions)
data_qu <- read_csv("data/manyfaces-pilot-quest_cleaned.csv",
                    show_col_types = FALSE)

# Exclusions
data_excl <- read_csv("data/exclusions.csv",
                    show_col_types = FALSE)
```

## Introduction

Faces are rich sources of information, playing a key role in human social perception. They strongly draw attention, serving as a primary means of person recognition and informing social inferences [e.g., @freeman_more_2016; @sutherland_understanding_2022; @young_recognizing_2017; @zebrowitz_reading_1997]. Various subfields within psychology—including vision science, person perception, social cognition, affective science, cognitive neuroscience, behavioural science, and person recognition—address research questions involving faces and thus necessitate the use of face images as stimuli. Here, we introduce a novel face image database for research use. This database was collected as part of an international multi-lab collaboration and addresses various limitations of existing databases. Crucially, we take a transparent approach aligned with Open Science best practices and make openly available the reproducible protocol we followed to collect the images, enabling future expansion of the database.

### Existing face databases

A wealth of face stimulus databases exists, created for various purposes. Many of these are available [see e.g., @workman_face_2021 face image meta-database to search many of these] for research purposes allowing access to a broad variety of face stimuli, well-suited to address questions in a given subfield. For example, there are many different databases in which photographed individuals (referred to as targets or models) display different facial expressions of emotion [e.g., @ebner_facesdatabase_2010; @van_der_schalk_moving_2011, of use in affective science], appear in varying lighting and angles [e.g., @burton_glasgow_2010; @gao_cas-peal_2008, useful for person recognition research], or belong to various age or racial/ethnic groups (for perception researcher). However, many studies tend to rely on a small number of those stimulus databases, which introduces potential problems. For example, 4107 published papers cite the Karolinska Directed Emotional Faces [@lundqvist_karolinska_2015], 4375 cite the NimStim Set of Facial Expressions [@tottenham_nimstim_2009], 2461 cite the Chicago Face Database [@ma_chicago_2015], and 3219 the Radboud Faces Database [@langner_presentation_2010]. Although citation counts are not a direct measure of the use of these databases, they nonetheless indicate their popularity. This frequent (over)use of the same sets of stimuli can compromise generalizability when various research questions are tested on only a limited sample of face images from specific backgrounds [@debruine_reproducible_2022; @yarkoni_generalizability_2022]. Participants, especially those participating online, may also become familiar with frequently used stimuli, potentially introducing biases to their responses. 

Of course, not all research involving face images uses stimuli from published databases. It is common for researchers to purposefully collect images themselves. But these stimulus sets are seldom openly shared (e.g., due to ethical limitations). Although this circumvents the issues with frequently-used databases, it introduces its own set of problems. Chiefly, face images collected by researchers for their own research use are often insufficiently documented. Although the resulting images may be described and/or illustrated with an example, the process of image collection is rarely described in adequate detail. This is often the case for even those databases openly available for research use. This lack of image acquisition (and processing) methods documentation limits transparency, replication attempts--for example, another researcher attempting to replicate a finding using new face images collected themselves, rather than the images collected by the original researchers [see @trebicky_grim_2024]--and assessments of results' validity (as undercharacterization of stimuli can preclude identification of issues like confounding factors). Furthermore, different research teams often employ protocols that can differ significantly from each other in a plethora of major and minor details, including hair (e.g., free hairstyle or pulled back; facial hair allowed or not), facial accessories, makeup, lighting conditions, and type of shirt worn. These often-unreported details further hinder the comparison between different studies. 

In many databases (both publicly available and not), there are also limitations in the diversity of stimuli. For example, many include only images of individuals who self-identify as White or Western [e.g., see @cook_why_2021 see also @martinez_facecraft_2025 for a discussion of the potential reification of race in face research]. This reinforces the centering of Whiteness and Western culture in psychological research, importantly limiting the ecological validity and generalizability of conclusions [see also @henrich_weirdest_2010; @roberts_challenging_2023]. Recently created databases have sought to address this, specifically collecting images of multiracial/multi-ethnic and non-Western samples, for example [e.g., @bastanfard_iranian_2007; @chen_broadening_2021; @courset_caucasian_2018; @meyers_hawaii_2024; @saribay_bogazici_2018; @trzewik_israeli_2025], but more such work is needed to further diversify the pool of available face research stimuli. Many world regions and ancestries remain underrepresented (e.g. East African, First Nations, Australian Aboriginal people). This may be due, in part, to resource limitations: Equipment and setups for collecting images suitable for research use can be prohibitively expensive and complex, making database collection not feasible for everyone and everywhere, thereby limiting database diversity. Additionally, researchers with the necessary equipment may be limited in terms of the diversity of their available participant sample in terms of ancestry and cultural heritage that may shape visible facial features. Furthermore, to date, no extant database contains images of individuals from *multiple* world regions. 

A final limitation of extant face image databases is their largely fixed state. That is, once collected, the number of stimuli available does not change. There are some exceptions to this, such as multiple waves of additions to the Chicago Face Database [@lakshmi_india_2021; @ma_chicago_2021]. Continued additions to databases are not the norm, however. This not only constrains the number of stimuli in any given database, increasing the likelihood of the frequent reuse of any individual stimulus, but can also render some databases obsolete over time (e.g., due to very dated model appearance/styling).


### Open Science considerations

Open and transparent methods are essential for fully understanding, evaluating, and replicating research. Although methods reporting has become more open and accessible with researchers sharing materials such as questionnaire wording, stimuli, and programming scripts, the reporting of face image database collection procedures often lacks crucial details for full transparency and reproducibility. This matters because differences in often-undocumented details such as focal length, for example, can importantly affect faces' appearance, subsequently impacting perceptions of them [e.g., @trebicky_focal_2016; see @trebicky_grim_2024 for further discussion].

Big Team Science (i.e., multi-lab) endeavours and large-scale collaborations are now recognized as a vital part of improving science. Within psychology, multiple such initiatives have importantly contributed to addressing a variety of research questions, improving diversity and generalizability in the field (e.g., in the domain of face perception, Jones et al.’s [-@jones_which_2021] Psychological Science Accelerator project). Such an approach has yet to be applied to stimulus database collection, but it represents an opportunity to address many of the limitations of face image databases raised.


### The current work

Here, we sought to address the limitations of existing face image databases through a Big Team Science approach. This served as the first project of ManyFaces, an international consortium of face perception and recognition researchers formed in 2022.

To address transparency and reproducibility issues, we developed an openly available, reproducible protocol for face image collection. The protocol covers collecting multiple images (varying in standardization, viewing angle, and facial expression) of each target/model to benefit a wide array of possible research areas and questions related to face perception and recognition. Following this protocol, we collected images (Phase 1) in 20 different labs across the world, spanning 11 countries and five world regions (Europe, Latin America, North America, the Middle East, and Southeast Asia). This resulted in a diverse set of models and images, more than would be possible to collect in or by a single lab. Moreover, this database is not static but can be added to in the future by any interested researcher following the protocol. We also validated the database (Phase 2) by recruiting online perceivers to rate a subset of the images (i.e., front-facing images) to generate norming data for key social trait perceptions and emotion recognition. We make this image database and norming data available for future research.

Altogether, we introduce a new, diverse, openly-available face stimulus set that cancontinue to grow in the future. We believe that this effort will broaden the generalizability of findings in face perception and recognition research.


## Study/phase 1: Protocol development & stimulus collection

To set face perception and recognition Big Team Science in motion and enable conducting multi-site studies, a set of stimuli suitable for most areas (e.g., in terms of research question, geographic location) is needed. Therefore, we developed a protocol allowing us to collect such a database of face images. We developed the protocol with maximum usability and accessibility in mind, making it transparent and readable to non-experts, and designed for use with attainable (vs. highly specialized and expensive) equipment, with minimal setup requirements, and without requiring expertise in photography. Note that the protocol is for in-lab image collection to minimize external noise, to enable the collection of models' self-report data, and to ensure the consensual use of models' images (vs. scraping images from online sources or generating them). Photographing faces in a controlled lab environment thus strikes a balance in terms of ecological validity and ethical concerns [see @trzewik_israeli_2025 for discussion of the value of lab-photographed vs. artificially generated and ambient face images]. 


### Method

For both studies/phases, the University of Glasgow provided ethical approval. We obtained additional local ethical approval at collaborating institutions where required.


#### Protocol Development 

The team curated a set of equipment and developed a protocol for collecting standardized, reproducible images of faces for research. Following a survey among the ManyFaces members^[A call to join the consortium was initially disseminated over social media, followed by an initial meeting with members to determine the general direction of the group and potential future projects. We then developed a webpage to enable more people to join and set up a network of communication to discuss and prioritise projects.], we constructed the protocol for the collection of a variety of images that would be useful for a broad variety of research questions. Specifically, we included images varying in their standardization of appearance (standardized and unstandardized, e.g., white t-shirt and hair pulled back vs. clothing and hair as worn by the participant/model on that day), viewing angle (frontal, profile, ¾), and facial expression (neutral, natural, angry, disgusted, fearful, happy, sad, surprised).

Prior to beginning data collection, the ManyFaces team pre-tested the protocol to ensure clarity of instructions and consistency of images collected across sites. We revised the protocol to address issues that arose (e.g., revised facial expression elicitation instructions, clarified camera settings). The final protocol can be found on the OSF (<https://osf.io/qxwvz>; see also <https://osf.io/nkahx> for all materials). In sum, the protocol detailed the following:


##### Image Types 

The protocol defined the categories of images in terms of their standardisation of appearance, viewing angle, and facial expression.

* Standardisation of Appearance. In the *standardised* images, models wore a white crew neck t-shirt, had their hair pulled back or covered, included no adornments/accessories (excepting those that could not be removed for cultural reasons), and wore minimal or no makeup (models were informed about this beforehand). In contrast, *unstandardised* images showed models in their own clothing, with their hair as they came into the lab, and with adornments, except for glasses or anything that obscured the face or neck.  
* Viewing Angle. *Full frontal portraits* showed models facing the camera, *left and right profile portraits* showed each side of models’ faces in profile, and *left and right ¾ portraits* showed models’ faces at a 45-degree angle from the axis of the camera.  
* Facial Expression. *Neutral* images were of models refraining from making any facial expression, *natural* images displayed a natural expression for the model, and each facial expression of emotion displayed the expression the model would make if they were feeling the specified emotion (models instructed e.g., "face the camera with the expression you would make if you were feeling happy").

##### Image Prioritisation

The protocol specified which images to take (i.e., the combinations of appearance standardisation, viewing angle, and facial expression), in what order, and which were most crucial to collect (as voted by the ManyFaces members) if researchers were under time constraints^[Pilot testing gave us the estimated/recommended amount of time for data collection, but some labs may have been able to devote less time due to resource or time constraints]. The order of image collection was as follows, with starred (*) images prioritised/required for each model:

* Exposure calibration photo*
* Identification & calibration photo*
* Unstandardized - Natural - Full frontal portrait
* Unstandardized - Neutral - Full frontal portrait*
* Unstandardized - Neutral - Left and Right profile portraits
* Unstandardized - Neutral - Left and Right ¾ portraits
* Unstandardized - Happy - Full frontal portrait*
* Unstandardized - Happy - Left and Right profile portraits
* Unstandardized - Happy - Left and Right ¾ portraits
* Unstandardized - All other expressions (in random or appropriate order) - Full frontal portrait
* Standardized - Natural - Full frontal portrait
* Standardized - Neutral - Full frontal portrait*
* Standardized - Neutral - Left and Right profile portraits*
* Standardized - Neutral - Left and Right ¾ portraits*
* Standardized - Happy - Full frontal portrait*
* Standardized - Happy - Left and Right profile portraits
* Standardized - Happy - Left and Right ¾ portraits
* Standardized - All other expressions (in random or appropriate order) - Full frontal portrait

Researchers with additional time could also capture the other expressions at different viewing angles (Left/Right profile, ¾) with either unstandardized or standardized appearance, as well as a video of standardized appearance to test out 3D image capture.


##### Equipment

All collaborating sites/labs used the same set of equipment. <!--note the 2 US labs used their own equipment - check whether exact same or comparable--> Images were captured using a Canon EOS 250d (also called Canon EOS Rebel SL3 in some regions) camera with a kit lens (Canon 18-55mm IS STM lens), and lit by an LED Ring light (Fovitec Bi-Colour LED 18” Ring Light) on a stand. For colour calibration, Calibrite ColourChecker Classic card was used. The image collection spaces were to have a white background with a chair for models to sit in. Researchers provided white t-shirts for models to wear for standardised images.

##### Setup

The setup of the room used for image collection at each site required removing or minimising external light sources (e.g., using a windowless room, window blinds, turning off overhead lights) and colour spill. The protocol specified the distances at which to set up the chair for models to sit in relative to the white background and the lighting and camera rig. Camera setup instructions, including the specified shooting and focusing modes, shutter speed, aperture, ISO, white balance, colour space, and file type.

##### Procedure

Finally, the protocol detailed the written informed consent process, how to prepare and position models, and how to take each kind of image listed in the image prioritisation section. This included the positioning of the model’s head for each viewing angle and facial expression elicitation instructions.

#### Stimulus Collection

```{r}
lab_n <- data_models$lab_id |> unique() |> length()

models_per_site <- count(data_models, lab_id)$n
mean_mps <- mean(models_per_site) |> round(1)
```


Following the protocol, `r lab_n` labs (1 in Austria, 2 in Brazil, 2 in Canada, 2 in Germany, 1 in Israel, 3 in Malaysia, 1 in Mexico, 1 in the Netherlands, 1 in Serbia, 4 in the UK, and 2 in the US) collected images of an average of `r mean_mps` models per site. As specified by the protocol, researchers collected multiple images of each model, within the time constraints of the study session.

Models furthermore completed a demographic questionnaire (<https://osf.io/7w5du/files/q3p7d>) on Experimentum, reporting their age, gender, ethnicity, height, and weight. They also reported whether or not they were wearing makeup (specifying what kinds, e.g., foundation, eye makeup, semi-permanent makeup) and whether they had ever experienced anything that could affect the shape of their face (e.g., broken nose, cosmetic surgery or injections, orthodontic work) and specified this if they were willing. Finally, they completed a debriefing questionnaire (<https://osf.io/7w5du/files/26b8c>) asking them about their experience of posing for the images (e.g., whether instructions were clear, whether any part of the process was uncomfortable).

### Results

#### Images

```{r}
model_n <- nrow(data_models)
```


`r model_n` total models provided their images, following withdrawals and exclusions for poor image quality. @tbl-images-per-type details the number of models for each kind of photo. Example images are shown in @fig-img-examples. Images are available for research use and can be requested here: [link to be added] <!--add: link to where the images can be requested--> Note that images cannot be shared without submitting a request and completing a research use agreement, in line with ethical constraints.

```{r}
#| label: tbl-images-per-type
#| tbl-cap: The number of images available for each image type. The column `std` is for standardised images, with neutral clothing, no makeup, and hair pulled back, while `unstd` is for unstandardised images with natural hair and makeup.

emo_levels <- c(neu = "neutral", 
                ang = "anger", 
                dis = "disgust", 
                fea = "fear", 
                hap = "happy", 
                sad = "sad", 
                sur = "surprised")

image_type_n <- data_models |>
  select(model_id:unstd_neu) |>
  pivot_longer(std_ang:unstd_neu) |>
  filter(value == 1) |>
  count(name) |>
  separate(name, c("type", "emotion")) |>
  pivot_wider(names_from = type, values_from = n) |>
  mutate(emotion = factor(emotion, names(emo_levels), emo_levels)) |>
  arrange(emotion)

image_type_n |>
  flextable()
```

```{r}
#| label: fig-img-examples
#| fig-cap: Example images of each level of standardisation, angle, and expression
#| fig-alt: "A 7x6 grid of pictures of a woman in a white t-shirt with a white headband, showing different angles and expressions. Top labels: neutral, anger, disgust, fear, happy, sad, surprised. Side labels: standardised front, standardised left 90 (side profile), standardised left 45 (three-quarter profile), standardised right 45, standardised right 90, unstandardised front."

knitr::include_graphics("fig/img-examples.png")
```


#### Model Demographics

```{r}
# count faces per ethnicity category
eth_rec <- count(data_models, ethnicity_rec, .drop = FALSE)
eth_n <- as.list(eth_rec$n)
names(eth_n) <- c(as.character(eth_rec$ethnicity_rec[1:7]), "NA")
```


For data cleaning, we first downloaded and reshaped the raw data from Experimentum. In the next step, we ensured that the models’ gender, race/ethnicity, and units of height and weight were consistently formatted across labs. 

For gender and race/ethnicity, words presented in languages other than English were recoded to be presented in English (e.g., “mulher” to “female”, “preta” to “Black”). We then classified self-described race/ethnicity into one of seven categories (“White” (n = `r eth_n$White`), “Black” (n = `r eth_n$Black`), “Asian” (n = `r eth_n$Asian`), “Indigenous” (n = `r eth_n$Indigenous`), “MENA” (n = `r eth_n$MENA`)<!-- this code leaves a blank--> (Middle Eastern or North African), “Latine” (n = `r eth_n$Latine`), or “Mixed” (n = `r eth_n$Mixed`)), where possible. Descriptions that could not be clearly sorted into these categories were given “Ambiguous label” (n = `r eth_n[["Ambiguous label"]]`)<!-- this code leaves a blank--> and non-entries were recoded as NA (n = `r eth_n[["NA"]]`). 

For height, we ensured all data were presented in centimeters, and for weight, we ensured that all data were presented in kilograms. Models could report height and weight in metric or imperial units, so we converted from imperial to metric where required. We also sanity-checked the reported units, assuming heights > 100 to be in centimeters and those < 100 to be in inches, and weights < 80 to be in kilograms. Any non-entries of height or weight for a model was recoded as NA .

```{r}
#| label: tbl-model-demog
#| tbl-cap: Summary of model demographic info

data_models |>
  mutate(gender = gender_rec) |>
  summarise(
    .by = gender,
    n = n(),
    age_min = min(age, na.rm = TRUE),
    age_max = max(age, na.rm = TRUE),
    age_mean = mean(age, na.rm = TRUE),
    height_min = min(height_cm, na.rm = TRUE),
    height_max = max(height_cm, na.rm = TRUE),
    height_mean = mean(height_cm, na.rm = TRUE),
    weight_min = min(weight_kg, na.rm = TRUE),
    weight_max = max(weight_kg, na.rm = TRUE),
    weight_mean = mean(weight_kg, na.rm = TRUE)
  ) |>
  flextable() |>
  add_header_row(values = c("", "Age (years)", "Height (cm)", "Weight (kg)"),
                colwidths = c(2, 3, 3, 3)) |>
  set_header_labels(age_min = "min", 
                    age_max = "max",
                    age_mean = "mean",
                    height_min = "min", 
                    height_max = "max",
                    height_mean = "mean",
                    weight_min = "min", 
                    weight_max = "max",
                    weight_mean = "mean") |>
  colformat_char(j = 1, na_str = "missing") |>
  colformat_double(digits = 1) |>
  flextable::width(j = 1, width = 1)
          
```


## Study/phase 2: Validation/norming data

We next obtained perceptions/ratings of a subset of the photos (front-facing) to validate the emotion expressions (their perceived emotion category and intensity) and collect norming data on central social perceptions, namely perceived attractiveness, dominance, trustworthiness, gender-typicality, memorability, and age. We chose these ratings due to their central importance in the person perception, face recognition, and emotion perception literatures [e.g., @bainbridge_intrinsic_2013, @jack_four_2016, @oosterhof_functional_2008; @sutherland_social_2013; @perrett_your_2017]. We preregistered this study on the OSF (<https://osf.io/4d5v9>).

### Method

#### Image Processing
<!--move this section to phase 1?-->

RAW images were processed using webmorphR [@debruine_reproducible_2022], which facilitates scriptable processing of images using imagemagick [@imagemagick] (a full script of the processing steps is available at <https://github.com/ManyFacesTeam/imgprep>). Briefly, 

* Each face was delineated using the Face++ automatic face detection algorithm (see <https://www.faceplusplus.com/>) to generate a 106-point template.
* All images were resized to 1000w by 1500h pixels to standardise size (two different RAW formats were used, which resulted in two different image sizes)
* The median RGB colour value of a 100×100 pixel patch at the upper left corner was calculated to fill in any edges from alignment.
* The image was repositioned and cropped (not rotated or resized) such that
    * The image size was 675w by 900h pixel
    * point 71 (between the eyes) was relocated to position [.5w, .4h]
* This aligned image was saved as a lossless PNG using imagemagick default settings (e.g., sRGB colour space).
* A white balance correction was applied to the resulting images, calculated from the mean RGB values in the 25x25 pixel top-right corner patch (white background) ^[We did not fully colour-calibrate the images here (see Discussion) but future researchers may wish to do so.] 
* These images were converted to JPEGs with a quality setting of 75 to reduce file size for stimulus display online.  



```{r}
#| label: fig-img-processing
#| fig-cap: "Image processing steps: (A) the raw image, (B) the image automatically delineated and cropped to a standard alignment, (C) white balance correction."

knitr::include_graphics("fig/img-processing.png")
```


#### Stimuli

```{r}
# count images for each emotion and unstandardised
emo_n <- as.list(image_type_n$std)
names(emo_n) <- image_type_n$emotion
unstd <- image_type_n$unstd[[1]]
```


The number of stimuli was determined by the number of models recruited across the 20 research labs and how many models posed for each image type. The maximum number of targets per image type was 205^[The preregistration mis-stated the total number of models, rather than the maximum number of images, to be 205. The total number of models was 211], meaning each rater saw up to 205 stimuli. We obtained ratings of the front-facing standardised neutral (n = `r emo_n$neutral`), unstandardised neutral (n = `r unstd`), standardised angry (n = `r emo_n$anger`), standardised disgusted (n = `r emo_n$digust`)<!-- this code leaves a blank-->, standardised fearful (n = `r emo_n$fear`), standardised happy (n = `r emo_n$happy`), standardised sad (n = `r emo_n$sad`), and standardised surprised (n = `r emo_n$surprised`).


```{r}
#| label: fig-stim-examples
#| fig-cap: An example of each type of stimulus, with the total number of stimuli of that type in parentheses.
#| fig-alt: "Example face images for the 8 types of stimuli: unstandardised (177), neutral (205), anger (187), disgust (184), fear (175), happy (199), sad (183), surprised (184)"

knitr::include_graphics("fig/stim-types.png")
```


#### Attention Check Stimuli

Additionally, we created stimuli for attention checks. These were white images with the same size and aspect ratio as the face stimuli, but contained only the written instruction to choose a specific response, (e.g., ‘Choose "fear"’ or ‘Choose "3"’). 

#### Measures

##### Standardised Neutral Faces

###### Trait Ratings

We obtained ratings of faces’ attractiveness, dominance, trustworthiness, memorability, and gender-typicality (‘How attractive [dominant, trustworthy, memorable, gender-typical] does this person look?’). Ratings were on scales ranging from 1 (*not at all*) to 7 (*very*).

###### Demographic Impressions

We obtained ratings of faces’ perceived age (‘How old does this person look?’), with responses collected in 5-year ranges/brackets (i.e., 16-20, 21-25, …, 76-80, 81+). 

<!--still to add perceived race/ethnicity-->

##### Unstandardised Neutral Faces

###### Trait Ratings

We obtained ratings of faces’ attractiveness, dominance, and trustworthiness on scales ranging from 1 (*not at all*) to 7 (*very*).

##### Standardised Emotional Faces

###### Emotion Categorisation

We obtained impressions of the emotion each person was expressing (‘What emotion is this person expressing?’), choosing one from: *anger, disgust, fear, happiness, sadness, surprise, other*. Here, raters categorized a counterbalanced mixture of expressions (from one of six counterbalanced conditions) rather than faces all showing the same expression. The 201 identities with emotion images were divided into six groups of up to 34 images, and each counterbalanced condition showed a different emotion for each of the six groups, such that no identity was shown more than once to each rater. Since not all identities had all six emotions, the number of images in each counterbalanced condition ranged from 179 to 193.


###### Emotion Intensity Ratings

We obtained ratings of how intensely faces expressed each intended emotion (‘How intensely is this person expressing anger [disgust, fear, happiness, sadness, surprise]?’) from 1 (*not at all)* to 7 (*very*). Here, raters only rated all faces showing one emotion expression (e.g., all angry faces) and rated the intensity only of the intended expression (e.g., angry faces only rated on anger intensity).

#### Procedure

We collected ratings via Experimentum [@debruine_experimentum_2020]; a structure file for the exact experimental setup is available on the OSF (<https://osf.io/7w5du/files/zc2b3>). After a brief introduction to the study and online informed consent, each participant (rater) was randomly allocated to one of the ratings (e.g., rating all 205 standardised neutral faces on how memorable they look). All available faces were displayed one at a time, in a randomised order for each rater. The question/prompt and response scale remained visible at the top of the screen, above the photo throughout the study. The study automatically progressed to the next trial once the rater responded by clicking on the response scale. There was no time limit to provide a response. We included seven attention checks embedded in the study, which directed raters to provide a specific response.

Following rating or categorising all stimuli, raters self-reported their gender, age, race/ethnicity, country of residence, and device type used for the study (desktop, laptop, tablet, mobile). They also completed an honesty/attention check question, asking them if they engaged with the study seriously, with assurance of payment regardless of response (choosing from ‘no, I was not really paying attention’ and ‘yes, I tried to give my authentic first impressions’). 


We recruited fluent English-speaking raters through Prolific <!--add prolific inclusion/exclusion criteria-->. We collected all data in May 2025^[This was slightly later than preregistered, due to pretesting and study setup delays. Note that we also planned to collect perceived race/ethnicity data, but decided against this, due to the regional specificity of race/ethnicity labels.].


#### Participants

We aimed to collect 100 raters per rating condition to achieve stable averages and allow for exclusions [@hehman_assessing_2025], totalling 2100 raters. Altogether, `r initial_raters` raters completed the study. See Results for exclusions and demographics of the final sample.

### Results

#### Data Cleaning and Exclusions

In the raters’ demographic questionnaire, we standardized participants’ recording of their race/ethnicity similarly to the models’ by recoding their inputs into one of seven categories (“White”, “Black”, “Asian”, “Indigenous/Pacific Islander”, “MENA” (Middle Eastern or North African), “Latine”, or “Mixed”), or as  “Ambiguous label” when this was not possible. Any non-entries were recoded as NA. 

A total of `r initial_raters` raters completed `r initial_completions` rating or categorisation tasks. We found that some raters did not complete all trials in tasks, some raters completed more than one task, and some raters completed more than the maximum number of trials in a task (likely by restarting the study or bypassing the back button block). Therefore, we removed incomplete tasks from our data, retained raters’ first complete tasks, and filtered our duplicate trials by keeping only raters’ first ratings for a duplicated trial. After these exclusions, and before implementing the pre-registered plan for data exclusions, we had complete and clean data from `r complete_raters` raters.
 
Our pre-registered plan for data exclusions included removing raters who gave overly consistent responses, committed overly fast responses, self-reported not taking the study seriously when asked whether or not they completed the study authentically, and failed attention checks. In total, we excluded `r excluded_raters` raters for our pre-registered reasons for data exclusions. @tbl-exclusions shows the number of raters we excluded for each of our reasons. 
 

```{r}
#| label: tbl-exclusions
#| tbl-cap: The number of raters excluded from analysis.
# tbl-subcap: Multiple raters met more than one exclusion criterion; therefore, the counts for individual criteria do not sum to the total number of raters excluded.

exclusions |> 
  summarise(
    `Overly consistent` = sum(!is.na(same_pcnt)),
    `Overly fast` = sum(!is.na(med_rt)),
    `Failed honesty check` = sum(!is.na(honesty_check)),
    `Failed attention checks` = sum(!is.na(attn_checks_passed)),
    `Total` = n()
  ) |> 
  pivot_longer(
    cols = everything(),
    names_to = "Reason for exclusion",
    values_to = "N"
  ) |>
  flextable() |>
  flextable::width(j = 1, 2)

```

We defined overly consistent responses as those raters who responded to at least 90% of trials identically. During agreement analyses (see Agreement Indicators for the Ratings), we found that `r additionally_excl` additional raters only gave two unique responses across all trials and also excluded these as overly consistent.  We defined overly fast responses as those raters whose median reaction time fell below the 1st percentile of the overall distribution of median reaction times (see Figure 1 for the distribution of median reaction times). For our attention checks, the threshold for inclusion was to accurately complete six or more attention checks (i.e., participants were excluded if they failed more than one of the checks). Lastly, participants were excluded based on a self-reported honesty check, i.e., if they reported not taking the study seriously rather than taking it authentically. After these exclusions, we had `r final_raters` raters in our sample (see @tbl-raters-per-task for the number of raters per task).

```{r}
#| label: fig-rt-distribution
#| fig-cap: Median reaction time distribution

med_rt |> 
  ggplot(aes(x = med_rt)) +
  geom_histogram(binwidth = 100, boundary = 0, fill = "dodgerblue") +
  geom_vline(xintercept = quantile(med_rt$med_rt, probs = 0.01), 
             color = "red", linetype = "dashed", linewidth = 1) +
  coord_cartesian(xlim = c(0, 8000)) +
  scale_x_continuous(expand = expansion(mult = 0)) +
  labs(x = "Median RT (ms)",
       y = "Count (per 100ms bin)")  +
  theme_bw()
```

 
```{r}
#| label: tbl-raters-per-task
#| tbl-cap: Number of raters per task. The emotion rating task was broken into 6 counterbalanced versions.

data_exp |>
  summarise(.by = exp,
            raters = unique(session_id) |> length()
            ) |>
  arrange(exp) |>
  flextable() |>
  set_header_labels(exp = "Rating Task",
                    raters = "N") |>
  flextable::width(j = 1, 2)
```

Anonymous data are available on Github (<https://github.com/ManyFacesTeam/pilot-ratings>).
 
#### Rater Demographics

```{r}
n_demog_provided <- data_quest$session_id |> unique() |> length()
```


We collected the following demographic data from our raters (N = `r n_demog_provided` provided data): Age, gender, residence, ethnicity, and devices on which the ratings were completed. See @fig-gender-age and Tables 3-5 for demographic information.

```{r}
#| label: fig-gender-age
#| fig-cap: Histogram of rater age, separated by gender

data_quest |> 
  ggplot(aes(x = age, fill = gender)) +
  geom_histogram(binwidth = 1) +
  scale_fill_manual(values = c("hotpink", "dodgerblue", "darkorchid", "gray")) +
  guides(fill  = guide_legend(position = "inside")) +
  theme(legend.position.inside = c(.7, .7)) +
  labs(x = "Age (years)", y = NULL)
```


```{r}
#| label: tbl-raters-country
#| tbl-cap: Rater country of residence

data_quest |> 
  count(residence) |>
  summarise(.by = n, res = paste(residence, collapse = ", ")) |>
  arrange(desc(n)) |>
  select(res, n) |>
  flextable() |>
  set_header_labels(res = "Country of Residence", n = "N") |>
  flextable::width(j = 1, 2)
```
 
```{r}
#| label: tbl-raters-eth
#| tbl-cap: Rater race/ethnicity

data_quest |> 
  count(ethnicity_rec) |>
  arrange(desc(n)) |>
  flextable() |>
  colformat_char(na_str = "Missing") |>
  set_header_labels(ethnicity_rec = "Ethnicity", n = "N") |>
  flextable::width(j = 1, 2)
```

```{r}
#| label: tbl-raters-dev
#| tbl-cap: Devices used by raters to complete the study

data_quest |> 
  count(device) |>
  arrange(desc(n)) |>
  flextable() |>
  colformat_char(na_str = "missing") |>
  set_header_labels(device = "Device Used", n = "N") |>
  flextable::width(j = 1, 2)
```


#### Agreement Indicators for the Ratings


##### Standardized Trait Ratings

```{r}
icc_table <- data_trait |> 
  group_by(exp) |> 
  group_map(calc_icc) |> 
  bind_rows() |>
  rename(Rating = experiment,
         N = n_raters,
         ICC = 3,
         lower = 4, 
         upper = 5)

s_n_min <- min(icc_table$N)
s_n_max <- max(icc_table$N)
```

In the first step, we calculated intraclass correlation coefficients [ICC(2,k)] for ratings of standardized images. The number of raters ranged from `r s_n_min` to `r s_n_max` for each trait. The average reliability across raters was excellent for ratings of attractiveness, dominance, trustworthiness, and gender-typicality, but poorer for ratings of memorability (see @tbl-std-neu-icc).

```{r}
#| label: tbl-std-neu-icc
#| tbl-cap: Intraclass correlation coefficients for ratings of standardized neutral faces

icc_table |>
  flextable() |>
  colformat_double(digits = 2) |>
  flextable::width(j = 1, 2) |>
  add_header_row(values = c("", "95% CI"), colwidths = c(3, 2)) |>
  align(i = 1, align = "center", part = "header")
```

 
Next, we examined the number of raters required for trait ratings to reach stable levels of reliability defined as ICC(2,k) values of 0.75 (see Figure 3). Attractiveness ratings stabilized with the fewest participants, reaching a median ICC(2,k) of 0.75 at approximately `r n_075[["attractive"]]` raters. It was also the only trait to exceed a median ICC(2,k) of 0.90, which occurred with `r n_090[["attractive"]]` raters. Ratings for dominance, trustworthiness, and gender-typicality reached a median ICC(2,k) of 0.75 with `r n_075[["dominant"]]` raters (see @fig-icc-cos). 

```{r}
#| label: fig-icc-cos
#| fig-cap: Corridors of stability for ICC
#| fig-subcap: Dashed horizontal lines indicate ICC(2,k) of 0.75 and 0.90, respectively.

icc2k_corridor_summary |> 
  ggplot(aes(x = n_raters_sampled)) +
  geom_ribbon(aes(ymin = ICC2_k_low, ymax = ICC2_k_high), 
              alpha = 0.15, fill = "grey60") +
  # Median ICC(2,k)
  geom_line(aes(y = ICC2_k_median), linewidth = 0.8, colour = "black") +
  # Thresholds: 0.75 grey, 0.9 black
  geom_hline(yintercept = 0.75, linetype = "dashed", colour = "grey40") +
  geom_hline(yintercept = 0.9, linetype = "dashed", colour = "black") +
  geom_vline(data = thresholds_median_075 |> filter(!is.na(N_for_075_median)),
             aes(xintercept = N_for_075_median), colour = "red", linetype = "dotted") +
  geom_text(data = thresholds_median_075 |> filter(!is.na(N_for_075_median)),
            aes(x = N_for_075_median, y = 0.60, label = paste0("N=", N_for_075_median)),
            angle = 0, hjust = -0.2, vjust = 0,, colour = "red", size = 3) +
  facet_wrap(~ experiment) +
  scale_y_continuous(limits = c(0, 1)) +
  labs(
    x = "Number of raters sampled", 
    y = "ICC(2,k)") +
  scale_x_continuous(breaks = seq(10, 100, 10), labels = seq(10, 100, 10)) +
  labs(x = "Number of raters sampled", y = "ICC(2,k)") +
  theme_minimal(base_size = 12) +
  theme(
    strip.text = element_text(size = 12, face = "bold"),
    axis.text = element_text(size = 11),
    axis.title = element_text(size = 12),
    plot.subtitle = element_text(size = 11)
  )
```

```{r}
#| warning: false
#| message: false
alpha_omega_table <- data_trait |> 
  group_by(exp) |> 
  group_map(purrr::partial(calc_alpha_omega)) |> 
  bind_rows() |>
  rename(Rating = experiment,
         N = n_raters,
         alpha = alpha,
         omega = omega_t)

mem <- filter(alpha_omega_table, Rating == "memorable")
a_mem <- mem$alpha |> format(digits = 2)
o_mem <- mem$omega |> format(digits = 2)
``` 

To further check inter-rater agreement, we used both Cronbach’s alpha (α) and McDonald’s omega total (ωt; see @tbl-std-neu-alpha-omega). For four of the five traits, both α and ωt exceeded 0.90, indicating excellent inter-rater agreement. For the fifth trait (memorability), α was lower (`r a_mem`), suggesting reduced inter-rater agreement. ωt for memorability was substantially higher (`r o_mem`), suggesting better agreement under the assumption that raters varied in how consistently they judged memorability, a pattern consistent with the lower ICC observed for this trait.

```{r}
#| label: tbl-std-neu-alpha-omega
#| tbl-cap: Cronbach’s alpha and McDonald’s omega total for ratings of standardized neutral faces.

alpha_omega_table |>
  flextable() |>
  colformat_double(digits = 2) |>
  flextable::width(j = 1, 2)
```
 
##### Unstandardized Trait Ratings

```{r}
#| warning: false
#| message: false

icc_table_unstd <- data_trait_unstd |> 
  group_by(exp) |> 
  group_map(calc_icc, check_dropped_raters = FALSE) |> 
  bind_rows() |>
  rename(Rating = experiment,
         N = n_raters,
         ICC = 3,
         lower = 4, 
         upper = 5)

u_n_min <- min(icc_table_unstd$N)
u_n_max <- max(icc_table_unstd$N)

alpha_omega_table_unstd <- data_trait_unstd |> 
  group_by(exp) |> 
  group_map(calc_alpha_omega) |> 
  bind_rows() |>
  rename(Rating = experiment,
         N = n_raters,
         alpha = alpha,
         omega = omega_t)

u_a_min <- min(alpha_omega_table_unstd$alpha) |> format(digits = 2)
u_a_max <- max(alpha_omega_table_unstd$alpha) |> format(digits = 2)
u_o_min <- min(alpha_omega_table_unstd$omega) |> format(digits = 2)
u_o_max <- max(alpha_omega_table_unstd$omega) |> format(digits = 2)
``` 

As with standardized trait ratings, we calculated intraclass correlation coefficients for ratings of unstandardized images. The number of raters ranged from  `r u_n_min` to `r u_n_max`. Average inter-rater reliability was excellent for all three traits (see @tbl-unstd-neu-icc). This was further supported by Cronbach's α ranging from `r u_a_min` to `r u_a_max` and McDonald's ωt ranging from `r u_o_min` to `r u_o_max` (see @tbl-unstd-neu-alpha-omega).


```{r}
#| label: tbl-unstd-neu-icc
#| tbl-cap: Agreement for ratings of unstandardized neutral faces

icc_table_unstd |>
  flextable() |>
  colformat_double(digits = 2) |>
  flextable::width(j = 1, 2) |>
  add_header_row(values = c("", "95% CI"), colwidths = c(3, 2)) |>
  align(i = 1, align = "center", part = "header")
```
 

```{r}
#| label: tbl-unstd-neu-alpha-omega
#| tbl-cap: Cronbach’s alpha and McDonald’s omega total for ratings of unstandardized neutral faces.

alpha_omega_table_unstd |>
  flextable() |>
  colformat_double(digits = 2) |>
  flextable::width(j = 1, 2)
```


##### Emotion Intensity Ratings

```{r}
#| warning: false
#| message: false
icc_table_emo <- data_emo_int |> 
  group_by(exp) |> 
  group_map(calc_icc) |> 
  bind_rows() |>
  rename(Rating = experiment,
         N = n_raters,
         ICC = 3,
         lower = 4, 
         upper = 5)

emo_n_min <- min(icc_table_emo$N)
emo_n_max <- max(icc_table_emo$N)

emo_icc_min <- min(icc_table_emo$ICC) |> format(digits = 2)
emo_icc_max <- max(icc_table_emo$ICC) |> format(digits = 2)

alpha_omega_table_emo <- data_emo_int |> 
  group_by(exp) |> 
  group_map(calc_alpha_omega) |> 
  bind_rows() |>
  rename(Rating = experiment,
         N = n_raters,
         alpha = alpha,
         omega = omega_t)

emo_a_min <- min(alpha_omega_table_emo$alpha) |> format(digits = 2)
emo_a_max <- max(alpha_omega_table_emo$alpha) |> format(digits = 2)
emo_o_min <- min(alpha_omega_table_emo$omega) |> format(digits = 2)
emo_o_max <- max(alpha_omega_table_emo$omega) |> format(digits = 2)
```

Next, we calculated intraclass correlation coefficients, Cronbach's α, and McDonald's ωt for emotion intensity ratings. The number of raters ranged from `r emo_n_min` to `r emo_n_max`. The average reliability across raters was excellent for all emotions, with ICC(2,k) ranging from `r emo_icc_min` to `r emo_icc_max`, Cronbach’s α ranging from `r emo_a_min` to `r `, and  McDonald's ωt ranging from `r emo_o_min` to `r emo_o_max` (see @tbl-emo-icc and @tbl-emo-alpha-omega).


```{r}
#| label: tbl-emo-icc
#| tbl-cap: Intraclass correlation coefficients for ratings of emotional faces’ emotion intensity

icc_table_emo |>
  flextable() |>
  colformat_double(digits = 2) |>
  flextable::width(j = 1, 2) |>
  add_header_row(values = c("", "95% CI"), colwidths = c(3, 2)) |>
  align(i = 1, align = "center", part = "header")
```

```{r}
#| label: tbl-emo-alpha-omega
#| tbl-cap: Cronbach’s alpha and McDonald’s omega total for ratings of emotional faces’ emotion intensity

alpha_omega_table_emo |>
  flextable() |>
  colformat_double(digits = 2) |>
  flextable::width(j = 1, 2)
```
 
#### Points of Stability

To determine the number of raters required for stable ratings, we computed the point of stability [POS; @hehman_assessing_2025], defined as the smallest sample size at which 95% of resampled means fall within a corridor of stability of ±0.5 points and do not again exceed it. As shown in Figures 4-6, the ratings of standardized images reached stability between 37 and 44 raters, the ratings of unstandardized images between 37 and 41 raters, and emotion intensity ratings of standardized emotional faces between 29 and 50 raters.

```{r}
#| label: fig-std-neu-pos
#| fig-cap: Points of stability for ratings of standardised neutral faces

stability_stats_traits$pos$label <- paste0("Point of stability\n(N = ", stability_stats_traits$pos$pos, ")")

summary_ci_traits |> 
  mutate(exp = factor(exp, exp_levels_neutral)) |>
  ggplot(aes(x = sample_size)) +
  geom_ribbon(aes(ymin = ll, ymax = ul, fill = exp),
              alpha = 0.5) +
  geom_vline(data = stability_stats_traits$pos,
             aes(xintercept = pos),
             colour = "black",
             linetype = "solid",
             linewidth = .75) +
  geom_hline(yintercept = c(-0.5, 0.5),
             linetype = "dashed",
             colour = "grey30",
             linewidth = 0.5) +
  facet_wrap(~ exp) +
  xlab("Sample size (N)") +
  ylab("Centered mean ratings") +
  scale_fill_manual(values = rainbow) +
  theme_minimal() +
  theme(panel.grid.major = element_line(colour = "#e8e8e8"),
        panel.grid.minor = element_line(colour = "#e8e8e8"),
        strip.text = element_text(size = 12, face = "bold"),
        axis.text = element_text(size = 10),
        axis.title = element_text(size = 12, face = "bold"),
        legend.position = "none") +
  geom_label(data = stability_stats_traits$pos,
             aes(x = pos + 30, y = 1.6, label = label),
             fill = NA,   
             linewidth = 0,
             colour = "black",    
             size = 3.5,
             inherit.aes = FALSE)
```


```{r}
#| label: fig-unstd-neu-pos
#| fig-cap: Points of stability for ratings of unstandardised neutral faces

stability_stats_traits_unstd$pos$label <- paste0("Point of stability\n(N = ", stability_stats_traits_unstd$pos$pos, ")")

summary_ci_traits_unstd |> 
  mutate(exp = factor(exp, exp_levels_neutral_unstd)) |>
  ggplot(aes(x = sample_size)) +
  geom_ribbon(aes(ymin = ll, ymax = ul, fill = exp),
              alpha = 0.5) +
  geom_vline(data = stability_stats_traits_unstd$pos,
             aes(xintercept = pos),
             colour = "black",
             linetype = "solid",
             linewidth = .75) +
  geom_hline(yintercept = c(-0.5, 0.5),
             linetype = "dashed",
             colour = "grey30",
             linewidth = 0.5) +
  facet_wrap(~ exp) +
  xlab("Sample size (N)") +
  ylab("Centered mean ratings") +
  scale_fill_manual(values = rainbow) +
  theme_minimal() +
  theme(panel.grid.major = element_line(colour = "#e8e8e8"),
        panel.grid.minor = element_line(colour = "#e8e8e8"),
        strip.text = element_text(size = 12, face = "bold"),
        axis.text = element_text(size = 10),
        axis.title = element_text(size = 12, face = "bold"),
        legend.position = "none") +
  geom_label(data = stability_stats_traits_unstd$pos,
             aes(x = pos + 30, y = 1.6, label = label),
             fill = NA,   
             linewidth = 0,
             colour = "black",    
             size = 3.5,
             inherit.aes = FALSE)
```

```{r}
#| label: fig-emo-pos
#| fig-cap: Points of stability for emotion intensity ratings of standardised emotional faces of each expression

stability_stats_emo_int$pos$label <- paste0("Point of stability\n(N = ", stability_stats_emo_int$pos$pos, ")")

summary_ci_emo_int |> 
  ggplot(aes(x = sample_size)) +
  geom_ribbon(aes(ymin = ll, ymax = ul, fill = exp),
              alpha = 0.5) +
  geom_vline(data = stability_stats_emo_int$pos,
             aes(xintercept = pos),
             colour = "black",
             linetype = "solid",
             linewidth = .75) +
  geom_hline(yintercept = c(-0.5, 0.5),
             linetype = "dashed",
             colour = "grey30",
             linewidth = 0.5) +
  facet_wrap(~ exp) +
  xlab("Sample size (N)") +
  ylab("Centered mean ratings") +
  scale_fill_manual(values = rainbow) +
  theme_minimal() +
  theme(panel.grid.major = element_line(colour = "#e8e8e8"),
        panel.grid.minor = element_line(colour = "#e8e8e8"),
        strip.text = element_text(size = 12, face = "bold"),
        axis.text = element_text(size = 10),
        axis.title = element_text(size = 12, face = "bold"),
        legend.position = "none") +
  geom_label(data = stability_stats_emo_int$pos,
             aes(x = pos + 30, y = 1.6, label = label),
             fill = NA,   
             linewidth = 0,
             colour = "black",    
             size = 3.5,
             inherit.aes = FALSE)
```

 
#### Descriptive Statistics

In this section, we report the descriptive statistics for ratings of the neutral standardized images, the neutral unstandardized images, the emotion categorization task, the intensity of expressed emotions, and the ages of the models.

##### Standardized Neutral Trait Ratings

Trustworthiness, dominance, and memorability ratings were approximately normally distributed and showed similar patterns of central tendency and variance. In contrast, attractiveness ratings were somewhat right-skewed, reflected in a relatively lower mean, while gender-typicality ratings were left-skewed, with a relatively higher mean (see @tbl-std-neu-desc for descriptive statistics and @fig-std-neu-hist for rating distributions).

```{r}
std_neutral <- data_exp |>
  filter(exp %in% exp_levels_neutral) |>
  mutate(dv = as.integer(dv),
         exp = factor(exp, exp_levels_neutral)) |> 
  separate_wider_delim(trial_name,
                       delim = "_",
                       names = c("lab_id", "model_id"),
                       too_many = "drop")
```

```{r}
#| label: tbl-std-neu-desc
#| tbl-cap: Descriptive statistics for ratings of standardized neutral faces

std_neutral |>
  summarise(N = n(),
            mean_rating = mean(dv),
            .by = c("exp", "lab_id", "model_id")) |> 
  summarise(N = first(N),
            M = mean(mean_rating),
            SD = sd(mean_rating),
            .by = "exp") |>
  rename(Rating = exp) |>
  flextable() |>
  colformat_double(digits = 2) |>
  flextable::width(j = 1, 2)
```

```{r}
#| label: fig-std-neu-hist
#| fig-cap: Histograms for ratings of standardized neutral faces

std_neutral |>
  ggplot(aes(x = dv, fill = exp)) +
  geom_histogram(binwidth = 1, show.legend = FALSE, color = "black") +
  facet_wrap(~exp, ncol = 3, axes = "all_x", drop = FALSE) +
  labs(x = NULL, y = NULL) +
  scale_x_continuous(breaks = 1:7) +
  scale_y_continuous(breaks = seq(0, 5000, 1000), labels = paste0(0:5, "K")) +
  scale_fill_manual(values = rainbow, drop = FALSE)
```




##### Unstandardized Neutral Trait Ratings

The ratings for dominance were approximately normally distributed, whereas the ratings for attractiveness were slightly skewed right and the ratings for trustworthiness were slightly skewed left, each reflected in their relatively low and high mean ratings (see @tbl-unstd-neu-desc for descriptive statistics and @fig-unstd-neu-hist for rating distributions).

```{r}
unstd_neutral <- data_exp |>
  filter(exp %in% exp_levels_neutral_unstd) |>
  mutate(dv = as.integer(dv),
         exp = factor(exp, exp_levels_neutral_unstd)) |> 
  separate_wider_delim(trial_name,
                       delim = "_",
                       names = c("lab_id", "model_id"),
                       too_many = "drop")
```

```{r}
#| label: tbl-unstd-neu-desc
#| tbl-cap: Descriptive statistics for ratings of unstandardized neutral faces

unstd_neutral |>
  summarise(N = n(),
            mean_rating = mean(dv),
            .by = c("exp", "lab_id", "model_id")) |> 
  summarise(N = first(N),
            M = mean(mean_rating),
            SD = sd(mean_rating),
            .by = "exp") |>
  rename(Rating = exp) |>
  flextable() |>
  colformat_double(digits = 2) |>
  flextable::width(j = 1, 2)
```

```{r}
#| label: fig-unstd-neu-hist
#| fig-cap: Histograms for ratings of unstandardized neutral faces
#| fig-height: 2.5
#| fig-width: 8

unstd_neutral |>
  ggplot(aes(x = dv, fill = exp)) +
  geom_histogram(binwidth = 1, show.legend = FALSE, color = "black") +
  facet_wrap(~exp, ncol = 5, drop = FALSE) +
  labs(x = NULL, y = NULL) +
  scale_x_continuous(breaks = 1:7) +
  scale_y_continuous(breaks = seq(0, 5000, 1000), labels = paste0(0:5, "K")) +
  scale_fill_manual(values = rainbow, drop = FALSE)
```


 
##### Emotion Categorization and Intensity Ratings

Next, we explored raters’ emotion categorizations and perceived emotion intensity expressed by our models. Raters’ categorizations generally aligned with the models’ expression of emotion, with the greatest alignment was observed for expression of happiness, and the least for fear (@tbl-emo-desc, @fig-emo-freq). 

```{r}
emo_cat <- data_exp |>
  filter(exp %in% exp_levels_emo_cat) |>
  mutate(exp = factor(exp, exp_levels_emo_cat),
         dv = factor(dv, levels = c("anger", "disgust", "fear", 
                                    "happiness", "sadness", "surprise",
                                    "other"))) |> 
  separate_wider_delim(trial_name,
                       delim = "_",
                       names = c("lab_id", "model_id", NA, "model_expression", NA),
                       too_many = "drop") |> 
  mutate(model_expression = forcats::fct_recode(
    model_expression,
    anger = "ang",
    disgust = "dis",
    fear = "fea",
    happiness = "hap",
    sadness = "sad",
    surprise = "sur"
  ))
```


```{r}
#| label: tbl-emo-desc
#| tbl-cap: Emotion categorisation proportions
# tbl-subcap: Note. Italicised values indicate correct categorisations (categorisations matching intended model expression).

emo_cat |> 
  group_by(model_expression) |> 
  count(model_expression, dv) |> 
  mutate(prop = n / sum(n)) |>
  ungroup() |> 
  dplyr::select(-n) |> 
  pivot_wider(names_from = dv,
              values_from = prop) |> 
  relocate(other, .after = surprise) |>
  rename("Actual Model Expression" = model_expression) |>
  flextable() |>
  colformat_double(digits = 2) |>
  flextable::width(j = 1, 2) |>
  add_header_row(values = c("", "Rated Expression"), colwidths = c(1, 7)) |>
  align(i = 1, align = "center", part = "header")
```


```{r}
#| label: fig-emo-freq
#| fig-cap: Emotion categorisation frequencies
#| fig-subcap: Downward arrows indicate correct categorisations (categorisations matching intended model expression). A=anger, D=disgust, F=fear, H=happiness, S=sadness, U=surprise, O=other

emo_levels <- levels(emo_cat$model_expression)
emo_labels <- paste(emo_levels, "faces")

emo_cat |>
  mutate(emo = factor(model_expression, emo_levels, emo_labels)) |>
  ggplot(aes(x = dv, fill = dv)) +
  geom_point(aes(x = x, colour = I(fill), fill = I(fill)), 
             data.frame(emo = factor(emo_levels, emo_levels, emo_labels),
                        x = 1:6,
                        fill = rainbow),
             size = 6.5, y = -60, shape = 18, show.legend = FALSE) +
  geom_bar(color = "transparent") +
  facet_wrap(~emo, axes = "all_x", drop = FALSE) +
  scale_x_discrete(labels = c("A", "D", "F", "H", "S", "U", "O")) +
  scale_y_continuous(breaks = seq(0, 15000, 5000), labels = paste0(c(0, 5, 10, 15), "K")) +
  scale_fill_manual(values = c(rainbow, "grey"), drop = FALSE) +
  labs(x = NULL, y = NULL,
       fill = "Categorised as") +
  coord_cartesian(clip = "off") +
  theme(axis.ticks.x = element_blank())
```


Next, we calculated the average ratings of emotion intensity for each emotion (@tbl-intensity-desc). These ratings were very similar, on average, between the different emotions except for happiness, which was rated with greater intensity than all other emotions. On average, these ratings were located around the middle of the 7-point scale. @fig-intensity-hist shows the distribution of ratings on the scale for each emotion.

```{r}
emo_int <- data_exp |>
  filter(exp %in% exp_levels_emo_int) |>
  mutate(dv = as.integer(dv),
         exp = factor(exp, exp_levels_emo_int)) |> 
  separate_wider_delim(trial_name,
                       delim = "_",
                       names = c("lab_id", "model_id"),
                       too_many = "drop")
```

```{r}
#| label: tbl-intensity-desc
#| tbl-cap: Descriptive statistics for emotion intensity ratings

emo_int |>
  summarise(N = n(),
            mean_rating = mean(dv),
            .by = c("exp", "lab_id", "model_id")) |> 
  summarise(N = first(N),
            M = mean(mean_rating),
            SD = sd(mean_rating),
            .by = "exp") |>
  arrange(exp) |>
  rename(Rating = exp) |>
  flextable() |>
  colformat_double(digits = 2) |>
  flextable::width(j = 1, 2)
```

```{r}
#| label: fig-intensity-hist
#| fig-cap: Histograms of emotion intensity ratings

emo_int |>
  ggplot(aes(x = dv, fill = exp)) +
  geom_histogram(binwidth = 1, show.legend = FALSE, color = "black") +
  facet_wrap(~exp, ncol = 3, axes = "all_x", drop = FALSE) +
  scale_y_continuous(breaks = seq(0, 5000, 1000), labels = paste0(0:5, "K")) +
  labs(x = NULL, y = NULL) +
  scale_fill_manual(values = rainbow, drop = FALSE) +
  scale_x_continuous(breaks = 1:7)
```


 
##### Perceived Age

The distribution of raters’ perceptions of models’ ages is shown in @fig-age-hist. 26-30 years was the most-chosen age across raters and models. The correlation between models’ mode perceived age and actual age appears in @fig-age-corr.

```{r}
dv_levels <- seq(20, 85, 5)
dv_labels <- paste(dv_levels-4, "-", dv_levels )
dv_labels[14] <- "81+"

age <- data_exp |>
  filter(exp == "age") |>
  mutate(dv = factor(dv, dv_levels, dv_labels)) |> 
  separate_wider_delim(trial_name,
                       delim = "_",
                       names = c("lab_id", "model_id"),
                       too_many = "drop")
```

```{r}
#| label: fig-age-hist
#| fig-cap: Histogram of perceived age
#| fig-width: 10
#| fig-height: 5

age |>
  ggplot(aes(x = dv)) +
  geom_bar(color = "black", fill = "gray") +
  scale_x_discrete(drop = FALSE) +
  scale_y_continuous(breaks = seq(0, 5000, 1000), labels = paste0(0:5, "K")) +
  labs(x = "Age Categories (in years)", y = NULL)
```


```{r}
# Save modal age for each face
age_modes <- age |>
  group_by(lab_id, model_id) |>
  summarise(out = list(get_mode_info(dv)),
            .groups = "drop") |>
  unnest_wider(out)

age_perc <- age_modes |> 
  mutate(age_mid = age_mode) |> 
  separate(age_mid, into = c("age_start", "age_end"), sep = "-", convert = TRUE) |> 
  mutate(age_perc = (age_start + age_end) / 2) |> 
  select(-age_start, -age_end)

age_by_face <- data_models |> 
  dplyr::select(lab_id, model_id, age) |> 
  rename(age_actual = age) |> 
  left_join(age_perc, by = c("lab_id", "model_id"))

# Get stats
age_by_face_cortest <- cor.test(age_by_face$age_actual, age_by_face$age_perc)
age_by_face_r <- age_by_face_cortest$estimate
age_by_face_p <- age_by_face_cortest$p.value
p_comp = "="
if (age_by_face_p < .001) {
  age_by_face_p = .001
  p_comp <- "<"
}
```

```{r}
#| label: fig-age-corr
#| fig-cap: Correlation between models’ actual age and mode perceived age

label_text <- sprintf("r = %.2f, p %s %.3f", 
                      age_by_face_r, p_comp, age_by_face_p)

age_by_face |> 
  filter(!is.na(age_actual) & !is.na(age_perc)) |>
  #count(age_actual, age_perc) |>
  ggplot(aes(x = age_actual, y = age_perc)) +
  geom_point(alpha = 0.25) +
  geom_smooth(method = "lm",  formula = y ~ x, se = TRUE, colour = "black") +
  annotate("text", x = 65, y = 20, label = label_text, hjust = 0) +
  labs(x = "Actual Age", y = "Perceived Age [Mode]") +
  theme_bw()
```
 

## Discussion

### The value of the current work

This database provides a uniquely diverse face image stimulus set in terms of model nationality and image variety (appearance standardization, viewing angle, and facial expression). The diversity of images within the database makes it potentially useful to address a variety of research questions in multiple subfields of psychology (e.g., face recognition, emotion). It accomplishes this while being transparent and reproducible via the openly available protocol. Additionally, the database need not remain static: Other researchers can follow the protocol and add to the database, further increasing its sample size and diversity. This protocol can also serve as a template for researchers interested in collecting stimulus images, in terms of highlighting the kinds of details that need to be considered and documented)

The data associated with the images are also valuable. First, we collected models' self-reported demographic information, which is useful for various kinds of research questions. The validation/norming data we collected also provide a useful starting point for researchers interested in using the images, for example enabling researchers to choose subsets of images most suited to address their research questions. These data also provide information about the minimum number of raters needed for reliable mean ratings for different judgments [see @hehman_assessing_2025]. 

This work also provides proof of concept that various labs, all following the same protocol and using the same equipment, can take comparable images to form a coherent face database. This broadens opportunities for future image database collection: A database does not need to all be collected by one lab in one location.

### Reflections on the Process & Limitations

The leadership team found that having a broad variety of areas of expertise and diverse research backgrounds within the broader team was invaluable in both developing the database and in troubleshooting issues. We moreover found that setting out expectations at the start of the project through a collaboration agreement was essential. However, we ran into various issues throughout the process, which are common to Big Team Science initiatives. For example, with so many labs and individual researchers involved, timelines for completion necessarily stretched. This made momentum difficult to maintain at times. Planning in generous buffer time for such large-scale projects and managing team members' expectations around timelines is therefore essential. We also faced difficulty in finding an optimal way to communicate effectively with all team members, given varying preferences (e.g., via email vs. other team communication platforms). It is perhaps worth surveying members of a big team at project start about communication possibilities, as well as outlining more specific communication expectations in a collaboration agreement. 

After Phase 1, we surveyed the ManyFaces research team to provide feedback on the process of collecting images using the protocol. Team members commonly expressed the desire for additional concise guidance in following the protocol, including an additional abbreviated protocol with numbered steps to follow during data collection and a video tutorial (in addition to the provided illustrative images) showing the necessary steps for researchers unfamiliar with camera equipment. These are requests that we can incorporate into future protocol updates. Doing so could both make the process easier for researchers and minimize errors and deviations from the protocol, ensuring greater consistency between sites. 

The major issue raised by the research team was the difficulty of eliciting facial expressions as described in the protocol. The protocol took a straightforward instructive approach to emotion elicitation (e.g., asking participants to face the camera with the expression they would make if they were feeling happy), following consultation with emotion experts in ManyFaces and discussion of feasibility with the wider team. However, models self-reported difficulty with the facial expression posing, in line with researchers’ feedback. The emotion validation data also indicate that models struggled to pose the facial expressions of emotion. The data furthermore suggest that perceivers had difficulty identifying emotions from these posed facial expressions. In support of this idea, recent research shows posed expressions to not appear genuine and reveals differing perceptions of posed and spontaneous expressions [@dawel_perceived_2017; @dawel_faking_2025]. Future work, including possible updates to our own protocol, may therefore focus on developing a better framework to elicit facial expressions. This could include taking inspiration from databases of naturally-induced emotion [e.g., @miolla_padova_2023; @sneddon_belfast_2012] and considering potential cultural differences and specific local needs.

The research team also raised several comments about the provided and not-provided equipment, specifically the flimsiness of the light stand and the missing standardized backdrop (due to shipping constraints, participating researchers were to source a white background, e.g., a wall, seamless fabric, or paper). It is worth highlighting that the images were captured with equipment selected due to several practical constraints: budget, university-approved vendors, shipping logistics, interoperability, and ease of use for non-experts in varying conditions. The available project budget and administrative restrictions on approved vendors (their offerings and stock) for the University of Glasgow primarily limited the attainable camera, lens, and lighting setup. In general, we opted to create a setup that would not be prohibitively expensive to acquire, could be shipped in a single parcel, and would be compatible (substitutable) with equipment other researchers may have available. The last consideration was creating a setup with the fewest degrees of freedom, and thus less room for error. Therefore, we opted to use an LED ring light with a camera mounted inside the ring in landscape orientation on a single stand (vs. portrait orientation using a different kind of mount or separate stands for the camera and light). Ring lights became ubiquitous and easy to acquire in recent years; they only need to be positioned square, level, and in front of the sitter. Other, more complex setups can be created; however, the price, mobility, and repeatability of such setups may represent major constraints to consistent and reliable image collection. It is therefore worth considering trade-offs between different kinds of equipment and setups in future work.

In addition to our choices of equipment, we also made certain pragmatic choices during image processing that future work could improve on. For example, we white-balanced the images rather than fully colour-correcting them, as we were able to create a fully reproducible scripted method for the former process, but not the latter. This was due to limitations such as varied colour checker chart placement and orientation in models’ photos and a lack of expertise on the research team to work around this in a reproducible way. We therefore opted to simply white-balance to keep the process entirely open and reproducible, rather than fully colour-correct using a manual and not fully reproducible method. The process of aligning and sizing the images was also driven by cropping needs. That is, due to the landscape orientation of the photos, there was limited vertical space that we could crop, constraining the possibilities for face alignment and image size. Future work could consider using alternative equipment setups to enable capture images in portrait mode. Variation in faces’ size in the images, due to some deviations from the distances specified in the protocol, also affected the image processing steps. This could be addressed in future work by more clearly highlighting not only the key aspects of the protocol that should be kept constant, but also why these aspects should be constant (i.e., clarifying the reasoning to all team members). 

Finally, we collected images of 211 models, but no single image type included images of all models. Rather, the maximum number of models represented in a single image type was 205 images (standardized neutral, front-facing). This led to missing perceived age/ethnicity data for six models who had unstandardized but not standardized neutral images available.

### Conclusion

Here, we introduce a new diverse face image database, which will be useful to researchers interested in a variety of questions related to social perception, person recognition, and vision science. We demonstrate that a cohesive database can be compiled across a variety of sites, opening doors for future additions to this database and the development of future multi-lab databases.

