Updating data processing code to add demographic data, post game surv…

…ey, playername-to-station mapping (#572)

This PR:

- Adds code to populate the DB with demographic data, postgame survey data, and a playername-to-station mapping.
- Adds *.egg-info to .gitignore, removes accidentally added .egg-info files from previous commit.
- Updates the inspect file creation script to point to the new location of the sqlite db
- Adds code for sentence labeling

---------

Co-authored-by: Paulo Soares <[email protected]>

.gitignore

@@ -109,3 +109,5 @@ HSR*
 
 # Texpad
 *.texpadtmp/
+
+*.egg-info

human_experiments/datasette_interface/README.md

@@ -14,22 +14,22 @@ The following `make` commands can be used to perform different operations on the
 
 # Populating the ToMCAT database
 
-The tomcat dataset is in a Postgres cluster that lives in `/space/paulosoares/postgres`. A series of `make` commands can be used to perform different operations on this dataset. The commands run by default in `development` mode, which uses an sqlite database created under `/space/USER/tomcat/.dev`. To modify the Postgres database, the commands need to be executed in `production` mode, which can be done by adding the following before each command:
+The tomcat dataset is in a Postgres cluster that lives in `/space/paulosoares/postgres`. A series of `make` commands can be used to perform different operations on this dataset. The commands run by default in `development` mode, which uses an sqlite database created under `/space/$USER/tomcat/.dev`. To modify the Postgres database, the commands need to be executed in `production` mode, which can be done by adding the following before each command:
 ```
 working_env=production db_pass=<user_postgres_pass>
 ```
-Your user must have permission to modify the Postgres database in production mode. 
+Your user must have permission to modify the Postgres database in production mode.
 
 ## Commands
 1. **clear_db**: [WARNING] drops the tables in the ToMCAT database. It must only be called if one really wants a fresh copy.
 2. **create_tables**: creates new tables and indices. If a table already exists, it won't be changed.
 3. **update_raw**: Adds new raw data to the relevant tables. This can be called to update the database with new experiment data. It will skip experiments already processed.
 4. **sync_raw**: Filters and synchronizes fNIRS and EEG signals with a main clock with frequency 200Hz that starts 1 minute before the rest_state task and ends 1 minute after end of the last minecraft trial. Synchronized signals are saved to the `fnirs_sync` and `eeg_sync` tables.
-5. **to_sqlite**: Copies the Postgres database to an SQLite database for publication in the datasette interface. It runs in production mode automatically to make sure to read from the Postgres database. IThe environment variable `TBS` can be used in conjunction with this command to specify a subset of tables one wants to copy, otherwise, all tables will be copied. Be patient and run this in a tmux session as this process can take several days depending on the size of the tables. 
+5. **to_sqlite**: Copies the Postgres database to an SQLite database for publication in the datasette interface. It runs in production mode automatically to make sure to read from the Postgres database. The environment variable `TBS` can be used in conjunction with this command to specify a subset of tables one wants to copy, otherwise, all tables will be copied. Be patient and run this in a tmux session as this process can take several days depending on the size of the tables.
 
 # Miscellaneous
 The following `make` commands can be used to perform different operations.
 
 1. **update_inspect_file**: Updates the `inspect-data.json` with the newest sqlite ToMCAT database.
 2. **generate_diagram**: Generates the database diagram from the newest sqlite ToMCAT database.
-3. **screenshots_to_server**: Copies screenshot images to the web server for access though a public URL.
+3. **screenshots_to_server**: Copies screenshot images to the web server for access though a public URL.

human_experiments/datasette_interface/bin/create_tables.py

@@ -6,6 +6,7 @@
 from datasette_interface.database.entity.base.group_session import GroupSession  # noqa f401
 from datasette_interface.database.entity.base.modality import Modality  # noqa f401
 from datasette_interface.database.entity.base.participant import Participant  # noqa f401
+from datasette_interface.database.entity.base.post_game_survey import PostGameSurvey  # noqa f401
 from datasette_interface.database.entity.base.station import Station  # noqa f401
 from datasette_interface.database.entity.base.task import Task  # noqa f401
 from datasette_interface.database.entity.derived.eeg_sync import EEGSync  # noqa f401

human_experiments/datasette_interface/bin/update_inspect_file

@@ -1,4 +1,4 @@
 #!/usr/bin/env bash
 
 # Update inspect-data.json
-time datasette inspect /space/paulosoares/tomcat/tomcat.db > inspect-data.json
+time datasette inspect /space/$USER/tomcat/.dev/tomcat.db > inspect-data.json

human_experiments/datasette_interface/datasette_interface/common/config.py

@@ -28,6 +28,28 @@ class Settings(BaseSettings):
         "/tomcat/data/raw/LangLab/experiments/study_3_pilot/"
         "station_to_eeg_amp_mapping.csv"
     )
+    station_to_minecraft_playername_mapping_path: str = (
+        "/tomcat/data/raw/LangLab/experiments/study_3_pilot/"
+        "station_to_minecraft_playername_mapping.csv"
+    )
+
+    self_report_data_path: str = (
+        "/tomcat/data/raw/LangLab/experiments/study_3_pilot/"
+        "ToMCATSelfReport_DATA_2023-06-11_2358.tsv"
+    )
+    self_report_data_dictionary_path: str = (
+        "/tomcat/data/raw/LangLab/experiments/study_3_pilot/"
+        "ToMCATSelfReport_DataDictionary_2023-08-08.tsv"
+    )
+
+    post_game_survey_data_path: str = (
+        "/tomcat/data/raw/LangLab/experiments/study_3_pilot/"
+        "ToMCATPostGameSurvey_DATA_2024-07-16_2050.csv"
+    )
+    post_game_survey_data_dictionary_path: str = (
+        "/tomcat/data/raw/LangLab/experiments/study_3_pilot/"
+        "ToMCATPostGameSurvey_DataDictionary_2024-07-17.csv"
+    )
 
     @classmethod
     @field_validator(

human_experiments/datasette_interface/datasette_interface/database/entity/base/group_session.py

@@ -9,3 +9,6 @@ class GroupSession(Base):
 
     id: Mapped[str] = mapped_column(Text, primary_key=True)
     advisor: Mapped[str] = mapped_column(Text)
+    lion_minecraft_playername: Mapped[str] = mapped_column(Text, nullable=True)
+    tiger_minecraft_playername: Mapped[str] = mapped_column(Text, nullable=True)
+    leopard_minecraft_playername: Mapped[str] = mapped_column(Text, nullable=True)

human_experiments/datasette_interface/datasette_interface/database/entity/base/participant.py

@@ -1,4 +1,5 @@
-from sqlalchemy import Integer
+from typing import Optional
+from sqlalchemy import Integer, Text, Boolean
 from sqlalchemy.orm import Mapped, mapped_column
 
 from datasette_interface.database.config import Base
@@ -8,3 +9,31 @@ class Participant(Base):
     __tablename__ = "participant"
 
     id: Mapped[int] = mapped_column(Integer, primary_key=True)
+    age: Mapped[Optional[int]] = mapped_column(Integer)
+    sex: Mapped[Optional[str]] = mapped_column(Text)
+    hisp: Mapped[Optional[bool]] = mapped_column(Boolean)
+    race: Mapped[Optional[str]] = mapped_column(Text)
+    income: Mapped[Optional[str]] = mapped_column(Text)
+    edu: Mapped[Optional[str]] = mapped_column(Text)
+    exp: Mapped[Optional[str]] = mapped_column(Text)
+    exp_mc: Mapped[Optional[str]] = mapped_column(Text)
+    handedness: Mapped[Optional[str]] = mapped_column(Text)
+    trackpad_preference: Mapped[Optional[str]] = mapped_column(Text)
+    sph_label: Mapped[Optional[str]] = mapped_column(Text)
+    shl_impairments: Mapped[Optional[str]] = mapped_column(Text)
+    shl_impairments_specify: Mapped[Optional[str]] = mapped_column(Text)
+    shl_impairments_agediagnosis: Mapped[Optional[str]] = mapped_column(Text)
+    shl_impairments_therapy: Mapped[Optional[str]] = mapped_column(Text)
+    first_language: Mapped[Optional[str]] = mapped_column(Text)
+    languages_spoken: Mapped[Optional[str]] = mapped_column(Text)
+    language_age_learned: Mapped[Optional[str]] = mapped_column(Text)
+    countries_live_one_year: Mapped[Optional[str]] = mapped_column(Text)
+    major_schooling_country: Mapped[Optional[str]] = mapped_column(Text)
+    health_label: Mapped[Optional[str]] = mapped_column(Text)
+    health_concussion: Mapped[Optional[str]] = mapped_column(Text)
+    health_seizure: Mapped[Optional[str]] = mapped_column(Text)
+    health_trauma: Mapped[Optional[str]] = mapped_column(Text)
+    health_other_trauma_specify: Mapped[Optional[str]] = mapped_column(Text)
+    health_medications: Mapped[Optional[str]] = mapped_column(Text)
+    health_vision: Mapped[Optional[str]] = mapped_column(Text)
+    health_vision_specify: Mapped[Optional[str]] = mapped_column(Text)

human_experiments/datasette_interface/datasette_interface/database/entity/base/post_game_survey.py

@@ -0,0 +1,140 @@
+from typing import Optional
+from sqlalchemy import Integer, Text, Boolean, ForeignKey
+from sqlalchemy.orm import Mapped, mapped_column
+
+from datasette_interface.database.config import Base
+from datasette_interface.database.entity.base.participant import Participant
+
+
+class PostGameSurvey(Base):
+    __tablename__ = "post_game_survey"
+
+    participant_id: Mapped[int] = mapped_column(
+        "participant", Integer, ForeignKey(Participant.id), primary_key=True
+    )
+
+    agent_calm: Mapped[Optional[str]] = mapped_column(Text)
+    agent_anxious: Mapped[Optional[str]] = mapped_column(Text)
+    agent_excited: Mapped[Optional[str]] = mapped_column(Text)
+    agent_sad: Mapped[Optional[str]] = mapped_column(Text)
+    agent_guilty: Mapped[Optional[str]] = mapped_column(Text)
+    agent_angry: Mapped[Optional[str]] = mapped_column(Text)
+    agent_happy: Mapped[Optional[str]] = mapped_column(Text)
+    agent_lonely: Mapped[Optional[str]] = mapped_column(Text)
+    agent_proud: Mapped[Optional[str]] = mapped_column(Text)
+    agent_friendly: Mapped[Optional[str]] = mapped_column(Text)
+
+    game_calm: Mapped[Optional[str]] = mapped_column(Text)
+    game_anxious: Mapped[Optional[str]] = mapped_column(Text)
+    game_excited: Mapped[Optional[str]] = mapped_column(Text)
+    game_sad: Mapped[Optional[str]] = mapped_column(Text)
+    game_guilty: Mapped[Optional[str]] = mapped_column(Text)
+    game_angry: Mapped[Optional[str]] = mapped_column(Text)
+    game_happy: Mapped[Optional[str]] = mapped_column(Text)
+    game_lonely: Mapped[Optional[str]] = mapped_column(Text)
+    game_proud: Mapped[Optional[str]] = mapped_column(Text)
+    game_friendly: Mapped[Optional[str]] = mapped_column(Text)
+
+    agent_intel: Mapped[Optional[int]] = mapped_column(Integer)
+    agent_care: Mapped[Optional[int]] = mapped_column(Integer)
+    agent_honest: Mapped[Optional[int]] = mapped_column(Integer)
+    agent_expert: Mapped[Optional[int]] = mapped_column(Integer)
+    agent_concern: Mapped[Optional[int]] = mapped_column(Integer)
+    agent_trust: Mapped[Optional[int]] = mapped_column(Integer)
+    agent_comp: Mapped[Optional[int]] = mapped_column(Integer)
+    agent_insens: Mapped[Optional[int]] = mapped_column(Integer)
+    agent_honor: Mapped[Optional[int]] = mapped_column(Integer)
+    agent_bright: Mapped[Optional[int]] = mapped_column(Integer)
+    agent_understand: Mapped[Optional[int]] = mapped_column(Integer)
+    agent_phoney: Mapped[Optional[int]] = mapped_column(Integer)
+
+    agent_well: Mapped[Optional[str]] = mapped_column(Text)
+    agent_smooth: Mapped[Optional[str]] = mapped_column(Text)
+    agent_acc: Mapped[Optional[str]] = mapped_column(Text)
+    agent_like: Mapped[Optional[str]] = mapped_column(Text)
+    agent_enjoy: Mapped[Optional[str]] = mapped_column(Text)
+    agent_awk: Mapped[Optional[str]] = mapped_column(Text)
+    agent_place: Mapped[Optional[str]] = mapped_column(Text)
+    agent_play: Mapped[Optional[str]] = mapped_column(Text)
+    agent_perform: Mapped[Optional[str]] = mapped_column(Text)
+
+    team_calm: Mapped[Optional[str]] = mapped_column(Text)
+    team_anxious: Mapped[Optional[str]] = mapped_column(Text)
+    team_excited: Mapped[Optional[str]] = mapped_column(Text)
+    team_sad: Mapped[Optional[str]] = mapped_column(Text)
+    team_guilty: Mapped[Optional[str]] = mapped_column(Text)
+    team_angry: Mapped[Optional[str]] = mapped_column(Text)
+    team_happy: Mapped[Optional[str]] = mapped_column(Text)
+    team_lonely: Mapped[Optional[str]] = mapped_column(Text)
+    team_proud: Mapped[Optional[str]] = mapped_column(Text)
+    team_friendly: Mapped[Optional[str]] = mapped_column(Text)
+
+    team_intel: Mapped[Optional[int]] = mapped_column(Integer)
+    team_care: Mapped[Optional[int]] = mapped_column(Integer)
+    team_honest: Mapped[Optional[int]] = mapped_column(Integer)
+    team_expert: Mapped[Optional[int]] = mapped_column(Integer)
+    team_concern: Mapped[Optional[int]] = mapped_column(Integer)
+    team_trust: Mapped[Optional[int]] = mapped_column(Integer)
+    team_comp: Mapped[Optional[int]] = mapped_column(Integer)
+    team_insens: Mapped[Optional[int]] = mapped_column(Integer)
+    team_honor: Mapped[Optional[int]] = mapped_column(Integer)
+    team_bright: Mapped[Optional[int]] = mapped_column(Integer)
+    team_understand: Mapped[Optional[int]] = mapped_column(Integer)
+    team_phoney: Mapped[Optional[int]] = mapped_column(Integer)
+
+    team_wrong: Mapped[Optional[str]] = mapped_column(Text)  # typo..
+    team_forget: Mapped[Optional[str]] = mapped_column(Text)
+    team_minimize: Mapped[Optional[str]] = mapped_column(Text)
+    team_insult: Mapped[Optional[str]] = mapped_column(Text)
+    team_irrat: Mapped[Optional[str]] = mapped_column(Text)
+    team_crit: Mapped[Optional[str]] = mapped_column(Text)
+    team_fault: Mapped[Optional[str]] = mapped_column(Text)
+    team_ignore: Mapped[Optional[str]] = mapped_column(Text)
+    team_impt: Mapped[Optional[str]] = mapped_column(Text)
+    team_weak: Mapped[Optional[str]] = mapped_column(Text)
+
+    team_along: Mapped[Optional[str]] = mapped_column(Text)
+    team_smooth: Mapped[Optional[str]] = mapped_column(Text)
+    team_accept: Mapped[Optional[str]] = mapped_column(Text)
+    team_like: Mapped[Optional[str]] = mapped_column(Text)
+    team_enjoy: Mapped[Optional[str]] = mapped_column(Text)
+    team_awk: Mapped[Optional[str]] = mapped_column(Text)
+    team_place: Mapped[Optional[str]] = mapped_column(Text)
+    team_likeme: Mapped[Optional[str]] = mapped_column(Text)
+    team_genuine: Mapped[Optional[str]] = mapped_column(Text)
+    team_part: Mapped[Optional[str]] = mapped_column(Text)
+    team_listen: Mapped[Optional[str]] = mapped_column(Text)
+    team_ilike: Mapped[Optional[str]] = mapped_column(Text)
+    team_interact: Mapped[Optional[str]] = mapped_column(Text)
+    team_itrust: Mapped[Optional[str]] = mapped_column(Text)
+    team_fit: Mapped[Optional[str]] = mapped_column(Text)
+    team_cohesion: Mapped[Optional[str]] = mapped_column(Text)
+    team_work: Mapped[Optional[str]] = mapped_column(Text)
+    team_knit: Mapped[Optional[str]] = mapped_column(Text)
+    team_like_mem: Mapped[Optional[str]] = mapped_column(Text)
+    team_work_well: Mapped[Optional[str]] = mapped_column(Text)
+    team_decision: Mapped[Optional[str]] = mapped_column(Text)
+    team_express: Mapped[Optional[str]] = mapped_column(Text)
+    team_organize: Mapped[Optional[str]] = mapped_column(Text)
+    team_accomplish: Mapped[Optional[str]] = mapped_column(Text)
+    team_approp: Mapped[Optional[str]] = mapped_column(Text)
+    team_alt: Mapped[Optional[str]] = mapped_column(Text)
+    team_influ: Mapped[Optional[str]] = mapped_column(Text)
+    team_contrib: Mapped[Optional[str]] = mapped_column(Text)
+
+    agent_emot1: Mapped[Optional[str]] = mapped_column(Text)
+    agent_emot2: Mapped[Optional[str]] = mapped_column(Text)
+    agent_emot3: Mapped[Optional[str]] = mapped_column(Text)
+    agent_emot4: Mapped[Optional[str]] = mapped_column(Text)
+    agent_emot5: Mapped[Optional[str]] = mapped_column(Text)
+    agent_emot6: Mapped[Optional[str]] = mapped_column(Text)
+    agent_emot7: Mapped[Optional[str]] = mapped_column(Text)
+    agent_emot8: Mapped[Optional[str]] = mapped_column(Text)
+    agent_emot9: Mapped[Optional[str]] = mapped_column(Text)
+    agent_emot10: Mapped[Optional[str]] = mapped_column(Text)
+
+    know_team_members: Mapped[Optional[bool]] = mapped_column(Boolean)
+    know_person_at_cheetah: Mapped[Optional[str]] = mapped_column(Text)
+    know_person_at_lion: Mapped[Optional[str]] = mapped_column(Text)
+    know_person_at_tiger: Mapped[Optional[str]] = mapped_column(Text)
+    know_person_at_leopard: Mapped[Optional[str]] = mapped_column(Text)