Skip to content

Commit

Permalink
Neon Companion app.
Browse files Browse the repository at this point in the history
  • Loading branch information
mikelgg93 committed Nov 11, 2024
1 parent 7b71c68 commit de43be5
Show file tree
Hide file tree
Showing 26 changed files with 90 additions and 78 deletions.
16 changes: 9 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,18 +1,20 @@
# Recommended Defaults

- When importing images, use markdown-style imports

```markdown
![alt text](/path/to/image.png)
```

- When trying to create a gallery view, consider using the `PhotoGrid` component
- Avoid using custom HTML and CSS as much as possible.


# Capitalization

- We use title case for all titles, headings, menu sections etc. We use title case as it is defined by the Associated Press Stylebook. There is a converter available [here](https://titlecaseconverter.com/) to double-check correctness.
- Fixed product and feature names are capitalized, like e.g.
- Reference Image Mapper
- Neon Companion App
- Neon Companion Device
- Pupil Invisible Glasses
- Video Renderer
- Heatmap Visualization
- Reference Image Mapper
- Neon Companion
- Pupil Invisible Glasses
- Video Renderer
- Heatmap Visualization
2 changes: 1 addition & 1 deletion alpha-lab/neon-with-capture/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,5 +16,5 @@ To get started with Neon in Capture, follow these steps:
5. Select either `Neon 3D` or `2D` as the **Gaze Mapping** option in the **Calibration** tab.

::: warning
Recordings made with the Neon Companion App (rather than Pupil Capture) are **NOT** compatible with Pupil Player.
Recordings made with the Neon Companion app (rather than Pupil Capture) are **NOT** compatible with Pupil Player.
:::
7 changes: 4 additions & 3 deletions invisible/data-collection/offset-correction/index.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,13 @@
# Using Offset Correction to Improve Gaze Accuracy
For some wearers, you may find a constant offset in their gaze estimates. Especially in setups where wearers gaze at somethings that is less than 1 meter away from them this can happen due to parallax error.

To compensate for those offsets, you can use the Offset Correction feature in the Pupil Invisible Companion App. See the video below to learn how it works! The video is using demonstrating the feature within the Neon Companion App, but the approach is very similar.
For some wearers, you may find a constant offset in their gaze estimates. Especially in setups where wearers gaze at somethings that is less than 1 meter away from them this can happen due to parallax error.

To compensate for those offsets, you can use the Offset Correction feature in the Pupil Invisible Companion App. See the video below to learn how it works! The video is using demonstrating the feature within the Neon Companion app, but the approach is very similar.

<Youtube src="7weK8UPLOzo" />

::: warning
Note, that the amount of offset introduced by parallax error is highly dependend on the distance between the wearer and the object they are looking at. The closer the object, the larger the offset.

The offset correction is only valid for the specific distance at which it was recorded. If the wearer changes their distance to the object, you will need to record a new offset correction.
:::
:::
8 changes: 4 additions & 4 deletions neon/data-collection/data-format/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,17 +19,17 @@ This file contains meta-information on the recording.
| **android_device_id** | Unique identifier of the Android device used as Companion. |
| **android_device_model** | Model name of the Companion device. |
| **android_device_name** | Device name of the Companion device. |
| **app_version** | Version of the Neon Companion App used to make the recording. |
| **calib_version** | Version of the offset correction used by the Neon Companion App. |
| **data_format_version** | Version of the data format used by the Neon Companion App. |
| **app_version** | Version of the Neon Companion app used to make the recording. |
| **calib_version** | Version of the offset correction used by the Neon Companion app. |
| **data_format_version** | Version of the data format used by the Neon Companion app. |
| **duration** | Duration of the recording in nanoseconds |
| **firmware_version** | Version numbers of the firmware and FPGA. |
| **frame_id** | Number identifying the type of frame used for this recording. |
| **frame_name** | Name of the frame used for this recording. |
| **gaze_offset** | Gaze offset applied to this recording using the offset correction. Values are in pixels. |
| **module_serial_number** | Serial number of the Neon module used for the recording. This number is encoded in the QR code on the back of the Neon module. |
| **os_version** | Version of the Android OS that was installed on the recording Companion device. |
| **pipeline_version** | Version of the gaze estimation pipeline used by the Neon Companion App. |
| **pipeline_version** | Version of the gaze estimation pipeline used by the Neon Companion app. |
| **recording_id** | Unique identifier of the recording. |
| **start_time** | Timestamp of when the recording was started. Given as UTC timestamp in nanoseconds. |
| **template_data** | Data regarding the selected template for the recording as well as the response values. |
Expand Down
8 changes: 4 additions & 4 deletions neon/data-collection/data-streams/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ The scene camera can be operated with automatic or manual exposure. In situation
## Gaze

Available in: <Badge>Real-time</Badge><Badge>Pupil Cloud</Badge><Badge>Neon Player</Badge>
The Neon Companion App can provide gaze data in real-time at up to 200 Hz. Gaze data is output in pixel space of the scene camera image. The origin is in the top-left corner of the image.
The Neon Companion app can provide gaze data in real-time at up to 200 Hz. Gaze data is output in pixel space of the scene camera image. The origin is in the top-left corner of the image.

![Gaze](./gaze.jpg)

Expand All @@ -47,7 +47,7 @@ The downloads for gaze mapping enrichments ([Reference Image Mapper](/pupil-clou
## 3D Eye States

Available in: <Badge>Real-time</Badge><Badge>Pupil Cloud</Badge>
The Neon Companion App provides 3D eye state data in real-time at up to 200 Hz. The 3D eye states are a time series of each eye's position and orientation in 3D space, given by the location of the eyeball center and the optical axis of each eye. The units are millimeters.
The Neon Companion app provides 3D eye state data in real-time at up to 200 Hz. The 3D eye states are a time series of each eye's position and orientation in 3D space, given by the location of the eyeball center and the optical axis of each eye. The units are millimeters.

The coordinate system is depicted below. The origin corresponds to the scene camera of the Neon Module.

Expand All @@ -64,7 +64,7 @@ If 200 Hz real-time data is essential, consider upgrading to a newer [Companion
## Pupil Diameters

Available in: <Badge>Real-time</Badge><Badge>Pupil Cloud</Badge>
The Neon Companion App provides pupil diameter data in real-time at up to 200 Hz. Separately for the left and right eye. The computed pupil diameters correspond to the physical pupil size in mm, rather than the apparent pupil size in pixels as observed in the eye videos. You can find a high-level description as well as a thorough evaluation of the accuracy and robustness of Neon’s pupil-size measurements in our [white paper](https://zenodo.org/records/10057185).
The Neon Companion app provides pupil diameter data in real-time at up to 200 Hz. Separately for the left and right eye. The computed pupil diameters correspond to the physical pupil size in mm, rather than the apparent pupil size in pixels as observed in the eye videos. You can find a high-level description as well as a thorough evaluation of the accuracy and robustness of Neon’s pupil-size measurements in our [white paper](https://zenodo.org/records/10057185).

Similar to the 3D eye states, the accuracy of the pupil diameter measurements improves when supplying the wearer's IED in the wearer profile before making a recording.

Expand All @@ -82,7 +82,7 @@ The blink detection algorithm is operating directly on the eye video to detect t
Available in: <Badge>Pupil Cloud</Badge><Badge>Neon Player</Badge>
Stereo microphones are integrated into the Neon module. Recorded audio will be part of the resulting scene video.

Audio recording is disabled in the Neon Companion App by default and can be enabled in the settings.
Audio recording is disabled in the Neon Companion app by default and can be enabled in the settings.

## Movement (IMU Data)

Expand Down
6 changes: 3 additions & 3 deletions neon/data-collection/ecosystem-overview/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,15 +4,15 @@ The Neon ecosystem contains a range of tools that support you during data collec

## Neon Companion App

You should have already used the Neon Companion App to [make your first recording](/data-collection/first-recording/). This app is the core of every Neon data collection.
You should have already used the Neon Companion app to [make your first recording](/data-collection/first-recording/). This app is the core of every Neon data collection.

When your Neon is connected to the Companion device, it supplies it with power and enables it to generate a real-time gaze signal as well as several other [data streams](/data-collection/data-streams/). When making a [recording](/data-collection/recordings/), all generated data is saved on the Companion device.

The app automatically saves [UTC timestamps](https://en.wikipedia.org/wiki/Coordinated_Universal_Time) for every generated data sample. This allows you to easily sync your data with other external data streams, or to sync recordings from multiple subjects that have been made in parallel.

## Other Data Collection Tools

Several other tools complement the Neon Companion App and can make data collection much easier in some scenarios.
Several other tools complement the Neon Companion app and can make data collection much easier in some scenarios.

### Neon Monitor

Expand Down Expand Up @@ -40,7 +40,7 @@ Neon is compatible with LSL and you can learn more about how to use Neon with LS

[Pupil Cloud](/pupil-cloud/) is our web-based storage and analysis platform located at [cloud.pupil-labs.com](https://cloud.pupil-labs.com/). It makes it easy to store all your data securely in one place and offers a variety of options for data analysis and visualization.

Pupil Cloud is the recommended tool for processing your Neon recordings and if you enable uploads in the Neon Companion App all recordings can be uploaded automatically.
Pupil Cloud is the recommended tool for processing your Neon recordings and if you enable uploads in the Neon Companion app all recordings can be uploaded automatically.

![Pupil Cloud](./pupil_cloud.webp)

Expand Down
6 changes: 3 additions & 3 deletions neon/data-collection/first-recording/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,11 +12,11 @@ Create a new Google account or use an existing Google account during setup.

## 2. Install and Start the Neon Companion App.

Next, install the Neon Companion App on your device:
Next, install the Neon Companion app on your device:

- Launch the **Google Play Store** app. It is already installed by default on your Companion Device.
- Search for [**Neon Companion**](https://play.google.com/store/apps/details?id=com.pupillabs.neoncomp) in the Google Play Store and click install.
- Start the Neon Companion App when the installation has finished.
- Start the Neon Companion app when the installation has finished.
- A couple of permission prompts will appear, which you need to accept.
- First-time users need to sign up for a [Pupil Cloud](https://cloud.pupil-labs.com/) account. Sign up using your Google account, or create an account with an email address and password.

Expand All @@ -27,7 +27,7 @@ Next, install the Neon Companion App on your device:

</a>

![Neon Companion App](/ne-companion_app_logo-bg.png)
![Neon Companion app](/ne-companion_app_logo-bg.png)

</div>

Expand Down
5 changes: 3 additions & 2 deletions neon/data-collection/index.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,11 @@
# Data Collection with Neon

Data collection is a key step for any application of Neon! In this section you can learn everything about it, starting with how to make [your first recording](/data-collection/first-recording/)!

You can find an overview of what data is contained in Neon recordings including all the [data streams](/data-collection/data-streams/) from the various sensors, as well as any additional data like [events](/data-collection/events/), [wearers](/data-collection/wearers/), and [templates](/data-collection/templates/).

You can find introductions on how to use the Neon Companion App, e.g. on [Offset Correction](/data-collection/offset-correction/), and how-to guides on common tasks during data collection, e.g. [Time Synchronization](/data-collection/time-synchronization/).
You can find introductions on how to use the Neon Companion app, e.g. on [Offset Correction](/data-collection/offset-correction/), and how-to guides on common tasks during data collection, e.g. [Time Synchronization](/data-collection/time-synchronization/).

Documentation on useful software and integrations for data collection is also available, see e.g. [Monitor App](/data-collection/monitor-app/) or [Lab Streaming Layer](/data-collection/lab-streaming-layer/).

Finally, you can find a list of [troubleshooting](/data-collection/troubleshooting/) tips and tricks for common issues during data collection.
Finally, you can find a list of [troubleshooting](/data-collection/troubleshooting/) tips and tricks for common issues during data collection.
16 changes: 8 additions & 8 deletions neon/data-collection/lab-streaming-layer/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,15 +2,15 @@

[Lab Streaming Layer](https://labstreaminglayer.org/) (LSL) is an open-source framework that connects, manages, and synchronizes data streams from multiple sources, such as EEG, GSK, and motion capture systems. Check out the [LSL documentation](https://labstreaminglayer.readthedocs.io/info/intro.html) for a full overview of supported devices.

The Neon Companion App has built-in support for LSL, streaming Neon’s real-time generated data over the LSL network. This allows you to easily synchronize Neon with other LSL-supported devices.
The Neon Companion app has built-in support for LSL, streaming Neon’s real-time generated data over the LSL network. This allows you to easily synchronize Neon with other LSL-supported devices.

## **Usage**

LSL streaming can be initiated in the Companion App by enabling the "Stream over LSL" setting.

When enabled, data will be streamed over the LSL network, and subsequently, to any connected LSL inlet (such as the LSL LabRecorder App, or another third-party system with inlet functionality) which is listening. Like the [Real-Time API](https://docs.pupil-labs.com/neon/real-time-api/tutorials/), it is not necessary for the Companion App to be actively recording, but simultaneously streaming LSL data while making a recording is supported.

Note that you'll need to ensure the Neon Companion App is connected to the same network as the other devices streaming via LSL.
Note that you'll need to ensure the Neon Companion app is connected to the same network as the other devices streaming via LSL.

## **LSL Outlets**

Expand Down Expand Up @@ -40,11 +40,11 @@ If your devices are on the same network but you have trouble connecting, it is l

- UDP broadcasts to port `16571` and/or
- UDP multicast to port `16571` at
- `FF02:113D:6FDD:2C17:A643:FFE2:1BD1:3CD2`
- `FF05:113D:6FDD:2C17:A643:FFE2:1BD1:3CD2`
- `FF08113D:6FDD:2C17:A643:FFE2:1BD1:3CD2`
- `FF0E:113D:6FDD:2C17:A643:FFE2:1BD1:3CD2`
- `224.0.0.1`, `224.0.0.183`, `239.255.172.215`
- `FF02:113D:6FDD:2C17:A643:FFE2:1BD1:3CD2`
- `FF05:113D:6FDD:2C17:A643:FFE2:1BD1:3CD2`
- `FF08113D:6FDD:2C17:A643:FFE2:1BD1:3CD2`
- `FF0E:113D:6FDD:2C17:A643:FFE2:1BD1:3CD2`
- `224.0.0.1`, `224.0.0.183`, `239.255.172.215`
- TCP and UDP connections to the ports `16572`-`16604`

More troubleshooting tips can be found in the [Network Troubleshooting](https://labstreaminglayer.readthedocs.io/info/network-connectivity.html) page in LSL’s documentation.
More troubleshooting tips can be found in the [Network Troubleshooting](https://labstreaminglayer.readthedocs.io/info/network-connectivity.html) page in LSL’s documentation.
4 changes: 2 additions & 2 deletions neon/data-collection/measuring-ied/index.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
# Measuring the Inter-Eye-Distance (IED)

The wearer's IED can be set in the Neon Companion App for applications requiring
The wearer's IED can be set in the Neon Companion app for applications requiring
precise pupillometry or eye-state measurements.
This does not require prior calibration of the device. However, the accuracy of Neon’s 3D eye-state and pupil-size
measurements can be enhanced by correctly setting the IED for each wearer.

To add the wearer's IED, input the value into their ‘Wearer Profile’ in the Neon Companion App before
To add the wearer's IED, input the value into their ‘Wearer Profile’ in the Neon Companion app before
starting
a recording. The default IED is set to 63 mm, which is the average for adults.

Expand Down
4 changes: 2 additions & 2 deletions neon/data-collection/monitor-app/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,10 @@

Using the **Neon Monitor** app you can easily monitor your data collection in real-time and remote control all your Neons from another device. It's a web-app that can be opened in any browser on a computer, tablet or phone! The only requirement is that the Neon Companion device and the device you use to access the Monitor app are connected to the same network.

To access the Monitor app make sure the Neon Companion App is running and visit the page [neon.local:8080](http://neon.local:8080) on your monitoring device.
To access the Monitor app make sure the Neon Companion app is running and visit the page [neon.local:8080](http://neon.local:8080) on your monitoring device.

::: tip
The Neon Companion App can display a QR code that gets you straight to the monitor website. Just select `Streaming` on the home screen!
The Neon Companion app can display a QR code that gets you straight to the monitor website. Just select `Streaming` on the home screen!
:::

## The App's User Interface
Expand Down
3 changes: 2 additions & 1 deletion neon/data-collection/offset-correction/index.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
# Using Offset Correction to Improve Gaze Accuracy
For some wearers, you may find a constant offset in their gaze estimates. To compensate for those, you can use the Offset Correction feature in the Neon Companion App. See the video below to learn how it works!

For some wearers, you may find a constant offset in their gaze estimates. To compensate for those, you can use the Offset Correction feature in the Neon Companion app. See the video below to learn how it works!

<Youtube src="7weK8UPLOzo" />

Expand Down
Loading

0 comments on commit de43be5

Please sign in to comment.