←←All cases

Eye-tracking technology solution for neurological research

Client

A university-based research team specializing in neurological studies

Industry

Biotech, life sciences

Services

Hardware design, embedded software development, desktop application development, technology consulting

Tech stack

STM32H750 microcontroller, MIPI CSI-2 cameras, MPU-6500 motion sensor, OpenCV, Python, C

Challenge

In late 2022, a research team from a California-based university approached ITRex with a non-trivial task. The team, under the direction of a well-known neuroscientist, was conducting research on a species of monkey known for its swift head and eye movements. The experiment focused on vision and how the brain regulates eye movement—specifically, how it stabilizes the image regardless of head position or movement speed. This study aims to help with the development of future prosthetic devices, such as bionic eyes. To carry out the experiment, the researchers required a custom helmet that could record high-quality video data from two angles: the monkey's eye and its field of view. They turned to a third-party vendor, who created a prototype of the eye-tracking solution. The device included a microcontroller, a VGA camera that captured images at 60 frames per second (FPS) with a resolution of 320 by 240 dots, an SD card for image storage, and a 400 mAh accumulator capable of powering the device for 60 minutes of recording. Since the experiments were conducted in a darkened room, the helmet also used an infrared LED light to illuminate the environment and the monkey's face, allowing for precise eye movement tracking. The prototype failed to meet the client’s requirements due to its low frame rate and poor image quality.

Subsequently, the client tasked the ITRex team with:

●

Modifying the eye-tracking solution to achieve desired image frequency and resolution without increasing device size and while using the cameras provided by the client. The latter requirement was especially important because the researchers needed to be able to disassemble the camera and remove the infrared filter.

●

Enhancing the custom eye-tracking device with a gyroscope to record the angle of head rotation relative to its starting position

●

Keeping the battery life of the eye-tracking technology solution at around 70 minutes

●

Optimizing camera and motion sensor data storage and processing

Solution

After meticulously studying the project requirements, the ITRex R&D team proposed two options:

●

Replacing the microcontroller with a more powerful microprocessor. To validate the concept, we created a prototype that, while providing excellent image quality, did not meet the device size requirements and drained the battery too quickly.

●

Making changes to the existing microcontroller. The client’s prototype featured an STM32H750 microcontroller. Despite being energy efficient and one of the most powerful development boards in its class, the microcontroller lacked an interface for connecting directly to cameras. It also failed to store a full frame transmitted by the camera in internal memory. As a result, the ITRex software engineers would have to provide a solution for adapting the microcontroller's standard DCMI interface to the chosen camera’s MIPI CSI-2 interface while optimizing its internal and external storage capabilities.

We went with the second option, addressing the following issues in the process:

●

Developing a custom driver for MIPI CSI-2 cameras. A dedicated embedded systems engineer created a hardware solution and a custom C driver that allowed us to interface the MIPI cameras with the microcontroller. To that end, we had to request documentation from the camera vendor, which turned out to be inaccurate and incomplete. The vendor also provided us with a reference Linux driver, but we had to recreate and modify its functionality from the ground up because the MIPI CSI-2 camera was not designed to work with microcontrollers in the first place. The documentation also lacked instructions for cropping the images transmitted from the camera; by default, the device sent images from the entire matrix (full HD, 2688 by 1520 pixels). The ITRex team needed to figure out how to adjust the image resolution given the microcontroller's limited memory capacity.

●

Addressing the memory constraints. The customer was unable to record images at 60 FPS with the first prototype due to SD card bus limitations. To increase throughput, we recommended replacing the old memory card with an SD card that supports ultra-high-speed (UHS-I) mode and can be connected to the microcontroller with minimum effort. To further optimize the eye-tracking solution’s performance, we turned to the bare-metal programming approach, writing custom firmware in C.

●

Enabling the motion-tracking functionality. In addition to capturing images showing the position of the monkey's eyes and field of view, the research team intended to record the angle of head rotation. For this, we equipped the eye-tracking solution with an MPU-6500 motion sensor. To achieve precise head position tracking, motion data had to be captured at a rate two times faster than the camera's frame rate.

●

Facilitating camera and motion sensor data processing. The MIPI camera captured data using a Bayer matrix. This method allows each pixel to detect a single color, and the complete image is later reconstructed with an accurate color representation through software processing. Given the project's goal of collecting extensive behavioral data without overloading the microcontroller, ITRex devised a strategy for managing the image processing workload. The method involved capturing the data in raw format directly onto the UHS SD card, resulting in approximately 55–60 GB of data for each hour of recording, and processing it using a custom desktop application written in Python. The application, which makes use of robust OpenCV libraries, can handle binary files such as camera-captured video frames. It separates and converts images from Bayer format to RGB before compiling them into an AVI video file. To address the issue of frame drops, a technique for duplicating the previous frame was used to ensure a smooth visual output. The app also provides text transcripts with specific frame numbers and sequences, as well as motion sensor data. Speaking of the latter, we decided not to analyze gyroscope data directly on the microcontroller because doing so would necessitate developing a computational model, which would increase the microprocessor's workload and affect image frequency. Furthermore, there were concerns about the monkeys' movements, which could affect the model's accuracy.

●

Achieving energy efficiency. We optimized the power consumption of the eye-tracking solution by selecting an energy-efficient microcontroller coupled with bare-metal firmware, foregoing a traditional operating system. This option, combined with the eye-tracking device’s ability to enter sleep mode between filming sessions and the decision to transfer data processing functions to a separate desktop app, significantly improves energy efficiency. During active recording, the system maximizes energy usage up to 150 mAh to maintain optimal image quality. Despite this, when operating two cameras simultaneously for an hour, the system's power consumption remains sufficiently low, ensuring the battery is not fully depleted.

Overall, ITRex significantly altered the original prototype by redesigning its hardware and performing a full board bring-up, which included driver and firmware development, functional testing, and power optimization.

Impact

The successfully developed eye-tracking device has transformed the client's research capabilities, providing unprecedented insights into the neural mechanisms that underpin vision and eye movement. Its lightweight, energy-efficient design has proven to be extremely effective in field tests, capturing and processing high-speed video data from monkeys in motion. The project took over a year to complete due to the complexity of the eye-tracking solution requirements and the need to overcome the challenges posed by the initial prototype design. It should be noted that the project was single-handedly carried out by a dedicated full-stack developer with experience in hardware design, embedded system engineering, and desktop application development. If the customer had contacted another company, the eye-tracking device could have been developed faster. However, the budget would have increased manyfold. ITRex, on the other hand, provided a more personalized approach and worked closely with the research team throughout the project, ensuring that the client's needs were fully understood and met. ITRex has also assisted the customer in negotiating a contract with a hardware manufacturer and is currently overseeing mass production of the eye-tracking solution.