Solving comma.ai 's openpilot design challenge
Multimodal messaging for openpilot confidence levels
Comma.ai's openpilot is a Level 2 driver-assistance system that can be installed in a vehicle to enable semi-autonomous driving capabilities. Openpilot monitors its own confidence in handling various driving situations, but it currently communicates this information through disruptive audible alerts when it nears system limits like steering torque, brake pressure, or acceleration.
The challenge was to design a clear, non-intrusive UI that effectively conveys three tiers of driving confidence—from reliable to expected to fail. The goal here is to meet user needs by providing timely, intuitive information that enhances safety without causing distraction or annoyance.
The proposed design communicates system status in a multimodal interface that prioritizes safety while minimizing driver distraction. Drawing on human factors research, such as ambient displays and peripheral vision principles, messages appear positioned on the left within or near the driver’s eyellipse (where the drivers eyes are likely to be located in the vehicle) for minimal glance time, integrating voice commands and tones to provide action-oriented prompts like "Brake" or "Ready Control." Using discrete states instead of continuous gauges ensures quick driver responses without requiring constant monitoring. The design is compatible with current hardware with suggested future enhancements like an LED light or Heads Up Display (HUD) integration. The design improves the driving experience by delivering timely alerts in an ergonomically comfortable manner, aligning with UX best practices and standards like ISO 15008:2017 [1] and ISO 4513:2022 [2].
High Confidence
When openpilot has high confidence, typically autopilot will be engaged. When autopilot is engaged, the green UI theme is applied and a small autopilot message is displayed to prioritize other information that might be of more importance.
When entering autopilot:
Tone Plays
Voice message: “Autopilot”
Audio alerts can be managed in settings
When openpilot has high confidence:
"Autopilot" on-screen message
Small size on-screen message
Green UI Theme
Low Confidence
When openpilot has low confidence, standard messaging will include the verb “Ready” and be tailored to the specific action the driver might need to prepare to take. This will allow a driver to observe road conditions and ideally be prepared to take action before they are given a message to take control. Specifically for steering, an on-screen message will be displayed that says “Ready Steer,” an auditory tone will play, and a voice message will say “Ready Steer.”
When entering low confidence:
Tone Plays
Voice message: “Ready [action]”
Audio alerts can be managed in settings
When openpilot has low confidence:
"Ready [action]" on-screen message
Medium size on-screen message
Tan UI Theme
No Confidence
When openpilot has no confidence, standard messaging will include a take over message and in some cases be tailored to the specific action the driver might need to take. Ideally, the driver is first given a message to be "Ready" and then when there is no confidence they can be prepared to take the needed action. Specifically for steering, an on-screen message will be displayed that says “Steer,” an auditory tone will play, and a voice message will say “Steer.” The large high-contrast message is used so that a driver can easily see the message at a glance or in their periphery. The abrupt change of the display will indicate they need to take control.
When entering no confidence:
Tone Plays
Voice message: “[Action]”
Audio alerts can be managed in settings
When openpilot has no confidence:
"[Action]" on-screen message
Large size on-screen message
Black UI Theme (high contrast)
Driver Control
When a driver has control of the car, a small message is displayed that says "Driver Control" and a blue UI theme is applied.
When entering driver control:
Tone Plays
Voice message: “Driver Control”
Audio alerts can be managed in settings
When driver has control:
"Driver Control" on-screen message
Small size on-screen message
Blue UI Theme
Solution highlights
Messages are positioned on the left of the screen to allow for minimal glance from the driver.
Messages communicate the specific action a driver needs to take.
Different states are communicated through voice audio alerts, audio tones, on screen messages, and colors.
Voice and tone audio alerts should be prioritized over on-screen messages. Although helpful, monitoring on screen messages will likely cause distractions from driving.
Non-actionable messages such as “Autopilot” and “Driver Control” are displayed small in the corner as they dont require any immediate action.
Actionable messages such as “Ready Steer” are displayed larger to differentiate them from autopilot
Messages that require immediate action such as “Steer” are presented prominently with the highest contrast colors and the largest size in order to grab the driver's attention.
States for Confidence Levels of Driver Controls
Depending on the unique driving situation, a different state will display depending on the confidence level of a specific driver control or confidence of a combinations of driver controls. The state that is displayed to the driver will provide feedback to effectively convey the action the driver must take to continue driving safely.
Case for non-analog states
Recommend not presenting states of confidence levels as an analog display even though the data openpilot might be analyzing is analog and could be shown on a continuous spectrum. For example from the driver's perspective, they need to know if the openpilot is in autopilot, whether they need to be ready to take over, or if they need to immediately take over. There are no actions a driver can take that are in between these discrete states. Presenting additional information outside of these states that a diver needs to constantly monitor and is not relevant to immediate actions they need to take is likely to distract from driving.
If an analog gauge were used for confidence levels, there could be confusion about what is displayed for steering, brake, and acceleration. If an analog gauge for acceleration is at 50 percent, could a driver interpret that as the car accelerating at 50 percent? Leaving this open to interpretation would likely be dangerous.
Feedback Methods
The main purpose of the feedback methods is to signal to the driver to take a specific action. For this reason, feedback is recommended to be limited to specific actions a user takes and will not include additional details such as “road conditions are not optimal” that will distract the driver with additional information they need to process but doesn't help them complete a required action.
settings
references
[1] ISO 15008:2017, Road vehicles — Ergonomic aspects of transport information and control systems — Specifications and test procedures for in-vehicle visual presentation, International Organization for Standardization, 2017.
[2] ISO 4513:2022, Road vehicles — Visibility — Method for establishment of eyellipses for driver’s eye location, International Organization for Standardization, 2022.
[4] Wickens, Christopher D., Justin G. Hollands, Simon Banbury, and Raja Parasuraman. 2015. Engineering Psychology and Human Performance. 4th ed. New York: Routledge.