July is almost over and the burning question is where is your board? We delivered our first boards to customers this month who have discovered a critical radio bug that potentially results in a total loss of throttle control. We immediately asked these customers to stop riding and halted further deliveries until we could identify, diagnose, and patch the issue.
For a more detailed technical explanation of the issue, you can read at the bottom of this post.
With the help of our early customers we have replicated and diagnosed the issue and our team is now implementing a patch that will be applied to all boards and tested before the rest ship out. Delays are never good and we do apologise profusely for this, however it is against our philosophy to knowingly put our customers at risk of potential injury.
Whilst we do have OTA updates available and could ship boards and push the update when ready, we decided to pause shipments out of concern for our customers. Because the triggering conditions for the issue can only be detected with advanced RF equipment, customers are not be able to identify and avoid the issue before they experience the loss of control. We recognise how excited everyone is to ride their boards and how by shipping boards before the firmware is updated that customers could ride the boards with the present software.
We know how frustrating this can be, our team is working diligently to rectify this issue as fast as possible and get the rest of the boards out the door to our incredibly patient customers. We are internally discussing compensation options for customers given the extended delays and will update customers waiting this week with a number of options.
We will provide a further update this week with more detailed timelines to resolution once our team has progressed further with the firmware development changes.
Thank you again for your support and patience as we work to deliver a reliable and amazing riding experience for our customers. We will be in touch with each customer regarding compensation options.
The technical details
The Jed Board uses finite state machines (you can read more about finite state machines here) to define the behaviour of each subsystem, thus restricting the overall behaviour of the product to a finite permutation of states. Our radio module features a state machine that is responsible for all aspects of communication and operation between the remote control and the radio in our electronic speed controller.
Our radios communicate using new BLE5.0 standard which offers a number of significant advantages over BLE4/BT4 and other radio options. This technology enables us to sustain two simultaneous radio communications and synchronise the states between the two independent ESCs in the AWD. There is a lot of complex processing to keep track of these states and the patch we are implementing further optimises these connections.
In areas with excessively abnormal radio activity in the 2.4Ghz spectrum, the present state machine can fail to recognise a disconnection event which is required to initiate the reconnecting process. The system becomes trapped in a state where it is expecting user input from a connected device that is in reality no longer connected, however it will not attempt to reconnect because the device is still seen as connected. Although the remote displays accurate speedometer and battery level in this state because those values are broadcasted as BLE characteristics to paired devices, control inputs are not communicated by the state machine because the state of the ESC and Remote radios are no longer in sync. Both radios must perceive a disconnect event to initiate re-connection. Thus, redefining and tuning the trigger event for the re-connection process to account for this unusual interference shall prevent de-synchronisation of the remote and ESC radios.
This software bug is especially more pronounced on the AWD as there are a total of 3 radios involved. The state of the three state machines must be synchronised and able to recognise all disconnection events. Unfortunately the side effect was not always a total loss of connectivity, but reduction to a single controller.
Rebooting the affected ESC by removing and reattaching batteries after 5 seconds clears the error.
We have spent over 12 months testing our radios in varying RF conditions, countries, and scenarios using measuring equipment such as the HackRF and various other spectrum analysis tools. Navigating the diversity of consumer and commercial products that operate on the 2.4 Ghz spectrum requires tuning as two devices rarely communicate using the same protocol.
By capturing the RF interference with a custom firmware and playing it back in the lab, we identified the bug’s mechanism of action and affected programming states of the radios.
We handled a similar bug months ago when we discovered interference from Apple devices caused a similar exception in our state machine. Because Apple devices are notorious at transmitting more than they need to, we implemented a software filter to attenuate the spurious data to resolve the problem. Apple products and Android phones are no longer a concern as we ride frequently with up to five iPhones and three apple watches of varying models, three android phones and two bluetooth speakers of various make and model – to say the least we have full pockets when riding)
We take pride in our remote radio’s conservative power settings which comply with all legal requirements in various countries around the world and provides us with weeks of riding between charges. We explored boosting radio power at the expense of battery life to further mitigate the potential for service interruptions that require a reconnection, but a hardware implementation would be more of a band-aid than an actual fix that addresses state desynchronisation.
Our team is making design changes to the state machine to better handle conditions like this and to help catch anything else that may potentially crop up in the future. This also involves changes to how we handle the AWD connections and will overall offer a more reliable and resilient connection in extensively excessive RF scenarios.