Things to consider before deploying Raspberry Pi

Many projects, especially in prototype or early pilot stages, do not fully consider the impact of having to frequently update multiple devices in the field. It is not uncommon to find that the update strategy is simply to SSH into the units and run the update commands, while others use sophisticated tools (e.g. Ansible), but most do not fully consider the risks of pushing significant updates to unattended units.

Pilot projects present a unique set of challenges when it comes to updates. The updates are more frequent as compared to stable production units, yet there is less infrastructure in place to reduce update-related risks. Consequently, pilot units actually need to be more robust than one might otherwise anticipate.

A structured approach for updating software on devices deployed in remote environments is particularly important for edge computing. A well-planned strategy ensures that updates are applied efficiently, securely, and with minimal downtime.

There are different methods to update devices remotely or Over the Air (OTA):

• Package Manager Updates: The OS and its packages may be updated using tools like apt. This is typically used during development but not recommended for unattended devices because the effects of package updates are hard to predict and may leave devices in inconsistent states across the fleet, or fail to boot or log in. These updates may also fail due to power outages or network errors.
• File Updates: Updates to files may be pushed via the internet or local network. A must have for remote devices, however, these updates don’t work too well for system files.
• Full Image Updates: These updates typically deploy a new fresh copy of /root filesystem and the associated boot artifacts (kernel, initramfs, etc.) This guarantees the entire system is always in a consistent and predictable state and eliminates issues caused by package mismatches, dependency conflicts, or incompatible software in partial updates.

The device network connection reliability and bandwidth are factors in determining an update strategy. It is also important to select a fail-safe update method so the device can handle a power-loss or communication failure during an update.

Typical Raspberry Pi installations have a single image partition containing the OS, application, and user data. If this filesystem is somehow corrupted for whatever reason, the device may never boot again. This may be due to an OS update error, power outage, misconfiguration, or a malware attack, and many of these causes are outside of the control of even the most perfect software development and testing process.

The only reliable way to recover is by re-imaging the entire flash via USB or replacing the microSD card. This recovery method is not practical, unless the device is easily accessible by trusted local personnel with appropriate technical skills and a computer.

A/B partitioning is used to improve reliability and stability. This approach ensures that devices can recover from failed updates, corruption, or malware without being on-site.

An A/B image has two bootable system partitions: A (active) and B (inactive). The system boots from Partition A while B remains idle. Software updates are installed on the inactive partition (B in this case). If the update is successful, then partition B becomes the active partition. If an update fails due to power loss or corruption, the system will revert to the previous partition. This mechanism provides fail-safe updates and prevents the device from becoming unbootable.

A variation on the schema is to designate one partition as the “safe mode” partition, while the other partition is the one that’s frequently updated. Both methods have their pros and cons but they both eliminate the single point of failure of having only one bootable /root on the device.

Another advantage to A/B partitioning is the A or B active partition can be selected manually. For example, suppose the device software was updated successfully, but after the update the application is not running properly. This could be resolved easily and quickly by a manual rollback to the previous version located on the other partition.

Edge compute devices operate in distributed, remote, and often unsecured environments, making them prime targets for cyber threats.

Physically accessible, un-encrypted flash storage is one of the most obvious attack vectors against edge devices. These attacks are extremely easy to perform and the impact can easily escalate to complete compromises of devices and/or fleets, depending on whether shared credentials are present on individual devices. Intellectual property compromises are also likely to result, which may be more significant in certain circumstances (e.g. AI/ML).

Filesystem encryption is an absolute must for edge devices, and it must be done in such a way that the filesystem decryption keys are not easily available on local storage. This typically requires a secure element or a hardware security module.

Lastly, maintaining filesystem encryption across updates, partitions, and a fleet of devices is an ongoing challenge that can easily result in bricked devices. The core issue is key management—issuing, protecting, and rotating keys across multiple dimensions. And it goes without saying that reusing filesystem encryption keys across an entire fleet is strongly discouraged, as it creates significant security risks.

Even when the local storage is properly protected, updates can be intercepted, modified, or replaced with malicious software, leading to data breaches, system takeovers, or malware infections. Secure updates and secure boots are essential to maintain system integrity, data security, and operational reliability as they ensure that the device is executing the code that the developer intended.

Code signing and verification are required to ensure that only authorized updates from the trusted source are installed. It is also advisable to encrypt the updates when they cross untrusted networks or are residing on untrusted storage, though proper signature enforcement will prevent unauthorized modifications even if the updates are deciphered by a malevolent party.

The other half of secure updates is to make sure only authorized software can run from boot. If an attacker has physical access to the device, they could attempt to replace or modify the firmware or replace the OS with a compromised version. Secure boot works by checking all boot components for valid signatures, where each component verifies the next, ensuring integrity from the firmware up to the OS and the application. If an unauthorized modification is detected, secure boot prevents the system from booting.

One particularly risky challenge with secure boot is the need to re-sign the boot artifacts whenever they are updated, which requires the use of the private (signing) key. Storing a private key on the device is very problematic because if any malware gets a hold of it, it will likely gain full control of the boot chain. A fleet wide secure boot signing key is also ill-advised because a single key compromise would instantly compromise the entire fleet.  This is where typical update methods fail because securing an update also means securing the boot chain.

Secure boot and secure updates work together to ensure that only authorized firmware and updates are installed and run on the device.

Unlike cloud servers, edge compute devices operate in untrusted environments, making them more vulnerable to physical and cyber threats. Depending on the application and deployment conditions, software-based security alone may not be sufficient. Implementing hardware-based security is essential to protect sensitive data and ensure system integrity.

 

Trusted Platform Module (TPM)

A TPM is a hardware security chip initially designed for x86 computers. Every TPM device has a unique root key, can store and provide keys to decrypt filesystems or other data and can perform some cryptographic operations internally. TPMs are ubiquitous and well-understood. Including a TPM on an edge device is much better than no hardware secure element at all.

TPMs are not designed for fully unattended devices as they fall short in the following areas:

• Communication between the CPU and the TPM is neither encrypted nor protected from man-in-the-middle attacks. This means that keys may be sniffed quite easily or changed in-flight. A basic version of this attack is easily accomplished with an I2C sniffer for about $10. A sniffer attack can compromise the keys during the boot process, resulting in a defeat of the filesystem encryption and loss of any secrets stored in files.
• While some TPMs have physical perimeter monitoring, they cannot do so for a reasonable amount of time without device power. This presents an obvious attack vector, especially in combination with the I2C sniffing.
• TPMs are typically resettable, but may not be fully resettable, and usually do not have proper self-destruct mechanisms. Combined with lack of reliable physical security, this leads to additional attack vectors, though those are more sophisticated and may not be an issue for some projects.

The bottom line with TPMs is that they were designed for a much milder environment than what a typical edge device would encounter. The Raspberry Pi’s secure boot feature does not have TPM support, which creates another layer for the developer to address.

 

HSM (Hardware Security Module)

An HSM includes a wide category of devices that provide security services for the devices they protect. In general, HSMs offer extensive functionality ranging from platform protection to cryptographic acceleration.

Zymbit HSMs are specifically designed to protect Raspberry Pi class devices deployed in untrusted environments and address the major limitations of TPMs:

• All communication between the HSM and the host is encrypted with ephemeral keys that are derived by periodic ECDH exchanges, protecting both the data and preventing man-in-the-middle attacks.
• Zymbit HSMs have battery-backed physical perimeter protection and HSM removal monitoring that can run for multiple years, using a small button style battery. Combined with a suitable enclosure design, these features are essential to ensuring key destruction if the device’s physical security is compromised. Many of Zymbit HSMs are also equipped with accelerometers and temperature sensors that may be armed to self-destruct keys.
• Once fully armed, Zymbit HSMs are not resettable and cannot be used to create impostor devices.

Lastly, a hardware security module is only as good as its software. TPM software is primarily centered on cryptographic services, which are similar between most TPMs and HSMs.

Unlike typical TPMs/HSMs, Zymbit HSMs have integrated software update mechanisms, and which can be used with developer’s own OS update strategy, as well as with Zymbit’s Bootware. Bootware supports most Raspberry Pi OS combinations while maintaining a secure boot chain, secure update chain and A/B updates with filesystem encryption, all anchored to the HSM.

Deploying an edge computing solution with Raspberry Pi is both exciting and challenging. A robust remote software update strategy is essential for maintaining security, stability, and functionality in remote deployments. By ensuring updates are secure, efficient, and fail-safe, you can minimize downtime and keep your system running smoothly.

Key strategies such as A/B image partitioning, secure remote updates, and secure boot enhance reliability and security, improving the success rate of device deployments. With proper planning and the right tools, Raspberry Pi can be a powerful asset in your edge computing strategy.

Zymbit offers software and hardware solutions to enhance the security and reliability of Linux-based edge computing devices.

Zymbit’s Bootware integrates hardware security modules (HSMs) with device image management, ensuring highly secure and reliable deployments while maintaining an agile development environment. It includes secure boot, A/B image management, fail-safe updates with digital signing, device recovery, and encryption.