Role of computer hardware in securing operating systems and hypervisors in trusted computing applications

Many software applications operate with “least privilege,” which means that the software is only granted minimal access to hardware, other applications, and other system resources. A separation of security context between an application and other resources such as operating systems and hypervisors ensures that less secure applications and software cannot access critical data from more secure trusted computing applications and reviews.

Highly sensitive data should be protected to ensure that only code that needs to operate on that data has access to it.

The responsibility for maintaining this kind of secure separation of applications lies with the operating system and the hypervisor, if one exists on the system. Think of an application that sits on top of a software stack; each lower layer of this stack must do its part to maintain application layer security.

At the bottom of the stack is hardware, which must be able to enforce access controls. On top of the hardware is an operating system or Type 1 hypervisor that handles access controls. Type 2 hypervisors, instead of running on hardware, run on an operating system.

For optimal protection, system designers should configure the system to ensure that the operating system or Type 1 hypervisor leverages all available hardware security to manage scheduling, resources, processes, and security at from the next layer. It is very difficult to create a secure application if these basic security essentials are missing.

Related: Reliable Computer Hardware Features to Maintain Cybersecurity During Operation

Let’s first consider the operating system and its vital role in ensuring system security on any moderately powerful processor.

For many years, operating systems have maintained a separation between kernel processes and user-space applications. In fact, the whole function of an operating system is to ensure the consistent operation of multiple applications running on a single piece of hardware. As part of their responsibility, operating systems have evolved to prevent or limit a malicious actor in one application from influencing other applications running concurrently.

As processors have become faster and more efficient, they have also become more complicated, which has made the responsibilities of operating systems even more complex. It is complicated, for example, for an operating system to manage cache management between tasks as processes move in and out of memory; the operating system must flush or invalidate any cache, which can be difficult and error-prone.

Add to that challenge factors such as direct memory access (DMA), side effects and security issues such as hammer blow, collapse and spectrum attacks, and the security responsibilities of the operating system become immense.

Hardware Security Capabilities

Most computing hardware today includes security features for the operating system or hypervisor. Indeed, many of these hardware features perform operations in supervisor mode. In processor-based trusted boot resources like Intel SGX or ARM TrustZone, the operating system must create and manage process and resource access to security domains.

Software engineers must design operating systems and the hypervisor to utilize the hardware’s built-in security features. The operating system and hypervisor typically run in privileged mode; they are the only entities able to use all the functionalities of the processing architecture.

Using an old operating system on new hardware can negate hardware security capabilities, and updating operating systems in military and aerospace systems can be costly. System designers should consider potential trade-offs between program risk and cost when determining when to insert new versions of operating systems during system refresh.

Related: Secure Boot: A Key Strategy for Ensuring Embedded Computer System Reliability

The operating system must also manage access to resources securely, in addition to maintaining security boundaries between processes and using hardware security to its full potential; it is not enough to maintain the separation of processes. It compromises trusted computing if one process can read another’s data from a device as data flows in and out.

Software engineers must design operating system software drivers that access devices with security in mind. They should understand the potential trade-offs between increasing access and availability of I/O resources, and maintaining and controlling access separation.

Some processors offer enhanced I/O management capabilities, such as an input-output memory management unit (IOMMU), which allows the operating system and software drivers to improve security while making the most of I/O resources.

Security aspects of hypervisors

A hypervisor manages virtualization resources to allow multiple operating systems to run on the same hardware at the same time. Operating systems and the hypervisor must work together when running in a virtualized environment to maintain a secure and reliable environment.

A set of guest operating systems operate in the system stack on top of type 1 hardware and hypervisor. Each of these guest operating systems operates the same as a single operating system when a hypervisor is not present. The type 1 hypervisor virtualizes all hardware resources and manages access to all operating systems running above it in the stack. VmWare ESX and Xen are examples of type 1 hypervisors.

In contrast, type 2 hypervisors run on top of another operating system, using that parent operating system to access hardware resources. The type 2 hypervisor virtualizes these resources to the guest operating systems that reside above it. VmWare Workstation and Oracle VirtualBox are examples of type 2 hypervisors.

Linux KVM is a hybrid hypervisor and runs in Linux kernel mode directly on the hardware, while using the Linux operating system architecture to manage virtualized resources.

Related: COTS-Based Trusted Computing: Getting Started in Next-Generation Critical Electronics

The type of hypervisor used determines where the basic security responsibilities for guest operating systems reside. The Type 1 hypervisor must use all security features available in the hardware to prevent a compromise in which one guest operating system leaks information or gains access to another guest operating system.

It is imperative that software engineers write host or parent operating systems to utilize the hardware security capabilities available with respect to Type 2 hypervisors; Type 2 hypervisor uses the host operating system to manage and control access to virtualized resources.

The operating system—or set of guest operating systems—separates user-space processes from supervisory processes. Each process that helps run the system has a defined interface that helps it communicate and interact with other processes. The operating system also ensures that processes operate within their defined roles and use interfaces to communicate. For example, it detects and prevents invalid operations and interface access, which prevents a faulty process from shutting down the entire system.

Operating system security

System designers must give the most stringent security requirements to the operating system or hypervisor when considering the layers of software running on the stack.

The strictest security should reside at the lowest level because of the risk of security failure at this level. Designers should analyze the operating system and hypervisor for bugs that could allow inappropriate access. If the lowest-level operating system or hypervisor has a bug, a failure can compromise all trusted systems that use that operating system or hypervisor. It could also allow elevation of privilege, which could exploit an application to compromise all other processes running in the operating system or hypervisor.

The risk of failure at the operating system or hypervisor level is normally much more limited, but system designers still need to review applications for security. An application-level failure can compromise an application or a guest operating system, but the next level should normally prevent such a compromise from spreading further into the system.

Related: Trusted Computer Hardware: What You Need to Know

It is very helpful to discuss with the hardware vendor as early as possible in the design cycle to better understand the security requirements of the system. A clear understanding will help designers give proper importance to security capabilities when evaluating operating systems or hypervisors.

Selecting the most recent operating system or hypervisor with built-in security will usually be the best choice. Better yet, minimize potential exploits and security vulnerabilities when downsizing your chosen operating system or hypervisor feature set.

Embedded systems often provide the ability to configure the operating system to remove unnecessary features, processes, and libraries. Adapting the operating system to minimize potential exploits is also good security practice.

More information about Curtiss-Wright’s Trusted COTS (TCOTS) program for protecting critical technologies and data in deployed embedded computing systems is available online at

David Sheets is a Senior Principal Security Architect at the Curtiss-Wright Corp. division. Defense Solutions in Ashburn, Virginia.

Ready to make a purchase? Search the Military and Aerospace Electronics Buyer’s Guide for companies, new products, press releases and videos

Comments are closed.