Fully segregated networks? Your dual-homed devices might disagree

TL;DR

• Using dual-homed devices as a segregation tool is not recommended as a security design solution
• Use dedicated hardware and robust firewalls to segregate networks to limit access to critical networks
• Proactively check for unintended exposure of network services and disable unnecessary services

Introduction

When we carry out security assessments in Operational Technology (OT) and Industrial Control System (ICS) environments, one thing that often stands out is the use of dual-homed devices.

In this blog post, I look at a recent OT / ICS engagement with a Critical National Infrastructure (CNI) client, which shows why dual-homed devices can be valuable targets for attackers.

What is a dual-homed device?

By design, dual-homed devices have multiple network interfaces (NICs) that are connected to two or more separate networks.

A dual-homed device may seem like a convenient solution, but the very same design that allows convenient communication between two networks can also provide a pathway for threat actors (and penetration testers) to move laterally in an organization’s infrastructure.

Bridging between different networks (often of varying trust levels) can undermine the very segregation intended to protect critical systems. While some OEM vendors historically promoted this topology, the current recommendation is to implement proper network segmentation.

Types of dual-homed device

When testing in OT / ICS networks, we typically find dual-homed devices falling into three main categories, with common but subtly different security implications:

• Windows hosts
• Linux hosts
• Embedded Systems

When these devices straddle network boundaries, host misconfigurations and security weaknesses can be exploited to gain network access to segregated networks.

• Host Access Control
  The nature of dual-homed hosts means compromise of the host grants network access to all connected network segments. It’s still mind-blowingly common (to me, anyway!) to find Embedded Systems using well-known default credentials. Within OT / ICS networks, legacy devices are often left unpatched for well-known exploitable vulnerabilities, such as authentication bypass or privilege escalation issues.

• Host-based Firewall
  It’s not uncommon to find host-based firewalls to be missing or disabled, particularly for Windows hosts and Embedded Systems. Even when a host firewall is enabled, overly permissive firewall rules often allow unintended network access.

• Unintended Service Exposure
  Services intended to be accessible only from a specific network can inadvertently be exposed to unintended, low-trust networks. A common example are management services such as SSH.

• Minimal Event Logging and Alerting Capabilities
  Some Embedded Systems often lack the capability to collect and collate event logs to a centralised location for SOC analysis. Even when they do have this capability, it’s not uncommon to find Windows and Linux hosts not configured to do so. Failure to detect and alert on potentially malicious activity can significantly increase the scale and risk of an attack and delay incident response efforts.

Example – OT / ICS Security Assessment for a CNI client operating in the energy sector

Summary

One of our CNI clients approached us to perform an onsite OT / ICS security assessment of one of their facilities.

We demonstrated to the client how their dual-homed devices could be exploited as pivot points to access segregated networks. Crucially, due to a combination of outdated firmware resulting in unintended exposure of network services and cleartext transmission of weak, reused and default passwords, these dual-homed devices could enable an attacker to compromise critical control and safety networks from untrusted network zones.

Background

When conducting security assessments in OT / ICS environments, we often recommend a bespoke and blended approach that combines aspects of consultancy and penetration testing. Collaborating closely with the client’s OT Engineering Teams provides access to their in-depth knowledge and understanding of their specific environment.

My colleague Andrew recently published a blog post on this topic, it’s worth a read.

The client OT Engineering Team shared network design and topology information which seemed to suggest the OT networks were well segmented following the Purdue Enterprise Reference Architecture (PERA). Firewalls were used to segment the OT networks based on varying trust levels – Remote Gateway network, DMZ network, Process network, Field Device network and SIS network. The firewalls had been securely configured to limit data flows to only those required for legitimate business functionality.

Figure 1: High level network topology showing segmented network using multiple dedicated firewalls

OT DMZ network

Testing in the DMZ network revealed a misconfigured GPS timeserver with default credentials still in use. Using these credentials, we connected to the device and discovered that the GPS timeserver had multiple NICs, including a secondary connection to the Field Device network.

Figure 2: Dual-homed GPS timeserver

Additionally, the firmware version installed on the GPS timeserver was outdated with several known vulnerabilities. A significant security issue was the lack of granularity when enabling SSH, as SSH could only be enabled or disabled for all interfaces collectively, without the option to enable it for a single interface. This granular functionality had since been implemented by the hardware vendor in a newer firmware version.

Unknowingly to the client, the current configuration resulted in the SSH service being enabled for both the DMZ network and the Field Device Network. The client’s OT Engineer explained that exposure of the SSH service on the Field Device Network was unintended and not required for any legitimate business function.

With administrative access to the GPS timeserver, we set up a SOCKS proxy over an SSH tunnel, allowing us to pivot between the Field Device network and the DMZ, or vice versa.

OT Process network

During testing in the OT Process network, multiple Linux-based PLCs were identified with telnet management services enabled. Telnet is a legacy shell service protocol that offers no security assurances, as traffic is exchanged in clear text. By intercepting the unencrypted telnet traffic, we captured the password used for administrative access. The password was extremely weak, memorable as it was based on the name of the OEM vendor application, and it was widely reused.

These PLCs also had secondary NICs with direct connectivity to the Field Device network. Crucially, and unbeknown to the client (and more worryingly, the OEM vendor was also unaware), the devices were configured such that the telnet service was enabled and exposed on all interfaces – including the Field Device interface.

Figure 3: Dual-homed PLCs

The unintended exposure of telnet services on the Field Device network allows an attacker administrative access to the PLC via the Field Device interface which can be used to reconfigure the PLC settings and potentially cause unsafe conditions within the facility. Additionally, pivoting through the PLC from the Field Device network would allow an attacker to gain network access to all other devices in the OT Process network.

OEM vendor network testing

During discussions with the site OT engineer, we learnt that multiple OEM vendor systems and networks were in use throughout the facility, separate from the main Process Control systems. Each of these had independent remote access solutions, allowing OEM vendors remote access for reporting, configuration, and troubleshooting of various non-Process Control systems. Our client’s OT Engineering team did not have full ownership of the assets in these OEM networks, so for obvious security reasons, these networks and systems were intended to remain segregated from the site’s Process Control systems.

However, our testing revealed that several OEM vendor solutions also had devices with secondary NICs, providing direct connectivity to the same Field Device network used by the GPS timeserver and PLCs.

Figure 4: Dual-homed vendor devices with connectivity to the Field Device network

The nature of the OEM vendor solutions meant that a supply-chain attacker could pivot through the dual-homed devices within the OEM vendor networks to gain access to the Field Device network. From here, they could then pivot through the GPS timeserver to access the OT DMZ network and pivot through a PLC to access the OT Process network.

Figure 5: Remote supply-chain attack of OT networks via shared Field Device network

SIS network testing

DMZ testing had previously identified an engineering workstation that was outdated and non-compliant with the organisation’s patching policies. This workstation was used by Engineers to configure the Safety Instrumented System (SIS), a critical set of hardware and software controls that are designed to monitor the facility and respond to unsafe conditions. Firewall rules had been configured to allow legitimate engineering traffic to traverse both firewalls.

A missing patch for a known vulnerability allowed us to compromise the engineering workstation and gain access to hosts within the SIS. By pivoting through various dual-homed devices, an attacker on an OEM vendor remote access network could target the SIS Engineering workstation to gain access to the SIS network and potentially tamper with its safety functions.

Figure 6: Engineering access to SIS from OT DMZ

Summary

At times during this engagement, it felt like we were involved in an OT / ICS CTF – it was genuinely like a tick list of some of the most common security weaknesses we still regularly find when testing OT/ICS environments.

• Outdated device firmware with known exploitable vulnerabilities
• Default, weak and reused passwords
• Unencrypted protocols allowing transmission of credentials in cleartext
• Unintended and unknown network services
• Dual-homed devices allowing bypass of security controls

We conducted this engagement collaboratively, using a hybrid approach of consultancy and bespoke targeted penetration testing. The client’s OT Engineering Team played an important role by sharing the detailed knowledge of their specific environment, helping us identify critical assets and understand operational workflows. While we saw plenty of good practices, a few significant weaknesses allowed us to bypass key firewall segmentation controls and move between the critical layers of the OT / ICS networks, potentially resulting in the security, operational status and safety of the facility being severely compromised.

Conclusion

So… what were the immediate outcomes?

• The default password for the GPS timeserver was changed, firmware was updated, and SSH access was restricted to only the necessary NICs.
• Our client reported the PLC telnet issue to their OEM vendor, who initially denied its possibility. However, after conducting their own internal tests, they confirmed the vulnerability and quickly developed a fix. In accordance with our client’s change control process, client testing began, and deployment across all affected facilities was scheduled.
• The SIS Engineering Workstation was updated, and the reasons for patching non-compliance were identified and addressed.
• For the OEM Vendor Networks, we recommended that our client urgently conduct a comprehensive review of all OEM vendor networks to identify all data flows into and out of their OT networks. Auditing and event logs should be captured to provide proactive network monitoring and detection capabilities. If an attacker gains access to their critical networks, they need to be alerted immediately!