The port monitor is an invaluable tool when commissioning communications with 3rd party systems. It is a function or device that allows engineers to read the data as it is transmitted over the buses and, if necessary, analyze the packets to immediately show the cause of the problem. Some systems and their development environments don't have detailed diagnostic tools and if the communication doesn't work as expected, it shows up at the application level only: data is not read, commands don't work, alarm mails are not sent... Then there is no choice but to dive deeper and follow the classic rules for diagnostics:
The problem is mostly at the last point: we need to conect directly to the interfaces, examine them, and record data which are being transmitted and received there. With serial buses such as RS232 or RS485, it is usually possible to connect physically directly to the line and read the data via a converter and monitoring program (terminal) in the PC. However, in the case of Ethernet communication, we may run into a problem: if one of the participants in the communication is not a PC, monitoring programs like Wireshark (https://www.wireshark.org) simply cannot be used. This is because the active network elements - routers and switches - only connect the source and destination ports to each other, and no communication is transmitted to the other ports. This significantly speeds up the data transfer in the network (the collision domain contains only two devices). The PC is, however, isolated from the communication between the PLC and the third-party system.
This also applies if the third party system is a service in the Internet, for example if the PLC communicates as a client using a TCP or UDP channel or serves as a server for data access using the web API - even then the monitoring PC is completely outside the data flow between the PLC and the third party device.
Ideally, the PLC allows to save a communication dump via the development environment or via the operating system itself. This is then analysed either "manually", if the lower layers of communication are filtered out for clarity, or by means of the appropriate program - usually the Wireshark format (.pcap files). We will now look at the various monitoring options in more detail.
The simplest option seems to be to use a hub. This now historical device basically electrically copies the signals from each port to the other ports (all devices connected to the hub are in the same collision domain). This means a much lower data throughput compared to a switch, but a PC with a monitoring program (typically Wireshark again) can then receive all the data exchanged between the PLC and the third-party system.
The problem is that hubs are not commercially available anymore, so either we bring an old one from the drawer, or buy a hardware device designed for packet sniffing – e.g. the Ethernet Tap https://ioninja.com/hardware/ethernet-tap.html - which even offers its own analytic software - or Ethernet Sniffer https://www.midbittech.com. However, a common configurable router can be used as well:
An affordable option is a configurable Mikrotik router which is able to scan network communication. The sniffer settings can be found in the Mikrotik menu under Tools > Packet Sniffer.
In the sniffer properties, the File name item sets the name of the file to which the sampled data will be written.
The Streaming tab allows you to set a specific address and port. The Sniffer then only stores data that matches the given parameters. This means saving memory space and increasing the clarity of the recordings. For short logs of tens of seconds (if we get an error within this time, such as when establishing a connection) it is not necessary to care about this, the logs can be filtered later in Wireshark.
The Filter tab allows to specify a particular Ethernet port to sniff the data from in the Interfaces option.
Start the sampling by pressing the Start button. When the sampling is finished (Stop button), the file with the sampled data can be downloaded in the Files menu:
The .pcap file can be opened in Wireshark and examined,see below.
Some PLCs (e.g. Domat IPLC510) allow access to their operating system. Then the tcpdump program can be used. The advantage of this method is that diagnostics can be performed remotely without having to install a hub or router to the PLC and connect another computer with a monitoring program. Terminal access to the PLC is good enough. When storing data, it is useful to filter the packets directly at the capture time to store only the data of interest and thus limit the size of the resulting file.
Note that te direct access to the operating systém of the PLC brings lots of risks and if possible at all, it shall be used only when testing rather than on live projects.
Example for BACnet (UDP protocol, port 47808):
Connect to the PLC using a TCP terminal. In PuTTY, run
tcpdump -i eth0 -s 1500 -w /tmp/out.pcap "udp and port 47808"
and the entire network communication which matches the filter criteria in quotation marks starts to be saved in a file named out.pcap. Other tcpdump options and parameters can be found e.g. at https://danielmiessler.com/study/tcpdump/.
The saving must be terminated by dropping the command using Ctrl-C. Be careful that the monitoring does not run so long that the disk is completely full! Before starting monitoring, check the amount of free disk space. It is recommended not to leave monitoring "unattended" so that the file does not grow uncontrollably.
In the tmp directory there is then the file out.pcap, which can be downloaded using WinSCP and opened normally on the PC with the Wireshark program. After downloading it is advisable to delete the file from the PLC.
As an example, let's take the older Domat IPLC510 process stations. For these, the above procedure is not necessary if we monitor communication on one of the standard communication channels. In the list of variables there are variables that enable logging of communication in the PLC. They are called LogPortMonitor....:
LogPortMonitor.CircularSize.[channel_name] – sets maximum file size with the dump. After this is exceeded, the oldest records are deleted. If set to the default of 0, the file size is not limited, and the PLC disk space may be filled completely!
LogPortMonitor.Format.[channel_name] – specifies the recorded data format. The most frequent are 2 (Hex) and 0 (ASCII), a 7-bit ASCII format (1) or decadic (3) can be chosen too.
LogPortMonitor.[channel_name] – if True, the communication is recorded into the file.
In this monitoring method, not all physical communication is stored in the PLC, but only the application layer data. For example, for the Modbus protocol, we only see Modbus-related data, not IP addresses and TCP ports, IP protocol flags, MAC addresses, etc. - i.e. we see the same as in the port monitor in the SoftPLC IDE. This is sufficient for application diagnostics. However, if we want to detect problems on the TCP layer, for example, such as connection establishment, socket maintenance, etc., or in communication where the PLC is a server, a scan of complete communication using the tcpdump command must be used - see previous point (or use external tapping – a hub, Mikrotik, etc.).
When searching for the problem cause, error messages shall be noted, or the communication flow shall be followed and we try to identify the moment when the data exchange stops making sense. In the following examples there are the most frequent problems which occur in the daily practice.
All screenshots are from the Wireshark software. In the lower part of the window, the complete contents of the packet are analyzed, but some information can be read in the upper part with the packet list. It is useful to first familiarize yourself with the filtering options of the listing, in order to limit the display to only relevant packets - those exchanged between the devices under investigation, and preferably only on a specific protocol. Then the listing is much clearer.
A response is returned from the destination address (here 192.168.1.10). However, an attempt to open a TCP connection to port 502 results in an error: a packet is returned (packet 12 in the listing, see the first column) with the attributes ACK (acknowledgement of receipt) and RST (connection termination), which means that the device is available on the IP address in the network, but no service is running on TCP port 502. Therefore, the Modbus TCP server is probably not enabled or configured on the target device.
The Modbus server reports an error – Illegal function when the client tries to write data. According to the documentation, the server supports only F16 (Write Multiple Registers) for writing, but here the client tried to write using the F06 (Write Single Register) function.
In the following packet, the server (address 192.168.1.10) responds to the request to send the value when changes are made (Subscribe COV) that this service is not known (unrecognized service). The client must therefore periodically query the value itself using the Read Property function - in this example it starts doing this automatically, as we see in the following packet No. 11.
In the first query to the DNS server, the PLC correctly receives the IP address of the mail server and then establishes communication with it. The name and password are sent in encrypted form (BASE64). However, the attempt to log in fails, the wrong login credentials were used. (The server does not specify the problem for security reasons.)
The PLC as a client establishes a connection to the e-mail server. In packet 372 (the number is in the first column from the left) the server offers possible ways of secure communication (... STARTTLS...), but the PLC tries to log in directly (packet 373, AUTH LOGIN). This attempt is rejected by the server and the connection is terminated, in packet 542 we see the explanation: unrecognized authentication type.
The last example is a bit more complicated, but it illustrates how useful the network communication dump is. The IEC 60870-5-104 protocol is used in the dispatching of power sources and appliances in the grid. The remote communication unit (RTU) was to receive a binary command with a time stamp, in the protocol this is the data type C_SC_TA_1. The dispatcher tried to command the unit repeatedly, but was unsuccessful - the unit at 192.168.1.82 always returned a negative response, so the command was not accepted:
Without scanning the communication, we would only know that the devices are not communicating, or that the unit is not responding to the commands. The dispatcher had no analytical tools at his disposal, he could only check that the connection with the unit was established and that he was sending a command to a specific (correct) object. Only after clarifying all possible parameters of the communication, it turned out that the dispatcher was using the Select/Execute command and not a direct command. This means that the first command (Select) tells the unit that a certain object will be set to a certain state, which the unit acknowledges, and the command is implemented only with the following telegram (Execute). However, the unit was expecting a direct command and responded to the Select command with a non-confirmation (Negative: True).
After the settings in the unit have been changed (Select/Execute was activated instead of direct commanding) the communication was correct and the command was accepted:
Network communication problems can be difficult to resolve because they can have a variety of causes - from badly crimped connectors to incorrect settings in the application program. A good development environment (IDE) can make the job easier thanks to port monitors and detailed error messages, but sometimes there is nothing more helpful than to look at the complete communication at all layers and use the dump (and of course knowledge of the principles of the respective communication) to find out why the two parties do not understand each other.
