Problem:
Niagara Workbench or Niagara Network FOX connection unsuccessful. Customers need guidance on diagnosing and troubleshooting Fox connectivity issues.
Solution:
Workbench uses FOX when connecting to a Niagara Station. Workbench is the client to the remote station's server.
Fox is also used by the Niagara Network for all Niagara Station connections. When troubleshooting a Niagara Station connection it is important to remember that each station can be a client and server of other stations. In this document the Client is the station initiating the Niagara Network Fox connection. The remote station is the station being connected to by the Client. The remote station is acting as the server to the client station.
The majority of the information contained in this article pertains to both N4 and AX. Any differences between Niagara 4 and Niagara AX will be noted.
Each station involved in a Niagara4 FOX connection must contain the ‘FoxService’ (located under the ‘Services’ folder) and the ‘NiagaraNetwork’ (located under the ‘Drivers’ folder). Note that NiagaraAX has the FoxService located under the 'NiagaraNetwork'.
Fox Service – This is where you configure Fox and Foxs connectivity properties such as TCP port and enabled.
Niagara Network - Under the Niagara Network are created the Niagara Stations that represent the other remote stations involved in the FOX connection(s). The clients in the Niagara Stations will be connecting with the Fox Service in the remote stations.
Often FOX connectivity failure is due to incorrect configuration of the NiagaraStation’s properties.
Another major cause of FOX connection failure is the Fox ports (defaults are 1911 and 4911) being blocked by a firewall.
Niagara N4 and AX Version Compatibility-
You must be running an 'AX Security Update' version of AX when connecting an AX station to N4 using Niagara Network (Niagara Driver). Also, the Cipher Suite Group setting in the Fox Service must be set to 'Supported' when connecting any AX version prior to 3.8u3 (3.8.311).
Note that currently the Security Update versions noted below will connect via Niagara Network. Future releases of Niagara4 could contain framework changes that break Niagara Network connectivity to some AX versions. The intention is that AX 3.8 will always have a version that will connect to N4.
NiagaraStation properties-
The key properties of the NiagaraStation involved in a FOX connection are listed below (also see Figure 2):
This section will cover key properties to check when having Fox connection issues.
Reciprocal Niagara Station component
Note that when you create a Niagara Station under the Niagara Network of any station, when it connects to the remote station represented by this Niagara Station, the remote station automatically places a Niagara Station under its Niagara Network. By default this Niagara Station is disabled. It is not configured except for the ‘Address’ property. It should be noted that at times the station will put its ‘Host Name’ (not its IP) into the address property of the newly created Niagara Station. When you configure this Niagara Station in the remote station you should remove the host name and replace it with the IP address of the remote computer.
Troubleshooting a Workbench connection failure-
When dealing with a station connection using Workbench, you need to know what ports are in use by the station's Fox Service. Is the station using FOX, FOXS or both? Are the ports in use open on the computer's firewall and the building network?
Sometimes the Workbench Nav side bar window will contain corrupt or incorrect remote hosts. If a node in the Nav fails to connect, first try re-entering the credentials. If re-entering the credentials is unsuccessful, try removing the Nav node and then using 'File->Open->Open Station' and start fresh. From a Windows Command Prompt, try 'Pinging' the IP to make sure there is network connectivity between the computer and remote host.
Having more than one host in the Workbench Nav with the same station name can cause the connection to hang when you attempt to connect using one of those hosts. To work around this you must remove hosts from the Nav until only one remains for each station name.
Troubleshooting a Niagara Network connection failure-
Things to check when there are Fox connectivity problems between Niagara Stations. Much of the troubleshooting can be performed using Workbench. This troubleshooting list assumes that any physical LAN issues affecting connectivity have been addressed.
Note the Niagara Stations circled in red. These Niagara Stations represent the other stations involved in the Fox connection with the station. These Niagara Stations will be clients to the remote station they represent. And these remote stations will be clients to the Fox Service of this station. You can view all server connections on the Property sheet of the Fox Service (see Figure 8).
Figure 1
Key properties involved in a Fox connection.
Note the ‘Address’ property. Be aware that when you create a Niagara Station and it connects to the other station, that station automatically places a Niagara Station into the Niagara Network of the station to which is just connected. If this station was a Windows computer, often the Niagara Station address will contain the ‘Host Name’ and not the IP address of the computer. If the client station is not set to do DHCP the Fox connection back to the station on the computer will fail. Changing the Host Name to the IP address of the computer will allow the connection to be successful.
Figure 2
Always check the ‘Last Failure Cause’ in the client connection property sheet.
Figure 3
If the Last Failure Cause is “javax.net.ssl.SSLException: Certificate private key for exemption <137.19.60.179:4911> has changed”
Then you need to delete the certificate in the client’s Certificate Manager ‘Allowed Hosts’ and attempt a new connection. After deleting the certificate, simply right click the Niagara Station in the property sheet and perform the action ‘ping’. This will cause the remote station to reply and resubmit its certificate.
Figure 4
Check the Certificate Manager ‘allowed hosts’ and approve the new certificate. Connection should now be successful.
Figure 5
New certificate in the client has been approved.
Figure 6
Client connection now successful.
Figure 7
The server connections can be viewed in the Fox Service property sheet. You can verify a Niagara Station has connected by looking at the State of the server connection.
Figure 8
Problem:
Supervisor will not stay connected to subordinate JACE hosts. Client connection state goes from connected to not-connected. Client Connection's Last Fault Cause is "java.io.EOFException: EOF". Points may be stuck in pending-subscribe state.
Solution:
This problem state can occur when there's an extra instance of the same station running on the network.
Login to the Station on one of the affected subordinate hosts. Navigate to Services>FoxService>ServerConnections. Find and expand the connection for the Supervisor Station connecting to the JACE, as seen in the image below:
If you observe that the blob-value in the state property keeps changing, in sync with the value for the last-login-address property also changing, then you know that two hosts with the same Station name, are both trying to stay connected simultaneously. This problem should be resolved by using the two changing values of the last-login-address property, to identify both hosts, and shut down the one that should not be connecting.
Problem:
Station object in Niagara Network won't ping.
On the Client Connection:
Solutions:
Option 1) Restarting the Station will fix this.
Option 2) On the Station object, modify the name, IP address, and set enabled=false . Create a new Station object that matches the original configuration, and invoke ping, insuring that ping is successful. If needed, make an exception for self-signed certificate in Certificate Manager, and invoke ping again. Once connection is established, delete the new Station object, and restore the original Station object properties, and then invoke the ping action.
Problem:
Fox connection unstable due to incorrect concurrent-session setting on User Object.
Solution:
The Allow Concurrent Sessions property should normally be set to true on the User that is used for Station to Station connections. When set to false, a new session connection will invalidate the previously opened session for that user, if one exists. If a second Niagara Station attempts a connection using the same User name, it can result in EOFException messages and other undesirable symptoms for the Station that previously had a session open.
Problem:
Fox connection unstable on VPN networks reporting invalid MTU
On several sites, customers have encountered unstable Fox connections on sites with networks (primarily VPN connected networks) in scenarios where the reported MTU size is not valid.
Fox is a protocol that uses a subscriber/publisher mechanism to optimize use of network bandwidth for points subscriptions. The publisher updates the points subscriber in batches, using packets that match the maximum transmission unit (MTU size). When a network router advertises an MTU size greater than what it can actually handle, Fox communications fail in unexpected ways. This may manifest problems like, failure to replicate in the case of Enterprise Security.
The following are some examples of Station Output that has a correlation with this problem:
Solution:
Ultimately, the solution will be provided by the site's IT department, or whoever manages the router that reports the MTU size to the networks that the Niagara hosts reside on. It must report the correct MTU size. But you can test to see whether or not this is truly the source of your problems.
Niagara Workbench or Niagara Network FOX connection unsuccessful. Customers need guidance on diagnosing and troubleshooting Fox connectivity issues.
Solution:
Workbench uses FOX when connecting to a Niagara Station. Workbench is the client to the remote station's server.
Fox is also used by the Niagara Network for all Niagara Station connections. When troubleshooting a Niagara Station connection it is important to remember that each station can be a client and server of other stations. In this document the Client is the station initiating the Niagara Network Fox connection. The remote station is the station being connected to by the Client. The remote station is acting as the server to the client station.
The majority of the information contained in this article pertains to both N4 and AX. Any differences between Niagara 4 and Niagara AX will be noted.
Each station involved in a Niagara4 FOX connection must contain the ‘FoxService’ (located under the ‘Services’ folder) and the ‘NiagaraNetwork’ (located under the ‘Drivers’ folder). Note that NiagaraAX has the FoxService located under the 'NiagaraNetwork'.
Fox Service – This is where you configure Fox and Foxs connectivity properties such as TCP port and enabled.
Niagara Network - Under the Niagara Network are created the Niagara Stations that represent the other remote stations involved in the FOX connection(s). The clients in the Niagara Stations will be connecting with the Fox Service in the remote stations.
Often FOX connectivity failure is due to incorrect configuration of the NiagaraStation’s properties.
Another major cause of FOX connection failure is the Fox ports (defaults are 1911 and 4911) being blocked by a firewall.
Niagara N4 and AX Version Compatibility-
You must be running an 'AX Security Update' version of AX when connecting an AX station to N4 using Niagara Network (Niagara Driver). Also, the Cipher Suite Group setting in the Fox Service must be set to 'Supported' when connecting any AX version prior to 3.8u3 (3.8.311).
Note that currently the Security Update versions noted below will connect via Niagara Network. Future releases of Niagara4 could contain framework changes that break Niagara Network connectivity to some AX versions. The intention is that AX 3.8 will always have a version that will connect to N4.
AX Security Update versions-
AX 3.5.406
AX 3.6.407 (or greater)
AX 3.7.108 (or greater)
AX 3.8.111 (or greater)
AX 3.6.407 (or greater)
AX 3.7.108 (or greater)
AX 3.8.111 (or greater)
NiagaraStation properties-
The key properties of the NiagaraStation involved in a FOX connection are listed below (also see Figure 2):
- Enabled – The Niagara station must be enabled.
- Address – This is the IP address of the remote station.
- Port – TCP port used. This property is part of the ‘Client Connection’. Make sure that the port is open through any firewall on the PC or LAN.
- Fox port default is 1911.
- Foxs port default is 4911.
- Use Foxs – This property is also part of the ‘Client Connection’. If this is set to ‘True’ then a secure FOX connection is being used. When using Foxs, the Fox Service of the remote station must have ‘Foxs Enabled’ property set to ‘True’ and the ‘Port’ property in the NiagaraStation must match the ‘Foxs Port’ set in the Fox Service. Also note that in the Certificate Manager, under the ‘Allowed Hosts’ tab, you must approve the certificate that has been provided by the remote station to which you are attempting connection. The first time a NiagaraStation attempts a secure connection with another station, that station will send its SSL certificate to the station attempting connection. You must manually approve this certificate. Note that Foxs is available in AX-3.7 and greater.
- Username – This is the user in the remote station. Make sure this user has been assigned a role with permissions. Also, if an AX station is attempting connection to an N4 station, the user in the N4 station must be assigned the AXDigest Scheme.
- Password – Remote station user’s password.
This section will cover key properties to check when having Fox connection issues.
- Fox port - This is the port used for making non-secure TCP Fox connections. Default port is 1911. Make sure that this port is not being blocked by internal or external firewalls. Also make sure that all station clients are configured to use the correct port number.
- Fox Enabled - Must be set to True to use non-secure Fox connections.
- Foxs Port - The port used for making secure TCP Fox connections. Default port is 4911. Make sure that this port is not being blocked by internal or external firewalls. Also make sure that all station clients are configured to use the correct port number.
- Foxs Enabled - Must be set to True to use the secure Foxs connections.
- Foxs Only - If set to True then only secure Foxs connections would be used, even if you have set 'Fox Enabled' to true.
- Foxs Min Protocol - The default of TLSv1.0+ is typically fine.
- Cipher Suite Group - Strong cipher suites were implemented in AX 3.8u3 and N4.4u1. Recommended is the default to use strong ciphers. It must be noted that if you are connecting Niagara AX stations running a version prior to AX 3.8u3, this property must be set to 'Supported' in the N4 station. When this property is set to 'Recommended' only strong ciphers are available and an older version of AX station will not be able to connect to an N4 station version N4.4u1 or above that is set to 'Recommended'.
- Note - An 'IncompatibleVersionException' Exception "NiagaraAX Station Connections Unsupported" is thrown when an AX station attempts a Fox connection to an N4 station and the wrong Cipher Suite Group has been selected. When you see this exception change the Cipher Suite Group to 'Supported'.
- Foxs Cert - The "alias" for the server certificate in the host platform's "key store" to use for any Foxs connection. The default is the "tridium" self-signed certificate, which is automatically created when SSL is first loaded (by presence of certain modules and proper host licensing). If other certificates are in the host platform's key store, you can use the drop-down control to select another one instead. Note that you will need to open the Certificate Manager of the station, select the 'Allowed Hosts' tab and approve all hosts that have attempted a connection and have an 'Approval' of 'no'.
- Request Timeout - Time to wait for a response before assuming a connection is dead. Normally the default of 1 minute is fine. You can increase this value if you determine a longer timeout is needed.
- Socket Option Timeout - Time to wait for on a socket read before assuming a connection is dead. Normally the default of 1 minute is fine. You can increase this value if you determine a longer timeout is needed.
- Authentication Policy - When connecting an N4 station to an AX station this FoxService policy in the AX station should be set to 'Digest'. Note that if you are viewing AX station running version '3.6' or below using Workbench '3.7' or above, some of the FoxService properties show that do not actually exist in a '3.6' station. Such as FoxS, which was not available until 'AX 3.7'. The Authentication Policy will show on the page and changes made will get updated correctly even if using newer AX Workbench.
Reciprocal Niagara Station component
Note that when you create a Niagara Station under the Niagara Network of any station, when it connects to the remote station represented by this Niagara Station, the remote station automatically places a Niagara Station under its Niagara Network. By default this Niagara Station is disabled. It is not configured except for the ‘Address’ property. It should be noted that at times the station will put its ‘Host Name’ (not its IP) into the address property of the newly created Niagara Station. When you configure this Niagara Station in the remote station you should remove the host name and replace it with the IP address of the remote computer.
Troubleshooting a Workbench connection failure-
When dealing with a station connection using Workbench, you need to know what ports are in use by the station's Fox Service. Is the station using FOX, FOXS or both? Are the ports in use open on the computer's firewall and the building network?
Sometimes the Workbench Nav side bar window will contain corrupt or incorrect remote hosts. If a node in the Nav fails to connect, first try re-entering the credentials. If re-entering the credentials is unsuccessful, try removing the Nav node and then using 'File->Open->Open Station' and start fresh. From a Windows Command Prompt, try 'Pinging' the IP to make sure there is network connectivity between the computer and remote host.
Having more than one host in the Workbench Nav with the same station name can cause the connection to hang when you attempt to connect using one of those hosts. To work around this you must remove hosts from the Nav until only one remains for each station name.
Troubleshooting a Niagara Network connection failure-
Things to check when there are Fox connectivity problems between Niagara Stations. Much of the troubleshooting can be performed using Workbench. This troubleshooting list assumes that any physical LAN issues affecting connectivity have been addressed.
- Often the fix is to simply re-enter the client connection password. The password could be lost during an upgrade or station install if the security folder is lost or corrupt.
- Check with IT and make sure that the Fox ports in use are not being blocked by any firewall (default 1911 and 4911). Also check the Supervisor PC for virus protection software and Windows Firewall. You may need to add an inbound rule to Windows Firewall to open the Fox port(s).
- You see the error in the Station Dialog ERROR: MessageReader: Expected 'f', got <something else> . This symptom is is consistent with attempting to connect to a foxs port with a fox connection. It can also occur when our device is scanned by a cybersecurity scanner (or a malicious tool probing for vulnerabilities). Insure that the Fox ports in all stations are correct. Make sure that foxs=true/false is set correctly. If the issue persists, then you should use Wireshark or a similar tool to determine what network traffic corresponds with the error message, and what host it is coming from.
- Using Workbench, open the client side Niagara Network property sheet. Expand the Niagara Station having connectivity issues. Review the Niagara Station properties listed above, making sure they are configured correctly. Verify that the ‘Address’ property is correct. It is recommended that the ‘Address’ property contain an IP address and not a Host Name.
- Using Workbench, open the server station (the remote station to which you are trying to connect via Niagara Network). Open the User Manager and verify the that the Fox user (#5 above) is correctly setup in the remote station. Verify the user in the remote station has an Authentication Scheme and a Role.
- Using Workbench, on the remote station, open the property sheet of the Fox Service. If using Foxs, make sure that 'Foxs enabled' is set to true and also make sure that the TCP port settings match.
- Duplicate station names will generate 'Rejected' messages in the application director, which is a clue that two or more stations are using the same station name. These duplicate stations will fight for a Fox connection, kicking each other out. Look in station 'Spy'. Click on 'fox', then 'Fox log index'. Scroll down the list of stations listed in the log and look for duplicate station names [inside square brackets] having different IP addresses. You must change the station names so they no longer match. This will involve saving the Jace station to your local PC, renaming the folder and then re-installing the station.
- Duplicate stations within the same station's NiagaraNetwork can cause failure to load the Niagara Station Manager and also failure to expand the nav tree.
- You will see a 'javax.baja.naming.UnresolvedException: e6b27b' exception (the characters after the colon will vary).
- You need to find and remove the duplicate station(s).
- Use this BQL to find the stations - 'station:|slot:/|bql:select * from niagaraDriver:NiagaraStation' (control key + L and paste in the ord).
- After the duplicate station is removed you will need to restart the station for the Niagara Station Manager view to display correctly.
- If using Foxs make sure the certificate in the server station is valid. Using Workbench open the station's Certificate Manager and view the certificate. Check the ‘Not Before’ and ‘Not After’ dates of the certificate. Verify that the certificate under 'Allowed Hosts' has been approved.
- If wanting to use Fox (not FoxS) and unable to connect, use FoxS in Workbench to open the Fox Service property sheet and make sure that 'Fox Enabled' is true and the correct TCP port is set (default is 1911). Also verify that 'Foxs only' is set to FALSE. Again, make sure the Fox TCP port is not being blocked.
- You can look in the Workbench Application Director and verify that the correct Fox ports are showing up under 'Details'.
- Check the Station Date and Time and make sure they are correct and that the certificate times are valid. A valid certificate is required for FoxS connectivity. Use Workbench, right click on the station in the tree, 'Views', 'Station Summary'. Check current time. You can also look in Services, Platform Services.
- Check the ‘Last Failure Cause’ in the Client Connection property sheet. This can be helpful in finding the reason for failure to connect.
- Using the property sheet of the Niagara Station that is acting as the client, right click on the ‘Client Connection’ and perform the action ‘Manual Connect’. You can also select the Niagara Station and perform the action ‘Ping’. This can quickly test connectivity to see if your changes have corrected the problem.
- If you are troubleshooting AX to AX fox connections, verify that the 'Legacy Authentication' property of the Fox Service matches in all stations.
- A more advanced check is using a serial shell connection. From the Main Menu in serial shell, try pinging the remote host to verify LAN connectivity. Correct any LAN issues or TCP/IP settings.
- PENDING SUBSCRIBE/UNSUBSCRIBE is typically caused by having more than one Fox Service in a station. Use the Batch Editor to search for 'Custom Type', 'Fox', 'FoxService'. AX 'FoxService' lives under the Niagara Network. N4 'FoxService' lives in the 'Services' folder. Remove any extra 'FoxService' from the station and restart.
- 'java.io.IOException: Could not acquire peer certificate to process exemption.' Generally seen when you ask for an encrypted connection but use the non-secure port.
Also could happen if the port needed is blocked. This error can happen when a legacy JACE (Jace600, Jace300) is slow to process a request for a certificate when using FoxS (TLS). In the last case adding a delay in the Supervisor's 'system.properties' file will often correct the problem.- Add this line and restart the Supervisor (time is milliseconds) - niagara.socketFactory.hardTimeout=75000
- The 'system.properties' is located in the 'defaults' folder where Niagara build is installed.
- Start with 75000 milliseconds. Might increase the time slightly by a few more milliseconds if needed.
- Seeing 'SEVERE [08:22:26 12-May-21 EDT][fox] Failsafe timeout on read 129610ms > 60000ms' in station dialog and fox connections closing. Followed by 'javax.baja.xml.XException: java.io.IOException: circuit closed'
- This could be caused by an actual network problem such as bad switch or NIC card.
- Using Wireshark to capture network traffic can be helpful.
- Look in station Spy, fox, log index and examine the fox connections.
Note the Niagara Stations circled in red. These Niagara Stations represent the other stations involved in the Fox connection with the station. These Niagara Stations will be clients to the remote station they represent. And these remote stations will be clients to the Fox Service of this station. You can view all server connections on the Property sheet of the Fox Service (see Figure 8).
Figure 1
Key properties involved in a Fox connection.
Note the ‘Address’ property. Be aware that when you create a Niagara Station and it connects to the other station, that station automatically places a Niagara Station into the Niagara Network of the station to which is just connected. If this station was a Windows computer, often the Niagara Station address will contain the ‘Host Name’ and not the IP address of the computer. If the client station is not set to do DHCP the Fox connection back to the station on the computer will fail. Changing the Host Name to the IP address of the computer will allow the connection to be successful.
Figure 2
Always check the ‘Last Failure Cause’ in the client connection property sheet.
Figure 3
If the Last Failure Cause is “javax.net.ssl.SSLException: Certificate private key for exemption <137.19.60.179:4911> has changed”
Then you need to delete the certificate in the client’s Certificate Manager ‘Allowed Hosts’ and attempt a new connection. After deleting the certificate, simply right click the Niagara Station in the property sheet and perform the action ‘ping’. This will cause the remote station to reply and resubmit its certificate.
Figure 4
Check the Certificate Manager ‘allowed hosts’ and approve the new certificate. Connection should now be successful.
Figure 5
New certificate in the client has been approved.
Figure 6
Client connection now successful.
Figure 7
The server connections can be viewed in the Fox Service property sheet. You can verify a Niagara Station has connected by looking at the State of the server connection.
Figure 8
Problem:
Supervisor will not stay connected to subordinate JACE hosts. Client connection state goes from connected to not-connected. Client Connection's Last Fault Cause is "java.io.EOFException: EOF". Points may be stuck in pending-subscribe state.
Solution:
This problem state can occur when there's an extra instance of the same station running on the network.
Login to the Station on one of the affected subordinate hosts. Navigate to Services>FoxService>ServerConnections. Find and expand the connection for the Supervisor Station connecting to the JACE, as seen in the image below:
If you observe that the blob-value in the state property keeps changing, in sync with the value for the last-login-address property also changing, then you know that two hosts with the same Station name, are both trying to stay connected simultaneously. This problem should be resolved by using the two changing values of the last-login-address property, to identify both hosts, and shut down the one that should not be connecting.
Problem:
Station object in Niagara Network won't ping.
On the Client Connection:
- lastConnectTime and lastDisconnectTime are hours or days old, despite recent attempts to ping
- lastFaultCause is consistent with a timeout
Solutions:
Option 1) Restarting the Station will fix this.
Option 2) On the Station object, modify the name, IP address, and set enabled=false . Create a new Station object that matches the original configuration, and invoke ping, insuring that ping is successful. If needed, make an exception for self-signed certificate in Certificate Manager, and invoke ping again. Once connection is established, delete the new Station object, and restore the original Station object properties, and then invoke the ping action.
Problem:
Fox connection unstable due to incorrect concurrent-session setting on User Object.
Solution:
The Allow Concurrent Sessions property should normally be set to true on the User that is used for Station to Station connections. When set to false, a new session connection will invalidate the previously opened session for that user, if one exists. If a second Niagara Station attempts a connection using the same User name, it can result in EOFException messages and other undesirable symptoms for the Station that previously had a session open.
Problem:
Fox connection unstable on VPN networks reporting invalid MTU
On several sites, customers have encountered unstable Fox connections on sites with networks (primarily VPN connected networks) in scenarios where the reported MTU size is not valid.
Fox is a protocol that uses a subscriber/publisher mechanism to optimize use of network bandwidth for points subscriptions. The publisher updates the points subscriber in batches, using packets that match the maximum transmission unit (MTU size). When a network router advertises an MTU size greater than what it can actually handle, Fox communications fail in unexpected ways. This may manifest problems like, failure to replicate in the case of Enterprise Security.
The following are some examples of Station Output that has a correlation with this problem:
| java.io.EOFException: EOF at com.tridium.fox.message.MessageReader.read(MessageReader.java:119) at com.tridium.fox.message.MessageReader.consume(MessageReader.java:389) at com.tridium.fox.message.FoxMessage.readValue(FoxMessage.java:83) at com.tridium.fox.session.FoxCircuit.readMessage(FoxCircuit.java:86) at com.tridium.fox.sys.broker.BBrokerChannel.syncFromMaster(BBrokerChannel.java:2510) at com.tridium.fox.sys.broker.BBrokerChannel.circuitOpened(BBrokerChannel.java:278) at com.tridium.fox.sys.BFoxConnection.circuitOpened(BFoxConnection.java:464) at com.tridium.fox.session.SessionCircuits$ServiceThread.run(SessionCircuits.java:442) at java.lang.Thread.run(Thread.java:748) |
| SEVERE [11:53:46 14-Apr-21 EDT][fox.point] Sending batches java.io.InterruptedIOException at com.tridium.fox.session.SessionBedroom.sleep(SessionBedroom.java:70) at com.tridium.fox.session.FoxSession.sendSync(FoxSession.java:1096) at com.tridium.fox.sys.BFoxConnection.sendSync(BFoxConnection.java:521) at com.tridium.fox.sys.BFoxChannel.sendSync(BFoxChannel.java:340) at com.tridium.nd.point.BPointChannel.subscribe(BPointChannel.java:264) at com.tridium.nd.point.ClientWorker.sendSub(ClientWorker.java:279) at com.tridium.nd.point.ClientWorker.workIt(ClientWorker.java:150) at com.tridium.nd.point.ClientWorker.run(ClientWorker.java:89) at com.tridium.nd.BCyclicThreadPoolWorker$Work.run(BCyclicThreadPoolWorker.java:735) at javax.baja.util.ThreadPoolWorker$WorkerThread.run(ThreadPoolWorker.java:279) |
| java.io.InterruptedIOException at com.tridium.fox.session.SessionBedroom.sleep(SessionBedroom.java:70) at com.tridium.fox.session.FoxSession.sendSync(FoxSession.java:1110) at com.tridium.fox.sys.BFoxConnection.sendSync(BFoxConnection.java:521) at com.tridium.fox.sys.BFoxChannel.sendSync(BFoxChannel.java:349) at com.tridium.fox.sys.BFoxChannel.initializeSharedKey(BFoxChannel.java:857) at com.tridium.fox.sys.BFoxChannel.fwSessionOpened(BFoxChannel.java:143) at com.tridium.fox.sys.BFoxChannelRegistry.sessionOpened(BFoxChannelRegistry.java:178) at com.tridium.fox.sys.BFoxChannelRegistry.sessionOpened(BFoxChannelRegistry.java:156) at com.tridium.fox.sys.BFoxConnection.sessionOpened(BFoxConnection.java:412) at com.tridium.fox.sys.BFoxClientConnection.sessionOpened(BFoxClientConnection.java:558) at com.tridium.fox.session.FoxSession.start(FoxSession.java:474) at com.tridium.fox.session.Tuner.openClient(Tuner.java:371) at com.tridium.fox.session.Fox.open(Fox.java:457) at com.tridium.fox.sys.BFoxClientConnection$ConnectPrivilegedAction.run(BFoxClientConnection.java:722) at java.security.AccessController.doPrivileged(Native Method) at com.tridium.fox.sys.BFoxClientConnection.connect(BFoxClientConnection.java:604) at com.tridium.fox.sys.BFoxClientConnection.doManualConnect(BFoxClientConnection.java:1091) at auto.com_tridium_fox_sys_BFoxClientConnection.invoke(AutoGenerated) at com.tridium.sys.schema.ComponentSlotMap.invoke(ComponentSlotMap.java:1906) at com.tridium.sys.schema.ComponentSlotMap.invoke(ComponentSlotMap.java:1871) at javax.baja.sys.BComponent.invoke(BComponent.java:1223) at com.tridium.fox.sys.broker.BBrokerChannel.invoke(BBrokerChannel.java:1877) at com.tridium.fox.sys.broker.BBrokerChannel.process(BBrokerChannel.java:245) at com.tridium.fox.sys.BFoxConnection.process(BFoxConnection.java:452) at com.tridium.fox.session.SessionDispatcher.dispatch(SessionDispatcher.java:84) at com.tridium.fox.session.SessionDispatcher.run(SessionDispatcher.java:63) at java.lang.Thread.run(Thread.java:748) |
| Failed [08:54:43 25-May-22] Job Failed javax.baja.sys.BajaRuntimeException at com.tridium.fox.sys.BFoxSession.toException(BFoxSession.java:1217) at com.tridium.fox.sys.broker.BFoxComponentSpace.handleToSlotPath(BFoxComponentSpace.java:161) at com.tridium.fox.sys.broker.BFoxComponentSpace.loadByHandle(BFoxComponentSpace.java:133) at javax.baja.sync.BProxyComponentSpace.findByHandle(BProxyComponentSpace.java:78) at javax.baja.space.BComponentSpace.findByHandle(BComponentSpace.java:544) at javax.baja.space.BComponentSpace.resolveByHandle(BComponentSpace.java:573) at javax.baja.space.BHandleScheme.resolve(BHandleScheme.java:73) at javax.baja.naming.BOrdScheme.resolve(BOrdScheme.java:107) at javax.baja.naming.BOrd.resolve(BOrd.java:274) at javax.baja.naming.BOrd.resolve(BOrd.java:250) at javax.baja.naming.BatchResolve.resolve(BatchResolve.java:182) at javax.baja.naming.BatchResolve.resolve(BatchResolve.java:154) at com.tridium.exporttags.BSupervisorJoinJob.run(BSupervisorJoinJob.java:388) at com.tridium.exporttags.BJoinJob.run(BJoinJob.java:141) at javax.baja.util.Worker.process(Worker.java:168) at javax.baja.util.Worker$Processor.run(Worker.java:141) at java.lang.Thread.run(Thread.java:748) Caused by: javax.baja.net.NotConnectedException at com.tridium.fox.sys.broker.BFoxComponentSpace.channel(BFoxComponentSpace.java:256) javax.baja.net.NotConnectedException at com.tridium.fox.sys.broker.BFoxComponentSpace.channel(BFoxComponentSpace.java:256) at com.tridium.fox.sys.broker.BFoxComponentSpace.handleToSlotPath(BFoxComponentSpace.java:156) at com.tridium.fox.sys.broker.BFoxComponentSpace.loadByHandle(BFoxComponentSpace.java:133) at javax.baja.sync.BProxyComponentSpace.findByHandle(BProxyComponentSpace.java:78) at javax.baja.space.BComponentSpace.findByHandle(BComponentSpace.java:544) at javax.baja.space.BComponentSpace.resolveByHandle(BComponentSpace.java:573) at javax.baja.space.BHandleScheme.resolve(BHandleScheme.java:73) at javax.baja.naming.BOrdScheme.resolve(BOrdScheme.java:107) at javax.baja.naming.BOrd.resolve(BOrd.java:274) at javax.baja.naming.BOrd.resolve(BOrd.java:250) at javax.baja.naming.BatchResolve.resolve(BatchResolve.java:182) at javax.baja.naming.BatchResolve.resolve(BatchResolve.java:154) at com.tridium.exporttags.BSupervisorJoinJob.run(BSupervisorJoinJob.java:388) at com.tridium.exporttags.BJoinJob.run(BJoinJob.java:141) at javax.baja.util.Worker.process(Worker.java:168) at javax.baja.util.Worker$Processor.run(Worker.java:141) at java.lang.Thread.run(Thread.java:748) |
| SEVERE [10:53:25 19-Jul-21 EDT][orionTools.replicate] Failed replicating station Bldg_40_Security: entsec:AccessControlService javax.baja.sys.ServiceNotFoundException: entsec:AccessControlService at com.tridium.fox.sys.BFoxSession.getService(BFoxSession.java:1177) at com.tridiumx.entsec.access.replicate.EntsecReplicator.doReplicate(EntsecReplicator.java:128) at com.tridiumx.entsec.orionTools.replicate.Replicator.replicate(Replicator.java:146) at com.tridiumx.entsec.orionTools.replicate.ReplicationManager.replicateStation(ReplicationManager.java:310) at com.tridiumx.entsec.orionTools.replicate.BReplicationStationRunner.run(BReplicationStationRunner.java:34) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: javax.baja.net.NotConnectedException at com.tridium.fox.sys.BFoxConnection.sendSync(BFoxConnection.java:520) at com.tridium.fox.sys.BFoxChannel.sendSync(BFoxChannel.java:348) at com.tridium.fox.sys.broker.BBrokerChannel.serviceToPath(BBrokerChannel.java:1804) at com.tridium.fox.sys.BFoxSession.getService(BFoxSession.java:1171) ... 9 more |
Solution:
Ultimately, the solution will be provided by the site's IT department, or whoever manages the router that reports the MTU size to the networks that the Niagara hosts reside on. It must report the correct MTU size. But you can test to see whether or not this is truly the source of your problems.
To test MTU size using Windows:
- From a command prompt on a Windows PC connected to the network, run "ipconfig /all". Identify which network adapter is connected to the network you are troubleshooting. Note its name, IP address, subnet mask, and default gateway.
- Run "netsh interface ipv4 show subinterfaces". In the table that is printed, find the row where _interface_ name matches network adapter name from step 1. Find the corresponding MTU (first column).
- Subtract 28 from the reported MTU size (that's what the router told us it can support). For example, if MTU is 1358, then 1358-28=1330. Use that number in the following command: "ping <host> -f -l 1330". The first argument will be an IP address or host name. Try a reachable IP address that will go through the router (i.e. a remote Niagara host). The "-f" flag instructs the ping command not to fragment. In this way, we get an error if we our packet is too big. The "-l" flag sets the buffer size. 28 bytes are used for headers - so, that's why we subtract 28. The result *should* be the same as the response to a standard ping message (i.e. "Reply from 4.2.2.1: bytes=1330 time=55ms TTL=54"). If standard ping works, but you get 100% packet loss when using the calculated maximum buffer size, then the network does not support the MTU size that it claims it does. Note: If the command prints "Packet needs to be fragmented but DF set", then the buffer size exceeds the reported MTU size and no packet has been sent. Check your math, and try a smaller buffer size.
C:\>ping 10.0.0.1 -f -l 1330
Pinging 10.0.0.1 with 1330 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Ping statistics for 10.0.0.1: Packets: Sent = 3, Received = 0, Lost = 3 (100% loss),
Control-C
^C
To test MTU size using QNX:
- Connect to serial shell or via SSH and enter "sh" to exit the menu and drop to a shell
- Send the command "ifconfig". The MTU value will be identified for each adapter.
- Subtract 28 from the reported MTU size (that's what the router told us it can support). For example, if MTU is 1358, then 1358-28=1330. Use that number in the following command: "ping -D -s 1330 <host>". The argument "-D" sets the "Don't Fragment" bit in the IP header. And "-s 1330" sets the buffer size. If standard ping works, but you get 100% packet loss when using the calculated maximum buffer size, then the network does not support the MTU size that it claims it does. Note: If the command prints "ping: sendto: Message too long", then the buffer size exceeds the reported MTU size and no packet has been sent. Check your math, and try a smaller buffer size.
$ ping -D -s 1330 10.0.0.1
PING 10.0.0.1 (10.0.0.1): 1330 data bytes
----10.0.0.1 PING Statistics----
8 packets transmitted, 0 packets received, 100% packet loss
To test MTU size using JACE-9000:
- Connect to serial shell and login.
- Choose the option to Ping Host.
- Subtract 28 from the reported MTU size (that's what the router told us it can support). For example, if MTU is 1358, then 1358-28=1330. Use that number in the following command options: "-D -s 1330 <host>". The argument "-D" sets the "Don't Fragment" bit in the IP header. And "-s 1330" sets the buffer size. If standard ping works, but you get 100 when using the calculated maximum buffer size, then the network does not support the MTU size that it claims it does. Note: If the command prints "ping: sendto: Message too long", then the buffer size exceeds the reported MTU size and no packet has been sent. Check your math, and try a smaller buffer size.
Enter the options to be passed to ping : -D -s 1330 10.0.0.1
ping -D -s 1330 10.0.0.1
PING 10.0.0.1 (10.0.0.1) 1330(1358) bytes of data.
--- 10.0.0.1 ping statistics ---
9 packets transmitted, 0 received, 100% packet loss, time 8191ms
Press ENTER to continue