Tags

, , ,

Introduction

This is in reference to the following article:
http://www.codeproject.com/Articles/85602/PortQry-Implementation-using-TcpClient-Socket-and

It’s been a few years since I’ve looked at this and recently I had received a notification that someone had posted a response. First I would like to agree with emilio_grv’s response in that Application Programmers should be very careful about handling timeouts within the application. As with any application development, make sure you cleanup any non-used resources as soon as possible, especially with sockets, as you’ll extinguish the available source ports that can be used. In windows environments, the default is 3977.

Background

As a connection timeout is not a parameter available within the TcpClient.BeginConnect() or TcpClient.Connect() functions, an issue arises for those who have large scale processes that must be accomplished in a timely fashion.

With the use of firewalls in the network we make a compromise, trading network visibility for security. We often no longer get a response from the TCP stack at the far end or sometimes even the ICMP message back when a packet passes a firewall whether there is a problem or not. Even more frustrating, we may have no choice in the matter, as another group or organization could be managing the firewalls, and so policy change can be difficult if not impossible altogether.

Layer 4 (TCP Session) Visibility Tools

There used to be an easy way to check for port availability programatically. With tools like L4Trace and L4Ping, the programmers used RAW sockets to create TCP Headers with empty payloads to attempt to establish a session, and timeout the connection early, terminating the session with a FIN. This allowed in-depth analysis of the path where firewalls were in place, seeing who was blocking the port (by seeing who the last person responding to a TCP Session Establishment request with incrementing TTLs.

Enter the Black Hats

Various malware manufacturers and other `ingenious’ sorts used this capability to SYN Flood hosts in the attempt to break into them, or to manufacture malformed TCP Session Run Conditions to crash the server or overrun buffers. This resulted in Microsoft deprecating the RAW Sockets functionality within the WinSock API as of XP Service Pack 3, hence breaking L4Trace and L4Ping as well as other very useful tools.

Response to Concerns of Dangling Connections

The concern raised by emilio_grv, that the timeout of a socket early within the application without terminating the request, is a valid concern. There is a limit to the use of ports 1025-5000 on XP and 49152-65535 on Windows Vista/2008/7. However to say you should never manage your application’s timeouts is short sighted.

The code provided (in the previously noted article) does terminate the connection (once we’ve determined it’s timed out) through the disconnect() function provided by the WinSock’s C# API stack when the specified timeout period expires. It’s not ideal that this should even have to be done. Microsoft, IHMO, should have provided two parameters in the TcpClient.BeginConnect() Function. The first to set the timeout (noted x) in the following flow chart, and second, the retries before calling the connection dead. The programmer could cause very little damage. At the most poor/malicous coding could result in the elimination of available sockets on the client system. This is due to the socket not being released (locking the local source socket) until the 30s TCP Cleanup Period expires.

Unfortunately, we don’t have that luxury, and more importantly, within the WinSock API, there is no (as far as I’ve seen) way to even generate your own TCP packet without reimplementing WinSock from the ground up, as RAW sockets are no longer available/implemented. With Windows 7 this becomes even more complicated, as you are required to build a Kernel driver in order to interface with the Network Card or Packets prior to processing by WinSock. Let’s just say it gets complicated quick.

Current Flow Diagram showing WinSock TCP Establishment:

      ##########################      ##########################      ##########################
      ### AVAILABLE TCP PORT ###      ## UNAVAILABLE TCP PORT ##      ## FIREWALLED TCP PORT ###
      --------------------------      --------------------------      --------------------------
      TcpClient.BeginConnect()        TcpClient.BeginConnect()        TcpClient.BeginConnect()

      CLIENT..............SERVER      CLIENT..............SERVER      CLIENT..............SERVER
      SYN --------------------->      SYN --------------------->      SYN --------------------->
      <----------------- SYN-ACK      <----------------- RST-ACK      ......WAIT 3 SECONDS......
      ACK --------------------->      SYN --------------------->
                                      <----------------- RST-ACK      SYN --------------------->
      ..x*3 SECONDS TRANSPIRED..      SYN --------------------->      ......WAIT 6 SECONDS......
                                      <----------------- RST-ACK
                                                                      SYN --------------------->
      ..CONNECTION ESTABLISHED..      ..x*6 SECONDS TRANSPIRED..      .....WAIT 12 SECONDS......

      OS Triggers Async Callback      OS Triggers Async Callback      OS Triggers Async Callback

      TcpClient.EndConnect()

      FIN --------------------->
      <----------------- FIN-ACK
      ACK --------------------->

      OS Triggers Async Callback

      ...WAIT 30s FOR CLEANUP...      ...WAIT 30s FOR CLEANUP...     ....WAIT 30s FOR CLEANUP...

      TcpClient.Dispose()             TcpClient.Dispose()            TcpClient.Dispose()

      Local and Foreign TCP Port      Local and Foreign TCP Port     .Local TCP Port freed for..
       are available again for         are available again for       ......reuse by system......
                 use.                            use.
      ##########################      ##########################     ###########################

Dissemination

With a perfect path (no data loss) and a 3 second RTT (which is all TCP accounts for, and really the bounds with which TCP will effectively work in) the turn around time in an application is 9 seconds, if the Port is up and available, and we just terminate the connection once established.

connect-time
(x*3) = ?, (3*3) = 9 seconds

When the port is unavailable, (no application is attached to the port at the server) but unfiltered by a firewall, we receive the TCP messages back, allowing us to get a response back within 18 seconds.

connect-reset-time
(x*6) = ?, (3*6) = 18 seconds

When the port is unavailable, there is a route to the server in the routing table, the router hosting the subnet the server has an ARP entry to translate the IP to a MAC address, and a firewall sits between the client and server, the total time time to return the application a connection-failure is 21 seconds.

tcp-timeout + (tcp-timeout*(2^1)) + (tcp-timeout*(2^2))
(3) + (6) + (12) = 21 secnods

Looking at the other extreme, lets assume 4 millisecond turn around:

Available TCP Port:(0.004*3+(0.004*3) = 24ms

Unavailable TCP Port:(0.004*6) = 24ms

Filtered TCP Port:(3) + (6) + (12) = 21 seconds

Application

In today’s day an age, a slow packet turn-around time is between 700ms and 900ms, with Satellite being the slowest due to the physical distance the signal has to travel from earth to LEO and back.

I ran a report on our network and 96% of the host-latency (RTT) from the management system was within 60ms, with 92% less than 40ms, and 99% less than 100ms. We have over 2M addresses in use within the network of the 19M addresses available within RFC1918 space.

Say I were to poll every single device on the most common 25 ports using the standard TCP timeouts defined within WinSock. From a multi-threaded application on a system that can establish 3900 tcp connections (WinSocks default dynamic port limit rounded down) and we say we average the latency at 40ms for hosts that respond. Let also say that only 5% of the ports on active hosts are accessible due to firewall policy, and 60% of the non-responsive hosts are not behind a firewall.

                                           RFC1918_HOSTS: =   19,000,000  Total RFC1918 Addressing
                                            ACTIVE_HOSTS: =    2,000,000  Active Hosts
                                          SOCKETS_POLLED: =           25  Polled Ports per Host
                                          %_RESP_SOCKETS: =          20%  Responsive Ports on Active Host
                                                 LATENCY: =           40  milliseconds host majority latency
                                                 TIMEOUT: =            3  WinSock TCP Timeout Default
                                                 RETRIES: =            3  Winsock TCP Retry Default
                                         PORTS_AVAILABLE: =        3,900  Winsock Open Dynamic Source TCP Max
                                        UNFILTERED_HOSTS: =          60%  Percentage of Non-Filtered unused endpoints

                                              DEAD-HOSTS:  
                           (RFC1918_HOSTS - ACTIVE_HOSTS) =   17,000,000  Unused Addressing

                                      RESPONSIVE_SOCKETS:               
     (ACTIVE_HOSTS * SOCKETS_POLLED * RESPONSIVE_SOCKETS) =   10,000,000  Responsive Ports

                                    UNRESPONSIVE_SOCKETS:
                           (ACTIVE_HOSTS * SOCKETS_POLLED *             
       (100% - RESPONSIVE_SOCKETS) ) +(TIMEOUT * RETRIES) =  465,000,000  Unresponsive Ports             

                                 RESPONSIVE_SOCKETS_TIME:
                   (RESPONSIVE_SOCKETS / PORTS_AVAILABLE) *               Time in Seconds to
                                   ((LATENCY / 1000) * 3) =          308  Poll Responsive Endpoints   

                                      RESET_SOCKETS_TIME:             
(RESPONSIVE_SOCKETS / PORTS_AVAILABLE * UNFILTERED_HOSTS) *               Time in Seconds to
                                   ((LATENCY / 1000) * 6) =       17,169  Poll Reset-Responsive Hosts           

                               UNRESPONSIVE_SOCKETS_TIME:
(RESPONSIVE_SOCKETS / PORTS_AVAILABLE * UNFILTERED_HOSTS) *               Time in Seconds to
                    S(TIMEOUT * 2^(N), 0, RETRIES - 1, N) =    1,001,538  Poll Unresponsive (Filtered) Endpoints
                                                                          
             
                                                                 1,019,015  seconds
                                                                    16,984  mintes
                                                                       283  hours
                                                                        12  days


                                 RESPONSIVE_SOCKETS_TIME:
                   (RESPONSIVE_SOCKETS / PORTS_AVAILABLE) *               Time in Seconds to    
                                   ((LATENCY / 1000) * 3) =          308  Poll Responsive Endpoints  
           
                                      RESET_SOCKETS_TIME:
(RESPONSIVE_SOCKETS / PORTS_AVAILABLE * UNFILTERED_HOSTS) *               Time in Seconds to
                         ((LATENCY / 1000) * RETRIES * 2) =        5,723  Poll Reset-Responsive Hosts

                               UNRESPONSIVE_SOCKETS_TIME:
(RESPONSIVE_SOCKETS / PORTS_AVAILABLE * UNFILTERED_HOSTS) *               Time in Seconds to
                    S(TIMEOUT * 2^(N), 0, RETRIES - 1, N) =      143,077  Poll Unresponsive (Filtered) Endpoints 
            
                                                                 149,108  seconds
                                                                   2,485  mintes
                                                                      41  hours
                                                                       2  days

                                           RFC1918_HOSTS: =   19,000,000  Total RFC1918 Addressing
                                            ACTIVE_HOSTS: =    2,000,000  Active Hosts
                                          SOCKETS_POLLED: =           25  Polled Ports per Host
                                          %_RESP_SOCKETS: =          20%  Responsive Ports on Active Host
                                                 LATENCY: =           40  milliseconds host majority latency
                                                 TIMEOUT: =            3  WinSock TCP Timeout Default
                                                 RETRIES: =            3  Winsock TCP Retry Default
                                         PORTS_AVAILABLE: =        3,900  Winsock Open Dynamic Source TCP Max
                                        UNFILTERED_HOSTS: =          60%  Percentage of Non-Filtered unused endpoints

                                              DEAD-HOSTS: 
                           (RFC1918_HOSTS - ACTIVE_HOSTS) =   17,000,000  Unused Addressing

                                      RESPONSIVE_SOCKETS:
     (ACTIVE_HOSTS * SOCKETS_POLLED * RESPONSIVE_SOCKETS) =   10,000,000  Responsive Ports

                                    UNRESPONSIVE_SOCKETS:
                           (ACTIVE_HOSTS * SOCKETS_POLLED *             
                            (100% - RESPONSIVE_SOCKETS) ) +             
                                      (TIMEOUT * RETRIES) =  465,000,000  Unresponsive Ports

                                 RESPONSIVE_SOCKETS_TIME:
                   (RESPONSIVE_SOCKETS / PORTS_AVAILABLE) *               Time in Seconds to   
                                   ((LATENCY / 1000) * 3) =          308  Poll Responsive Endpoints

                                      RESET_SOCKETS_TIME:
(RESPONSIVE_SOCKETS / PORTS_AVAILABLE * UNFILTERED_HOSTS) *               Time in Seconds to
                                   ((LATENCY / 1000) * 6) =       17,169  Poll Reset-Responsive Hosts

                               UNRESPONSIVE_SOCKETS_TIME:
(RESPONSIVE_SOCKETS / PORTS_AVAILABLE * UNFILTERED_HOSTS) *               Time in Seconds to
                    S(TIMEOUT * 2^(N), 0, RETRIES - 1, N) =    1,001,538  Poll Unresponsive (Filtered) Endpoints

                                                               1,019,015  seconds
                                                                  16,984  mintes
                                                                     283  hours
                                                                      12  days


                                 RESPONSIVE_SOCKETS_TIME:
                   (RESPONSIVE_SOCKETS / PORTS_AVAILABLE) *               Time in Seconds to
                                   ((LATENCY / 1000) * 3) =          308  Poll Responsive Endpoints
           
                                      RESET_SOCKETS_TIME:
(RESPONSIVE_SOCKETS / PORTS_AVAILABLE * UNFILTERED_HOSTS) *               Time in Seconds to
                         ((LATENCY / 1000) * RETRIES * 2) =        5,723  Poll Reset-Responsive Hosts

                               UNRESPONSIVE_SOCKETS_TIME:
(RESPONSIVE_SOCKETS / PORTS_AVAILABLE * UNFILTERED_HOSTS) *               Time in Seconds to
                  * S(TIMEOUT * 2^(N), 0, RETRIES - 1, N) =      143,077  Poll Unresponsive (Filtered) Endpoints

                                                                 149,108  seconds
                                                                   2,485  mintes
                                                                      41  hours
                                                                       2  days

As depicted above, the first set of calculations are application latency occured it would take to perform a basic TCP Session Poll of the entire system using the previously defined parameters with WinSock’s Defaults for timeout and retries. By simply changing the timeout to 2 seconds and no retries we can cut the time it takes to poll every device within RFC1918 space to only 2 days from the 12 using the defaults.

I personally believe this provides plenty of merit to the need for the ability to either adjust the TCP parameters within the TcpClient.BeginConnect and TcpClient.Connect Functions or implementation of an exterior timing mechanism to track and terminate stale sessions at the Application Layer. In my job in particular, SCADA requires we poll thousands of well heads every hour. And this requires a TCP connection to each one, with a fairly lengthy GET/RESPONSE session for each device gathering the information stored on the PLCs and Meters. As you can imagine, this is difficult to scale, so any added efficiency that can be recovered can have drastic effects on the overall performance of the system.

I hope this was of some value to those reading. And I appreciate the response by emilio_grv. It gave me the opportunity to go back and look both at the WinSock implementation of TCP/IP and Sockets as well as get some study time on TCP itself.

Advertisements