Details

    • Type: Task Task
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 1.0.0-M1, 1.0.0-M2
    • Fix Version/s: 1.1.0
    • Component/s: Core
    • Labels:
      None

      Description

      There are indications of FtpServer not releasing file handles appropriately. This needs to be profiled and fixed.

        Activity

        Hide
        gerard added a comment -

        FtpServer uses intrinsically many loopback tcp connections on Windows, not on Linux
        Maybe Java 5 use different implementations of the same methods on these OS
        And these connections end with time_wait state, an explanation to /Device/Afd/EndPoint handles leak

        Here is a netstat result, all the time_wait sockets come from FtpServer, and only on Windows

        TCP 127.0.0.1:1068 0.0.0.0:0 LISTENING
        TCP 127.0.0.1:1621 127.0.0.1:1622 ESTABLISHED
        TCP 127.0.0.1:1622 127.0.0.1:1621 ESTABLISHED
        TCP 127.0.0.1:1623 127.0.0.1:1624 ESTABLISHED
        TCP 127.0.0.1:1624 127.0.0.1:1623 ESTABLISHED
        TCP 127.0.0.1:3187 127.0.0.1:9930 ESTABLISHED
        TCP 127.0.0.1:3190 127.0.0.1:3191 ESTABLISHED
        TCP 127.0.0.1:3191 127.0.0.1:3190 ESTABLISHED
        TCP 127.0.0.1:3325 127.0.0.1:3326 TIME_WAIT
        TCP 127.0.0.1:3329 127.0.0.1:3330 TIME_WAIT
        TCP 127.0.0.1:3339 127.0.0.1:3340 TIME_WAIT
        TCP 127.0.0.1:3341 127.0.0.1:3342 TIME_WAIT
        TCP 127.0.0.1:3343 127.0.0.1:3344 TIME_WAIT
        TCP 127.0.0.1:3345 127.0.0.1:3346 TIME_WAIT
        TCP 127.0.0.1:3347 127.0.0.1:3348 TIME_WAIT
        TCP 127.0.0.1:3349 127.0.0.1:3350 TIME_WAIT
        TCP 127.0.0.1:3353 127.0.0.1:3354 TIME_WAIT
        TCP 127.0.0.1:3355 127.0.0.1:3356 TIME_WAIT
        TCP 127.0.0.1:3357 127.0.0.1:3358 TIME_WAIT
        TCP 127.0.0.1:3359 127.0.0.1:3360 TIME_WAIT
        TCP 127.0.0.1:3361 127.0.0.1:3362 TIME_WAIT
        TCP 127.0.0.1:3367 127.0.0.1:3368 TIME_WAIT
        TCP 127.0.0.1:3369 127.0.0.1:3370 TIME_WAIT
        TCP 127.0.0.1:3371 127.0.0.1:3372 TIME_WAIT
        TCP 127.0.0.1:3379 127.0.0.1:3380 TIME_WAIT
        TCP 127.0.0.1:3381 127.0.0.1:3382 TIME_WAIT
        TCP 127.0.0.1:3387 127.0.0.1:3388 TIME_WAIT
        TCP 127.0.0.1:3390 127.0.0.1:3391 TIME_WAIT
        TCP 127.0.0.1:3398 127.0.0.1:3399 TIME_WAIT
        TCP 127.0.0.1:3402 127.0.0.1:3403 TIME_WAIT
        TCP 127.0.0.1:3404 127.0.0.1:3405 TIME_WAIT
        TCP 127.0.0.1:3406 127.0.0.1:3407 TIME_WAIT
        TCP 127.0.0.1:3408 127.0.0.1:3409 TIME_WAIT
        TCP 127.0.0.1:3416 127.0.0.1:3417 TIME_WAIT
        TCP 127.0.0.1:3418 127.0.0.1:3419 TIME_WAIT
        TCP 127.0.0.1:3426 127.0.0.1:3427 TIME_WAIT
        TCP 127.0.0.1:3428 127.0.0.1:3429 TIME_WAIT
        TCP 127.0.0.1:3430 127.0.0.1:3431 TIME_WAIT
        TCP 127.0.0.1:3434 127.0.0.1:3435 TIME_WAIT
        TCP 127.0.0.1:3436 127.0.0.1:3437 TIME_WAIT
        TCP 127.0.0.1:3440 127.0.0.1:3441 TIME_WAIT
        TCP 127.0.0.1:3452 127.0.0.1:3453 TIME_WAIT
        TCP 127.0.0.1:3454 127.0.0.1:3455 ESTABLISHED
        TCP 127.0.0.1:3455 127.0.0.1:3454 ESTABLISHED
        TCP 127.0.0.1:9930 127.0.0.1:3187 ESTABLISHED
        UDP 127.0.0.1:123 :
        UDP 127.0.0.1:1028 :
        UDP 127.0.0.1:1044 :
        UDP 127.0.0.1:1056 :
        UDP 127.0.0.1:1149 :
        UDP 127.0.0.1:1266 :
        UDP 127.0.0.1:1900 :
        UDP 127.0.0.1:2233 :
        UDP 127.0.0.1:3654 :
        UDP 127.0.0.1:4162 :

        Show
        gerard added a comment - FtpServer uses intrinsically many loopback tcp connections on Windows, not on Linux Maybe Java 5 use different implementations of the same methods on these OS And these connections end with time_wait state, an explanation to /Device/Afd/EndPoint handles leak Here is a netstat result, all the time_wait sockets come from FtpServer, and only on Windows TCP 127.0.0.1:1068 0.0.0.0:0 LISTENING TCP 127.0.0.1:1621 127.0.0.1:1622 ESTABLISHED TCP 127.0.0.1:1622 127.0.0.1:1621 ESTABLISHED TCP 127.0.0.1:1623 127.0.0.1:1624 ESTABLISHED TCP 127.0.0.1:1624 127.0.0.1:1623 ESTABLISHED TCP 127.0.0.1:3187 127.0.0.1:9930 ESTABLISHED TCP 127.0.0.1:3190 127.0.0.1:3191 ESTABLISHED TCP 127.0.0.1:3191 127.0.0.1:3190 ESTABLISHED TCP 127.0.0.1:3325 127.0.0.1:3326 TIME_WAIT TCP 127.0.0.1:3329 127.0.0.1:3330 TIME_WAIT TCP 127.0.0.1:3339 127.0.0.1:3340 TIME_WAIT TCP 127.0.0.1:3341 127.0.0.1:3342 TIME_WAIT TCP 127.0.0.1:3343 127.0.0.1:3344 TIME_WAIT TCP 127.0.0.1:3345 127.0.0.1:3346 TIME_WAIT TCP 127.0.0.1:3347 127.0.0.1:3348 TIME_WAIT TCP 127.0.0.1:3349 127.0.0.1:3350 TIME_WAIT TCP 127.0.0.1:3353 127.0.0.1:3354 TIME_WAIT TCP 127.0.0.1:3355 127.0.0.1:3356 TIME_WAIT TCP 127.0.0.1:3357 127.0.0.1:3358 TIME_WAIT TCP 127.0.0.1:3359 127.0.0.1:3360 TIME_WAIT TCP 127.0.0.1:3361 127.0.0.1:3362 TIME_WAIT TCP 127.0.0.1:3367 127.0.0.1:3368 TIME_WAIT TCP 127.0.0.1:3369 127.0.0.1:3370 TIME_WAIT TCP 127.0.0.1:3371 127.0.0.1:3372 TIME_WAIT TCP 127.0.0.1:3379 127.0.0.1:3380 TIME_WAIT TCP 127.0.0.1:3381 127.0.0.1:3382 TIME_WAIT TCP 127.0.0.1:3387 127.0.0.1:3388 TIME_WAIT TCP 127.0.0.1:3390 127.0.0.1:3391 TIME_WAIT TCP 127.0.0.1:3398 127.0.0.1:3399 TIME_WAIT TCP 127.0.0.1:3402 127.0.0.1:3403 TIME_WAIT TCP 127.0.0.1:3404 127.0.0.1:3405 TIME_WAIT TCP 127.0.0.1:3406 127.0.0.1:3407 TIME_WAIT TCP 127.0.0.1:3408 127.0.0.1:3409 TIME_WAIT TCP 127.0.0.1:3416 127.0.0.1:3417 TIME_WAIT TCP 127.0.0.1:3418 127.0.0.1:3419 TIME_WAIT TCP 127.0.0.1:3426 127.0.0.1:3427 TIME_WAIT TCP 127.0.0.1:3428 127.0.0.1:3429 TIME_WAIT TCP 127.0.0.1:3430 127.0.0.1:3431 TIME_WAIT TCP 127.0.0.1:3434 127.0.0.1:3435 TIME_WAIT TCP 127.0.0.1:3436 127.0.0.1:3437 TIME_WAIT TCP 127.0.0.1:3440 127.0.0.1:3441 TIME_WAIT TCP 127.0.0.1:3452 127.0.0.1:3453 TIME_WAIT TCP 127.0.0.1:3454 127.0.0.1:3455 ESTABLISHED TCP 127.0.0.1:3455 127.0.0.1:3454 ESTABLISHED TCP 127.0.0.1:9930 127.0.0.1:3187 ESTABLISHED UDP 127.0.0.1:123 : UDP 127.0.0.1:1028 : UDP 127.0.0.1:1044 : UDP 127.0.0.1:1056 : UDP 127.0.0.1:1149 : UDP 127.0.0.1:1266 : UDP 127.0.0.1:1900 : UDP 127.0.0.1:2233 : UDP 127.0.0.1:3654 : UDP 127.0.0.1:4162 :
        Hide
        Niklas Gustavsson added a comment -

        Windows has a TIME_WAIT timeout of 240 seconds while Linux times out in 60 seconds. Could that explain the difference you're seeing? I assume the sockets are terminated after 4 minutes?

        Show
        Niklas Gustavsson added a comment - Windows has a TIME_WAIT timeout of 240 seconds while Linux times out in 60 seconds. Could that explain the difference you're seeing? I assume the sockets are terminated after 4 minutes?
        Hide
        Dave Roberts added a comment -

        TIME_WAIT is the final state of a closed socket, which is kept around by the OS. The length of time it is kept in this state depends upon the OS, and Windows hangs on to them longer than most. This can be a problem on system with a high number of connections, as you can effectively run out of sockets if you exhaust the dynamic port allocation space within the 4 minute period.

        If you want to reduce the time a socket is in TIME_WAIT on Windows, you'll need to make a change to the registry.

        At the following location:
        HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters

        Set TCPTimedWaitDelay to 30 or 60 (seconds). If the key doesn't exist, simply add it. Of course, you'll need to reboot for this to take effect.

        Show
        Dave Roberts added a comment - TIME_WAIT is the final state of a closed socket, which is kept around by the OS. The length of time it is kept in this state depends upon the OS, and Windows hangs on to them longer than most. This can be a problem on system with a high number of connections, as you can effectively run out of sockets if you exhaust the dynamic port allocation space within the 4 minute period. If you want to reduce the time a socket is in TIME_WAIT on Windows, you'll need to make a change to the registry. At the following location: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters Set TCPTimedWaitDelay to 30 or 60 (seconds). If the key doesn't exist, simply add it. Of course, you'll need to reboot for this to take effect.
        Hide
        gerard added a comment -

        I will soon check the 4 minutes delay on WIndows
        On my Linux, time_wait is 60 seconds, i have checked this by a little program, seeing time_wait connections during this minute
        BUT i confirm Linux FtpServer does not use loopback sockets : I transfer about 20 files, using a continuous 'netstat -a', and I don't see any additional loopback connection nor time_wait state connections...

        Show
        gerard added a comment - I will soon check the 4 minutes delay on WIndows On my Linux, time_wait is 60 seconds, i have checked this by a little program, seeing time_wait connections during this minute BUT i confirm Linux FtpServer does not use loopback sockets : I transfer about 20 files, using a continuous 'netstat -a', and I don't see any additional loopback connection nor time_wait state connections...
        Hide
        gerard added a comment -

        Yes, about 4 minutes by default for TIME_WAIT state timeout, all is ok for that
        After a stress test, and using the latest release of Process Explorer (11.13), we clearly see green (creation) and red(removing) status for these /device/afd/endpoint and /device/tcp at the same time, but at the end we obtain .... more than 1500 handles for the jvm, with about 1400 only for /device/afd/endpoint, and they do not disapear at all even after 15 minutes so far
        Also observed at this moment:

        • 12 threads
        • the ftp(s) listener + one tcp loopback connection (local ports 3416/3417), as at the first launching of the jvm

        I'm going to try the upgrade java 5 u 14 -> u 15, nothing convincing in release notes, but maybe ...

        Show
        gerard added a comment - Yes, about 4 minutes by default for TIME_WAIT state timeout, all is ok for that After a stress test, and using the latest release of Process Explorer (11.13), we clearly see green (creation) and red(removing) status for these /device/afd/endpoint and /device/tcp at the same time, but at the end we obtain .... more than 1500 handles for the jvm, with about 1400 only for /device/afd/endpoint, and they do not disapear at all even after 15 minutes so far Also observed at this moment: 12 threads the ftp(s) listener + one tcp loopback connection (local ports 3416/3417), as at the first launching of the jvm I'm going to try the upgrade java 5 u 14 -> u 15, nothing convincing in release notes, but maybe ...
        Hide
        gerard added a comment -

        The issue remains with JRE 5 u15 and JRE 6 u 6

        Show
        gerard added a comment - The issue remains with JRE 5 u15 and JRE 6 u 6

          People

          • Assignee:
            Niklas Gustavsson
            Reporter:
            Niklas Gustavsson
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:

              Development