Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.0
    • Component/s: Resource Manager
    • Labels:
      None

      Description

      The worker registration takes too much time.
      When I tested on an in-house cluster which consists of one master and 32 workers, 31 workers are registered in about 2 minutes and one worker is failed to register.
      My guess from the log is that the failed worker does not receive any start messages.

      1. TAJO-296.patch
        2 kB
        Hyoungjun Kim

        Activity

        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Tajo-trunk-postcommit #563 (See https://builds.apache.org/job/Tajo-trunk-postcommit/563/)
        TAJO-296: Late registration of Tajo workers. (hyoungjunkim via hyunsik) (hyunsik: https://git-wip-us.apache.org/repos/asf?p=incubator-tajo.git&a=commit&h=b37453d004f0be10864e1d1188f28a768f5141ca)

        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/webapp/HttpServer.java
        • CHANGES.txt
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Tajo-trunk-postcommit #563 (See https://builds.apache.org/job/Tajo-trunk-postcommit/563/ ) TAJO-296 : Late registration of Tajo workers. (hyoungjunkim via hyunsik) (hyunsik: https://git-wip-us.apache.org/repos/asf?p=incubator-tajo.git&a=commit&h=b37453d004f0be10864e1d1188f28a768f5141ca ) tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/webapp/HttpServer.java CHANGES.txt
        Hide
        hyunsik Hyunsik Choi added a comment -

        Committed to the master branch.

        Show
        hyunsik Hyunsik Choi added a comment - Committed to the master branch.
        Hide
        hyunsik Hyunsik Choi added a comment -

        +1
        Thank you for your contribution. Nice fix!

        Show
        hyunsik Hyunsik Choi added a comment - +1 Thank you for your contribution. Nice fix!
        Hide
        hjkim Hyoungjun Kim added a comment -

        I changed jetty's SessionIdManager to HashSessionIdManager.
        (http://docs.codehaus.org/display/JETTY/Connectors+slow+to+startup)

        Show
        hjkim Hyoungjun Kim added a comment - I changed jetty's SessionIdManager to HashSessionIdManager. ( http://docs.codehaus.org/display/JETTY/Connectors+slow+to+startup )
        Hide
        hjkim Hyoungjun Kim added a comment -

        When running in debug mode, takes a long time in the following section

        2013-11-23 13:20:38,759 DEBUG mortbay.log (?:invoke(?)) - Container Server@5975d6ab + org.mortbay.jetty.servlet.HashSessionIdManager@7c65216 as sessionIdManager
        2013-11-23 13:20:38,759 DEBUG mortbay.log (?:invoke(?)) - Init SecureRandom.
        2013-11-23 13:22:45,941 DEBUG mortbay.log (?:invoke(?)) - started org.mortbay.jetty.servlet.HashSessionIdManager@7c65216
        
        Show
        hjkim Hyoungjun Kim added a comment - When running in debug mode, takes a long time in the following section 2013-11-23 13:20:38,759 DEBUG mortbay.log (?:invoke(?)) - Container Server@5975d6ab + org.mortbay.jetty.servlet.HashSessionIdManager@7c65216 as sessionIdManager 2013-11-23 13:20:38,759 DEBUG mortbay.log (?:invoke(?)) - Init SecureRandom. 2013-11-23 13:22:45,941 DEBUG mortbay.log (?:invoke(?)) - started org.mortbay.jetty.servlet.HashSessionIdManager@7c65216
        Hide
        jihoonson Jihoon Son added a comment - - edited

        We upgraded the java to 1.7 sometime ago. Here is the jvm version.

        $ java -version
        java version "1.7.0_17"
        OpenJDK Runtime Environment (IcedTea7 2.3.8) (Gentoo build 1.7.0_17-b02)
        OpenJDK 64-Bit Server VM (build 23.7-b01, mixed mode)
        
        Show
        jihoonson Jihoon Son added a comment - - edited We upgraded the java to 1.7 sometime ago. Here is the jvm version. $ java -version java version "1.7.0_17" OpenJDK Runtime Environment (IcedTea7 2.3.8) (Gentoo build 1.7.0_17-b02) OpenJDK 64-Bit Server VM (build 23.7-b01, mixed mode)
        Hide
        hyunsik Hyunsik Choi added a comment -

        In more detail, the jetty HTTP server on each worker starts at the init phase. The jetty can start slowly in a certain JVM,

        Show
        hyunsik Hyunsik Choi added a comment - In more detail, the jetty HTTP server on each worker starts at the init phase. The jetty can start slowly in a certain JVM,
        Hide
        hyunsik Hyunsik Choi added a comment -

        Could you share the JVM version that used in cluster nodes? I'm suspecting the jetty problem on a certain JVM version.

        Show
        hyunsik Hyunsik Choi added a comment - Could you share the JVM version that used in cluster nodes? I'm suspecting the jetty problem on a certain JVM version.

          People

          • Assignee:
            hjkim Hyoungjun Kim
            Reporter:
            jihoonson Jihoon Son
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development