Hive
  1. Hive
  2. HIVE-7169

HiveServer2 in Http Mode should have a configurable IdleMaxTime timeout

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.14.0
    • Component/s: HiveServer2
    • Labels:
      None

      Description

      Currently, in HiveServer2 we use Jetty Server to start the Http Server. The connector used for this Thrift Http Cli Service has maximum idle time as the default timeout as mentioned in http://grepcode.com/file/repo1.maven.org/maven2/org.eclipse.jetty/jetty-server/7.0.0.v20091005/org/eclipse/jetty/server/AbstractConnector.java#AbstractConnector.0_maxIdleTime.

      This should be manually configurable using connector.setMaxIdleTime(maxIdleTime);

      1. HIVE-7169.4.patch
        2 kB
        Hari Sankar Sivarama Subramaniyan
      2. HIVE-7169.3.patch
        2 kB
        Hari Sankar Sivarama Subramaniyan
      3. HIVE-7169.2.patch
        2 kB
        Hari Sankar Sivarama Subramaniyan
      4. HIVE-7169.1.patch
        2 kB
        Hari Sankar Sivarama Subramaniyan

        Activity

        Hari Sankar Sivarama Subramaniyan created issue -
        Hide
        Hari Sankar Sivarama Subramaniyan added a comment -

        cc-ing [~vaibhavgumashta] for review.

        Show
        Hari Sankar Sivarama Subramaniyan added a comment - cc-ing [~vaibhavgumashta] for review.
        Hari Sankar Sivarama Subramaniyan made changes -
        Field Original Value New Value
        Attachment HIVE-7169.1.patch [ 12648033 ]
        Hari Sankar Sivarama Subramaniyan made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Vaibhav Gumashta added a comment -

        Looks good. Can you also add the property description to hive-default.xml.template? Thanks!

        Show
        Vaibhav Gumashta added a comment - Looks good. Can you also add the property description to hive-default.xml.template? Thanks!
        Hide
        Vaibhav Gumashta added a comment -

        You might also want to add a little parameter description in HiveConf. The same can go in hive-default.xml.template. Thanks again!

        Show
        Vaibhav Gumashta added a comment - You might also want to add a little parameter description in HiveConf. The same can go in hive-default.xml.template. Thanks again!
        Hide
        Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12648033/HIVE-7169.1.patch

        ERROR: -1 due to 18 failed/errored test(s), 5585 tests executed
        Failed tests:

        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_display_colstats_tbllvl
        org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_optimization
        org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_insert1
        org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_metadata_only_queries
        org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_ptf
        org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_scriptfile1
        org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_dml
        org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_schema_evolution
        org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin6
        org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
        org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority
        org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_ctas
        org.apache.hadoop.hive.metastore.TestMetastoreVersion.testDefaults
        org.apache.hadoop.hive.ql.exec.tez.TestTezTask.testSubmit
        org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimal
        org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalX
        org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalXY
        org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
        

        Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/375/testReport
        Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/375/console
        Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-375/

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests exited with: TestsFailedException: 18 tests failed
        

        This message is automatically generated.

        ATTACHMENT ID: 12648033

        Show
        Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12648033/HIVE-7169.1.patch ERROR: -1 due to 18 failed/errored test(s), 5585 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_display_colstats_tbllvl org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_optimization org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_insert1 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_metadata_only_queries org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_ptf org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_scriptfile1 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_dml org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_schema_evolution org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin6 org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_ctas org.apache.hadoop.hive.metastore.TestMetastoreVersion.testDefaults org.apache.hadoop.hive.ql.exec.tez.TestTezTask.testSubmit org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimal org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalX org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalXY org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/375/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/375/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-375/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 18 tests failed This message is automatically generated. ATTACHMENT ID: 12648033
        Hari Sankar Sivarama Subramaniyan made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        Hari Sankar Sivarama Subramaniyan added a comment -

        cc-ing [~vaibhavgumashta] and Thejas M Nair for reviewing the patch.

        Thanks
        Hari

        Show
        Hari Sankar Sivarama Subramaniyan added a comment - cc-ing [~vaibhavgumashta] and Thejas M Nair for reviewing the patch. Thanks Hari
        Hari Sankar Sivarama Subramaniyan made changes -
        Attachment HIVE-7169.2.patch [ 12661862 ]
        Hari Sankar Sivarama Subramaniyan made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Thejas M Nair added a comment -

        Hari Sankar Sivarama Subramaniyan Can you please update the description of the param with the unit (millisecond), and also clarify that it is max idle time for a connection

        Show
        Thejas M Nair added a comment - Hari Sankar Sivarama Subramaniyan Can you please update the description of the param with the unit (millisecond), and also clarify that it is max idle time for a connection
        Hide
        Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12661862/HIVE-7169.2.patch

        ERROR: -1 due to 2 failed/errored test(s), 5894 tests executed
        Failed tests:

        org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_opt_vectorization
        org.apache.hive.jdbc.TestJdbcDriver2.testDatabaseMetaData
        

        Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/310/testReport
        Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/310/console
        Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-310/

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests exited with: TestsFailedException: 2 tests failed
        

        This message is automatically generated.

        ATTACHMENT ID: 12661862

        Show
        Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12661862/HIVE-7169.2.patch ERROR: -1 due to 2 failed/errored test(s), 5894 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_opt_vectorization org.apache.hive.jdbc.TestJdbcDriver2.testDatabaseMetaData Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/310/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/310/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-310/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed This message is automatically generated. ATTACHMENT ID: 12661862
        Hari Sankar Sivarama Subramaniyan made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        Hari Sankar Sivarama Subramaniyan added a comment -

        Thanks @thejas . Made changes as per the previous comment.

        Thanks
        Hari

        Show
        Hari Sankar Sivarama Subramaniyan added a comment - Thanks @thejas . Made changes as per the previous comment. Thanks Hari
        Hari Sankar Sivarama Subramaniyan made changes -
        Attachment HIVE-7169.3.patch [ 12661897 ]
        Hari Sankar Sivarama Subramaniyan made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Thejas M Nair added a comment -

        +1

        Show
        Thejas M Nair added a comment - +1
        Hide
        Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12661897/HIVE-7169.3.patch

        ERROR: -1 due to 2 failed/errored test(s), 5896 tests executed
        Failed tests:

        org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_opt_vectorization
        org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
        

        Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/316/testReport
        Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/316/console
        Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-316/

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests exited with: TestsFailedException: 2 tests failed
        

        This message is automatically generated.

        ATTACHMENT ID: 12661897

        Show
        Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12661897/HIVE-7169.3.patch ERROR: -1 due to 2 failed/errored test(s), 5896 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_opt_vectorization org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/316/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/316/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-316/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed This message is automatically generated. ATTACHMENT ID: 12661897
        Hide
        Lefty Leverenz added a comment -

        Just a little doc nitpick: "Maximum Idle time" shouldn't capitalize "Idle" (otherwise looks good).

        + "Maximum Idle time in milliseconds for a connection on the server when in HTTP mode."),

        Show
        Lefty Leverenz added a comment - Just a little doc nitpick: "Maximum Idle time" shouldn't capitalize "Idle" (otherwise looks good). + "Maximum Idle time in milliseconds for a connection on the server when in HTTP mode."),
        Hari Sankar Sivarama Subramaniyan made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hari Sankar Sivarama Subramaniyan made changes -
        Attachment HIVE-7169.4.patch [ 12662013 ]
        Hari Sankar Sivarama Subramaniyan made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Hive QA added a comment -

        Overall: -1 no tests executed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12662013/HIVE-7169.4.patch

        Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/331/testReport
        Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/331/console
        Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-331/

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Tests exited with: IllegalArgumentException: No propertifies found in file: mainProperties for property: spark.query.files
        

        This message is automatically generated.

        ATTACHMENT ID: 12662013

        Show
        Hive QA added a comment - Overall : -1 no tests executed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12662013/HIVE-7169.4.patch Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/331/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/331/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-331/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Tests exited with: IllegalArgumentException: No propertifies found in file: mainProperties for property: spark.query.files This message is automatically generated. ATTACHMENT ID: 12662013
        Hide
        Lefty Leverenz added a comment -

        Thanks for fixing the capitalization, Hari Sankar Sivarama Subramaniyan.

        +1 for the parameter description.

        Show
        Lefty Leverenz added a comment - Thanks for fixing the capitalization, Hari Sankar Sivarama Subramaniyan . +1 for the parameter description.
        Hide
        Thejas M Nair added a comment -

        Patch committed to trunk. Thanks for the contribution Hari Sankar Sivarama Subramaniyan

        Show
        Thejas M Nair added a comment - Patch committed to trunk. Thanks for the contribution Hari Sankar Sivarama Subramaniyan
        Thejas M Nair made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Fix Version/s 0.14.0 [ 12326450 ]
        Resolution Fixed [ 1 ]
        Lefty Leverenz made changes -
        Labels TODOC14
        Hide
        Lefty Leverenz added a comment -

        This adds configuration parameter hive.server2.thrift.http.max.idle.time so it needs to be documented in the wiki by the time 0.14.0 is released.

        Show
        Lefty Leverenz added a comment - This adds configuration parameter hive.server2.thrift.http.max.idle.time so it needs to be documented in the wiki by the time 0.14.0 is released. Configuration Properties – HiveServer2
        Hide
        Szehon Ho added a comment -

        Added info of:
        hive.server2.thrift.http.max.idle.time
        to https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties

        under the flag:
        hive.server2.thrift.http.max.worker.threads

        Show
        Szehon Ho added a comment - Added info of: hive.server2.thrift.http.max.idle.time to https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties under the flag: hive.server2.thrift.http.max.worker.threads
        Szehon Ho made changes -
        Labels TODOC14
        Hide
        Thejas M Nair added a comment -

        This has been fixed in 0.14 release. Please open new jira if you see any issues.

        Show
        Thejas M Nair added a comment - This has been fixed in 0.14 release. Please open new jira if you see any issues.
        Thejas M Nair made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Hari Sankar Sivarama Subramaniyan
            Reporter:
            Hari Sankar Sivarama Subramaniyan
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development