Hive
  1. Hive
  2. HIVE-5403

Move loading of filesystem, ugi, metastore client to hive session

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.13.0
    • Fix Version/s: 0.13.0
    • Component/s: None
    • Labels:
      None

      Description

      As part of HIVE-5184, the metastore connection, loading filesystem were done as part of the tez session so as to speed up query times while paying a cost at startup. We can do this more generally in hive to apply to both the mapreduce and tez side of things.

      1. HIVE-5403.1.patch
        3 kB
        Vikram Dixit K
      2. HIVE-5403.2.patch
        3 kB
        Vikram Dixit K
      3. HIVE-5403.3.patch
        3 kB
        Vikram Dixit K
      4. HIVE-5403.4.patch
        4 kB
        Vikram Dixit K

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hive-trunk-hadoop1-ptest #215 (See https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/215/)
          HIVE-5403: Perflogger broken due to HIVE-5403 (Vikram Dixit K via Gunther Hagleitner) (gunther: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1535598)

          • /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetastoreVersion.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java
            HIVE-5403 : Move loading of filesystem, ugi, metastore client to hive session (Vikram Dixit via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1535039)
          • /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetastoreVersion.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java
          Show
          Hudson added a comment - SUCCESS: Integrated in Hive-trunk-hadoop1-ptest #215 (See https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/215/ ) HIVE-5403 : Perflogger broken due to HIVE-5403 (Vikram Dixit K via Gunther Hagleitner) (gunther: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1535598 ) /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetastoreVersion.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java HIVE-5403 : Move loading of filesystem, ugi, metastore client to hive session (Vikram Dixit via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1535039 ) /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetastoreVersion.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hive-trunk-hadoop2-ptest #152 (See https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/152/)
          HIVE-5403: Perflogger broken due to HIVE-5403 (Vikram Dixit K via Gunther Hagleitner) (gunther: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1535598)

          • /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetastoreVersion.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java
            HIVE-5403 : Move loading of filesystem, ugi, metastore client to hive session (Vikram Dixit via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1535039)
          • /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetastoreVersion.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hive-trunk-hadoop2-ptest #152 (See https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/152/ ) HIVE-5403 : Perflogger broken due to HIVE-5403 (Vikram Dixit K via Gunther Hagleitner) (gunther: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1535598 ) /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetastoreVersion.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java HIVE-5403 : Move loading of filesystem, ugi, metastore client to hive session (Vikram Dixit via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1535039 ) /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetastoreVersion.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java
          Hide
          Hudson added a comment -

          ABORTED: Integrated in Hive-trunk-hadoop2 #521 (See https://builds.apache.org/job/Hive-trunk-hadoop2/521/)
          HIVE-5403: Perflogger broken due to HIVE-5403 (Vikram Dixit K via Gunther Hagleitner) (gunther: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1535598)

          • /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetastoreVersion.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java
          Show
          Hudson added a comment - ABORTED: Integrated in Hive-trunk-hadoop2 #521 (See https://builds.apache.org/job/Hive-trunk-hadoop2/521/ ) HIVE-5403 : Perflogger broken due to HIVE-5403 (Vikram Dixit K via Gunther Hagleitner) (gunther: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1535598 ) /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetastoreVersion.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hive-trunk-h0.21 #2420 (See https://builds.apache.org/job/Hive-trunk-h0.21/2420/)
          HIVE-5403: Perflogger broken due to HIVE-5403 (Vikram Dixit K via Gunther Hagleitner) (gunther: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1535598)

          • /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetastoreVersion.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hive-trunk-h0.21 #2420 (See https://builds.apache.org/job/Hive-trunk-h0.21/2420/ ) HIVE-5403 : Perflogger broken due to HIVE-5403 (Vikram Dixit K via Gunther Hagleitner) (gunther: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1535598 ) /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetastoreVersion.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hive-trunk-h0.21 #2417 (See https://builds.apache.org/job/Hive-trunk-h0.21/2417/)
          HIVE-5403 : Move loading of filesystem, ugi, metastore client to hive session (Vikram Dixit via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1535039)

          • /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetastoreVersion.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hive-trunk-h0.21 #2417 (See https://builds.apache.org/job/Hive-trunk-h0.21/2417/ ) HIVE-5403 : Move loading of filesystem, ugi, metastore client to hive session (Vikram Dixit via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1535039 ) /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetastoreVersion.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java
          Hide
          Vikram Dixit K added a comment -

          Raised HIVE-5633 for the same and uploaded the simple fix to unblock. I think the issue is that the perf logger should not depend on the session state in backend at least. I think some clean up may be required in perf logger.

          Show
          Vikram Dixit K added a comment - Raised HIVE-5633 for the same and uploaded the simple fix to unblock. I think the issue is that the perf logger should not depend on the session state in backend at least. I think some clean up may be required in perf logger.
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hive-trunk-hadoop2 #519 (See https://builds.apache.org/job/Hive-trunk-hadoop2/519/)
          HIVE-5403 : Move loading of filesystem, ugi, metastore client to hive session (Vikram Dixit via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1535039)

          • /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetastoreVersion.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hive-trunk-hadoop2 #519 (See https://builds.apache.org/job/Hive-trunk-hadoop2/519/ ) HIVE-5403 : Move loading of filesystem, ugi, metastore client to hive session (Vikram Dixit via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1535039 ) /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetastoreVersion.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java
          Hide
          Gunther Hagleitner added a comment -

          Another one: PerfLogger doesn't work on the backend with this change anymore. The problem is that the SessionState now uses MetaException in the code path to start the session. That's not available on the backend. PerfLogger has logic to determine whether it runs front or backend. It does so by checking SessionState.get() == null.

          That check cannot be executed anymore because loading the SessionState tries to resolve the MetaException (MetaStore api).

          The easiest fix would be to collapse the exception handlers to one that catches exception (super class of meta store) and wraps that into a runtime exception. Logically that's no different from what's performed right now.
          Can we have a follow up to this one as well?

          Show
          Gunther Hagleitner added a comment - Another one: PerfLogger doesn't work on the backend with this change anymore. The problem is that the SessionState now uses MetaException in the code path to start the session. That's not available on the backend. PerfLogger has logic to determine whether it runs front or backend. It does so by checking SessionState.get() == null. That check cannot be executed anymore because loading the SessionState tries to resolve the MetaException (MetaStore api). The easiest fix would be to collapse the exception handlers to one that catches exception (super class of meta store) and wraps that into a runtime exception. Logically that's no different from what's performed right now. Can we have a follow up to this one as well?
          Hide
          Brock Noland added a comment -

          Hey guys, thank you very much for your work on this! I know this is already committed, but the following is incorrect:

          +    // session creation should fail since the schema didn't get created
          +    try {
          +      SessionState.start(new CliSessionState(hiveConf));
          +    } catch (RuntimeException re) {
          +      assertTrue(re.getCause().getCause() instanceof MetaException);
          +    }
          

          It should be

          +    // session creation should fail since the schema didn't get created
          +    try {
          +      SessionState.start(new CliSessionState(hiveConf));
                  fail("Expected exception");
          +    } catch (RuntimeException re) {
          +      assertTrue(re.getCause().getCause() instanceof MetaException);
          +    }
          

          Can you do a follow up jira?

          Show
          Brock Noland added a comment - Hey guys, thank you very much for your work on this! I know this is already committed, but the following is incorrect: + // session creation should fail since the schema didn't get created + try { + SessionState.start(new CliSessionState(hiveConf)); + } catch (RuntimeException re) { + assertTrue(re.getCause().getCause() instanceof MetaException); + } It should be + // session creation should fail since the schema didn't get created + try { + SessionState.start(new CliSessionState(hiveConf)); fail("Expected exception"); + } catch (RuntimeException re) { + assertTrue(re.getCause().getCause() instanceof MetaException); + } Can you do a follow up jira?
          Hide
          Ashutosh Chauhan added a comment -

          Committed to trunk. Thanks, Vikram!

          Show
          Ashutosh Chauhan added a comment - Committed to trunk. Thanks, Vikram!
          Hide
          Hive QA added a comment -

          Overall: +1 all checks pass

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12609737/HIVE-5403.4.patch

          SUCCESS: +1 4430 tests passed

          Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1201/testReport
          Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1201/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          

          This message is automatically generated.

          Show
          Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12609737/HIVE-5403.4.patch SUCCESS: +1 4430 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1201/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1201/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated.
          Hide
          Vikram Dixit K added a comment -

          Fixed failing testcase. The test was failing because the show tables issued by the test was expected to fail upon creation of the metastore client (connection to metastore). However, since the metastore client is created upon session start by this patch, the testcase failed earlier than expected. Fixed it accordingly.

          Show
          Vikram Dixit K added a comment - Fixed failing testcase. The test was failing because the show tables issued by the test was expected to fail upon creation of the metastore client (connection to metastore). However, since the metastore client is created upon session start by this patch, the testcase failed earlier than expected. Fixed it accordingly.
          Hide
          Ashutosh Chauhan added a comment -

          +1

          Show
          Ashutosh Chauhan added a comment - +1
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12608592/HIVE-5403.3.patch

          ERROR: -1 due to 2 failed/errored test(s), 4411 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_script_broken_pipe1
          org.apache.hadoop.hive.metastore.TestMetastoreVersion.testVersionRestriction
          

          Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1134/testReport
          Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1134/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests failed with: TestsFailedException: 2 tests failed
          

          This message is automatically generated.

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12608592/HIVE-5403.3.patch ERROR: -1 due to 2 failed/errored test(s), 4411 tests executed Failed tests: org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_script_broken_pipe1 org.apache.hadoop.hive.metastore.TestMetastoreVersion.testVersionRestriction Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1134/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1134/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests failed with: TestsFailedException: 2 tests failed This message is automatically generated.
          Hide
          Vikram Dixit K added a comment -

          Updated to latest trunk. Eclipse has been broken since the merge with the vectorization branch. I will raise another JIRA for the same but with this patch I am able to successfully run tests on the command line/terminal.

          Show
          Vikram Dixit K added a comment - Updated to latest trunk. Eclipse has been broken since the merge with the vectorization branch. I will raise another JIRA for the same but with this patch I am able to successfully run tests on the command line/terminal.
          Hide
          Gunther Hagleitner added a comment -

          Ran into some problems running tests thru eclipse with this change. Need to take a look at this before committing it.

          Show
          Gunther Hagleitner added a comment - Ran into some problems running tests thru eclipse with this change. Need to take a look at this before committing it.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12606196/HIVE-5403.2.patch

          ERROR: -1 due to 1 failed/errored test(s), 4077 tests executed
          Failed tests:

          org.apache.hadoop.hive.metastore.TestMetastoreVersion.testVersionRestriction
          

          Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/988/testReport
          Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/988/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests failed with: TestsFailedException: 1 tests failed
          

          This message is automatically generated.

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12606196/HIVE-5403.2.patch ERROR: -1 due to 1 failed/errored test(s), 4077 tests executed Failed tests: org.apache.hadoop.hive.metastore.TestMetastoreVersion.testVersionRestriction Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/988/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/988/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests failed with: TestsFailedException: 1 tests failed This message is automatically generated.
          Hide
          Gunther Hagleitner added a comment -

          Looks good to me. +1

          Show
          Gunther Hagleitner added a comment - Looks good to me. +1
          Hide
          Vikram Dixit K added a comment -

          Updated to address Gunther's comments.

          Show
          Vikram Dixit K added a comment - Updated to address Gunther's comments.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12606025/HIVE-5403.1.patch

          ERROR: -1 due to 2 failed/errored test(s), 4070 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket_num_reducers
          org.apache.hadoop.hive.metastore.TestMetastoreVersion.testVersionRestriction
          

          Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/980/testReport
          Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/980/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests failed with: TestsFailedException: 2 tests failed
          

          This message is automatically generated.

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12606025/HIVE-5403.1.patch ERROR: -1 due to 2 failed/errored test(s), 4070 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket_num_reducers org.apache.hadoop.hive.metastore.TestMetastoreVersion.testVersionRestriction Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/980/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/980/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests failed with: TestsFailedException: 2 tests failed This message is automatically generated.
          Hide
          Gunther Hagleitner added a comment -

          comment on rb

          Show
          Gunther Hagleitner added a comment - comment on rb
          Hide
          Vikram Dixit K added a comment -
          Show
          Vikram Dixit K added a comment - Review board: https://reviews.apache.org/r/14425/

            People

            • Assignee:
              Vikram Dixit K
              Reporter:
              Vikram Dixit K
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development