Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-9234

HiveServer2 leaks FileSystem objects in FileSystem.CACHE

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.12.0, 0.13.0, 0.12.1, 0.14.0, 0.13.1
    • Fix Version/s: 1.0.0
    • Component/s: HiveServer2
    • Labels:
      None

      Description

      Running over extended period (48+ hrs), we've noticed HiveServer2 leaking FileSystem objects in FileSystem.CACHE. Linked jiras were previous attempts to fix it, but the issue still seems to be there. A workaround is to disable the caching (by setting fs.hdfs.impl.disable.cache and fs.file.impl.disable.cache to true), but creating new FileSystem objects is expensive.

      1. HIVE-9234.1.patch
        11 kB
        Vaibhav Gumashta
      2. HIVE-9234.2.patch
        11 kB
        Vaibhav Gumashta
      3. HIVE-9234.2.patch
        11 kB
        Vaibhav Gumashta
      4. HIVE-9234.branch-14.patch
        1 kB
        Vaibhav Gumashta

        Issue Links

          Activity

          Hide
          vgumashta Vaibhav Gumashta added a comment -

          cc Thejas M Nair Patch based on trunk. Will attach one for 14.1 as well. Vikram Dixit K Can we include this in 14.1? Without fs caching, there is significant performance loss.

          Show
          vgumashta Vaibhav Gumashta added a comment - cc Thejas M Nair Patch based on trunk. Will attach one for 14.1 as well. Vikram Dixit K Can we include this in 14.1? Without fs caching, there is significant performance loss.
          Hide
          hiveqa Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12690398/HIVE-9234.1.patch

          ERROR: -1 due to 1 failed/errored test(s), 6723 tests executed
          Failed tests:

          org.apache.hive.hcatalog.streaming.TestStreaming.testEndpointConnection
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2273/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2273/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2273/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 1 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12690398 - PreCommit-HIVE-TRUNK-Build

          Show
          hiveqa Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12690398/HIVE-9234.1.patch ERROR: -1 due to 1 failed/errored test(s), 6723 tests executed Failed tests: org.apache.hive.hcatalog.streaming.TestStreaming.testEndpointConnection Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2273/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2273/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2273/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed This message is automatically generated. ATTACHMENT ID: 12690398 - PreCommit-HIVE-TRUNK-Build
          Hide
          vgumashta Vaibhav Gumashta added a comment -

          Test failure is unrelated.

          Show
          vgumashta Vaibhav Gumashta added a comment - Test failure is unrelated.
          Hide
          vgumashta Vaibhav Gumashta added a comment -

          Attaching patch for 14.1.

          Show
          vgumashta Vaibhav Gumashta added a comment - Attaching patch for 14.1.
          Hide
          vgumashta Vaibhav Gumashta added a comment -

          Thejas M Nair New patch attached incorporating the feedback.

          Show
          vgumashta Vaibhav Gumashta added a comment - Thejas M Nair New patch attached incorporating the feedback.
          Hide
          thejas Thejas M Nair added a comment -

          +1
          Vaibhav, As discussed offline, for 0.14, it would be good to have smaller patch with just the change to call closeAllForUGI at the right place.

          Show
          thejas Thejas M Nair added a comment - +1 Vaibhav, As discussed offline, for 0.14, it would be good to have smaller patch with just the change to call closeAllForUGI at the right place.
          Hide
          vgumashta Vaibhav Gumashta added a comment -

          Thejas M Nair Patch for 14.1 focussing just on the fix, without any code cleanup.

          Show
          vgumashta Vaibhav Gumashta added a comment - Thejas M Nair Patch for 14.1 focussing just on the fix, without any code cleanup.
          Hide
          thejas Thejas M Nair added a comment -

          +1 for the 0.14 patch.

          Show
          thejas Thejas M Nair added a comment - +1 for the 0.14 patch.
          Hide
          vgumashta Vaibhav Gumashta added a comment -

          Reattaching for precommit run on new patch.

          Show
          vgumashta Vaibhav Gumashta added a comment - Reattaching for precommit run on new patch.
          Hide
          hiveqa Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12692020/HIVE-9234.2.patch

          ERROR: -1 due to 2 failed/errored test(s), 7313 tests executed
          Failed tests:

          org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1
          org.apache.hive.hcatalog.streaming.TestStreaming.testEndpointConnection
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2367/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2367/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2367/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 2 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12692020 - PreCommit-HIVE-TRUNK-Build

          Show
          hiveqa Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12692020/HIVE-9234.2.patch ERROR: -1 due to 2 failed/errored test(s), 7313 tests executed Failed tests: org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 org.apache.hive.hcatalog.streaming.TestStreaming.testEndpointConnection Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2367/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2367/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2367/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed This message is automatically generated. ATTACHMENT ID: 12692020 - PreCommit-HIVE-TRUNK-Build
          Hide
          vgumashta Vaibhav Gumashta added a comment -

          Test failure unrelated. Patch committed to trunk and 14.1. Thanks for reviewing Thejas M Nair and Brock Noland.

          Show
          vgumashta Vaibhav Gumashta added a comment - Test failure unrelated. Patch committed to trunk and 14.1. Thanks for reviewing Thejas M Nair and Brock Noland .
          Hide
          brocknoland Brock Noland added a comment -

          Hey guys, I don't actually see the leak fix unless acquire throws some kind of exception? Can you elaborate on:

          call closeAllForUGI at the right place.

          ?

          Show
          brocknoland Brock Noland added a comment - Hey guys, I don't actually see the leak fix unless acquire throws some kind of exception? Can you elaborate on: call closeAllForUGI at the right place. ?
          Hide
          thejas Thejas M Nair added a comment -

          The leak was in FileSystem.CACHE, it would keep growing. The Key in the CACHE includes UGI object. To clear the entries in CACHE corresponding to the UGI object, closeAllForUGI should be called once the UGI is no longer going to be used.
          The fix is in changing the sequence - to do ShimLoader.getHadoopShims().closeAllForUGI(sessionUgi) as the final step of HiveSessionImplwithUGI.close, after super.close(). super.close() was indirectly adding another entry into the CACHE.

          Show
          thejas Thejas M Nair added a comment - The leak was in FileSystem.CACHE, it would keep growing. The Key in the CACHE includes UGI object. To clear the entries in CACHE corresponding to the UGI object, closeAllForUGI should be called once the UGI is no longer going to be used. The fix is in changing the sequence - to do ShimLoader.getHadoopShims().closeAllForUGI(sessionUgi) as the final step of HiveSessionImplwithUGI.close, after super.close(). super.close() was indirectly adding another entry into the CACHE.
          Hide
          thejas Thejas M Nair added a comment -

          Updating release version for jiras resolved in 1.0.0 .

          Show
          thejas Thejas M Nair added a comment - Updating release version for jiras resolved in 1.0.0 .
          Hide
          thejas Thejas M Nair added a comment -

          This issue has been fixed in Apache Hive 1.0.0. If there is any issue with the fix, please open a new jira to address it.

          Show
          thejas Thejas M Nair added a comment - This issue has been fixed in Apache Hive 1.0.0. If there is any issue with the fix, please open a new jira to address it.

            People

            • Assignee:
              vgumashta Vaibhav Gumashta
              Reporter:
              vgumashta Vaibhav Gumashta
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development