Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.4
    • 0.4
    • None
    • None

    Description

      PigHCatUtil::getTable() creates a metastore client but never call close on it. As a result, underlying connection will not be released. Metastore client is created in multiple different files, all of those should be examined to make sure, client is properly closed at the end.

      Attachments

        1. HCATALOG-363.patch
          16 kB
          Rohini Palaniswamy
        2. HCATALOG-363-1.patch
          15 kB
          Rohini Palaniswamy

        Activity

          As part of HCAT-36 I added HCatUtil.closeHiveClientQuietly() to cases where the client was created using HCatUtil as I encountered HCAT-236 with client.close(). Probably that should also become part of the patch for this jira instead of being in HCAT-36.

          rohini Rohini Palaniswamy added a comment - As part of HCAT-36 I added HCatUtil.closeHiveClientQuietly() to cases where the client was created using HCatUtil as I encountered HCAT-236 with client.close(). Probably that should also become part of the patch for this jira instead of being in HCAT-36.

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4701/
          -----------------------------------------------------------

          Review request for hcatalog.

          Summary
          -------

          Changes done:
          1) Ensured that whenever there is a createHiveMetastoreClient there is a close() so that connections are released.
          2) Tried to close the connections quietly. Does not have much effect as of now as HiveMetaStoreClient already catches the exception and does a LOG.error. So the stack trace still appears in the logs.

          This addresses bug HCATALOG-363.
          https://issues.apache.org/jira/browse/HCATALOG-363

          Diffs


          http://svn.apache.org/repos/asf/incubator/hcatalog/branches/branch-0.4/src/java/org/apache/hcatalog/common/HCatUtil.java 1311907
          http://svn.apache.org/repos/asf/incubator/hcatalog/branches/branch-0.4/src/java/org/apache/hcatalog/mapreduce/DefaultOutputCommitterContainer.java 1311907
          http://svn.apache.org/repos/asf/incubator/hcatalog/branches/branch-0.4/src/java/org/apache/hcatalog/mapreduce/FileOutputCommitterContainer.java 1311907
          http://svn.apache.org/repos/asf/incubator/hcatalog/branches/branch-0.4/src/java/org/apache/hcatalog/mapreduce/FileOutputFormatContainer.java 1311907
          http://svn.apache.org/repos/asf/incubator/hcatalog/branches/branch-0.4/src/java/org/apache/hcatalog/mapreduce/HCatBaseOutputFormat.java 1311907
          http://svn.apache.org/repos/asf/incubator/hcatalog/branches/branch-0.4/src/java/org/apache/hcatalog/mapreduce/HCatOutputFormat.java 1311907
          http://svn.apache.org/repos/asf/incubator/hcatalog/branches/branch-0.4/src/java/org/apache/hcatalog/mapreduce/InitializeInput.java 1311907
          http://svn.apache.org/repos/asf/incubator/hcatalog/branches/branch-0.4/src/java/org/apache/hcatalog/pig/PigHCatUtil.java 1311907

          Diff: https://reviews.apache.org/r/4701/diff

          Testing
          -------

          Unit tests pass. Did an integration test issuing pig commands. e2e tests not run.

          Thanks,

          Rohini

          jiraposter@reviews.apache.org jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4701/ ----------------------------------------------------------- Review request for hcatalog. Summary ------- Changes done: 1) Ensured that whenever there is a createHiveMetastoreClient there is a close() so that connections are released. 2) Tried to close the connections quietly. Does not have much effect as of now as HiveMetaStoreClient already catches the exception and does a LOG.error. So the stack trace still appears in the logs. This addresses bug HCATALOG-363 . https://issues.apache.org/jira/browse/HCATALOG-363 Diffs http://svn.apache.org/repos/asf/incubator/hcatalog/branches/branch-0.4/src/java/org/apache/hcatalog/common/HCatUtil.java 1311907 http://svn.apache.org/repos/asf/incubator/hcatalog/branches/branch-0.4/src/java/org/apache/hcatalog/mapreduce/DefaultOutputCommitterContainer.java 1311907 http://svn.apache.org/repos/asf/incubator/hcatalog/branches/branch-0.4/src/java/org/apache/hcatalog/mapreduce/FileOutputCommitterContainer.java 1311907 http://svn.apache.org/repos/asf/incubator/hcatalog/branches/branch-0.4/src/java/org/apache/hcatalog/mapreduce/FileOutputFormatContainer.java 1311907 http://svn.apache.org/repos/asf/incubator/hcatalog/branches/branch-0.4/src/java/org/apache/hcatalog/mapreduce/HCatBaseOutputFormat.java 1311907 http://svn.apache.org/repos/asf/incubator/hcatalog/branches/branch-0.4/src/java/org/apache/hcatalog/mapreduce/HCatOutputFormat.java 1311907 http://svn.apache.org/repos/asf/incubator/hcatalog/branches/branch-0.4/src/java/org/apache/hcatalog/mapreduce/InitializeInput.java 1311907 http://svn.apache.org/repos/asf/incubator/hcatalog/branches/branch-0.4/src/java/org/apache/hcatalog/pig/PigHCatUtil.java 1311907 Diff: https://reviews.apache.org/r/4701/diff Testing ------- Unit tests pass. Did an integration test issuing pig commands. e2e tests not run. Thanks, Rohini

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4701/
          -----------------------------------------------------------

          (Updated 2012-04-17 18:24:45.529279)

          Review request for hcatalog.

          Changes
          -------

          Updated the patch to recent revision and fixed conflict in HCatBaseOutputFormat.

          Summary
          -------

          Changes done:
          1) Ensured that whenever there is a createHiveMetastoreClient there is a close() so that connections are released.
          2) Tried to close the connections quietly. Does not have much effect as of now as HiveMetaStoreClient already catches the exception and does a LOG.error. So the stack trace still appears in the logs.

          This addresses bug HCATALOG-363.
          https://issues.apache.org/jira/browse/HCATALOG-363

          Diffs (updated)


          http://svn.apache.org/repos/asf/incubator/hcatalog/branches/branch-0.4/src/java/org/apache/hcatalog/common/HCatUtil.java 1327178
          http://svn.apache.org/repos/asf/incubator/hcatalog/branches/branch-0.4/src/java/org/apache/hcatalog/mapreduce/DefaultOutputCommitterContainer.java 1327178
          http://svn.apache.org/repos/asf/incubator/hcatalog/branches/branch-0.4/src/java/org/apache/hcatalog/mapreduce/FileOutputCommitterContainer.java 1327178
          http://svn.apache.org/repos/asf/incubator/hcatalog/branches/branch-0.4/src/java/org/apache/hcatalog/mapreduce/FileOutputFormatContainer.java 1327178
          http://svn.apache.org/repos/asf/incubator/hcatalog/branches/branch-0.4/src/java/org/apache/hcatalog/mapreduce/HCatOutputFormat.java 1327178
          http://svn.apache.org/repos/asf/incubator/hcatalog/branches/branch-0.4/src/java/org/apache/hcatalog/mapreduce/InitializeInput.java 1327178
          http://svn.apache.org/repos/asf/incubator/hcatalog/branches/branch-0.4/src/java/org/apache/hcatalog/pig/PigHCatUtil.java 1327178

          Diff: https://reviews.apache.org/r/4701/diff

          Testing
          -------

          Unit tests pass. Did an integration test issuing pig commands. e2e tests not run.

          Thanks,

          Rohini

          jiraposter@reviews.apache.org jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4701/ ----------------------------------------------------------- (Updated 2012-04-17 18:24:45.529279) Review request for hcatalog. Changes ------- Updated the patch to recent revision and fixed conflict in HCatBaseOutputFormat. Summary ------- Changes done: 1) Ensured that whenever there is a createHiveMetastoreClient there is a close() so that connections are released. 2) Tried to close the connections quietly. Does not have much effect as of now as HiveMetaStoreClient already catches the exception and does a LOG.error. So the stack trace still appears in the logs. This addresses bug HCATALOG-363 . https://issues.apache.org/jira/browse/HCATALOG-363 Diffs (updated) http://svn.apache.org/repos/asf/incubator/hcatalog/branches/branch-0.4/src/java/org/apache/hcatalog/common/HCatUtil.java 1327178 http://svn.apache.org/repos/asf/incubator/hcatalog/branches/branch-0.4/src/java/org/apache/hcatalog/mapreduce/DefaultOutputCommitterContainer.java 1327178 http://svn.apache.org/repos/asf/incubator/hcatalog/branches/branch-0.4/src/java/org/apache/hcatalog/mapreduce/FileOutputCommitterContainer.java 1327178 http://svn.apache.org/repos/asf/incubator/hcatalog/branches/branch-0.4/src/java/org/apache/hcatalog/mapreduce/FileOutputFormatContainer.java 1327178 http://svn.apache.org/repos/asf/incubator/hcatalog/branches/branch-0.4/src/java/org/apache/hcatalog/mapreduce/HCatOutputFormat.java 1327178 http://svn.apache.org/repos/asf/incubator/hcatalog/branches/branch-0.4/src/java/org/apache/hcatalog/mapreduce/InitializeInput.java 1327178 http://svn.apache.org/repos/asf/incubator/hcatalog/branches/branch-0.4/src/java/org/apache/hcatalog/pig/PigHCatUtil.java 1327178 Diff: https://reviews.apache.org/r/4701/diff Testing ------- Unit tests pass. Did an integration test issuing pig commands. e2e tests not run. Thanks, Rohini

          committed to trunk and branch-0.4. Thanks Rohini

          toffer Francis Christopher Liu added a comment - committed to trunk and branch-0.4. Thanks Rohini
          gates Alan Gates added a comment -

          Issue closed with 0.4 release.

          gates Alan Gates added a comment - Issue closed with 0.4 release.

          People

            rohini Rohini Palaniswamy
            ashutoshc Ashutosh Chauhan
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: