Hive
  1. Hive
  2. HIVE-2151

Too many open files in running negative cli tests

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.0
    • Component/s: CLI, Testing Infrastructure
    • Labels:
      None
    1. HIVE-2151.1.patch
      1 kB
      He Yongqiang
    2. HIVE-2151.2.patch
      1 kB
      He Yongqiang
    3. HIVE-2151.3.patch
      4 kB
      He Yongqiang
    4. HIVE-2151.4.patch
      4 kB
      He Yongqiang

      Activity

      He Yongqiang created issue -
      Hide
      Ashutosh Chauhan added a comment -

      I have hit this issue too.

      Show
      Ashutosh Chauhan added a comment - I have hit this issue too.
      He Yongqiang made changes -
      Field Original Value New Value
      Attachment HIVE-2151.1.patch [ 12478385 ]
      He Yongqiang made changes -
      Status Open [ 1 ] Patch Available [ 10002 ]
      Hide
      jiraposter@reviews.apache.org added a comment -

      -----------------------------------------------------------
      This is an automatically generated e-mail. To reply, visit:
      https://reviews.apache.org/r/696/
      -----------------------------------------------------------

      Review request for hive.

      Summary
      -------

      Too many open files in running negative cli tests

      This addresses bug HIVE-2151.
      https://issues.apache.org/jira/browse/HIVE-2151

      Diffs


      trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1100117

      Diff: https://reviews.apache.org/r/696/diff

      Testing
      -------

      Thanks,

      Yongqiang

      Show
      jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/696/ ----------------------------------------------------------- Review request for hive. Summary ------- Too many open files in running negative cli tests This addresses bug HIVE-2151 . https://issues.apache.org/jira/browse/HIVE-2151 Diffs trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1100117 Diff: https://reviews.apache.org/r/696/diff Testing ------- Thanks, Yongqiang
      Hide
      Ning Zhang added a comment -

      We did an offline review and found a better fix. Yongqiang will upload a new patch soon.

      Show
      Ning Zhang added a comment - We did an offline review and found a better fix. Yongqiang will upload a new patch soon.
      Ning Zhang made changes -
      Status Patch Available [ 10002 ] Open [ 1 ]
      He Yongqiang made changes -
      Attachment HIVE-2151.2.patch [ 12478461 ]
      Hide
      jiraposter@reviews.apache.org added a comment -

      -----------------------------------------------------------
      This is an automatically generated e-mail. To reply, visit:
      https://reviews.apache.org/r/696/
      -----------------------------------------------------------

      (Updated 2011-05-06 19:17:48.260294)

      Review request for hive.

      Changes
      -------

      new diff

      Summary
      -------

      Too many open files in running negative cli tests

      This addresses bug HIVE-2151.
      https://issues.apache.org/jira/browse/HIVE-2151

      Diffs (updated)


      trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1100117

      Diff: https://reviews.apache.org/r/696/diff

      Testing
      -------

      Thanks,

      Yongqiang

      Show
      jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/696/ ----------------------------------------------------------- (Updated 2011-05-06 19:17:48.260294) Review request for hive. Changes ------- new diff Summary ------- Too many open files in running negative cli tests This addresses bug HIVE-2151 . https://issues.apache.org/jira/browse/HIVE-2151 Diffs (updated) trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1100117 Diff: https://reviews.apache.org/r/696/diff Testing ------- Thanks, Yongqiang
      He Yongqiang made changes -
      Attachment HIVE-2151.3.patch [ 12478472 ]
      He Yongqiang made changes -
      Status Open [ 1 ] Patch Available [ 10002 ]
      Hide
      Ning Zhang added a comment -

      Can you update the review board with the latest patch? It's much easier to add comments.

      In ZooKeeperHiveLockManager.close(), I'm not sure if the change is desired. According to the comment, it simply close the client to release transient locks. The later removeAllRedundantNodes() call may be needed after the client close to clean up further. So changing the order may results in undesired behavior. Namit, can you comment on this?

      Driver.java:148, add comments wht ctx needs to set HiveLockManager.

      Show
      Ning Zhang added a comment - Can you update the review board with the latest patch? It's much easier to add comments. In ZooKeeperHiveLockManager.close(), I'm not sure if the change is desired. According to the comment, it simply close the client to release transient locks. The later removeAllRedundantNodes() call may be needed after the client close to clean up further. So changing the order may results in undesired behavior. Namit, can you comment on this? Driver.java:148, add comments wht ctx needs to set HiveLockManager.
      Hide
      He Yongqiang added a comment -

      removeAllRedundantNodes() leaks a zookeeper connection. it should be put before closing zookeeper instance because it is still using the zookeeper instance.

      Show
      He Yongqiang added a comment - removeAllRedundantNodes() leaks a zookeeper connection. it should be put before closing zookeeper instance because it is still using the zookeeper instance.
      He Yongqiang made changes -
      Attachment HIVE-2151.4.patch [ 12478621 ]
      Hide
      jiraposter@reviews.apache.org added a comment -

      -----------------------------------------------------------
      This is an automatically generated e-mail. To reply, visit:
      https://reviews.apache.org/r/696/
      -----------------------------------------------------------

      (Updated 2011-05-09 17:15:15.221624)

      Review request for hive.

      Changes
      -------

      reverted the change in ZooKeeperHiveLockManager's close, but put a zookeeper close in removeReduantNodes
      added more comments in Driver.

      Summary
      -------

      Too many open files in running negative cli tests

      This addresses bug HIVE-2151.
      https://issues.apache.org/jira/browse/HIVE-2151

      Diffs (updated)


      trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1101116
      trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java 1101116

      Diff: https://reviews.apache.org/r/696/diff

      Testing
      -------

      Thanks,

      Yongqiang

      Show
      jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/696/ ----------------------------------------------------------- (Updated 2011-05-09 17:15:15.221624) Review request for hive. Changes ------- reverted the change in ZooKeeperHiveLockManager's close, but put a zookeeper close in removeReduantNodes added more comments in Driver. Summary ------- Too many open files in running negative cli tests This addresses bug HIVE-2151 . https://issues.apache.org/jira/browse/HIVE-2151 Diffs (updated) trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1101116 trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java 1101116 Diff: https://reviews.apache.org/r/696/diff Testing ------- Thanks, Yongqiang
      Hide
      He Yongqiang added a comment -

      Talked with namit offline, this new patch reverted the change in ZooKeeperHiveLockManager's close, but put a zookeeper close in removeReduantNodes.
      Also added more comments in Driver to address Ning's last comments.

      Show
      He Yongqiang added a comment - Talked with namit offline, this new patch reverted the change in ZooKeeperHiveLockManager's close, but put a zookeeper close in removeReduantNodes. Also added more comments in Driver to address Ning's last comments.
      Hide
      Ning Zhang added a comment -

      +1. Will commit if tests pass

      Show
      Ning Zhang added a comment - +1. Will commit if tests pass
      Hide
      Ning Zhang added a comment -

      Yongqiang, the TestClilDriver failed (index_auto.q, etc.). It seems the change is conflict with the recent bitmap index changes. Can you take a look?

      Show
      Ning Zhang added a comment - Yongqiang, the TestClilDriver failed (index_auto.q, etc.). It seems the change is conflict with the recent bitmap index changes. Can you take a look?
      Ning Zhang made changes -
      Status Patch Available [ 10002 ] Open [ 1 ]
      Hide
      He Yongqiang added a comment -

      Ning, it seems the index_auto.q still fail without this patch. Can you verify?

      Show
      He Yongqiang added a comment - Ning, it seems the index_auto.q still fail without this patch. Can you verify?
      Hide
      John Sichi added a comment -
      Show
      John Sichi added a comment - It has been passing fine in Jenkins: https://builds.apache.org/hudson/job/Hive-trunk-h0.20/717/
      Hide
      He Yongqiang added a comment -

      i got a failure on a clean check out when running index_auto.q. But when i run again, the error is not there. And also after apply the patch, there is also no error.

      Ning, can you do a 'ant clean run'?

      Show
      He Yongqiang added a comment - i got a failure on a clean check out when running index_auto.q. But when i run again, the error is not there. And also after apply the patch, there is also no error. Ning, can you do a 'ant clean run'?
      Hide
      Ning Zhang added a comment -

      OK. I got why it failed. index_auto.q build index to /tmp/index_where. I had run the test before under root, which makes the owner of /tmp/index_where root. Running the same test using my own user will fail because of write permission.

      Going forward I think we should change all index building tests to make results under trunk/build/ rather than /tmp, so that ant clean will clean all side effects.

      Show
      Ning Zhang added a comment - OK. I got why it failed. index_auto.q build index to /tmp/index_where. I had run the test before under root, which makes the owner of /tmp/index_where root. Running the same test using my own user will fail because of write permission. Going forward I think we should change all index building tests to make results under trunk/build/ rather than /tmp, so that ant clean will clean all side effects.
      Hide
      He Yongqiang added a comment -

      Good catch, Ning! At least we should make sure the /tmp/xxx dir got removed at the end of the testcase, if making it use build/tmp/ is not easy to do.

      Show
      He Yongqiang added a comment - Good catch, Ning! At least we should make sure the /tmp/xxx dir got removed at the end of the testcase, if making it use build/tmp/ is not easy to do.
      Hide
      Ning Zhang added a comment -

      Yongqiang, you can reference the build directory using '$

      {system:build.dir.hive}

      ', which is defined in build-common.xml. Can you open a JIRA and fix all such index tests?

      Show
      Ning Zhang added a comment - Yongqiang, you can reference the build directory using '$ {system:build.dir.hive} ', which is defined in build-common.xml. Can you open a JIRA and fix all such index tests?
      Hide
      Ning Zhang added a comment -

      Committed. Thanks Yongqiang!

      Show
      Ning Zhang added a comment - Committed. Thanks Yongqiang!
      Ning Zhang made changes -
      Status Open [ 1 ] Resolved [ 5 ]
      Fix Version/s 0.8.0 [ 12316178 ]
      Resolution Fixed [ 1 ]
      Hide
      Hudson added a comment -

      Integrated in Hive-trunk-h0.20 #726 (See https://builds.apache.org/hudson/job/Hive-trunk-h0.20/726/)

      Show
      Hudson added a comment - Integrated in Hive-trunk-h0.20 #726 (See https://builds.apache.org/hudson/job/Hive-trunk-h0.20/726/ )
      Carl Steinbach made changes -
      Status Resolved [ 5 ] Closed [ 6 ]

        People

        • Assignee:
          He Yongqiang
          Reporter:
          He Yongqiang
        • Votes:
          0 Vote for this issue
          Watchers:
          0 Start watching this issue

          Dates

          • Created:
            Updated:
            Resolved:

            Development