Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-4161

HDFS keeps a thread open for every file writer

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.0
    • Fix Version/s: 1.1.1
    • Component/s: hdfs-client
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      In 1.0 release DFSClient uses a thread per file writer. In some use cases (dynamic partions in hive) that use a large number of file writers a large number of threads are created. The file writer thread has the following stack:

      at java.lang.Thread.sleep(Native Method)
      at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1462)
      at java.lang.Thread.run(Thread.java:662)
      

      This problem has been fixed in later releases. This jira will post a consolidated patch from various jiras that addresses the issue.

      1. h4161_20121108_b-1.patch
        41 kB
        Tsz Wo Nicholas Sze

        Issue Links

          Activity

          Hide
          Matt Foley added a comment -

          Closed upon release of 1.1.1.

          Show
          Matt Foley added a comment - Closed upon release of 1.1.1.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I have committed this.

          Show
          Tsz Wo Nicholas Sze added a comment - I have committed this.
          Hide
          Suresh Srinivas added a comment -

          +1 for the change.

          Show
          Suresh Srinivas added a comment - +1 for the change.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          After updated with HDFS-4168, all the failed tests are now passing.

          Show
          Tsz Wo Nicholas Sze added a comment - After updated with HDFS-4168 , all the failed tests are now passing.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          The findbugs warnings are not related to the patch.

          All unit test passed except for the ones mentioned in HDFS-4168.

          Show
          Tsz Wo Nicholas Sze added a comment - The findbugs warnings are not related to the patch. All unit test passed except for the ones mentioned in HDFS-4168 .
          Hide
          Tsz Wo Nicholas Sze added a comment -

          test-patch results:

               [exec] -1 overall.  
               [exec] 
               [exec]     +1 @author.  The patch does not contain any @author tags.
               [exec] 
               [exec]     +1 tests included.  The patch appears to include 21 new or modified tests.
               [exec] 
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
               [exec] 
               [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
               [exec] 
               [exec]     -1 findbugs.  The patch appears to introduce 9 new Findbugs (version 1.3.9) warnings.
          
          Show
          Tsz Wo Nicholas Sze added a comment - test-patch results: [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 21 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] -1 findbugs. The patch appears to introduce 9 new Findbugs (version 1.3.9) warnings.
          Hide
          Suresh Srinivas added a comment -

          Thank you Nicholas. Can you post test-patch output. Perhaps we should also run all the unit tests.

          Show
          Suresh Srinivas added a comment - Thank you Nicholas. Can you post test-patch output. Perhaps we should also run all the unit tests.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          h4161_20121108_b-1.patch: consolidated patch with all the changes.

          Show
          Tsz Wo Nicholas Sze added a comment - h4161_20121108_b-1.patch: consolidated patch with all the changes.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          HDFS-1865 "Share LeaseChecker thread among DFSClients" is the main JIRA for solving this problem. In order to backport it and the related bug fixes, we have the following JIRA sequence:

          • HDFS-278: Should DFS outputstream's close wait forever?
          • HDFS-1840: Terminate LeaseChecker when all writing files are closed.
          • HDFS-1870: Refactor DFSClient.LeaseChecker
          • HDFS-1890: A few improvements on the LeaseRenewer.pendingCreates map
          • HDFS-1865: Share LeaseChecker thread among DFSClients
          • HDFS-2810: Leases not properly getting renewed by clients
          • HDFS-3646: LeaseRenewer can hold reference to inactive DFSClient instances forever
          • HDFS-2240: Possible deadlock between LeaseRenewer and its factory
          Show
          Tsz Wo Nicholas Sze added a comment - HDFS-1865 "Share LeaseChecker thread among DFSClients" is the main JIRA for solving this problem. In order to backport it and the related bug fixes, we have the following JIRA sequence: HDFS-278 : Should DFS outputstream's close wait forever? HDFS-1840 : Terminate LeaseChecker when all writing files are closed. HDFS-1870 : Refactor DFSClient.LeaseChecker HDFS-1890 : A few improvements on the LeaseRenewer.pendingCreates map HDFS-1865 : Share LeaseChecker thread among DFSClients HDFS-2810 : Leases not properly getting renewed by clients HDFS-3646 : LeaseRenewer can hold reference to inactive DFSClient instances forever HDFS-2240 : Possible deadlock between LeaseRenewer and its factory

            People

            • Assignee:
              Tsz Wo Nicholas Sze
              Reporter:
              Suresh Srinivas
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development