Hadoop Common
  1. Hadoop Common
  2. HADOOP-4961

ConcurrentModificationException in lease recovery of empty files.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.18.0
    • Fix Version/s: 0.18.3
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The problem is that internalReleaseLease() finalizes empty files, which removes these file path names from the lease. So this modifies the Collection of file names which LeaseManager.checkLeases() is iterating on.

      1. TestLeaseExpire.java
        3 kB
        Tsz Wo Nicholas Sze
      2. CMEinLeaseRecovery-0-19.patch
        5 kB
        Konstantin Shvachko
      3. CMEinLeaseRecovery-0-18.patch
        4 kB
        Konstantin Shvachko
      4. CMEinLeaseRecovery.patch
        4 kB
        Konstantin Shvachko

        Issue Links

          Activity

          Hide
          Tsz Wo Nicholas Sze added a comment -

          TestLeaseExpire.java: a unit test for the problem.

          Show
          Tsz Wo Nicholas Sze added a comment - TestLeaseExpire.java: a unit test for the problem.
          Hide
          Konstantin Shvachko added a comment -

          This exception is found in the name-node *.out file:

          Exception in thread "org.apache.hadoop.dfs.LeaseManager$Monitor@2244d990" java.util.ConcurrentModificationException
                  at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1100)
                  at java.util.TreeMap$KeyIterator.next(TreeMap.java:1154)
                  at org.apache.hadoop.dfs.LeaseManager.checkLeases(LeaseManager.java:371)
                  at org.apache.hadoop.dfs.LeaseManager.access$800(LeaseManager.java:51)
                  at org.apache.hadoop.dfs.LeaseManager$Monitor.run(LeaseManager.java:346)
                  at java.lang.Thread.run(Thread.java:619)
          
          Show
          Konstantin Shvachko added a comment - This exception is found in the name-node *.out file: Exception in thread "org.apache.hadoop.dfs.LeaseManager$Monitor@2244d990" java.util.ConcurrentModificationException at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1100) at java.util.TreeMap$KeyIterator.next(TreeMap.java:1154) at org.apache.hadoop.dfs.LeaseManager.checkLeases(LeaseManager.java:371) at org.apache.hadoop.dfs.LeaseManager.access$800(LeaseManager.java:51) at org.apache.hadoop.dfs.LeaseManager$Monitor.run(LeaseManager.java:346) at java.lang. Thread .run( Thread .java:619)
          Hide
          Konstantin Shvachko added a comment -

          This fixes the problem. The test provided by Nicholas fails on with current code but succeeds with the changes.

          Show
          Konstantin Shvachko added a comment - This fixes the problem. The test provided by Nicholas fails on with current code but succeeds with the changes.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          +1 patch looks good

          Show
          Tsz Wo Nicholas Sze added a comment - +1 patch looks good
          Hide
          Tsz Wo Nicholas Sze added a comment -
               [exec] +1 overall.  
               [exec] 
               [exec]     +1 @author.  The patch does not contain any @author tags.
               [exec] 
               [exec]     +1 tests included.  The patch appears to include 3 new or modified tests.
               [exec] 
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
               [exec] 
               [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
               [exec] 
               [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
               [exec] 
               [exec]     +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
          
          Show
          Tsz Wo Nicholas Sze added a comment - [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
          Hide
          Konstantin Shvachko added a comment -

          Unit tests: BUILD SUCCESSFUL

          Show
          Konstantin Shvachko added a comment - Unit tests: BUILD SUCCESSFUL
          Hide
          Konstantin Shvachko added a comment -

          A separate patch for branch 0.19 is required.

          Show
          Konstantin Shvachko added a comment - A separate patch for branch 0.19 is required.
          Hide
          Konstantin Shvachko added a comment -

          I just committed this.
          Thank you Nicholas and Sameer.

          Show
          Konstantin Shvachko added a comment - I just committed this. Thank you Nicholas and Sameer.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-trunk #706 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/706/)
          . Fix ConcurrentModificationException in lease recovery of empty files. Contributed by Konstantin Shvachko and Nicholas.

          Show
          Hudson added a comment - Integrated in Hadoop-trunk #706 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/706/ ) . Fix ConcurrentModificationException in lease recovery of empty files. Contributed by Konstantin Shvachko and Nicholas.

            People

            • Assignee:
              Konstantin Shvachko
              Reporter:
              Konstantin Shvachko
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development