Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-5944

LeaseManager:findLeaseWithPrefixPath can't handle path like /a/b/ right and cause SecondaryNameNode failed do checkpoint

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.2.0, 2.2.0
    • Fix Version/s: 1.3.0, 2.4.0
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      In our cluster, we encountered error like this:
      java.io.IOException: saveLeases found path /XXX/20140206/04_30/_SUCCESS.slc.log but is not under construction.
      at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.saveFilesUnderConstruction(FSNamesystem.java:6217)
      at org.apache.hadoop.hdfs.server.namenode.FSImageFormat$Saver.save(FSImageFormat.java:607)
      at org.apache.hadoop.hdfs.server.namenode.FSImage.saveCurrent(FSImage.java:1004)
      at org.apache.hadoop.hdfs.server.namenode.FSImage.saveNamespace(FSImage.java:949)

      What happened:
      Client A open file /XXX/20140206/04_30/_SUCCESS.slc.log for write.
      And Client A continue refresh it's lease.
      Client B deleted /XXX/20140206/04_30/
      Client C open file /XXX/20140206/04_30/_SUCCESS.slc.log for write
      Client C closed the file /XXX/20140206/04_30/_SUCCESS.slc.log
      Then secondaryNameNode try to do checkpoint and failed due to failed to delete lease hold by Client A when Client B deleted /XXX/20140206/04_30/.

      The reason is a bug in findLeaseWithPrefixPath:
      int srclen = prefix.length();
      if (p.length() == srclen || p.charAt(srclen) == Path.SEPARATOR_CHAR)

      { entries.put(entry.getKey(), entry.getValue()); }

      Here when prefix is /XXX/20140206/04_30/, and p is /XXX/20140206/04_30/SUCCESS.slc.log, p.charAt(srcllen) is ''.
      The fix is simple, I'll upload patch later.

        Attachments

        1. HDFS-5944.patch
          3 kB
          yunjiong zhao
        2. HDFS-5944.test.txt
          0.8 kB
          Brandon Li
        3. HDFS-5944.trunk.patch
          3 kB
          Brandon Li
        4. HDFS-5944-branch-1.2.patch
          3 kB
          yunjiong zhao

          Activity

            People

            • Assignee:
              zhaoyunjiong yunjiong zhao
              Reporter:
              zhaoyunjiong yunjiong zhao
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: