Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-438

DFS pathname limitation.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.1.0, 0.1.1, 0.2.0, 0.2.1, 0.3.0, 0.3.1, 0.3.2, 0.4.0, 0.5.0
    • Fix Version/s: 0.7.0
    • Component/s: None
    • Labels:
      None

      Description

      I was trying to create a deep hierarchy of directories using DFS mkdirs().
      When the path to the leaf directory became long (~20000) DFS was still able to create
      directories with these names, but UTF8 started truncating long strings resulting in
      incorrect logging of namespace edits. That later crashed the namenode during restart,
      when it was trying to reproduce file creation logged in the edits file with truncated names.
      UTF8 is deprecated now so we will have to replace it with Text.
      With UTF8 we should enforce a pathname limit of 0xffff/3 = 21845
      With Text it is going to be larger. Not sure what the exact number is.

        Attachments

        1. pathlength.patch
          4 kB
          Wendy Chien

          Activity

            People

            • Assignee:
              wchien Wendy Chien
              Reporter:
              shv Konstantin Shvachko
            • Votes:
              1 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: