Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-2305

Running multiple 2NNs can result in corrupt file system


    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.2
    • Fix Version/s: 1.1.0
    • Component/s: namenode
    • Labels:
    • Hadoop Flags:


      Here's the scenario:

      • You run the NN and 2NN (2NN A) on the same machine.
      • You don't have the address of the 2NN configured, so it's defaulting to
      • There's another 2NN (2NN B) running on a second machine.
      • When a 2NN is done checkpointing, it says "hey NN, I have an updated fsimage for you. You can download it from this URL, which includes my IP address, which is x"

      And here's the steps that occur to cause this issue:

      1. Some edits happen.
      2. 2NN A (on the NN machine) does a checkpoint. All is dandy.
      3. Some more edits happen.
      4. 2NN B (on a different machine) does a checkpoint. It tells the NN "grab the newly-merged fsimage file from"
      5. NN happily grabs the fsimage from 2NN A (the 2NN on the NN machine), which is stale.
      6. NN renames edits.new file to edits. At this point the in-memory FS state is fine, but the on-disk state is missing edits.
      7. The next time a 2NN (any 2NN) tries to do a checkpoint, it gets an up-to-date edits file, with an outdated fsimage, and tries to apply those edits to that fsimage.
      8. Kaboom.
      1. hdfs-2305-test.patch
        3 kB
        Aaron T. Myers
      2. hdfs-2305.1.patch
        16 kB
        Aaron T. Myers
      3. hdfs-2305.0.patch
        15 kB
        Aaron T. Myers

        Issue Links


          Matt Foley made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Aaron T. Myers made changes -
          Link This issue relates to HDFS-2549 [ HDFS-2549 ]
          Aaron T. Myers made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Fix Version/s [ 12317959 ]
          Resolution Fixed [ 1 ]
          Aaron T. Myers made changes -
          Attachment hdfs-2305.1.patch [ 12494158 ]
          Aaron T. Myers made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Aaron T. Myers made changes -
          Attachment hdfs-2305.0.patch [ 12492676 ]
          Aaron T. Myers made changes -
          Field Original Value New Value
          Attachment hdfs-2305-test.patch [ 12492562 ]
          Aaron T. Myers created issue -


            • Assignee:
              Aaron T. Myers
              Aaron T. Myers
            • Votes:
              0 Vote for this issue
              12 Start watching this issue


              • Created: