Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-1623 High Availability Framework for HDFS NN
  3. HDFS-2874

HA: edit log should log to shared dirs before local dirs

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • HA branch (HDFS-1623)
    • HA branch (HDFS-1623)
    • ha, namenode
    • None

    Description

      Currently, the NN logs its edits to each of its edits directories in sequence. This can produce the following bad sequence:

      • NN accumulates 100 edits (tx 1-100) in the buffer. Writes and syncs to local drive, then crashes
      • Failover occurs. SBN takes over at txid=1, since txid 1 never got writen.
      • First NN restarts. It reads up to txid 100 from its local directories. It is now "ahead" of the active NN with inconsistent state.
        The solution is to write to the shared edits dir, and sync that, before writing to any local drives.

      Attachments

        1. hdfs-2874.txt
          24 kB
          Todd Lipcon
        2. hdfs-2874.txt
          24 kB
          Todd Lipcon
        3. hdfs-2874.txt
          9 kB
          Todd Lipcon

        Issue Links

          Activity

            People

              tlipcon Todd Lipcon
              tlipcon Todd Lipcon
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: