Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-5535 Umbrella jira for improved HDFS rolling upgrades
  3. HDFS-6038

Allow JournalNode to handle editlog produced by new release with future layoutversion

    Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.4.0
    • Component/s: journal-node, namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      In HA setup, the JNs receive edit logs (blob) from the NN and write into edit log files. In order to write well-formed edit log files, the JNs prepend a header for each edit log file. The problem is that the JN hard-codes the version (i.e., NameNodeLayoutVersion in the edit log, therefore it generates incorrect edit logs when the newer release bumps the NameNodeLayoutVersion during rolling upgrade.

      In the meanwhile, currently JN tries to decode the in-progress editlog segment in order to know the last txid in the segment. In the rolling upgrade scenario, the JN with the old software may not be able to correctly decode the editlog generated by the new software.

      This jira makes the following changes to allow JN to handle editlog produced by software with future layoutversion:
      1. Change the NN--JN startLogSegment RPC signature and let NN specify the layoutversion for the new editlog segment.
      2. Persist a length field for each editlog op to indicate the total length of the op. Instead of calling EditLogFileInputStream#validateEditLog to get the last txid of an in-progress editlog segment, a new method scanEditLog is added and used by JN which does not decode each editlog op but uses the length to quickly jump to the next op.

        Attachments

        1. editsStored
          5 kB
          Jing Zhao
        2. HDFS-6038.000.patch
          50 kB
          Jing Zhao
        3. HDFS-6038.001.patch
          65 kB
          Jing Zhao
        4. HDFS-6038.002.patch
          88 kB
          Jing Zhao
        5. HDFS-6038.003.patch
          107 kB
          Jing Zhao
        6. HDFS-6038.004.patch
          107 kB
          Jing Zhao
        7. HDFS-6038.005.patch
          109 kB
          Jing Zhao
        8. HDFS-6038.006.patch
          109 kB
          Jing Zhao
        9. HDFS-6038.007.patch
          109 kB
          Jing Zhao
        10. HDFS-6038.008.patch
          117 kB
          Jing Zhao

          Issue Links

            Activity

              People

              • Assignee:
                jingzhao Jing Zhao
                Reporter:
                wheat9 Haohui Mai
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: