Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-3050

rework OEV to share more code with the NameNode

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.0-alpha
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Current, OEV (the offline edits viewer) re-implements all of the opcode parsing logic found in the NameNode. This duplicated code creates a maintenance burden for us.

      OEV should be refactored to simply use the normal EditLog parsing code, rather than rolling its own. By using the existing FSEditLogLoader code to load edits in OEV, we can avoid having to update two places when the format changes.

      We should not put opcode checksums into the XML, because they are a serialization detail, not related to what the data is what we're storing. This will also make it possible to modify the XML file and translate this modified file back to a binary edits log file.

      Finally, this changes introduces --fix-txids. When OEV is passed this flag, it will close gaps in the transaction log by modifying the sequence numbers. This is useful if you want to modify the edit log XML (say, by removing a transaction), and transform the modified XML back into a valid binary edit log file.

        Attachments

        1. HDFS-3050.006.patch
          119 kB
          Colin P. McCabe
        2. HDFS-3050.007.patch
          110 kB
          Colin P. McCabe
        3. HDFS-3050.008.patch
          119 kB
          Colin P. McCabe
        4. HDFS-3050.009.patch
          121 kB
          Colin P. McCabe
        5. HDFS-3050.010.patch
          120 kB
          Colin P. McCabe
        6. HDFS-3050.011.patch
          120 kB
          Colin P. McCabe
        7. HDFS-3050.012.patch
          120 kB
          Colin P. McCabe
        8. HDFS-3050.014.patch
          200 kB
          Colin P. McCabe
        9. HDFS-3050.015.patch
          186 kB
          Colin P. McCabe
        10. HDFS-3050.016.patch
          185 kB
          Colin P. McCabe
        11. HDFS-3050.017.patch
          184 kB
          Colin P. McCabe
        12. HDFS-3050.018.patch
          184 kB
          Colin P. McCabe
        13. HDFS-3050.019.patch
          184 kB
          Colin P. McCabe
        14. HDFS-3050.020.patch
          184 kB
          Colin P. McCabe

          Issue Links

            Activity

              People

              • Assignee:
                cmccabe Colin P. McCabe
                Reporter:
                cmccabe Colin P. McCabe
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: