Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-1448

Create multi-format parser for edits logs file, support binary and XML formats initially

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 0.22.0
    • 0.23.0
    • tools
    • None
    • Reviewed
    • Hide
      Offline edits viewer feature adds oev tool to hdfs script. Oev makes it possible to convert edits logs to/from native binary and XML formats. It uses the same framework as Offline image viewer.

      Example usage:

      $HADOOP_HOME/bin/hdfs oev -i edits -o output.xml
      Show
      Offline edits viewer feature adds oev tool to hdfs script. Oev makes it possible to convert edits logs to/from native binary and XML formats. It uses the same framework as Offline image viewer. Example usage: $HADOOP_HOME/bin/hdfs oev -i edits -o output.xml

    Description

      Create multi-format parser for edits logs file, support binary and XML formats initially.

      Parsing should work from any supported format to any other supported format (e.g. from binary to XML and from XML to binary).

      The binary format is the format used by FSEditLog class to read/write edits file.

      Primary reason to develop this tool is to help with troubleshooting, the binary format is hard to read and edit (for human troubleshooters).

      Longer term it could be used to clean up and minimize parsers for fsimage and edits files. Edits parser OfflineEditsViewer is written in a very similar fashion to OfflineImageViewer. Next step would be to merge OfflineImageViewer and OfflineEditsViewer and use the result in both FSImage and FSEditLog. This is subject to change, specifically depending on adoption of avro (which would completely change how objects are serialized as well as provide ways to convert files to different formats).

      Attachments

        1. CDH-4355.txt
          1 kB
          Colin McCabe
        2. editsStored
          1.00 MB
          Erik Steffl
        3. HDFS-1448-0.22.patch
          94 kB
          Erik Steffl
        4. HDFS-1448-0.22-1.patch
          110 kB
          Erik Steffl
        5. HDFS-1448-0.22-2.patch
          110 kB
          Erik Steffl
        6. HDFS-1448-0.22-3.patch
          138 kB
          Erik Steffl
        7. HDFS-1448-0.22-4.patch
          146 kB
          Erik Steffl
        8. HDFS-1448-0.22-5.patch
          147 kB
          Erik Steffl
        9. Viewer hierarchy.pdf
          44 kB
          Jakob Homan

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            zasran Erik Steffl
            zasran Erik Steffl
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment