Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1448

Create multi-format parser for edits logs file, support binary and XML formats initially

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.23.0
    • Component/s: tools
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Offline edits viewer feature adds oev tool to hdfs script. Oev makes it possible to convert edits logs to/from native binary and XML formats. It uses the same framework as Offline image viewer.

      Example usage:

      $HADOOP_HOME/bin/hdfs oev -i edits -o output.xml
      Show
      Offline edits viewer feature adds oev tool to hdfs script. Oev makes it possible to convert edits logs to/from native binary and XML formats. It uses the same framework as Offline image viewer. Example usage: $HADOOP_HOME/bin/hdfs oev -i edits -o output.xml

      Description

      Create multi-format parser for edits logs file, support binary and XML formats initially.

      Parsing should work from any supported format to any other supported format (e.g. from binary to XML and from XML to binary).

      The binary format is the format used by FSEditLog class to read/write edits file.

      Primary reason to develop this tool is to help with troubleshooting, the binary format is hard to read and edit (for human troubleshooters).

      Longer term it could be used to clean up and minimize parsers for fsimage and edits files. Edits parser OfflineEditsViewer is written in a very similar fashion to OfflineImageViewer. Next step would be to merge OfflineImageViewer and OfflineEditsViewer and use the result in both FSImage and FSEditLog. This is subject to change, specifically depending on adoption of avro (which would completely change how objects are serialized as well as provide ways to convert files to different formats).

      1. CDH-4355.txt
        1 kB
        Colin Patrick McCabe
      2. HDFS-1448-0.22-5.patch
        147 kB
        Erik Steffl
      3. HDFS-1448-0.22-4.patch
        146 kB
        Erik Steffl
      4. HDFS-1448-0.22-3.patch
        138 kB
        Erik Steffl
      5. HDFS-1448-0.22-2.patch
        110 kB
        Erik Steffl
      6. HDFS-1448-0.22-1.patch
        110 kB
        Erik Steffl
      7. Viewer hierarchy.pdf
        44 kB
        Jakob Homan
      8. editsStored
        1.00 MB
        Erik Steffl
      9. HDFS-1448-0.22.patch
        94 kB
        Erik Steffl

        Issue Links

          Activity

          Erik Steffl created issue -
          Erik Steffl made changes -
          Field Original Value New Value
          Attachment HDFS-1448-0.22.patch [ 12456746 ]
          Attachment editsStored [ 12456747 ]
          Jeff Hammerbacher made changes -
          Link This issue is related to HDFS-842 [ HDFS-842 ]
          Jakob Homan made changes -
          Attachment Viewer hierarchy.pdf [ 12457088 ]
          Jakob Homan made changes -
          Priority Minor [ 4 ] Major [ 3 ]
          Konstantin Shvachko made changes -
          Assignee Erik Steffl [ zasran ]
          Jakob Homan made changes -
          Link This issue is duplicated by HDFS-312 [ HDFS-312 ]
          Jakob Homan made changes -
          Link This issue is duplicated by HDFS-209 [ HDFS-209 ]
          Erik Steffl made changes -
          Attachment HDFS-1448-0.22-1.patch [ 12458974 ]
          Erik Steffl made changes -
          Attachment HDFS-1448-0.22-2.patch [ 12459099 ]
          Erik Steffl made changes -
          Attachment HDFS-1448-0.22-3.patch [ 12460087 ]
          Erik Steffl made changes -
          Attachment HDFS-1448-0.22-4.patch [ 12466278 ]
          Erik Steffl made changes -
          Attachment HDFS-1448-0.22-5.patch [ 12466777 ]
          Jakob Homan made changes -
          Fix Version/s 0.23.0 [ 12315571 ]
          Fix Version/s 0.22.0 [ 12314241 ]
          Tsz Wo Nicholas Sze made changes -
          Hadoop Flags [Reviewed]
          Tsz Wo Nicholas Sze made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Erik Steffl made changes -
          Release Note Offline edits viewer feature adds oev tool to hdfs script. Oev makes it possible to convert edits logs to/from native binary and XML formats. It uses the same framework as Offline image viewer.

          Example usage:

          $HADOOP_HOME/bin/hdfs oev -i edits -o output.xml
          Colin Patrick McCabe made changes -
          Attachment CDH-4355.txt [ 12515043 ]

            People

            • Assignee:
              Erik Steffl
              Reporter:
              Erik Steffl
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development