Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-4122

Cleanup HDFS logs and reduce the size of logged messages

    Details

    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Hide
      The change from this jira changes the content of some of the log messages. No log message are removed. Only the content of the log messages is changed to reduce the size. If you have a tool that depends on the exact content of the log, please look at the patch and make appropriate updates to the tool.
      Show
      The change from this jira changes the content of some of the log messages. No log message are removed. Only the content of the log messages is changed to reduce the size. If you have a tool that depends on the exact content of the log, please look at the patch and make appropriate updates to the tool.

      Description

      I have attached the patch that removes unnecessary information from the log such as "Namesystem." and prefixing file information with "file" and block information with "block" etc.

      On a branch-1 log I saw it reduce the log size by ~10%.

      1. log.chatty.cleanup.branch1.patch
        41 kB
        Suresh Srinivas
      2. log.chatty.cleanup.trunk.patch
        25 kB
        Suresh Srinivas
      3. log.chatty.cleanup.trunk.patch
        56 kB
        Suresh Srinivas
      4. log.chatty.cleanup.trunk.patch
        56 kB
        Suresh Srinivas
      5. log.chatty.cleanup.branch1.patch
        67 kB
        Suresh Srinivas
      6. log.chatty.cleanup.branch1.patch
        84 kB
        Suresh Srinivas

        Issue Links

          Activity

          Allen Wittenauer made changes -
          Fix Version/s 3.0.0 [ 12320356 ]
          Chris Nauroth made changes -
          Link This issue is related to HDFS-4782 [ HDFS-4782 ]
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Tsz Wo Nicholas Sze made changes -
          Link This issue breaks HDFS-4180 [ HDFS-4180 ]
          Todd Lipcon made changes -
          Hadoop Flags Reviewed [ 10343 ] Incompatible change,Reviewed [ 10342, 10343 ]
          Suresh Srinivas made changes -
          Fix Version/s 2.0.3-alpha [ 12323274 ]
          Suresh Srinivas made changes -
          Release Note The change from this jira changes the content of some of the log messages. No log message are removed. Only the content of the log messages is changed to reduce the size. If you have a tool that depends on the exact content of the log, please look at the patch and make appropriate updates to the tool.
          Suresh Srinivas made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Fix Version/s 1.2.0 [ 12321657 ]
          Fix Version/s 3.0.0 [ 12320356 ]
          Resolution Fixed [ 1 ]
          Tsz Wo Nicholas Sze made changes -
          Component/s hdfs client [ 12312928 ]
          Suresh Srinivas made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Suresh Srinivas made changes -
          Attachment log.chatty.cleanup.branch1.patch [ 12551140 ]
          Suresh Srinivas made changes -
          Attachment log.chatty.cleanup.branch1.patch [ 12551135 ]
          Tsz Wo Nicholas Sze made changes -
          Hadoop Flags Reviewed [ 10343 ]
          Component/s data-node [ 12312927 ]
          Component/s name-node [ 12312926 ]
          Suresh Srinivas made changes -
          Attachment log.chatty.cleanup.trunk.patch [ 12551134 ]
          Suresh Srinivas made changes -
          Attachment log.chatty.cleanup.trunk.patch [ 12551094 ]
          Suresh Srinivas made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Suresh Srinivas made changes -
          Attachment log.chatty.cleanup.trunk.patch [ 12551093 ]
          Suresh Srinivas made changes -
          Field Original Value New Value
          Attachment log.chatty.cleanup.branch1.patch [ 12551091 ]
          Suresh Srinivas created issue -

            People

            • Assignee:
              Suresh Srinivas
              Reporter:
              Suresh Srinivas
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development