Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-4122

Cleanup HDFS logs and reduce the size of logged messages

    Details

    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Hide
      The change from this jira changes the content of some of the log messages. No log message are removed. Only the content of the log messages is changed to reduce the size. If you have a tool that depends on the exact content of the log, please look at the patch and make appropriate updates to the tool.
      Show
      The change from this jira changes the content of some of the log messages. No log message are removed. Only the content of the log messages is changed to reduce the size. If you have a tool that depends on the exact content of the log, please look at the patch and make appropriate updates to the tool.

      Description

      I have attached the patch that removes unnecessary information from the log such as "Namesystem." and prefixing file information with "file" and block information with "block" etc.

      On a branch-1 log I saw it reduce the log size by ~10%.

      1. log.chatty.cleanup.branch1.patch
        41 kB
        Suresh Srinivas
      2. log.chatty.cleanup.trunk.patch
        25 kB
        Suresh Srinivas
      3. log.chatty.cleanup.trunk.patch
        56 kB
        Suresh Srinivas
      4. log.chatty.cleanup.trunk.patch
        56 kB
        Suresh Srinivas
      5. log.chatty.cleanup.branch1.patch
        67 kB
        Suresh Srinivas
      6. log.chatty.cleanup.branch1.patch
        84 kB
        Suresh Srinivas

        Issue Links

          Activity

          Suresh Srinivas created issue -
          Suresh Srinivas made changes -
          Field Original Value New Value
          Attachment log.chatty.cleanup.branch1.patch [ 12551091 ]
          Suresh Srinivas made changes -
          Attachment log.chatty.cleanup.trunk.patch [ 12551093 ]
          Suresh Srinivas made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Suresh Srinivas made changes -
          Attachment log.chatty.cleanup.trunk.patch [ 12551094 ]
          Suresh Srinivas made changes -
          Attachment log.chatty.cleanup.trunk.patch [ 12551134 ]
          Tsz Wo Nicholas Sze made changes -
          Hadoop Flags Reviewed [ 10343 ]
          Component/s data-node [ 12312927 ]
          Component/s name-node [ 12312926 ]
          Suresh Srinivas made changes -
          Attachment log.chatty.cleanup.branch1.patch [ 12551135 ]
          Suresh Srinivas made changes -
          Attachment log.chatty.cleanup.branch1.patch [ 12551140 ]
          Suresh Srinivas made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Tsz Wo Nicholas Sze made changes -
          Component/s hdfs client [ 12312928 ]
          Suresh Srinivas made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Fix Version/s 1.2.0 [ 12321657 ]
          Fix Version/s 3.0.0 [ 12320356 ]
          Resolution Fixed [ 1 ]
          Suresh Srinivas made changes -
          Release Note The change from this jira changes the content of some of the log messages. No log message are removed. Only the content of the log messages is changed to reduce the size. If you have a tool that depends on the exact content of the log, please look at the patch and make appropriate updates to the tool.
          Suresh Srinivas made changes -
          Fix Version/s 2.0.3-alpha [ 12323274 ]
          Todd Lipcon made changes -
          Hadoop Flags Reviewed [ 10343 ] Incompatible change,Reviewed [ 10342, 10343 ]
          Tsz Wo Nicholas Sze made changes -
          Link This issue breaks HDFS-4180 [ HDFS-4180 ]
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Chris Nauroth made changes -
          Link This issue is related to HDFS-4782 [ HDFS-4782 ]
          Allen Wittenauer made changes -
          Fix Version/s 3.0.0 [ 12320356 ]

            People

            • Assignee:
              Suresh Srinivas
              Reporter:
              Suresh Srinivas
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development