Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-4080

Add a separate logger for block state change logs to enable turning off those logs

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.3-alpha, 3.0.0-alpha1
    • Fix Version/s: 2.0.3-alpha, 0.23.5
    • Component/s: namenode
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Add a separate logger "BlockStateChange" for block state change logs.

      Description

      Although the block-level logging in namenode is useful for debugging, it can add a significant overhead to busy hdfs clusters since they are done while the namespace write lock is held. One example is shown in HDFS-4075. In this example, the write lock was held for 5 minutes while logging 11 million log messages for 5.5 million block invalidation events.

      It will be useful if we have an option to disable these block-level log messages and keep other state change messages going. If others feel that they can turned into DEBUG (with addition of isDebugEnabled() checks), that may also work too, but there might be people depending on the messages.

        Attachments

        1. hdfs-4080.patch
          33 kB
          Kihwal Lee
        2. hdfs-4080.1.patch
          32 kB
          Kihwal Lee
        3. hdfs-4080.patch
          23 kB
          Kihwal Lee
        4. hdfs-4080-branch-0.23.patch
          23 kB
          Kihwal Lee
        5. hdfs-4080-trunk.patch
          24 kB
          Kihwal Lee

          Issue Links

            Activity

              People

              • Assignee:
                kihwal Kihwal Lee
                Reporter:
                kihwal Kihwal Lee
              • Votes:
                0 Vote for this issue
                Watchers:
                11 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: