Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-14693

NameNode should log a warning when EditLog IPC logger's pending size exceeds limit.

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 3.1.2
    • Fix Version/s: 3.3.0, 3.2.1, 3.1.3
    • Component/s: namenode
    • Labels:
      None

      Description

      In a production environment, there may be some differences in each JouranlNode (e.g. network condition, disk condition, and so on). For example, If a JN's network is much worse than other JNs, then the time taken by the NN to write this JN will be much greater than other JNs, in this case, it will cause the IPC Logger thread corresponding to this JN to have many pending edits, when the pending edits exceeds the maximum limit (default 10MB), the new edits about to write to this JN will be silently dropped, and will result gaps in the editlog segment, which causing this JN and NN repeatedly reporting the following errors: 

      org.apache.hadoop.hdfs.qjournal.protocol.JournalOutOfSyncException: Can't write txid 1904164873 expecting nextTxId=1904164871

      Unfortunately, the above error message can not help us quickly find the root cause, It took more time to find the cause, so it's better to add a warning log here, like this: 

      2019-08-02 04:55:05,879 WARN org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager:Pending edits to 192.168.202.13:8485 is going to exceed limit size:10240, current queued edits size:10224, will silently drop 174 bytes of edits!

       This is just a very small improvement.

        Attachments

        1. HDFS-14693.001.patch
          1 kB
          Xudong Cao

          Activity

            People

            • Assignee:
              xudongcao Xudong Cao
              Reporter:
              xudongcao Xudong Cao
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: