Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-16074

Remove an expensive debug string concatenation

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha1
    • Fix Version/s: 3.4.0, 3.2.3, 3.3.2
    • Component/s: None

      Description

      Running a YCSB load query, found that we do an expensive string concatenation on the write path in DFSOutputStream.writeChunkPrepare().

      Nearly 25% of HDFS client write CPU time is spent here. It is not necessary because it's supposed to be a debug message. So let's remove it.

       if (currentPacket == null) {
            currentPacket = createPacket(packetSize, chunksPerPacket, getStreamer()
                .getBytesCurBlock(), getStreamer().getAndIncCurrentSeqno(), false);
            DFSClient.LOG.debug("WriteChunk allocating new packet seqno={},"
                    + " src={}, packetSize={}, chunksPerPacket={}, bytesCurBlock={}",
                currentPacket.getSeqno(), src, packetSize, chunksPerPacket,
                getStreamer().getBytesCurBlock() + ", " + this); <---- here
          }
      

        Attachments

        1. Screen Shot 2021-06-17 at 10.32.21 AM.png
          352 kB
          Wei-Chiu Chuang
        2. Screen Shot 2021-06-16 at 2.32.29 PM.png
          477 kB
          Wei-Chiu Chuang

          Issue Links

            Activity

              People

              • Assignee:
                weichiu Wei-Chiu Chuang
                Reporter:
                weichiu Wei-Chiu Chuang
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 40m
                  40m