Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-16074

Remove an expensive debug string concatenation

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.0.0-alpha1
    • 3.4.0, 3.2.3, 3.3.2
    • None

    Description

      Running a YCSB load query, found that we do an expensive string concatenation on the write path in DFSOutputStream.writeChunkPrepare().

      Nearly 25% of HDFS client write CPU time is spent here. It is not necessary because it's supposed to be a debug message. So let's remove it.

       if (currentPacket == null) {
            currentPacket = createPacket(packetSize, chunksPerPacket, getStreamer()
                .getBytesCurBlock(), getStreamer().getAndIncCurrentSeqno(), false);
            DFSClient.LOG.debug("WriteChunk allocating new packet seqno={},"
                    + " src={}, packetSize={}, chunksPerPacket={}, bytesCurBlock={}",
                currentPacket.getSeqno(), src, packetSize, chunksPerPacket,
                getStreamer().getBytesCurBlock() + ", " + this); <---- here
          }
      

      Attachments

        1. Screen Shot 2021-06-17 at 10.32.21 AM.png
          352 kB
          Wei-Chiu Chuang
        2. Screen Shot 2021-06-16 at 2.32.29 PM.png
          477 kB
          Wei-Chiu Chuang

        Issue Links

          Activity

            People

              weichiu Wei-Chiu Chuang
              weichiu Wei-Chiu Chuang
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 40m
                  40m