Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6729

Accurately compute the test execute time in DFSIO

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      When doing DFSIO test as a distributed i/o benchmark tool. Then especially writes plenty of files to disk or read from, both can cause performance issue and imprecise value in a way. The question is that existing practices needs to delete files when before running a job and that will cause extra time consumption and furthermore cause performance issue, statistical time error and imprecise throughput as the files are lots of. So we need to replace or improve this hack to prevent this from happening in the future.

      public static void testWrite() throws Exception {
          FileSystem fs = cluster.getFileSystem();
          long tStart = System.currentTimeMillis();
          bench.writeTest(fs); // this line of code will cause extra time consumption as fs.delete(*,*) by the writeTest method
          long execTime = System.currentTimeMillis() - tStart;
          bench.analyzeResult(fs, TestType.TEST_TYPE_WRITE, execTime);
        }
      
      private void writeTest(FileSystem fs) throws IOException {
        Path writeDir = getWriteDir(config);
        fs.delete(getDataDir(config), true);
        fs.delete(writeDir, true);    
        runIOTest(WriteMapper.class, writeDir);
        }
      

       

      https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/TestDFSIO.java

        Attachments

        1. MAPREDUCE-6729.002.patch
          6 kB
          Kai Zheng
        2. MAPREDUCE-6729.001.patch
          6 kB
          zhangminglei

          Issue Links

            Activity

              People

              • Assignee:
                mingleizhang zhangminglei
                Reporter:
                mingleizhang zhangminglei
              • Votes:
                0 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: