Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6758

TestDFSIO should parallelize its creation of control files on setup

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: test
    • Labels:
      None

      Description

      TestDFSIO currently performs a sequential for-loop to create nrFiles control files in the controlDir which is a subdirectory of the overall test.build.data directory, which may be a non-HDFS FileSystem implementation:

      private void createControlFile(FileSystem fs,
                                      long nrBytes, // in bytes
                                      int nrFiles
                                    ) throws IOException {
        LOG.info("creating control file: "+nrBytes+" bytes, "+nrFiles+" files");
      
        Path controlDir = getControlDir(config);
        fs.delete(controlDir, true);
      
        for(int i=0; i < nrFiles; i++) {
          String name = getFileName(i);
          Path controlFile = new Path(controlDir, "in_file_" + name);
          SequenceFile.Writer writer = null;
          try {
            writer = SequenceFile.createWriter(fs, config, controlFile,
                                               Text.class, LongWritable.class,
                                               CompressionType.NONE);
            writer.append(new Text(name), new LongWritable(nrBytes));
          } catch(Exception e) {
            throw new IOException(e.getLocalizedMessage());
          } finally {
            if (writer != null)
              writer.close();
            writer = null;
          }
        }
        LOG.info("created control files for: "+nrFiles+" files");
      }
      

      When testing in an object-store based filesystem with higher round-trip latency than HDFS (like S3 or GCS), this means job setup that might only take seconds in HDFS ends up taking minutes or even tens of minutes against the object stores if the test is using thousands of control files. In the same vein as other JIRAs in https://issues.apache.org/jira/browse/HADOOP-11694, the control-file creation should be parallelized/multithreaded to efficiently launch large TestDFSIO jobs against FileSystem impls with high round-trip latency but which can still support high overall throughput/QPS.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                dennishuo Dennis Huo
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated: