Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6806

Trim configuration values in gridmix/DistributedCacheEmulator.java

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Patch Available
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 3.0.0-alpha1
    • Fix Version/s: None
    • Component/s: contrib/gridmix
    • Labels:
      None

      Description

      The current implementation of DistributedCacheEmulator.java in gridmix does not follow the practice of trimming configuration values. This leads to errors if users set values containing space or newline.

      see the previous commits as reference (just list a few):
      HADOOP-6578. Configuration should trim whitespace around a lot of value types
      HADOOP-6534. Trim whitespace from directory lists initializing
      Patch is available against trunk
      HDFS-9708. FSNamesystem.initAuditLoggers() doesn't trim classnames
      HDFS-2799. Trim fs.checkpoint.dir values.
      YARN-3395. FairScheduler: Trim whitespaces when using username for queuename.
      YARN-2869. CapacityScheduler should trim sub queue names when parse configuration.

      Patch is available against trunk (tested):

      DistributedCacheEmulator.java
      index 72027c1..eb12723 100644
      --- a/hadoop-tools/hadoop-gridmix/src/main/java/org/apache/hadoop/mapred/gridmix/DistributedCacheEmulator.java
      +++ b/hadoop-tools/hadoop-gridmix/src/main/java/org/apache/hadoop/mapred/gridmix/DistributedCacheEmulator.java
      @@ -315,14 +315,14 @@ void updateHDFSDistCacheFilesList(JobStory jobdesc) throws IOException {
           // paths, to be used by this simulated job.
           JobConf jobConf = jobdesc.getJobConf();
      
      -    String[] files = jobConf.getStrings(MRJobConfig.CACHE_FILES);
      +    String[] files = jobConf.getTrimmedStrings(MRJobConfig.CACHE_FILES);
           if (files != null) {
      
      -      String[] fileSizes = jobConf.getStrings(MRJobConfig.CACHE_FILES_SIZES);
      +      String[] fileSizes = jobConf.getTrimmedStrings(MRJobConfig.CACHE_FILES_SIZES);
             String[] visibilities =
      -        jobConf.getStrings(MRJobConfig.CACHE_FILE_VISIBILITIES);
      +        jobConf.getTrimmedStrings(MRJobConfig.CACHE_FILE_VISIBILITIES);
             String[] timeStamps =
      -        jobConf.getStrings(MRJobConfig.CACHE_FILE_TIMESTAMPS);
      +        jobConf.getTrimmedStrings(MRJobConfig.CACHE_FILE_TIMESTAMPS);
      
             FileSystem fs = FileSystem.get(conf);
             String user = jobConf.getUser();
      @@ -495,7 +495,7 @@ void configureDistCacheFiles(Configuration conf, JobConf jobConf)
             throws IOException {
           if (shouldEmulateDistCacheLoad()) {
      
      -      String[] files = jobConf.getStrings(MRJobConfig.CACHE_FILES);
      +      String[] files = jobConf.getTrimmedStrings(MRJobConfig.CACHE_FILES);
             if (files != null) {
               // hdfs based distributed cache files to be configured for simulated job
               List<String> cacheFiles = new ArrayList<String>();
      @@ -504,10 +504,10 @@ void configureDistCacheFiles(Configuration conf, JobConf jobConf)
               List<String> localCacheFiles = new ArrayList<String>();
      
               String[] visibilities =
      -          jobConf.getStrings(MRJobConfig.CACHE_FILE_VISIBILITIES);
      +          jobConf.getTrimmedStrings(MRJobConfig.CACHE_FILE_VISIBILITIES);
               String[] timeStamps =
      -          jobConf.getStrings(MRJobConfig.CACHE_FILE_TIMESTAMPS);
      -        String[] fileSizes = jobConf.getStrings(MRJobConfig.CACHE_FILES_SIZES);
      +          jobConf.getTrimmedStrings(MRJobConfig.CACHE_FILE_TIMESTAMPS);
      +        String[] fileSizes = jobConf.getTrimmedStrings(MRJobConfig.CACHE_FILES_SIZES);
      
               String user = jobConf.getUser();
               for (int i = 0; i < files.length; i++) {
      

        Attachments

        1. MAPREDUCE-6806.0000.patch
          3 kB
          Tianyin Xu

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              tianyin Tianyin Xu
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: