Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2153

Bring in more job configuration properties in to the trace file

        Details

        • Type: Improvement Improvement
        • Status: Closed
        • Priority: Major Major
        • Resolution: Fixed
        • Affects Version/s: 0.23.0
        • Fix Version/s: 0.23.0
        • Component/s: tools/rumen
        • Labels:
          None
        • Hadoop Flags:
          Reviewed
        • Release Note:
          Adds job configuration parameters to the job trace. The configuration parameters are stored under the 'jobProperties' field as key-value pairs.
        • Tags:
          rumen, job-conf, job-properties

          Description

          To emulate distributed cache usage in gridmix jobs, there are 9 configuration properties needed to be available in trace file:
          (1) mapreduce.job.cache.files
          (2) mapreduce.job.cache.files.visibilities
          (3) mapreduce.job.cache.files.filesizes
          (4) mapreduce.job.cache.files.timestamps

          (5) mapreduce.job.cache.archives
          (6) mapreduce.job.cache.archives.visibilities
          (7) mapreduce.job.cache.archives.filesizes
          (8) mapreduce.job.cache.archives.timestamps

          (9) mapreduce.job.cache.symlink.create

          To emulate data compression in gridmix jobs, trace file should contain the following configuration properties:
          (1) mapreduce.map.output.compress
          (2) mapreduce.map.output.compress.codec
          (3) mapreduce.output.fileoutputformat.compress
          (4) mapreduce.output.fileoutputformat.compress.codec
          (5) mapreduce.output.fileoutputformat.compress.type

          Ideally, gridmix should set many job specific configuration properties like io.sort.mb, io.sort.factor, etc when running simulated jobs to get the same effect of original/real job in terms of spilled records, number of merges, etc.

          TraceBuilder should bring in all these properties into the generated trace file.

          1. Text File
            MapReduce-2153-trunk.patch
            130 kB
            Rajesh Balamohan
          2. Text File
            mr-2153-test-patch-results.txt
            213 kB
            Rajesh Balamohan
          3. Text File
            MapReduce-2153-trunk.patch
            130 kB
            Rajesh Balamohan
          4. Text File
            MR-2153-patch.txt
            128 kB
            Rajesh Balamohan

            Issue Links

            blocks

            New Feature - A new feature of the product, which has yet to be developed. MAPREDUCE-2407 Make Gridmix emulate usage of Distributed Cache files

            • Major - Major loss of function.
            • Closed

            New Feature - A new feature of the product, which has yet to be developed. MAPREDUCE-2408 Make Gridmix emulate usage of data compression

            • Major - Major loss of function.
            • Closed

            Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-2725 Make Gridmix configure job specific config properties for the simulated jobs

            • Major - Major loss of function.
            • Open

              Activity

              Ascending order - Click to sort in descending order
              Ravi Gummadi created issue -
              Ravi Gummadi made changes -
              Field Original Value New Value
              Link This issue blocks MAPREDUCE-2408 [ MAPREDUCE-2408 ]
              Ravi Gummadi made changes -
              Link This issue blocks MAPREDUCE-2407 [ MAPREDUCE-2407 ]
              Rajesh Balamohan made changes -
              Attachment MapReduce-2153-trunk.patch [ 12475568 ]
              Rajesh Balamohan made changes -
              Attachment mr-2153-test-patch-results.txt [ 12475671 ]
              Rajesh Balamohan made changes -
              Status Open [ 1 ] Patch Available [ 10002 ]
              Affects Version/s 0.23.0 [ 12315570 ]
              Rajesh Balamohan made changes -
              Attachment MapReduce-2153-trunk.patch [ 12476085 ]
              Amar Kamat made changes -
              Status Patch Available [ 10002 ] Open [ 1 ]
              Rajesh Balamohan made changes -
              Status Open [ 1 ] Patch Available [ 10002 ]
              Rajesh Balamohan made changes -
              Status Patch Available [ 10002 ] Open [ 1 ]
              Assignee Rajesh Balamohan [ rajesh.balamohan ]
              Rajesh Balamohan made changes -
              Attachment MR-2153-patch.txt [ 12476206 ]
              Rajesh Balamohan made changes -
              Status Open [ 1 ] Patch Available [ 10002 ]
              Amar Kamat made changes -
              Status Patch Available [ 10002 ] Resolved [ 5 ]
              Hadoop Flags [Reviewed]
              Release Note Adds job configuration parameters to the job trace. The configuration parameters are stored under the 'jobProperties' field as key-value pairs.
              Fix Version/s 0.23.0 [ 12315570 ]
              Resolution Fixed [ 1 ]
              Tags rumen, job-conf, job-properties
              Ravi Gummadi made changes -
              Link This issue blocks MAPREDUCE-2725 [ MAPREDUCE-2725 ]
              Arun C Murthy made changes -
              Status Resolved [ 5 ] Closed [ 6 ]

                People

                • Assignee:
                  Rajesh Balamohan
                  Reporter:
                  Ravi Gummadi
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  2 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Development