Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-1280 [Umbrella] Ensure YARN+MR works with https on the web interfaces
  3. YARN-1283

Invalid 'url of job' mentioned in Job output with yarn.http.policy=HTTPS_ONLY

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 2.1.1-beta
    • 2.3.0
    • None
    • Reviewed

    Description

      After setting yarn.http.policy=HTTPS_ONLY, the job output shows incorrect "The url to track the job".

      Currently, its printing http://RM:<httpsport>/proxy/application_1381162886563_0001/ instead https://RM:<httpsport>/proxy/application_1381162886563_0001/

      http://hostname:8088/proxy/application_1381162886563_0001/ is invalid

      hadoop jar hadoop-mapreduce-client-jobclient-tests.jar sleep -m 1 -r 1
      13/10/07 18:39:39 INFO client.RMProxy: Connecting to ResourceManager at hostname/100.00.00.000:8032
      13/10/07 18:39:40 INFO mapreduce.JobSubmitter: number of splits:1
      13/10/07 18:39:40 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name
      13/10/07 18:39:40 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
      13/10/07 18:39:40 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
      13/10/07 18:39:40 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
      13/10/07 18:39:40 INFO Configuration.deprecation: mapreduce.partitioner.class is deprecated. Instead, use mapreduce.job.partitioner.class
      13/10/07 18:39:40 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
      13/10/07 18:39:40 INFO Configuration.deprecation: mapred.mapoutput.value.class is deprecated. Instead, use mapreduce.map.output.value.class
      13/10/07 18:39:40 INFO Configuration.deprecation: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
      13/10/07 18:39:40 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name
      13/10/07 18:39:40 INFO Configuration.deprecation: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class
      13/10/07 18:39:40 INFO Configuration.deprecation: mapreduce.inputformat.class is deprecated. Instead, use mapreduce.job.inputformat.class
      13/10/07 18:39:40 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
      13/10/07 18:39:40 INFO Configuration.deprecation: mapreduce.outputformat.class is deprecated. Instead, use mapreduce.job.outputformat.class
      13/10/07 18:39:40 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
      13/10/07 18:39:40 INFO Configuration.deprecation: mapred.mapoutput.key.class is deprecated. Instead, use mapreduce.map.output.key.class
      13/10/07 18:39:40 INFO Configuration.deprecation: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
      13/10/07 18:39:40 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1381162886563_0001
      13/10/07 18:39:40 INFO impl.YarnClientImpl: Submitted application application_1381162886563_0001 to ResourceManager at hostname/100.00.00.000:8032
      13/10/07 18:39:40 INFO mapreduce.Job: The url to track the job: http://hostname:8088/proxy/application_1381162886563_0001/
      13/10/07 18:39:40 INFO mapreduce.Job: Running job: job_1381162886563_0001
      13/10/07 18:39:46 INFO mapreduce.Job: Job job_1381162886563_0001 running in uber mode : false
      13/10/07 18:39:46 INFO mapreduce.Job: map 0% reduce 0%
      13/10/07 18:39:53 INFO mapreduce.Job: map 100% reduce 0%
      13/10/07 18:39:58 INFO mapreduce.Job: map 100% reduce 100%
      13/10/07 18:39:58 INFO mapreduce.Job: Job job_1381162886563_0001 completed successfully
      13/10/07 18:39:58 INFO mapreduce.Job: Counters: 43
      File System Counters
      FILE: Number of bytes read=26
      FILE: Number of bytes written=177279
      FILE: Number of read operations=0
      FILE: Number of large read operations=0
      FILE: Number of write operations=0
      HDFS: Number of bytes read=48
      HDFS: Number of bytes written=0
      HDFS: Number of read operations=1
      HDFS: Number of large read operations=0
      HDFS: Number of write operations=0
      Job Counters
      Launched map tasks=1
      Launched reduce tasks=1
      Other local map tasks=1
      Total time spent by all maps in occupied slots (ms)=7136
      Total time spent by all reduces in occupied slots (ms)=6062
      Map-Reduce Framework
      Map input records=1
      Map output records=1
      Map output bytes=4
      Map output materialized bytes=22
      Input split bytes=48
      Combine input records=0
      Combine output records=0
      Reduce input groups=1
      Reduce shuffle bytes=22
      Reduce input records=1
      Reduce output records=0
      Spilled Records=2
      Shuffled Maps =1
      Failed Shuffles=0
      Merged Map outputs=1
      GC time elapsed (ms)=60
      CPU time spent (ms)=1700
      Physical memory (bytes) snapshot=567582720
      Virtual memory (bytes) snapshot=4292997120
      Total committed heap usage (bytes)=846594048
      Shuffle Errors
      BAD_ID=0
      CONNECTION=0
      IO_ERROR=0
      WRONG_LENGTH=0
      WRONG_MAP=0
      WRONG_REDUCE=0
      File Input Format Counters
      Bytes Read=0
      File Output Format Counters
      Bytes Written=0

      Attachments

        1. YARN-1283.3.patch
          10 kB
          Omkar Vinit Joshi
        2. YARN-1283.20131008.2.patch
          9 kB
          Omkar Vinit Joshi
        3. YARN-1283.20131008.1.patch
          7 kB
          Omkar Vinit Joshi
        4. YARN-1283.20131007.1.patch
          7 kB
          Omkar Vinit Joshi

        Activity

          People

            ojoshi Omkar Vinit Joshi
            yeshavora Yesha Vora
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: