Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-10448

SLS should set default user to handle SYNTH format

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.2.1, 3.4.0
    • 3.4.0
    • None
    • Reviewed

    Description

      When using the synthetic generator json file example from the doc ( https://hadoop.apache.org/docs/current/hadoop-sls/SchedulerLoadSimulator.html#SYNTH_JSON_input_file_format ), it throws the following exception:

      java.lang.IllegalArgumentException: Null user
      at org.apache.hadoop.security.UserGroupInformation.createRemoteUser(UserGroupInformation.java:1269)
      at org.apache.hadoop.security.UserGroupInformation.createRemoteUser(UserGroupInformation.java:1256)
      at org.apache.hadoop.yarn.sls.appmaster.AMSimulator.submitReservationWhenSpecified(AMSimulator.java:191)
      at org.apache.hadoop.yarn.sls.appmaster.AMSimulator.firstStep(AMSimulator.java:161)
      at org.apache.hadoop.yarn.sls.scheduler.TaskRunner$Task.run(TaskRunner.java:88)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      

      So the solution is either:
      1) to make user_name a mandatory field, or
      2) to set default user in SLS code if the json file does not define it.

      IMO, solution 2 might be better, because in most cases (if not all) user_name has no impact on scheduler performance, thus it is reasonable to make it an optional field, which is also consistent with the job.user field in SLS JSON file.

      Attachments

        1. image-2020-10-11-22-01-37-227.png
          125 kB
          Qi Zhu
        2. image-2020-10-11-22-02-17-166.png
          14 kB
          Qi Zhu
        3. YARN-10448.001.patch
          0.9 kB
          Qi Zhu
        4. YARN-10448.002.patch
          0.9 kB
          Qi Zhu
        5. YARN-10448.003.patch
          1 kB
          Qi Zhu
        6. YARN-10448.004.patch
          1 kB
          Qi Zhu

        Activity

          People

            zhuqi Qi Zhu
            zhuqi Qi Zhu
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: