Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-8472 YARN Container Phase 2
  3. YARN-8569

Create an interface to provide cluster information to application

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 3.3.0
    • None

    Description

      Some program requires container hostnames to be known for application to run. For example, distributed tensorflow requires launch_command that looks like:

      # On ps0.example.com:
      $ python trainer.py \
           --ps_hosts=ps0.example.com:2222,ps1.example.com:2222 \
           --worker_hosts=worker0.example.com:2222,worker1.example.com:2222 \
           --job_name=ps --task_index=0
      # On ps1.example.com:
      $ python trainer.py \
           --ps_hosts=ps0.example.com:2222,ps1.example.com:2222 \
           --worker_hosts=worker0.example.com:2222,worker1.example.com:2222 \
           --job_name=ps --task_index=1
      # On worker0.example.com:
      $ python trainer.py \
           --ps_hosts=ps0.example.com:2222,ps1.example.com:2222 \
           --worker_hosts=worker0.example.com:2222,worker1.example.com:2222 \
           --job_name=worker --task_index=0
      # On worker1.example.com:
      $ python trainer.py \
           --ps_hosts=ps0.example.com:2222,ps1.example.com:2222 \
           --worker_hosts=worker0.example.com:2222,worker1.example.com:2222 \
           --job_name=worker --task_index=1
      

      This is a bit cumbersome to orchestrate via Distributed Shell, or YARN services launch_command. In addition, the dynamic parameters do not work with YARN flex command. This is the classic pain point for application developer attempt to automate system environment settings as parameter to end user application.

      It would be great if YARN Docker integration can provide a simple option to expose hostnames of the yarn service via a mounted file. The file content gets updated when flex command is performed. This allows application developer to consume system environment settings via a standard interface. It is like /proc/devices for Linux, but for Hadoop. This may involve updating a file in distributed cache, and allow mounting of the file via container-executor.

      Attachments

        1. YARN-8569.001.patch
          5 kB
          Eric Yang
        2. YARN-8569.002.patch
          45 kB
          Eric Yang
        3. YARN-8569.003.patch
          53 kB
          Eric Yang
        4. YARN-8569.004.patch
          52 kB
          Eric Yang
        5. YARN-8569.005.patch
          54 kB
          Eric Yang
        6. YARN-8569.006.patch
          65 kB
          Eric Yang
        7. YARN-8569.007.patch
          65 kB
          Eric Yang
        8. YARN-8569.008.patch
          64 kB
          Eric Yang
        9. YARN-8569.009.patch
          63 kB
          Eric Yang
        10. YARN-8569.010.patch
          62 kB
          Eric Yang
        11. YARN-8569.011.patch
          58 kB
          Eric Yang
        12. YARN-8569.012.patch
          58 kB
          Eric Yang
        13. YARN-8569.013.patch
          59 kB
          Eric Yang
        14. YARN-8569.014.patch
          59 kB
          Eric Yang
        15. YARN-8569.015.patch
          59 kB
          Eric Yang
        16. YARN-8569.016.patch
          59 kB
          Eric Yang
        17. YARN-8569 YARN sysfs interface to provide cluster information to application.pdf
          77 kB
          Eric Yang

        Issue Links

          Activity

            People

              eyang Eric Yang
              eyang Eric Yang
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: