Hadoop YARN
  1. Hadoop YARN
  2. YARN-82

YARN local-dirs defaults to /tmp/nm-local-dir

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 2.0.3-alpha
    • Component/s: nodemanager
    • Labels:
      None

      Description

      yarn.nodemanager.local-dirs defaults to /tmp/nm-local-dir. It should be

      {hadoop.tmp.dir}

      /nm-local-dir or similar. Among other problems, this can prevent multiple test clusters from starting on the same machine.

      Thanks to Hemanth Yamijala for reporting this issue.

      1. YARN-82.patch
        0.9 kB
        Hemanth Yamijala
      2. YARN-82.patch
        0.9 kB
        Hemanth Yamijala

        Activity

        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Patch Available Patch Available Open Open
        8d 7h 56m 1 Hemanth Yamijala 20/Sep/12 11:18
        Open Open Patch Available Patch Available
        7d 7h 48m 2 Hemanth Yamijala 20/Sep/12 11:18
        Patch Available Patch Available Resolved Resolved
        20h 22m 1 Siddharth Seth 21/Sep/12 07:41
        Resolved Resolved Closed Closed
        147d 6h 30m 1 Arun C Murthy 15/Feb/13 13:12
        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1203 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1203/)
        YARN-82. Correct the fix version in CHANGES.txt (Revision 1388340)
        YARN-82. Change the default local and log dirs to be based on hadoop.tmp.dir and yarn.log.dir. (Contributed by Hemanth Yamijala) (Revision 1388339)

        Result = SUCCESS
        sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1388340
        Files :

        • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt

        sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1388339
        Files :

        • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1203 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1203/ ) YARN-82 . Correct the fix version in CHANGES.txt (Revision 1388340) YARN-82 . Change the default local and log dirs to be based on hadoop.tmp.dir and yarn.log.dir. (Contributed by Hemanth Yamijala) (Revision 1388339) Result = SUCCESS sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1388340 Files : /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1388339 Files : /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1172 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1172/)
        YARN-82. Correct the fix version in CHANGES.txt (Revision 1388340)
        YARN-82. Change the default local and log dirs to be based on hadoop.tmp.dir and yarn.log.dir. (Contributed by Hemanth Yamijala) (Revision 1388339)

        Result = SUCCESS
        sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1388340
        Files :

        • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt

        sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1388339
        Files :

        • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1172 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1172/ ) YARN-82 . Correct the fix version in CHANGES.txt (Revision 1388340) YARN-82 . Change the default local and log dirs to be based on hadoop.tmp.dir and yarn.log.dir. (Contributed by Hemanth Yamijala) (Revision 1388339) Result = SUCCESS sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1388340 Files : /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1388339 Files : /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #2774 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2774/)
        YARN-82. Correct the fix version in CHANGES.txt (Revision 1388340)
        YARN-82. Change the default local and log dirs to be based on hadoop.tmp.dir and yarn.log.dir. (Contributed by Hemanth Yamijala) (Revision 1388339)

        Result = FAILURE
        sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1388340
        Files :

        • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt

        sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1388339
        Files :

        • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2774 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2774/ ) YARN-82 . Correct the fix version in CHANGES.txt (Revision 1388340) YARN-82 . Change the default local and log dirs to be based on hadoop.tmp.dir and yarn.log.dir. (Contributed by Hemanth Yamijala) (Revision 1388339) Result = FAILURE sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1388340 Files : /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1388339 Files : /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2815 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2815/)
        YARN-82. Correct the fix version in CHANGES.txt (Revision 1388340)

        Result = SUCCESS
        sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1388340
        Files :

        • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2815 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2815/ ) YARN-82 . Correct the fix version in CHANGES.txt (Revision 1388340) Result = SUCCESS sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1388340 Files : /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
        Siddharth Seth made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Fix Version/s 2.0.3-alpha [ 12323272 ]
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #2752 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2752/)
        YARN-82. Correct the fix version in CHANGES.txt (Revision 1388340)
        YARN-82. Change the default local and log dirs to be based on hadoop.tmp.dir and yarn.log.dir. (Contributed by Hemanth Yamijala) (Revision 1388339)

        Result = SUCCESS
        sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1388340
        Files :

        • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt

        sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1388339
        Files :

        • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2752 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2752/ ) YARN-82 . Correct the fix version in CHANGES.txt (Revision 1388340) YARN-82 . Change the default local and log dirs to be based on hadoop.tmp.dir and yarn.log.dir. (Contributed by Hemanth Yamijala) (Revision 1388339) Result = SUCCESS sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1388340 Files : /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1388339 Files : /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2814 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2814/)
        YARN-82. Change the default local and log dirs to be based on hadoop.tmp.dir and yarn.log.dir. (Contributed by Hemanth Yamijala) (Revision 1388339)

        Result = SUCCESS
        sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1388339
        Files :

        • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2814 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2814/ ) YARN-82 . Change the default local and log dirs to be based on hadoop.tmp.dir and yarn.log.dir. (Contributed by Hemanth Yamijala) (Revision 1388339) Result = SUCCESS sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1388339 Files : /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
        Hide
        Siddharth Seth added a comment -

        +1. Committing this. Thanks for the patch Hemanth, and Andy for the review.

        Show
        Siddharth Seth added a comment - +1. Committing this. Thanks for the patch Hemanth, and Andy for the review.
        Hemanth Yamijala made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hemanth Yamijala made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hemanth Yamijala made changes -
        Attachment YARN-82.patch [ 12545859 ]
        Hide
        Hemanth Yamijala added a comment -

        Modified the patch per Sid's comments. I have changed the default to userlogs - to be similar to what was there in 1.x.

        Show
        Hemanth Yamijala added a comment - Modified the patch per Sid's comments. I have changed the default to userlogs - to be similar to what was there in 1.x.
        Hide
        Siddharth Seth added a comment -

        log-dirs could be $

        {yarn.log.dir}/app-logs or even ${yarn.log.dir}

        /userlogs like in 1.0 ?, since the property is used to configure the log location for containers run on behalf of a user.

        Show
        Siddharth Seth added a comment - log-dirs could be $ {yarn.log.dir}/app-logs or even ${yarn.log.dir} /userlogs like in 1.0 ?, since the property is used to configure the log location for containers run on behalf of a user.
        Andy Isaacson made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Andy Isaacson made changes -
        Assignee Andy Isaacson [ adi2 ] Hemanth Yamijala [ yhemanth ]
        Hide
        Andy Isaacson added a comment -

        Thanks for the patch, Hemanth.

        Show
        Andy Isaacson added a comment - Thanks for the patch, Hemanth.
        Hide
        Andy Isaacson added a comment -

        Andy, I have a patch for this. Is it OK if I assign this JIRA to myself and upload the same ?

        Definitely! Thanks for the patch.

        +1, I agree with the change to yarn.log.dir. Someone who understands the YARN requirements should review as well.

        Show
        Andy Isaacson added a comment - Andy, I have a patch for this. Is it OK if I assign this JIRA to myself and upload the same ? Definitely! Thanks for the patch. +1, I agree with the change to yarn.log.dir. Someone who understands the YARN requirements should review as well.
        Hemanth Yamijala made changes -
        Attachment YARN-82.patch [ 12544590 ]
        Hide
        Hemanth Yamijala added a comment -

        Attaching a patch which seems to work for me. Andy / others, would be good if you can review and let me know if its ok.

        I've changed the following two properties in the patch:
        yarn.nodemanager.local-dirs now points to $

        {hadoop.tmp.dir}

        /nm-local-dir
        yarn.nodemanager.log-dirs now points to $

        {yarn.log.dir}

        /nm-logs.

        The latter is an incompatible change. The default value was /tmp/logs. However, since this was logs, I thought it was better to default to yarn.log.dir. Let me know if this is fine.

        The other directory based configuration is yarn.nodemanager.remote-app-log-dir. However, since this is a location on HDFS typically, it seems better not to default it to variables that point to local logs.

        Show
        Hemanth Yamijala added a comment - Attaching a patch which seems to work for me. Andy / others, would be good if you can review and let me know if its ok. I've changed the following two properties in the patch: yarn.nodemanager.local-dirs now points to $ {hadoop.tmp.dir} /nm-local-dir yarn.nodemanager.log-dirs now points to $ {yarn.log.dir} /nm-logs. The latter is an incompatible change. The default value was /tmp/logs. However, since this was logs, I thought it was better to default to yarn.log.dir. Let me know if this is fine. The other directory based configuration is yarn.nodemanager.remote-app-log-dir. However, since this is a location on HDFS typically, it seems better not to default it to variables that point to local logs.
        Hide
        Hemanth Yamijala added a comment -

        Andy, I have a patch for this. Is it OK if I assign this JIRA to myself and upload the same ?

        Show
        Hemanth Yamijala added a comment - Andy, I have a patch for this. Is it OK if I assign this JIRA to myself and upload the same ?
        Arun C Murthy made changes -
        Affects Version/s 2.0.0-alpha [ 12323271 ]
        Affects Version/s 2.1.0-alpha [ 12322547 ]
        Hide
        Arun C Murthy added a comment -

        Hemanth - yes, agreed.

        Btw, great to see you around!

        Show
        Arun C Murthy added a comment - Hemanth - yes, agreed. Btw, great to see you around!
        Hide
        Hemanth Yamijala added a comment -

        Arun, assume it is ok to add deps on configs in Common (i.e core-site), but not with Mapreduce/HDFS. Is that correct ?

        Show
        Hemanth Yamijala added a comment - Arun, assume it is ok to add deps on configs in Common (i.e core-site), but not with Mapreduce/HDFS. Is that correct ?
        Hide
        Andy Isaacson added a comment -

        Let's be careful, we don't need to introduce explicit deps on configs from HDFS/MR. Let's keep a separate namespace.

        Thanks for the idea, Arun. Can you suggest a good name or existing variable I should depend on?

        Show
        Andy Isaacson added a comment - Let's be careful, we don't need to introduce explicit deps on configs from HDFS/MR. Let's keep a separate namespace. Thanks for the idea, Arun. Can you suggest a good name or existing variable I should depend on?
        Andy Isaacson made changes -
        Field Original Value New Value
        Assignee Andy Isaacson [ adi2 ]
        Hide
        Arun C Murthy added a comment -

        Let's be careful, we don't need to introduce explicit deps on configs from HDFS/MR. Let's keep a separate namespace.

        Show
        Arun C Murthy added a comment - Let's be careful, we don't need to introduce explicit deps on configs from HDFS/MR. Let's keep a separate namespace.
        Hide
        Vinod Kumar Vavilapalli added a comment -

        +1, can one of you take a stab at it. While you are at it, can you please also address any other hard-coded temp paths like this? Thanks.

        Show
        Vinod Kumar Vavilapalli added a comment - +1, can one of you take a stab at it. While you are at it, can you please also address any other hard-coded temp paths like this? Thanks.
        Andy Isaacson created issue -

          People

          • Assignee:
            Hemanth Yamijala
            Reporter:
            Andy Isaacson
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development