Hadoop YARN
  1. Hadoop YARN
  2. YARN-1442

change yarn minicluster base directory via system property

    Details

    • Type: New Feature New Feature
    • Status: Open
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: 2.2.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      The yarn minicluster used for testing uses the "target" directory by default. We use gradle for building our projects and we would like to see it using a different directory. This patch makes it possible to use a different directory by setting the yarn.minicluster.directory system property.

        Activity

        Hide
        Ken Krugler added a comment -

        I'm curious why we wouldn't use the existing yarn.nodemanager.xxx conf settings for controlling where to put files. That's what I had originally done, and would seem like the most consistent approach.

        Related, I was assuming dfs.data.dir would control where to put HDFS blocks, but instead there's an undocumented MiniDFSCluster.HDFS_MINIDFS_BASEDIR property...why?

        Show
        Ken Krugler added a comment - I'm curious why we wouldn't use the existing yarn.nodemanager.xxx conf settings for controlling where to put files. That's what I had originally done, and would seem like the most consistent approach. Related, I was assuming dfs.data.dir would control where to put HDFS blocks, but instead there's an undocumented MiniDFSCluster.HDFS_MINIDFS_BASEDIR property...why?
        Hide
        Mark Miller added a comment -

        +1 - The Apache Solr project runs Yarn in it's tests and currently has to duplicate a bunch of Yarn mini cluster code to work around this issue.

        Show
        Mark Miller added a comment - +1 - The Apache Solr project runs Yarn in it's tests and currently has to duplicate a bunch of Yarn mini cluster code to work around this issue.
        Hide
        Alejandro Abdelnur added a comment -

        Instead introducing a new system property "yarn.minicluster.directory" we should have a common "hadoop.test.dir" used by all minicluster and tests, if different components need their own subdir, they can created it under it. Else we need to chase all properties to set.

        Show
        Alejandro Abdelnur added a comment - Instead introducing a new system property "yarn.minicluster.directory" we should have a common "hadoop.test.dir" used by all minicluster and tests, if different components need their own subdir, they can created it under it. Else we need to chase all properties to set.
        Hide
        André Kelpe added a comment -

        any chance that this patch could be applied?

        Show
        André Kelpe added a comment - any chance that this patch could be applied?

          People

          • Assignee:
            Unassigned
            Reporter:
            André Kelpe
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:

              Development