Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-1442

change yarn minicluster base directory via system property

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 2.2.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      The yarn minicluster used for testing uses the "target" directory by default. We use gradle for building our projects and we would like to see it using a different directory. This patch makes it possible to use a different directory by setting the yarn.minicluster.directory system property.

        Activity

        Hide
        fs111 André Kelpe added a comment -

        any chance that this patch could be applied?

        Show
        fs111 André Kelpe added a comment - any chance that this patch could be applied?
        Hide
        tucu00 Alejandro Abdelnur added a comment -

        Instead introducing a new system property "yarn.minicluster.directory" we should have a common "hadoop.test.dir" used by all minicluster and tests, if different components need their own subdir, they can created it under it. Else we need to chase all properties to set.

        Show
        tucu00 Alejandro Abdelnur added a comment - Instead introducing a new system property "yarn.minicluster.directory" we should have a common "hadoop.test.dir" used by all minicluster and tests, if different components need their own subdir, they can created it under it. Else we need to chase all properties to set.
        Hide
        markrmiller@gmail.com Mark Miller added a comment -

        +1 - The Apache Solr project runs Yarn in it's tests and currently has to duplicate a bunch of Yarn mini cluster code to work around this issue.

        Show
        markrmiller@gmail.com Mark Miller added a comment - +1 - The Apache Solr project runs Yarn in it's tests and currently has to duplicate a bunch of Yarn mini cluster code to work around this issue.
        Hide
        kkrugler Ken Krugler added a comment -

        I'm curious why we wouldn't use the existing yarn.nodemanager.xxx conf settings for controlling where to put files. That's what I had originally done, and would seem like the most consistent approach.

        Related, I was assuming dfs.data.dir would control where to put HDFS blocks, but instead there's an undocumented MiniDFSCluster.HDFS_MINIDFS_BASEDIR property...why?

        Show
        kkrugler Ken Krugler added a comment - I'm curious why we wouldn't use the existing yarn.nodemanager.xxx conf settings for controlling where to put files. That's what I had originally done, and would seem like the most consistent approach. Related, I was assuming dfs.data.dir would control where to put HDFS blocks, but instead there's an undocumented MiniDFSCluster.HDFS_MINIDFS_BASEDIR property...why?

          People

          • Assignee:
            Unassigned
            Reporter:
            fs111 André Kelpe
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:

              Development