Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3654

MiniMRYarnCluster should set MASTER_ADDRESS to "local"

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Invalid
    • Affects Version/s: 0.23.1
    • Fix Version/s: None
    • Component/s: mrv2, test
    • Labels:
      None

      Description

      I needed to make the attached change in order for MiniMRCluster based HBase tests to get past job client initialization.

        Activity

        Hide
        Mahadev konar added a comment -

        @Andrew,
        I was able to run the test all fine after applying the patch on HBASE-5212. The only issue being HBASE-5191. Ill have some bandwidth this week to work on 0.23 and HBase issues. Ill update the hbase jiras.

        Show
        Mahadev konar added a comment - @Andrew, I was able to run the test all fine after applying the patch on HBASE-5212 . The only issue being HBASE-5191 . Ill have some bandwidth this week to work on 0.23 and HBase issues. Ill update the hbase jiras.
        Hide
        Hitesh Shah added a comment -

        In the TestTableMapReduce test, I ended up doing something along the lines of:

        table.getConfiguration().set("mapreduce.framework.name", "yarn");
        job = new Job(table.getConfiguration(), "process column contents");

        to get around the clientprovider issue which seemed to have the test run successfully but I could not completely verify as there are no logs generated even though the hbase/hadoop log levels are set to debug. However, the above is in any case not the correct solution.

        When submitting a job to a cluster, the assumption would be to use the relevant client-side aspects of the cluster's configuration. However in the hbase test, what is happening is that the job and htable objects are being initialized via HbaseTestingUtility.getConfiguration(). The test framework encapsulates the dfs/mr clusters but currently it does set internal configs like the jobtracker address for the test to work. It would need to change to probably use the new cluster factory created as part of MAPREDUCE-3169 to switch between mr1/mr2 and also likely set some config values if the job configs are not initialized using the mrCluster's configuration ( such as jobtracker address for MR1 and for MR2, yarn's resource/node mgr addresses in addition to the framework name being set to yarn).

        Show
        Hitesh Shah added a comment - In the TestTableMapReduce test, I ended up doing something along the lines of: table.getConfiguration().set("mapreduce.framework.name", "yarn"); job = new Job(table.getConfiguration(), "process column contents"); to get around the clientprovider issue which seemed to have the test run successfully but I could not completely verify as there are no logs generated even though the hbase/hadoop log levels are set to debug. However, the above is in any case not the correct solution. When submitting a job to a cluster, the assumption would be to use the relevant client-side aspects of the cluster's configuration. However in the hbase test, what is happening is that the job and htable objects are being initialized via HbaseTestingUtility.getConfiguration(). The test framework encapsulates the dfs/mr clusters but currently it does set internal configs like the jobtracker address for the test to work. It would need to change to probably use the new cluster factory created as part of MAPREDUCE-3169 to switch between mr1/mr2 and also likely set some config values if the job configs are not initialized using the mrCluster's configuration ( such as jobtracker address for MR1 and for MR2, yarn's resource/node mgr addresses in addition to the framework name being set to yarn).
        Hide
        Andrew Purtell added a comment -

        I can accept the resolution for this issue in particular. However, with MRv1 we could simply instantiate a MiniMRCluster and it would start without additional configuration. It is the Hadoop code that delegates MiniMRCluster to the MiniYarnMRCluster. Our client code shouldn't have to set any arguably internal configuration variables to avoid cryptic MiniMRCluster failures.

        Mahadev helped us update our POM. I applied these changes locally and still all MiniMRCluster based HBase tests fail to launch. Have you been able to successfully run the HBase unit tests like TestTableMapReduce successfully with 0.23.1-SNAPSHOT?

        Show
        Andrew Purtell added a comment - I can accept the resolution for this issue in particular. However, with MRv1 we could simply instantiate a MiniMRCluster and it would start without additional configuration. It is the Hadoop code that delegates MiniMRCluster to the MiniYarnMRCluster. Our client code shouldn't have to set any arguably internal configuration variables to avoid cryptic MiniMRCluster failures. Mahadev helped us update our POM. I applied these changes locally and still all MiniMRCluster based HBase tests fail to launch. Have you been able to successfully run the HBase unit tests like TestTableMapReduce successfully with 0.23.1-SNAPSHOT?
        Hitesh Shah made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Invalid [ 6 ]
        Hide
        Hitesh Shah added a comment -

        Looking more into the hbase test, a couple of issues were found ( pom fix to depend on the required jar so as to find the YarnClientProtocol Provider and a change required to the job client submission to use the conf from the initialized MiniMRYarnCluster object ).

        Marking this as invalid as it seems to me that the fix needs to be done in the hbase test code.

        Show
        Hitesh Shah added a comment - Looking more into the hbase test, a couple of issues were found ( pom fix to depend on the required jar so as to find the YarnClientProtocol Provider and a change required to the job client submission to use the conf from the initialized MiniMRYarnCluster object ). Marking this as invalid as it seems to me that the fix needs to be done in the hbase test code.
        Hide
        Hitesh Shah added a comment -

        Forgot to add, there still might be an issue with the service loading as it does not seem to find the yarn client provider.

        Show
        Hitesh Shah added a comment - Forgot to add, there still might be an issue with the service loading as it does not seem to find the yarn client provider.
        Hide
        Hitesh Shah added a comment -

        The service loader for the ClientProtocolProvider should cycle through a list containing both the local and yarn client providers. The debug logs seem to indicate that it only sees the local client provider as an option and fails as the framework is set to yarn.

        The fix probably might be in the unit test itself - could you ensure the job/jobconf is being initialized using MiniMrYarnCluster#getConfig ?

        Show
        Hitesh Shah added a comment - The service loader for the ClientProtocolProvider should cycle through a list containing both the local and yarn client providers. The debug logs seem to indicate that it only sees the local client provider as an option and fails as the framework is set to yarn. The fix probably might be in the unit test itself - could you ensure the job/jobconf is being initialized using MiniMrYarnCluster#getConfig ?
        Hide
        Hitesh Shah added a comment -

        Nevermind - saw the mail on the user list.

        Adding the info here for reference:

        2012-01-10 18:01:02,489 INFO [main] mapreduce.TestTableMapReduce(153): Started mrtest
        2012-01-10 18:01:02,493 DEBUG [main] mapreduce.Cluster(93): Trying ClientProtocolProvider : org.apache.hadoop.mapred.LocalClientProtocolProvider
        2012-01-10 18:01:02,494 INFO [main] mapreduce.Cluster(116): Failed to use org.apache.hadoop.mapred.LocalClientProtocolProvider due to error: Invalid "mapreduce.jobtracker.address" configuration value for LocalJobRunner : "test"
        2012-01-10 18:01:02,494 ERROR [main] security.UserGroupInformation(1160): PriviledgedActionException as:apurtell (auth:SIMPLE) cause:java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
        2012-01-10 18:01:02,497 INFO [main] hbase.HBaseTestingUtility(1173): Stopping mini mapreduce cluster...
        2

        Show
        Hitesh Shah added a comment - Nevermind - saw the mail on the user list. Adding the info here for reference: 2012-01-10 18:01:02,489 INFO [main] mapreduce.TestTableMapReduce(153): Started mrtest 2012-01-10 18:01:02,493 DEBUG [main] mapreduce.Cluster(93): Trying ClientProtocolProvider : org.apache.hadoop.mapred.LocalClientProtocolProvider 2012-01-10 18:01:02,494 INFO [main] mapreduce.Cluster(116): Failed to use org.apache.hadoop.mapred.LocalClientProtocolProvider due to error: Invalid "mapreduce.jobtracker.address" configuration value for LocalJobRunner : "test" 2012-01-10 18:01:02,494 ERROR [main] security.UserGroupInformation(1160): PriviledgedActionException as:apurtell (auth:SIMPLE) cause:java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses. 2012-01-10 18:01:02,497 INFO [main] hbase.HBaseTestingUtility(1173): Stopping mini mapreduce cluster... 2
        Hide
        Hitesh Shah added a comment -

        The change seems a bit odd as master address which is the jobtracker address should not be referenced in a yarn cluster.
        Could you attach logs or add more information as to how this solves the hbase job client init failure?

        Show
        Hitesh Shah added a comment - The change seems a bit odd as master address which is the jobtracker address should not be referenced in a yarn cluster. Could you attach logs or add more information as to how this solves the hbase job client init failure?
        Mahadev konar made changes -
        Assignee Andrew Purtell [ apurtell ]
        Priority Major [ 3 ] Critical [ 2 ]
        Component/s mrv2 [ 12314301 ]
        Andrew Purtell made changes -
        Field Original Value New Value
        Attachment MAPREDUCE-3654.patch [ 12510136 ]
        Andrew Purtell created issue -

          People

          • Assignee:
            Andrew Purtell
            Reporter:
            Andrew Purtell
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development