Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-16375

Mapreduce mini cluster using HBaseTestingUtility not setting correct resourcemanager and jobhistory webapp address of MapReduceTestingShim

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.2.0
    • Fix Version/s: 1.3.0, 0.98.22, 1.1.7, 1.2.4, 2.0.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Starting mapreduce mini cluster using HBaseTestingUtility is not setting "yarn.resourcemanager.webapp.address" and "mapreduce.jobhistory.webapp.address" which are required for getting the submitted yarn apps using mapreduce webapp. These properties are not being copied from jobConf of MapReduceTestingShim resulting in default values.

      HBaseTestingUtility.java
      // Allow the user to override FS URI for this map-reduce cluster to use.
      mrCluster = new MiniMRCluster(servers,
      FS_URI != null ? FS_URI : FileSystem.get(conf).getUri().toString(), 1,
      null, null, new JobConf(this.conf));
      JobConf jobConf = MapreduceTestingShim.getJobConf(mrCluster);
      if (jobConf == null)

      Unknown macro: { jobConf = mrCluster.createJobConf(); }

      jobConf.set("mapreduce.cluster.local.dir",
      conf.get("mapreduce.cluster.local.dir")); //Hadoop MiniMR overwrites this while it should not
      LOG.info("Mini mapreduce cluster started");

      // In hadoop2, YARN/MR2 starts a mini cluster with its own conf instance and updates settings.
      // Our HBase MR jobs need several of these settings in order to properly run. So we copy the
      // necessary config properties here. YARN-129 required adding a few properties.
      conf.set("mapreduce.jobtracker.address", jobConf.get("mapreduce.jobtracker.address"));
      // this for mrv2 support; mr1 ignores this
      conf.set("mapreduce.framework.name", "yarn");
      conf.setBoolean("yarn.is.minicluster", true);
      String rmAddress = jobConf.get("yarn.resourcemanager.address");
      if (rmAddress != null)

      Unknown macro: { conf.set("yarn.resourcemanager.address", rmAddress); }

      String historyAddress = jobConf.get("mapreduce.jobhistory.address");
      if (historyAddress != null)

      Unknown macro: { conf.set("mapreduce.jobhistory.address", historyAddress); }

      String schedulerAddress =
      jobConf.get("yarn.resourcemanager.scheduler.address");
      if (schedulerAddress != null)

      Unknown macro: { conf.set("yarn.resourcemanager.scheduler.address", schedulerAddress); }

      As a immediate fix for phoenix e2e test to succeed, need the below lines to be added as well

      String rmWebappAddress = jobConf.get("yarn.resourcemanager.webapp.address");
      if (rmWebappAddress != null)

      Unknown macro: { conf.set("yarn.resourcemanager.webapp.address", rmWebappAddress); }

      String historyWebappAddress = jobConf.get("mapreduce.jobhistory.webapp.address");
      if (historyWebappAddress != null)

      Unknown macro: { conf.set("mapreduce.jobhistory.webapp.address", historyWebappAddress); }

      Eventually, we should also see if we can copy over all the jobConf properties to HBaseTestingUtility conf object.

        Attachments

        1. HBASE-16375.patch
          5 kB
          Andrew Kyle Purtell
        2. HBASE-16375.master.001.patch
          5 kB
          Loknath Priyatham Teja Singamsetty
        3. HBASE-16375_0.98_and_above.patch
          1 kB
          Loknath Priyatham Teja Singamsetty
        4. HBASE-16375_0.98_and_above_with_tests.patch
          4 kB
          Loknath Priyatham Teja Singamsetty
        5. HBASE-16375_0.98_and_above_with_tests_format.patch
          5 kB
          Loknath Priyatham Teja Singamsetty

          Activity

            People

            • Assignee:
              singamteja Loknath Priyatham Teja Singamsetty
              Reporter:
              singamteja Loknath Priyatham Teja Singamsetty
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: