Uploaded image for project: 'Oozie'
  1. Oozie
  2. OOZIE-1516

Usability issues submitting jobs

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Critical
    • Resolution: Not A Bug
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: examples
    • Labels:
      None
    • Environment:

      Cloudera CDH4 latest on Centos

      Description

      J Harsh suggested filing a usability issue concerning a problem he helped me resolve via email. Problem seems to boil down to differences between localhost and localhost.localdomain that I don't understand well enough to explain here. I've enclosed the email thread as documentation.

      J Harsh...
      I believe we seem to be doing a strict hostname check in Apache Oozie
      and not a resolving check. Could you report an improvement on the
      Apache Oozie JIRA (https://issues.apache.org/jira/browse/OOZIE)
      upstream to perhaps improve this for usability?

      On Sun, Sep 1, 2013 at 2:41 AM, Brad Cox <bradjcox@gmail.com> wrote:
      Copied examples to /user/cloudera/examples and changed nameNode to
      hdfs://localhost.localdomain in job.properties. Results look like a good
      sign.

      [cloudera@localhost Desktop]$ oozie job -oozie http://localhost:11000/oozie
      -config examples/apps/map-reduce/job.properties -run
      job: 0000000-130831130415538-oozie-oozi-W

      But in oozie console, got a job fail because jobTracker's not in whitelist
      so changed that to just localhost.localdomain by analogy.

      [cloudera@localhost Desktop]$ oozie job -oozie http://localhost:11000/oozie
      -config examples/apps/map-reduce/job.properties -run
      job: 0000001-130831130415538-oozie-oozi-W

      Console says that try succeeded. So thanks for the help!!

      All I'm missing is an understanding of why cloudera comes outOfBox with some
      services configured as localhost and others localhost.localdomain. I always
      thought they were equivalent. But these are huge differences; success vs
      fail.

      On Aug 31, 2013, at 4:58 PM, Brad Cox <bradjcox@gmail.com> wrote:

      You mean use localhost.localdomain instead of localhost? Or should I define
      a unique name with hostname cmd, restart all, and use that instead?

      [cloudera@localhost Desktop]$ hostname
      localhost.localdomain

      On Aug 31, 2013, at 4:54 PM, Harsh J <harsh@cloudera.com> wrote:

      The whitelist message is close to the solution. Do not use 'localhost'
      in your job.properties, but replace it with whatever the command
      'hostname' produces on your terminal, and I believe it should do the
      trick.

      On Sat, Aug 31, 2013 at 11:11 PM, Brad Cox <bradjcox@gmail.com> wrote:

      That's what I tried in previous attempts. That causes this

      oozie job -oozie http://localhost:11000/oozie -config
      examples/apps/map-reduce/job.properties -run
      Error: HTTP error code: 500 : Internal Server Error

      /var/log/lochost.date.log contains
      Caused by: java.net.URISyntaxException: Relative path in absolute URI:
      localhost:8020/user/cloudera/examples/apps/map-reduce
      at java.net.URI.checkPath(URI.java:1788)

      examples is where I put it as you said
      [cloudera@localhost Desktop]$ hdfs dfs -ls /
      Found 8 items
      drwxr-xr-x - hdfs supergroup 0 2013-08-30 15:28 /accumulo
      drwxr-xr-x - oozie supergroup 0 2013-08-31 04:52 /examples
      drwxr-xr-x - hbase hbase 0 2013-08-31 07:22 /hbase
      drwxr-xr-x - hdfs supergroup 0 2013-08-30 16:38 /share
      drwxr-xr-x - solr solr 0 2013-07-17 00:03 /solr
      drwxrwxrwx - hdfs supergroup 0 2013-08-30 16:32 /tmp
      drwxr-xr-x - hdfs supergroup 0 2013-08-30 16:32 /user
      drwxr-xr-x - hdfs supergroup 0 2013-07-17 00:03 /var

      All this is after hacking nameNode=localhost:8020 (was hdfs://...) in
      job.properties based on the Running the Examples section of the Oozie docs.
      On reverting that change, the message changed to the problem I had before:

      oozie job -oozie http://localhost:11000/oozie -config
      examples/apps/map-reduce/job.properties -run
      Error: E0901 : E0901: Namenode [localhost:8020] not allowed, not in Oozies
      whitelist

      Based on that msg, I'd already changed
      oozie.service.HadoopAccessorService.nameNode.whitelist to localhost:8020 in
      /etc/oozie/conf/oozie-site.xml and just made a similar change to
      jobTracker.whitelist to localhost:8021. Best I've been able to do is switch
      mystifying messages.

      On Aug 31, 2013, at 12:22 PM, Harsh J <harsh@cloudera.com> wrote:

      Hi Brad,

      In Oozie, an simple "application" consists of a directory on HDFS that
      carries workflow.xml, and associated libraries/files.

      To invoke an instance of such an application, we use a local
      job.properties file that carries any variable parameter values, to
      begin the application with.

      Typically, to run the examples out of the box on a standalone cluster,
      you can run the following set of commands (assuming you do have the
      oozie-examples tarball):

      $ tar xf oozie-examples.tar.gz
      $ hadoop fs -put examples examples
      $ oozie job -oozie http://localhost:11000/oozie -config
      examples/apps/map-reduce/job.properties -run

      Can you tell me what error you face if you try this out?

      On Sat, Aug 31, 2013 at 8:22 PM, Brad Cox <bradjcox@gmail.com> wrote:

      I'm trying the ozzie examples in CDH4.1 and the error messages make no
      sense. Got through the others by editing the properties file but this one
      really stumps me.

      I've copied the ozzie examples folder to /home/cloudera/Desktop and have
      verified that job.properties path is right.

      [cloudera@localhost Desktop]$ whoami
      cloudera
      [cloudera@localhost Desktop]$ ls
      /home/cloudera/Desktop/examples/apps/map-reduce/job.properties

      Run as cloudera user and trying various permutations on the path
      (with/without file:// prefix) I keep getting this.

      oozie job -oozie http://localhost:11000/oozie -config
      file:///home/cloudera/Desktop/examples/apps/map-reduce/job.properties -run
      java.io.IOException: configuration file
      file:///home/cloudera/Desktop/examples/apps/map-reduce/job.properties not
      found
      at org.apache.oozie.cli.OozieCLI.getConfiguration(OozieCLI.java:641)
      at org.apache.oozie.cli.OozieCLI.jobCommand(OozieCLI.java:795)
      at org.apache.oozie.cli.OozieCLI.processCommand(OozieCLI.java:512)
      at org.apache.oozie.cli.OozieCLI.run(OozieCLI.java:485)
      at org.apache.oozie.cli.OozieCLI.main(OozieCLI.java:182)
      configuration file
      file:///home/cloudera/Desktop/examples/apps/map-reduce/job.properties not
      found

      I also tried putting the examples in hdfs as the instructions said but that
      doesn't help. In fact that's entirely the wrong idea according to some
      emails I've seen.

      Dr. Brad J. Cox Cell: 703-594-1883 Blog: http://bradjcox.blogspot.com
      http://virtualschool.edu

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              bradjcox Brad Cox
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: