Details
-
Improvement
-
Status: Resolved
-
Critical
-
Resolution: Not A Bug
-
None
-
None
-
None
-
Cloudera CDH4 latest on Centos
Description
J Harsh suggested filing a usability issue concerning a problem he helped me resolve via email. Problem seems to boil down to differences between localhost and localhost.localdomain that I don't understand well enough to explain here. I've enclosed the email thread as documentation.
J Harsh...
I believe we seem to be doing a strict hostname check in Apache Oozie
and not a resolving check. Could you report an improvement on the
Apache Oozie JIRA (https://issues.apache.org/jira/browse/OOZIE)
upstream to perhaps improve this for usability?
On Sun, Sep 1, 2013 at 2:41 AM, Brad Cox <bradjcox@gmail.com> wrote:
Copied examples to /user/cloudera/examples and changed nameNode to
hdfs://localhost.localdomain in job.properties. Results look like a good
sign.
[cloudera@localhost Desktop]$ oozie job -oozie http://localhost:11000/oozie
-config examples/apps/map-reduce/job.properties -run
job: 0000000-130831130415538-oozie-oozi-W
But in oozie console, got a job fail because jobTracker's not in whitelist
so changed that to just localhost.localdomain by analogy.
[cloudera@localhost Desktop]$ oozie job -oozie http://localhost:11000/oozie
-config examples/apps/map-reduce/job.properties -run
job: 0000001-130831130415538-oozie-oozi-W
Console says that try succeeded. So thanks for the help!!
All I'm missing is an understanding of why cloudera comes outOfBox with some
services configured as localhost and others localhost.localdomain. I always
thought they were equivalent. But these are huge differences; success vs
fail.
On Aug 31, 2013, at 4:58 PM, Brad Cox <bradjcox@gmail.com> wrote:
You mean use localhost.localdomain instead of localhost? Or should I define
a unique name with hostname cmd, restart all, and use that instead?
[cloudera@localhost Desktop]$ hostname
localhost.localdomain
On Aug 31, 2013, at 4:54 PM, Harsh J <harsh@cloudera.com> wrote:
The whitelist message is close to the solution. Do not use 'localhost'
in your job.properties, but replace it with whatever the command
'hostname' produces on your terminal, and I believe it should do the
trick.
On Sat, Aug 31, 2013 at 11:11 PM, Brad Cox <bradjcox@gmail.com> wrote:
That's what I tried in previous attempts. That causes this
oozie job -oozie http://localhost:11000/oozie -config
examples/apps/map-reduce/job.properties -run
Error: HTTP error code: 500 : Internal Server Error
/var/log/lochost.date.log contains
Caused by: java.net.URISyntaxException: Relative path in absolute URI:
localhost:8020/user/cloudera/examples/apps/map-reduce
at java.net.URI.checkPath(URI.java:1788)
examples is where I put it as you said
[cloudera@localhost Desktop]$ hdfs dfs -ls /
Found 8 items
drwxr-xr-x - hdfs supergroup 0 2013-08-30 15:28 /accumulo
drwxr-xr-x - oozie supergroup 0 2013-08-31 04:52 /examples
drwxr-xr-x - hbase hbase 0 2013-08-31 07:22 /hbase
drwxr-xr-x - hdfs supergroup 0 2013-08-30 16:38 /share
drwxr-xr-x - solr solr 0 2013-07-17 00:03 /solr
drwxrwxrwx - hdfs supergroup 0 2013-08-30 16:32 /tmp
drwxr-xr-x - hdfs supergroup 0 2013-08-30 16:32 /user
drwxr-xr-x - hdfs supergroup 0 2013-07-17 00:03 /var
All this is after hacking nameNode=localhost:8020 (was hdfs://...) in
job.properties based on the Running the Examples section of the Oozie docs.
On reverting that change, the message changed to the problem I had before:
oozie job -oozie http://localhost:11000/oozie -config
examples/apps/map-reduce/job.properties -run
Error: E0901 : E0901: Namenode [localhost:8020] not allowed, not in Oozies
whitelist
Based on that msg, I'd already changed
oozie.service.HadoopAccessorService.nameNode.whitelist to localhost:8020 in
/etc/oozie/conf/oozie-site.xml and just made a similar change to
jobTracker.whitelist to localhost:8021. Best I've been able to do is switch
mystifying messages.
On Aug 31, 2013, at 12:22 PM, Harsh J <harsh@cloudera.com> wrote:
Hi Brad,
In Oozie, an simple "application" consists of a directory on HDFS that
carries workflow.xml, and associated libraries/files.
To invoke an instance of such an application, we use a local
job.properties file that carries any variable parameter values, to
begin the application with.
Typically, to run the examples out of the box on a standalone cluster,
you can run the following set of commands (assuming you do have the
oozie-examples tarball):
$ tar xf oozie-examples.tar.gz
$ hadoop fs -put examples examples
$ oozie job -oozie http://localhost:11000/oozie -config
examples/apps/map-reduce/job.properties -run
Can you tell me what error you face if you try this out?
On Sat, Aug 31, 2013 at 8:22 PM, Brad Cox <bradjcox@gmail.com> wrote:
I'm trying the ozzie examples in CDH4.1 and the error messages make no
sense. Got through the others by editing the properties file but this one
really stumps me.
I've copied the ozzie examples folder to /home/cloudera/Desktop and have
verified that job.properties path is right.
[cloudera@localhost Desktop]$ whoami
cloudera
[cloudera@localhost Desktop]$ ls
/home/cloudera/Desktop/examples/apps/map-reduce/job.properties
Run as cloudera user and trying various permutations on the path
(with/without file:// prefix) I keep getting this.
oozie job -oozie http://localhost:11000/oozie -config
file:///home/cloudera/Desktop/examples/apps/map-reduce/job.properties -run
java.io.IOException: configuration file
file:///home/cloudera/Desktop/examples/apps/map-reduce/job.properties not
found
at org.apache.oozie.cli.OozieCLI.getConfiguration(OozieCLI.java:641)
at org.apache.oozie.cli.OozieCLI.jobCommand(OozieCLI.java:795)
at org.apache.oozie.cli.OozieCLI.processCommand(OozieCLI.java:512)
at org.apache.oozie.cli.OozieCLI.run(OozieCLI.java:485)
at org.apache.oozie.cli.OozieCLI.main(OozieCLI.java:182)
configuration file
file:///home/cloudera/Desktop/examples/apps/map-reduce/job.properties not
found
I also tried putting the examples in hdfs as the instructions said but that
doesn't help. In fact that's entirely the wrong idea according to some
emails I've seen.
Dr. Brad J. Cox Cell: 703-594-1883 Blog: http://bradjcox.blogspot.com
http://virtualschool.edu