Spec file and SRPM for building a Hadoop-0.19.1 RPM

• Key: HADOOP-5615
• Type: Improvement
• Status: Open
• Priority: Minor
• Assignee: Unassigned
• Reporter: Ian Soboroff
• Component/s: build
• Affects Version/s: 0.19.1

Description

I like the idea of Cloudera's RPMs, in that packages are convenient and the boot scripts are very handy, but they are for a patched 0.18.3 and include other stuff. Here I offer a spec file for 0.19.1 without extra cruft. It is essentially the spec from Cloudera's RPM with suitable edits.

Comments

Here is the edited Cloudera spec file for building an RPM from vanilla 0.19.1. I was going to also include the SRPM but it's too big to attach to a JIRA. Until I find somewhere else to post it, here's how to build it yourself.

1. get the SRPM from Cloudera and 'rpm -i' it.
2. replace SPECS/hadoop.spec with this one.
3. put a vanilla hadoop-0.19.1.tar.gz in SOURCES
4. replace Cloudera's SOURCES/logo.pnm with the Hadoop logo
5. do 'rpmbuild --target=noarch -ba hadoop.spec'.

The Cloudera spec file has no license in it, so I assume that it can be licensed ASF.

[Michael Thomas]

The update to 0.19.1 is very helpful. But there are a number of style issues in cloudera's spec file that I hope we can get cleaned up:

• Using %define to define the standard system directories is a bit overkill. Use the predefined macros %{_bindir}, %{_libdir}, etc. to get the standard system locations for these files.

• Using the hadoop binary distribution to build the rpm seems a bit backwards. It would be better of there was a source-only tarball on the download page to reduce the size of the src rpm.

• On a similar note, it would be really nice to split the (nonexistent) hadoop source tarball into a noarch and arch-specific component. This would allow building arch and arch-specific rpms without resorting to the "%ifarch noarch" frangments in the spec file, which can be awkward to handle for automated build systems.

• The %changelog section is missing at the end of the spec file.

• Missing "BuildRequires: subversion" to generate the build/src/org/apache/hadoop/package-info.java file

• Use 'getent' before creating the hadoop user to avoid errors if the user already exists (from a directory service, for example):
  getent passwd hadoop >/dev/null || \
  useradd -r -g hadoop -c "Hadoop runtime user" \
  -s /bin/bash hadoop

• Move the pid file to /var/run/hadoop/*.pid, the standard place to put pid files.

I've attaching the spec files that I've been using for hadoop and hdfs, for reference. Mine are probably not suitable for general use because I cut out most of the contrib packages, and my init script doesn't support the job tracker (I'm only interested in HDFS).

[Michael Thomas]

Another hadoop spec file, for reference

[Christophe Bisciglia]

To remove any ambiguity - if the license isn't clearly specified, it was intended to be Apache 2.0 - we are happy to see this patch. We'll clarify this in future releases.

We're also happy to help get this right in hadoop proper, and are willing to host proper hadoop RPMs in our development repository. Depending on how quickly this patch is completed, we may start hosting 19.1 RPM's sooner than later.