GitHub user chinmaykolhatkar opened a pull request:

https://github.com/apache/bigtop/pull/92

BIGTOP-2314 Added deb and rpm package of Apache Apex to bigtop.

This pull request add deb and rpm packaging of apex as a part of bigtop.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chinmaykolhatkar/bigtop BIGTOP-2314

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/bigtop/pull/92.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #92

commit 3986ea7a623570dd2f88a7469d369451bdb65c20
Author: chinmaykolhatkar <chinmay@datatorrent.com>
Date: 2016-02-09T16:17:34Z

BIGTOP-2314 Added deb and rpm package of Apache Apex to bigtop.

Patch Available at:
https://github.com/apache/bigtop/pull/92

Github user youngwookim commented on a diff in the pull request:

https://github.com/apache/bigtop/pull/92#discussion_r53780176

— Diff: bigtop-packages/src/common/apex/do-component-build —
@@ -0,0 +1,81 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+. `dirname $0`/bigtop.bom
+
+HADOOP_VERSION=2.2.0
— End diff –

Is it intended? if you need a version of the dependency, just use the HADOOP_VERSION from bigtop.bom like this https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hive/do-component-build#L24

Github user youngwookim commented on a diff in the pull request:

https://github.com/apache/bigtop/pull/92#discussion_r53784215

— Diff: bigtop-packages/src/common/apex/install_apex.sh —
@@ -0,0 +1,111 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -e
+
+usage()

+
+OPTS=$(getopt \
+ -n $0 \
+ -o '' \
+ -l 'prefix:' \
+ -l 'doc-dir:' \
+ -l 'lib-dir:' \
+ -l 'bin-dir:' – "$@")
+
+if [ $? != 0 ] ; then
+ usage
+fi
+
+eval set – "$OPTS"
+while true ; do
+ case "$1" in
+ --prefix)
+ PREFIX=$2 ; shift 2
+ ;;
+ --build-dir)
+ BUILD_DIR=$2 ; shift 2
+ ;;
+ --doc-dir)
+ DOC_DIR=$2 ; shift 2
+ ;;
+ --lib-dir)
+ LIB_DIR=$2 ; shift 2
+ ;;
+ --bin-dir)
+ BIN_DIR=$2 ; shift 2
+ ;;
+ --)
+ shift ; break
+ ;;
+ *)
+ echo "Unknown option: $1"
+ usage
+ exit 1
+ ;;
+ esac
+done
+
+for var in PREFIX ; do
+ if [ -z "$(eval "echo \$$var")" ]; then
+ echo Missing param: $var
+ usage
+ fi
+done
+
+DOC_DIR=$

+LIB_DIR=$

+BIN_DIR=$

+MAN_DIR=/usr/share/man/man1
+
+install -d -m 0755 $PREFIX/$LIB_DIR
+install -d -m 0755 $PREFIX/$LIB_DIR/bin
+install -d -m 0755 $PREFIX/$LIB_DIR/lib
+install -d -m 0755 $PREFIX/$BIN_DIR
+install -d -m 0755 $PREFIX/$DOC_DIR
+install -d -m 0755 $PREFIX/$MAN_DIR
+
+# Install dtcli
+cp engine/src/main/scripts/dtcli $PREFIX/$LIB_DIR/bin/apex
+chmod 755 $PREFIX/$LIB_DIR/bin/apex
+# Create symlinks
— End diff –

Not sure why it is needed. if we install $PREFIX/$LIB_DIR/bin/apex then, it would be nice to have wrappers for that like these: https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/install_hadoop.sh#L167 and https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hive/install_hive.sh#L135

Github user chinmaykolhatkar commented on a diff in the pull request:

https://github.com/apache/bigtop/pull/92#discussion_r53926545

— Diff: bigtop-packages/src/common/apex/do-component-build —
@@ -0,0 +1,81 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+. `dirname $0`/bigtop.bom
+
+HADOOP_VERSION=2.2.0
— End diff –

I removed this... Though we need this to exclude the dependencies of hadoop 2.2 while copying min set of runtime libraries.

Github user chinmaykolhatkar commented on a diff in the pull request:

https://github.com/apache/bigtop/pull/92#discussion_r53926567

— Diff: bigtop-packages/src/common/apex/install_apex.sh —
@@ -0,0 +1,111 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -e
+
+usage()

+
+OPTS=$(getopt \
+ -n $0 \
+ -o '' \
+ -l 'prefix:' \
+ -l 'doc-dir:' \
+ -l 'lib-dir:' \
+ -l 'bin-dir:' – "$@")
+
+if [ $? != 0 ] ; then
+ usage
+fi
+
+eval set – "$OPTS"
+while true ; do
+ case "$1" in
+ --prefix)
+ PREFIX=$2 ; shift 2
+ ;;
+ --build-dir)
+ BUILD_DIR=$2 ; shift 2
+ ;;
+ --doc-dir)
+ DOC_DIR=$2 ; shift 2
+ ;;
+ --lib-dir)
+ LIB_DIR=$2 ; shift 2
+ ;;
+ --bin-dir)
+ BIN_DIR=$2 ; shift 2
+ ;;
+ --)
+ shift ; break
+ ;;
+ *)
+ echo "Unknown option: $1"
+ usage
+ exit 1
+ ;;
+ esac
+done
+
+for var in PREFIX ; do
+ if [ -z "$(eval "echo \$$var")" ]; then
+ echo Missing param: $var
+ usage
+ fi
+done
+
+DOC_DIR=$

+LIB_DIR=$

+BIN_DIR=$

+MAN_DIR=/usr/share/man/man1
+
+install -d -m 0755 $PREFIX/$LIB_DIR
+install -d -m 0755 $PREFIX/$LIB_DIR/bin
+install -d -m 0755 $PREFIX/$LIB_DIR/lib
+install -d -m 0755 $PREFIX/$BIN_DIR
+install -d -m 0755 $PREFIX/$DOC_DIR
+install -d -m 0755 $PREFIX/$MAN_DIR
+
+# Install dtcli
+cp engine/src/main/scripts/dtcli $PREFIX/$LIB_DIR/bin/apex
+chmod 755 $PREFIX/$LIB_DIR/bin/apex
+# Create symlinks
— End diff –

Modified as per the suggestion.

Chinmay Kolhatkar Thanks for the update! minor nit:

$ rpm -qpl output/apex/noarch/apex-3.3.0-1.el6.noarch.rpm 
......
/usr/lib/apex/lib/hadoop-common-2.2.0-tests.jar

Should be remove the test jar from package.

Overall looks good to me just curious and I don't have expertise on Apex, can you elaborate on the pom with Hadoop version 2.2?

YoungWoo Kim I've remove hadoop-common-2.2.0-tests.jar. The PR is updated.

Now, about hadoop version 2.2:
Apex is supported with minimum version of hadoop 2.2. Hence this is added as a dependency for apex maven.
But the runtime dependency of apex needs hadoop to be in classpath + some other apex related dependencies.

As apex cli class is run with hadoop binary, all hadoop runtime dependencies gets into classpath automatically.
Hence what we need to include, in /usr/lib/apex/lib, only those dependencies which are only specific to apex.

Basically, we want to do (A - B) where A is all apex runtime dependencies and B is hadoop runtime dependencies.

Hence to find hadoop dependencies there is a pom with hadoop version 2.2.

Chinmay Kolhatkar Thanks for your explanation.

Will commit this shortly. great work Chinmay Kolhatkar! Thanks for your contribution.

Github user chinmaykolhatkar commented on the pull request:

https://github.com/apache/bigtop/pull/92#issuecomment-211263273

Closing as this PR is merged.

Github user chinmaykolhatkar closed the pull request at:

https://github.com/apache/bigtop/pull/92