Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-2283 QFS Integration
  3. BIGTOP-2285

Add qfs rpm and debian packaging code

        Details

        • Type: Sub-task
        • Status: Closed
        • Priority: Major
        • Resolution: Fixed
        • Affects Version/s: 1.1.0
        • Fix Version/s: 1.2.0
        • Component/s: debian, rpm, toolchain
        • Labels:

          Activity

          Ascending order - Click to sort in descending order
          Hide
          Permalink
          githubbot ASF GitHub Bot added a comment -

          GitHub user fsareshwala opened a pull request:

          https://github.com/apache/bigtop/pull/77

          BIGTOP-2285. add qfs rpm and debian packaging code

          This change adds code to bigtop to package qfs for debian and rpm systems. The
          packages produced are specific to their functions, e.g. a separate package for
          the metaserver, chunkserver, webui, client tools, development headers and
          libraries, etc. This way, users can select which packages they wish to install
          on which system in their infrastructure.

          In order to build qfs and its respective packages, various dependencies were
          added to the toolchain puppet tasks (e.g. boost-devel). These will be installed
          along with the other dependencies that get installed during the toolchain gradle
          task. In doing so, users should be presented with a seamless setup process to
          get qfs packages building.

          I have tested to make sure that packaging is correct by using the qfs-rpm and
          qfs-deb grandle tasks to generate the respective rpm and debian packages. I then
          manually inspected package contents to ensure that files were being installed
          into the correct location.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/fsareshwala/bigtop master

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/bigtop/pull/77.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #77

          commit dd33286309ef111d2ac23bf2ab1df327b64c971a
          Author: Faraaz Sareshwala <fsareshwala@quantcast.com>
          Date: 2016-01-13T23:29:46Z

          BIGTOP-2285. add qfs rpm and debian packaging code

          This change adds code to bigtop to package qfs for debian and rpm systems. The
          packages produced are specific to their functions, e.g. a separate package for
          the metaserver, chunkserver, webui, client tools, development headers and
          libraries, etc. This way, users can select which packages they wish to install
          on which system in their infrastructure.

          In order to build qfs and its respective packages, various dependencies were
          added to the toolchain puppet tasks (e.g. boost-devel). These will be installed
          along with the other dependencies that get installed during the toolchain gradle
          task. In doing so, users should be presented with a seamless setup process to
          get qfs packages building.

          I have tested to make sure that packaging is correct by using the qfs-rpm and
          qfs-deb grandle tasks to generate the respective rpm and debian packages. I then
          manually inspected package contents to ensure that files were being installed
          into the correct location.

          Show
          githubbot ASF GitHub Bot added a comment - GitHub user fsareshwala opened a pull request: https://github.com/apache/bigtop/pull/77 BIGTOP-2285 . add qfs rpm and debian packaging code This change adds code to bigtop to package qfs for debian and rpm systems. The packages produced are specific to their functions, e.g. a separate package for the metaserver, chunkserver, webui, client tools, development headers and libraries, etc. This way, users can select which packages they wish to install on which system in their infrastructure. In order to build qfs and its respective packages, various dependencies were added to the toolchain puppet tasks (e.g. boost-devel). These will be installed along with the other dependencies that get installed during the toolchain gradle task. In doing so, users should be presented with a seamless setup process to get qfs packages building. I have tested to make sure that packaging is correct by using the qfs-rpm and qfs-deb grandle tasks to generate the respective rpm and debian packages. I then manually inspected package contents to ensure that files were being installed into the correct location. You can merge this pull request into a Git repository by running: $ git pull https://github.com/fsareshwala/bigtop master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/bigtop/pull/77.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #77 commit dd33286309ef111d2ac23bf2ab1df327b64c971a Author: Faraaz Sareshwala <fsareshwala@quantcast.com> Date: 2016-01-13T23:29:46Z BIGTOP-2285 . add qfs rpm and debian packaging code This change adds code to bigtop to package qfs for debian and rpm systems. The packages produced are specific to their functions, e.g. a separate package for the metaserver, chunkserver, webui, client tools, development headers and libraries, etc. This way, users can select which packages they wish to install on which system in their infrastructure. In order to build qfs and its respective packages, various dependencies were added to the toolchain puppet tasks (e.g. boost-devel). These will be installed along with the other dependencies that get installed during the toolchain gradle task. In doing so, users should be presented with a seamless setup process to get qfs packages building. I have tested to make sure that packaging is correct by using the qfs-rpm and qfs-deb grandle tasks to generate the respective rpm and debian packages. I then manually inspected package contents to ensure that files were being installed into the correct location.
          Hide
          Permalink
          fsareshwala Faraaz Sareshwala added a comment -

          https://github.com/apache/bigtop/pull/77

          Show
          fsareshwala Faraaz Sareshwala added a comment - https://github.com/apache/bigtop/pull/77
          Hide
          Permalink
          oflebbe Olaf Flebbe added a comment -

          Added Faraaz Sareshwala to contributors and assigned ticket.

          Show
          oflebbe Olaf Flebbe added a comment - Added Faraaz Sareshwala to contributors and assigned ticket.
          Hide
          Permalink
          githubbot ASF GitHub Bot added a comment -

          Github user fsareshwala commented on the pull request:

          https://github.com/apache/bigtop/pull/77#issuecomment-181655944

          Closing this pull request out so that I can fix the source branch.

          Show
          githubbot ASF GitHub Bot added a comment - Github user fsareshwala commented on the pull request: https://github.com/apache/bigtop/pull/77#issuecomment-181655944 Closing this pull request out so that I can fix the source branch.
          Hide
          Permalink
          githubbot ASF GitHub Bot added a comment -

          Github user fsareshwala closed the pull request at:

          https://github.com/apache/bigtop/pull/77

          Show
          githubbot ASF GitHub Bot added a comment - Github user fsareshwala closed the pull request at: https://github.com/apache/bigtop/pull/77
          Hide
          Permalink
          githubbot ASF GitHub Bot added a comment -

          GitHub user fsareshwala opened a pull request:

          https://github.com/apache/bigtop/pull/83

          BIGTOP-2285. add qfs rpm and debian packaging code

          This change adds code to bigtop to package qfs for debian and rpm systems. The
          packages produced are specific to their functions, e.g. a separate package for
          the metaserver, chunkserver, webui, client tools, development headers and
          libraries, etc. This way, users can select which packages they wish to install
          on which system in their infrastructure. One caveat worth mentioning is that the
          packages themselves don't contain configuration files for the services in
          question. The configuration files are provided by the puppet deploy scripts that
          come with the bigtop repository. Those will be submitted in a separate change.

          In order to build qfs and its respective packages, various dependencies were
          added to the toolchain puppet tasks (e.g. boost-devel). These will be installed
          along with the other dependencies that get installed during the toolchain gradle
          task. In doing so, users should be presented with a seamless setup process to
          get qfs packages building.

          I have tested to make sure that packaging is correct by using the qfs-rpm and
          qfs-deb grandle tasks to generate the respective rpm and debian packages. I then
          manually inspected package contents to ensure that files were being installed
          into the correct location.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/fsareshwala/bigtop qfs-packaging

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/bigtop/pull/83.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #83

          commit c5f45c8f4b6e43e06f439a0cdb89844be65571fc
          Author: Faraaz Sareshwala <fsareshwala@quantcast.com>
          Date: 2016-01-13T23:29:46Z

          BIGTOP-2285. add qfs rpm and debian packaging code

          This change adds code to bigtop to package qfs for debian and rpm systems. The
          packages produced are specific to their functions, e.g. a separate package for
          the metaserver, chunkserver, webui, client tools, development headers and
          libraries, etc. This way, users can select which packages they wish to install
          on which system in their infrastructure. One caveat worth mentioning is that the
          packages themselves don't contain configuration files for the services in
          question. The configuration files are provided by the puppet deploy scripts that
          come with the bigtop repository. Those will be submitted in a separate change.

          In order to build qfs and its respective packages, various dependencies were
          added to the toolchain puppet tasks (e.g. boost-devel). These will be installed
          along with the other dependencies that get installed during the toolchain gradle
          task. In doing so, users should be presented with a seamless setup process to
          get qfs packages building.

          I have tested to make sure that packaging is correct by using the qfs-rpm and
          qfs-deb grandle tasks to generate the respective rpm and debian packages. I then
          manually inspected package contents to ensure that files were being installed
          into the correct location.

          Show
          githubbot ASF GitHub Bot added a comment - GitHub user fsareshwala opened a pull request: https://github.com/apache/bigtop/pull/83 BIGTOP-2285 . add qfs rpm and debian packaging code This change adds code to bigtop to package qfs for debian and rpm systems. The packages produced are specific to their functions, e.g. a separate package for the metaserver, chunkserver, webui, client tools, development headers and libraries, etc. This way, users can select which packages they wish to install on which system in their infrastructure. One caveat worth mentioning is that the packages themselves don't contain configuration files for the services in question. The configuration files are provided by the puppet deploy scripts that come with the bigtop repository. Those will be submitted in a separate change. In order to build qfs and its respective packages, various dependencies were added to the toolchain puppet tasks (e.g. boost-devel). These will be installed along with the other dependencies that get installed during the toolchain gradle task. In doing so, users should be presented with a seamless setup process to get qfs packages building. I have tested to make sure that packaging is correct by using the qfs-rpm and qfs-deb grandle tasks to generate the respective rpm and debian packages. I then manually inspected package contents to ensure that files were being installed into the correct location. You can merge this pull request into a Git repository by running: $ git pull https://github.com/fsareshwala/bigtop qfs-packaging Alternatively you can review and apply these changes as the patch at: https://github.com/apache/bigtop/pull/83.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #83 commit c5f45c8f4b6e43e06f439a0cdb89844be65571fc Author: Faraaz Sareshwala <fsareshwala@quantcast.com> Date: 2016-01-13T23:29:46Z BIGTOP-2285 . add qfs rpm and debian packaging code This change adds code to bigtop to package qfs for debian and rpm systems. The packages produced are specific to their functions, e.g. a separate package for the metaserver, chunkserver, webui, client tools, development headers and libraries, etc. This way, users can select which packages they wish to install on which system in their infrastructure. One caveat worth mentioning is that the packages themselves don't contain configuration files for the services in question. The configuration files are provided by the puppet deploy scripts that come with the bigtop repository. Those will be submitted in a separate change. In order to build qfs and its respective packages, various dependencies were added to the toolchain puppet tasks (e.g. boost-devel). These will be installed along with the other dependencies that get installed during the toolchain gradle task. In doing so, users should be presented with a seamless setup process to get qfs packages building. I have tested to make sure that packaging is correct by using the qfs-rpm and qfs-deb grandle tasks to generate the respective rpm and debian packages. I then manually inspected package contents to ensure that files were being installed into the correct location.
          Hide
          Permalink
          githubbot ASF GitHub Bot added a comment -

          Github user fsareshwala commented on the pull request:

          https://github.com/apache/bigtop/pull/77#issuecomment-181656131

          New packaging pull request: https://github.com/apache/bigtop/pull/83

          Show
          githubbot ASF GitHub Bot added a comment - Github user fsareshwala commented on the pull request: https://github.com/apache/bigtop/pull/77#issuecomment-181656131 New packaging pull request: https://github.com/apache/bigtop/pull/83
          Hide
          Permalink
          oflebbe Olaf Flebbe added a comment -

          Thank you very much for your contribution! Testing it right now, if it compiles in my environment.

          Show
          oflebbe Olaf Flebbe added a comment - Thank you very much for your contribution! Testing it right now, if it compiles in my environment.
          Hide
          Permalink
          fsareshwala Faraaz Sareshwala added a comment -

          Thanks Olaf! Let me know if you need any help with anything. I'm available on IRC as well as email if you want a more realtime communication mechanism.

          Show
          fsareshwala Faraaz Sareshwala added a comment - Thanks Olaf! Let me know if you need any help with anything. I'm available on IRC as well as email if you want a more realtime communication mechanism.
          Hide
          Permalink
          oflebbe Olaf Flebbe added a comment -

          The debian and ubuntu packages do not compile because of the unmet build requirement "maven" ,
          since we do not install the "maven" package from debian/ubuntu (for a purpose) .

          opensuse fails at 

          running build_ext
building 'qfs' extension
creating build
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/ws
creating build/temp.linux-x86_64-2.7/ws/build
creating build/temp.linux-x86_64-2.7/ws/build/qfs
creating build/temp.linux-x86_64-2.7/ws/build/qfs/rpm
creating build/temp.linux-x86_64-2.7/ws/build/qfs/rpm/BUILD
creating build/temp.linux-x86_64-2.7/ws/build/qfs/rpm/BUILD/qfs-1.1.4
creating build/temp.linux-x86_64-2.7/ws/build/qfs/rpm/BUILD/qfs-1.1.4/src
creating build/temp.linux-x86_64-2.7/ws/build/qfs/rpm/BUILD/qfs-1.1.4/src/cc
creating build/temp.linux-x86_64-2.7/ws/build/qfs/rpm/BUILD/qfs-1.1.4/src/cc/access
gcc -pthread -fno-strict-aliasing -fmessage-length=0 -grecord-gcc-switches -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables -fasynchronous-unwind-tables -g -DNDEBUG -fmessage-length=0 -grecord-gcc-switches -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables -fasynchronous-unwind-tables -g -DOPENSSL_LOAD_CONF -fPIC -I/ws/build/qfs/rpm/BUILD/qfs-1.1.4/src/cc -I/usr/include/python2.7 -c /ws/build/qfs/rpm/BUILD/qfs-1.1.4/src/cc/access/kfs_module_py.cc -o build/temp.linux-x86_64-2.7/ws/build/qfs/rpm/BUILD/qfs-1.1.4/src/cc/access/kfs_module_py.o
creating build/lib.linux-x86_64-2.7
g++ -pthread -shared build/temp.linux-x86_64-2.7/ws/build/qfs/rpm/BUILD/qfs-1.1.4/src/cc/access/kfs_module_py.o -Lsrc/cc/libclient -Lsrc/cc/common -Lsrc/cc/kfsio -Lsrc/cc/qcdio -Lsrc/cc/qcrs -L/usr/lib64 -lqfs_client -lqfs_common -lqfs_io -lqfs_qcdio -lqfs_qcrs -lboost_regex -lpython2.7 -o build/lib.linux-x86_64-2.7/qfs.so
/usr/lib64/gcc/x86_64-suse-linux/4.8/../../../../x86_64-suse-linux/bin/ld: cannot find -lqfs_client
/usr/lib64/gcc/x86_64-suse-linux/4.8/../../../../x86_64-suse-linux/bin/ld: cannot find -lqfs_common
/usr/lib64/gcc/x86_64-suse-linux/4.8/../../../../x86_64-suse-linux/bin/ld: cannot find -lqfs_io
/usr/lib64/gcc/x86_64-suse-linux/4.8/../../../../x86_64-suse-linux/bin/ld: cannot find -lqfs_qcdio
/usr/lib64/gcc/x86_64-suse-linux/4.8/../../../../x86_64-suse-linux/bin/ld: cannot find -lqfs_qcrs
collect2: error: ld returned 1 exit status
error: command 'g++' failed with exit status 1

          Could you please handle at least the ubuntu/debian issue?

          Show
          oflebbe Olaf Flebbe added a comment - The debian and ubuntu packages do not compile because of the unmet build requirement "maven" , since we do not install the "maven" package from debian/ubuntu (for a purpose) . opensuse fails at running build_ext building 'qfs' extension creating build creating build/temp.linux-x86_64-2.7 creating build/temp.linux-x86_64-2.7/ws creating build/temp.linux-x86_64-2.7/ws/build creating build/temp.linux-x86_64-2.7/ws/build/qfs creating build/temp.linux-x86_64-2.7/ws/build/qfs/rpm creating build/temp.linux-x86_64-2.7/ws/build/qfs/rpm/BUILD creating build/temp.linux-x86_64-2.7/ws/build/qfs/rpm/BUILD/qfs-1.1.4 creating build/temp.linux-x86_64-2.7/ws/build/qfs/rpm/BUILD/qfs-1.1.4/src creating build/temp.linux-x86_64-2.7/ws/build/qfs/rpm/BUILD/qfs-1.1.4/src/cc creating build/temp.linux-x86_64-2.7/ws/build/qfs/rpm/BUILD/qfs-1.1.4/src/cc/access gcc -pthread -fno-strict-aliasing -fmessage-length=0 -grecord-gcc-switches -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables -fasynchronous-unwind-tables -g -DNDEBUG -fmessage-length=0 -grecord-gcc-switches -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables -fasynchronous-unwind-tables -g -DOPENSSL_LOAD_CONF -fPIC -I/ws/build/qfs/rpm/BUILD/qfs-1.1.4/src/cc -I/usr/include/python2.7 -c /ws/build/qfs/rpm/BUILD/qfs-1.1.4/src/cc/access/kfs_module_py.cc -o build/temp.linux-x86_64-2.7/ws/build/qfs/rpm/BUILD/qfs-1.1.4/src/cc/access/kfs_module_py.o creating build/lib.linux-x86_64-2.7 g++ -pthread -shared build/temp.linux-x86_64-2.7/ws/build/qfs/rpm/BUILD/qfs-1.1.4/src/cc/access/kfs_module_py.o -Lsrc/cc/libclient -Lsrc/cc/common -Lsrc/cc/kfsio -Lsrc/cc/qcdio -Lsrc/cc/qcrs -L/usr/lib64 -lqfs_client -lqfs_common -lqfs_io -lqfs_qcdio -lqfs_qcrs -lboost_regex -lpython2.7 -o build/lib.linux-x86_64-2.7/qfs.so /usr/lib64/gcc/x86_64-suse-linux/4.8/../../../../x86_64-suse-linux/bin/ld: cannot find -lqfs_client /usr/lib64/gcc/x86_64-suse-linux/4.8/../../../../x86_64-suse-linux/bin/ld: cannot find -lqfs_common /usr/lib64/gcc/x86_64-suse-linux/4.8/../../../../x86_64-suse-linux/bin/ld: cannot find -lqfs_io /usr/lib64/gcc/x86_64-suse-linux/4.8/../../../../x86_64-suse-linux/bin/ld: cannot find -lqfs_qcdio /usr/lib64/gcc/x86_64-suse-linux/4.8/../../../../x86_64-suse-linux/bin/ld: cannot find -lqfs_qcrs collect2: error: ld returned 1 exit status error: command 'g++' failed with exit status 1 Could you please handle at least the ubuntu/debian issue?
          Hide
          Permalink
          fsareshwala Faraaz Sareshwala added a comment -

          Thanks for testing this Olaf. It looks like the vm I was using to test the packaging already had maven installed. I've fixed up the code and tested in using the bigtop docker provisioner to make sure it all works. I've also updated the pull request with the latest patch.

          I tried to set up opensuse under docker to test this myself but I couldn't get things quite working. I did manage to test it under centos and ubuntu so I believe that things should be building correctly now for both build types. Could you take a look again?

          Show
          fsareshwala Faraaz Sareshwala added a comment - Thanks for testing this Olaf. It looks like the vm I was using to test the packaging already had maven installed. I've fixed up the code and tested in using the bigtop docker provisioner to make sure it all works. I've also updated the pull request with the latest patch. I tried to set up opensuse under docker to test this myself but I couldn't get things quite working. I did manage to test it under centos and ubuntu so I believe that things should be building correctly now for both build types. Could you take a look again?
          Hide
          Permalink
          oflebbe Olaf Flebbe added a comment -

          opensuse compile fails because krb5-config ist not found. You have to manually add /usr/lib/mit/bin/ into your PATH. That's one of the peculiarities of SuSE distributions.

          You should be able to download a suitable machine (w/o your additions) with 

          docker pull bigtop/slaves:trunk-opensuse-13.2
          Show
          oflebbe Olaf Flebbe added a comment - opensuse compile fails because krb5-config ist not found. You have to manually add /usr/lib/mit/bin/ into your PATH. That's one of the peculiarities of SuSE distributions. You should be able to download a suitable machine (w/o your additions) with docker pull bigtop/slaves:trunk-opensuse-13.2
          Hide
          Permalink
          oflebbe Olaf Flebbe added a comment - - edited

          At least it builds now on debian, ubuntu, centos-7. (didn't try centos-6 and fedora right now)

          There are a couple of lintian errors (debian). I like to get Errors out and number of Warnings reduced. (forget about the force-reload Error)

          W: qfs-python: missing-depends-line
E: qfs-metaserver: dir-or-file-in-var-run var/run/qfs/
E: qfs-metaserver: postrm-does-not-call-updaterc.d-for-init.d-script etc/init.d/qfs-metaserver
E: qfs-metaserver: postrm-contains-additional-updaterc.d-calls etc/init.d/qfs-chunkserver
E: qfs-metaserver: init.d-script-does-not-implement-required-option etc/init.d/qfs-metaserver force-reload
W: qfs-metaserver: binary-without-manpage usr/bin/filelister
W: qfs-metaserver: binary-without-manpage usr/bin/logcompactor
W: qfs-metaserver: binary-without-manpage usr/bin/metaserver
W: qfs-metaserver: binary-without-manpage usr/bin/qfsfsck
W: qfs-client: hardening-no-relro usr/lib/qfs/libJerasure.so
W: qfs-client: hardening-no-relro usr/lib/qfs/libJerasure.so.2
W: qfs-client: hardening-no-relro usr/lib/qfs/libJerasure.so.2.0.0
W: qfs-client: binary-without-manpage usr/bin/cpfromqfs
W: qfs-client: binary-without-manpage usr/bin/cptoqfs
W: qfs-client: binary-without-manpage usr/bin/qfs
W: qfs-client: binary-without-manpage usr/bin/qfs_backup
W: qfs-client: binary-without-manpage usr/bin/qfs_checkpoint_prune.py
W: qfs-client: binary-without-manpage usr/bin/qfs_log_prune.py
W: qfs-client: binary-without-manpage usr/bin/qfsadmin
W: qfs-client: binary-without-manpage usr/bin/qfscat
W: qfs-client: binary-without-manpage usr/bin/qfsdataverify
W: qfs-client: binary-without-manpage usr/bin/qfsfileenum
W: qfs-client: binary-without-manpage usr/bin/qfsfsck
W: qfs-client: binary-without-manpage usr/bin/qfshibernate
W: qfs-client: binary-without-manpage usr/bin/qfsping
W: qfs-client: binary-without-manpage usr/bin/qfsput
W: qfs-client: binary-without-manpage usr/bin/qfssample
W: qfs-client: binary-without-manpage usr/bin/qfsshell
W: qfs-client: binary-without-manpage usr/bin/qfsstats
W: qfs-client: binary-without-manpage usr/bin/qfstoggleworm
E: qfs-client: python-script-but-no-python-dep usr/bin/qfs_backup
E: qfs-client: python-script-but-no-python-dep usr/bin/qfs_checkpoint_prune.py
E: qfs-client: python-script-but-no-python-dep usr/bin/qfs_log_prune.py
W: qfs-client: postinst-has-useless-call-to-ldconfig
W: qfs-client: postrm-has-useless-call-to-ldconfig
E: qfs-webui: dir-or-file-in-var-run var/run/qfs/
E: qfs-webui: postrm-does-not-call-updaterc.d-for-init.d-script etc/init.d/qfs-webui
E: qfs-webui: postrm-contains-additional-updaterc.d-calls etc/init.d/qfs-chunkserver
E: qfs-webui: init.d-script-does-not-implement-required-option etc/init.d/qfs-webui force-reload
E: qfs-webui: python-script-but-no-python-dep usr/share/qfs/webui/qfsstatus.py
W: qfs-webui: executable-not-elf-or-script usr/share/qfs/webui/files/sorttable/arrow-down.gif
W: qfs-webui: executable-not-elf-or-script usr/share/qfs/webui/files/sorttable/arrow-none.gif
W: qfs-webui: executable-not-elf-or-script usr/share/qfs/webui/files/sorttable/arrow-up.gif
W: qfs-fuse: binary-without-manpage usr/bin/qfs_fuse
E: qfs-chunkserver: dir-or-file-in-var-run var/run/qfs/
E: qfs-chunkserver: init.d-script-does-not-implement-required-option etc/init.d/qfs-chunkserver force-reload
W: qfs-chunkserver: binary-without-manpage usr/bin/chunkscrubber
W: qfs-chunkserver: binary-without-manpage usr/bin/chunkserver
W: qfs: empty-binary-package
W: qfs-dev: binary-without-manpage usr/bin/checksum
W: qfs-dev: binary-without-manpage usr/bin/dirtree_creator
W: qfs-dev: binary-without-manpage usr/bin/dtokentest
W: qfs-dev: binary-without-manpage usr/bin/qfslogger
W: qfs-dev: binary-without-manpage usr/bin/rand-sfmt
W: qfs-dev: binary-without-manpage usr/bin/rebalanceexecutor
W: qfs-dev: binary-without-manpage usr/bin/rebalanceplanner
W: qfs-dev: binary-without-manpage usr/bin/replicachecker
W: qfs-dev: binary-without-manpage usr/bin/requestparser
W: qfs-dev: binary-without-manpage usr/bin/sortedhash
W: qfs-dev: binary-without-manpage usr/bin/sslfiltertest
W: qfs-dev: binary-without-manpage usr/bin/stlset
          Show
          oflebbe Olaf Flebbe added a comment - - edited At least it builds now on debian, ubuntu, centos-7. (didn't try centos-6 and fedora right now) There are a couple of lintian errors (debian). I like to get Errors out and number of Warnings reduced. (forget about the force-reload Error) W: qfs-python: missing-depends-line E: qfs-metaserver: dir-or-file-in- var -run var /run/qfs/ E: qfs-metaserver: postrm-does-not-call-updaterc.d- for -init.d-script etc/init.d/qfs-metaserver E: qfs-metaserver: postrm-contains-additional-updaterc.d-calls etc/init.d/qfs-chunkserver E: qfs-metaserver: init.d-script-does-not-implement-required-option etc/init.d/qfs-metaserver force-reload W: qfs-metaserver: binary-without-manpage usr/bin/filelister W: qfs-metaserver: binary-without-manpage usr/bin/logcompactor W: qfs-metaserver: binary-without-manpage usr/bin/metaserver W: qfs-metaserver: binary-without-manpage usr/bin/qfsfsck W: qfs-client: hardening-no-relro usr/lib/qfs/libJerasure.so W: qfs-client: hardening-no-relro usr/lib/qfs/libJerasure.so.2 W: qfs-client: hardening-no-relro usr/lib/qfs/libJerasure.so.2.0.0 W: qfs-client: binary-without-manpage usr/bin/cpfromqfs W: qfs-client: binary-without-manpage usr/bin/cptoqfs W: qfs-client: binary-without-manpage usr/bin/qfs W: qfs-client: binary-without-manpage usr/bin/qfs_backup W: qfs-client: binary-without-manpage usr/bin/qfs_checkpoint_prune.py W: qfs-client: binary-without-manpage usr/bin/qfs_log_prune.py W: qfs-client: binary-without-manpage usr/bin/qfsadmin W: qfs-client: binary-without-manpage usr/bin/qfscat W: qfs-client: binary-without-manpage usr/bin/qfsdataverify W: qfs-client: binary-without-manpage usr/bin/qfsfileenum W: qfs-client: binary-without-manpage usr/bin/qfsfsck W: qfs-client: binary-without-manpage usr/bin/qfshibernate W: qfs-client: binary-without-manpage usr/bin/qfsping W: qfs-client: binary-without-manpage usr/bin/qfsput W: qfs-client: binary-without-manpage usr/bin/qfssample W: qfs-client: binary-without-manpage usr/bin/qfsshell W: qfs-client: binary-without-manpage usr/bin/qfsstats W: qfs-client: binary-without-manpage usr/bin/qfstoggleworm E: qfs-client: python-script-but-no-python-dep usr/bin/qfs_backup E: qfs-client: python-script-but-no-python-dep usr/bin/qfs_checkpoint_prune.py E: qfs-client: python-script-but-no-python-dep usr/bin/qfs_log_prune.py W: qfs-client: postinst-has-useless-call-to-ldconfig W: qfs-client: postrm-has-useless-call-to-ldconfig E: qfs-webui: dir-or-file-in- var -run var /run/qfs/ E: qfs-webui: postrm-does-not-call-updaterc.d- for -init.d-script etc/init.d/qfs-webui E: qfs-webui: postrm-contains-additional-updaterc.d-calls etc/init.d/qfs-chunkserver E: qfs-webui: init.d-script-does-not-implement-required-option etc/init.d/qfs-webui force-reload E: qfs-webui: python-script-but-no-python-dep usr/share/qfs/webui/qfsstatus.py W: qfs-webui: executable-not-elf-or-script usr/share/qfs/webui/files/sorttable/arrow-down.gif W: qfs-webui: executable-not-elf-or-script usr/share/qfs/webui/files/sorttable/arrow-none.gif W: qfs-webui: executable-not-elf-or-script usr/share/qfs/webui/files/sorttable/arrow-up.gif W: qfs-fuse: binary-without-manpage usr/bin/qfs_fuse E: qfs-chunkserver: dir-or-file-in- var -run var /run/qfs/ E: qfs-chunkserver: init.d-script-does-not-implement-required-option etc/init.d/qfs-chunkserver force-reload W: qfs-chunkserver: binary-without-manpage usr/bin/chunkscrubber W: qfs-chunkserver: binary-without-manpage usr/bin/chunkserver W: qfs: empty-binary- package W: qfs-dev: binary-without-manpage usr/bin/checksum W: qfs-dev: binary-without-manpage usr/bin/dirtree_creator W: qfs-dev: binary-without-manpage usr/bin/dtokentest W: qfs-dev: binary-without-manpage usr/bin/qfslogger W: qfs-dev: binary-without-manpage usr/bin/rand-sfmt W: qfs-dev: binary-without-manpage usr/bin/rebalanceexecutor W: qfs-dev: binary-without-manpage usr/bin/rebalanceplanner W: qfs-dev: binary-without-manpage usr/bin/replicachecker W: qfs-dev: binary-without-manpage usr/bin/requestparser W: qfs-dev: binary-without-manpage usr/bin/sortedhash W: qfs-dev: binary-without-manpage usr/bin/sslfiltertest W: qfs-dev: binary-without-manpage usr/bin/stlset
          Hide
          Permalink
          cos Konstantin Boudnik added a comment -

          I think we can safely skip centos-6, as I believe we have decided not to support it anymore.

          Show
          cos Konstantin Boudnik added a comment - I think we can safely skip centos-6, as I believe we have decided not to support it anymore.
          Hide
          Permalink
          cos Konstantin Boudnik added a comment -

          Please reduce the size of the commit message. git log isn't to replace JIRA Thanks!

          Show
          cos Konstantin Boudnik added a comment - Please reduce the size of the commit message. git log isn't to replace JIRA Thanks!
          Hide
          Permalink
          fsareshwala Faraaz Sareshwala added a comment -

          Thanks Olaf. I've updated the code on the github pull request to eliminate all errors and have reduced the warnings by quite a bit. I've set a few warnings and errors as overridden and ignored (e.g. binary-without-manpage). Is that okay?

          Show
          fsareshwala Faraaz Sareshwala added a comment - Thanks Olaf. I've updated the code on the github pull request to eliminate all errors and have reduced the warnings by quite a bit. I've set a few warnings and errors as overridden and ignored (e.g. binary-without-manpage). Is that okay?
          Hide
          Permalink
          fsareshwala Faraaz Sareshwala added a comment -

          Thanks for the reviews guys. Really appreciate it.

          Does anyone have any comments on the packaging structure? Are files where they should be to conform to the bigtop standards?

          Show
          fsareshwala Faraaz Sareshwala added a comment - Thanks for the reviews guys. Really appreciate it. Does anyone have any comments on the packaging structure? Are files where they should be to conform to the bigtop standards?
          Hide
          Permalink
          oflebbe Olaf Flebbe added a comment -

          I looked again at the debian package.

          Building is o.k. But I do not agree reducing the errors by overriding lintian defaults. My intention was that you fix the packages instead. If something has to be fixed upstream, o.k. But please remove the lintian overrides.

          Installation errors:
          There are conflicts between qfs-client and qfs-metaserver . Both own qfsfsck.

          Installing qfs-metaserver has errors while installing

          root@e534e7776385:/qfs# dpkg -i qfs-metaserver_1.1.4-1_amd64.deb 
(Reading database ... 14261 files and directories currently installed.)
Preparing to unpack qfs-metaserver_1.1.4-1_amd64.deb ...
Unpacking qfs-metaserver (1.1.4-1) ...
Setting up qfs-metaserver (1.1.4-1) ...
chown: cannot access '/var/run/qfs': No such file or directory
dpkg: error processing package qfs-metaserver (--install):
 subprocess installed post-installation script returned error exit status 1
Processing triggers for systemd (215-17+deb8u2) ...
Errors were encountered while processing:
 qfs-metaserver
root@e534e7776385:/qfs#  cannot access '/var/run/qfs': No such file or directory^C

          This seems to be because /var/run aka /run is a temporary dir, you have to create directories in your start script.

          Show
          oflebbe Olaf Flebbe added a comment - I looked again at the debian package. Building is o.k. But I do not agree reducing the errors by overriding lintian defaults. My intention was that you fix the packages instead. If something has to be fixed upstream, o.k. But please remove the lintian overrides. Installation errors: There are conflicts between qfs-client and qfs-metaserver . Both own qfsfsck. Installing qfs-metaserver has errors while installing root@e534e7776385:/qfs# dpkg -i qfs-metaserver_1.1.4-1_amd64.deb (Reading database ... 14261 files and directories currently installed.) Preparing to unpack qfs-metaserver_1.1.4-1_amd64.deb ... Unpacking qfs-metaserver (1.1.4-1) ... Setting up qfs-metaserver (1.1.4-1) ... chown: cannot access '/ var /run/qfs': No such file or directory dpkg: error processing package qfs-metaserver (--install): subprocess installed post-installation script returned error exit status 1 Processing triggers for systemd (215-17+deb8u2) ... Errors were encountered while processing: qfs-metaserver root@e534e7776385:/qfs# cannot access '/ var /run/qfs': No such file or directory^C This seems to be because /var/run aka /run is a temporary dir, you have to create directories in your start script.
          Hide
          Permalink
          fsareshwala Faraaz Sareshwala added a comment -

          Thanks for taking another look Olaf. I've updated the code to fix all the issues you mentioned. I'm still ignoring the binary-without-manpage lintian warning since I am not able to provide manpages at the moment. I will do that in a future release. I took care of various other errors so the lintian overrides are significantly less now. Some are proper to ignore (e.g. new-package-should-close-itp-bug).

          Please let me know what you think about the latest code.

          Show
          fsareshwala Faraaz Sareshwala added a comment - Thanks for taking another look Olaf. I've updated the code to fix all the issues you mentioned. I'm still ignoring the binary-without-manpage lintian warning since I am not able to provide manpages at the moment. I will do that in a future release. I took care of various other errors so the lintian overrides are significantly less now. Some are proper to ignore (e.g. new-package-should-close-itp-bug). Please let me know what you think about the latest code.
          Hide
          Permalink
          githubbot ASF GitHub Bot added a comment -

          Github user youngwookim commented on a diff in the pull request:

          https://github.com/apache/bigtop/pull/83#discussion_r54212291

          — Diff: bigtop-packages/src/common/qfs/do-component-build —
          @@ -0,0 +1,133 @@
          +#!/bin/bash
          +
          +# Licensed to the Apache Software Foundation (ASF) under one or more
          +# contributor license agreements. See the NOTICE file distributed with
          +# this work for additional information regarding copyright ownership.
          +# The ASF licenses this file to You under the Apache License, Version 2.0
          +# (the "License"); you may not use this file except in compliance with
          +# the License. You may obtain a copy of the License at
          +#
          +# http://www.apache.org/licenses/LICENSE-2.0
          +#
          +# Unless required by applicable law or agreed to in writing, software
          +# distributed under the License is distributed on an "AS IS" BASIS,
          +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          +# See the License for the specific language governing permissions and
          +# limitations under the License.
          +
          +# For debugging:
          +# set -ex
          +
          +# Autodetect JAVA_HOME if not defined
          +source /usr/lib/bigtop-utils/bigtop-detect-javahome
          — End diff –

          I believe JAVA_HOME detection is not necessary at build stage because we have bigtop toolchain. So, please remove this bits.

          Show
          githubbot ASF GitHub Bot added a comment - Github user youngwookim commented on a diff in the pull request: https://github.com/apache/bigtop/pull/83#discussion_r54212291 — Diff: bigtop-packages/src/common/qfs/do-component-build — @@ -0,0 +1,133 @@ +#!/bin/bash + +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# For debugging: +# set -ex + +# Autodetect JAVA_HOME if not defined +source /usr/lib/bigtop-utils/bigtop-detect-javahome — End diff – I believe JAVA_HOME detection is not necessary at build stage because we have bigtop toolchain. So, please remove this bits.
          Hide
          Permalink
          githubbot ASF GitHub Bot added a comment -

          Github user youngwookim commented on a diff in the pull request:

          https://github.com/apache/bigtop/pull/83#discussion_r54212818

          — Diff: bigtop-packages/src/deb/qfs/rules —
          @@ -0,0 +1,81 @@
          +#!/usr/bin/make -f
          +
          +# Licensed to the Apache Software Foundation (ASF) under one or more
          +# contributor license agreements. See the NOTICE file distributed with
          +# this work for additional information regarding copyright ownership.
          +# The ASF licenses this file to You under the Apache License, Version 2.0
          +# (the "License"); you may not use this file except in compliance with
          +# the License. You may obtain a copy of the License at
          +#
          +# http://www.apache.org/licenses/LICENSE-2.0
          +#
          +# Unless required by applicable law or agreed to in writing, software
          +# distributed under the License is distributed on an "AS IS" BASIS,
          +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          +# See the License for the specific language governing permissions and
          +# limitations under the License.
          +#
          +# makefile
          +
          +# Uncomment this to turn on verbose mode.
          +export DH_VERBOSE=1
          +
          +# This has to be exported to make some magic below work.
          +export DH_OPTIONS
          +
          +# Build configuration – override with environment variables
          +SHELL := /bin/bash
          +HADOOP_VERSION ?= 1.0.4
          — End diff –

          Bigtop defines HADOOP_VERSION in bom so, just pass the variable $

          {HADOOP_VERSION}

          or $

          {HADOOP_BASE_VERSION}

          . ref. https://github.com/apache/bigtop/blob/master/bigtop-packages/src/deb/zookeeper/rules#L39

          Show
          githubbot ASF GitHub Bot added a comment - Github user youngwookim commented on a diff in the pull request: https://github.com/apache/bigtop/pull/83#discussion_r54212818 — Diff: bigtop-packages/src/deb/qfs/rules — @@ -0,0 +1,81 @@ +#!/usr/bin/make -f + +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# makefile + +# Uncomment this to turn on verbose mode. +export DH_VERBOSE=1 + +# This has to be exported to make some magic below work. +export DH_OPTIONS + +# Build configuration – override with environment variables +SHELL := /bin/bash +HADOOP_VERSION ?= 1.0.4 — End diff – Bigtop defines HADOOP_VERSION in bom so, just pass the variable $ {HADOOP_VERSION} or $ {HADOOP_BASE_VERSION} . ref. https://github.com/apache/bigtop/blob/master/bigtop-packages/src/deb/zookeeper/rules#L39
          Hide
          Permalink
          githubbot ASF GitHub Bot added a comment -

          Github user youngwookim commented on a diff in the pull request:

          https://github.com/apache/bigtop/pull/83#discussion_r54212968

          — Diff: bigtop-packages/src/rpm/qfs/SPECS/qfs.spec —
          @@ -0,0 +1,360 @@
          +# Licensed to the Apache Software Foundation (ASF) under one or more
          +# contributor license agreements. See the NOTICE file distributed with
          +# this work for additional information regarding copyright ownership.
          +# The ASF licenses this file to You under the Apache License, Version 2.0
          +# (the "License"); you may not use this file except in compliance with
          +# the License. You may obtain a copy of the License at
          +#
          +# http://www.apache.org/licenses/LICENSE-2.0
          +#
          +# Unless required by applicable law or agreed to in writing, software
          +# distributed under the License is distributed on an "AS IS" BASIS,
          +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          +# See the License for the specific language governing permissions and
          +# limitations under the License.
          +
          +%define qfs_name qfs
          +
          +%define bin_dir /usr/bin
          +%define lib_dir /usr/lib/qfs
          +%define etc_dir %{_sysconfdir}/qfs
          +%define include_dir /usr/include
          +%define data_dir /usr/share/qfs
          +%define var_dir /var
          +%define qfs_services chunkserver metaserver webui
          +
          +%global HADOOP_VERSION %

          {hadoop_version}

          +
          +%if %{?!HADOOP_VERSION:1}0
          — End diff –

          Would be good to have default Hadoop version from bigtop bom. Bigtop does not support Hadoop 1.x at this point.

          Show
          githubbot ASF GitHub Bot added a comment - Github user youngwookim commented on a diff in the pull request: https://github.com/apache/bigtop/pull/83#discussion_r54212968 — Diff: bigtop-packages/src/rpm/qfs/SPECS/qfs.spec — @@ -0,0 +1,360 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +%define qfs_name qfs + +%define bin_dir /usr/bin +%define lib_dir /usr/lib/qfs +%define etc_dir %{_sysconfdir}/qfs +%define include_dir /usr/include +%define data_dir /usr/share/qfs +%define var_dir /var +%define qfs_services chunkserver metaserver webui + +%global HADOOP_VERSION % {hadoop_version} + +%if %{?!HADOOP_VERSION:1}0 — End diff – Would be good to have default Hadoop version from bigtop bom. Bigtop does not support Hadoop 1.x at this point.
          Hide
          Permalink
          cos Konstantin Boudnik added a comment -

          It does support 1.0, but in 0.4.0 we moved to Hadoop 2.x, so 0.3.x line is for Hadoop 1.x. Which is long time abandoned. Besides, is there anyone who still cares about Hadoop 1.x?

          Show
          cos Konstantin Boudnik added a comment - It does support 1.0, but in 0.4.0 we moved to Hadoop 2.x, so 0.3.x line is for Hadoop 1.x. Which is long time abandoned. Besides, is there anyone who still cares about Hadoop 1.x?
          Hide
          Permalink
          githubbot ASF GitHub Bot added a comment -

          Github user fsareshwala commented on a diff in the pull request:

          https://github.com/apache/bigtop/pull/83#discussion_r54285663

          — Diff: bigtop-packages/src/common/qfs/do-component-build —
          @@ -0,0 +1,133 @@
          +#!/bin/bash
          +
          +# Licensed to the Apache Software Foundation (ASF) under one or more
          +# contributor license agreements. See the NOTICE file distributed with
          +# this work for additional information regarding copyright ownership.
          +# The ASF licenses this file to You under the Apache License, Version 2.0
          +# (the "License"); you may not use this file except in compliance with
          +# the License. You may obtain a copy of the License at
          +#
          +# http://www.apache.org/licenses/LICENSE-2.0
          +#
          +# Unless required by applicable law or agreed to in writing, software
          +# distributed under the License is distributed on an "AS IS" BASIS,
          +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          +# See the License for the specific language governing permissions and
          +# limitations under the License.
          +
          +# For debugging:
          +# set -ex
          +
          +# Autodetect JAVA_HOME if not defined
          +source /usr/lib/bigtop-utils/bigtop-detect-javahome
          — End diff –

          If I remove this, then JAVA_HOME isn't passed on through the makefile for qfs and we can't locate jni and build the jni bindings. Should this not be the case?

          Show
          githubbot ASF GitHub Bot added a comment - Github user fsareshwala commented on a diff in the pull request: https://github.com/apache/bigtop/pull/83#discussion_r54285663 — Diff: bigtop-packages/src/common/qfs/do-component-build — @@ -0,0 +1,133 @@ +#!/bin/bash + +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# For debugging: +# set -ex + +# Autodetect JAVA_HOME if not defined +source /usr/lib/bigtop-utils/bigtop-detect-javahome — End diff – If I remove this, then JAVA_HOME isn't passed on through the makefile for qfs and we can't locate jni and build the jni bindings. Should this not be the case?
          Hide
          Permalink
          githubbot ASF GitHub Bot added a comment -

          Github user fsareshwala commented on a diff in the pull request:

          https://github.com/apache/bigtop/pull/83#discussion_r54285917

          — Diff: bigtop-packages/src/deb/qfs/rules —
          @@ -0,0 +1,81 @@
          +#!/usr/bin/make -f
          +
          +# Licensed to the Apache Software Foundation (ASF) under one or more
          +# contributor license agreements. See the NOTICE file distributed with
          +# this work for additional information regarding copyright ownership.
          +# The ASF licenses this file to You under the Apache License, Version 2.0
          +# (the "License"); you may not use this file except in compliance with
          +# the License. You may obtain a copy of the License at
          +#
          +# http://www.apache.org/licenses/LICENSE-2.0
          +#
          +# Unless required by applicable law or agreed to in writing, software
          +# distributed under the License is distributed on an "AS IS" BASIS,
          +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          +# See the License for the specific language governing permissions and
          +# limitations under the License.
          +#
          +# makefile
          +
          +# Uncomment this to turn on verbose mode.
          +export DH_VERBOSE=1
          +
          +# This has to be exported to make some magic below work.
          +export DH_OPTIONS
          +
          +# Build configuration – override with environment variables
          +SHELL := /bin/bash
          +HADOOP_VERSION ?= 1.0.4
          — End diff –

          We do pass it through in the packages.gradle. This is just a default in case it isn't passed through for some reason.

          Show
          githubbot ASF GitHub Bot added a comment - Github user fsareshwala commented on a diff in the pull request: https://github.com/apache/bigtop/pull/83#discussion_r54285917 — Diff: bigtop-packages/src/deb/qfs/rules — @@ -0,0 +1,81 @@ +#!/usr/bin/make -f + +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# makefile + +# Uncomment this to turn on verbose mode. +export DH_VERBOSE=1 + +# This has to be exported to make some magic below work. +export DH_OPTIONS + +# Build configuration – override with environment variables +SHELL := /bin/bash +HADOOP_VERSION ?= 1.0.4 — End diff – We do pass it through in the packages.gradle. This is just a default in case it isn't passed through for some reason.
          Hide
          Permalink
          githubbot ASF GitHub Bot added a comment -

          Github user fsareshwala commented on a diff in the pull request:

          https://github.com/apache/bigtop/pull/83#discussion_r54286010

          — Diff: bigtop-packages/src/rpm/qfs/SPECS/qfs.spec —
          @@ -0,0 +1,360 @@
          +# Licensed to the Apache Software Foundation (ASF) under one or more
          +# contributor license agreements. See the NOTICE file distributed with
          +# this work for additional information regarding copyright ownership.
          +# The ASF licenses this file to You under the Apache License, Version 2.0
          +# (the "License"); you may not use this file except in compliance with
          +# the License. You may obtain a copy of the License at
          +#
          +# http://www.apache.org/licenses/LICENSE-2.0
          +#
          +# Unless required by applicable law or agreed to in writing, software
          +# distributed under the License is distributed on an "AS IS" BASIS,
          +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          +# See the License for the specific language governing permissions and
          +# limitations under the License.
          +
          +%define qfs_name qfs
          +
          +%define bin_dir /usr/bin
          +%define lib_dir /usr/lib/qfs
          +%define etc_dir %{_sysconfdir}/qfs
          +%define include_dir /usr/include
          +%define data_dir /usr/share/qfs
          +%define var_dir /var
          +%define qfs_services chunkserver metaserver webui
          +
          +%global HADOOP_VERSION %

          {hadoop_version}

          +
          +%if %{?!HADOOP_VERSION:1}0
          — End diff –

          We do pass it through from the bigtop bom. See my changes to packages.gradle to see how this is handled and flows through.

          Show
          githubbot ASF GitHub Bot added a comment - Github user fsareshwala commented on a diff in the pull request: https://github.com/apache/bigtop/pull/83#discussion_r54286010 — Diff: bigtop-packages/src/rpm/qfs/SPECS/qfs.spec — @@ -0,0 +1,360 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +%define qfs_name qfs + +%define bin_dir /usr/bin +%define lib_dir /usr/lib/qfs +%define etc_dir %{_sysconfdir}/qfs +%define include_dir /usr/include +%define data_dir /usr/share/qfs +%define var_dir /var +%define qfs_services chunkserver metaserver webui + +%global HADOOP_VERSION % {hadoop_version} + +%if %{?!HADOOP_VERSION:1}0 — End diff – We do pass it through from the bigtop bom. See my changes to packages.gradle to see how this is handled and flows through.
          Hide
          Permalink
          fsareshwala Faraaz Sareshwala added a comment -

          Sorry Olaf. I just realized that the master branch hadn't been updated with the latest code. Just the branches that were the sources of the pull request. I just updated the master branch so it should be good to review now. Sorry about that!

          Show
          fsareshwala Faraaz Sareshwala added a comment - Sorry Olaf. I just realized that the master branch hadn't been updated with the latest code. Just the branches that were the sources of the pull request. I just updated the master branch so it should be good to review now. Sorry about that!
          Hide
          Permalink
          oflebbe Olaf Flebbe added a comment -

          There are a couple of fine points which can be solved later on, I think.

          Thank you very much for your contribution!

          Will commit current #83 to bigtop .

          Show
          oflebbe Olaf Flebbe added a comment - There are a couple of fine points which can be solved later on, I think. Thank you very much for your contribution! Will commit current #83 to bigtop .
          Hide
          Permalink
          oflebbe Olaf Flebbe added a comment -

          committed

          Show
          oflebbe Olaf Flebbe added a comment - committed
          Hide
          Permalink
          githubbot ASF GitHub Bot added a comment -

          Github user c0s commented on a diff in the pull request:

          https://github.com/apache/bigtop/pull/83#discussion_r55145881

          — Diff: bigtop-packages/src/rpm/qfs/SPECS/qfs.spec —
          @@ -0,0 +1,360 @@
          +# Licensed to the Apache Software Foundation (ASF) under one or more
          +# contributor license agreements. See the NOTICE file distributed with
          +# this work for additional information regarding copyright ownership.
          +# The ASF licenses this file to You under the Apache License, Version 2.0
          +# (the "License"); you may not use this file except in compliance with
          +# the License. You may obtain a copy of the License at
          +#
          +# http://www.apache.org/licenses/LICENSE-2.0
          +#
          +# Unless required by applicable law or agreed to in writing, software
          +# distributed under the License is distributed on an "AS IS" BASIS,
          +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          +# See the License for the specific language governing permissions and
          +# limitations under the License.
          +
          +%define qfs_name qfs
          +
          +%define bin_dir /usr/bin
          +%define lib_dir /usr/lib/qfs
          +%define etc_dir %{_sysconfdir}/qfs
          +%define include_dir /usr/include
          +%define data_dir /usr/share/qfs
          +%define var_dir /var
          +%define qfs_services chunkserver metaserver webui
          +
          +%global HADOOP_VERSION %

          {hadoop_version}

          +
          +%if %{?!HADOOP_VERSION:1}0
          — End diff –

          Hmm, my bad then. I though the up-level build.gradle will enforce it. I will fix it later, for now just disregard this comment. Thanks

          Show
          githubbot ASF GitHub Bot added a comment - Github user c0s commented on a diff in the pull request: https://github.com/apache/bigtop/pull/83#discussion_r55145881 — Diff: bigtop-packages/src/rpm/qfs/SPECS/qfs.spec — @@ -0,0 +1,360 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +%define qfs_name qfs + +%define bin_dir /usr/bin +%define lib_dir /usr/lib/qfs +%define etc_dir %{_sysconfdir}/qfs +%define include_dir /usr/include +%define data_dir /usr/share/qfs +%define var_dir /var +%define qfs_services chunkserver metaserver webui + +%global HADOOP_VERSION % {hadoop_version} + +%if %{?!HADOOP_VERSION:1}0 — End diff – Hmm, my bad then. I though the up-level build.gradle will enforce it. I will fix it later, for now just disregard this comment. Thanks
          Hide
          Permalink
          githubbot ASF GitHub Bot added a comment -

          Github user fsareshwala commented on the pull request:

          https://github.com/apache/bigtop/pull/83#issuecomment-192962740

          Olaf committed this code into bigtop: https://issues.apache.org/jira/browse/BIGTOP-2285

          I am going to close this pull request out.

          Show
          githubbot ASF GitHub Bot added a comment - Github user fsareshwala commented on the pull request: https://github.com/apache/bigtop/pull/83#issuecomment-192962740 Olaf committed this code into bigtop: https://issues.apache.org/jira/browse/BIGTOP-2285 I am going to close this pull request out.
          Hide
          Permalink
          githubbot ASF GitHub Bot added a comment -

          Github user fsareshwala closed the pull request at:

          https://github.com/apache/bigtop/pull/83

          Show
          githubbot ASF GitHub Bot added a comment - Github user fsareshwala closed the pull request at: https://github.com/apache/bigtop/pull/83
          Hide
          Permalink
          jonathak Jonathan Kelly added a comment -

          I'm confused about why HADOOP_VERSION is explicitly passed through to qfs's do-component-build and install_qfs.sh scripts, when the bigtop.bom already provides it via the HADOOP_VERSION environment variable? For instance, note how the spark's do-component-build and install_spark.sh scripts reference $HADOOP_VERSION without having to pass it via the command line args. Also, you are setting HADOOP_VERSION to the Hadoop package version rather than the base version, which is normally OK (since they are usually the same value), but it can cause a problem if they are different.

          Show
          jonathak Jonathan Kelly added a comment - I'm confused about why HADOOP_VERSION is explicitly passed through to qfs's do-component-build and install_qfs.sh scripts, when the bigtop.bom already provides it via the HADOOP_VERSION environment variable? For instance, note how the spark's do-component-build and install_spark.sh scripts reference $HADOOP_VERSION without having to pass it via the command line args. Also, you are setting HADOOP_VERSION to the Hadoop package version rather than the base version, which is normally OK (since they are usually the same value), but it can cause a problem if they are different.
          Hide
          Permalink
          jonathak Jonathan Kelly added a comment -

          Oh, I see. It's because you are not sourcing the bigtop.bom file in qfs's do-component-build or install_qfs.sh scripts, as is done for other apps like Spark.

          Show
          jonathak Jonathan Kelly added a comment - Oh, I see. It's because you are not sourcing the bigtop.bom file in qfs's do-component-build or install_qfs.sh scripts, as is done for other apps like Spark.
          Hide
          Permalink
          fsareshwala Faraaz Sareshwala added a comment -

          Thanks for the note! Should I be doing that?

          Show
          fsareshwala Faraaz Sareshwala added a comment - Thanks for the note! Should I be doing that?
          Hide
          Permalink
          fsareshwala Faraaz Sareshwala added a comment -

          I've created BIGTOP-2364 to track this. Patch coming up.

          Show
          fsareshwala Faraaz Sareshwala added a comment - I've created BIGTOP-2364 to track this. Patch coming up.
          Hide
          Permalink
          jonathak Jonathan Kelly added a comment -

          Thanks a lot!

          Show
          jonathak Jonathan Kelly added a comment - Thanks a lot!
          Hide
          Permalink
          fsareshwala Faraaz Sareshwala added a comment -

          Jonathan Kelly I've submitted a patch (https://github.com/apache/bigtop/pull/98) to fix this. Thanks!

          Show
          fsareshwala Faraaz Sareshwala added a comment - Jonathan Kelly I've submitted a patch ( https://github.com/apache/bigtop/pull/98 ) to fix this. Thanks!

            People

            • Assignee:
              fsareshwala Faraaz Sareshwala
              Reporter:
              fsareshwala Faraaz Sareshwala
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development