Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.4.0
    • Fix Version/s: 0.4.0
    • Component/s: None
    • Labels:
      None

      Description

      httpfs is a new daemon that has been recently introduced in Hadoop. We need to provide packaging infrastructure for it.

      1. BIGTOP-333-final.patch.txt
        2 kB
        Roman Shaposhnik
      2. BIGTOP-333.patch.txt
        31 kB
        Roman Shaposhnik

        Activity

        Hide
        Roman Shaposhnik added a comment -

        While at it, we can also tackle webhdfs. Well either that, or it could be a separate JIRA

        Show
        Roman Shaposhnik added a comment - While at it, we can also tackle webhdfs. Well either that, or it could be a separate JIRA
        Hide
        Bruno Mahé added a comment -

        In the init script:

        • Use $HTTPFS_USER instead of hardcoding httpfs
        • I understand it's copy/paste from hadoop init script, but I would rather use one of the other init scripts. This is one is costly in maintenance.

        In the spec file:

        • There is a macro to handle pre/post -un
        Show
        Bruno Mahé added a comment - In the init script: Use $HTTPFS_USER instead of hardcoding httpfs I understand it's copy/paste from hadoop init script, but I would rather use one of the other init scripts. This is one is costly in maintenance. In the spec file: There is a macro to handle pre/post -un
        Hide
        Roman Shaposhnik added a comment -

        Which script do you suggest as a model?

        As for the spec file – I can't seem to be able to use the macro since I have to do additional stuff in pre/post for httpfs and RPM complains about multiple pre/post sections (I thought they would be cumulative, but they seem to be not).

        Show
        Roman Shaposhnik added a comment - Which script do you suggest as a model? As for the spec file – I can't seem to be able to use the macro since I have to do additional stuff in pre/post for httpfs and RPM complains about multiple pre/post sections (I thought they would be cumulative, but they seem to be not).
        Show
        Bruno Mahé added a comment - https://github.com/apache/bigtop/blob/hadoop-0.23/bigtop-packages/src/rpm/hadoop/SOURCES/hadoop-init.tmpl.suse ? or https://github.com/apache/bigtop/blob/hadoop-0.23/bigtop-packages/src/common/flume-ng/flume-ng-node.init Regarding the spec file, my bad. I forgot the service macro also sets %pre/post
        Hide
        Roman Shaposhnik added a comment -

        Updated the patch. It is now based on flume-ng-node.init and it also includes more elaborate puppet code for httpfs deployment. Please kindly review.

        Show
        Roman Shaposhnik added a comment - Updated the patch. It is now based on flume-ng-node.init and it also includes more elaborate puppet code for httpfs deployment. Please kindly review.
        Hide
        Bruno Mahé added a comment -

        Thanks a lot!
        +1

        Show
        Bruno Mahé added a comment - Thanks a lot! +1
        Hide
        Bruno Mahé added a comment -

        Roman, I am reopening this issue since it appears to have broken the build.

        See below for the error message:

        13:21:38  Requires(pre): hadoop = 0.23.1.20120204130424-1
        13:21:38  Requires(post): /bin/sh
        13:21:38  Requires(preun): /bin/sh
        13:21:38  Requires(postun): /bin/sh
        13:21:38  Processing files: hadoop-httpfs-0.23.1.20120204130424-1.x86_64
        13:21:38  error: File not found: /home/jenkins/workspace/BigTop-Hadoop-0.23-Fedora/build/hadoop/rpm/BUILDROOT/hadoop-0.23.1.20120204130424-1.x86_64/etc/rc.d/init.d/hadoop-%1
        13:21:38  
        13:21:38  
        13:21:38  RPM build errors:
        13:21:38      File listed twice: /var/lib/hadoop/cache
        13:21:38      File listed twice: /var/lib/hdfs/cache
        13:21:38      File listed twice: /var/lib/yarn/cache
        13:21:38      File listed twice: /var/lib/mapreduce/cache
        13:21:38      File not found: /home/jenkins/workspace/BigTop-Hadoop-0.23-Fedora/build/hadoop/rpm/BUILDROOT/hadoop-0.23.1.20120204130424-1.x86_64/etc/rc.d/init.d/hadoop-%1
        13:21:38  make: *** [/home/jenkins/workspace/BigTop-Hadoop-0.23-Fedora/build/hadoop//.rpm] Error 1
        13:21:38  Build step 'Execute shell' marked build as failure
        13:21:38  [WARNINGS] Skipping publisher since build result is FAILURE
        13:21:38  Archiving artifacts
        13:22:10  Notifying upstream projects of job completion
        13:22:10  Finished: FAILURE
        
        Show
        Bruno Mahé added a comment - Roman, I am reopening this issue since it appears to have broken the build. See below for the error message: 13:21:38 Requires(pre): hadoop = 0.23.1.20120204130424-1 13:21:38 Requires(post): /bin/sh 13:21:38 Requires(preun): /bin/sh 13:21:38 Requires(postun): /bin/sh 13:21:38 Processing files: hadoop-httpfs-0.23.1.20120204130424-1.x86_64 13:21:38 error: File not found: /home/jenkins/workspace/BigTop-Hadoop-0.23-Fedora/build/hadoop/rpm/BUILDROOT/hadoop-0.23.1.20120204130424-1.x86_64/etc/rc.d/init.d/hadoop-%1 13:21:38 13:21:38 13:21:38 RPM build errors: 13:21:38 File listed twice: /var/lib/hadoop/cache 13:21:38 File listed twice: /var/lib/hdfs/cache 13:21:38 File listed twice: /var/lib/yarn/cache 13:21:38 File listed twice: /var/lib/mapreduce/cache 13:21:38 File not found: /home/jenkins/workspace/BigTop-Hadoop-0.23-Fedora/build/hadoop/rpm/BUILDROOT/hadoop-0.23.1.20120204130424-1.x86_64/etc/rc.d/init.d/hadoop-%1 13:21:38 make: *** [/home/jenkins/workspace/BigTop-Hadoop-0.23-Fedora/build/hadoop//.rpm] Error 1 13:21:38 Build step 'Execute shell' marked build as failure 13:21:38 [WARNINGS] Skipping publisher since build result is FAILURE 13:21:38 Archiving artifacts 13:22:10 Notifying upstream projects of job completion 13:22:10 Finished: FAILURE
        Hide
        Roman Shaposhnik added a comment -

        Weird. I tried building it on CentOS5 before submitting. Anyway, apologies for the breakage – I'm looking into this now.

        Show
        Roman Shaposhnik added a comment - Weird. I tried building it on CentOS5 before submitting. Anyway, apologies for the breakage – I'm looking into this now.
        Hide
        Bruno Mahé added a comment -

        Thanks! I noticed your patch. I will try it as soon as it hits github mirror.

        The build seems to have passed without your latest fix on centos 6:
        http://bigtop01.cloudera.org:8080/job/Bigtop-hadoop-0.23-Hadoop/label=centos6/6/console

        But I see this warning message at the end:

        00:06:37  warning: Installed (but unpackaged) file(s) found:
        00:06:37     /etc/hadoop/conf.empty/httpfs-env.sh
        00:06:37     /etc/hadoop/conf.empty/httpfs-log4j.properties
        00:06:37     /etc/hadoop/conf.empty/httpfs-signature.secret
        00:06:37     /etc/hadoop/conf.empty/httpfs-site.xml
        00:06:37     /usr/lib/hadoop/hadoop-distcp-0.23.1-SNAPSHOT.jar
        00:06:37     /usr/lib/hadoop/hadoop-distcp.jar
        00:06:37     /usr/lib/hadoop/hadoop-extras-0.23.1-SNAPSHOT.jar
        00:06:37     /usr/lib/hadoop/hadoop-extras.jar
        00:06:37     /usr/lib/hadoop/hadoop-rumen-0.23.1-SNAPSHOT.jar
        00:06:37     /usr/lib/hadoop/hadoop-rumen.jar
        00:06:37     /usr/lib/hadoop/libexec/httpfs-config.sh
        00:06:37     /usr/lib/hadoop/sbin/hdfs-config.sh
        00:06:37     /usr/lib/hadoop/sbin/httpfs.sh
        

        A good chunck of these files seem to be related to httpfs though. So is everything ok?

        Show
        Bruno Mahé added a comment - Thanks! I noticed your patch. I will try it as soon as it hits github mirror. The build seems to have passed without your latest fix on centos 6: http://bigtop01.cloudera.org:8080/job/Bigtop-hadoop-0.23-Hadoop/label=centos6/6/console But I see this warning message at the end: 00:06:37 warning: Installed (but unpackaged) file(s) found: 00:06:37 /etc/hadoop/conf.empty/httpfs-env.sh 00:06:37 /etc/hadoop/conf.empty/httpfs-log4j.properties 00:06:37 /etc/hadoop/conf.empty/httpfs-signature.secret 00:06:37 /etc/hadoop/conf.empty/httpfs-site.xml 00:06:37 /usr/lib/hadoop/hadoop-distcp-0.23.1-SNAPSHOT.jar 00:06:37 /usr/lib/hadoop/hadoop-distcp.jar 00:06:37 /usr/lib/hadoop/hadoop-extras-0.23.1-SNAPSHOT.jar 00:06:37 /usr/lib/hadoop/hadoop-extras.jar 00:06:37 /usr/lib/hadoop/hadoop-rumen-0.23.1-SNAPSHOT.jar 00:06:37 /usr/lib/hadoop/hadoop-rumen.jar 00:06:37 /usr/lib/hadoop/libexec/httpfs-config.sh 00:06:37 /usr/lib/hadoop/sbin/hdfs-config.sh 00:06:37 /usr/lib/hadoop/sbin/httpfs.sh A good chunck of these files seem to be related to httpfs though. So is everything ok?
        Hide
        Roman Shaposhnik added a comment -

        Most of these files are new artifacts that are now appearing because we've bumped the Hadoop version to the HEAD of branch-0.23. We'll have to figure out where they belong as far as packaging. I'll open a separate JIRA for that.

        The rest of them (httpfs-*) are moved to a different place during the packaging step so everything is fine there.

        The point is well taken though – it might be nice to not keep the second copy in the locations where RPM is warning us about.

        Anyway, I've checked the resulting hadoop-httpfs package and it works fine.

        Show
        Roman Shaposhnik added a comment - Most of these files are new artifacts that are now appearing because we've bumped the Hadoop version to the HEAD of branch-0.23. We'll have to figure out where they belong as far as packaging. I'll open a separate JIRA for that. The rest of them (httpfs-*) are moved to a different place during the packaging step so everything is fine there. The point is well taken though – it might be nice to not keep the second copy in the locations where RPM is warning us about. Anyway, I've checked the resulting hadoop-httpfs package and it works fine.
        Hide
        Bruno Mahé added a comment -

        +1

        Show
        Bruno Mahé added a comment - +1
        Hide
        Peter Linnell added a comment -

        +1 LGTM on the 2nd patch as well.

        Show
        Peter Linnell added a comment - +1 LGTM on the 2nd patch as well.

          People

          • Assignee:
            Roman Shaposhnik
            Reporter:
            Roman Shaposhnik
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development