Bigtop
  1. Bigtop
  2. BIGTOP-674

Installing hadoop-mapreduce package uses 'hadoop' group before creation

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Not a Problem
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Environment:

      Debian 12.04, probably other DEB and RPM-based distros

      Description

      When installing the hadoop-mapreduce package, users and files are assigned to the 'hadoop' group before that group has been created.

      I'm still working on verifying all the details for RPMs, but I will attached a patch that fixes this for DEBs (and quite possibly for RPMs too). It doesn't make sense to me why the group has not been added earlier. The hadoop group is added in the hadoop.spec file on line 510, and in the hadoop.preinst script - and I think both of these are supposed to execute before the lines where I've added the group in the attached patch.

      1. BIGTOP-674.patch
        1 kB
        Sean Mackrory

        Activity

        Hide
        Bruno Mahé added a comment -

        This is actually a feature, not a bug

        dpkg and rpm do not deal with dependencies. Meaning they will verify the dependencies are already installed, but they will not go fetch them and wouldn't know how to.
        In order to solve your issue, you can either:

        • Specify all the dependencies on the command line. ex: "dpkg -i bigtop-utils.deb, hadoop-yarn.deb hadoop-mapreduce.deb"
        • Try "sudo apt-get install -f" to fix the state of your system
        • Use a higher level command which will know how to handle dependencies. For debian/ubuntu it will be apt-get, for CentOS/Fedora/RHEL/SL it will be yum and for Mageia it will be urpmi
        Show
        Bruno Mahé added a comment - This is actually a feature, not a bug dpkg and rpm do not deal with dependencies. Meaning they will verify the dependencies are already installed, but they will not go fetch them and wouldn't know how to. In order to solve your issue, you can either: Specify all the dependencies on the command line. ex: "dpkg -i bigtop-utils.deb, hadoop-yarn.deb hadoop-mapreduce.deb" Try "sudo apt-get install -f" to fix the state of your system Use a higher level command which will know how to handle dependencies. For debian/ubuntu it will be apt-get, for CentOS/Fedora/RHEL/SL it will be yum and for Mageia it will be urpmi
        Hide
        Sean Mackrory added a comment -

        Although the package hadoop is ultimately a dependency, the group hadoop is assumed to have been created in the preinst stage of the package hadoop-mapreduce. Debian Policy Manual states "Only essential packages and pre-dependencies (Pre-Depends) may be assumed to be available." As you'll see in the output I've copied below, dpkg does indeed list hadoop-yarn as a dependency (which subsequently lists hadoop as a dependency), but it does so only after the error is encountered, so it's confusing to a user at best. I've had this problem when using an aptitude repository as well (albeit not as consistently), which I think is even worse. Perhaps a more correct solution would be to add "Pre-Depends: hadoop (= $

        {source:Version}

        )" to the deb package hadoop-mapreduce.

        As for the rpm package, I'm not sure I can find a better solution for RPMs than what I already submitted. Perhaps using "PreReq" instrad of "Requires", but it sounds like that difference will not be reliable anyway: http://www.rpm.org/max-rpm-snapshot/s1-rpm-depend-manual-dependencies.html#S3-RPM-DEPEND-PREREQ

        $ sudo dpkg -i hadoop-mapreduce_2.0.0-1_all.deb
        Selecting previously unselected package hadoop-mapreduce.
        (Reading database ... 51084 files and directories currently installed.)
        Unpacking hadoop-mapreduce (from hadoop-mapreduce_2.0.0-1_all.deb) ...
        usermod: group 'hadoop' does not exist
        dpkg: dependency problems prevent configuration of hadoop-mapreduce:
        hadoop-mapreduce depends on bigtop-utils; however:
        Package bigtop-utils is not installed.
        hadoop-mapreduce depends on hadoop-yarn (= 2.0.0-1); however:
        Package hadoop-yarn is not installed.
        dpkg: error processing hadoop-mapreduce (--install):
        dependency problems - leaving unconfigured
        Errors were encountered while processing:
        hadoop-mapreduce

        Show
        Sean Mackrory added a comment - Although the package hadoop is ultimately a dependency, the group hadoop is assumed to have been created in the preinst stage of the package hadoop-mapreduce. Debian Policy Manual states "Only essential packages and pre-dependencies (Pre-Depends) may be assumed to be available." As you'll see in the output I've copied below, dpkg does indeed list hadoop-yarn as a dependency (which subsequently lists hadoop as a dependency), but it does so only after the error is encountered, so it's confusing to a user at best. I've had this problem when using an aptitude repository as well (albeit not as consistently), which I think is even worse. Perhaps a more correct solution would be to add "Pre-Depends: hadoop (= $ {source:Version} )" to the deb package hadoop-mapreduce. As for the rpm package, I'm not sure I can find a better solution for RPMs than what I already submitted. Perhaps using "PreReq" instrad of "Requires", but it sounds like that difference will not be reliable anyway: http://www.rpm.org/max-rpm-snapshot/s1-rpm-depend-manual-dependencies.html#S3-RPM-DEPEND-PREREQ $ sudo dpkg -i hadoop-mapreduce_2.0.0-1_all.deb Selecting previously unselected package hadoop-mapreduce. (Reading database ... 51084 files and directories currently installed.) Unpacking hadoop-mapreduce (from hadoop-mapreduce_2.0.0-1_all.deb) ... usermod: group 'hadoop' does not exist dpkg: dependency problems prevent configuration of hadoop-mapreduce: hadoop-mapreduce depends on bigtop-utils; however: Package bigtop-utils is not installed. hadoop-mapreduce depends on hadoop-yarn (= 2.0.0-1); however: Package hadoop-yarn is not installed. dpkg: error processing hadoop-mapreduce (--install): dependency problems - leaving unconfigured Errors were encountered while processing: hadoop-mapreduce
        Hide
        Bruno Mahé added a comment -

        How did you install hadoop-mapreduce package?
        What error do you see?
        What steps have you been through?

        The group hadoop is created as part of the package hadoop and therefore will be created before hadoop-mapreduce gets installed.
        See https://github.com/apache/bigtop/blob/trunk/bigtop-packages/src/deb/hadoop/hadoop.preinst#L35

        Show
        Bruno Mahé added a comment - How did you install hadoop-mapreduce package? What error do you see? What steps have you been through? The group hadoop is created as part of the package hadoop and therefore will be created before hadoop-mapreduce gets installed. See https://github.com/apache/bigtop/blob/trunk/bigtop-packages/src/deb/hadoop/hadoop.preinst#L35
        Hide
        Sean Mackrory added a comment - - edited

        Patch tested on DEBs, still working on verifying on RPMs, but that might me take a little while.

        Show
        Sean Mackrory added a comment - - edited Patch tested on DEBs, still working on verifying on RPMs, but that might me take a little while.

          People

          • Assignee:
            Bruno Mahé
            Reporter:
            Sean Mackrory
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development