Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-2802

Some packages don't create necessary groups on debian-based system if the users correspond to them already exist

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.2.0
    • Fix Version/s: 1.2.1
    • Component/s: None
    • Labels:
      None

      Description

      For example, user and group creation for yarn are independent on rpm-based systems:

      bigtop-packages/src/rpm/hadoop/SPECS/hadoop.spec
      557 %pre yarn
      558 getent group yarn >/dev/null   || groupadd -r yarn
      559 getent passwd yarn >/dev/null || /usr/sbin/useradd --comment "Hadoop Yarn" --shell /bin/bash -M -r -g yarn -G hadoop --home %{state_yarn} yarn
      

      but on debian-based systems, yarn group creation is skipped if yarn user is created in advance:

      bigtop-packages/src/deb/hadoop/hadoop-yarn.preinst
       34     install|upgrade)
       35         if ! getent passwd yarn >/dev/null; then
       36             # Adding system user: yarn.
       37             adduser \
       38                 --system \
       39                 --group \
       40                 --home /var/lib/hadoop-yarn \
       41                 --gecos "Hadoop YARN" \
       42                 --shell /bin/bash \
       43                 yarn >/dev/null 2>/dev/null || :
       44             usermod --append --groups hadoop yarn || :
       45         fi
       46     ;;
      

      It causes a problem when the package is used with Ambari. When Ambari installs a stack, it creates users such as hdfs, yarn, mapred besides hadoop, but it creates only hadoop group.
      So installing hadoop-yarn skips the above path and fails with the following error:

      Setting up hadoop-yarn (2.7.3-1) ...
      chown: invalid group: 'root:yarn'
      dpkg: error processing package hadoop-yarn (--configure):
       subprocess installed post-installation script returned error exit status 1
      Errors were encountered while processing:
       hadoop-yarn
      

      at line 24 in the following code:

      bigtop-packages/src/deb/hadoop/hadoop-yarn.postinst
       23     configure)
       24         chown root:yarn /usr/lib/hadoop-yarn/bin/container-executor
       25         chmod 4754 /usr/lib/hadoop-yarn/bin/container-executor
       26         mkdir -p /var/log/hadoop-yarn /var/run/hadoop-yarn /var/lib/hadoop-yarn/cache || :
       27         chown yarn:hadoop /var/log/hadoop-yarn /var/run/hadoop-yarn
       28         chmod g+w /var/log/hadoop-yarn /var/run/hadoop-yarn
       29         chown yarn:hadoop /var/lib/hadoop-yarn/ /var/lib/hadoop-yarn/cache
       30         chmod 0755 /var/lib/hadoop-yarn
       31         chmod 1777 /var/lib/hadoop-yarn/cache
       32     ;;
      

        Issue Links

          Activity

          Hide
          sekikn Kengo Seki added a comment -

          Committed and pushed. Thanks for the review!

          Show
          sekikn Kengo Seki added a comment - Committed and pushed. Thanks for the review!
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/bigtop/pull/234

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/bigtop/pull/234
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user sekikn commented on the issue:

          https://github.com/apache/bigtop/pull/234

          @c0s @oflebbe @rvs thanks for the reviews! I'm going to check this in.

          Show
          githubbot ASF GitHub Bot added a comment - Github user sekikn commented on the issue: https://github.com/apache/bigtop/pull/234 @c0s @oflebbe @rvs thanks for the reviews! I'm going to check this in.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user oflebbe commented on the issue:

          https://github.com/apache/bigtop/pull/234

          Oh yes, it was too late that night. The patch looks very good for debian. Please apply

          Show
          githubbot ASF GitHub Bot added a comment - Github user oflebbe commented on the issue: https://github.com/apache/bigtop/pull/234 Oh yes, it was too late that night. The patch looks very good for debian. Please apply
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user rvs commented on the issue:

          https://github.com/apache/bigtop/pull/234

          @sekikn I think the patch looks reasonable, @oflebbe did you mistype? This patch only applies on Debian side – so if you know any caveats of using --ingroup on that platform – please let us know. Otherwise – I guess the patch can go in?

          Show
          githubbot ASF GitHub Bot added a comment - Github user rvs commented on the issue: https://github.com/apache/bigtop/pull/234 @sekikn I think the patch looks reasonable, @oflebbe did you mistype? This patch only applies on Debian side – so if you know any caveats of using --ingroup on that platform – please let us know. Otherwise – I guess the patch can go in?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user oflebbe commented on the issue:

          https://github.com/apache/bigtop/pull/234

          IMHO ambari should be fixed. Why does it create broken user/group configs?

          Show
          githubbot ASF GitHub Bot added a comment - Github user oflebbe commented on the issue: https://github.com/apache/bigtop/pull/234 IMHO ambari should be fixed. Why does it create broken user/group configs?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user oflebbe commented on the issue:

          https://github.com/apache/bigtop/pull/234

          adduser on centos-7 does not have that flag '--ingroup'. Sadly.

          Show
          githubbot ASF GitHub Bot added a comment - Github user oflebbe commented on the issue: https://github.com/apache/bigtop/pull/234 adduser on centos-7 does not have that flag '--ingroup'. Sadly.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user sekikn commented on the issue:

          https://github.com/apache/bigtop/pull/234

          Thanks! In addition to that, this fix also covers the case that a user already exists but its group is not created. In such a case, the original scripts don't create the groups, so the final state differs from the normal case. It's undesirable.

          Show
          githubbot ASF GitHub Bot added a comment - Github user sekikn commented on the issue: https://github.com/apache/bigtop/pull/234 Thanks! In addition to that, this fix also covers the case that a user already exists but its group is not created. In such a case, the original scripts don't create the groups, so the final state differs from the normal case. It's undesirable.
          Hide
          sekikn Kengo Seki added a comment -

          Submitted PR. I confirmed that all components except for solr (because of the reason described in BIGTOP-2784) were successfully built and most of them were installed on Ubuntu 16.04.
          I got the following errors for hue, oozie and phoenix while installing them, but they seem irrelevant to this fix. I'll inspect them as other issues.

          hue
          # apt install hue-common
          Reading package lists... Done
          Building dependency tree       
          Reading state information... Done
          Some packages could not be installed. This may mean that you have
          requested an impossible situation or if you are using the unstable
          distribution that some required packages have not yet been created
          or been moved out of Incoming.
          The following information may help to resolve the situation:
          
          The following packages have unmet dependencies:
           hue-common : Depends: libffi but it is not installable
          E: Unable to correct problems, you have held broken packages.
          
          oozie
          # apt-get install oozie
          Reading package lists... Done
          Building dependency tree       
          Reading state information... Done
          The following NEW packages will be installed:
            oozie
          0 upgraded, 1 newly installed, 0 to remove and 67 not upgraded.
          Need to get 0 B/401 MB of archives.
          After this operation, 415 MB of additional disk space will be used.
          WARNING: The following packages cannot be authenticated!
            oozie
          Authentication warning overridden.
          Get:1 file:/bigtop-home/output/apt bigtop/contrib amd64 oozie all 4.3.0-1 [401 MB]
          debconf: unable to initialize frontend: Dialog
          debconf: (TERM is not set, so the dialog frontend is not usable.)
          debconf: falling back to frontend: Readline
          (Reading database ... 57747 files and directories currently installed.)
          Preparing to unpack .../o/oozie/oozie_4.3.0-1_all.deb ...
          Unpacking oozie (4.3.0-1) ...
          dpkg: error processing archive /bigtop-home/output/apt/pool/contrib/o/oozie/oozie_4.3.0-1_all.deb (--unpack):
           trying to overwrite '/usr/lib/oozie/lib/apacheds-kerberos-codec-2.0.0-M15.jar', which is also in package oozie-client 4.3.0-1
          dpkg-deb: error: subprocess paste was killed by signal (Broken pipe)
          Errors were encountered while processing:
           /bigtop-home/output/apt/pool/contrib/o/oozie/oozie_4.3.0-1_all.deb
          E: Sub-process /usr/bin/dpkg returned an error code (1)
          
          phoenix
          # apt-get install phoenix-queryserver
          Reading package lists... Done
          Building dependency tree       
          Reading state information... Done
          The following NEW packages will be installed:
            phoenix-queryserver
          0 upgraded, 1 newly installed, 0 to remove and 67 not upgraded.
          Need to get 0 B/1930 B of archives.
          After this operation, 12.3 kB of additional disk space will be used.
          WARNING: The following packages cannot be authenticated!
            phoenix-queryserver
          Authentication warning overridden.
          Get:1 file:/bigtop-home/output/apt bigtop/contrib amd64 phoenix-queryserver all 4.9.0-1 [1930 B]
          debconf: unable to initialize frontend: Dialog
          debconf: (TERM is not set, so the dialog frontend is not usable.)
          debconf: falling back to frontend: Readline
          (Reading database ... 57817 files and directories currently installed.)
          Preparing to unpack .../phoenix-queryserver_4.9.0-1_all.deb ...
          Unpacking phoenix-queryserver (4.9.0-1) ...
          dpkg: error processing archive /bigtop-home/output/apt/pool/contrib/p/phoenix/phoenix-queryserver_4.9.0-1_all.deb (--unpack):
           trying to overwrite '/etc/default/phoenix-queryserver', which is also in package phoenix 4.9.0-1
          Errors were encountered while processing:
           /bigtop-home/output/apt/pool/contrib/p/phoenix/phoenix-queryserver_4.9.0-1_all.deb
          E: Sub-process /usr/bin/dpkg returned an error code (1)
          
          Show
          sekikn Kengo Seki added a comment - Submitted PR. I confirmed that all components except for solr (because of the reason described in BIGTOP-2784 ) were successfully built and most of them were installed on Ubuntu 16.04. I got the following errors for hue, oozie and phoenix while installing them, but they seem irrelevant to this fix. I'll inspect them as other issues. hue # apt install hue-common Reading package lists... Done Building dependency tree Reading state information... Done Some packages could not be installed. This may mean that you have requested an impossible situation or if you are using the unstable distribution that some required packages have not yet been created or been moved out of Incoming. The following information may help to resolve the situation: The following packages have unmet dependencies: hue-common : Depends: libffi but it is not installable E: Unable to correct problems, you have held broken packages. oozie # apt-get install oozie Reading package lists... Done Building dependency tree Reading state information... Done The following NEW packages will be installed: oozie 0 upgraded, 1 newly installed, 0 to remove and 67 not upgraded. Need to get 0 B/401 MB of archives. After this operation, 415 MB of additional disk space will be used. WARNING: The following packages cannot be authenticated! oozie Authentication warning overridden. Get:1 file:/bigtop-home/output/apt bigtop/contrib amd64 oozie all 4.3.0-1 [401 MB] debconf: unable to initialize frontend: Dialog debconf: (TERM is not set, so the dialog frontend is not usable.) debconf: falling back to frontend: Readline (Reading database ... 57747 files and directories currently installed.) Preparing to unpack .../o/oozie/oozie_4.3.0-1_all.deb ... Unpacking oozie (4.3.0-1) ... dpkg: error processing archive /bigtop-home/output/apt/pool/contrib/o/oozie/oozie_4.3.0-1_all.deb (--unpack): trying to overwrite '/usr/lib/oozie/lib/apacheds-kerberos-codec-2.0.0-M15.jar', which is also in package oozie-client 4.3.0-1 dpkg-deb: error: subprocess paste was killed by signal (Broken pipe) Errors were encountered while processing: /bigtop-home/output/apt/pool/contrib/o/oozie/oozie_4.3.0-1_all.deb E: Sub-process /usr/bin/dpkg returned an error code (1) phoenix # apt-get install phoenix-queryserver Reading package lists... Done Building dependency tree Reading state information... Done The following NEW packages will be installed: phoenix-queryserver 0 upgraded, 1 newly installed, 0 to remove and 67 not upgraded. Need to get 0 B/1930 B of archives. After this operation, 12.3 kB of additional disk space will be used. WARNING: The following packages cannot be authenticated! phoenix-queryserver Authentication warning overridden. Get:1 file:/bigtop-home/output/apt bigtop/contrib amd64 phoenix-queryserver all 4.9.0-1 [1930 B] debconf: unable to initialize frontend: Dialog debconf: (TERM is not set, so the dialog frontend is not usable.) debconf: falling back to frontend: Readline (Reading database ... 57817 files and directories currently installed.) Preparing to unpack .../phoenix-queryserver_4.9.0-1_all.deb ... Unpacking phoenix-queryserver (4.9.0-1) ... dpkg: error processing archive /bigtop-home/output/apt/pool/contrib/p/phoenix/phoenix-queryserver_4.9.0-1_all.deb (--unpack): trying to overwrite '/etc/ default /phoenix-queryserver', which is also in package phoenix 4.9.0-1 Errors were encountered while processing: /bigtop-home/output/apt/pool/contrib/p/phoenix/phoenix-queryserver_4.9.0-1_all.deb E: Sub-process /usr/bin/dpkg returned an error code (1)
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user sekikn opened a pull request:

          https://github.com/apache/bigtop/pull/234

          BIGTOP-2802. Some packages don't create necessary groups on debian-based system if the users correspond to them already exist

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/sekikn/bigtop BIGTOP-2802

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/bigtop/pull/234.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #234


          commit b43a1772d147e5b57c051b25df97bbf1a921687d
          Author: Kengo Seki <sekikn@apache.org>
          Date: 2017-06-09T01:13:25Z

          BIGTOP-2802. Some packages don't create necessary groups on debian-based system if the users correspond to them already exist


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user sekikn opened a pull request: https://github.com/apache/bigtop/pull/234 BIGTOP-2802 . Some packages don't create necessary groups on debian-based system if the users correspond to them already exist You can merge this pull request into a Git repository by running: $ git pull https://github.com/sekikn/bigtop BIGTOP-2802 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/bigtop/pull/234.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #234 commit b43a1772d147e5b57c051b25df97bbf1a921687d Author: Kengo Seki <sekikn@apache.org> Date: 2017-06-09T01:13:25Z BIGTOP-2802 . Some packages don't create necessary groups on debian-based system if the users correspond to them already exist
          Hide
          sekikn Kengo Seki added a comment -

          Also, I found typos in bigtop-packages/src/deb/phoenix/phoenix.preinst (phenix instead of phoenix). I'll fix them together.

          bigtop-packages/src/deb/phoenix/phoenix.preinst
           26         if ! getent passwd phenix >/dev/null; then
           27                 # Adding system user: phenix.
           28                 adduser \
           29                   --system \
           30                   --group \
           31                   --home /var/lib/phenix \
           32                   --gecos "Phoenix User" \
           33                   --shell /bin/false \
           34                   phoenix >/dev/null
           35         fi
          
          Show
          sekikn Kengo Seki added a comment - Also, I found typos in bigtop-packages/src/deb/phoenix/phoenix.preinst (phenix instead of phoenix). I'll fix them together. bigtop-packages/src/deb/phoenix/phoenix.preinst 26 if ! getent passwd phenix >/dev/ null ; then 27 # Adding system user: phenix. 28 adduser \ 29 --system \ 30 --group \ 31 --home / var /lib/phenix \ 32 --gecos "Phoenix User" \ 33 --shell /bin/ false \ 34 phoenix >/dev/ null 35 fi

            People

            • Assignee:
              sekikn Kengo Seki
              Reporter:
              sekikn Kengo Seki
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development