Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-1865

circular dependency between bigtop-groovy and bigtop-utils

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Pending Closed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 1.0.0
    • Component/s: debian, rpm
    • Labels:
      None

      Description

      While building rpm packages for some of the Cascading bits I noticed that there is a circular dependency between bigtop-groovy and bigtop-utils:

      [vagrant@localhost bigtop]$ rpm -ivh build/bigtop-groovy/rpm/RPMS/noarch/bigtop-groovy-2.3.8-1.el6.noarch.rpm
      error: Failed dependencies:
      bigtop-utils >= 0.7 is needed by bigtop-groovy-2.3.8-1.el6.noarch
      [vagrant@localhost bigtop]$ rpm -ivh build/bigtop-utils/rpm/RPMS/noarch/bigtop-utils-1.0.0-1.el6.noarch.rpm
      error: Failed dependencies:
      bigtop-groovy is needed by bigtop-utils-1.0.0-1.el6.noarch
      [vagrant@localhost bigtop]$

      1. BIGTOP-1865.patch
        31 kB
        Konstantin Boudnik
      2. BIGTOP-1865.patch
        28 kB
        Konstantin Boudnik

        Issue Links

          Activity

          Hide
          evans_ye Evans Ye added a comment -

          Yup. I encounter this once before. When installing by yum it some how works.
          IIRC, this was introduced by a JIRA that replace the hadoop cmds by groovy API calls in init-hdfs.sh. Which we have a init-hcfs.groovy packaged in bigtop-utils. I guess that's the reason bigtop-utils is needed by bigtop-groovy. no?
          Maybe Konstantin Boudnik knows more.

          Show
          evans_ye Evans Ye added a comment - Yup. I encounter this once before. When installing by yum it some how works. IIRC, this was introduced by a JIRA that replace the hadoop cmds by groovy API calls in init-hdfs.sh. Which we have a init-hcfs.groovy packaged in bigtop-utils. I guess that's the reason bigtop-utils is needed by bigtop-groovy. no? Maybe Konstantin Boudnik knows more.
          Hide
          cos Konstantin Boudnik added a comment -

          oh, damn... That's right Evans Ye: the init-hcfs needs to depend on groovy package, which relies, in order, on utils to find $JAVA_HOME. We can move the bigtop-groovy dependency into hadoop-hdfs-namenode package perhaps. What do you think about the above?
          I believe Olaf Flebbe was trying to do something like that once, but I foolishly stopped him ;(

          Show
          cos Konstantin Boudnik added a comment - oh, damn... That's right Evans Ye : the init-hcfs needs to depend on groovy package, which relies, in order, on utils to find $JAVA_HOME . We can move the bigtop-groovy dependency into hadoop-hdfs-namenode package perhaps. What do you think about the above? I believe Olaf Flebbe was trying to do something like that once, but I foolishly stopped him ;(
          Hide
          oflebbe Olaf Flebbe added a comment -

          Feel free to reconsider BIGTOP-1798 Patch 1

          Show
          oflebbe Olaf Flebbe added a comment - Feel free to reconsider BIGTOP-1798 Patch 1
          Hide
          evans_ye Evans Ye added a comment - - edited

          if I get the things right:

          • bigtop-utils is needed by bigtop-groovy because bigtop-utils helps to find $JAVA_HOME
          • bigtop-groovy is needed by bigtop-utils because bigtop-groovy provides groovy installation for init-hcfs.groovy

          My thought is to move init-hcfs.groovy from bigtop-utils to bigtop-groovy. This based on a concept that we don't treat bigtop-groovy only as a groovy installation package, but rather than a package that any groovy stuff bigtop needed resides in this package.
          This is much easier for people to locate a groovy script. That is to say: If I don't have the context of bigtop's packaging rule, I'll guess xxx.groovy should be reside in the bigtop-groovy package.
          Thoughts?

          Show
          evans_ye Evans Ye added a comment - - edited if I get the things right: bigtop-utils is needed by bigtop-groovy because bigtop-utils helps to find $JAVA_HOME bigtop-groovy is needed by bigtop-utils because bigtop-groovy provides groovy installation for init-hcfs.groovy My thought is to move init-hcfs.groovy from bigtop-utils to bigtop-groovy. This based on a concept that we don't treat bigtop-groovy only as a groovy installation package, but rather than a package that any groovy stuff bigtop needed resides in this package. This is much easier for people to locate a groovy script. That is to say: If I don't have the context of bigtop's packaging rule, I'll guess xxx.groovy should be reside in the bigtop-groovy package. Thoughts?
          Hide
          cos Konstantin Boudnik added a comment -

          I think it might be a better solution than shoving it into a hadoop* package. I don't like it much from concerns separation standpoint, but it might be an ok solution. Ideally, the scripts would go to something like bigtop-scripts, but we always can do it in a later time. I am +1 on the idea.

          Show
          cos Konstantin Boudnik added a comment - I think it might be a better solution than shoving it into a hadoop* package. I don't like it much from concerns separation standpoint, but it might be an ok solution. Ideally, the scripts would go to something like bigtop-scripts, but we always can do it in a later time. I am +1 on the idea.
          Hide
          evans_ye Evans Ye added a comment -

          Correct me if I am wrong.
          My understanding is that init-hcfs.groovy can be used to arrange directories and libraries on any kind of DistributedFileSystem. That's why we name it hcfs. Wouldn't it be to specific if we put it into a hadoop* package?

          Show
          evans_ye Evans Ye added a comment - Correct me if I am wrong. My understanding is that init-hcfs.groovy can be used to arrange directories and libraries on any kind of DistributedFileSystem. That's why we name it hcfs. Wouldn't it be to specific if we put it into a hadoop* package?
          Hide
          oflebbe Olaf Flebbe added a comment -

          I would be happier if this script will go into bigtop-utils.

          The puppet deploy scripts can be adapted to use init-hcfs.groovy instead of init-hdfs.sh

          So we have a more or less vanilla hadoop package.

          Show
          oflebbe Olaf Flebbe added a comment - I would be happier if this script will go into bigtop-utils. The puppet deploy scripts can be adapted to use init-hcfs.groovy instead of init-hdfs.sh So we have a more or less vanilla hadoop package.
          Hide
          evans_ye Evans Ye added a comment -

          Do you mean the script init-hcfs.groovy will go into bigtop-utils?
          Since its already packaged in bigtop-utils.
          If we keep init-hcfs.groovy packaged in bigtop-utils, then we're back to the circular dependency problem.

          • bigtop-utils -> bigtop-groovy: groovy need to be installed for init-hcfs.groovy to work
          • bigtop-groovy -> bigtop-utils: util to detect JAVA_HOME need to be installed for groovy to work
          Show
          evans_ye Evans Ye added a comment - Do you mean the script init-hcfs.groovy will go into bigtop-utils? Since its already packaged in bigtop-utils. If we keep init-hcfs.groovy packaged in bigtop-utils, then we're back to the circular dependency problem. bigtop-utils -> bigtop-groovy: groovy need to be installed for init-hcfs.groovy to work bigtop-groovy -> bigtop-utils: util to detect JAVA_HOME need to be installed for groovy to work
          Hide
          cos Konstantin Boudnik added a comment -

          Good point, I've missed that angle. So please scratch my comment above.

          Show
          cos Konstantin Boudnik added a comment - Good point, I've missed that angle. So please scratch my comment above.
          Hide
          cos Konstantin Boudnik added a comment -

          So, what's the consensus then?

          Show
          cos Konstantin Boudnik added a comment - So, what's the consensus then?
          Hide
          evans_ye Evans Ye added a comment -

          Yes. I propose to move init-hcfs.groovy from bigtop-utils to bigtop-groovy. So that we can remove the dependency setting that bigtop-utils depends on bigtop-groovy.

          Show
          evans_ye Evans Ye added a comment - Yes. I propose to move init-hcfs.groovy from bigtop-utils to bigtop-groovy. So that we can remove the dependency setting that bigtop-utils depends on bigtop-groovy.
          Hide
          cos Konstantin Boudnik added a comment -

          I've looked at it some more and here's what I think:

          • hadoop-hdfs package already contains init-hdfs.sh and init-hcfs.json
          • having the groovy script right there (instead of current placement in bigtop-utils) just makes sense as it'd keep all the bits of the formatter together, not scattered together. At the same time - it doesn't create a bit artificial inclusion of the script into runtime package.
          Show
          cos Konstantin Boudnik added a comment - I've looked at it some more and here's what I think: hadoop-hdfs package already contains init-hdfs.sh and init-hcfs.json having the groovy script right there (instead of current placement in bigtop-utils) just makes sense as it'd keep all the bits of the formatter together, not scattered together. At the same time - it doesn't create a bit artificial inclusion of the script into runtime package.
          Hide
          cos Konstantin Boudnik added a comment - - edited

          How about something like this?

          Show
          cos Konstantin Boudnik added a comment - - edited How about something like this ?
          Hide
          evans_ye Evans Ye added a comment -

          I'd say the patch and your idea is, generally, good.
          Although putting hcfs files in hadoop might not be the best choice, putting them in bigtop-groovy seems not an elegant way as well.
          I like the idea to have all the related files contained in one single package, that's way easier to locate each of them.

          The patch just need to add line 188 in bigtop-packages/src/common/hadoop/install_hadoop.sh so that it's ready to commit:

          186 install -m 0755 ${DISTRO_DIR}/init-hdfs.sh ${SYSTEM_LIBEXEC_DIR}/
          187 install -m 0755 ${DISTRO_DIR}/init-hcfs.json ${SYSTEM_LIBEXEC_DIR}/
          188 install -m 0755 ${DISTRO_DIR}/init-hcfs.groovy ${SYSTEM_LIBEXEC_DIR}/
          189 rm -rf ${SYSTEM_LIBEXEC_DIR}/*.cmd
          

          Otherwise, missing init-hcfs.groovy when building hadoop.

          Show
          evans_ye Evans Ye added a comment - I'd say the patch and your idea is, generally, good. Although putting hcfs files in hadoop might not be the best choice, putting them in bigtop-groovy seems not an elegant way as well. I like the idea to have all the related files contained in one single package, that's way easier to locate each of them. The patch just need to add line 188 in bigtop-packages/src/common/hadoop/install_hadoop.sh so that it's ready to commit: 186 install -m 0755 ${DISTRO_DIR}/init-hdfs.sh ${SYSTEM_LIBEXEC_DIR}/ 187 install -m 0755 ${DISTRO_DIR}/init-hcfs.json ${SYSTEM_LIBEXEC_DIR}/ 188 install -m 0755 ${DISTRO_DIR}/init-hcfs.groovy ${SYSTEM_LIBEXEC_DIR}/ 189 rm -rf ${SYSTEM_LIBEXEC_DIR}/*.cmd Otherwise, missing init-hcfs.groovy when building hadoop.
          Hide
          cos Konstantin Boudnik added a comment -

          Good catch - thanks for spotting the missing line! I will build/test the pkgs today before posting new version of the patch.

          WRT placement of the hcfs stuff in HDFS: I agree it is utterly ugly. The only reason I am doing this is to unblock 1.0 with least changes and disruptions. I will open a new JIRA to extract all of this into a separate package. Does it sound ok?

          Show
          cos Konstantin Boudnik added a comment - Good catch - thanks for spotting the missing line! I will build/test the pkgs today before posting new version of the patch. WRT placement of the hcfs stuff in HDFS: I agree it is utterly ugly. The only reason I am doing this is to unblock 1.0 with least changes and disruptions. I will open a new JIRA to extract all of this into a separate package. Does it sound ok?
          Hide
          evans_ye Evans Ye added a comment -

          That's perfectly fine with me. Thanks!

          Show
          evans_ye Evans Ye added a comment - That's perfectly fine with me. Thanks!
          Hide
          cos Konstantin Boudnik added a comment - - edited

          Ah, and of course, hadoop-hdfs needs to declare the dependency on bigtop-groovy - otherwise the init script will be failing to start. Ah.... that's ugly ;(

          Show
          cos Konstantin Boudnik added a comment - - edited Ah, and of course, hadoop-hdfs needs to declare the dependency on bigtop-groovy - otherwise the init script will be failing to start. Ah.... that's ugly ;(
          Hide
          cos Konstantin Boudnik added a comment -

          Ok, tested on Ubuntu and CentOS

          Show
          cos Konstantin Boudnik added a comment - Ok, tested on Ubuntu and CentOS
          Hide
          evans_ye Evans Ye added a comment -

          Yup, you're right. There should be a dependency hadoop-hdfs -> bigtop-groovy.
          +1 to the updated patch.
          I've committed this. Thanks for addressing the issue Konstantin Boudnik.

          Show
          evans_ye Evans Ye added a comment - Yup, you're right. There should be a dependency hadoop-hdfs -> bigtop-groovy . +1 to the updated patch. I've committed this. Thanks for addressing the issue Konstantin Boudnik .
          Hide
          cos Konstantin Boudnik added a comment -

          Thanks for committing this!

          Show
          cos Konstantin Boudnik added a comment - Thanks for committing this!

            People

            • Assignee:
              cos Konstantin Boudnik
              Reporter:
              fs111 André Kelpe
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development