Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.6.0
    • Component/s: None
    • Labels:
      None

      Description

      OOZIE-1054 allows Oozie to automatically upload the sharelib to HDFS; Bigtop should either use that mechanism or do something similar.

        Activity

        Robert Kanter created issue -
        Robert Kanter made changes -
        Field Original Value New Value
        Description OOZIE-1054 allows Oozie to automatically upload the sharelib to HDFS; BigTop should either use that mechanism or do something similar. OOZIE-1054 allows Oozie to automatically upload the sharelib to HDFS; Bigtop should either use that mechanism or do something similar.
        Hide
        Konstantin Boudnik added a comment -

        DUP of BIGTOP-852

        Show
        Konstantin Boudnik added a comment - DUP of BIGTOP-852
        Konstantin Boudnik made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Duplicate [ 3 ]
        Hide
        Konstantin Boudnik added a comment -

        Thanks for digging up the dub, Bruno

        Show
        Konstantin Boudnik added a comment - Thanks for digging up the dub, Bruno
        Hide
        Robert Kanter added a comment -

        Forgive me if I'm missing something, but BIGTOP-852 doesn't seem to be addressing this issue. It's to create user directories in HDFS and this issue is to upload the Oozie sharelib (a bunch of jar files in a directory structure) to HDFS. While related, it doesn't look like the patch in BIGTOP-852 actually does this.

        Show
        Robert Kanter added a comment - Forgive me if I'm missing something, but BIGTOP-852 doesn't seem to be addressing this issue. It's to create user directories in HDFS and this issue is to upload the Oozie sharelib (a bunch of jar files in a directory structure) to HDFS. While related, it doesn't look like the patch in BIGTOP-852 actually does this.
        Hide
        Konstantin Boudnik added a comment -

        Yes, right on. I did it in a hurry with a vague recollection of

        # FIXME: OOZIE-1287
        for i in `cd ${SERVER_LIB_DIR}/libserver ; ls slf4j*.jar log4j*.jar` ; do
          cp ${SERVER_LIB_DIR}/libserver/$i ${SERVER_LIB_DIR}/lib/$i
        done
        

        that Roman recommended for install_oozie.sh as a workaround.

        Show
        Konstantin Boudnik added a comment - Yes, right on. I did it in a hurry with a vague recollection of # FIXME: OOZIE-1287 for i in `cd ${SERVER_LIB_DIR}/libserver ; ls slf4j*.jar log4j*.jar` ; do cp ${SERVER_LIB_DIR}/libserver/$i ${SERVER_LIB_DIR}/lib/$i done that Roman recommended for install_oozie.sh as a workaround.
        Konstantin Boudnik made changes -
        Resolution Duplicate [ 3 ]
        Status Resolved [ 5 ] Reopened [ 4 ]
        Hide
        Robert Kanter added a comment -

        That looks like its for the oozie server libraries; and OOOZIE-1287 is about the client missing slf4j.

        These are not related to the sharelib. The sharelib is the set of Hive, Sqoop, Pig, etc jars that Oozie needs in hdfs for those actions to work properly; its typically under /user/oozie/share/lib and contains a subfolder for each action (e.g. /user/oozie/share/lib/pig/ contains all of the Pig jars that Oozie needs). I don't see this anywhere.

        Show
        Robert Kanter added a comment - That looks like its for the oozie server libraries; and OOOZIE-1287 is about the client missing slf4j. These are not related to the sharelib. The sharelib is the set of Hive, Sqoop, Pig, etc jars that Oozie needs in hdfs for those actions to work properly; its typically under /user/oozie/share/lib and contains a subfolder for each action (e.g. /user/oozie/share/lib/pig/ contains all of the Pig jars that Oozie needs). I don't see this anywhere.
        Hide
        Konstantin Boudnik added a comment -

        I am using puppet deployment from Bigtop trunk to deploy my clusters and I see
        all these libs in /user/oozie/share/lib, so I am certain that the issue is fixed. Now we need to find out when/how it got fixed and close this ticket accordingly

        Show
        Konstantin Boudnik added a comment - I am using puppet deployment from Bigtop trunk to deploy my clusters and I see all these libs in /user/oozie/share/lib , so I am certain that the issue is fixed. Now we need to find out when/how it got fixed and close this ticket accordingly
        Hide
        Sean Mackrory added a comment - - edited

        It looks like this was originally done in cluster.pp BIGTOP-797, but was most recently modified to use the init-hdfs.sh command in BIGTOP-893. If stuff is added to init-hdfs.sh in the future that not all users want, we ought to split it into individual commands that can also be invoked individually, but for now it merely creates directories that some services depend on, and uploads share-lib, which seems good. Thoughts on that? Would we rather split it up now so there's a distinct "upload-oozie-sharelib" command?

        Show
        Sean Mackrory added a comment - - edited It looks like this was originally done in cluster.pp BIGTOP-797 , but was most recently modified to use the init-hdfs.sh command in BIGTOP-893 . If stuff is added to init-hdfs.sh in the future that not all users want, we ought to split it into individual commands that can also be invoked individually, but for now it merely creates directories that some services depend on, and uploads share-lib, which seems good. Thoughts on that? Would we rather split it up now so there's a distinct "upload-oozie-sharelib" command?
        Hide
        Roman Shaposhnik added a comment -

        I think if a component takes onto the ownership of parts of init-hdfs.sh job it would be more than reasonable to call into the component's provided script instead. In this particular case we just need to make sure that the functionality actually provided by Oozie works in Bigtop context.

        Show
        Roman Shaposhnik added a comment - I think if a component takes onto the ownership of parts of init-hdfs.sh job it would be more than reasonable to call into the component's provided script instead. In this particular case we just need to make sure that the functionality actually provided by Oozie works in Bigtop context.
        Roman Shaposhnik made changes -
        Assignee Sean Mackrory [ mackrorysd ]
        Hide
        Robert Kanter added a comment -

        If you want to try using the functionality provided by Oozie, the script for this is part of the oozie-setup.sh script under the bin dir. You would use one of the following commands:

        To create the sharelib:

        oozie-setup.sh sharelib create -fs FS_URI [-locallib SHARED_LIBRARY]
        

        where FS_URI is the HDFS URI (i.e. the value of fs.default.name): e.g. hdfs://localhost:8020
        and SHARED_LIBRARY is the local path to the sharelib tarball or expanded tarball (i.e. a folder)

        To upgrade (i.e. replace) an existing sharelib:

        oozie-setup.sh sharelib upgrade -fs FS_URI [-locallib SHARED_LIBRARY]
        

        where the arguments are the same as above.

        Either command should be run as the user who will be starting the Oozie server (typically "oozie").

        Behind the scenes those two commands are fairly basic, they essentially just upload a directory to HDFS; its not like oozie-setup.sh is doing anything special. So if its easier for Bigtop to do the equivalent in init-hdfs.sh, then feel free to keep doing that.

        Show
        Robert Kanter added a comment - If you want to try using the functionality provided by Oozie, the script for this is part of the oozie-setup.sh script under the bin dir. You would use one of the following commands: To create the sharelib: oozie-setup.sh sharelib create -fs FS_URI [-locallib SHARED_LIBRARY] where FS_URI is the HDFS URI (i.e. the value of fs.default.name ): e.g. hdfs://localhost:8020 and SHARED_LIBRARY is the local path to the sharelib tarball or expanded tarball (i.e. a folder) To upgrade (i.e. replace) an existing sharelib: oozie-setup.sh sharelib upgrade -fs FS_URI [-locallib SHARED_LIBRARY] where the arguments are the same as above. Either command should be run as the user who will be starting the Oozie server (typically "oozie"). Behind the scenes those two commands are fairly basic, they essentially just upload a directory to HDFS; its not like oozie-setup.sh is doing anything special. So if its easier for Bigtop to do the equivalent in init-hdfs.sh , then feel free to keep doing that.
        Hide
        Sean Mackrory added a comment - - edited

        Had a quick offline conversation about some options. I'll do my best to recreate the details here - forgive any clumsiness as this is my first exposure to this issue. There are a few options:

        • Using init-hdfs.sh in its current form
        • Using Oozie's Java tools to upload the generated tarball (or a directory listing all the libraries)

        The tarball may not contain the correct versions initially, and will cease to contain the correct versions when other components are updated (unless they rebuild Oozie - not reasonable). We could generate a directory containing symlinks to the current libraries, and then use Oozie's tool to use that, but this doesn't offer any clear benefits over using init-hdfs in the first place. The main difference would be that init-hdfs.sh doesn't require you to enter the fs url, but it also doesn't support the property in Oozie that specifies arbitrary locations for the sharelib. Given all that, we decided that the cleanest solution for Bigtop right now was to continue using init-hdfs.sh in its current form. If users install or upgrade new components, they need only run the script again to sync their sharelib. I'm certainly open to improving this in the future if Oozie changes the way this is managed on the server, but I think this is the all-around best option right now.

        If I'm missing any details or you have anything else to add, please speak up, otherwise I'll re-close this as resolved.

        Show
        Sean Mackrory added a comment - - edited Had a quick offline conversation about some options. I'll do my best to recreate the details here - forgive any clumsiness as this is my first exposure to this issue. There are a few options: Using init-hdfs.sh in its current form Using Oozie's Java tools to upload the generated tarball (or a directory listing all the libraries) The tarball may not contain the correct versions initially, and will cease to contain the correct versions when other components are updated (unless they rebuild Oozie - not reasonable). We could generate a directory containing symlinks to the current libraries, and then use Oozie's tool to use that, but this doesn't offer any clear benefits over using init-hdfs in the first place. The main difference would be that init-hdfs.sh doesn't require you to enter the fs url, but it also doesn't support the property in Oozie that specifies arbitrary locations for the sharelib. Given all that, we decided that the cleanest solution for Bigtop right now was to continue using init-hdfs.sh in its current form. If users install or upgrade new components, they need only run the script again to sync their sharelib. I'm certainly open to improving this in the future if Oozie changes the way this is managed on the server, but I think this is the all-around best option right now. If I'm missing any details or you have anything else to add, please speak up, otherwise I'll re-close this as resolved.
        Sean Mackrory made changes -
        Status Reopened [ 4 ] Reopened [ 4 ]
        Sean Mackrory made changes -
        Status Reopened [ 4 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Roman Shaposhnik made changes -
        Fix Version/s 0.6.0 [ 12323895 ]
        Roman Shaposhnik made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Sean Mackrory
            Reporter:
            Robert Kanter
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development