Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.0.0
    • Component/s: build, debian, rpm
    • Labels:
      None

      Description

      Bump up Apache Pig version to 0.14.0

      1. BIGTOP-1594.1.patch
        16 kB
        YoungWoo Kim
      2. BIGTOP-1594.2.patch
        16 kB
        YoungWoo Kim
      3. BIGTOP-1594.3.patch
        17 kB
        YoungWoo Kim
      4. BIGTOP-1594.4.patch
        17 kB
        YoungWoo Kim
      5. BIGTOP-1594.5.patch
        17 kB
        YoungWoo Kim
      6. BIGTOP-1594.6.patch
        17 kB
        YoungWoo Kim

        Activity

        Hide
        warwithin YoungWoo Kim added a comment -

        First patch.

        Show
        warwithin YoungWoo Kim added a comment - First patch.
        Hide
        warwithin YoungWoo Kim added a comment -

        Revised patch:

        • Fix invalid symlinks
        Show
        warwithin YoungWoo Kim added a comment - Revised patch: Fix invalid symlinks
        Hide
        evans_ye Evans Ye added a comment - - edited

        Hey YoungWoo Kim, awesome patch!
        I've tested it on bigtop docker build slave:

        docker run -ti --rm -v /root/bigtop:/bigtop bigtop/slaves:centos-6.4 bash -l
        cd /bigtop; ./gradew pig-rpm
        

        The 0.14 RPM has been successfully built.
        I also ran a pig job against the built RPM on docker-puppet cluster and it works like a charm!
        So +1 on the RPM side.
        Will take a further look on Debian tomorrow.
        In general, I think we pretty much will get this in shortly!

        Show
        evans_ye Evans Ye added a comment - - edited Hey YoungWoo Kim , awesome patch! I've tested it on bigtop docker build slave: docker run -ti --rm -v /root/bigtop:/bigtop bigtop/slaves:centos-6.4 bash -l cd /bigtop; ./gradew pig-rpm The 0.14 RPM has been successfully built. I also ran a pig job against the built RPM on docker-puppet cluster and it works like a charm! So +1 on the RPM side. Will take a further look on Debian tomorrow. In general, I think we pretty much will get this in shortly!
        Hide
        cos Konstantin Boudnik added a comment -

        Olaf Flebbe, you mentioned that you had some issues with this version of the Pig - could you please chime in?
        YoungWoo Kim in the other ticket you've referred to some documentation build issue. If this is still the case - could you put more details in here? Thanks.

        Show
        cos Konstantin Boudnik added a comment - Olaf Flebbe , you mentioned that you had some issues with this version of the Pig - could you please chime in? YoungWoo Kim in the other ticket you've referred to some documentation build issue. If this is still the case - could you put more details in here? Thanks.
        Hide
        warwithin YoungWoo Kim added a comment -

        Revised patch (BIGTOP-1594.3.patch):

        • Remove unused dir
        • Fix a build issue on examples
          # cd /usr/share/doc/pig-0.14.0/examples/
          # ant
          Buildfile: /usr/share/doc/pig-0.14.0/examples/build.xml
          
          ...
          BUILD SUCCESSFUL
          Total time: 3 seconds
          # ls -als pigtutorial.tar.gz 
          10224 -rw-r--r-- 1 root root 10467298 2015-01-14 11:40 pigtutorial.tar.gz
          # 
           
        Show
        warwithin YoungWoo Kim added a comment - Revised patch ( BIGTOP-1594 .3.patch): Remove unused dir Fix a build issue on examples # cd /usr/share/doc/pig-0.14.0/examples/ # ant Buildfile: /usr/share/doc/pig-0.14.0/examples/build.xml ... BUILD SUCCESSFUL Total time: 3 seconds # ls -als pigtutorial.tar.gz 10224 -rw-r--r-- 1 root root 10467298 2015-01-14 11:40 pigtutorial.tar.gz #
        Hide
        evans_ye Evans Ye added a comment -

        Nice catch Konstantin Boudnik. I was naively assuming that HDP2.2 somehow indicating 0.14 is ready.
        The revised patch can have RPM successfully built, but I don't have luck on the deb side, will have more try on it.

        Show
        evans_ye Evans Ye added a comment - Nice catch Konstantin Boudnik . I was naively assuming that HDP2.2 somehow indicating 0.14 is ready. The revised patch can have RPM successfully built, but I don't have luck on the deb side, will have more try on it.
        Hide
        warwithin YoungWoo Kim added a comment -

        Evans,
        Please elaborate on the failures. I'll look into that.

        Show
        warwithin YoungWoo Kim added a comment - Evans, Please elaborate on the failures. I'll look into that.
        Hide
        warwithin YoungWoo Kim added a comment -

        Revised patch (BIGTOP-1594.4.patch):

        • Make Deb packaging works.

        Evans,
        I fixed failures with Deb packaging, so please try the latest patch.

        Show
        warwithin YoungWoo Kim added a comment - Revised patch ( BIGTOP-1594 .4.patch): Make Deb packaging works. Evans, I fixed failures with Deb packaging, so please try the latest patch.
        Hide
        evans_ye Evans Ye added a comment -

        Thanks for the update!
        I'm testing now, will give comments tomorrow.

        Show
        evans_ye Evans Ye added a comment - Thanks for the update! I'm testing now, will give comments tomorrow.
        Hide
        cos Konstantin Boudnik added a comment -

        Well, what do you see in HDP or CDH doesn't reflect the reality of Apache release. It could be an official release + X patches; where X could be quite large and not all x(X) might be even published.

        Show
        cos Konstantin Boudnik added a comment - Well, what do you see in HDP or CDH doesn't reflect the reality of Apache release. It could be an official release + X patches; where X could be quite large and not all x(X) might be even published.
        Hide
        evans_ye Evans Ye added a comment -

        YoungWoo Kim the build works now!
        I'll go further to test in on cluster.
        recall Olaf Flebbe's comment: Pig 0.14.0 had other issues. It did not build as well for me.
        Could you point out the issue?

        Show
        evans_ye Evans Ye added a comment - YoungWoo Kim the build works now! I'll go further to test in on cluster. recall Olaf Flebbe 's comment: Pig 0.14.0 had other issues. It did not build as well for me. Could you point out the issue?
        Hide
        evans_ye Evans Ye added a comment - - edited

        That's true, cos. Distributions always have their release contains tuns of back ported features.
        But in HDP2.2 it shows(as the link I provided) only PIG-4138 has been back ported to its pig-0.14.0. So they're almost the same

        Show
        evans_ye Evans Ye added a comment - - edited That's true, cos. Distributions always have their release contains tuns of back ported features. But in HDP2.2 it shows(as the link I provided) only PIG-4138 has been back ported to its pig-0.14.0. So they're almost the same
        Hide
        evans_ye Evans Ye added a comment -

        OK, the test result is good.
        I think it's ready to commit on the functionality aspect.
        So +1 by me.
        But I'm not quite familiar with our build system.
        It would be great if someone can take a glance on the patch, Konstantin Boudnik, maybe?

        Show
        evans_ye Evans Ye added a comment - OK, the test result is good. I think it's ready to commit on the functionality aspect. So +1 by me. But I'm not quite familiar with our build system. It would be great if someone can take a glance on the patch, Konstantin Boudnik , maybe?
        Hide
        rvs Roman Shaposhnik added a comment -

        The patch looks good. A few nits, though:

        • we seem to be missing /usr/lib/pig/ {CHANGES,LICENSE,NOTICE,README,RELEASE_NOTES}

          .txt

        • /usr/lib/pig/conf is no longer a symlink to /etc/...
        • /usr/lib/pig/pig.jar symlink is gone
        • /usr/share/doc/pig-0.14.0/api is empty
        Show
        rvs Roman Shaposhnik added a comment - The patch looks good. A few nits, though: we seem to be missing /usr/lib/pig/ {CHANGES,LICENSE,NOTICE,README,RELEASE_NOTES} .txt /usr/lib/pig/conf is no longer a symlink to /etc/... /usr/lib/pig/pig.jar symlink is gone /usr/share/doc/pig-0.14.0/api is empty
        Hide
        warwithin YoungWoo Kim added a comment -

        Evans Ye Thanks for testing and reviewing.

        Roman Shaposhnik Thanks for your comments. I'll update the patch.

        Show
        warwithin YoungWoo Kim added a comment - Evans Ye Thanks for testing and reviewing. Roman Shaposhnik Thanks for your comments. I'll update the patch.
        Hide
        evans_ye Evans Ye added a comment -

        No prob YoungWoo Kim. I'm also learning.

        Show
        evans_ye Evans Ye added a comment - No prob YoungWoo Kim . I'm also learning.
        Hide
        warwithin YoungWoo Kim added a comment - - edited

        Updated patch (BIGTOP-1594.5.patch):

        • /usr/lib/pig/*.txt
        • Fix the symlink for /usr/lib/pig/conf and /etc/pig/conf
        • Fix the javadoc issue
          # ls -als
          합계 4768
             4 drwxr-xr-x   5 root root    4096 2015-01-19 14:45 .
             4 dr-xr-xr-x. 22 root root    4096 2015-01-19 14:44 ..
           176 -rw-r--r--   1 root root  178355 2015-01-19 14:11 CHANGES.txt
            12 -rw-r--r--   1 root root   11358 2015-01-19 14:11 LICENSE.txt
             4 -rw-r--r--   1 root root    2125 2015-01-19 14:11 NOTICE.txt
             4 -rw-r--r--   1 root root    1307 2015-01-19 14:11 README.txt
             4 -rw-r--r--   1 root root    1871 2015-01-19 14:11 RELEASE_NOTES.txt
             4 drwxr-xr-x   2 root root    4096 2015-01-19 14:44 bin
             0 lrwxrwxrwx   1 root root      13 2015-01-19 14:44 conf -> /etc/pig/conf
             4 drwxr-xr-x   3 root root    4096 2015-01-19 14:45 lib
          4156 -rw-r--r--   1 root root 4253917 2015-01-19 14:11 pig-0.14.0-core-h2.jar
            12 -rw-r--r--   1 root root    9746 2015-01-19 14:12 pig-0.14.0-smoketests.jar
           380 -rw-r--r--   1 root root  385490 2015-01-19 14:12 piggybank.jar
           
          # ls -als /usr/share/doc/pig-0.14.0/api/
          합계 5892
             4 drwxr-xr-x 4 root root    4096 2015-01-19 14:45 .
             4 drwxr-xr-x 7 root root    4096 2015-01-19 14:45 ..
           220 -rw-r--r-- 1 root root  223749 2015-01-19 14:11 allclasses-frame.html
           196 -rw-r--r-- 1 root root  198669 2015-01-19 14:11 allclasses-noframe.html
           200 -rw-r--r-- 1 root root  203721 2015-01-19 14:11 constant-values.html
            36 -rw-r--r-- 1 root root   36532 2015-01-19 14:11 deprecated-list.html
            12 -rw-r--r-- 1 root root    8661 2015-01-19 14:11 help-doc.html
          4508 -rw-r--r-- 1 root root 4613073 2015-01-19 14:11 index-all.html
             4 -rw-r--r-- 1 root root    2832 2015-01-19 14:11 index.html
             4 drwxr-xr-x 3 root root    4096 2015-01-19 14:45 org
            12 -rw-r--r-- 1 root root   11178 2015-01-19 14:11 overview-frame.html
            24 -rw-r--r-- 1 root root   24141 2015-01-19 14:11 overview-summary.html
           380 -rw-r--r-- 1 root root  388231 2015-01-19 14:11 overview-tree.html
             4 -rw-r--r-- 1 root root    2905 2015-01-19 14:11 package-list
             4 drwxr-xr-x 2 root root    4096 2015-01-19 14:45 resources
           268 -rw-r--r-- 1 root root  272562 2015-01-19 14:11 serialized-form.html
            12 -rw-r--r-- 1 root root   11139 2015-01-19 14:11 stylesheet.css
          
           

        I removed /usr/lib/pig.jar symlink. in 0.14.0, uber jar is removed and '$PIG_HOME/bin/pig' pick up $PIG_HOME/pig*core*.jar automatically.

        Roman Shaposhnik Please review above changes. thanks.

        Show
        warwithin YoungWoo Kim added a comment - - edited Updated patch ( BIGTOP-1594 .5.patch): /usr/lib/pig/*.txt Fix the symlink for /usr/lib/pig/conf and /etc/pig/conf Fix the javadoc issue # ls -als 합계 4768 4 drwxr-xr-x 5 root root 4096 2015-01-19 14:45 . 4 dr-xr-xr-x. 22 root root 4096 2015-01-19 14:44 .. 176 -rw-r--r-- 1 root root 178355 2015-01-19 14:11 CHANGES.txt 12 -rw-r--r-- 1 root root 11358 2015-01-19 14:11 LICENSE.txt 4 -rw-r--r-- 1 root root 2125 2015-01-19 14:11 NOTICE.txt 4 -rw-r--r-- 1 root root 1307 2015-01-19 14:11 README.txt 4 -rw-r--r-- 1 root root 1871 2015-01-19 14:11 RELEASE_NOTES.txt 4 drwxr-xr-x 2 root root 4096 2015-01-19 14:44 bin 0 lrwxrwxrwx 1 root root 13 2015-01-19 14:44 conf -> /etc/pig/conf 4 drwxr-xr-x 3 root root 4096 2015-01-19 14:45 lib 4156 -rw-r--r-- 1 root root 4253917 2015-01-19 14:11 pig-0.14.0-core-h2.jar 12 -rw-r--r-- 1 root root 9746 2015-01-19 14:12 pig-0.14.0-smoketests.jar 380 -rw-r--r-- 1 root root 385490 2015-01-19 14:12 piggybank.jar # ls -als /usr/share/doc/pig-0.14.0/api/ 합계 5892 4 drwxr-xr-x 4 root root 4096 2015-01-19 14:45 . 4 drwxr-xr-x 7 root root 4096 2015-01-19 14:45 .. 220 -rw-r--r-- 1 root root 223749 2015-01-19 14:11 allclasses-frame.html 196 -rw-r--r-- 1 root root 198669 2015-01-19 14:11 allclasses-noframe.html 200 -rw-r--r-- 1 root root 203721 2015-01-19 14:11 constant-values.html 36 -rw-r--r-- 1 root root 36532 2015-01-19 14:11 deprecated-list.html 12 -rw-r--r-- 1 root root 8661 2015-01-19 14:11 help-doc.html 4508 -rw-r--r-- 1 root root 4613073 2015-01-19 14:11 index-all.html 4 -rw-r--r-- 1 root root 2832 2015-01-19 14:11 index.html 4 drwxr-xr-x 3 root root 4096 2015-01-19 14:45 org 12 -rw-r--r-- 1 root root 11178 2015-01-19 14:11 overview-frame.html 24 -rw-r--r-- 1 root root 24141 2015-01-19 14:11 overview-summary.html 380 -rw-r--r-- 1 root root 388231 2015-01-19 14:11 overview-tree.html 4 -rw-r--r-- 1 root root 2905 2015-01-19 14:11 package-list 4 drwxr-xr-x 2 root root 4096 2015-01-19 14:45 resources 268 -rw-r--r-- 1 root root 272562 2015-01-19 14:11 serialized-form.html 12 -rw-r--r-- 1 root root 11139 2015-01-19 14:11 stylesheet.css I removed /usr/lib/pig.jar symlink. in 0.14.0, uber jar is removed and '$PIG_HOME/bin/pig' pick up $PIG_HOME/pig* core *.jar automatically. Roman Shaposhnik Please review above changes. thanks.
        Hide
        oflebbe Olaf Flebbe added a comment -

        Please add an

        export ANT_OPTS=-Xmx4096m
        before running ant in do_component_build

        ant dies while building java docs for me without increasing java memory limits.

        With this change pig compiles and an package is created.

        Show
        oflebbe Olaf Flebbe added a comment - Please add an export ANT_OPTS=-Xmx4096m before running ant in do_component_build ant dies while building java docs for me without increasing java memory limits. With this change pig compiles and an package is created.
        Hide
        warwithin YoungWoo Kim added a comment -

        Olaf Flebbe Thanks for your comments I'll update the patch.

        Show
        warwithin YoungWoo Kim added a comment - Olaf Flebbe Thanks for your comments I'll update the patch.
        Hide
        warwithin YoungWoo Kim added a comment -

        Updated patch (BIGTOP-1594.6.patch):

        • Add 'ANT_OPTS' env var
        Show
        warwithin YoungWoo Kim added a comment - Updated patch ( BIGTOP-1594 .6.patch): Add 'ANT_OPTS' env var
        Hide
        rvs Roman Shaposhnik added a comment -

        +1 and committed! Thanks for the patch!

        Show
        rvs Roman Shaposhnik added a comment - +1 and committed! Thanks for the patch!
        Hide
        rvs Roman Shaposhnik added a comment -

        YoungWoo Kim

        Actually, I spoke too soon. It appears that if executed under non-root the build fails as follows:

        [exec] BUILD FAILED
             [exec] /usr/local/apache-forrest/main/targets/plugins.xml:374: Directory /usr/local/apache-forrest/build/plugins creation was not successful for an unknown reason
        

        Can you, please, take care of the forrest issue?

        As an aside: I though Pig migrated away from forrest. Am I missing something?

        Show
        rvs Roman Shaposhnik added a comment - YoungWoo Kim Actually, I spoke too soon. It appears that if executed under non-root the build fails as follows: [exec] BUILD FAILED [exec] /usr/local/apache-forrest/main/targets/plugins.xml:374: Directory /usr/local/apache-forrest/build/plugins creation was not successful for an unknown reason Can you, please, take care of the forrest issue? As an aside: I though Pig migrated away from forrest. Am I missing something?
        Hide
        warwithin YoungWoo Kim added a comment -

        Roman Shaposhnik I'll look into that.

        Show
        warwithin YoungWoo Kim added a comment - Roman Shaposhnik I'll look into that.
        Hide
        rvs Roman Shaposhnik added a comment -

        YoungWoo Kim thanks! Also note, that your patch is now applied, so make sure that you post a solution on top of the current state of our source tree.

        Show
        rvs Roman Shaposhnik added a comment - YoungWoo Kim thanks! Also note, that your patch is now applied, so make sure that you post a solution on top of the current state of our source tree.
        Hide
        oflebbe Olaf Flebbe added a comment - - edited

        This issue is already addressed by BIGTOP-1596

        Just run the current bigtop_toolchain puppet scripts. However BIGTOP-1610 has to be applied now as well after running the toolchain

        If there is another solution for this I will back out BIGTOP-1596 again for sure.

        Show
        oflebbe Olaf Flebbe added a comment - - edited This issue is already addressed by BIGTOP-1596 Just run the current bigtop_toolchain puppet scripts. However BIGTOP-1610 has to be applied now as well after running the toolchain If there is another solution for this I will back out BIGTOP-1596 again for sure.
        Hide
        warwithin YoungWoo Kim added a comment -

        Olaf Flebbe Thanks for your explanation.

        Show
        warwithin YoungWoo Kim added a comment - Olaf Flebbe Thanks for your explanation.

          People

          • Assignee:
            warwithin YoungWoo Kim
            Reporter:
            warwithin YoungWoo Kim
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development