Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-1830

Move apache-forrest installation from bigtop_toolchain to pig compile

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 1.0.0
    • Component/s: build
    • Labels:
      None

      Description

      The apache forrest installation within bigtop_toolchain is a bad idea.

      apache-forrest is only needed AFAIK for "pig"

      It has to be made r/w in bigtop_toolchain for all in order to work, since usage of forrest tries to write into forrest.

      The solution is to download it in do-component-build, unpack and use it privately there.

      In order to address security issues: Download from https and check md5sum

        Issue Links

          Activity

          Hide
          oflebbe Olaf Flebbe added a comment -

          Konstantin Boudnik Thanks for reviewing and encouragement to look for a better solution

          Show
          oflebbe Olaf Flebbe added a comment - Konstantin Boudnik Thanks for reviewing and encouragement to look for a better solution
          Hide
          cos Konstantin Boudnik added a comment -

          Sorry, I haven't realized there's a newer version of it. Thanks!
          +1 please commit

          Show
          cos Konstantin Boudnik added a comment - Sorry, I haven't realized there's a newer version of it. Thanks! +1 please commit
          Hide
          oflebbe Olaf Flebbe added a comment -

          The patch is final from my side and tested on Debian and Centos

          Show
          oflebbe Olaf Flebbe added a comment - The patch is final from my side and tested on Debian and Centos
          Hide
          cos Konstantin Boudnik added a comment -

          I like the idea. It is clean/easy/and seem to simplify out stuff quite a bit. Looking forward for the final version of it!

          Show
          cos Konstantin Boudnik added a comment - I like the idea. It is clean/easy/and seem to simplify out stuff quite a bit. Looking forward for the final version of it!
          Hide
          oflebbe Olaf Flebbe added a comment -

          Download once would be easy, just using option -N for wget. Will even reload file, if it is corrupted in the first place. But where to cache the file?

          Unfortunately apache forrest is not deployed in Maven, so I cannot misuse Maven for download and cache an source artifact.

          So I came up with the idea to store the download in the maven repository itself in an private directory which should not harm normal operation. Since Users will have a lot of stuff downloaded in $HOME/.m2 the apache-forrest download will likely not overflow anything. If users want to have a clean bootstrap, the will remote ~/.m2/repository , so user experience should be o.k.

          Konstantin Boudnik What do you think ?

          Show
          oflebbe Olaf Flebbe added a comment - Download once would be easy, just using option -N for wget. Will even reload file, if it is corrupted in the first place. But where to cache the file? Unfortunately apache forrest is not deployed in Maven, so I cannot misuse Maven for download and cache an source artifact. So I came up with the idea to store the download in the maven repository itself in an private directory which should not harm normal operation. Since Users will have a lot of stuff downloaded in $HOME/.m2 the apache-forrest download will likely not overflow anything. If users want to have a clean bootstrap, the will remote ~/.m2/repository , so user experience should be o.k. Konstantin Boudnik What do you think ?
          Hide
          cos Konstantin Boudnik added a comment -

          I love the idea! I wish we could get rid of Forrest completely ;(
          One possible improvement (please feel free to reject if my premise is wrong): Downloading Forrest on every build seems to be a bit costly - it's a large tarball afterall. Can we optimize it somehow, so we set it up only on the first Pig build and the consequent will be reusing the earlier copy? I am not sure if this will safe us any significant time, hence my offer to reject the idea.

          Other than that - the patch looks good. +1
          Please commit if you choose not to make the optimization. Thanks!

          Show
          cos Konstantin Boudnik added a comment - I love the idea! I wish we could get rid of Forrest completely ;( One possible improvement (please feel free to reject if my premise is wrong): Downloading Forrest on every build seems to be a bit costly - it's a large tarball afterall. Can we optimize it somehow, so we set it up only on the first Pig build and the consequent will be reusing the earlier copy? I am not sure if this will safe us any significant time, hence my offer to reject the idea. Other than that - the patch looks good. +1 Please commit if you choose not to make the optimization. Thanks!

            People

            • Assignee:
              oflebbe Olaf Flebbe
              Reporter:
              oflebbe Olaf Flebbe
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development