Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-1488

bigtop_toolchain hardcode the apache mirror sites for downloading maven

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 1.0.0
    • Component/s: build, general
    • Labels:
      None

      Description

      In the bigtop_toolchain module, the puppet modules use hardcoded apache mirror sites to download component tar balls(for example, use ftp://mirror.reverse.net/pub/apache/maven/maven-3/3.0.5/binaries/apache-maven-3.0.5-bin.tar.gz to download maven).
      In some cases the hardcoded mirror maybe very slow, causing the installation to be very slow or even fail.
      We can use the "www.apache.org/dyn/closer.cgi" to choose apache mirrors automatically and faster the installatin.

      1. BIGTOP-1488.v1.patch
        5 kB
        Newton Alex
      2. BIGTOP-1488.v2.patch
        4 kB
        Newton Alex

        Activity

        Hide
        cos Konstantin Boudnik added a comment -

        That be great! Could you provide a patch?!

        Show
        cos Konstantin Boudnik added a comment - That be great! Could you provide a patch?!
        Hide
        stanley_shi stanley shi added a comment -

        Here's my plan:
        1. implement a custom function and put at "bigtop_toolchain/lib/puppet/parser/functions/nearest_apache_mirror.rb", it will return the nearest apache mirror by parsing the json response from the URL "https://www.apache.org/dyn/closer.cgi?as_json=1";
        2. in the puppet code, use the custom function to replace the apache mirror prefixes;

        But I have a questions that cannot be decided:
        how to parsing the http/json response from the apache site:
        a. the ideal way would be to use ruby json gem; but this will need more dependency to be pulled in;
        b. another way is to parse the json string as simple string and extract the content from it;
        I am not sure which is the better way;

        Show
        stanley_shi stanley shi added a comment - Here's my plan: 1. implement a custom function and put at "bigtop_toolchain/lib/puppet/parser/functions/nearest_apache_mirror.rb", it will return the nearest apache mirror by parsing the json response from the URL "https://www.apache.org/dyn/closer.cgi?as_json=1"; 2. in the puppet code, use the custom function to replace the apache mirror prefixes; But I have a questions that cannot be decided: how to parsing the http/json response from the apache site: a. the ideal way would be to use ruby json gem; but this will need more dependency to be pulled in; b. another way is to parse the json string as simple string and extract the content from it; I am not sure which is the better way;
        Hide
        cos Konstantin Boudnik added a comment -

        I'd literally go with simple grep by pattern and pick the first entry, eg

        curl --stderr /dev/null https://www.apache.org/dyn/closer.cgi?as_json=1 | grep "http:\/\/" | head -1 | sed -e 's#"##g' -e 's#,##' | tr -s '[:blank:]'
        
        Show
        cos Konstantin Boudnik added a comment - I'd literally go with simple grep by pattern and pick the first entry, eg curl --stderr /dev/ null https: //www.apache.org/dyn/closer.cgi?as_json=1 | grep "http:\/\/" | head -1 | sed -e 's#"##g' -e 's#,##' | tr -s '[:blank:]'
        Hide
        rvs Roman Shaposhnik added a comment -

        +1 to what Konstantin Boudnik said!

        Show
        rvs Roman Shaposhnik added a comment - +1 to what Konstantin Boudnik said!
        Hide
        stanley_shi stanley shi added a comment -

        Thanks, I will use grep to do this:
        curl --stderr /dev/null https://www.apache.org/dyn/closer.cgi?as_json=1 | grep preferred |cut -d '"' -f 4

        Show
        stanley_shi stanley shi added a comment - Thanks, I will use grep to do this: curl --stderr /dev/null https://www.apache.org/dyn/closer.cgi?as_json=1 | grep preferred |cut -d '"' -f 4
        Hide
        nalex Newton Alex added a comment -

        Patch Attached

        Show
        nalex Newton Alex added a comment - Patch Attached
        Hide
        cos Konstantin Boudnik added a comment -

        Looks ok. Are ant and maven the only once we download from an Apache mirror? (considering the Ant will go away with next Pig release it doesn't look worthy doing this just for one tarball; but whatever).

        I will try to run it and if it works I will commit it. Thanks Alex!

        Show
        cos Konstantin Boudnik added a comment - Looks ok. Are ant and maven the only once we download from an Apache mirror? (considering the Ant will go away with next Pig release it doesn't look worthy doing this just for one tarball; but whatever). I will try to run it and if it works I will commit it. Thanks Alex!
        Hide
        cos Konstantin Boudnik added a comment - - edited

        With patch

        sudo puppet apply --modulepath=`pwd` -e "include bigtop_toolchain::ant"

        fails

        Could not autoload nearest_apache_mirror: /home/cos/workspaces/bigtop/bigtop_toolchain/lib/puppet/parser/functions/nearest_apache_mirror.rb:10: invalid regular expression; there's no previous pattern, to which '*' would define cardinality at 1: /*
         * Licensed to the Apache Software Foundation (ASF) under one
         * or more contributor license agreements.  See the NOTICE file
         * distributed with this work for additional information
         * regarding copyright ownership.  The ASF licenses this file
         * to you under the Apache License, Version 2.0 (the
         * "License"); you may not use this file except in compliance
         * with the License.  You may obtain a copy of the License at
         *
         *     http:/
        /home/cos/workspaces/bigtop/bigtop_toolchain/lib/puppet/parser/functions/nearest_apache_mirror.rb:12: syntax error, unexpected tSTAR, expecting '='
         * Unless required by applicable law or agreed to in writing, software
          ^
        /home/cos/workspaces/bigtop/bigtop_toolchain/lib/puppet/parser/functions/nearest_apache_mirror.rb:12: syntax error, unexpected kIN, expecting $end
         * Unless required by applicable law or agreed to in writing, software
                                                            ^ at /home/cos/workspaces/bigtop/bigtop_toolchain/manifests/deps.pp:19 on node tcos.sanr.wandisco.com
        
        Show
        cos Konstantin Boudnik added a comment - - edited With patch sudo puppet apply --modulepath=`pwd` -e "include bigtop_toolchain::ant" fails Could not autoload nearest_apache_mirror: /home/cos/workspaces/bigtop/bigtop_toolchain/lib/puppet/parser/functions/nearest_apache_mirror.rb:10: invalid regular expression; there's no previous pattern, to which '*' would define cardinality at 1: /* * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file * distributed with this work for additional information * regarding copyright ownership. The ASF licenses this file * to you under the Apache License, Version 2.0 (the * "License"); you may not use this file except in compliance * with the License. You may obtain a copy of the License at * * http:/ /home/cos/workspaces/bigtop/bigtop_toolchain/lib/puppet/parser/functions/nearest_apache_mirror.rb:12: syntax error, unexpected tSTAR, expecting '=' * Unless required by applicable law or agreed to in writing, software ^ /home/cos/workspaces/bigtop/bigtop_toolchain/lib/puppet/parser/functions/nearest_apache_mirror.rb:12: syntax error, unexpected kIN, expecting $end * Unless required by applicable law or agreed to in writing, software ^ at /home/cos/workspaces/bigtop/bigtop_toolchain/manifests/deps.pp:19 on node tcos.sanr.wandisco.com
        Hide
        nalex Newton Alex added a comment -

        Revised patch attached.

        Konstantin Boudnik Sorry about the delay. Have attached the revised patch.

        Here are the test results:
        -----------------
        bash-4.1# puppet apply --modulepath=`pwd` -e "include bigtop_toolchain::ant"
        notice: /Stage[main]/Bigtop_toolchain::Packages/Package[libtool]/ensure: created
        notice: /Stage[main]/Bigtop_toolchain::Packages/Package[git]/ensure: created
        notice: /Stage[main]/Bigtop_toolchain::Packages/Package[gcc-c++]/ensure: created
        notice: /Stage[main]/Bigtop_toolchain::Packages/Package[openssl-devel]/ensure: created
        notice: /Stage[main]/Bigtop_toolchain::Packages/Package[createrepo]/ensure: created
        notice: /Stage[main]/Bigtop_toolchain::Packages/Package[rpm-build]/ensure: created
        notice: /Stage[main]/Bigtop_toolchain::Packages/Package[fuse]/ensure: created
        notice: /Stage[main]/Bigtop_toolchain::Packages/Package[cppunit-devel]/ensure: created
        notice: /Stage[main]/Bigtop_toolchain::Packages/Package[lzo-devel]/ensure: created
        notice: /Stage[main]/Bigtop_toolchain::Packages/Package[cmake]/ensure: created
        notice: /Stage[main]/Bigtop_toolchain::Packages/Package[redhat-rpm-config]/ensure: created
        notice: /Stage[main]/Bigtop_toolchain::Packages/Package[fuse-devel]/ensure: created
        notice: /Stage[main]/Bigtop_toolchain::Deps/Exec[/usr/bin/wget http://archive.apache.org/dist/forrest/0.9/apache-forrest-0.9.tar.gz]/returns: executed successfully
        notice: /Stage[main]/Bigtop_toolchain::Deps/Exec[/usr/bin/wget https://launchpad.net/ubuntu/+archive/primary/+files/libprotobuf8_2.5.0-9ubuntu1_amd64.deb]/returns: executed successfully
        notice: /Stage[main]/Bigtop_toolchain::Deps/Exec[/usr/bin/wget http://mirror.reverse.net/pub/apache//maven/maven-3/3.0.5/binaries/apache-maven-3.0.5-bin.tar.gz]/returns: executed successfully
        notice: /Stage[main]/Bigtop_toolchain::Deps/Exec[/usr/bin/wget http://services.gradle.org/distributions/gradle-2.0-bin.zip]/returns: executed successfully
        notice: /Stage[main]/Bigtop_toolchain::Deps/Exec[/usr/bin/wget https://launchpad.net/ubuntu/+archive/primary/+files/protobuf-compiler_2.5.0-9ubuntu1_amd64.deb]/returns: executed successfully
        notice: /Stage[main]/Bigtop_toolchain::Deps/Exec[/usr/bin/wget http://mirror.reverse.net/pub/apache//ant/binaries/apache-ant-1.9.4-bin.tar.gz]/returns: executed successfully
        notice: /Stage[main]/Bigtop_toolchain::Ant/Exec[/bin/tar xvzf /usr/src/apache-ant-1.9.4-bin.tar.gz]: Triggered 'refresh' from 1 events
        notice: /Stage[main]/Bigtop_toolchain::Ant/File[/usr/local/ant]/ensure: created
        notice: /Stage[main]/Bigtop_toolchain::Deps/Exec[/usr/bin/wget https://launchpad.net/ubuntu/+archive/primary/+files/libprotoc8_2.5.0-9ubuntu1_amd64.deb]/returns: executed successfully
        notice: Finished catalog run in 162.71 seconds
        bash-4.1#

        ------------------

        Show
        nalex Newton Alex added a comment - Revised patch attached. Konstantin Boudnik Sorry about the delay. Have attached the revised patch. Here are the test results: ----------------- bash-4.1# puppet apply --modulepath=`pwd` -e "include bigtop_toolchain::ant" notice: /Stage [main] /Bigtop_toolchain::Packages/Package [libtool] /ensure: created notice: /Stage [main] /Bigtop_toolchain::Packages/Package [git] /ensure: created notice: /Stage [main] /Bigtop_toolchain::Packages/Package [gcc-c++] /ensure: created notice: /Stage [main] /Bigtop_toolchain::Packages/Package [openssl-devel] /ensure: created notice: /Stage [main] /Bigtop_toolchain::Packages/Package [createrepo] /ensure: created notice: /Stage [main] /Bigtop_toolchain::Packages/Package [rpm-build] /ensure: created notice: /Stage [main] /Bigtop_toolchain::Packages/Package [fuse] /ensure: created notice: /Stage [main] /Bigtop_toolchain::Packages/Package [cppunit-devel] /ensure: created notice: /Stage [main] /Bigtop_toolchain::Packages/Package [lzo-devel] /ensure: created notice: /Stage [main] /Bigtop_toolchain::Packages/Package [cmake] /ensure: created notice: /Stage [main] /Bigtop_toolchain::Packages/Package [redhat-rpm-config] /ensure: created notice: /Stage [main] /Bigtop_toolchain::Packages/Package [fuse-devel] /ensure: created notice: /Stage [main] /Bigtop_toolchain::Deps/Exec [/usr/bin/wget http://archive.apache.org/dist/forrest/0.9/apache-forrest-0.9.tar.gz] /returns: executed successfully notice: /Stage [main] /Bigtop_toolchain::Deps/Exec [/usr/bin/wget https://launchpad.net/ubuntu/+archive/primary/+files/libprotobuf8_2.5.0-9ubuntu1_amd64.deb] /returns: executed successfully notice: /Stage [main] /Bigtop_toolchain::Deps/Exec [/usr/bin/wget http://mirror.reverse.net/pub/apache//maven/maven-3/3.0.5/binaries/apache-maven-3.0.5-bin.tar.gz] /returns: executed successfully notice: /Stage [main] /Bigtop_toolchain::Deps/Exec [/usr/bin/wget http://services.gradle.org/distributions/gradle-2.0-bin.zip] /returns: executed successfully notice: /Stage [main] /Bigtop_toolchain::Deps/Exec [/usr/bin/wget https://launchpad.net/ubuntu/+archive/primary/+files/protobuf-compiler_2.5.0-9ubuntu1_amd64.deb] /returns: executed successfully notice: /Stage [main] /Bigtop_toolchain::Deps/Exec [/usr/bin/wget http://mirror.reverse.net/pub/apache//ant/binaries/apache-ant-1.9.4-bin.tar.gz] /returns: executed successfully notice: /Stage [main] /Bigtop_toolchain::Ant/Exec [/bin/tar xvzf /usr/src/apache-ant-1.9.4-bin.tar.gz] : Triggered 'refresh' from 1 events notice: /Stage [main] /Bigtop_toolchain::Ant/File [/usr/local/ant] /ensure: created notice: /Stage [main] /Bigtop_toolchain::Deps/Exec [/usr/bin/wget https://launchpad.net/ubuntu/+archive/primary/+files/libprotoc8_2.5.0-9ubuntu1_amd64.deb] /returns: executed successfully notice: Finished catalog run in 162.71 seconds bash-4.1# ------------------
        Hide
        cos Konstantin Boudnik added a comment -

        It works and looks good. +1
        Committing.

        Show
        cos Konstantin Boudnik added a comment - It works and looks good. +1 Committing.
        Hide
        cos Konstantin Boudnik added a comment -

        Committed and pushed as
        70c8dec..7d406ee HEAD -> master

        Thanks Alex!

        Show
        cos Konstantin Boudnik added a comment - Committed and pushed as 70c8dec..7d406ee HEAD -> master Thanks Alex!

          People

          • Assignee:
            nalex Newton Alex
            Reporter:
            stanley_shi stanley shi
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development