Bigtop
  1. Bigtop
  2. BIGTOP-593

improve the package test to handle [arch=xx] in the repo file

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Not a Problem
    • Affects Version/s: 0.4.0
    • Fix Version/s: 0.4.0
    • Component/s: Tests
    • Labels:
      None

      Description

      the Ubuntu precise repo file contains the symbol [arch=amd64] , the current Bigtop package test doesn't handle it

        Activity

        Hide
        Johnny Zhang added a comment -

        as discussed, it is not a real, please close it

        Show
        Johnny Zhang added a comment - as discussed, it is not a real, please close it
        Hide
        Roman Shaposhnik added a comment -

        Johnny, that's part of the reason I'm confused as well. On our precise package testing EC2 slave the following works perfectly well:

        deb http://bigtop01.cloudera.org:8080/job/Bigtop-trunk-Repository/./label=precise//lastSuccessfulBuild/artifact/repo/ bigtop contrib
        

        See http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-packagetest/label=precise-slave/74/consoleFull
        for more details

        Show
        Roman Shaposhnik added a comment - Johnny, that's part of the reason I'm confused as well. On our precise package testing EC2 slave the following works perfectly well: deb http://bigtop01.cloudera.org:8080/job/Bigtop-trunk-Repository/./label=precise//lastSuccessfulBuild/artifact/repo/ bigtop contrib See http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-packagetest/label=precise-slave/74/consoleFull for more details
        Hide
        Johnny Zhang added a comment -

        np. yes, make sense, let me think about a fix.

        But seems no repo in 4.0 release has [arch=xx], the percise repo is http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-Repository/label=precise/lastSuccessfulBuild/artifact/repo/bigtop.list

        is it modified to so to work around this issue? (just make sure we can have a env to test the fix)

        Show
        Johnny Zhang added a comment - np. yes, make sense, let me think about a fix. But seems no repo in 4.0 release has [arch=xx] , the percise repo is http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-Repository/label=precise/lastSuccessfulBuild/artifact/repo/bigtop.list is it modified to so to work around this issue? (just make sure we can have a env to test the fix)
        Hide
        Roman Shaposhnik added a comment -

        After thinking about this for a quite some time (apologies for the delay) I think it would make much more sense not to special case this directly, but to modify our AptCmdLinePackageManager class to handle this additional multi-arch requirement.

        Makes sense?

        Show
        Roman Shaposhnik added a comment - After thinking about this for a quite some time (apologies for the delay) I think it would make much more sense not to special case this directly, but to modify our AptCmdLinePackageManager class to handle this additional multi-arch requirement. Makes sense?
        Hide
        Johnny Zhang added a comment -

        please discard the previous two and use BIGTOP-593_3.txt, I have verified it in my env, it works.

        Show
        Johnny Zhang added a comment - please discard the previous two and use BIGTOP-593 _3.txt, I have verified it in my env, it works.
        Hide
        Roman Shaposhnik added a comment -

        Johnny, you can test your changes by actually executing package tests:

        #!/bin/bash
        
        export HADOOP_HOME=/usr/lib/hadoop
        export HADOOP_CONF_DIR=/etc/hadoop/conf
        export REPO_FILE_URL="http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-Repository/label=${label/-slave/}/lastSuccessfulBuild/artifact/repo/bigtop"
        export REPO_KEY_URL="http://archive.apache.org/dist/incubator/bigtop/bigtop-0.3.0-incubating/repos/GPG-KEY-bigtop"
        
        if [ "$label" = "lucid-slave" ]; then
          REPO_FILE_URL="${REPO_FILE_URL}.list"
        else
          REPO_FILE_URL="${REPO_FILE_URL}.repo"
        fi
        
        env
        
        /root/slave/tools/mvn_303/bin/mvn -U -f bigtop-tests/test-execution/package/pom.xml \
          -Dcdh.repo.key.url="${REPO_KEY_URL}" -Dcdh.repo.file.url="${REPO_FILE_URL}"       \
          -Dorg.apache.bigtop.itest.log4j.level=TRACE -Dlog4j.debug=true                    \
          -Dorg.apache.maven-failsafe-plugin.testInclude='**/'$PKG_SUITE'*'                 \
        clean verify
        
        Show
        Roman Shaposhnik added a comment - Johnny, you can test your changes by actually executing package tests: #!/bin/bash export HADOOP_HOME=/usr/lib/hadoop export HADOOP_CONF_DIR=/etc/hadoop/conf export REPO_FILE_URL="http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-Repository/label=${label/-slave/}/lastSuccessfulBuild/artifact/repo/bigtop" export REPO_KEY_URL="http://archive.apache.org/dist/incubator/bigtop/bigtop-0.3.0-incubating/repos/GPG-KEY-bigtop" if [ "$label" = "lucid-slave" ]; then REPO_FILE_URL="${REPO_FILE_URL}.list" else REPO_FILE_URL="${REPO_FILE_URL}.repo" fi env /root/slave/tools/mvn_303/bin/mvn -U -f bigtop-tests/test-execution/package/pom.xml \ -Dcdh.repo.key.url="${REPO_KEY_URL}" -Dcdh.repo.file.url="${REPO_FILE_URL}" \ -Dorg.apache.bigtop.itest.log4j.level=TRACE -Dlog4j.debug=true \ -Dorg.apache.maven-failsafe-plugin.testInclude='**/'$PKG_SUITE'*' \ clean verify
        Hide
        Johnny Zhang added a comment - - edited

        I tried to verify this patch locally. My steps are:

        1. make the changes is under bigtop/bigtop-test-framework/src/main/groovy/org/apache/bigtop/itest/pmanager/ folder
        2. cd to bigtop, do 'mvn -DskipTests clean install'
        3. cd to bigtop/bigtop-tests/test-artifacts/package, do "mvn ... verify",

        I get a BUILD SUCCESS, but the test doesn't run actually (apply repo key, repo file), install package...
        How can I verify the fix by running the test? Thanks.

        # mvn -f pom.xml -Dcom.cloudera.itest.log4j.level=TRACE -Dcdh.repo.host=$REPOHOST -Dcdh.repo.file.url.${flavor}=http://$REPOHOST/$repofile -Dcdh.repo.url=http://$REPOHOST/$repourl -Dcdh.repo.key.url=http://$REPOHOST/$repokeyurl -Dorg.apache.maven-failsafe-plugin.testInclude=**/TestPackagesReadiness.* -Darch.platform='[arch=amd64]' -Darch.platform="[arch=amd64]" verify
        [INFO] Scanning for projects...
        [INFO]                                                                         
        [INFO] ------------------------------------------------------------------------
        [INFO] Building packagesmoke 0.4.0-incubating-SNAPSHOT
        [INFO] ------------------------------------------------------------------------
        [INFO] 
        [INFO] --- gmaven-plugin:1.0:generateStubs (default) @ package-smoke ---
        [INFO]  Generated 24 Java stubs
        [INFO] 
        [INFO] --- maven-remote-resources-plugin:1.1:process (default) @ package-smoke ---
        [INFO] Setting property: classpath.resource.loader.class => 'org.codehaus.plexus.velocity.ContextClassLoaderResourceLoader'.
        [INFO] Setting property: velocimacro.messages.on => 'false'.
        [INFO] Setting property: resource.loader => 'classpath'.
        [INFO] Setting property: resource.manager.logwhenfound => 'false'.
        [INFO] 
        [INFO] --- maven-resources-plugin:2.4.3:resources (default-resources) @ package-smoke ---
        [INFO] Using 'UTF-8' encoding to copy filtered resources.
        [INFO] Copying 161 resources
        [INFO] Copying 3 resources
        [INFO] 
        [INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ package-smoke ---
        [INFO] Nothing to compile - all classes are up to date
        [INFO] 
        [INFO] --- gmaven-plugin:1.0:compile (default) @ package-smoke ---
        [INFO]  Compiled 82 Groovy classes
        [INFO] 
        [INFO] --- gmaven-plugin:1.0:generateTestStubs (default) @ package-smoke ---
        [INFO]  No sources found for Java stub generation
        [INFO] 
        [INFO] --- maven-resources-plugin:2.4.3:testResources (default-testResources) @ package-smoke ---
        [INFO] Using 'UTF-8' encoding to copy filtered resources.
        [INFO] skip non existing resourceDirectory /root/bigtop/bigtop-tests/test-artifacts/package/src/test/resources
        [INFO] Copying 3 resources
        [INFO] 
        [INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ package-smoke ---
        [INFO] No sources to compile
        [INFO] 
        [INFO] --- gmaven-plugin:1.0:testCompile (default) @ package-smoke ---
        [INFO]  No sources found to compile
        [INFO] 
        [INFO] --- gmaven-plugin:1.0:execute (check-manifest) @ package-smoke ---
        Manifest XML validated
        [INFO] 
        [INFO] --- maven-surefire-plugin:2.7:test (default-test) @ package-smoke ---
        [INFO] Tests are skipped.
        [INFO] 
        [INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ package-smoke ---
        [INFO] Building jar: /root/bigtop/bigtop-tests/test-artifacts/package/target/package-smoke-0.4.0-incubating-SNAPSHOT.jar
        [INFO] 
        [INFO] --- maven-failsafe-plugin:2.7.1:integration-test (integration-test) @ package-smoke ---
        [INFO] No tests to run.
        [INFO] 
        [INFO] --- maven-failsafe-plugin:2.7.1:verify (verify) @ package-smoke ---
        [INFO] No tests to run.
        [INFO] ------------------------------------------------------------------------
        [INFO] BUILD SUCCESS
        [INFO] ------------------------------------------------------------------------
        [INFO] Total time: 13.413s
        [INFO] Finished at: Fri May 18 16:56:24 PDT 2012
        [INFO] Final Memory: 15M/62M
        [INFO] ------------------------------------------------------------------------
        
        Show
        Johnny Zhang added a comment - - edited I tried to verify this patch locally. My steps are: 1. make the changes is under bigtop/bigtop-test-framework/src/main/groovy/org/apache/bigtop/itest/pmanager/ folder 2. cd to bigtop, do 'mvn -DskipTests clean install' 3. cd to bigtop/bigtop-tests/test-artifacts/package, do "mvn ... verify", I get a BUILD SUCCESS, but the test doesn't run actually (apply repo key, repo file), install package... How can I verify the fix by running the test? Thanks. # mvn -f pom.xml -Dcom.cloudera.itest.log4j.level=TRACE -Dcdh.repo.host=$REPOHOST -Dcdh.repo.file.url.${flavor}=http://$REPOHOST/$repofile -Dcdh.repo.url=http://$REPOHOST/$repourl -Dcdh.repo.key.url=http://$REPOHOST/$repokeyurl -Dorg.apache.maven-failsafe-plugin.testInclude=**/TestPackagesReadiness.* -Darch.platform='[arch=amd64]' -Darch.platform="[arch=amd64]" verify [INFO] Scanning for projects... [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building packagesmoke 0.4.0-incubating-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- gmaven-plugin:1.0:generateStubs (default) @ package-smoke --- [INFO] Generated 24 Java stubs [INFO] [INFO] --- maven-remote-resources-plugin:1.1:process (default) @ package-smoke --- [INFO] Setting property: classpath.resource.loader.class => 'org.codehaus.plexus.velocity.ContextClassLoaderResourceLoader'. [INFO] Setting property: velocimacro.messages.on => 'false'. [INFO] Setting property: resource.loader => 'classpath'. [INFO] Setting property: resource.manager.logwhenfound => 'false'. [INFO] [INFO] --- maven-resources-plugin:2.4.3:resources (default-resources) @ package-smoke --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 161 resources [INFO] Copying 3 resources [INFO] [INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ package-smoke --- [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] --- gmaven-plugin:1.0:compile (default) @ package-smoke --- [INFO] Compiled 82 Groovy classes [INFO] [INFO] --- gmaven-plugin:1.0:generateTestStubs (default) @ package-smoke --- [INFO] No sources found for Java stub generation [INFO] [INFO] --- maven-resources-plugin:2.4.3:testResources (default-testResources) @ package-smoke --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /root/bigtop/bigtop-tests/test-artifacts/package/src/test/resources [INFO] Copying 3 resources [INFO] [INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ package-smoke --- [INFO] No sources to compile [INFO] [INFO] --- gmaven-plugin:1.0:testCompile (default) @ package-smoke --- [INFO] No sources found to compile [INFO] [INFO] --- gmaven-plugin:1.0:execute (check-manifest) @ package-smoke --- Manifest XML validated [INFO] [INFO] --- maven-surefire-plugin:2.7:test (default-test) @ package-smoke --- [INFO] Tests are skipped. [INFO] [INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ package-smoke --- [INFO] Building jar: /root/bigtop/bigtop-tests/test-artifacts/package/target/package-smoke-0.4.0-incubating-SNAPSHOT.jar [INFO] [INFO] --- maven-failsafe-plugin:2.7.1:integration-test (integration-test) @ package-smoke --- [INFO] No tests to run. [INFO] [INFO] --- maven-failsafe-plugin:2.7.1:verify (verify) @ package-smoke --- [INFO] No tests to run. [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 13.413s [INFO] Finished at: Fri May 18 16:56:24 PDT 2012 [INFO] Final Memory: 15M/62M [INFO] ------------------------------------------------------------------------
        Hide
        Johnny Zhang added a comment - - edited

        new attached BIGTOP_593_2.txt
        this is more straight forward fix, which get rid of the if else block

        Show
        Johnny Zhang added a comment - - edited new attached BIGTOP_593_2.txt this is more straight forward fix, which get rid of the if else block
        Hide
        Johnny Zhang added a comment -

        the patch only makes the change for apt, not sure if the [arch=xx] label will be a trend for yum, zypper, if it is the case, I can change them too

        Show
        Johnny Zhang added a comment - the patch only makes the change for apt, not sure if the [arch=xx] label will be a trend for yum, zypper, if it is the case, I can change them too

          People

          • Assignee:
            Johnny Zhang
            Reporter:
            Johnny Zhang
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development