Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-11911

test-patch should allow configuration of default branch

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None

      Description

      right now test-patch.sh forces a default branch of 'trunk'. would be better to allow it to be configurable.

      1. HADOOP-11911.4.patch
        3 kB
        Sean Busbey
      2. HADOOP-11911.3.patch
        3 kB
        Sean Busbey
      3. HADOOP-11911.2.patch
        3 kB
        Sean Busbey
      4. HADOOP-11911.1.patch
        3 kB
        Sean Busbey

        Activity

        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #186 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/186/)
        HADOOP-11911. test-patch should allow configuration of default branch (Sean Busbey via aw) (aw: rev 9b01f81eb874cd63e6b9ae2d09d94fc8bf4fcd7d)

        • hadoop-common-project/hadoop-common/CHANGES.txt
        • dev-support/test-patch.sh
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #186 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/186/ ) HADOOP-11911 . test-patch should allow configuration of default branch (Sean Busbey via aw) (aw: rev 9b01f81eb874cd63e6b9ae2d09d94fc8bf4fcd7d) hadoop-common-project/hadoop-common/CHANGES.txt dev-support/test-patch.sh
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2135 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2135/)
        HADOOP-11911. test-patch should allow configuration of default branch (Sean Busbey via aw) (aw: rev 9b01f81eb874cd63e6b9ae2d09d94fc8bf4fcd7d)

        • hadoop-common-project/hadoop-common/CHANGES.txt
        • dev-support/test-patch.sh
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2135 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2135/ ) HADOOP-11911 . test-patch should allow configuration of default branch (Sean Busbey via aw) (aw: rev 9b01f81eb874cd63e6b9ae2d09d94fc8bf4fcd7d) hadoop-common-project/hadoop-common/CHANGES.txt dev-support/test-patch.sh
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #176 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/176/)
        HADOOP-11911. test-patch should allow configuration of default branch (Sean Busbey via aw) (aw: rev 9b01f81eb874cd63e6b9ae2d09d94fc8bf4fcd7d)

        • dev-support/test-patch.sh
        • hadoop-common-project/hadoop-common/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #176 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/176/ ) HADOOP-11911 . test-patch should allow configuration of default branch (Sean Busbey via aw) (aw: rev 9b01f81eb874cd63e6b9ae2d09d94fc8bf4fcd7d) dev-support/test-patch.sh hadoop-common-project/hadoop-common/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2117 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2117/)
        HADOOP-11911. test-patch should allow configuration of default branch (Sean Busbey via aw) (aw: rev 9b01f81eb874cd63e6b9ae2d09d94fc8bf4fcd7d)

        • hadoop-common-project/hadoop-common/CHANGES.txt
        • dev-support/test-patch.sh
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2117 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2117/ ) HADOOP-11911 . test-patch should allow configuration of default branch (Sean Busbey via aw) (aw: rev 9b01f81eb874cd63e6b9ae2d09d94fc8bf4fcd7d) hadoop-common-project/hadoop-common/CHANGES.txt dev-support/test-patch.sh
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Yarn-trunk #919 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/919/)
        HADOOP-11911. test-patch should allow configuration of default branch (Sean Busbey via aw) (aw: rev 9b01f81eb874cd63e6b9ae2d09d94fc8bf4fcd7d)

        • hadoop-common-project/hadoop-common/CHANGES.txt
        • dev-support/test-patch.sh
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #919 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/919/ ) HADOOP-11911 . test-patch should allow configuration of default branch (Sean Busbey via aw) (aw: rev 9b01f81eb874cd63e6b9ae2d09d94fc8bf4fcd7d) hadoop-common-project/hadoop-common/CHANGES.txt dev-support/test-patch.sh
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #185 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/185/)
        HADOOP-11911. test-patch should allow configuration of default branch (Sean Busbey via aw) (aw: rev 9b01f81eb874cd63e6b9ae2d09d94fc8bf4fcd7d)

        • hadoop-common-project/hadoop-common/CHANGES.txt
        • dev-support/test-patch.sh
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #185 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/185/ ) HADOOP-11911 . test-patch should allow configuration of default branch (Sean Busbey via aw) (aw: rev 9b01f81eb874cd63e6b9ae2d09d94fc8bf4fcd7d) hadoop-common-project/hadoop-common/CHANGES.txt dev-support/test-patch.sh
        Hide
        aw Allen Wittenauer added a comment -

        +1 committed to the real source tree and branch-2.

        Show
        aw Allen Wittenauer added a comment - +1 committed to the real source tree and branch-2.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #7734 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7734/)
        HADOOP-11911. test-patch should allow configuration of default branch (Sean Busbey via aw) (aw: rev 9b01f81eb874cd63e6b9ae2d09d94fc8bf4fcd7d)

        • dev-support/test-patch.sh
        • hadoop-common-project/hadoop-common/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7734 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7734/ ) HADOOP-11911 . test-patch should allow configuration of default branch (Sean Busbey via aw) (aw: rev 9b01f81eb874cd63e6b9ae2d09d94fc8bf4fcd7d) dev-support/test-patch.sh hadoop-common-project/hadoop-common/CHANGES.txt
        Hide
        hadoopqa Hadoop QA added a comment -



        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s dev-support patch detected.
        0 pre-patch 0m 0s Pre-patch trunk compilation is healthy.
        0 @author 0m 0s Skipping @author checks as test-patch has been patched.
        +1 release audit 0m 14s The applied patch does not increase the total number of release audit warnings.
        0 shellcheck 0m 14s Shellcheck was not available.
        +1 whitespace 0m 1s The patch has no lines that end in whitespace.
            0m 24s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12730498/HADOOP-11911.4.patch
        Optional Tests shellcheck
        git revision trunk / 9356cf8
        Java 1.7.0_55
        uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6484/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s dev-support patch detected. 0 pre-patch 0m 0s Pre-patch trunk compilation is healthy. 0 @author 0m 0s Skipping @author checks as test-patch has been patched. +1 release audit 0m 14s The applied patch does not increase the total number of release audit warnings. 0 shellcheck 0m 14s Shellcheck was not available. +1 whitespace 0m 1s The patch has no lines that end in whitespace.     0m 24s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12730498/HADOOP-11911.4.patch Optional Tests shellcheck git revision trunk / 9356cf8 Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6484/console This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -

        A patch to test-patch or smart-apply-patch has been detected.
        Re-executing against the patched versions to perform further tests.
        The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/6484/console in case of problems.

        Show
        hadoopqa Hadoop QA added a comment - A patch to test-patch or smart-apply-patch has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/6484/console in case of problems.
        Hide
        busbey Sean Busbey added a comment -

        -04

        • correct shellcheck errors.
        Show
        busbey Sean Busbey added a comment - -04 correct shellcheck errors.
        Hide
        aw Allen Wittenauer added a comment -

        shellcheck says there are two new errors with this patch:

        ./dev-support/test-patch.sh:840:29: note: Double quote to prevent globbing and word splitting. [SC2086]
        ./dev-support/test-patch.sh:1373:14: note: Double quote to prevent globbing and word splitting. [SC2086]
        
        Show
        aw Allen Wittenauer added a comment - shellcheck says there are two new errors with this patch: ./dev-support/test-patch.sh:840:29: note: Double quote to prevent globbing and word splitting. [SC2086] ./dev-support/test-patch.sh:1373:14: note: Double quote to prevent globbing and word splitting. [SC2086]
        Hide
        aw Allen Wittenauer added a comment -

        So then I'd just do a force checkout to PATCH_BRANCH_DEFAULT?

        Yup.

        Here's the good news: this is all very fast and relatively cheap in the grand scheme of things. The biggest (and unavoidable) cost is the initial pull when we're already on trunk.. err PATCH_BRANCH_DEFAULT.

        Also keep in mind that you pretty much MUST do a fetch or a pull prior to determining the branch. Otherwise you won't be able to test against branches you don't know about yet...

        Show
        aw Allen Wittenauer added a comment - So then I'd just do a force checkout to PATCH_BRANCH_DEFAULT? Yup. Here's the good news: this is all very fast and relatively cheap in the grand scheme of things. The biggest (and unavoidable) cost is the initial pull when we're already on trunk.. err PATCH_BRANCH_DEFAULT. Also keep in mind that you pretty much MUST do a fetch or a pull prior to determining the branch. Otherwise you won't be able to test against branches you don't know about yet...
        Hide
        busbey Sean Busbey added a comment -

        quote
        oh! that's why you do the second set of pull --rebase. yikes.
        quote

        This "yikes" should be read as "the student was enlightened", btw.

        Show
        busbey Sean Busbey added a comment - quote oh! that's why you do the second set of pull --rebase. yikes. quote This "yikes" should be read as "the student was enlightened", btw.
        Hide
        hadoopqa Hadoop QA added a comment -



        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s dev-support patch detected.
        0 pre-patch 0m 0s Pre-patch trunk compilation is healthy.
        0 @author 0m 0s Skipping @author checks as test-patch has been patched.
        +1 release audit 0m 14s The applied patch does not increase the total number of release audit warnings.
        0 shellcheck 0m 14s Shellcheck was not available.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
            0m 23s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12730200/HADOOP-11911.3.patch
        Optional Tests shellcheck
        git revision trunk / 8f65c79
        Java 1.7.0_55
        uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6476/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s dev-support patch detected. 0 pre-patch 0m 0s Pre-patch trunk compilation is healthy. 0 @author 0m 0s Skipping @author checks as test-patch has been patched. +1 release audit 0m 14s The applied patch does not increase the total number of release audit warnings. 0 shellcheck 0m 14s Shellcheck was not available. +1 whitespace 0m 0s The patch has no lines that end in whitespace.     0m 23s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12730200/HADOOP-11911.3.patch Optional Tests shellcheck git revision trunk / 8f65c79 Java 1.7.0_55 uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6476/console This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -

        A patch to test-patch or smart-apply-patch has been detected.
        Re-executing against the patched versions to perform further tests.
        The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/6476/console in case of problems.

        Show
        hadoopqa Hadoop QA added a comment - A patch to test-patch or smart-apply-patch has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/6476/console in case of problems.
        Hide
        busbey Sean Busbey added a comment -

        -03

        • revert to prior checkout, rebase, branch, rebase workflow.
        • force checkout to default branch instead of trunk
        • add a clarifying note about the second rebase.
        Show
        busbey Sean Busbey added a comment - -03 revert to prior checkout, rebase, branch, rebase workflow. force checkout to default branch instead of trunk add a clarifying note about the second rebase.
        Hide
        busbey Sean Busbey added a comment -

        oh! that's why you do the second set of pull --rebase. yikes.

        Show
        busbey Sean Busbey added a comment - oh! that's why you do the second set of pull --rebase. yikes.
        Hide
        busbey Sean Busbey added a comment -

        So then I'd just do a force checkout to PATCH_BRANCH_DEFAULT?

        Doesn't the pull --rebase only update trunk in that case? For patches that are on long-lived feature branches, won't they remain stale? I guess this is sounding like a different jira.

        Show
        busbey Sean Busbey added a comment - So then I'd just do a force checkout to PATCH_BRANCH_DEFAULT? Doesn't the pull --rebase only update trunk in that case? For patches that are on long-lived feature branches, won't they remain stale? I guess this is sounding like a different jira.
        Hide
        aw Allen Wittenauer added a comment - - edited

        that would be better because as is if the repo ends up not on branch then git pull --rebase will fail because it won't know what to rebase.

        BTW, that doesn't happen with the --force checkout to trunk....

        So really, if you remove all of those changes and leave just the ones that set and work with PATCH_BRANCH_DEFAULT, this could get committed...

        Show
        aw Allen Wittenauer added a comment - - edited that would be better because as is if the repo ends up not on branch then git pull --rebase will fail because it won't know what to rebase. BTW, that doesn't happen with the --force checkout to trunk.... So really, if you remove all of those changes and leave just the ones that set and work with PATCH_BRANCH_DEFAULT, this could get committed...
        Hide
        aw Allen Wittenauer added a comment -

        FWIW, my normal test universe these days it to have two clones of Apache's repo: one named test and one named work. The former is used for test-patch and the latter is where I'm doing things. Then in the parent of those two directories, I have a simple script:

        $#/bin/bash
        export BUILD_URL=https://builds.apache.org/job/PreCommit-HADOOP-Build/fake/
        work/dev-support/test-patch.sh --basedir=test $@
        

        Then I can pass --resetrepo (or --dirty-workspace if I'm testing two patches together) as needed.

        Show
        aw Allen Wittenauer added a comment - FWIW, my normal test universe these days it to have two clones of Apache's repo: one named test and one named work. The former is used for test-patch and the latter is where I'm doing things. Then in the parent of those two directories, I have a simple script: $#/bin/bash export BUILD_URL=https: //builds.apache.org/job/PreCommit-HADOOP-Build/fake/ work/dev-support/test-patch.sh --basedir=test $@ Then I can pass --resetrepo (or --dirty-workspace if I'm testing two patches together) as needed.
        Hide
        aw Allen Wittenauer added a comment -

        Right. the --dirty-workspace + --resetrepo modes changes the rules a bit. There is zero guarantee that the repo is a fresh one.

        Show
        aw Allen Wittenauer added a comment - Right. the --dirty-workspace + --resetrepo modes changes the rules a bit. There is zero guarantee that the repo is a fresh one.
        Hide
        busbey Sean Busbey added a comment -

        that would be better because as is if the repo ends up not on branch then git pull --rebase will fail because it won't know what to rebase.

        Show
        busbey Sean Busbey added a comment - that would be better because as is if the repo ends up not on branch then git pull --rebase will fail because it won't know what to rebase.
        Hide
        busbey Sean Busbey added a comment -

        ah! that means you have a local git repo that isn't getting wiped out between runs. I can update to rebase any local branches.

        Show
        busbey Sean Busbey added a comment - ah! that means you have a local git repo that isn't getting wiped out between runs. I can update to rebase any local branches.
        Hide
        aw Allen Wittenauer added a comment -

        updated to use git fetch instead of git pull --rebase sine all we need is a fetch and we immediately then do a checkout.

        I originally had this and it didn't always update non-checked out branches, IIRC. Then when a branch switch happened, it wasn't the most recent commit.

        Show
        aw Allen Wittenauer added a comment - updated to use git fetch instead of git pull --rebase sine all we need is a fetch and we immediately then do a checkout. I originally had this and it didn't always update non-checked out branches, IIRC. Then when a branch switch happened, it wasn't the most recent commit.
        Hide
        hadoopqa Hadoop QA added a comment -



        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s dev-support patch detected.
        0 pre-patch 0m 0s Pre-patch trunk compilation is healthy.
        0 @author 0m 0s Skipping @author checks as test-patch has been patched.
        +1 release audit 0m 14s The applied patch does not increase the total number of release audit warnings.
        0 shellcheck 0m 14s Shellcheck was not available.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
            0m 16s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12730076/HADOOP-11911.2.patch
        Optional Tests shellcheck
        git revision trunk / a319771
        Java 1.7.0_55
        uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6468/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s dev-support patch detected. 0 pre-patch 0m 0s Pre-patch trunk compilation is healthy. 0 @author 0m 0s Skipping @author checks as test-patch has been patched. +1 release audit 0m 14s The applied patch does not increase the total number of release audit warnings. 0 shellcheck 0m 14s Shellcheck was not available. +1 whitespace 0m 0s The patch has no lines that end in whitespace.     0m 16s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12730076/HADOOP-11911.2.patch Optional Tests shellcheck git revision trunk / a319771 Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6468/console This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -

        A patch to test-patch or smart-apply-patch has been detected.
        Re-executing against the patched versions to perform further tests.
        The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/6468/console in case of problems.

        Show
        hadoopqa Hadoop QA added a comment - A patch to test-patch or smart-apply-patch has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/6468/console in case of problems.
        Hide
        busbey Sean Busbey added a comment -

        -02

        • updated to use git fetch instead of git pull --rebase sine all we need is a fetch and we immediately then do a checkout.
        Show
        busbey Sean Busbey added a comment - -02 updated to use git fetch instead of git pull --rebase sine all we need is a fetch and we immediately then do a checkout.
        Hide
        busbey Sean Busbey added a comment -

        makes the default branch configurable by cli, defaulting to 'trunk'

        also removed an extra checkout --force trunk done before we determine the branch (it's a no-op because we then do another forced checkout to whatever the patch branch is, including trunk).

        also removed an extra pull --rebase done after we do our final forced checkout.

        Show
        busbey Sean Busbey added a comment - makes the default branch configurable by cli, defaulting to 'trunk' also removed an extra checkout --force trunk done before we determine the branch (it's a no-op because we then do another forced checkout to whatever the patch branch is, including trunk). also removed an extra pull --rebase done after we do our final forced checkout.

          People

          • Assignee:
            busbey Sean Busbey
            Reporter:
            busbey Sean Busbey
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development