Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-6938

Convert build to work with Git rather than SVN.

    Details

    • Type: Task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 6.0, 5.x
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      We assume an SVN checkout in parts of our build and will need to move to assuming a Git checkout.

      Patches against https://github.com/dweiss/lucene-solr-svn2git from LUCENE-6933.

      1. LUCENE-6938-wc-checker.patch
        5 kB
        Uwe Schindler
      2. LUCENE-6938-wc-checker.patch
        5 kB
        Uwe Schindler
      3. LUCENE-6938-1.patch
        10 kB
        Uwe Schindler
      4. LUCENE-6938.patch
        10 kB
        Mark Miller
      5. LUCENE-6938.patch
        11 kB
        Mark Miller
      6. LUCENE-6938.patch
        10 kB
        Mark Miller
      7. LUCENE-6938.patch
        10 kB
        Mark Miller

        Issue Links

          Activity

          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit c19b1b90c9812c48712fbaf1b55af49169993766 in lucene-solr's branch refs/heads/branch_5x from Mike McCandless
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=c19b1b9 ]

          LUCENE-6938: addVersion can no longer do a --reord-only merge with git
          (cherry picked from commit 2514521)

          Show
          jira-bot ASF subversion and git services added a comment - Commit c19b1b90c9812c48712fbaf1b55af49169993766 in lucene-solr's branch refs/heads/branch_5x from Mike McCandless [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=c19b1b9 ] LUCENE-6938 : addVersion can no longer do a --reord-only merge with git (cherry picked from commit 2514521)
          Hide
          janhoy Jan Høydahl added a comment -

          Opened LUCENE-7155 for this, keeping this JIRA as resolved.

          Show
          janhoy Jan Høydahl added a comment - Opened LUCENE-7155 for this, keeping this JIRA as resolved.
          Hide
          rjernst Ryan Ernst added a comment -

          It looks like the branch detection logic isn't working given our current naming conventions we have in git now. This is the current logic, which worked in svn:

            if branchName == b'master':
              return 'master'
            if branchName.startswith(b'branch_'):
              return 'stable'
            if branchName.startswith(b'lucene_solr_'):
              return 'release'
          

          But we now name our release branches like branch_5_5 instead of lucene_solr_5_5. So in this case, the script thought it was a stable branch, and thus the version was added as deprecated.

          Show
          rjernst Ryan Ernst added a comment - It looks like the branch detection logic isn't working given our current naming conventions we have in git now. This is the current logic, which worked in svn: if branchName == b'master': return 'master' if branchName.startswith(b'branch_'): return 'stable' if branchName.startswith(b'lucene_solr_'): return 'release' But we now name our release branches like branch_5_5 instead of lucene_solr_5_5 . So in this case, the script thought it was a stable branch, and thus the version was added as deprecated.
          Hide
          elyograg Shawn Heisey added a comment -

          Apologies for the incorrect credit.

          I've never done it, but ReleaseTodo seems to indicate that running addVersion is done as one of the early steps in the process.

          For major and minor releases, I think this makes sense, because you'll be making a new branch early in the process. At that point the parent branch should get a version bump, and the subsequent release work will be done in the new branch, presumably with the correct version number already present.

          For bugfix releases, I think it makes more sense to run addVersion just after tagging the release – one of the last steps. That would have prevented the problem I ran into. I ran "ant package" on branch_5_5 some time after 5.5.0 was fully released, but I got 5.5.0-SNAPSHOT filenames instead of 5.5.1-SNAPSHOT.

          Regarding the brand-new bug-fix Version entry being deprecated: This makes no sense to me, especially since it causes an immediate test failure in the test that the addVersion script runs after making changes. I can see with "git diff" that the script did correctly add deprecation annotations to LUCENE_5_5_0.

          If the addVersion script were being used to add 5.5.1 to one of the 6x branches or the master branch, then it WOULD make sense for the new entry to be deprecated. Perhaps I was not using the script correctly for my use case, or the script needs some detection code or the option you mentioned to skip deprecation.

          Show
          elyograg Shawn Heisey added a comment - Apologies for the incorrect credit. I've never done it, but ReleaseTodo seems to indicate that running addVersion is done as one of the early steps in the process. For major and minor releases, I think this makes sense, because you'll be making a new branch early in the process. At that point the parent branch should get a version bump, and the subsequent release work will be done in the new branch, presumably with the correct version number already present. For bugfix releases, I think it makes more sense to run addVersion just after tagging the release – one of the last steps. That would have prevented the problem I ran into. I ran "ant package" on branch_5_5 some time after 5.5.0 was fully released, but I got 5.5.0-SNAPSHOT filenames instead of 5.5.1-SNAPSHOT. Regarding the brand-new bug-fix Version entry being deprecated: This makes no sense to me, especially since it causes an immediate test failure in the test that the addVersion script runs after making changes. I can see with "git diff" that the script did correctly add deprecation annotations to LUCENE_5_5_0. If the addVersion script were being used to add 5.5.1 to one of the 6x branches or the master branch, then it WOULD make sense for the new entry to be deprecated. Perhaps I was not using the script correctly for my use case, or the script needs some detection code or the option you mentioned to skip deprecation.
          Hide
          janhoy Jan Høydahl added a comment -

          It was I who cherry-picked Mike's fix to the 5_5 branch, but the git mail bot logs author, not committer
          I also saw the same regarding deprecation. Had to remove the deprecations manually, then tests pass. Should the script have a switch for skipping deprecation?
          A bit unclear to me after reading RM docs: Should bugfix version bump be performed by RM after releasing previous minor release, or by the be done by the RM for the bugfix release just prior to release?

          Show
          janhoy Jan Høydahl added a comment - It was I who cherry-picked Mike's fix to the 5_5 branch, but the git mail bot logs author, not committer I also saw the same regarding deprecation. Had to remove the deprecations manually, then tests pass. Should the script have a switch for skipping deprecation? A bit unclear to me after reading RM docs: Should bugfix version bump be performed by RM after releasing previous minor release, or by the be done by the RM for the bugfix release just prior to release?
          Hide
          hossman Hoss Man added a comment -

          that sounds correct ... on the 5.5 branch LUCENE_5_5_1 should not be deprecated ... but on all subsequent branches (6x, 6_1, master) it should be.

          Show
          hossman Hoss Man added a comment - that sounds correct ... on the 5.5 branch LUCENE_5_5_1 should not be deprecated ... but on all subsequent branches (6x, 6_1, master) it should be.
          Hide
          elyograg Shawn Heisey added a comment -

          Thanks, Michael McCandless. That last commit made the script run with a "5.5.1" argument.

          One problem, though: It added the new LUCENE_5_5_1 version as already deprecated, which caused ant test -Dtestcase=TestVersion to fail. Removing the deprecation allowed the test to pass.

          Show
          elyograg Shawn Heisey added a comment - Thanks, Michael McCandless . That last commit made the script run with a "5.5.1" argument. One problem, though: It added the new LUCENE_5_5_1 version as already deprecated, which caused ant test -Dtestcase=TestVersion to fail. Removing the deprecation allowed the test to pass.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 7105d451777c2c30e7f2b48260265b352fbb3472 in lucene-solr's branch refs/heads/branch_5_5 from Mike McCandless
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=7105d45 ]

          LUCENE-6938: addVersion can no longer do a --reord-only merge with git
          (cherry picked from commit 2514521)

          Show
          jira-bot ASF subversion and git services added a comment - Commit 7105d451777c2c30e7f2b48260265b352fbb3472 in lucene-solr's branch refs/heads/branch_5_5 from Mike McCandless [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=7105d45 ] LUCENE-6938 : addVersion can no longer do a --reord-only merge with git (cherry picked from commit 2514521)
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit ef965661abf108f10b3da78aaec27576a7ef00c7 in lucene-solr's branch refs/heads/master from Steve Rowe
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=ef96566 ]

          LUCENE-7024, LUCENE-6938: fix smoke tester to pull pom.xml.template files from the Solr source distribution instead of from svn
          (cherry-picked branch_5_5 commit f6a1bbf)

          Show
          jira-bot ASF subversion and git services added a comment - Commit ef965661abf108f10b3da78aaec27576a7ef00c7 in lucene-solr's branch refs/heads/master from Steve Rowe [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=ef96566 ] LUCENE-7024 , LUCENE-6938 : fix smoke tester to pull pom.xml.template files from the Solr source distribution instead of from svn (cherry-picked branch_5_5 commit f6a1bbf)
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 0bc239b16a1f411ab9c3426dbf8190019356edc0 in lucene-solr's branch refs/heads/branch_5x from Steve Rowe
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=0bc239b ]

          LUCENE-7204, LUCENE-6938: fix smoke tester to pull pom.xml.template files from the Solr source distribution instead of from svn

          Show
          jira-bot ASF subversion and git services added a comment - Commit 0bc239b16a1f411ab9c3426dbf8190019356edc0 in lucene-solr's branch refs/heads/branch_5x from Steve Rowe [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=0bc239b ] LUCENE-7204 , LUCENE-6938 : fix smoke tester to pull pom.xml.template files from the Solr source distribution instead of from svn
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit f6a1bbf8bec8b83ce68b697b9905048b44ef80f6 in lucene-solr's branch refs/heads/branch_5_5 from Steve Rowe
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=f6a1bbf ]

          LUCENE-7204, LUCENE-6938: fix smoke tester to pull pom.xml.template files from the Solr source distribution instead of from svn

          Show
          jira-bot ASF subversion and git services added a comment - Commit f6a1bbf8bec8b83ce68b697b9905048b44ef80f6 in lucene-solr's branch refs/heads/branch_5_5 from Steve Rowe [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=f6a1bbf ] LUCENE-7204 , LUCENE-6938 : fix smoke tester to pull pom.xml.template files from the Solr source distribution instead of from svn
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 4d094975d96455fae1877b0a7ee2dafef83a5828 in lucene-solr's branch refs/heads/master from Mike McCandless
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=4d09497 ]

          LUCENE-6938: fix some places in smoke tester to accept git commit hash instead of svn revison

          Conflicts:
          dev-tools/scripts/smokeTestRelease.py

          Show
          jira-bot ASF subversion and git services added a comment - Commit 4d094975d96455fae1877b0a7ee2dafef83a5828 in lucene-solr's branch refs/heads/master from Mike McCandless [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=4d09497 ] LUCENE-6938 : fix some places in smoke tester to accept git commit hash instead of svn revison Conflicts: dev-tools/scripts/smokeTestRelease.py
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 383c3ca976a3f216afbfdfbff02619a21dbd2d9c in lucene-solr's branch refs/heads/branch_5x from Mike McCandless
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=383c3ca ]

          LUCENE-6938: fix some places in smoke tester to accept git commit hash instead of svn revison

          Show
          jira-bot ASF subversion and git services added a comment - Commit 383c3ca976a3f216afbfdfbff02619a21dbd2d9c in lucene-solr's branch refs/heads/branch_5x from Mike McCandless [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=383c3ca ] LUCENE-6938 : fix some places in smoke tester to accept git commit hash instead of svn revison
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 0cd6d229b3c7d73e7d83eb687c2862fa8eeeb703 in lucene-solr's branch refs/heads/branch_5_5 from Mike McCandless
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=0cd6d22 ]

          LUCENE-6938: fix some places in smoke tester to accept git commit hash instead of svn revison

          Show
          jira-bot ASF subversion and git services added a comment - Commit 0cd6d229b3c7d73e7d83eb687c2862fa8eeeb703 in lucene-solr's branch refs/heads/branch_5_5 from Mike McCandless [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=0cd6d22 ] LUCENE-6938 : fix some places in smoke tester to accept git commit hash instead of svn revison
          Hide
          dsmiley David Smiley added a comment -

          +1 to that! It can be frustrating to forget to push and find other people expectantly not seeing what you said you did.

          Show
          dsmiley David Smiley added a comment - +1 to that! It can be frustrating to forget to push and find other people expectantly not seeing what you said you did.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 7509b9c9691514c3f6c231a35f29e340b3484fc1 in lucene-solr's branch refs/heads/master from Mike McCandless
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=7509b9c ]

          LUCENE-6938: add TODO that we should also detect unpushed commits

          Show
          jira-bot ASF subversion and git services added a comment - Commit 7509b9c9691514c3f6c231a35f29e340b3484fc1 in lucene-solr's branch refs/heads/master from Mike McCandless [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=7509b9c ] LUCENE-6938 : add TODO that we should also detect unpushed commits
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit de9d4ac3b7370df8e3fd5418b7811ecb44c62998 in lucene-solr's branch refs/heads/master from Mike McCandless
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=de9d4ac ]

          LUCENE-6938: fix buggy detection of dirty git checkout

          Show
          jira-bot ASF subversion and git services added a comment - Commit de9d4ac3b7370df8e3fd5418b7811ecb44c62998 in lucene-solr's branch refs/heads/master from Mike McCandless [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=de9d4ac ] LUCENE-6938 : fix buggy detection of dirty git checkout
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 9fe6a8f20a8e6ff1fded6379a096db2e390c9675 in lucene-solr's branch refs/heads/branch_5x from Mike McCandless
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=9fe6a8f ]

          LUCENE-6938: add TODO that we should also detect unpushed commits

          Show
          jira-bot ASF subversion and git services added a comment - Commit 9fe6a8f20a8e6ff1fded6379a096db2e390c9675 in lucene-solr's branch refs/heads/branch_5x from Mike McCandless [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=9fe6a8f ] LUCENE-6938 : add TODO that we should also detect unpushed commits
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 3989732f70df3981ad0533929f64af3a8d30b92d in lucene-solr's branch refs/heads/branch_5x from Mike McCandless
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=3989732 ]

          LUCENE-6938: fix buggy detection of dirty git checkout

          Show
          jira-bot ASF subversion and git services added a comment - Commit 3989732f70df3981ad0533929f64af3a8d30b92d in lucene-solr's branch refs/heads/branch_5x from Mike McCandless [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=3989732 ] LUCENE-6938 : fix buggy detection of dirty git checkout
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 9a4ff707ed8d4aafa20dd3cc9c0fd4c7378046f2 in lucene-solr's branch refs/heads/branch_5_5 from Mike McCandless
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=9a4ff70 ]

          LUCENE-6938: add TODO that we should also detect unpushed commits

          Show
          jira-bot ASF subversion and git services added a comment - Commit 9a4ff707ed8d4aafa20dd3cc9c0fd4c7378046f2 in lucene-solr's branch refs/heads/branch_5_5 from Mike McCandless [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=9a4ff70 ] LUCENE-6938 : add TODO that we should also detect unpushed commits
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit e3a06f0334034c1280f416b4afd6a6249cda395e in lucene-solr's branch refs/heads/branch_5_5 from Mike McCandless
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=e3a06f0 ]

          LUCENE-6938: fix buggy detection of dirty git checkout

          Show
          jira-bot ASF subversion and git services added a comment - Commit e3a06f0334034c1280f416b4afd6a6249cda395e in lucene-solr's branch refs/heads/branch_5_5 from Mike McCandless [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=e3a06f0 ] LUCENE-6938 : fix buggy detection of dirty git checkout
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit f8be973b9473a250ba50746a0b548f6521f012ed in lucene-solr's branch refs/heads/master from Mike McCandless
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=f8be973 ]

          LUCENE-6938: switch from svn to git

          Show
          jira-bot ASF subversion and git services added a comment - Commit f8be973b9473a250ba50746a0b548f6521f012ed in lucene-solr's branch refs/heads/master from Mike McCandless [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=f8be973 ] LUCENE-6938 : switch from svn to git
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 144ab5b107aeb7cf79d7e5fd97b8ff5d2ba2ba2f in lucene-solr's branch refs/heads/branch_5x from Mike McCandless
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=144ab5b ]

          LUCENE-6938: switch from svn to git

          Show
          jira-bot ASF subversion and git services added a comment - Commit 144ab5b107aeb7cf79d7e5fd97b8ff5d2ba2ba2f in lucene-solr's branch refs/heads/branch_5x from Mike McCandless [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=144ab5b ] LUCENE-6938 : switch from svn to git
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 329167872371d19f37f9f48f1822014dc87a7eb6 in lucene-solr's branch refs/heads/branch_5_5 from Mike McCandless
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=3291678 ]

          LUCENE-6938: switch from svn to git

          Show
          jira-bot ASF subversion and git services added a comment - Commit 329167872371d19f37f9f48f1822014dc87a7eb6 in lucene-solr's branch refs/heads/branch_5_5 from Mike McCandless [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=3291678 ] LUCENE-6938 : switch from svn to git
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 2514521b5506760f81a4c23b0685769e4eefea88 in lucene-solr's branch refs/heads/master from Mike McCandless
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=2514521 ]

          LUCENE-6938: addVersion can no longer do a --reord-only merge with git

          Show
          jira-bot ASF subversion and git services added a comment - Commit 2514521b5506760f81a4c23b0685769e4eefea88 in lucene-solr's branch refs/heads/master from Mike McCandless [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=2514521 ] LUCENE-6938 : addVersion can no longer do a --reord-only merge with git
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 70e61fd9e04ba0312b9c1d3f6d6e8313ab0dce75 in lucene-solr's branch refs/heads/master from Mike McCandless
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=70e61fd ]

          LUCENE-6938: switch from svn to git

          Show
          jira-bot ASF subversion and git services added a comment - Commit 70e61fd9e04ba0312b9c1d3f6d6e8313ab0dce75 in lucene-solr's branch refs/heads/master from Mike McCandless [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=70e61fd ] LUCENE-6938 : switch from svn to git
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 8b71a1baf5b9c6d16d24134cebeaf7f22333580d in lucene-solr's branch refs/heads/branch_5x from Mike McCandless
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=8b71a1b ]

          LUCENE-6938: switch from svn to git

          Show
          jira-bot ASF subversion and git services added a comment - Commit 8b71a1baf5b9c6d16d24134cebeaf7f22333580d in lucene-solr's branch refs/heads/branch_5x from Mike McCandless [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=8b71a1b ] LUCENE-6938 : switch from svn to git
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 7a329d4e299f364a716ca7e3d786684f280d0100 in lucene-solr's branch refs/heads/branch_5_5 from Mike McCandless
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=7a329d4 ]

          LUCENE-6938: switch from svn to git

          Show
          jira-bot ASF subversion and git services added a comment - Commit 7a329d4e299f364a716ca7e3d786684f280d0100 in lucene-solr's branch refs/heads/branch_5_5 from Mike McCandless [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=7a329d4 ] LUCENE-6938 : switch from svn to git
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit b0e769c3ec598dd7398cc8df123bc4c41069e2c3 in lucene-solr's branch refs/heads/branch_5_4 from Uwe Schindler
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=b0e769c ]

          LUCENE-6995, LUCENE-6938: Add branch change trigger to common-build.xml to keep sane build on GIT branch change

          Show
          jira-bot ASF subversion and git services added a comment - Commit b0e769c3ec598dd7398cc8df123bc4c41069e2c3 in lucene-solr's branch refs/heads/branch_5_4 from Uwe Schindler [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=b0e769c ] LUCENE-6995 , LUCENE-6938 : Add branch change trigger to common-build.xml to keep sane build on GIT branch change
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 0d53dce184674dfc8c23879c0e0648b0bd6ae1b8 in lucene-solr's branch refs/heads/branch_5_4 from Uwe Schindler
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=0d53dce ]

          LUCENE-6938: Add WC checks back, now based on JGit

          Show
          jira-bot ASF subversion and git services added a comment - Commit 0d53dce184674dfc8c23879c0e0648b0bd6ae1b8 in lucene-solr's branch refs/heads/branch_5_4 from Uwe Schindler [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=0d53dce ] LUCENE-6938 : Add WC checks back, now based on JGit
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit d4a8bbbf2b1effc2f166530fcd4720127eafc9a9 in lucene-solr's branch refs/heads/branch_5_4 from Uwe Schindler
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=d4a8bbb ]

          LUCENE-6938: Improve output of Git Hash if no GIT available or no GIT checkout (this restores previous behaviour)

          Show
          jira-bot ASF subversion and git services added a comment - Commit d4a8bbbf2b1effc2f166530fcd4720127eafc9a9 in lucene-solr's branch refs/heads/branch_5_4 from Uwe Schindler [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=d4a8bbb ] LUCENE-6938 : Improve output of Git Hash if no GIT available or no GIT checkout (this restores previous behaviour)
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 016b26675efbd25b9907115b400bacb55b840af3 in lucene-solr's branch refs/heads/branch_5_4 from Steve Rowe
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=016b266 ]

          LUCENE-6938: Maven build: Switch SCM descriptors from svn to git; buildnumber-maven-plugin's buildNumberPropertyName property (used in Maven-built artifact manifests) renamed from svn.revision to checkoutid; removed Subversion-specific stuff from README.maven

          Show
          jira-bot ASF subversion and git services added a comment - Commit 016b26675efbd25b9907115b400bacb55b840af3 in lucene-solr's branch refs/heads/branch_5_4 from Steve Rowe [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=016b266 ] LUCENE-6938 : Maven build: Switch SCM descriptors from svn to git; buildnumber-maven-plugin's buildNumberPropertyName property (used in Maven-built artifact manifests) renamed from svn.revision to checkoutid; removed Subversion-specific stuff from README.maven
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 83112977e8fa66615d23e57697b2743052c71098 in lucene-solr's branch refs/heads/branch_5_4 from Uwe Schindler
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=8311297 ]

          LUCENE-6938: fix typo, sorry

          Show
          jira-bot ASF subversion and git services added a comment - Commit 83112977e8fa66615d23e57697b2743052c71098 in lucene-solr's branch refs/heads/branch_5_4 from Uwe Schindler [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=8311297 ] LUCENE-6938 : fix typo, sorry
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit f4b228b34174e87b1ed43e330b500d8b795604ca in lucene-solr's branch refs/heads/branch_5_4 from Uwe Schindler
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=f4b228b ]

          LUCENE-6938: Fix Lucene's src.tgz file; remove svnkit stuff

          Show
          jira-bot ASF subversion and git services added a comment - Commit f4b228b34174e87b1ed43e330b500d8b795604ca in lucene-solr's branch refs/heads/branch_5_4 from Uwe Schindler [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=f4b228b ] LUCENE-6938 : Fix Lucene's src.tgz file; remove svnkit stuff
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 7f9506ca82032804f2354fef71201366fcbf9932 in lucene-solr's branch refs/heads/branch_5_4 from Dawid Weiss
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=7f9506c ]

          LUCENE-6938: Convert build to work with Git rather than SVN. (Mark Miller
          via Dawid Weiss).

          Show
          jira-bot ASF subversion and git services added a comment - Commit 7f9506ca82032804f2354fef71201366fcbf9932 in lucene-solr's branch refs/heads/branch_5_4 from Dawid Weiss [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=7f9506c ] LUCENE-6938 : Convert build to work with Git rather than SVN. (Mark Miller via Dawid Weiss).
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 0ef36fcdd107084a2ac3156943f01eb5f7dd9c74 in lucene-solr's branch refs/heads/branch_5x from Uwe Schindler
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=0ef36fc ]

          LUCENE-6995, LUCENE-6938: Add branch change trigger to common-build.xml to keep sane build on GIT branch change

          Show
          jira-bot ASF subversion and git services added a comment - Commit 0ef36fcdd107084a2ac3156943f01eb5f7dd9c74 in lucene-solr's branch refs/heads/branch_5x from Uwe Schindler [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=0ef36fc ] LUCENE-6995 , LUCENE-6938 : Add branch change trigger to common-build.xml to keep sane build on GIT branch change
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 424a647af4d093915108221bcd4390989303b426 in lucene-solr's branch refs/heads/master from Uwe Schindler
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=424a647 ]

          LUCENE-6995, LUCENE-6938: Add branch change trigger to common-build.xml to keep sane build on GIT branch change

          Show
          jira-bot ASF subversion and git services added a comment - Commit 424a647af4d093915108221bcd4390989303b426 in lucene-solr's branch refs/heads/master from Uwe Schindler [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=424a647 ] LUCENE-6995 , LUCENE-6938 : Add branch change trigger to common-build.xml to keep sane build on GIT branch change
          Hide
          thetaphi Uwe Schindler added a comment -

          Thanks Mark & Dawid! I close this issue, build seems to be fully converted to GIT.

          Show
          thetaphi Uwe Schindler added a comment - Thanks Mark & Dawid! I close this issue, build seems to be fully converted to GIT.
          Hide
          thetaphi Uwe Schindler added a comment -

          Small update.

          Show
          thetaphi Uwe Schindler added a comment - Small update.
          Hide
          thetaphi Uwe Schindler added a comment - - edited

          Hi,
          here is the former "ant check-svn-working-copy" now ritten for git. Its running almost the same checks:

          • It reports unversioned or missing files and fails build
          • If it runs as Jenkins (or for checking regeneration of sources), it also fails build on changes in working copy
          • if it is not a working copy, it prints a warning and proceeds

          As GIT has no file properties, we don't do property checks like EOL-style or MIME-type.

          The usage is same, it runs by ant validate/precommit. The target name changes a bit, I removed the "svn".

          I will commit this a bit later. I do some tests with different (non-)working copies.

          Show
          thetaphi Uwe Schindler added a comment - - edited Hi, here is the former "ant check-svn-working-copy" now ritten for git. Its running almost the same checks: It reports unversioned or missing files and fails build If it runs as Jenkins (or for checking regeneration of sources), it also fails build on changes in working copy if it is not a working copy, it prints a warning and proceeds As GIT has no file properties, we don't do property checks like EOL-style or MIME-type. The usage is same, it runs by ant validate/precommit. The target name changes a bit, I removed the "svn". I will commit this a bit later. I do some tests with different (non-)working copies.
          Hide
          markrmiller@gmail.com Mark Miller added a comment -

          Thanks for taking this up Uwe! I'm away for the weekend.

          Show
          markrmiller@gmail.com Mark Miller added a comment - Thanks for taking this up Uwe! I'm away for the weekend.
          Hide
          thetaphi Uwe Schindler added a comment -

          As we currently have no auto-comments from the Git Bot:

          • I improved the Git hash behaviour for non-Git checkouts or if Git executable is missing (like SVN did before)
          • Steve Rowe added Git to the Maven build
          Show
          thetaphi Uwe Schindler added a comment - As we currently have no auto-comments from the Git Bot: I improved the Git hash behaviour for non-Git checkouts or if Git executable is missing (like SVN did before) Steve Rowe added Git to the Maven build
          Hide
          thetaphi Uwe Schindler added a comment -

          I cherry-picked the 3 commit (Mark/Dawids and 2 of mine). Worked like a charm with the GUI of Tortoise. The conflicts Dawid found was just java8 vs. java7 strings.

          I'll now reenable 5.x builds on jenkins

          Show
          thetaphi Uwe Schindler added a comment - I cherry-picked the 3 commit (Mark/Dawids and 2 of mine). Worked like a charm with the GUI of Tortoise. The conflicts Dawid found was just java8 vs. java7 strings. I'll now reenable 5.x builds on jenkins
          Hide
          thetaphi Uwe Schindler added a comment -

          It looks like JIRA does not yet reports GIT commits, we should fix that.

          Show
          thetaphi Uwe Schindler added a comment - It looks like JIRA does not yet reports GIT commits, we should fix that.
          Hide
          thetaphi Uwe Schindler added a comment -

          I committed everything. Dawid Weiss, can you merge my recent commit to 5.x, too?

          The Jenkins Jobs for 5.x are not yet enabled, I am waiting for backport.

          Show
          thetaphi Uwe Schindler added a comment - I committed everything. Dawid Weiss , can you merge my recent commit to 5.x, too? The Jenkins Jobs for 5.x are not yet enabled, I am waiting for backport.
          Hide
          thetaphi Uwe Schindler added a comment -

          Final version, previous one was not as effective as current one.

          Show
          thetaphi Uwe Schindler added a comment - Final version, previous one was not as effective as current one.
          Hide
          thetaphi Uwe Schindler added a comment -

          Updated patch also disabling svnkit stuff in root.

          Will commit in a moment to make smoker work

          Show
          thetaphi Uwe Schindler added a comment - Updated patch also disabling svnkit stuff in root. Will commit in a moment to make smoker work
          Hide
          thetaphi Uwe Schindler added a comment -

          Here is the patch to fix the Lucene src.tgz. The reason was simple.

          src.export.dir contains now root folder, but previously it was only the lucene/ subfolder. The logic in the macro and git's behaviour did not reflect this. Te quick workaround is to define additional property for Lucene's build.xml and append this while tarring and running scripts.

          Show
          thetaphi Uwe Schindler added a comment - Here is the patch to fix the Lucene src.tgz. The reason was simple. src.export.dir contains now root folder, but previously it was only the lucene/ subfolder. The logic in the macro and git's behaviour did not reflect this. Te quick workaround is to define additional property for Lucene's build.xml and append this while tarring and running scripts.
          Hide
          thetaphi Uwe Schindler added a comment -

          In trunk, the smoke tester on Jenkins did not pass: https://builds.apache.org/job/Lucene-Solr-SmokeRelease-trunk/395/console

          To me it looks like some magic we do with the src-export folder did not fully work, so missing some files:

          package-tgz-src:
              [mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-trunk/lucene/build/src-export/docs/changes
                [get] Getting: https://issues.apache.org/jira/rest/api/2/project/LUCENE
                [get] To: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-trunk/lucene/build/src-export/docs/changes/jiraVersionList.json
               [exec] Failed to open /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-trunk/lucene/build/src-export/CHANGES.txt
               [exec] Use of uninitialized value $first_relid_regex in substitution (s///) at /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-trunk/lucene/site/changes/changes2html.pl line 223.
               [exec] Use of uninitialized value $second_relid_regex in substitution (s///) at /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-trunk/lucene/site/changes/changes2html.pl line 225.
               [exec] Use of uninitialized value $first_relid_regex in concatenation (.) or string at /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-trunk/lucene/site/changes/changes2html.pl line 226.
               [exec] Use of uninitialized value $second_relid_regex in concatenation (.) or string at /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-trunk/lucene/site/changes/changes2html.pl line 226.
               [exec] Use of uninitialized value $title in concatenation (.) or string at /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-trunk/lucene/site/changes/changes2html.pl line 231.
               [exec] Use of uninitialized value $first_relid in concatenation (.) or string at /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-trunk/lucene/site/changes/changes2html.pl line 231.
               [exec] Use of uninitialized value $second_relid in concatenation (.) or string at /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-trunk/lucene/site/changes/changes2html.pl line 231.
               [exec] Use of uninitialized value $first_relid in concatenation (.) or string at /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-trunk/lucene/site/changes/changes2html.pl line 231.
               [exec] Use of uninitialized value $second_relid in concatenation (.) or string at /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-trunk/lucene/site/changes/changes2html.pl line 231.
               [exec] Use of uninitialized value $title in concatenation (.) or string at /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-trunk/lucene/site/changes/changes2html.pl line 231.
             [delete] Deleting: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-trunk/lucene/build/src-export/docs/changes/jiraVersionList.json
               [copy] Copying 3 files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-trunk/lucene/build/src-export/docs/changes
                [tar] Building tar: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-trunk/lucene/dist/lucene-6.0.0-src.tgz
               [echo] Building checksums for '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-trunk/lucene/dist/lucene-6.0.0-src.tgz'
          

          Not sure what is broken there.

          Show
          thetaphi Uwe Schindler added a comment - In trunk, the smoke tester on Jenkins did not pass: https://builds.apache.org/job/Lucene-Solr-SmokeRelease-trunk/395/console To me it looks like some magic we do with the src-export folder did not fully work, so missing some files: package-tgz-src: [mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-trunk/lucene/build/src-export/docs/changes [get] Getting: https://issues.apache.org/jira/rest/api/2/project/LUCENE [get] To: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-trunk/lucene/build/src-export/docs/changes/jiraVersionList.json [exec] Failed to open /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-trunk/lucene/build/src-export/CHANGES.txt [exec] Use of uninitialized value $first_relid_regex in substitution (s///) at /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-trunk/lucene/site/changes/changes2html.pl line 223. [exec] Use of uninitialized value $second_relid_regex in substitution (s///) at /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-trunk/lucene/site/changes/changes2html.pl line 225. [exec] Use of uninitialized value $first_relid_regex in concatenation (.) or string at /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-trunk/lucene/site/changes/changes2html.pl line 226. [exec] Use of uninitialized value $second_relid_regex in concatenation (.) or string at /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-trunk/lucene/site/changes/changes2html.pl line 226. [exec] Use of uninitialized value $title in concatenation (.) or string at /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-trunk/lucene/site/changes/changes2html.pl line 231. [exec] Use of uninitialized value $first_relid in concatenation (.) or string at /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-trunk/lucene/site/changes/changes2html.pl line 231. [exec] Use of uninitialized value $second_relid in concatenation (.) or string at /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-trunk/lucene/site/changes/changes2html.pl line 231. [exec] Use of uninitialized value $first_relid in concatenation (.) or string at /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-trunk/lucene/site/changes/changes2html.pl line 231. [exec] Use of uninitialized value $second_relid in concatenation (.) or string at /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-trunk/lucene/site/changes/changes2html.pl line 231. [exec] Use of uninitialized value $title in concatenation (.) or string at /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-trunk/lucene/site/changes/changes2html.pl line 231. [delete] Deleting: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-trunk/lucene/build/src-export/docs/changes/jiraVersionList.json [copy] Copying 3 files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-trunk/lucene/build/src-export/docs/changes [tar] Building tar: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-trunk/lucene/dist/lucene-6.0.0-src.tgz [echo] Building checksums for '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-trunk/lucene/dist/lucene-6.0.0-src.tgz' Not sure what is broken there.
          Hide
          dweiss Dawid Weiss added a comment -

          Mark, I applied your patch to master (temporary migrated repo at git@github.com:dweiss/lucene-solr-svn2git.git). ant precommit worked for me without any problems. I could not apply it cleanly to branch_5x though – I didn't look closely, just proceeded with the import, I'm sure we can figure it out later.

          Show
          dweiss Dawid Weiss added a comment - Mark, I applied your patch to master (temporary migrated repo at git@github.com:dweiss/lucene-solr-svn2git.git). ant precommit worked for me without any problems. I could not apply it cleanly to branch_5x though – I didn't look closely, just proceeded with the import, I'm sure we can figure it out later.
          Show
          markrmiller@gmail.com Mark Miller added a comment - https://github.com/markrmiller/lucene-solr-svn2git/commit/c587241d35f3dc641a2de26eff3ba2dc2f6eca59
          Hide
          markrmiller@gmail.com Mark Miller added a comment -

          Okay, here is what I think is the minimum patch that replicates what we had. Let's do that for this issue and open new issues for any improvements. That way we won't hold up the conversion at all and we can try and keep some momentum.

          Show
          markrmiller@gmail.com Mark Miller added a comment - Okay, here is what I think is the minimum patch that replicates what we had. Let's do that for this issue and open new issues for any improvements. That way we won't hold up the conversion at all and we can try and keep some momentum.
          Hide
          thetaphi Uwe Schindler added a comment -

          I will open serparate issue once we moved to Git and fix that, together with the check-svn-working-copy-task!

          So lets do the move now!

          Show
          thetaphi Uwe Schindler added a comment - I will open serparate issue once we moved to Git and fix that, together with the check-svn-working-copy-task! So lets do the move now!
          Hide
          thetaphi Uwe Schindler added a comment -

          How about the main build gets the git hash and writes it to a temporary property file. All other build phases can then just read that file and we are done. Am I missing something?

          Thats not good, because on git update it would not update that file unless you do ant clean. The solution proposed before is easy, I will implement it later, no worries. We have the infrastructure:

          • a task will be added to common-build.xml that gets git hash and saves it in property. This task has unless="property"
          • property is added to the patternset of properties to pass down the sub-modules
          • all submodules JAR tasks call the task as dependency. But if the upper builds has already defined the property it is a no-op

          No worries, I will take care! For now just use the approach we had with Subversion.

          Show
          thetaphi Uwe Schindler added a comment - How about the main build gets the git hash and writes it to a temporary property file. All other build phases can then just read that file and we are done. Am I missing something? Thats not good, because on git update it would not update that file unless you do ant clean. The solution proposed before is easy, I will implement it later, no worries. We have the infrastructure: a task will be added to common-build.xml that gets git hash and saves it in property. This task has unless="property" property is added to the patternset of properties to pass down the sub-modules all submodules JAR tasks call the task as dependency. But if the upper builds has already defined the property it is a no-op No worries, I will take care! For now just use the approach we had with Subversion.
          Hide
          upayavira Upayavira added a comment -

          How about the main build gets the git hash and writes it to a temporary property file. All other build phases can then just read that file and we are done. Am I missing something?

          Show
          upayavira Upayavira added a comment - How about the main build gets the git hash and writes it to a temporary property file. All other build phases can then just read that file and we are done. Am I missing something?
          Hide
          dweiss Dawid Weiss added a comment - - edited

          I think I slightly misunderstood this the first time. You meant making it more efficient for windows was not a requirement?

          Yes, exactly. I work on Windows so it also affects me, but I don't think it's critical.

          It does not look so easy though

          I think it's doable, but far from convenient. It's a similar situation as to what happens with "checking whether any tests executed" – you want a property or a marked passed down to sub-builds... it's a pain to maintain. Perhaps we should look at the core of the problem and somehow fix it in Ant itself...

          Show
          dweiss Dawid Weiss added a comment - - edited I think I slightly misunderstood this the first time. You meant making it more efficient for windows was not a requirement? Yes, exactly. I work on Windows so it also affects me, but I don't think it's critical. It does not look so easy though I think it's doable, but far from convenient. It's a similar situation as to what happens with "checking whether any tests executed" – you want a property or a marked passed down to sub-builds... it's a pain to maintain. Perhaps we should look at the core of the problem and somehow fix it in Ant itself...
          Hide
          markrmiller@gmail.com Mark Miller added a comment -

          I think this is an improvement, not a requirement?

          I think I slightly misunderstood this the first time. You meant making it more efficient for windows was not a requirement?

          In that case I agree, though I figured if it was easy, we should just do it. It does not look so easy though. So I suggest a switch to turn it off in build.properties. But right, I don't think it's a requirement that we make it more efficient, just that we keep an id in the jars.

          Show
          markrmiller@gmail.com Mark Miller added a comment - I think this is an improvement, not a requirement? I think I slightly misunderstood this the first time. You meant making it more efficient for windows was not a requirement? In that case I agree, though I figured if it was easy, we should just do it. It does not look so easy though. So I suggest a switch to turn it off in build.properties. But right, I don't think it's a requirement that we make it more efficient, just that we keep an id in the jars.
          Hide
          markrmiller@gmail.com Mark Miller added a comment -

          Doesn't look easy to share any state between multiple inits.

          I don't even know if doing it at the top level appears any better than per jar. It's still a ton of calls per run.

          We can simply allow it to be disabled via build.properties if it's an issue for some Windows devs.

          Show
          markrmiller@gmail.com Mark Miller added a comment - Doesn't look easy to share any state between multiple inits. I don't even know if doing it at the top level appears any better than per jar. It's still a ton of calls per run. We can simply allow it to be disabled via build.properties if it's an issue for some Windows devs.
          Hide
          markrmiller@gmail.com Mark Miller added a comment -

          there must be some way to just get the checkout sha once

          The key word is once. Yes, of course we can get the sha the same way as we can get an svn version Uwe's concern is how many times we execute a program to do it. Our ant scripts init 8 billion times per target.

          I'll look into trying to exec a minimal number of times.

          Show
          markrmiller@gmail.com Mark Miller added a comment - there must be some way to just get the checkout sha once The key word is once. Yes, of course we can get the sha the same way as we can get an svn version Uwe's concern is how many times we execute a program to do it. Our ant scripts init 8 billion times per target. I'll look into trying to exec a minimal number of times.
          Hide
          markrmiller@gmail.com Mark Miller added a comment -

          I think this is an improvement, not a requirement?

          I think because we had this feature with svn and there is no consensus about dropping it and it affects releases, we want it before the move. I'm sure it will be simple to add.

          Show
          markrmiller@gmail.com Mark Miller added a comment - I think this is an improvement, not a requirement? I think because we had this feature with svn and there is no consensus about dropping it and it affects releases, we want it before the move. I'm sure it will be simple to add.
          Hide
          dweiss Dawid Weiss added a comment -

          > I think the main thing left to do in this issue is put the git hash in efficiently.

          I think this is an improvement, not a requirement? Don't we call SVN multiple times already? Other than that I agree with you – get the baseline targets working, clean up everything that doesn't work after the transition.

          Show
          dweiss Dawid Weiss added a comment - > I think the main thing left to do in this issue is put the git hash in efficiently. I think this is an improvement, not a requirement? Don't we call SVN multiple times already? Other than that I agree with you – get the baseline targets working, clean up everything that doesn't work after the transition.
          Hide
          markrmiller@gmail.com Mark Miller added a comment -

          We don't need to vote yet - that only happens when consensus fails or someone wants to force something. We can warn the dev list again to make sure everyone is caught up, but no need to force a vote unless someone comes out against. There is a very visible discussion and a few JIRA issues that have been in progress for a long time now. Once we are ready to go, we can sum things up in a new dev thread.

          I think in terms of what needs to be covered here, Uwe has detailed it pretty well. We want all the targets to work really - or to understand why any target does not work. We can wait for Uwe to create a new git validator though - all targets still work without that. 'svn' does not really have a very deep imprint in our build targets.

          I think the main thing left to do in this issue is put the git hash in efficiently.

          Some other things people are concerned about can get further JIRA issues, but I imagine a lot of that (such as python scripts) can be updated as used / needed by those that use them.

          Show
          markrmiller@gmail.com Mark Miller added a comment - We don't need to vote yet - that only happens when consensus fails or someone wants to force something. We can warn the dev list again to make sure everyone is caught up, but no need to force a vote unless someone comes out against. There is a very visible discussion and a few JIRA issues that have been in progress for a long time now. Once we are ready to go, we can sum things up in a new dev thread. I think in terms of what needs to be covered here, Uwe has detailed it pretty well. We want all the targets to work really - or to understand why any target does not work. We can wait for Uwe to create a new git validator though - all targets still work without that. 'svn' does not really have a very deep imprint in our build targets. I think the main thing left to do in this issue is put the git hash in efficiently. Some other things people are concerned about can get further JIRA issues, but I imagine a lot of that (such as python scripts) can be updated as used / needed by those that use them.
          Hide
          dweiss Dawid Weiss added a comment -

          Can we specify the commands (build/ precommit checks) that need to "work" with a git clone so that we can orderly go through them and know where we are with the migration process? It'd be good to have it done and then vote/ move on with the development to git. My candidates would be:

          • ant clean test
          • ant jar
          • ant validate precommit

          Then there are follow-ups:

          • Maven POMs (scm defs)
          • README and other help files referring to SVN
          • various python scripts under dev-tools/scripts invoke SVN
          • Jenkins CI job definitions, etc.
          Show
          dweiss Dawid Weiss added a comment - Can we specify the commands (build/ precommit checks) that need to "work" with a git clone so that we can orderly go through them and know where we are with the migration process? It'd be good to have it done and then vote/ move on with the development to git. My candidates would be: ant clean test ant jar ant validate precommit Then there are follow-ups: Maven POMs (scm defs) README and other help files referring to SVN various python scripts under dev-tools/scripts invoke SVN Jenkins CI job definitions, etc.
          Hide
          dpgove Dennis Gove added a comment -

          You can get the current sha1 with the command

          $> git rev-parse HEAD
          

          And you can replace HEAD with the name of a branch/tag to get the sha1 of that. See
          $> git help rev-parse
          for all the options

          Show
          dpgove Dennis Gove added a comment - You can get the current sha1 with the command $> git rev-parse HEAD And you can replace HEAD with the name of a branch/tag to get the sha1 of that. See $> git help rev-parse for all the options
          Hide
          paul.elschot@xs4all.nl Paul Elschot added a comment - - edited

          there must be some way to just get the checkout sha once

          This will work on the current branch:

          git log --format="%H" -n 1

          But also this, for example for the trunk branch:

          cat .git/refs/heads/trunk

          This is why branching in git it so cheap, a branch is no more than a file with a sha1.

          the command line of Git is the most confusing an user-unfriendly thing

          Indeed, and this makes the learning curve steeper than it would need to be.

          Show
          paul.elschot@xs4all.nl Paul Elschot added a comment - - edited there must be some way to just get the checkout sha once This will work on the current branch: git log --format= "%H" -n 1 But also this, for example for the trunk branch: cat .git/refs/heads/trunk This is why branching in git it so cheap, a branch is no more than a file with a sha1. the command line of Git is the most confusing an user-unfriendly thing Indeed, and this makes the learning curve steeper than it would need to be.
          Hide
          thetaphi Uwe Schindler added a comment -

          Wow I bet there is a good chance this is much faster with Git. Otherwise, there must be some way to just get the checkout sha once and use it for every jar?

          Yes that would work. I was about to do that on svn already. The trick is to just populate the property once and pass it down to sub-builds (using the patternset of properties to pass down). We then just need a task that populates the property if it doest not yet exists (unless="property")

          I think we should just replace the old SVN revision in JAR files by the sha1. It should be easy to get it with a single "git" command (no idea how: I am still not firm in using Git's CLI; I always use TortoiseGit, because the command line of Git is the most confusing an user-unfriendly thing I have ever seen).

          FYI: I would not call "jgit" to populate the property at the moment - if we cache the result its fine - because this will cause permgen errors in Java 7 (the well-known Ant Classloader bug). Otherwise I would have implemented the same for SVN already (I have a patch here for SVN similar to Dawid's code, but this breaks the whole build after a complete build with many JAR files).

          Show
          thetaphi Uwe Schindler added a comment - Wow I bet there is a good chance this is much faster with Git. Otherwise, there must be some way to just get the checkout sha once and use it for every jar? Yes that would work. I was about to do that on svn already. The trick is to just populate the property once and pass it down to sub-builds (using the patternset of properties to pass down). We then just need a task that populates the property if it doest not yet exists (unless="property") I think we should just replace the old SVN revision in JAR files by the sha1. It should be easy to get it with a single "git" command (no idea how: I am still not firm in using Git's CLI; I always use TortoiseGit, because the command line of Git is the most confusing an user-unfriendly thing I have ever seen). FYI: I would not call "jgit" to populate the property at the moment - if we cache the result its fine - because this will cause permgen errors in Java 7 (the well-known Ant Classloader bug). Otherwise I would have implemented the same for SVN already (I have a patch here for SVN similar to Dawid's code, but this breaks the whole build after a complete build with many JAR files).
          Hide
          markrmiller@gmail.com Mark Miller added a comment -

          We can investigate further, if we really need the git commit hash in every JAR file. to me this was always a large slowdown on windows where creating a new process costs much time.

          Wow I bet there is a good chance this is much faster with Git. Otherwise, there must be some way to just get the checkout sha once and use it for every jar?

          Show
          markrmiller@gmail.com Mark Miller added a comment - We can investigate further, if we really need the git commit hash in every JAR file. to me this was always a large slowdown on windows where creating a new process costs much time. Wow I bet there is a good chance this is much faster with Git. Otherwise, there must be some way to just get the checkout sha once and use it for every jar?
          Hide
          markrmiller@gmail.com Mark Miller added a comment -

          you can do git-archive to get a tarball without any intermediate filesystem index.

          Yeah, that is the first thing I found when looking for a git export. I tried to keep things as similar to they were as possible though - and I'd also like to keep as much logic (like the compressing) out of exec and in ant as possible.

          I strongly opt for keeping the commit's md5 inside build artefacts.

          I have no problem with including the hash myself.

          Show
          markrmiller@gmail.com Mark Miller added a comment - you can do git-archive to get a tarball without any intermediate filesystem index. Yeah, that is the first thing I found when looking for a git export. I tried to keep things as similar to they were as possible though - and I'd also like to keep as much logic (like the compressing) out of exec and in ant as possible. I strongly opt for keeping the commit's md5 inside build artefacts. I have no problem with including the hash myself.
          Hide
          dweiss Dawid Weiss added a comment - - edited

          Hi. Catching up with what's been said, here's my opinion.

          1) I didn't go into the specific of what the scripts were doing to get a "pristine" copy of a built, but with git you can do git-archive to get a tarball without any intermediate filesystem index. This has advantages for Windows (permissions are stored properly) and for speed (much faster on slower filesystems). Perhaps the scripts could be improved and do a two-phase check: git stat to verify the current checkout isn't dirty (locally ignored files remain ignored) and then create a tarball, from which all the tests, etc. would be executed in any follow-up steps.

          https://git-scm.com/docs/git-archive

          2) I strongly opt for keeping the commit's md5 inside build artefacts. This has helped me enormously in the past a few times. These hashes are not linear, correct, but they are even better at locating a particular commit the build was executed against, be it a branch or whatever. We personally use two git markers – the commit hash and a more human-friendly last-tag + dirty flag. We call jgit from ant, but here's the beanshell script that collects the required properties:

                import org.eclipse.jgit.api.*;
                import org.eclipse.jgit.lib.*;
                import org.eclipse.jgit.storage.file.*;
          
                Repository repository = new FileRepositoryBuilder()
                  .findGitDir(project.getBaseDir())
                  .build();
          
                Git git = new Git(repository);
                String revLine =
                    git.log().setMaxCount(1).call().iterator().next().name() +
                    (git.status().call().isClean() ? "" : "-dirty");
          
                project.setNewProperty("product.gitrev", revLine);
          
          Show
          dweiss Dawid Weiss added a comment - - edited Hi. Catching up with what's been said, here's my opinion. 1) I didn't go into the specific of what the scripts were doing to get a "pristine" copy of a built, but with git you can do git-archive to get a tarball without any intermediate filesystem index. This has advantages for Windows (permissions are stored properly) and for speed (much faster on slower filesystems). Perhaps the scripts could be improved and do a two-phase check: git stat to verify the current checkout isn't dirty (locally ignored files remain ignored) and then create a tarball, from which all the tests, etc. would be executed in any follow-up steps. https://git-scm.com/docs/git-archive 2) I strongly opt for keeping the commit's md5 inside build artefacts. This has helped me enormously in the past a few times. These hashes are not linear, correct, but they are even better at locating a particular commit the build was executed against, be it a branch or whatever. We personally use two git markers – the commit hash and a more human-friendly last-tag + dirty flag. We call jgit from ant, but here's the beanshell script that collects the required properties: import org.eclipse.jgit.api.*; import org.eclipse.jgit.lib.*; import org.eclipse.jgit.storage.file.*; Repository repository = new FileRepositoryBuilder() .findGitDir(project.getBaseDir()) .build(); Git git = new Git(repository); String revLine = git.log().setMaxCount(1).call().iterator().next().name() + (git.status().call().isClean() ? "" : " -dirty"); project.setNewProperty( "product.gitrev" , revLine);
          Hide
          thetaphi Uwe Schindler added a comment -

          Exactly. We don't want to have anything ignored or other leftovers in it. Basically we want to tar what "ant clean clean-jars" leaves back (or should). Or better: the state of a fresh checkout without any additional files (not even ignroed ones) and no special files like .svn or .git.

          The reason why git is faster than svn on exporting: it does the whole export stuff from your local GIT database wthout networks. Because you have the whole repository local already.

          Show
          thetaphi Uwe Schindler added a comment - Exactly. We don't want to have anything ignored or other leftovers in it. Basically we want to tar what "ant clean clean-jars" leaves back (or should). Or better: the state of a fresh checkout without any additional files (not even ignroed ones) and no special files like .svn or .git. The reason why git is faster than svn on exporting: it does the whole export stuff from your local GIT database wthout networks. Because you have the whole repository local already.
          Hide
          steve_rowe Steve Rowe added a comment -

          FWIW, building the source release from svn export was put in place as a result of problems building the 3.1 release (previously there were fragile inclusion/exclusions rules, the rough equivalent of which are preserved as lucene.local.src.package.patterns for the package-local-src-tgz target). See http://markmail.org/message/nfon2anpgzdja2st and LUCENE-2973.

          Show
          steve_rowe Steve Rowe added a comment - FWIW, building the source release from svn export was put in place as a result of problems building the 3.1 release (previously there were fragile inclusion/exclusions rules, the rough equivalent of which are preserved as lucene.local.src.package.patterns for the package-local-src-tgz target). See http://markmail.org/message/nfon2anpgzdja2st and LUCENE-2973 .
          Hide
          steve_rowe Steve Rowe added a comment -

          Presumably the reason for building the tarball from an export is to avoid including uncommitted changes

          I'm not sure - we have checks for that done in another way in another place. I think it was also to avoid things like the .svn folders and what not.

          Also it's to avoid including locally-built ignored things in tarballs. This will remain an issue with git, I think.

          Show
          steve_rowe Steve Rowe added a comment - Presumably the reason for building the tarball from an export is to avoid including uncommitted changes I'm not sure - we have checks for that done in another way in another place. I think it was also to avoid things like the .svn folders and what not. Also it's to avoid including locally-built ignored things in tarballs. This will remain an issue with git, I think.
          Hide
          markrmiller@gmail.com Mark Miller added a comment - - edited

          Presumably the reason for building the tarball from an export is to avoid including uncommitted changes

          I'm not sure - we have checks for that done in another way in another place. I think it was also to avoid things like the .svn folders and what not. i.e. what an svn export is for.

          Anyway, given this operation is so fast with the git method, I like keeping the old macro and behavior initially. Seems simpler and safer to mimic old behavior while we make the migration and update the release doc, etc.

          Show
          markrmiller@gmail.com Mark Miller added a comment - - edited Presumably the reason for building the tarball from an export is to avoid including uncommitted changes I'm not sure - we have checks for that done in another way in another place. I think it was also to avoid things like the .svn folders and what not. i.e. what an svn export is for. Anyway, given this operation is so fast with the git method, I like keeping the old macro and behavior initially. Seems simpler and safer to mimic old behavior while we make the migration and update the release doc, etc.
          Hide
          upayavira Upayavira added a comment -

          Presumably the reason for building the tarball from an export is to avoid including uncommitted changes. Could we achieve the same end by just confirming that the git checkout is clean and not proceeding if there are any changes? Like so:

          if [ -z "$(git status --porcelain)" ]; then 
            # Working directory clean
          else 
            # Uncommitted changes
          fi
          

          (courtesy of http://unix.stackexchange.com/questions/155046/determine-if-git-working-directory-is-clean-from-a-script)

          Show
          upayavira Upayavira added a comment - Presumably the reason for building the tarball from an export is to avoid including uncommitted changes. Could we achieve the same end by just confirming that the git checkout is clean and not proceeding if there are any changes? Like so: if [ -z "$(git status --porcelain)" ]; then # Working directory clean else # Uncommitted changes fi (courtesy of http://unix.stackexchange.com/questions/155046/determine-if-git-working-directory-is-clean-from-a-script )
          Hide
          markrmiller@gmail.com Mark Miller added a comment -

          over 7 minutes

          I guess that was a bit of a long run. I've since seen it take 5 and a half minutes, while git has been pretty consistent at 4 and a half minutes. Probably is just the difference in export speed.

          Show
          markrmiller@gmail.com Mark Miller added a comment - over 7 minutes I guess that was a bit of a long run. I've since seen it take 5 and a half minutes, while git has been pretty consistent at 4 and a half minutes. Probably is just the difference in export speed.
          Hide
          markrmiller@gmail.com Mark Miller added a comment -

          That export runs so much faster than svn export.

          Overall, precommit dropped from like over 7 minutes to 4 and a half for me. Unless it's the SVN stuff, but that seemed to be relatively quick based on the output. It will be quick with jgit regardless I'm sure.

          Show
          markrmiller@gmail.com Mark Miller added a comment - That export runs so much faster than svn export. Overall, precommit dropped from like over 7 minutes to 4 and a half for me. Unless it's the SVN stuff, but that seemed to be relatively quick based on the output. It will be quick with jgit regardless I'm sure.
          Hide
          thetaphi Uwe Schindler added a comment -

          Looks good for now. As I see, you removed the svnversion from JAR files - like suggested. We can investigate further, if we really need the git commit hash in every JAR file. to me this was always a large slowdown on windows where creating a new process costs much time.

          Show
          thetaphi Uwe Schindler added a comment - Looks good for now. As I see, you removed the svnversion from JAR files - like suggested. We can investigate further, if we really need the git commit hash in every JAR file. to me this was always a large slowdown on windows where creating a new process costs much time.
          Hide
          markrmiller@gmail.com Mark Miller added a comment -

          Here is a patch with some early exploration. I have an alternate for the svn export I think, and have done some initial renaming, cleaning up.

          Show
          markrmiller@gmail.com Mark Miller added a comment - Here is a patch with some early exploration. I have an alternate for the svn export I think, and have done some initial renaming, cleaning up.
          Hide
          markrmiller@gmail.com Mark Miller added a comment -

          Thanks Uwe!

          This is not critical and can be fixed later.

          +1.

          I'll open separate issue if needed.

          Yeah, let's see how long this takes vs a jgit version. If we just finish this issue real quick, we might as well spin it off into it's own issue.

          but somehow this makes no sense, as the commit hashes cannot be sorted and don't have a "version like" character

          But it does tell you how to get to the code that created that JAR still, right?

          The source tar gz/zip files use svn export. We have to fix this, otherwise we cannot release and test.

          Yeah, this seems like the key issue to address

          Show
          markrmiller@gmail.com Mark Miller added a comment - Thanks Uwe! This is not critical and can be fixed later. +1. I'll open separate issue if needed. Yeah, let's see how long this takes vs a jgit version. If we just finish this issue real quick, we might as well spin it off into it's own issue. but somehow this makes no sense, as the commit hashes cannot be sorted and don't have a "version like" character But it does tell you how to get to the code that created that JAR still, right? The source tar gz/zip files use svn export. We have to fix this, otherwise we cannot release and test. Yeah, this seems like the key issue to address
          Hide
          thetaphi Uwe Schindler added a comment - - edited

          Hi,
          the current build relies on SVN at 3 places:

          • "ant check-svn-working-copy" as part of "ant validate" and "ant precommit". This is not critical and can be fixed later. The task gets a no-op if executed from non-svn, so we can fix later, no need to hurry. I will take care of that and prepare patch to use "jgit" for similar checks (https://www.eclipse.org/jgit/). I'll open separate issue if needed.
          • Currently we call "svnversion" and place output inside all JAR files. I would like to remove this completely. We can of course add the commit hash, but somehow this makes no sense, as the commit hashes cannot be sorted and don't have a "version like" character. So +1 to remove "svnversion" calls during JAR building and just place version number in JAR files, but no revision/hash.
          • The source tar gz/zip files use svn export. We have to fix this, otherwise we cannot release and test.

          Jenkins builds need to be updated, but thats easy. Policeman is already prepared to do git checkouts (using jgit, see above).

          Show
          thetaphi Uwe Schindler added a comment - - edited Hi, the current build relies on SVN at 3 places: "ant check-svn-working-copy" as part of "ant validate" and "ant precommit". This is not critical and can be fixed later. The task gets a no-op if executed from non-svn, so we can fix later, no need to hurry. I will take care of that and prepare patch to use "jgit" for similar checks ( https://www.eclipse.org/jgit/ ). I'll open separate issue if needed. Currently we call "svnversion" and place output inside all JAR files. I would like to remove this completely. We can of course add the commit hash, but somehow this makes no sense, as the commit hashes cannot be sorted and don't have a "version like" character. So +1 to remove "svnversion" calls during JAR building and just place version number in JAR files, but no revision/hash. The source tar gz/zip files use svn export. We have to fix this, otherwise we cannot release and test. Jenkins builds need to be updated, but thats easy. Policeman is already prepared to do git checkouts (using jgit, see above).

            People

            • Assignee:
              markrmiller@gmail.com Mark Miller
              Reporter:
              markrmiller@gmail.com Mark Miller
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development