Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-169

Better overall build process without using a shell script

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.0.1
    • Fix Version/s: 0.0.1
    • Component/s: Tools and Build
    • Labels:
      None

      Description

      I would suggest to create a better build process which takes advantage of the multi module capabilities of Maven and furthermore of multi threading which can decrease the build time dramatically.

      Apart from that currently the build process is spreaded over different areas which are configured differently (differen plugin versions etc.) which leads not in a realiable build process.

      Furthermore this would make integration into a CI environment like Apache area simpler.

        Issue Links

          Activity

          Hide
          joewitt Joseph Witt added a comment -

          This is really important topic so you'll find a very receptive group here. Our most recent discussion:

          http://mail-archives.apache.org/mod_mbox/incubator-nifi-dev/201412.mbox/%3CCALJK9a5X6D9T%2BPakKuUFhVX8OptcZB%3DUg9gvT3gfea0mf-mT7w%40mail.gmail.com%3E

          and to some extent here:

          http://mail-archives.apache.org/mod_mbox/incubator-nifi-dev/201412.mbox/%3CCALJK9a66dNTDdmmhiFG-DVkcDZTn4V7YmwGGLbW8_JeO8pA36w%40mail.gmail.com%3E

          The key is figuring out a holistic process that supports effective builds, releases, CI, and that reflects the project's different release frequencies.

          The discussion thus far does suggest there is a good openness for change once we narrow in on a good path.

          Show
          joewitt Joseph Witt added a comment - This is really important topic so you'll find a very receptive group here. Our most recent discussion: http://mail-archives.apache.org/mod_mbox/incubator-nifi-dev/201412.mbox/%3CCALJK9a5X6D9T%2BPakKuUFhVX8OptcZB%3DUg9gvT3gfea0mf-mT7w%40mail.gmail.com%3E and to some extent here: http://mail-archives.apache.org/mod_mbox/incubator-nifi-dev/201412.mbox/%3CCALJK9a66dNTDdmmhiFG-DVkcDZTn4V7YmwGGLbW8_JeO8pA36w%40mail.gmail.com%3E The key is figuring out a holistic process that supports effective builds, releases, CI, and that reflects the project's different release frequencies. The discussion thus far does suggest there is a good openness for change once we narrow in on a good path.
          Hide
          joewitt Joseph Witt added a comment -

          All

          In an effort to stop swimming upstream i've got a branch NIFI-169 which is an attempt to just move to a single Maven project build structure. We can sort out concerns from there

          Show
          joewitt Joseph Witt added a comment - All In an effort to stop swimming upstream i've got a branch NIFI-169 which is an attempt to just move to a single Maven project build structure. We can sort out concerns from there
          Hide
          khmarbaise Karl Heinz Marbaise added a comment -

          Ok. I have started locally with a branch (a few hours ago) .....and got many modules already working but I can take a look and may be i can deliver patches against that branch...

          Show
          khmarbaise Karl Heinz Marbaise added a comment - Ok. I have started locally with a branch (a few hours ago) .....and got many modules already working but I can take a look and may be i can deliver patches against that branch...
          Hide
          joewitt Joseph Witt added a comment -

          Sounds good. I will continue on this path. Will push it to origin in a bit. I am also happy to look at a PR or whatever. I am frankly quite honored to have your help.

          Show
          joewitt Joseph Witt added a comment - Sounds good. I will continue on this path. Will push it to origin in a bit. I am also happy to look at a PR or whatever. I am frankly quite honored to have your help.
          Hide
          khmarbaise Karl Heinz Marbaise added a comment - - edited

          At the moment i have that state https://github.com/khmarbaise/incubator-nifi/tree/NIFI-169-khmarbaise ...not ready but...it's a step forward...If you can show your branch so we can combine the efforts...

          Show
          khmarbaise Karl Heinz Marbaise added a comment - - edited At the moment i have that state https://github.com/khmarbaise/incubator-nifi/tree/NIFI-169-khmarbaise ...not ready but...it's a step forward...If you can show your branch so we can combine the efforts...
          Hide
          joewitt Joseph Witt added a comment -

          Just took a look. We're on very much the same path. I'm going a slowly through the overall parent pom as I just learned about the org.apache:apache artifact. So learning what other projects do here as well.

          I am happy to push forward. Or to hold up a bit and then pull from yours. Whatever you're comfortable with.

          Show
          joewitt Joseph Witt added a comment - Just took a look. We're on very much the same path. I'm going a slowly through the overall parent pom as I just learned about the org.apache:apache artifact. So learning what other projects do here as well. I am happy to push forward. Or to hold up a bit and then pull from yours. Whatever you're comfortable with.
          Hide
          bmargulies Benson Margulies added a comment -

          I'd volunteer here, but KHM is an even older Maven hand than I am.

          Show
          bmargulies Benson Margulies added a comment - I'd volunteer here, but KHM is an even older Maven hand than I am.
          Hide
          khmarbaise Karl Heinz Marbaise added a comment -

          Thanks Benson .

          @Joseph i would suggest you put your state on a public branch so i can take a look and create appropriate PR's for it, cause you know the project more well than i do. I'm looking at the moment only with the Maven glasses on it.

          Show
          khmarbaise Karl Heinz Marbaise added a comment - Thanks Benson . @Joseph i would suggest you put your state on a public branch so i can take a look and create appropriate PR's for it, cause you know the project more well than i do. I'm looking at the moment only with the Maven glasses on it.
          Hide
          joewitt Joseph Witt added a comment -

          Will do. Working toward having it in a state to push the branch to origin

          Show
          joewitt Joseph Witt added a comment - Will do. Working toward having it in a state to push the branch to origin
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 19d4a150a875e0862d458e03ffb413797f2bf81b in incubator-nifi's branch refs/heads/NIFI-169 from Joseph Witt
          [ https://git-wip-us.apache.org/repos/asf?p=incubator-nifi.git;h=19d4a15 ]

          NIFI-169 well it finally all builds. There is a classpath issue still to sort out which impacts startup

          Show
          jira-bot ASF subversion and git services added a comment - Commit 19d4a150a875e0862d458e03ffb413797f2bf81b in incubator-nifi's branch refs/heads/ NIFI-169 from Joseph Witt [ https://git-wip-us.apache.org/repos/asf?p=incubator-nifi.git;h=19d4a15 ] NIFI-169 well it finally all builds. There is a classpath issue still to sort out which impacts startup
          Hide
          joewitt Joseph Witt added a comment -

          Ok well the build is totally rearranged now and does create the final result. Still need to sort out a classpath issue.

          We cannot have a single command build because of the fact that Maven will apparently not allow you to have a plugin with extensions built within the same reactor as other artifacts that use that plugin. So unless there is a good workaround there we will still have to do two different commands (build nar-plugin, then build all others)

          Show
          joewitt Joseph Witt added a comment - Ok well the build is totally rearranged now and does create the final result. Still need to sort out a classpath issue. We cannot have a single command build because of the fact that Maven will apparently not allow you to have a plugin with extensions built within the same reactor as other artifacts that use that plugin. So unless there is a good workaround there we will still have to do two different commands (build nar-plugin, then build all others)
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 675ea6611c3e941df05f88dacb6c691bd9712206 in incubator-nifi's branch refs/heads/NIFI-169 from Joseph Witt
          [ https://git-wip-us.apache.org/repos/asf?p=incubator-nifi.git;h=675ea66 ]

          NIFI-169 Fixed classpath issue affecting startup

          Show
          jira-bot ASF subversion and git services added a comment - Commit 675ea6611c3e941df05f88dacb6c691bd9712206 in incubator-nifi's branch refs/heads/ NIFI-169 from Joseph Witt [ https://git-wip-us.apache.org/repos/asf?p=incubator-nifi.git;h=675ea66 ] NIFI-169 Fixed classpath issue affecting startup
          Hide
          joewitt Joseph Witt added a comment -

          ok app builds and runs. Please take a look over branch NIFI-169 on nifi's incubator git in ASF

          Show
          joewitt Joseph Witt added a comment - ok app builds and runs. Please take a look over branch NIFI-169 on nifi's incubator git in ASF
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 614349bdfdb4b73509d6c9384b9094aefc41d7a8 in incubator-nifi's branch refs/heads/NIFI-169 from Joseph Witt
          [ https://git-wip-us.apache.org/repos/asf?p=incubator-nifi.git;h=614349b ]

          NIFI-169 updating a few tests and build

          Show
          jira-bot ASF subversion and git services added a comment - Commit 614349bdfdb4b73509d6c9384b9094aefc41d7a8 in incubator-nifi's branch refs/heads/ NIFI-169 from Joseph Witt [ https://git-wip-us.apache.org/repos/asf?p=incubator-nifi.git;h=614349b ] NIFI-169 updating a few tests and build
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 3a4c6ed887cd2e13aa861ed99028de33678bcbe8 in incubator-nifi's branch refs/heads/NIFI-169 from Joseph Witt
          [ https://git-wip-us.apache.org/repos/asf?p=incubator-nifi.git;h=3a4c6ed ]

          Merge branch 'develop' into NIFI-169

          Show
          jira-bot ASF subversion and git services added a comment - Commit 3a4c6ed887cd2e13aa861ed99028de33678bcbe8 in incubator-nifi's branch refs/heads/ NIFI-169 from Joseph Witt [ https://git-wip-us.apache.org/repos/asf?p=incubator-nifi.git;h=3a4c6ed ] Merge branch 'develop' into NIFI-169
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit a94cf4be4b339bf520517d2d00390f8bc3b249e1 in incubator-nifi's branch refs/heads/NIFI-169 from Joseph Witt
          [ https://git-wip-us.apache.org/repos/asf?p=incubator-nifi.git;h=a94cf4b ]

          Merge branch 'develop' into NIFI-169

          Show
          jira-bot ASF subversion and git services added a comment - Commit a94cf4be4b339bf520517d2d00390f8bc3b249e1 in incubator-nifi's branch refs/heads/ NIFI-169 from Joseph Witt [ https://git-wip-us.apache.org/repos/asf?p=incubator-nifi.git;h=a94cf4b ] Merge branch 'develop' into NIFI-169
          Hide
          mcgilman Matt Gilman added a comment - - edited

          +1 for merging to develop

          My environment:
          Apache Maven 3.2.3 (33f8c3e1027c3ddde99d3cdebad2656a31e8fdf4; 2014-08-11T16:58:10-04:00)
          Maven home: /opt/maven/apache-maven-3.2.3
          Java version: 1.7.0_45, vendor: Oracle Corporation
          Java home: /Library/Java/JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home/jre
          Default locale: en_US, platform encoding: UTF-8
          OS name: "mac os x", version: "10.9.5", arch: "x86_64", family: "mac"

          Show
          mcgilman Matt Gilman added a comment - - edited +1 for merging to develop My environment: Apache Maven 3.2.3 (33f8c3e1027c3ddde99d3cdebad2656a31e8fdf4; 2014-08-11T16:58:10-04:00) Maven home: /opt/maven/apache-maven-3.2.3 Java version: 1.7.0_45, vendor: Oracle Corporation Java home: /Library/Java/JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home/jre Default locale: en_US, platform encoding: UTF-8 OS name: "mac os x", version: "10.9.5", arch: "x86_64", family: "mac"
          Hide
          markap14 Mark Payne added a comment -

          +1

          Build works well on Windows and is MUCH faster and easier than the previous requirement of building many modules.

          My environment:
          Apache Maven 3.2.3 (33f8c3e1027c3ddde99d3cdebad2656a31e8fdf4; 2014-08-11T16:58:10-04:00)
          Maven home: C:\apps\apache-maven-3.2.3\bin\..
          Java version: 1.7.0, vendor: Oracle Corporation
          Java home: C:\Program Files\Java\jdk1.7.0\jre
          Default locale: en_US, platform encoding: Cp1252
          OS name: "windows 7", version: "6.1", arch: "amd64", family: "windows"

          Show
          markap14 Mark Payne added a comment - +1 Build works well on Windows and is MUCH faster and easier than the previous requirement of building many modules. My environment: Apache Maven 3.2.3 (33f8c3e1027c3ddde99d3cdebad2656a31e8fdf4; 2014-08-11T16:58:10-04:00) Maven home: C:\apps\apache-maven-3.2.3\bin\.. Java version: 1.7.0, vendor: Oracle Corporation Java home: C:\Program Files\Java\jdk1.7.0\jre Default locale: en_US, platform encoding: Cp1252 OS name: "windows 7", version: "6.1", arch: "amd64", family: "windows"
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 614349bdfdb4b73509d6c9384b9094aefc41d7a8 in incubator-nifi's branch refs/heads/develop from Joseph Witt
          [ https://git-wip-us.apache.org/repos/asf?p=incubator-nifi.git;h=614349b ]

          NIFI-169 updating a few tests and build

          Show
          jira-bot ASF subversion and git services added a comment - Commit 614349bdfdb4b73509d6c9384b9094aefc41d7a8 in incubator-nifi's branch refs/heads/develop from Joseph Witt [ https://git-wip-us.apache.org/repos/asf?p=incubator-nifi.git;h=614349b ] NIFI-169 updating a few tests and build
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 3a4c6ed887cd2e13aa861ed99028de33678bcbe8 in incubator-nifi's branch refs/heads/develop from Joseph Witt
          [ https://git-wip-us.apache.org/repos/asf?p=incubator-nifi.git;h=3a4c6ed ]

          Merge branch 'develop' into NIFI-169

          Show
          jira-bot ASF subversion and git services added a comment - Commit 3a4c6ed887cd2e13aa861ed99028de33678bcbe8 in incubator-nifi's branch refs/heads/develop from Joseph Witt [ https://git-wip-us.apache.org/repos/asf?p=incubator-nifi.git;h=3a4c6ed ] Merge branch 'develop' into NIFI-169
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit a94cf4be4b339bf520517d2d00390f8bc3b249e1 in incubator-nifi's branch refs/heads/develop from Joseph Witt
          [ https://git-wip-us.apache.org/repos/asf?p=incubator-nifi.git;h=a94cf4b ]

          Merge branch 'develop' into NIFI-169

          Show
          jira-bot ASF subversion and git services added a comment - Commit a94cf4be4b339bf520517d2d00390f8bc3b249e1 in incubator-nifi's branch refs/heads/develop from Joseph Witt [ https://git-wip-us.apache.org/repos/asf?p=incubator-nifi.git;h=a94cf4b ] Merge branch 'develop' into NIFI-169
          Hide
          joewitt Joseph Witt added a comment -

          updated nifi website to reflect new guidance for build process.

          Show
          joewitt Joseph Witt added a comment - updated nifi website to reflect new guidance for build process.

            People

            • Assignee:
              joewitt Joseph Witt
              Reporter:
              khmarbaise Karl Heinz Marbaise
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development