Infrastructure
  1. Infrastructure
  2. INFRA-5819

Request to set up Buildbot for OpenJPA documentation build

    Details

    • Type: Task Task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Fix Version/s: Initial Clearing
    • Component/s: Buildbot
    • Labels:
      None

      Description

      OpenJPA project would like to use the Buildbot feature for its daily documentation build process. We plan to upload the nightly built documentation to ci.apache.org which allows for public access.

      Please also let us know:
      - where we can get more info on how to set up the build and up-load process
      - where in http://ci.apache.org/projects/** OpenJPA can use to deposit our nightly built documentation

      Thanks,
      Albert

        Activity

        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Waiting for Infra Waiting for Infra Waiting for user Waiting for user
        33d 11h 10m 4 Gavin 11/Mar/13 01:16
        Waiting for user Waiting for user Waiting for Infra Waiting for Infra
        6d 18h 43m 4 Albert Lee 12/Mar/13 22:13
        Waiting for Infra Waiting for Infra Closed Closed
        2d 22h 57m 1 Albert Lee 15/Mar/13 21:11
        Gavin made changes -
        Fix Version/s Initial Clearing [ 12325964 ]
        Albert Lee made changes -
        Status Waiting for Infra [ 10011 ] Closed [ 6 ]
        Resolution Fixed [ 1 ]
        Hide
        Albert Lee added a comment -
        I have removed 1.1.x doc build from the list in openjpa.conf since it is a old-ball release and not really used. The rest of the releases look good.

        Thanks,
        Albert Lee.
        Show
        Albert Lee added a comment - I have removed 1.1.x doc build from the list in openjpa.conf since it is a old-ball release and not really used. The rest of the releases look good. Thanks, Albert Lee.
        Albert Lee made changes -
        Status Waiting for user [ 10010 ] Waiting for Infra [ 10011 ]
        Hide
        Albert Lee added a comment -
        1a) OpenJPA 1.0.x and 1.1.x use different mvn options to process. I have updated the OpenJPA build to include a new Maven profile for these releases. I have also added the new profile usage to openjpa.conf and should fix this problem.

        1b) OpenJPA 1.*.* release needs JDK 1.5 to compile. I have no idea how the buildbot script changes needed to accommodate this requirement.

        2) As far as trigger frequency. we can use Nightly Scheduler; if possible trigger only if any commit has been performed.

        Thanks,
        Albert Lee.
        Show
        Albert Lee added a comment - 1a) OpenJPA 1.0.x and 1.1.x use different mvn options to process. I have updated the OpenJPA build to include a new Maven profile for these releases. I have also added the new profile usage to openjpa.conf and should fix this problem. 1b) OpenJPA 1.*.* release needs JDK 1.5 to compile. I have no idea how the buildbot script changes needed to accommodate this requirement. 2) As far as trigger frequency. we can use Nightly Scheduler; if possible trigger only if any commit has been performed. Thanks, Albert Lee.
        Gavin made changes -
        Status Waiting for Infra [ 10011 ] Waiting for user [ 10010 ]
        Hide
        Gavin added a comment -
        Ok so, now we have quite a bit more done.

        Passing Builds:

        http://ci.apache.org/builders/openjpa-2.0.x-docs
        http://ci.apache.org/builders/openjpa-2.1.x-docs
        http://ci.apache.org/builders/openjpa-2.2.1.x-docs
        http://ci.apache.org/builders/openjpa-2.2.x-docs
        http://ci.apache.org/builders/openjpa-trunk-docs

        Passing builds have their docs all uploaded successfully to:

        http://ci.apache.org/projects/openjpa/$version/**

        Failed Builds:

        http://ci.apache.org/builders/openjpa-1.0.x-docs
        http://ci.apache.org/builders/openjpa-1.1.x-docs
        http://ci.apache.org/builders/openjpa-1.2.x-docs
        http://ci.apache.org/builders/openjpa-1.3.x-docs


        So, questions for you:

        1. what to do with the failed builds ? Do we need different mvn commands or mvn/java versions to run these ?

        2. Currently all builds I trigger manually for testing purposes. Going forward, how do you want these docs triggered ?
             Options are a) per commit trigger; b) Nightly Scheduler; c) Weekly Scheduler; d) something else(?)


        Feel free to check the updated config file:

        https://svn.apache.org/repos/infra/infrastructure/buildbot/aegis/buildmaster/master1/projects/openjpa.conf

        Thx.
        Show
        Gavin added a comment - Ok so, now we have quite a bit more done. Passing Builds: http://ci.apache.org/builders/openjpa-2.0.x-docs http://ci.apache.org/builders/openjpa-2.1.x-docs http://ci.apache.org/builders/openjpa-2.2.1.x-docs http://ci.apache.org/builders/openjpa-2.2.x-docs http://ci.apache.org/builders/openjpa-trunk-docs Passing builds have their docs all uploaded successfully to: http://ci.apache.org/projects/openjpa/$version/** Failed Builds: http://ci.apache.org/builders/openjpa-1.0.x-docs http://ci.apache.org/builders/openjpa-1.1.x-docs http://ci.apache.org/builders/openjpa-1.2.x-docs http://ci.apache.org/builders/openjpa-1.3.x-docs So, questions for you: 1. what to do with the failed builds ? Do we need different mvn commands or mvn/java versions to run these ? 2. Currently all builds I trigger manually for testing purposes. Going forward, how do you want these docs triggered ?      Options are a) per commit trigger; b) Nightly Scheduler; c) Weekly Scheduler; d) something else(?) Feel free to check the updated config file: https://svn.apache.org/repos/infra/infrastructure/buildbot/aegis/buildmaster/master1/projects/openjpa.conf Thx.
        Hide
        Gavin added a comment -
        Got it working for trunk

        http://ci.apache.org/projects/openjpa/trunk/

        all there.
        Show
        Gavin added a comment - Got it working for trunk http://ci.apache.org/projects/openjpa/trunk/ all there.
        Hide
        Albert Lee added a comment -
        Anything I can help to keep this moving forward?

        Thanks,
        Albert Lee.
        Show
        Albert Lee added a comment - Anything I can help to keep this moving forward? Thanks, Albert Lee.
        Albert Lee made changes -
        Status Waiting for user [ 10010 ] Waiting for Infra [ 10011 ]
        Hide
        Albert Lee added a comment -
        Gavin,

        There are 3 sets of artifacts we need from the build to ci.apache.org (cao) site:

        1) manual - This the 3 links you had picked up and copied to the cao. This is what we need..
        2) javadoc - The javadoc are built and zipped in openjpa-project\target\site\downloads\*-binary.zip!\apache-openjpa-*-SNAPSHOT\docs\javadoc\**
        3) schema - There are 2 schema files built and zipped in openjpa-project\target\site\downloads\*-binary.zip!\apache-openjpa-*-SNAPSHOT\docs\schema\*.xsd

        The manual (1) is also zipped in openjpa-project\target\site\downloads\*-binary.zip!\apache-openjpa-*-SNAPSHOT\docs\docbook\manual\**, You may want to unzip openjpa-project\target\site\downloads\*-binary.zip once and use it for the copy for all 3 folders.

        The ant script we used to extract is:

                <unzip dest="target/site/tempDocs">
                    <fileset dir="target/site/downloads">
                        <include name="*binary*.zip" />
                    </fileset>
                    <patternset>
                        <include name="**/javadoc/**" />
                        <include name="**/docbook/**" />
                        <include name="**/schema/**"/>
                    </patternset>
                </unzip>

        These 3 sets of artifact can go to :
        1) /home/buildmaster/master1/public_html/projects/openjpa/trunk/manual
        2) /home/buildmaster/master1/public_html/projects/openjpa/trunk/javadoc
        3) /home/buildmaster/master1/public_html/projects/openjpa/trunk/schema

        or something similar that fit the perl script.

        Once the trunk is complete, we need the same buildbot configuration for the following releases:

        svn.apache.org/repos/asf/openjpa/branches/1.0.x
        svn.apache.org/repos/asf/openjpa/branches/1.1.x
        svn.apache.org/repos/asf/openjpa/branches/1.2.x
        svn.apache.org/repos/asf/openjpa/branches/1.3.x
        svn.apache.org/repos/asf/openjpa/branches/2.0.x
        svn.apache.org/repos/asf/openjpa/branches/2.1.x
        svn.apache.org/repos/asf/openjpa/branches/2.2.x
        svn.apache.org/repos/asf/openjpa/branches/2.2.1.x

        Thanks again for all the help. If there is anything we can do, let us know.

        Albert Lee.
        Show
        Albert Lee added a comment - Gavin, There are 3 sets of artifacts we need from the build to ci.apache.org (cao) site: 1) manual - This the 3 links you had picked up and copied to the cao. This is what we need.. 2) javadoc - The javadoc are built and zipped in openjpa-project\target\site\downloads\*-binary.zip!\apache-openjpa-*-SNAPSHOT\docs\javadoc\** 3) schema - There are 2 schema files built and zipped in openjpa-project\target\site\downloads\*-binary.zip!\apache-openjpa-*-SNAPSHOT\docs\schema\*.xsd The manual (1) is also zipped in openjpa-project\target\site\downloads\*-binary.zip!\apache-openjpa-*-SNAPSHOT\docs\docbook\manual\**, You may want to unzip openjpa-project\target\site\downloads\*-binary.zip once and use it for the copy for all 3 folders. The ant script we used to extract is:         <unzip dest="target/site/tempDocs">             <fileset dir="target/site/downloads">                 <include name="*binary*.zip" />             </fileset>             <patternset>                 <include name="**/javadoc/**" />                 <include name="**/docbook/**" />                 <include name="**/schema/**"/>             </patternset>         </unzip> These 3 sets of artifact can go to : 1) /home/buildmaster/master1/public_html/projects/openjpa/trunk/manual 2) /home/buildmaster/master1/public_html/projects/openjpa/trunk/javadoc 3) /home/buildmaster/master1/public_html/projects/openjpa/trunk/schema or something similar that fit the perl script. Once the trunk is complete, we need the same buildbot configuration for the following releases: svn.apache.org/repos/asf/openjpa/branches/1.0.x svn.apache.org/repos/asf/openjpa/branches/1.1.x svn.apache.org/repos/asf/openjpa/branches/1.2.x svn.apache.org/repos/asf/openjpa/branches/1.3.x svn.apache.org/repos/asf/openjpa/branches/2.0.x svn.apache.org/repos/asf/openjpa/branches/2.1.x svn.apache.org/repos/asf/openjpa/branches/2.2.x svn.apache.org/repos/asf/openjpa/branches/2.2.1.x Thanks again for all the help. If there is anything we can do, let us know. Albert Lee.
        Gavin made changes -
        Status Waiting for Infra [ 10011 ] Waiting for user [ 10010 ]
        Hide
        Gavin added a comment -
        See svn commit: r851481 - This uploads the docbook , can you specify any other directories from the build that you would like uploading? Or is this mission complete?

        See

        http://ci.apache.org/projects/openjpa/trunk/manual/manual.pdf
        http://ci.apache.org/projects/openjpa/trunk/manual/manual.html
        http://ci.apache.org/projects/openjpa/trunk/manual/manual/index.html
        Show
        Gavin added a comment - See svn commit: r851481 - This uploads the docbook , can you specify any other directories from the build that you would like uploading? Or is this mission complete? See http://ci.apache.org/projects/openjpa/trunk/manual/manual.pdf http://ci.apache.org/projects/openjpa/trunk/manual/manual.html http://ci.apache.org/projects/openjpa/trunk/manual/manual/index.html
        Hide
        Albert Lee added a comment -
        The basic buildbot configuration seems to be working for trunk-docs as in:

        http://ci.apache.org/builders/openjpa-trunk-docs

        I wonder how we can move/copy the artifacts built by this configuration to the appropriate location in ci.apache.org for public viewing.

        If there is anything we can help, please give us the instructions how to proceed.

        Thanks,
        Albert Lee.
        Show
        Albert Lee added a comment - The basic buildbot configuration seems to be working for trunk-docs as in: http://ci.apache.org/builders/openjpa-trunk-docs I wonder how we can move/copy the artifacts built by this configuration to the appropriate location in ci.apache.org for public viewing. If there is anything we can help, please give us the instructions how to proceed. Thanks, Albert Lee.
        Albert Lee made changes -
        Status Waiting for user [ 10010 ] Waiting for Infra [ 10011 ]
        Hide
        Albert Lee added a comment -
         I have updated the openjpa.conf to the mvn parameters we use for our nightly build. Mainly as in:

        f_ojpa_1.addStep(Compile(command=["mvn" , "-U" , "install" , "-DskipTests" , "-Pjavadoc-profile,docbook-profile"]))

        For the locations where the built artifacts for the upload, it is described in the "nightly-upload" profile in openjpa-project/pom.xml. The ant file it launches is openjpa-project/upload.xml.

        Thank you so much for working on this request.

        Albert Lee.
        Show
        Albert Lee added a comment -  I have updated the openjpa.conf to the mvn parameters we use for our nightly build. Mainly as in: f_ojpa_1.addStep(Compile(command=["mvn" , "-U" , "install" , "-DskipTests" , "-Pjavadoc-profile,docbook-profile"])) For the locations where the built artifacts for the upload, it is described in the "nightly-upload" profile in openjpa-project/pom.xml. The ant file it launches is openjpa-project/upload.xml. Thank you so much for working on this request. Albert Lee.
        Gavin made changes -
        Status Waiting for Infra [ 10011 ] Waiting for user [ 10010 ]
        Hide
        Gavin added a comment - - edited
        [~allee8285]

        I've made a start.

        http://ci.apache.org/builders/openjpa-trunk-docs/builds/1/steps/compile/logs/stdio

        I guess from the failure my mvn command isnt quite right, do I need to do an install first?
        (docs say all that is needed is the command Im using:

                    <!--
                        Javadoc profile. Docs can be built by running:
                            mvn process-resources -Dtest=false -Pjavadoc-profile
                    -->
        )

        All I want at this stage is the complete docs and javadocs, pdfs etc all in one target directory,
        we can then use another buildbot step to upload them to ci.apache.org/projects/openjpa

        So we do not want to do any external deploy , we'll let buildbot handle the upload from a local
        target directory.

        Please let me know how I should proceed.

        OR - feel free to edit the config yourself at:

        https://svn.apache.org/repos/infra/infrastructure/buildbot/aegis/buildmaster/master1/projects/openjpa.conf

        we'll just get trunk running at this stage, then you can add others later.
        Show
        Gavin added a comment - - edited [~allee8285] I've made a start. http://ci.apache.org/builders/openjpa-trunk-docs/builds/1/steps/compile/logs/stdio I guess from the failure my mvn command isnt quite right, do I need to do an install first? (docs say all that is needed is the command Im using:             <!--                 Javadoc profile. Docs can be built by running:                     mvn process-resources -Dtest=false -Pjavadoc-profile             --> ) All I want at this stage is the complete docs and javadocs, pdfs etc all in one target directory, we can then use another buildbot step to upload them to ci.apache.org/projects/openjpa So we do not want to do any external deploy , we'll let buildbot handle the upload from a local target directory. Please let me know how I should proceed. OR - feel free to edit the config yourself at: https://svn.apache.org/repos/infra/infrastructure/buildbot/aegis/buildmaster/master1/projects/openjpa.conf we'll just get trunk running at this stage, then you can add others later.
        Gavin made changes -
        Assignee Gavin [ ipv6guru ]
        Albert Lee made changes -
        Status Waiting for user [ 10010 ] Waiting for Infra [ 10011 ]
        Hide
        Albert Lee added a comment -
        OpenJPA uses svn.apache.org for source repository (http://svn.apache.org/repos/asf/openjpa/[branches/**/ | trunk/ ] and maven to drive build processes.

        Currently we need to build 8 releases (from 1.1.x to trunk). For documentation build, we do something like:

        mvn -Pdeploy-external clean deploy

        This builds all the OptnJPA documentations and at the end of the process, an ant script is run to upload the artifacts to people.apache.org/~person/**. This includes some manual in pdf format and fully expanded javadoc etc. See links in http://openjpa.apache.org/documentation.html.

        I guess we can create a new script after the document build and upload the artifacts to http://ci.apache.org/projects/openjpa/**.

        I am new to the Apache build infrastructure. Any more detail info, instructions or sample would help to facilitate this migration.

        Thanks,
        Albert Lee.
        Show
        Albert Lee added a comment - OpenJPA uses svn.apache.org for source repository ( http://svn.apache.org/repos/asf/openjpa/ [branches/**/ | trunk/ ] and maven to drive build processes. Currently we need to build 8 releases (from 1.1.x to trunk). For documentation build, we do something like: mvn -Pdeploy-external clean deploy This builds all the OptnJPA documentations and at the end of the process, an ant script is run to upload the artifacts to people.apache.org/~person/**. This includes some manual in pdf format and fully expanded javadoc etc. See links in http://openjpa.apache.org/documentation.html . I guess we can create a new script after the document build and upload the artifacts to http://ci.apache.org/projects/openjpa/** . I am new to the Apache build infrastructure. Any more detail info, instructions or sample would help to facilitate this migration. Thanks, Albert Lee.
        #asfinfra Bot made changes -
        Field Original Value New Value
        Status Waiting for Infra [ 10011 ] Waiting for user [ 10010 ]
        Hide
        #asfinfra Bot added a comment -
        <danielsh> What commands (or script) build the documentation from a working copy?
        Show
        #asfinfra Bot added a comment - <danielsh> What commands (or script) build the documentation from a working copy?
        Albert Lee created issue -

          People

          • Assignee:
            Gavin
            Reporter:
            Albert Lee
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development