Incubator
  1. Incubator
  2. INCUBATOR-78

Unreliable report schedule and list of podlings

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      As discussed in http://markmail.org/message/le55q754m4eypu7r it would be useful to generate the podlings report schedule and the list of projects on the http://incubator.apache.org/ automatically from a master file.

        Activity

        Hide
        Bertrand Delacretaz added a comment -

        As a suggestion, each incubating project could have a status file at http://svn.apache.org/repos/asf/incubator/PROJECT/trunk/INCUBATOR-INFO.txt, where PROJECT is the project name.

        It's easy to verify with a script that this file is present, and putting it in the project's trunk makes it visible to and easily maintainable by the project community, as opposed to something that's stored away in incubator/trunk.

        The info might be something like:

        name: Buzzword
        description: Buzzword is a framework to create buzzword-compliant webapps in Java
        incubating-since: 2008-05-21
        incubation-status: incubating
        mentors: jdoe@apache.org, davinci@apache.org, spongebob@apache.org
        reporting-months: May, June, July, October
        homepage: http://incubator.apache.org/buzzword
        committers-list: http://incubator.apache.org/buzzword/whoweare.html

        Show
        Bertrand Delacretaz added a comment - As a suggestion, each incubating project could have a status file at http://svn.apache.org/repos/asf/incubator/PROJECT/trunk/INCUBATOR-INFO.txt , where PROJECT is the project name. It's easy to verify with a script that this file is present, and putting it in the project's trunk makes it visible to and easily maintainable by the project community, as opposed to something that's stored away in incubator/trunk. The info might be something like: name: Buzzword description: Buzzword is a framework to create buzzword-compliant webapps in Java incubating-since: 2008-05-21 incubation-status: incubating mentors: jdoe@apache.org, davinci@apache.org, spongebob@apache.org reporting-months: May, June, July, October homepage: http://incubator.apache.org/buzzword committers-list: http://incubator.apache.org/buzzword/whoweare.html
        Hide
        Roland Weber added a comment -

        Projects could use svn:externals to link their status file from a location in incubator/trunk into their project tree. Then people who just want to track the status files don't have to check out each project trunk individually.

        While it would also be possible to use svn:externals in incubator/trunk to link to all the status files in the project trunks, that would create yet another metadata location.

        cheers,
        Roland

        Show
        Roland Weber added a comment - Projects could use svn:externals to link their status file from a location in incubator/trunk into their project tree. Then people who just want to track the status files don't have to check out each project trunk individually. While it would also be possible to use svn:externals in incubator/trunk to link to all the status files in the project trunks, that would create yet another metadata location. cheers, Roland
        Hide
        Bertrand Delacretaz added a comment -

        With your proposal, if the svn:externals are not setup correctly, a project might be missing without it being noticed.

        The idea with http://svn.apache.org/repos/asf/incubator/PROJECT/trunk/INCUBATOR-INFO.txt is to use a recursive wget job, for example, which starts at http://svn.apache.org/repos/asf/incubator, visits all subfolders and checks for that file.

        In this case, as soon as a project has an svn repository there, the script can check its status file and complain if absent or incomplete - this would make it impossible for projects to be missed, as is the case now.

        Show
        Bertrand Delacretaz added a comment - With your proposal, if the svn:externals are not setup correctly, a project might be missing without it being noticed. The idea with http://svn.apache.org/repos/asf/incubator/PROJECT/trunk/INCUBATOR-INFO.txt is to use a recursive wget job, for example, which starts at http://svn.apache.org/repos/asf/incubator , visits all subfolders and checks for that file. In this case, as soon as a project has an svn repository there, the script can check its status file and complain if absent or incomplete - this would make it impossible for projects to be missed, as is the case now.
        Hide
        Sebb added a comment -

        Note that some incubator projects don't currently have trunk directories.
        This probably needs to be sorted out.

        Having the file under trunk means that it will appear in branches and tags, and will probably appear in releases as well.
        This may or may not be seen as a problem.

        Might be worth considering storing the file at the same level as trunk?
        This is a bit more awkward to edit (need to check out non-recursively) but is easier to find when navigating SVN.

        Show
        Sebb added a comment - Note that some incubator projects don't currently have trunk directories. This probably needs to be sorted out. Having the file under trunk means that it will appear in branches and tags, and will probably appear in releases as well. This may or may not be seen as a problem. Might be worth considering storing the file at the same level as trunk? This is a bit more awkward to edit (need to check out non-recursively) but is easier to find when navigating SVN.
        Hide
        Robert Burrell Donkin added a comment -

        That's why I was wondering about reusing DOAP. We could ask all podlings to register DOAP with http//projects.apache.org and then just analyse the RDF collected under the Incubator PMC.

        Show
        Robert Burrell Donkin added a comment - That's why I was wondering about reusing DOAP. We could ask all podlings to register DOAP with http//projects.apache.org and then just analyse the RDF collected under the Incubator PMC.
        Hide
        Robert Burrell Donkin added a comment -

        I think using an algorithm to assign podlings to reporting months would be better than manual assignment.

        Show
        Robert Burrell Donkin added a comment - I think using an algorithm to assign podlings to reporting months would be better than manual assignment.
        Hide
        Henning Schmiedehausen added a comment -

        why not re-use marvin? It works fine for the TLPs.

        Show
        Henning Schmiedehausen added a comment - why not re-use marvin? It works fine for the TLPs.
        Hide
        Bertrand Delacretaz added a comment -

        Do you know where the marvin code can be found?

        Show
        Bertrand Delacretaz added a comment - Do you know where the marvin code can be found?
        Show
        Jukka Zitting added a comment - +1 to Marvin. AFAIK it's in https://svn.apache.org/repos/asf/infrastructure/trunk/tools/board_reminders/
        Hide
        Craig L Russell added a comment -

        If we all agree that http://wiki.apache.org/incubator/ReportingSchedule is normative, basing the reminders on this page seems to be non-controversial.

        Let's not create yet another duplicate place to keep this information.

        And I'll repeat my earlier comment that a global reminder sent to each podling's dev list that refers to the above page would be fine pending implementation of a smarter reminder service.

        Show
        Craig L Russell added a comment - If we all agree that http://wiki.apache.org/incubator/ReportingSchedule is normative, basing the reminders on this page seems to be non-controversial. Let's not create yet another duplicate place to keep this information. And I'll repeat my earlier comment that a global reminder sent to each podling's dev list that refers to the above page would be fine pending implementation of a smarter reminder service.
        Hide
        Robert Burrell Donkin added a comment -

        Posting a reminder to every podling's dev list requires a list of dev lists. ATM we don't maindate names so we'd still need to address the meta-data problem.

        Show
        Robert Burrell Donkin added a comment - Posting a reminder to every podling's dev list requires a list of dev lists. ATM we don't maindate names so we'd still need to address the meta-data problem.
        Hide
        Henning Schmiedehausen added a comment -

        I thought every podling has a ppmc list? Aren't they recorded somewhere?

        Show
        Henning Schmiedehausen added a comment - I thought every podling has a ppmc list? Aren't they recorded somewhere?
        Hide
        Craig L Russell added a comment -

        Robert Burrell Donkin opined:
        > I think using an algorithm to assign podlings to reporting months would be better than manual assignment.

        I'm mentoring a few projects and prefer that they all have the same reporting schedule, so for me, manual assignment is preferable.

        Show
        Craig L Russell added a comment - Robert Burrell Donkin opined: > I think using an algorithm to assign podlings to reporting months would be better than manual assignment. I'm mentoring a few projects and prefer that they all have the same reporting schedule, so for me, manual assignment is preferable.
        Hide
        Bertrand Delacretaz added a comment -

        > If we all agree that http://wiki.apache.org/incubator/ReportingSchedule is normative, basing the reminders on
        > this page seems to be non-controversial.

        As recently missed report reminders show, projects are missing (or at least were missing) from that page - basing the list of projects on something more automated like svn repositories or a master file stored in svn sounds much safer and scalable.

        That wouldn't prevent creating report schedules by hand, but for the schedule to be reliable there needs IMHO to be an automated way of checking it against reference info, to make sure there are no phantom projects as is the case now.

        From memory, at least Pig, Rat and Bluesky have been forgotten in the reporting schedule, without anybody noticing for quite some time.

        Show
        Bertrand Delacretaz added a comment - > If we all agree that http://wiki.apache.org/incubator/ReportingSchedule is normative, basing the reminders on > this page seems to be non-controversial. As recently missed report reminders show, projects are missing (or at least were missing) from that page - basing the list of projects on something more automated like svn repositories or a master file stored in svn sounds much safer and scalable. That wouldn't prevent creating report schedules by hand, but for the schedule to be reliable there needs IMHO to be an automated way of checking it against reference info, to make sure there are no phantom projects as is the case now. From memory, at least Pig, Rat and Bluesky have been forgotten in the reporting schedule, without anybody noticing for quite some time.
        Hide
        Robert Burrell Donkin added a comment -

        Re: PIG, RAT, Bluesky - the current process doesn't take into account a long bootstrap period

        Show
        Robert Burrell Donkin added a comment - Re: PIG, RAT, Bluesky - the current process doesn't take into account a long bootstrap period
        Hide
        Robert Burrell Donkin added a comment -

        Re: Mailing lists AIUI we don't maintain meta- data for podlings. So we don't know what mailing lists they have.

        It should be possible to scrape mail-archives.a.org and use some heuristics to generate a list

        Show
        Robert Burrell Donkin added a comment - Re: Mailing lists AIUI we don't maintain meta- data for podlings. So we don't know what mailing lists they have. It should be possible to scrape mail-archives.a.org and use some heuristics to generate a list
        Hide
        Robert Burrell Donkin added a comment -

        Re: maunal verses automatic reporting schedules. I would be happy with a manual list provided that it was:

        • on the main website
        • easily machine readable
        Show
        Robert Burrell Donkin added a comment - Re: maunal verses automatic reporting schedules. I would be happy with a manual list provided that it was: on the main website easily machine readable
        Hide
        Noel J. Bergman added a comment -

        > As a suggestion, each incubating project could have a status file at [...] where PROJECT is the project name.

        They already do. See all of the $

        {PROJECT}

        .xml files under http://svn.apache.org/viewvc/incubator/public/trunk/site-author/projects

        Show
        Noel J. Bergman added a comment - > As a suggestion, each incubating project could have a status file at [...] where PROJECT is the project name. They already do. See all of the $ {PROJECT} .xml files under http://svn.apache.org/viewvc/incubator/public/trunk/site-author/projects
        Hide
        David Crossley added a comment -

        We can parse the generated table "Currently in incubation" at http://incubator.apache.org/projects/ which has most of the information that we need.

        However, this is still not reliable because we need to know which projects were accepted. Often they miss the first two steps: adding their status to that table, and adding themselves to the reporting schedule Wiki page: http://wiki.apache.org/incubator/ReportingSchedule

        See some more discussion at:
        Re: up-to-date list of projects in incubation
        http://markmail.org/message/qxemvfbsljdwbflj

        If their mentors don't add to the reporting schedule soon after acceptance, then another IPMC member needs to jump in to do it.

        Then we can parse that wiki ReportingSchedule and correlate with the projects index. Less maintenance of source data, and alerts to missing status info pages.

        Show
        David Crossley added a comment - We can parse the generated table "Currently in incubation" at http://incubator.apache.org/projects/ which has most of the information that we need. However, this is still not reliable because we need to know which projects were accepted. Often they miss the first two steps: adding their status to that table, and adding themselves to the reporting schedule Wiki page: http://wiki.apache.org/incubator/ReportingSchedule See some more discussion at: Re: up-to-date list of projects in incubation http://markmail.org/message/qxemvfbsljdwbflj If their mentors don't add to the reporting schedule soon after acceptance, then another IPMC member needs to jump in to do it. Then we can parse that wiki ReportingSchedule and correlate with the projects index. Less maintenance of source data, and alerts to missing status info pages.
        Hide
        David Crossley added a comment -

        Today i added a tool called "clutch" which addresses some of this need. See r705975.

        See the notice to the general mailing list:
        Re: new tool "clutch" assist oversight of all podlings
        http://markmail.org/message/yy6cb2gokk5p2ur7

        and the website at http://incubator.apache.org/clutch.html

        Show
        David Crossley added a comment - Today i added a tool called "clutch" which addresses some of this need. See r705975. See the notice to the general mailing list: Re: new tool "clutch" assist oversight of all podlings http://markmail.org/message/yy6cb2gokk5p2ur7 and the website at http://incubator.apache.org/clutch.html
        Hide
        Gavin added a comment -

        The use of Marvin for report reminders is the only outstanding item I can see for this issue, and that is now in operation, so closing.

        Show
        Gavin added a comment - The use of Marvin for report reminders is the only outstanding item I can see for this issue, and that is now in operation, so closing.

          People

          • Assignee:
            Unassigned
            Reporter:
            Bertrand Delacretaz
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development