Hadoop Common
  1. Hadoop Common
  2. HADOOP-7723

Automatically generate good Release Notes

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.20.204.0, 0.23.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Target Version/s:

      Description

      In branch-0.20-security, there is a tool src/docs/relnotes.py, that automatically generates Release Notes. Fix deficiencies and port it up to trunk.

        Activity

        Hide
        Matt Foley added a comment -

        Many of our peer projects generate trivial "release notes" as a list of bugs fixed, giving the bug number and one-line description, for instance as auto-generated by Jira under:

        Project > Road Map (or Change Log) > Release Notes

        e.g., for 0.20.205.0:

        https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310942&version=12316392

        However, the Hadoop project has tried to do better than this, by actually collecting "Release Note" field values from fixed bugs (or Description fields from bugs with empty Release Note fields), and presenting them in the form of the releasenotes.html document template, e.g., for 0.20.205.0:

        http://hadoop.apache.org/common/docs/r0.20.205.0/releasenotes.html

        When doing the notes for 0.20.205.0, I found that the tool for doing these collected Release Notes (src/docs/relnotes.py) was broken in a couple of respects:

        • It was inconsistent with the documented process in HowToRelease, because it wanted bug lists piped in somewhat differently.
        • It assumed that Jira's report on "Resolved" bugs was sufficient, while that list often differs somewhat from CHANGES.txt. In particular, bugs held open for ports to other branches would not be reported as Resolved in the current branch.
        • Most critically, the feature to extract the "Release Note" field from jira issues doesn't work unless the person running it has top-level Jira admin privs (not just admin privs for the Hadoop projects). This restriction is built in to the Jira CLI tool ('jira.sh').

        I fixed these issues, and will submit the improved tool for review. It now does the following:

        • Query Jira for bugs resolved in the current release.
        • Query CHANGES.txt for bugs resolved in the current release.
        • Merge and diff the two lists, reporting the result and giving the Release Manager an opportunity to resolve the variances.
        • Look up the Release Note field for each resolved bug, scraping it from a 'curl' call rather than the admin-restricted Jira CLI tool.
        • If there is no Release Note, use the Description field but limit it to the first 500 characters, in case the Description is long.
        • Format as before.

        I also suggest these enhancements:

        • List the jiras with Release Notes first. These are usually the larger or incompatible changes that most readers will care about most. Then list the other jiras with their descriptions.
        • Sort in forward numerical order, instead of reverse.
        Show
        Matt Foley added a comment - Many of our peer projects generate trivial "release notes" as a list of bugs fixed, giving the bug number and one-line description, for instance as auto-generated by Jira under: Project > Road Map (or Change Log) > Release Notes e.g., for 0.20.205.0: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310942&version=12316392 However, the Hadoop project has tried to do better than this, by actually collecting "Release Note" field values from fixed bugs (or Description fields from bugs with empty Release Note fields), and presenting them in the form of the releasenotes.html document template, e.g., for 0.20.205.0: http://hadoop.apache.org/common/docs/r0.20.205.0/releasenotes.html When doing the notes for 0.20.205.0, I found that the tool for doing these collected Release Notes (src/docs/relnotes.py) was broken in a couple of respects: It was inconsistent with the documented process in HowToRelease, because it wanted bug lists piped in somewhat differently. It assumed that Jira's report on "Resolved" bugs was sufficient, while that list often differs somewhat from CHANGES.txt. In particular, bugs held open for ports to other branches would not be reported as Resolved in the current branch. Most critically, the feature to extract the "Release Note" field from jira issues doesn't work unless the person running it has top-level Jira admin privs (not just admin privs for the Hadoop projects). This restriction is built in to the Jira CLI tool ('jira.sh'). I fixed these issues, and will submit the improved tool for review. It now does the following: Query Jira for bugs resolved in the current release. Query CHANGES.txt for bugs resolved in the current release. Merge and diff the two lists, reporting the result and giving the Release Manager an opportunity to resolve the variances. Look up the Release Note field for each resolved bug, scraping it from a 'curl' call rather than the admin-restricted Jira CLI tool. If there is no Release Note, use the Description field but limit it to the first 500 characters, in case the Description is long. Format as before. I also suggest these enhancements: List the jiras with Release Notes first. These are usually the larger or incompatible changes that most readers will care about most. Then list the other jiras with their descriptions. Sort in forward numerical order, instead of reverse.
        Hide
        Matt Foley added a comment -

        Just found that all instances of <br/> in body text is being html-ified to "<br/>"
        Should make it stop doing that.

        Show
        Matt Foley added a comment - Just found that all instances of <br/> in body text is being html-ified to "<br/>" Should make it stop doing that.
        Hide
        Matt Foley added a comment -

        the above should read: is being html-ified to "& lt ; br / & gt ;"

        Show
        Matt Foley added a comment - the above should read: is being html-ified to "& lt ; br / & gt ;"
        Hide
        Matt Foley added a comment -

        Changed Target Version to 1.3.0 upon release of 1.2.0. Please change to 1.2.1 if you intend to submit a fix for branch-1.2.

        Show
        Matt Foley added a comment - Changed Target Version to 1.3.0 upon release of 1.2.0. Please change to 1.2.1 if you intend to submit a fix for branch-1.2.
        Hide
        Mit Desai added a comment -

        I see that this Jira is still targated to 0.23.0. Is this still needed in 0.23 or 2.X? If so we can re-target this accordingly. Or else, we can close this Jira for now.

        Show
        Mit Desai added a comment - I see that this Jira is still targated to 0.23.0. Is this still needed in 0.23 or 2.X? If so we can re-target this accordingly. Or else, we can close this Jira for now.
        Hide
        Mit Desai added a comment -

        Haven't heard back since a while. As 0.23 is going into maintenance mode, removing 0.23 from the target versions.

        Show
        Mit Desai added a comment - Haven't heard back since a while. As 0.23 is going into maintenance mode, removing 0.23 from the target versions.
        Hide
        Allen Wittenauer added a comment -

        ping

        It'd be good to get these improvements in....

        Show
        Allen Wittenauer added a comment - ping It'd be good to get these improvements in....

          People

          • Assignee:
            Matt Foley
            Reporter:
            Matt Foley
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:

              Development