Details

    • Type: Task
    • Status: Reopened
    • Priority: Blocker
    • Resolution: Unresolved
    • Affects Version/s: Impala 2.8.0, Impala 2.9.0
    • Fix Version/s: Impala 2.9.0
    • Component/s: Docs
    • Labels:

      Description

      2.8.0 was released without any notes on new features. 2.9 should have release notes, including backfilled ones for 2.8.

        Activity

        Hide
        mikesbrown Michael Brown added a comment - - edited

        I talked with Greg Rahn about this, and the acceptable release notes he was envisioning is automation-generated, changelog-style. The example he cited was https://github.com/apache/parquet-cpp/blob/apache-parquet-cpp-1.0.0/CHANGELOG . I think this is something that I can do, reducing the burden of doc writers.

        Show
        mikesbrown Michael Brown added a comment - - edited I talked with Greg Rahn about this, and the acceptable release notes he was envisioning is automation-generated, changelog-style. The example he cited was https://github.com/apache/parquet-cpp/blob/apache-parquet-cpp-1.0.0/CHANGELOG . I think this is something that I can do, reducing the burden of doc writers.
        Hide
        jbapple Jim Apple added a comment -

        When you have your script for this, can you add instructions to <https://cwiki.apache.org/confluence/display/IMPALA/DRAFT%3A+How+to+Release>?

        Show
        jbapple Jim Apple added a comment - When you have your script for this, can you add instructions to < https://cwiki.apache.org/confluence/display/IMPALA/DRAFT%3A+How+to+Release >?
        Hide
        mikesbrown Michael Brown added a comment -

        Absolutely; thank you for the pointer.

        Show
        mikesbrown Michael Brown added a comment - Absolutely; thank you for the pointer.
        Hide
        grahn Greg Rahn added a comment -
        Show
        grahn Greg Rahn added a comment - Looks like those CHANGELOG notes are JIRA based. See https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12321021&version=12340208
        Hide
        mikesbrown Michael Brown added a comment -

        Thanks for that tip Greg Rahn. I found the instructions here https://confluence.atlassian.com/adminjiraserver071/creating-release-notes-802592502.html

        Upside: Really easy to generate by hand.

        Potential, minor downsides if we care about some level of polish:

        1. No curating on the queries; you have to manually remove uninteresting items (like random broken build infra fixes) instead of using a JQL filter.
        2. Sub-tasks are included and in their own first-order group
        3. Sub-tasks are listed first, then bugs, improvements, etc. This is probably due to just how the issue types happen to be stored in Jira.

        If we use this feature, it would require a user to manually curate the list to change the ordering to something nicer, and perhaps prune uninteresting issues from the set. It seems like this is something that could be at the discretion of a release manager and should take maybe 5 minutes. I claim too that we will never be able to write the perfect JQL query to capture the exact Jiras we'd want in release notes, so there will necessarily always be some manual curation of the release notes.

        It seems reasonable to provide guidelines on how to do this at https://cwiki.apache.org/confluence/display/IMPALA/DRAFT%3A+How+to+Release and see how that goes.

        Show
        mikesbrown Michael Brown added a comment - Thanks for that tip Greg Rahn . I found the instructions here https://confluence.atlassian.com/adminjiraserver071/creating-release-notes-802592502.html Upside: Really easy to generate by hand. Potential, minor downsides if we care about some level of polish: No curating on the queries; you have to manually remove uninteresting items (like random broken build infra fixes) instead of using a JQL filter. Sub-tasks are included and in their own first-order group Sub-tasks are listed first, then bugs, improvements, etc. This is probably due to just how the issue types happen to be stored in Jira. If we use this feature, it would require a user to manually curate the list to change the ordering to something nicer, and perhaps prune uninteresting issues from the set. It seems like this is something that could be at the discretion of a release manager and should take maybe 5 minutes. I claim too that we will never be able to write the perfect JQL query to capture the exact Jiras we'd want in release notes, so there will necessarily always be some manual curation of the release notes. It seems reasonable to provide guidelines on how to do this at https://cwiki.apache.org/confluence/display/IMPALA/DRAFT%3A+How+to+Release and see how that goes.
        Hide
        mikesbrown Michael Brown added a comment -

        I also don't mind trying this proposal retroactively for 2.8 and see how that goes. There shouldn't be anything wrong with generating release notes of 2.8 and putting them on the docs site.

        Show
        mikesbrown Michael Brown added a comment - I also don't mind trying this proposal retroactively for 2.8 and see how that goes. There shouldn't be anything wrong with generating release notes of 2.8 and putting them on the docs site.
        Hide
        jrussell John Russell added a comment -

        If I create a docs JIRA, e.g. https://issues.apache.org/jira/browse/IMPALA-5326 , is that going to end up in the report or is there a way to label/filter those out?

        Last week Greg and I looked specifically at the list of new features (since those by nature tend to require more doc work), using this JIRA report:

        https://issues.apache.org/jira/issues/?jql=project%20%3D%20IMPALA%20AND%20type%20in%20(%22New%20Feature%22)%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20in%20(%22Impala%202.9.0%22)%20AND%20component%20not%20in%20(infrastructure)%20AND%20labels%20not%20in%20(broken-build%2C%20toolchain%2C%20crash%2C%20correctness%2C%20regression%2C%20stress)%20ORDER%20BY%20key%20ASC%2C%20issuetype%20DESC

        I see that doesn't quite match up with what's in the auto-generated list. The JIRA-created release notes document includes these new features:

        IMPALA-4686
        IMPALA-4734
        IMPALA-4740
        IMPALA-4810
        IMPALA-4883

        I don't see anything obvious when I spot-check those issues to indicate my report was doing over-aggressive filtering. Do we know all the JQL clauses for the new features list produced by JIRA?

        Both my new features query and the JIRA-produced one include:

        IMPALA-4460

        which is resolved as 'Duplicate', and the one that it's a duplicate of (IMPALA-4932) is currently still open. I'll fine-tune my list. Can the JIRA-produced one also be fine-tuned to omit 'duplicate' ones? (Or is it worthwhile to let both original + duplicate JIRAs appear in the list?)

        Show
        jrussell John Russell added a comment - If I create a docs JIRA, e.g. https://issues.apache.org/jira/browse/IMPALA-5326 , is that going to end up in the report or is there a way to label/filter those out? Last week Greg and I looked specifically at the list of new features (since those by nature tend to require more doc work), using this JIRA report: https://issues.apache.org/jira/issues/?jql=project%20%3D%20IMPALA%20AND%20type%20in%20(%22New%20Feature%22)%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20in%20(%22Impala%202.9.0%22)%20AND%20component%20not%20in%20(infrastructure)%20AND%20labels%20not%20in%20(broken-build%2C%20toolchain%2C%20crash%2C%20correctness%2C%20regression%2C%20stress)%20ORDER%20BY%20key%20ASC%2C%20issuetype%20DESC I see that doesn't quite match up with what's in the auto-generated list. The JIRA-created release notes document includes these new features: IMPALA-4686 IMPALA-4734 IMPALA-4740 IMPALA-4810 IMPALA-4883 I don't see anything obvious when I spot-check those issues to indicate my report was doing over-aggressive filtering. Do we know all the JQL clauses for the new features list produced by JIRA? Both my new features query and the JIRA-produced one include: IMPALA-4460 which is resolved as 'Duplicate', and the one that it's a duplicate of ( IMPALA-4932 ) is currently still open. I'll fine-tune my list. Can the JIRA-produced one also be fine-tuned to omit 'duplicate' ones? (Or is it worthwhile to let both original + duplicate JIRAs appear in the list?)
        Hide
        mikesbrown Michael Brown added a comment -

        Thanks for taking a look at the report John Russell.

        If I create a docs JIRA, e.g. https://issues.apache.org/jira/browse/IMPALA-5326 , is that going to end up in the report or is there a way to label/filter those out?

        There is no way to filter those out using the release notes flow from Jira. https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12321021&version=12340208 shows IMPALA-3398, IMPALA-4984, and IMPALA-5006. For Apache Impala (incubating), I don't necessarily see a reason to filter these out.

        Do we know all the JQL clauses for the new features list produced by JIRA?

        No. My guess it's a simply "fixed in <version>" list group-by type. I have not done a thorough cross reference of this, though.

        IMPALA-4686

        Missed by your filter because its component is infrastructure. I'll follow up to make sure that's correct.

        IMPALA-4734

        IMPALA-4740

        IMPALA-4810

        IMPALA-4883

        I looked into this, and it's a Jira bug. https://jira.atlassian.com/browse/JRASERVER-20788 When filtering on NOT IN with labels (or any nullable field that supports NOT IN, I guess) you need to include OR labels IS EMPTY. If I adjust your JQL, these get included.

        Can the JIRA-produced one also be fine-tuned to omit 'duplicate' ones?

        Unfortunately not, since there is no way to have JQL in Jira-produced release notes. But yours can, if you include AND resolution = "Fixed".

        Or is it worthwhile to let both original + duplicate JIRAs appear in the list?

        In an ideal changelog, this is not worthwhile.

        Last, John Russell, not said, but implied here, is that you rely heavily on the "New Feature" issue type to produce documentation, and we should encourage contributors to use this correctly. For example, if I add new testing infrastructure, you would prefer that not be included as a "New Feature", because it's not a new feature of Impala proper, right?

        Show
        mikesbrown Michael Brown added a comment - Thanks for taking a look at the report John Russell . If I create a docs JIRA, e.g. https://issues.apache.org/jira/browse/IMPALA-5326 , is that going to end up in the report or is there a way to label/filter those out? There is no way to filter those out using the release notes flow from Jira. https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12321021&version=12340208 shows IMPALA-3398 , IMPALA-4984 , and IMPALA-5006 . For Apache Impala (incubating), I don't necessarily see a reason to filter these out. Do we know all the JQL clauses for the new features list produced by JIRA? No. My guess it's a simply "fixed in <version>" list group-by type. I have not done a thorough cross reference of this, though. IMPALA-4686 Missed by your filter because its component is infrastructure. I'll follow up to make sure that's correct. IMPALA-4734 IMPALA-4740 IMPALA-4810 IMPALA-4883 I looked into this, and it's a Jira bug. https://jira.atlassian.com/browse/JRASERVER-20788 When filtering on NOT IN with labels (or any nullable field that supports NOT IN , I guess) you need to include OR labels IS EMPTY . If I adjust your JQL, these get included. Can the JIRA-produced one also be fine-tuned to omit 'duplicate' ones? Unfortunately not, since there is no way to have JQL in Jira-produced release notes. But yours can, if you include AND resolution = "Fixed" . Or is it worthwhile to let both original + duplicate JIRAs appear in the list? In an ideal changelog, this is not worthwhile. Last, John Russell , not said, but implied here, is that you rely heavily on the "New Feature" issue type to produce documentation, and we should encourage contributors to use this correctly. For example, if I add new testing infrastructure, you would prefer that not be included as a "New Feature", because it's not a new feature of Impala proper, right?
        Hide
        mikesbrown Michael Brown added a comment -

        > IMPALA-4686

        Missed by your filter because its component is infrastructure. I'll follow up to make sure that's correct.

        Changed to Task.

        Show
        mikesbrown Michael Brown added a comment - > IMPALA-4686 Missed by your filter because its component is infrastructure. I'll follow up to make sure that's correct. Changed to Task.
        Hide
        mikesbrown Michael Brown added a comment -
        commit 0493c6261528d9c3d7a52ec6f1aa28b191ec0e6a
        Author: Michael Brown <mikeb@cloudera.com>
        Date:   Thu May 18 08:11:55 2017 -0700
        
            IMPALA-4803: add 2.8 change log
        
            The ASF Jira site provides a mechanism to render so-called "release
            notes" in HTML. The rendered HTML is more like what I'd call a change
            log, so I try to use that phrase here and on the site. The rendered HTML
            shows resolved Jira issues for a given release, grouped by issueType.
            Curating the HTML requires
        
            1. Reordering the groups: the Jira release notes lists Sub-tasks first,
            which are less interesting than New Features, Epics, or Improvements. I
            manually reordered the groups to have the more interesting groups first,
            and less interesting groups later.
        
            2. Adding some preamble HTML to make this a distinct page.
        
            Change-Id: Icaaa6ee1182702b2b5986ff44041fcf40fe7af0e
            Reviewed-on: http://gerrit.cloudera.org:8080/6919
            Reviewed-by: Jim Apple <jbapple-impala@apache.org>
            Tested-by: Michael Brown <mikeb@cloudera.com>
        
        Show
        mikesbrown Michael Brown added a comment - commit 0493c6261528d9c3d7a52ec6f1aa28b191ec0e6a Author: Michael Brown <mikeb@cloudera.com> Date: Thu May 18 08:11:55 2017 -0700 IMPALA-4803: add 2.8 change log The ASF Jira site provides a mechanism to render so-called "release notes" in HTML. The rendered HTML is more like what I'd call a change log, so I try to use that phrase here and on the site. The rendered HTML shows resolved Jira issues for a given release, grouped by issueType. Curating the HTML requires 1. Reordering the groups: the Jira release notes lists Sub-tasks first, which are less interesting than New Features, Epics, or Improvements. I manually reordered the groups to have the more interesting groups first, and less interesting groups later. 2. Adding some preamble HTML to make this a distinct page. Change-Id: Icaaa6ee1182702b2b5986ff44041fcf40fe7af0e Reviewed-on: http://gerrit.cloudera.org:8080/6919 Reviewed-by: Jim Apple <jbapple-impala@apache.org> Tested-by: Michael Brown <mikeb@cloudera.com>
        Hide
        mikesbrown Michael Brown added a comment -

        In my opinion, this can be resolved. We have a clear mechanism for doing this, which is part of the steps a release lead will take. See https://cwiki.apache.org/confluence/display/IMPALA/DRAFT%3A+How+to+Release step 15

        Show
        mikesbrown Michael Brown added a comment - In my opinion, this can be resolved. We have a clear mechanism for doing this, which is part of the steps a release lead will take. See https://cwiki.apache.org/confluence/display/IMPALA/DRAFT%3A+How+to+Release step 15
        Hide
        mikesbrown Michael Brown added a comment -

        Reopening and assigning to John Russell

        Show
        mikesbrown Michael Brown added a comment - Reopening and assigning to John Russell

          People

          • Assignee:
            jrussell John Russell
            Reporter:
            jbapple Jim Apple
          • Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:

              Development