Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13688

Stop bundling HTML source code in javadoc JARs

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.6.4
    • Fix Version/s: 3.0.0-alpha2
    • Component/s: build
    • Labels:
      None

      Description

      We generate source code with line numbers for inclusion in the javadoc JARs. Given that there's github and other online viewers, this doesn't seem so useful these days.

      Disabling the "linkSource" option saves us 40MB for the hadoop-common javadoc jar:

      -rw-r--r-- 1 andrew andrew 98M Oct  5 14:44 hadoop-common-3.0.0-alpha2-SNAPSHOT-javadoc.jar
      -rw-r--r-- 1 andrew andrew 58M Oct  5 15:00 ./hadoop-common-project/hadoop-common/target/hadoop-common-3.0.0-alpha2-SNAPSHOT-javadoc.jar
      

        Activity

        Hide
        andrew.wang Andrew Wang added a comment -

        Straightforward patch. Tested per description.

        Show
        andrew.wang Andrew Wang added a comment - Straightforward patch. Tested per description.
        Hide
        andrew.wang Andrew Wang added a comment -

        The other issue I noticed is that there seems to be tons of extra content in the javadoc jars, like other jars and classfiles. It looks like it's sucking down the entire target directory:

        After unzipping the hadoop-common javadoc jar:

        -> % ls
        allclasses-frame.html    findbugsXml.xml                                       hadoop-maven-plugins-protoc-checksums.json  META-INF               src-html
        allclasses-noframe.html  generated-sources                                     help-doc.html                               org                    stylesheet.css
        antrun                   generated-test-sources                                index-all.html                              overview-frame.html    test
        archive-tmp              hadoop-annotations.jar                                index.html                                  overview-summary.html  test-classes
        classes                  hadoop-common-3.0.0-alpha2-SNAPSHOT                   javadoc-bundle-options                      overview-tree.html     test-dir
        constant-values.html     hadoop-common-3.0.0-alpha2-SNAPSHOT.jar               jdiff.jar                                   package-list           xerces.jar
        deprecated-list.html     hadoop-common-3.0.0-alpha2-SNAPSHOT-sources.jar       log                                         script.js
        docs-src                 hadoop-common-3.0.0-alpha2-SNAPSHOT-tests.jar         maven-archiver                              serialized-form.html
        findbugsExcludeFile.xml  hadoop-common-3.0.0-alpha2-SNAPSHOT-test-sources.jar  maven-shared-archive-resources              site
        andrew@zanarkand [03:21:07 PM] [/tmp/javadoc] 
        -> % find . -name "*.class" | wc -l 
        3455
        andrew@zanarkand [03:21:11 PM] [/tmp/javadoc] 
        -> % find . -name "*.jar" | wc -l
        85
        -> % find . -name "*.jar" -exec wc -c {} \; | awk -e '{SUM+=$1} END {print SUM}' 
        41692556
        

        Since the jarfiles by themselves are another 41MB, there's the potential to really streamline the javadoc jars.

        Show
        andrew.wang Andrew Wang added a comment - The other issue I noticed is that there seems to be tons of extra content in the javadoc jars, like other jars and classfiles. It looks like it's sucking down the entire target directory: After unzipping the hadoop-common javadoc jar: -> % ls allclasses-frame.html findbugsXml.xml hadoop-maven-plugins-protoc-checksums.json META-INF src-html allclasses-noframe.html generated-sources help-doc.html org stylesheet.css antrun generated-test-sources index-all.html overview-frame.html test archive-tmp hadoop-annotations.jar index.html overview-summary.html test-classes classes hadoop-common-3.0.0-alpha2-SNAPSHOT javadoc-bundle-options overview-tree.html test-dir constant-values.html hadoop-common-3.0.0-alpha2-SNAPSHOT.jar jdiff.jar package-list xerces.jar deprecated-list.html hadoop-common-3.0.0-alpha2-SNAPSHOT-sources.jar log script.js docs-src hadoop-common-3.0.0-alpha2-SNAPSHOT-tests.jar maven-archiver serialized-form.html findbugsExcludeFile.xml hadoop-common-3.0.0-alpha2-SNAPSHOT-test-sources.jar maven-shared-archive-resources site andrew@zanarkand [03:21:07 PM] [/tmp/javadoc] -> % find . -name "*.class" | wc -l 3455 andrew@zanarkand [03:21:11 PM] [/tmp/javadoc] -> % find . -name "*.jar" | wc -l 85 -> % find . -name "*.jar" -exec wc -c {} \; | awk -e '{SUM+=$1} END {print SUM}' 41692556 Since the jarfiles by themselves are another 41MB, there's the potential to really streamline the javadoc jars.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 13s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        0 mvndep 0m 15s Maven dependency ordering for branch
        +1 mvninstall 7m 1s trunk passed
        +1 compile 6m 54s trunk passed
        +1 mvnsite 9m 18s trunk passed
        +1 mvneclipse 0m 59s trunk passed
        +1 javadoc 4m 37s trunk passed
        0 mvndep 0m 14s Maven dependency ordering for patch
        +1 mvninstall 7m 14s the patch passed
        +1 compile 7m 38s the patch passed
        +1 javac 7m 38s the patch passed
        +1 mvnsite 10m 40s the patch passed
        +1 mvneclipse 1m 4s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 4s The patch has no ill-formed XML file.
        +1 javadoc 5m 0s the patch passed
        -1 unit 3m 47s root in the patch failed.
        -1 asflicense 0m 22s The patch generated 2 ASF License warnings.
        66m 5s



        Reason Tests
        Failed junit tests hadoop.security.authentication.util.TestZKSignerSecretProvider



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue HADOOP-13688
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12831823/HADOOP-13688.001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
        uname Linux 4b43ffcdda22 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / e68c7b9
        Default Java 1.8.0_101
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10681/artifact/patchprocess/patch-unit-root.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10681/testReport/
        asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/10681/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-project-dist hadoop-hdfs-project/hadoop-hdfs-httpfs . U: .
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10681/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 15s Maven dependency ordering for branch +1 mvninstall 7m 1s trunk passed +1 compile 6m 54s trunk passed +1 mvnsite 9m 18s trunk passed +1 mvneclipse 0m 59s trunk passed +1 javadoc 4m 37s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 7m 14s the patch passed +1 compile 7m 38s the patch passed +1 javac 7m 38s the patch passed +1 mvnsite 10m 40s the patch passed +1 mvneclipse 1m 4s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 4s The patch has no ill-formed XML file. +1 javadoc 5m 0s the patch passed -1 unit 3m 47s root in the patch failed. -1 asflicense 0m 22s The patch generated 2 ASF License warnings. 66m 5s Reason Tests Failed junit tests hadoop.security.authentication.util.TestZKSignerSecretProvider Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HADOOP-13688 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12831823/HADOOP-13688.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux 4b43ffcdda22 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e68c7b9 Default Java 1.8.0_101 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10681/artifact/patchprocess/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10681/testReport/ asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/10681/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-project-dist hadoop-hdfs-project/hadoop-hdfs-httpfs . U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10681/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        andrew.wang Andrew Wang added a comment -

        test-patch issues look unrelated.

        Show
        andrew.wang Andrew Wang added a comment - test-patch issues look unrelated.
        Hide
        stevel@apache.org Steve Loughran added a comment -

        +1

        Show
        stevel@apache.org Steve Loughran added a comment - +1
        Hide
        ajisakaa Akira Ajisaka added a comment -

        +1

        Show
        ajisakaa Akira Ajisaka added a comment - +1
        Hide
        andrew.wang Andrew Wang added a comment -

        Thanks for reviewing Steve and Akira, I've pushed this to trunk.

        Show
        andrew.wang Andrew Wang added a comment - Thanks for reviewing Steve and Akira, I've pushed this to trunk.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10559 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10559/)
        HADOOP-13688. Stop bundling HTML source code in javadoc JARs. (wang: rev 72a2ae6452e615c66d10829da38737896814e02b)

        • (edit) hadoop-project-dist/pom.xml
        • (edit) pom.xml
        • (edit) hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10559 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10559/ ) HADOOP-13688 . Stop bundling HTML source code in javadoc JARs. (wang: rev 72a2ae6452e615c66d10829da38737896814e02b) (edit) hadoop-project-dist/pom.xml (edit) pom.xml (edit) hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml
        Hide
        elek Elek, Marton added a comment -

        Same problem has been questioned during the 2.8.0 release vote.

        I suggest to cherry-pick the patch to the branch-2/branch-2.8/branch-2.7. Hopefully we will get releases from those lines. I wouldn't call it incompatible change.

        Show
        elek Elek, Marton added a comment - Same problem has been questioned during the 2.8.0 release vote. I suggest to cherry-pick the patch to the branch-2/branch-2.8/branch-2.7. Hopefully we will get releases from those lines. I wouldn't call it incompatible change.
        Hide
        andrew.wang Andrew Wang added a comment -

        I think the 2.8.0 release vote issue might be a bit different, since this removes the content from the javadoc jars, and the 2.8.0 tarball had these in expanded form. Also, 2.7.2 seemed to be okay and didn't have this change, so it's unclear why 2.8.0 blew up.

        Show
        andrew.wang Andrew Wang added a comment - I think the 2.8.0 release vote issue might be a bit different, since this removes the content from the javadoc jars, and the 2.8.0 tarball had these in expanded form. Also, 2.7.2 seemed to be okay and didn't have this change, so it's unclear why 2.8.0 blew up.

          People

          • Assignee:
            andrew.wang Andrew Wang
            Reporter:
            andrew.wang Andrew Wang
          • Votes:
            1 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development