Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13688

Stop bundling HTML source code in javadoc JARs

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.6.4
    • Fix Version/s: 2.9.0, 3.0.0-alpha2, 2.8.2
    • Component/s: build
    • Labels:
      None

      Description

      We generate source code with line numbers for inclusion in the javadoc JARs. Given that there's github and other online viewers, this doesn't seem so useful these days.

      Disabling the "linkSource" option saves us 40MB for the hadoop-common javadoc jar:

      -rw-r--r-- 1 andrew andrew 98M Oct  5 14:44 hadoop-common-3.0.0-alpha2-SNAPSHOT-javadoc.jar
      -rw-r--r-- 1 andrew andrew 58M Oct  5 15:00 ./hadoop-common-project/hadoop-common/target/hadoop-common-3.0.0-alpha2-SNAPSHOT-javadoc.jar
      

        Issue Links

          Activity

          Hide
          andrew.wang Andrew Wang added a comment -

          Straightforward patch. Tested per description.

          Show
          andrew.wang Andrew Wang added a comment - Straightforward patch. Tested per description.
          Hide
          andrew.wang Andrew Wang added a comment -

          The other issue I noticed is that there seems to be tons of extra content in the javadoc jars, like other jars and classfiles. It looks like it's sucking down the entire target directory:

          After unzipping the hadoop-common javadoc jar:

          -> % ls
          allclasses-frame.html    findbugsXml.xml                                       hadoop-maven-plugins-protoc-checksums.json  META-INF               src-html
          allclasses-noframe.html  generated-sources                                     help-doc.html                               org                    stylesheet.css
          antrun                   generated-test-sources                                index-all.html                              overview-frame.html    test
          archive-tmp              hadoop-annotations.jar                                index.html                                  overview-summary.html  test-classes
          classes                  hadoop-common-3.0.0-alpha2-SNAPSHOT                   javadoc-bundle-options                      overview-tree.html     test-dir
          constant-values.html     hadoop-common-3.0.0-alpha2-SNAPSHOT.jar               jdiff.jar                                   package-list           xerces.jar
          deprecated-list.html     hadoop-common-3.0.0-alpha2-SNAPSHOT-sources.jar       log                                         script.js
          docs-src                 hadoop-common-3.0.0-alpha2-SNAPSHOT-tests.jar         maven-archiver                              serialized-form.html
          findbugsExcludeFile.xml  hadoop-common-3.0.0-alpha2-SNAPSHOT-test-sources.jar  maven-shared-archive-resources              site
          andrew@zanarkand [03:21:07 PM] [/tmp/javadoc] 
          -> % find . -name "*.class" | wc -l 
          3455
          andrew@zanarkand [03:21:11 PM] [/tmp/javadoc] 
          -> % find . -name "*.jar" | wc -l
          85
          -> % find . -name "*.jar" -exec wc -c {} \; | awk -e '{SUM+=$1} END {print SUM}' 
          41692556
          

          Since the jarfiles by themselves are another 41MB, there's the potential to really streamline the javadoc jars.

          Show
          andrew.wang Andrew Wang added a comment - The other issue I noticed is that there seems to be tons of extra content in the javadoc jars, like other jars and classfiles. It looks like it's sucking down the entire target directory: After unzipping the hadoop-common javadoc jar: -> % ls allclasses-frame.html findbugsXml.xml hadoop-maven-plugins-protoc-checksums.json META-INF src-html allclasses-noframe.html generated-sources help-doc.html org stylesheet.css antrun generated-test-sources index-all.html overview-frame.html test archive-tmp hadoop-annotations.jar index.html overview-summary.html test-classes classes hadoop-common-3.0.0-alpha2-SNAPSHOT javadoc-bundle-options overview-tree.html test-dir constant-values.html hadoop-common-3.0.0-alpha2-SNAPSHOT.jar jdiff.jar package-list xerces.jar deprecated-list.html hadoop-common-3.0.0-alpha2-SNAPSHOT-sources.jar log script.js docs-src hadoop-common-3.0.0-alpha2-SNAPSHOT-tests.jar maven-archiver serialized-form.html findbugsExcludeFile.xml hadoop-common-3.0.0-alpha2-SNAPSHOT-test-sources.jar maven-shared-archive-resources site andrew@zanarkand [03:21:07 PM] [/tmp/javadoc] -> % find . -name "*.class" | wc -l 3455 andrew@zanarkand [03:21:11 PM] [/tmp/javadoc] -> % find . -name "*.jar" | wc -l 85 -> % find . -name "*.jar" -exec wc -c {} \; | awk -e '{SUM+=$1} END {print SUM}' 41692556 Since the jarfiles by themselves are another 41MB, there's the potential to really streamline the javadoc jars.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 0m 15s Maven dependency ordering for branch
          +1 mvninstall 7m 1s trunk passed
          +1 compile 6m 54s trunk passed
          +1 mvnsite 9m 18s trunk passed
          +1 mvneclipse 0m 59s trunk passed
          +1 javadoc 4m 37s trunk passed
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 7m 14s the patch passed
          +1 compile 7m 38s the patch passed
          +1 javac 7m 38s the patch passed
          +1 mvnsite 10m 40s the patch passed
          +1 mvneclipse 1m 4s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 4s The patch has no ill-formed XML file.
          +1 javadoc 5m 0s the patch passed
          -1 unit 3m 47s root in the patch failed.
          -1 asflicense 0m 22s The patch generated 2 ASF License warnings.
          66m 5s



          Reason Tests
          Failed junit tests hadoop.security.authentication.util.TestZKSignerSecretProvider



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HADOOP-13688
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12831823/HADOOP-13688.001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
          uname Linux 4b43ffcdda22 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / e68c7b9
          Default Java 1.8.0_101
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10681/artifact/patchprocess/patch-unit-root.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10681/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/10681/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-project-dist hadoop-hdfs-project/hadoop-hdfs-httpfs . U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10681/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 15s Maven dependency ordering for branch +1 mvninstall 7m 1s trunk passed +1 compile 6m 54s trunk passed +1 mvnsite 9m 18s trunk passed +1 mvneclipse 0m 59s trunk passed +1 javadoc 4m 37s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 7m 14s the patch passed +1 compile 7m 38s the patch passed +1 javac 7m 38s the patch passed +1 mvnsite 10m 40s the patch passed +1 mvneclipse 1m 4s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 4s The patch has no ill-formed XML file. +1 javadoc 5m 0s the patch passed -1 unit 3m 47s root in the patch failed. -1 asflicense 0m 22s The patch generated 2 ASF License warnings. 66m 5s Reason Tests Failed junit tests hadoop.security.authentication.util.TestZKSignerSecretProvider Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HADOOP-13688 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12831823/HADOOP-13688.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux 4b43ffcdda22 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e68c7b9 Default Java 1.8.0_101 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10681/artifact/patchprocess/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10681/testReport/ asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/10681/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-project-dist hadoop-hdfs-project/hadoop-hdfs-httpfs . U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10681/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          andrew.wang Andrew Wang added a comment -

          test-patch issues look unrelated.

          Show
          andrew.wang Andrew Wang added a comment - test-patch issues look unrelated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          +1

          Show
          stevel@apache.org Steve Loughran added a comment - +1
          Hide
          ajisakaa Akira Ajisaka added a comment -

          +1

          Show
          ajisakaa Akira Ajisaka added a comment - +1
          Hide
          andrew.wang Andrew Wang added a comment -

          Thanks for reviewing Steve and Akira, I've pushed this to trunk.

          Show
          andrew.wang Andrew Wang added a comment - Thanks for reviewing Steve and Akira, I've pushed this to trunk.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10559 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10559/)
          HADOOP-13688. Stop bundling HTML source code in javadoc JARs. (wang: rev 72a2ae6452e615c66d10829da38737896814e02b)

          • (edit) hadoop-project-dist/pom.xml
          • (edit) pom.xml
          • (edit) hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10559 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10559/ ) HADOOP-13688 . Stop bundling HTML source code in javadoc JARs. (wang: rev 72a2ae6452e615c66d10829da38737896814e02b) (edit) hadoop-project-dist/pom.xml (edit) pom.xml (edit) hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml
          Hide
          elek Elek, Marton added a comment -

          Same problem has been questioned during the 2.8.0 release vote.

          I suggest to cherry-pick the patch to the branch-2/branch-2.8/branch-2.7. Hopefully we will get releases from those lines. I wouldn't call it incompatible change.

          Show
          elek Elek, Marton added a comment - Same problem has been questioned during the 2.8.0 release vote. I suggest to cherry-pick the patch to the branch-2/branch-2.8/branch-2.7. Hopefully we will get releases from those lines. I wouldn't call it incompatible change.
          Hide
          andrew.wang Andrew Wang added a comment -

          I think the 2.8.0 release vote issue might be a bit different, since this removes the content from the javadoc jars, and the 2.8.0 tarball had these in expanded form. Also, 2.7.2 seemed to be okay and didn't have this change, so it's unclear why 2.8.0 blew up.

          Show
          andrew.wang Andrew Wang added a comment - I think the 2.8.0 release vote issue might be a bit different, since this removes the content from the javadoc jars, and the 2.8.0 tarball had these in expanded form. Also, 2.7.2 seemed to be okay and didn't have this change, so it's unclear why 2.8.0 blew up.
          Hide
          djp Junping Du added a comment -

          I think there are related because our tar ball include these javadoc jars (with new create release script), so remove the link source will significantly reduce the tar ball in branch-2 releases. I verified that the size of tar ball is reduced from 410 M to 242M for the same branch-2.8.
          I plan to merge the patch to branch-2/branch-2.8 if no one objects.

          Show
          djp Junping Du added a comment - I think there are related because our tar ball include these javadoc jars (with new create release script), so remove the link source will significantly reduce the tar ball in branch-2 releases. I verified that the size of tar ball is reduced from 410 M to 242M for the same branch-2.8. I plan to merge the patch to branch-2/branch-2.8 if no one objects.
          Hide
          djp Junping Du added a comment -

          I have merged the patch to branch-2 and branch-2.8.

          Show
          djp Junping Du added a comment - I have merged the patch to branch-2 and branch-2.8.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          2.8.1 became a security release. Moving fix-version to 2.8.2 after the fact.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - 2.8.1 became a security release. Moving fix-version to 2.8.2 after the fact.

            People

            • Assignee:
              andrew.wang Andrew Wang
              Reporter:
              andrew.wang Andrew Wang
            • Votes:
              1 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development