Hadoop Common
  1. Hadoop Common
  2. HADOOP-8064

Remove unnecessary dependency on w3c.org in document processing

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0, 0.23.1
    • Fix Version/s: 0.23.2
    • Component/s: build
    • Labels:
      None

      Description

      Our builds fail occasionally due to validation failures caused by unavailability of w3.org. In fact, w3.org has been fed up with this kind of requests, blacklisting/throttling have been in place for some time.

      The ones I've seen are coming from maven pdf plugin v.1.1 used in hadoop-tools/hadoop-distcp. The input validation has been made configurable in MPDF-39, which requires DOXIA-392 (fixed since doxia-1.1.3). Unfortunately, the maven pdf plugin 1.2 has been cooking for long time and not been released yet. May be we could force it to use a later version of doxia?

      1. hadoop-8064.patch.txt
        0.9 kB
        Kihwal Lee
      2. hadoop-8064.patch.txt
        0.7 kB
        Kihwal Lee

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1011 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1011/)
        HADOOP-8064. Remove unnecessary dependency on w3c.org in document processing (Khiwal Lee via bobby) (Revision 1297274)

        Result = FAILURE
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297274
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-tools/hadoop-distcp/pom.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1011 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1011/ ) HADOOP-8064 . Remove unnecessary dependency on w3c.org in document processing (Khiwal Lee via bobby) (Revision 1297274) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297274 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-tools/hadoop-distcp/pom.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Build #217 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/217/)
        svn merge -c 1297274 from trunk to branch-0.23 FIXES HADOOP-8064. Remove unnecessary dependency on w3c.org in document processing (Khiwal Lee via bobby) (Revision 1297275)

        Result = FAILURE
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297275
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-distcp/pom.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #217 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/217/ ) svn merge -c 1297274 from trunk to branch-0.23 FIXES HADOOP-8064 . Remove unnecessary dependency on w3c.org in document processing (Khiwal Lee via bobby) (Revision 1297275) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297275 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-distcp/pom.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #976 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/976/)
        HADOOP-8064. Remove unnecessary dependency on w3c.org in document processing (Khiwal Lee via bobby) (Revision 1297274)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297274
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-tools/hadoop-distcp/pom.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #976 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/976/ ) HADOOP-8064 . Remove unnecessary dependency on w3c.org in document processing (Khiwal Lee via bobby) (Revision 1297274) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297274 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-tools/hadoop-distcp/pom.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #189 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/189/)
        svn merge -c 1297274 from trunk to branch-0.23 FIXES HADOOP-8064. Remove unnecessary dependency on w3c.org in document processing (Khiwal Lee via bobby) (Revision 1297275)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297275
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-distcp/pom.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #189 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/189/ ) svn merge -c 1297274 from trunk to branch-0.23 FIXES HADOOP-8064 . Remove unnecessary dependency on w3c.org in document processing (Khiwal Lee via bobby) (Revision 1297275) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297275 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-distcp/pom.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #1844 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1844/)
        HADOOP-8064. Remove unnecessary dependency on w3c.org in document processing (Khiwal Lee via bobby) (Revision 1297274)

        Result = FAILURE
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297274
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-tools/hadoop-distcp/pom.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1844 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1844/ ) HADOOP-8064 . Remove unnecessary dependency on w3c.org in document processing (Khiwal Lee via bobby) (Revision 1297274) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297274 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-tools/hadoop-distcp/pom.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Commit #641 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/641/)
        svn merge -c 1297274 from trunk to branch-0.23 FIXES HADOOP-8064. Remove unnecessary dependency on w3c.org in document processing (Khiwal Lee via bobby) (Revision 1297275)

        Result = FAILURE
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297275
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-distcp/pom.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #641 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/641/ ) svn merge -c 1297274 from trunk to branch-0.23 FIXES HADOOP-8064 . Remove unnecessary dependency on w3c.org in document processing (Khiwal Lee via bobby) (Revision 1297275) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297275 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-distcp/pom.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #1911 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1911/)
        HADOOP-8064. Remove unnecessary dependency on w3c.org in document processing (Khiwal Lee via bobby) (Revision 1297274)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297274
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-tools/hadoop-distcp/pom.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1911 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1911/ ) HADOOP-8064 . Remove unnecessary dependency on w3c.org in document processing (Khiwal Lee via bobby) (Revision 1297274) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297274 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-tools/hadoop-distcp/pom.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-0.23-Commit #641 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/641/)
        svn merge -c 1297274 from trunk to branch-0.23 FIXES HADOOP-8064. Remove unnecessary dependency on w3c.org in document processing (Khiwal Lee via bobby) (Revision 1297275)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297275
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-distcp/pom.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #641 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/641/ ) svn merge -c 1297274 from trunk to branch-0.23 FIXES HADOOP-8064 . Remove unnecessary dependency on w3c.org in document processing (Khiwal Lee via bobby) (Revision 1297275) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297275 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-distcp/pom.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Commit #631 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/631/)
        svn merge -c 1297274 from trunk to branch-0.23 FIXES HADOOP-8064. Remove unnecessary dependency on w3c.org in document processing (Khiwal Lee via bobby) (Revision 1297275)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297275
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-distcp/pom.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #631 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/631/ ) svn merge -c 1297274 from trunk to branch-0.23 FIXES HADOOP-8064 . Remove unnecessary dependency on w3c.org in document processing (Khiwal Lee via bobby) (Revision 1297275) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297275 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-distcp/pom.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #1837 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1837/)
        HADOOP-8064. Remove unnecessary dependency on w3c.org in document processing (Khiwal Lee via bobby) (Revision 1297274)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297274
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-tools/hadoop-distcp/pom.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1837 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1837/ ) HADOOP-8064 . Remove unnecessary dependency on w3c.org in document processing (Khiwal Lee via bobby) (Revision 1297274) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297274 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-tools/hadoop-distcp/pom.xml
        Hide
        Robert Joseph Evans added a comment -

        Thanks Kihwal I just put this into trunk, branch-0.23 and branch-0.23.2

        Show
        Robert Joseph Evans added a comment - Thanks Kihwal I just put this into trunk, branch-0.23 and branch-0.23.2
        Hide
        Robert Joseph Evans added a comment -

        The patch looks good to me. I manually compiled and ran packaging. +1

        Show
        Robert Joseph Evans added a comment - The patch looks good to me. I manually compiled and ran packaging. +1
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12517137/hadoop-8064.patch.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/675//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12517137/hadoop-8064.patch.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/675//console This message is automatically generated.
        Hide
        Kihwal Lee added a comment -

        The new patch disables the pdf generation in DistCp.

        Show
        Kihwal Lee added a comment - The new patch disables the pdf generation in DistCp.
        Hide
        Kihwal Lee added a comment -

        Thanks Alejandro! since DistCp is the only thing that uses maven-dpf-plugin, it will be better to disable the pdf generation temorarilly until we have a better way or v.1.2 is releaed.

        Show
        Kihwal Lee added a comment - Thanks Alejandro! since DistCp is the only thing that uses maven-dpf-plugin, it will be better to disable the pdf generation temorarilly until we have a better way or v.1.2 is releaed.
        Hide
        Alejandro Abdelnur added a comment -

        Is there any other module generating PDFs? Wouldn't make more sense to remove the PDF generation for now?

        Show
        Alejandro Abdelnur added a comment - Is there any other module generating PDFs? Wouldn't make more sense to remove the PDF generation for now?
        Hide
        Kihwal Lee added a comment -

        The precommit tried to apply the patch from inside hadoop-common-project, so it didn't apply. smart-apply-patch.sh does not catch this case since the patch contains none of the three subproject directory names. The patch actually applies fine.

        Show
        Kihwal Lee added a comment - The precommit tried to apply the patch from inside hadoop-common-project, so it didn't apply. smart-apply-patch.sh does not catch this case since the patch contains none of the three subproject directory names. The patch actually applies fine.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12516723/hadoop-8064.patch.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/652//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12516723/hadoop-8064.patch.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/652//console This message is automatically generated.
        Hide
        Kihwal Lee added a comment -

        The attached patch specifies maven-pdf-plugin version to be 1.2-SNAPSHOT.

        I've verified that the pdf is generates for hadoop-distcp correctly while w3c.org is no longer contacted. It also seems all docs build fine.

        Since this is not part of code generation, the risk is low.

        Show
        Kihwal Lee added a comment - The attached patch specifies maven-pdf-plugin version to be 1.2-SNAPSHOT. I've verified that the pdf is generates for hadoop-distcp correctly while w3c.org is no longer contacted. It also seems all docs build fine. Since this is not part of code generation, the risk is low.

          People

          • Assignee:
            Kihwal Lee
            Reporter:
            Kihwal Lee
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development