Giraph
  1. Giraph
  2. GIRAPH-159

Case insensitive file/directory name matching will produce errors on M/R jar unpack.

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.0
    • Fix Version/s: None
    • Component/s: build
    • Labels:
      None
    • Environment:

      OSX 10.6.8

      Description

      This only seems to affect platforms where there can be a file/directory naming conflicts
      from case insensitive matches.

      I was able to reproduce running the pseudo-distributed unit tests within OSX.

      This has affected other projects:
      https://issues.apache.org/jira/browse/MAHOUT-780

      I've been able to reproduce this on my local OSX install with the following error:
      https://groups.google.com/a/cloudera.org/group/cdh-user/browse_thread/thread/a201218000e956d3/cc6eca3ef9f80ff8

      Since LICENSE.txt contains the same content as the file LICENSE, I propose we exclude any LICENSE matches found in the unpacked dependency jars
      when the maven assembly phase hits 'jar-with-dependencies'.

      I have a patch which moves the 'jar-with-dependencies' descriptor to an external compile.xml file which has the proper excludes. This might also
      come in handy down the road should any additional tweaks be needed to the compile phase.

      1. GIRAPH-159.patch
        2 kB
        Brian Femiano
      2. compile.xml
        2 kB
        Brian Femiano

        Issue Links

          Activity

          Avery Ching made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Assignee Brian Femiano [ bfem ]
          Resolution Fixed [ 1 ]
          Hide
          Avery Ching added a comment -

          Hudson agrees, resolving.

          Show
          Avery Ching added a comment - Hudson agrees, resolving.
          Hide
          Hudson added a comment -

          Integrated in Giraph-trunk-Commit #94 (See https://builds.apache.org/job/Giraph-trunk-Commit/94/)
          GIRAPH-159: Case insensitive file/directory name matching will produce
          errors on M/R jar unpack (bfem via aching). (Revision 1305073)

          Result = SUCCESS
          aching : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1305073
          Files :

          • /incubator/giraph/trunk/CHANGELOG
          • /incubator/giraph/trunk/pom.xml
          • /incubator/giraph/trunk/src/main/assembly/compile.xml
          Show
          Hudson added a comment - Integrated in Giraph-trunk-Commit #94 (See https://builds.apache.org/job/Giraph-trunk-Commit/94/ ) GIRAPH-159 : Case insensitive file/directory name matching will produce errors on M/R jar unpack (bfem via aching). (Revision 1305073) Result = SUCCESS aching : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1305073 Files : /incubator/giraph/trunk/CHANGELOG /incubator/giraph/trunk/pom.xml /incubator/giraph/trunk/src/main/assembly/compile.xml
          Hide
          Avery Ching added a comment -

          +1. I left out your GiraphRunner changes since they were fixed by earlier JIRAs, but verified both the problem and the solution you proposed. Looks good! Thanks for the fix! Committing.

          Show
          Avery Ching added a comment - +1. I left out your GiraphRunner changes since they were fixed by earlier JIRAs, but verified both the problem and the solution you proposed. Looks good! Thanks for the fix! Committing.
          Hide
          Brian Femiano added a comment -

          giraph-0.2-SNAPSHOT-jar-with-dependencies.jar goes from being ~5MB in size to ~34MB once all the
          hbase dependencies are unpacked.

          mvn verify takes about 1.5 hours to run with the pseudo-distributed unit tests.

          Show
          Brian Femiano added a comment - giraph-0.2-SNAPSHOT-jar-with-dependencies.jar goes from being ~5MB in size to ~34MB once all the hbase dependencies are unpacked. mvn verify takes about 1.5 hours to run with the pseudo-distributed unit tests.
          Brian Femiano made changes -
          Link This issue relates to GIRAPH-153 [ GIRAPH-153 ]
          Hide
          Brian Femiano added a comment -

          I figured out what's causing it.

          It's a result of adding my hbase dependency to the pom.xml

          <dependency>
          <groupId>org.apache.hbase</groupId>
          <artifactId>hbase</artifactId>
          <version>0.92.1</version>
          </dependency>

          Compile the jar and you should see a new 'license' directory.

          jar tvf giraph-0.2-SNAPSHOT-jar-with-dependencies.jar | grep -i 'license'

          1358 Mon Mar 16 00:31:16 EDT 2009 META-INF/LICENSE.txt
          11358 Mon Nov 19 00:16:46 EST 2007 META-INF/LICENSE
          1596 Mon Dec 20 14:42:08 EST 2010 LICENSE
          11560 Tue Aug 23 13:48:08 EDT 2011 META-INF/maven/org.xerial.snappy/snappy-java/LICENSE
          0 Mon Feb 07 21:38:56 EST 2011 META-INF/license/
          1592 Mon Feb 07 21:38:38 EST 2011 META-INF/license/LICENSE.base64.txt
          10174 Mon Feb 07 21:38:38 EST 2011 META-INF/license/LICENSE.commons-logging.txt
          10174 Mon Feb 07 21:38:38 EST 2011 META-INF/license/LICENSE.felix.txt
          26441 Mon Feb 07 21:38:38 EST 2011 META-INF/license/LICENSE.jboss-logging.txt
          1592 Mon Feb 07 21:38:38 EST 2011 META-INF/license/LICENSE.jsr166y.txt
          1465 Mon Feb 07 21:38:38 EST 2011 META-INF/license/LICENSE.jzlib.txt
          10174 Mon Feb 07 21:38:38 EST 2011 META-INF/license/LICENSE.log4j.txt
          1732 Mon Feb 07 21:38:38 EST 2011 META-INF/license/LICENSE.protobuf.txt
          1203 Mon Feb 07 21:38:38 EST 2011 META-INF/license/LICENSE.slf4j.txt
          11358 Fri Jan 21 17:06:30 EST 2011 LICENSE.txt
          1062 Tue Oct 25 10:29:02 EDT 2011 META-INF/jruby.home/lib/ruby/gems/1.8/gems/rake-0.8.7/MIT-LICENSE

          Show
          Brian Femiano added a comment - I figured out what's causing it. It's a result of adding my hbase dependency to the pom.xml <dependency> <groupId>org.apache.hbase</groupId> <artifactId>hbase</artifactId> <version>0.92.1</version> </dependency> Compile the jar and you should see a new 'license' directory. jar tvf giraph-0.2-SNAPSHOT-jar-with-dependencies.jar | grep -i 'license' 1358 Mon Mar 16 00:31:16 EDT 2009 META-INF/LICENSE.txt 11358 Mon Nov 19 00:16:46 EST 2007 META-INF/LICENSE 1596 Mon Dec 20 14:42:08 EST 2010 LICENSE 11560 Tue Aug 23 13:48:08 EDT 2011 META-INF/maven/org.xerial.snappy/snappy-java/LICENSE 0 Mon Feb 07 21:38:56 EST 2011 META-INF/license/ 1592 Mon Feb 07 21:38:38 EST 2011 META-INF/license/LICENSE.base64.txt 10174 Mon Feb 07 21:38:38 EST 2011 META-INF/license/LICENSE.commons-logging.txt 10174 Mon Feb 07 21:38:38 EST 2011 META-INF/license/LICENSE.felix.txt 26441 Mon Feb 07 21:38:38 EST 2011 META-INF/license/LICENSE.jboss-logging.txt 1592 Mon Feb 07 21:38:38 EST 2011 META-INF/license/LICENSE.jsr166y.txt 1465 Mon Feb 07 21:38:38 EST 2011 META-INF/license/LICENSE.jzlib.txt 10174 Mon Feb 07 21:38:38 EST 2011 META-INF/license/LICENSE.log4j.txt 1732 Mon Feb 07 21:38:38 EST 2011 META-INF/license/LICENSE.protobuf.txt 1203 Mon Feb 07 21:38:38 EST 2011 META-INF/license/LICENSE.slf4j.txt 11358 Fri Jan 21 17:06:30 EST 2011 LICENSE.txt 1062 Tue Oct 25 10:29:02 EDT 2011 META-INF/jruby.home/lib/ruby/gems/1.8/gems/rake-0.8.7/MIT-LICENSE
          Hide
          Avery Ching added a comment -

          Brian, I am able to run the distributed unittests just fine on my single node hadoop install.

          jar tvf giraph-0.2-SNAPSHOT-jar-with-dependencies.jar | grep -i license
          11358 Mon Mar 16 00:31:16 PDT 2009 META-INF/LICENSE.txt
          580 Mon Jun 07 20:50:48 PDT 2010 META-INF/LICENSE
          11358 Fri Jan 21 17:06:30 PST 2011 LICENSE.txt

          I don't have a license directory. What does your jar say?

          Show
          Avery Ching added a comment - Brian, I am able to run the distributed unittests just fine on my single node hadoop install. jar tvf giraph-0.2-SNAPSHOT-jar-with-dependencies.jar | grep -i license 11358 Mon Mar 16 00:31:16 PDT 2009 META-INF/LICENSE.txt 580 Mon Jun 07 20:50:48 PDT 2010 META-INF/LICENSE 11358 Fri Jan 21 17:06:30 PST 2011 LICENSE.txt I don't have a license directory. What does your jar say?
          Brian Femiano made changes -
          Priority Minor [ 4 ] Major [ 3 ]
          Hide
          Brian Femiano added a comment -

          Any luck recreating this? I have to keep this change local until it's committed.

          Show
          Brian Femiano added a comment - Any luck recreating this? I have to keep this change local until it's committed.
          Eugene Koontz made changes -
          Link This issue is related to GIRAPH-165 [ GIRAPH-165 ]
          Brian Femiano made changes -
          Link This issue relates to GIRAPH-164 [ GIRAPH-164 ]
          Hide
          Brian Femiano added a comment -

          Steps to run.

          1) Follow http://ssc.io/running-giraphs-unit-tests-in-pseudo-distributed-mode/ for running the single node unit tests on an OSX.
          2) Before the M/R jobs can even begin, the JobTracker will throw an IOException indicating
          it cannot mkdirs on 'license'.
          3) The operating system does not distinguish between the directory 'license' and the file 'LICENSE'.

          Show
          Brian Femiano added a comment - Steps to run. 1) Follow http://ssc.io/running-giraphs-unit-tests-in-pseudo-distributed-mode/ for running the single node unit tests on an OSX. 2) Before the M/R jobs can even begin, the JobTracker will throw an IOException indicating it cannot mkdirs on 'license'. 3) The operating system does not distinguish between the directory 'license' and the file 'LICENSE'.
          Hide
          Avery Ching added a comment -

          Brian, can you show me how to recreate this issue on OSX?

          Show
          Avery Ching added a comment - Brian, can you show me how to recreate this issue on OSX?
          Brian Femiano made changes -
          Attachment GIRAPH-159.patch [ 12518972 ]
          Attachment compile.xml [ 12518973 ]
          Brian Femiano made changes -
          Attachment GIRAPH-159.patch [ 12518953 ]
          Hide
          Brian Femiano added a comment -

          To get mvn verify to pass the checkstyle requirements, I had to wrap 5 lines in GiraphRunner. I also
          had to remove the private method scoped 'conf' instance to a chained call.

          Show
          Brian Femiano added a comment - To get mvn verify to pass the checkstyle requirements, I had to wrap 5 lines in GiraphRunner. I also had to remove the private method scoped 'conf' instance to a chained call.
          Brian Femiano made changes -
          Attachment GIRAPH-159.patch [ 12518953 ]
          Brian Femiano made changes -
          Field Original Value New Value
          Priority Major [ 3 ] Minor [ 4 ]
          Brian Femiano created issue -

            People

            • Assignee:
              Brian Femiano
              Reporter:
              Brian Femiano
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development