Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1628

HarFileSystem shows incorrect replication numbers and permissions

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: harchive
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      In the har dir, the replication # of part-0 is 3.

      -bash-3.1$ hadoop fs -ls  ${DIR}.har
      Found 3 items
      -rw-------   5 tsz users       1141 2010-02-10 18:34 /user/tsz/t20.har/_index
      -rw-------   5 tsz users         24 2010-02-10 18:34 /user/tsz/t20.har/_masterindex
      -rw-------   3 tsz users      15052 2010-02-10 18:34 /user/tsz/t20.har/part-0
      

      but the replication # of the individual har:// files is shown as 5.

      -bash-3.1$ hadoop fs -lsr  ${HAR_FULL}/
      drw-------   - tsz users          0 2010-02-10 18:34 /user/tsz/t20.har/t20
      -rw-------   5 tsz users        723 2010-02-10 18:34 /user/tsz/t20.har/t20/text-00000000
      -rw-------   5 tsz users        779 2010-02-10 18:34 /user/tsz/t20.har/t20/text-00000001
      -rw-------   5 tsz users        818 2010-02-10 18:34 /user/tsz/t20.har/t20/text-00000002
      ...
      

      The permission also has similar problem. Clearly, the permission of t20.har/t20 shown above is incorrect.

      1. m1628_20100329.patch
        6 kB
        Tsz Wo Nicholas Sze
      2. m1628_20100329b.patch
        6 kB
        Tsz Wo Nicholas Sze
      3. m1628_20100330.patch
        9 kB
        Tsz Wo Nicholas Sze
      4. m1628_20100331.patch
        9 kB
        Tsz Wo Nicholas Sze

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #276 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/276/)
          . HarFileSystem shows incorrect replication numbers and permissions (tsz via mahadev)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #276 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/276/ ) . HarFileSystem shows incorrect replication numbers and permissions (tsz via mahadev)
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #296 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/296/)
          . HarFileSystem shows incorrect replication numbers and permissions (tsz via mahadev)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #296 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/296/ ) . HarFileSystem shows incorrect replication numbers and permissions (tsz via mahadev)
          Hide
          Mahadev konar added a comment -

          I just committed this. thanks nicholas!

          Show
          Mahadev konar added a comment - I just committed this. thanks nicholas!
          Hide
          Mahadev konar added a comment -

          +1 the patch looks good!

          Show
          Mahadev konar added a comment - +1 the patch looks good!
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > slightly changed the codes.
          More details: use HarStatus.isDir instead of comparing "none".

          Show
          Tsz Wo Nicholas Sze added a comment - > slightly changed the codes. More details: use HarStatus.isDir instead of comparing "none".
          Hide
          Tsz Wo Nicholas Sze added a comment -

          m1628_20100331.patch: slightly changed the codes.

          Show
          Tsz Wo Nicholas Sze added a comment - m1628_20100331.patch: slightly changed the codes.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12440285/m1628_20100330.patch
          against trunk revision 928104.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 5 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/73/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/73/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/73/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/73/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12440285/m1628_20100330.patch against trunk revision 928104. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 5 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/73/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/73/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/73/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/73/console This message is automatically generated.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          m1628_20100330.patch: updated the listing test in testMRConf.xml

          Show
          Tsz Wo Nicholas Sze added a comment - m1628_20100330.patch: updated the listing test in testMRConf.xml
          Hide
          Tsz Wo Nicholas Sze added a comment -

          m1628_20100329b.patch: fixed a bug.

          Show
          Tsz Wo Nicholas Sze added a comment - m1628_20100329b.patch: fixed a bug.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          m1628_20100329.patch:

          • for files, use part file status;
          • for dir, use .har dir status
          • need some tests.
          Show
          Tsz Wo Nicholas Sze added a comment - m1628_20100329.patch: for files, use part file status; for dir, use .har dir status need some tests.

            People

            • Assignee:
              Tsz Wo Nicholas Sze
              Reporter:
              Tsz Wo Nicholas Sze
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development