Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1628

HarFileSystem shows incorrect replication numbers and permissions

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: harchive
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      In the har dir, the replication # of part-0 is 3.

      -bash-3.1$ hadoop fs -ls  ${DIR}.har
      Found 3 items
      -rw-------   5 tsz users       1141 2010-02-10 18:34 /user/tsz/t20.har/_index
      -rw-------   5 tsz users         24 2010-02-10 18:34 /user/tsz/t20.har/_masterindex
      -rw-------   3 tsz users      15052 2010-02-10 18:34 /user/tsz/t20.har/part-0
      

      but the replication # of the individual har:// files is shown as 5.

      -bash-3.1$ hadoop fs -lsr  ${HAR_FULL}/
      drw-------   - tsz users          0 2010-02-10 18:34 /user/tsz/t20.har/t20
      -rw-------   5 tsz users        723 2010-02-10 18:34 /user/tsz/t20.har/t20/text-00000000
      -rw-------   5 tsz users        779 2010-02-10 18:34 /user/tsz/t20.har/t20/text-00000001
      -rw-------   5 tsz users        818 2010-02-10 18:34 /user/tsz/t20.har/t20/text-00000002
      ...
      

      The permission also has similar problem. Clearly, the permission of t20.har/t20 shown above is incorrect.

      1. m1628_20100329.patch
        6 kB
        Tsz Wo Nicholas Sze
      2. m1628_20100329b.patch
        6 kB
        Tsz Wo Nicholas Sze
      3. m1628_20100330.patch
        9 kB
        Tsz Wo Nicholas Sze
      4. m1628_20100331.patch
        9 kB
        Tsz Wo Nicholas Sze

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Patch Available Patch Available Open Open
          20h 18m 1 Tsz Wo Nicholas Sze 31/Mar/10 19:01
          Open Open Patch Available Patch Available
          31d 21h 42m 2 Tsz Wo Nicholas Sze 31/Mar/10 19:03
          Patch Available Patch Available Resolved Resolved
          43m 24s 1 Mahadev konar 31/Mar/10 19:46
          Resolved Resolved Closed Closed
          146d 2h 34m 1 Tom White 24/Aug/10 22:21
          Tom White made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Tom White made changes -
          Fix Version/s 0.21.0 [ 12314045 ]
          Fix Version/s 0.22.0 [ 12314184 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #276 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/276/)
          . HarFileSystem shows incorrect replication numbers and permissions (tsz via mahadev)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #276 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/276/ ) . HarFileSystem shows incorrect replication numbers and permissions (tsz via mahadev)
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #296 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/296/)
          . HarFileSystem shows incorrect replication numbers and permissions (tsz via mahadev)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #296 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/296/ ) . HarFileSystem shows incorrect replication numbers and permissions (tsz via mahadev)
          Mahadev konar made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Resolution Fixed [ 1 ]
          Hide
          Mahadev konar added a comment -

          I just committed this. thanks nicholas!

          Show
          Mahadev konar added a comment - I just committed this. thanks nicholas!
          Hide
          Mahadev konar added a comment -

          +1 the patch looks good!

          Show
          Mahadev konar added a comment - +1 the patch looks good!
          Tsz Wo Nicholas Sze made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > slightly changed the codes.
          More details: use HarStatus.isDir instead of comparing "none".

          Show
          Tsz Wo Nicholas Sze added a comment - > slightly changed the codes. More details: use HarStatus.isDir instead of comparing "none".
          Tsz Wo Nicholas Sze made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Tsz Wo Nicholas Sze made changes -
          Attachment m1628_20100331.patch [ 12440369 ]
          Hide
          Tsz Wo Nicholas Sze added a comment -

          m1628_20100331.patch: slightly changed the codes.

          Show
          Tsz Wo Nicholas Sze added a comment - m1628_20100331.patch: slightly changed the codes.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12440285/m1628_20100330.patch
          against trunk revision 928104.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 5 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/73/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/73/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/73/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/73/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12440285/m1628_20100330.patch against trunk revision 928104. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 5 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/73/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/73/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/73/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/73/console This message is automatically generated.
          Tsz Wo Nicholas Sze made changes -
          Component/s harchive [ 12312903 ]
          Tsz Wo Nicholas Sze made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Assignee Mahadev konar [ mahadev ] Tsz Wo (Nicholas), SZE [ szetszwo ]
          Tsz Wo Nicholas Sze made changes -
          Attachment m1628_20100330.patch [ 12440285 ]
          Hide
          Tsz Wo Nicholas Sze added a comment -

          m1628_20100330.patch: updated the listing test in testMRConf.xml

          Show
          Tsz Wo Nicholas Sze added a comment - m1628_20100330.patch: updated the listing test in testMRConf.xml
          Tsz Wo Nicholas Sze made changes -
          Attachment m1628_20100329b.patch [ 12440140 ]
          Hide
          Tsz Wo Nicholas Sze added a comment -

          m1628_20100329b.patch: fixed a bug.

          Show
          Tsz Wo Nicholas Sze added a comment - m1628_20100329b.patch: fixed a bug.
          Tsz Wo Nicholas Sze made changes -
          Attachment m1628_20100329.patch [ 12440137 ]
          Hide
          Tsz Wo Nicholas Sze added a comment -

          m1628_20100329.patch:

          • for files, use part file status;
          • for dir, use .har dir status
          • need some tests.
          Show
          Tsz Wo Nicholas Sze added a comment - m1628_20100329.patch: for files, use part file status; for dir, use .har dir status need some tests.
          Mahadev konar made changes -
          Project Hadoop Common [ 12310240 ] Hadoop Map/Reduce [ 12310941 ]
          Key HADOOP-6601 MAPREDUCE-1628
          Fix Version/s 0.22.0 [ 12314184 ]
          Fix Version/s 0.22.0 [ 12314296 ]
          Component/s fs [ 12310689 ]
          Mahadev konar made changes -
          Fix Version/s 0.22.0 [ 12314296 ]
          Tsz Wo Nicholas Sze made changes -
          Link This issue is related to MAPREDUCE-1548 [ MAPREDUCE-1548 ]
          Tsz Wo Nicholas Sze made changes -
          Field Original Value New Value
          Summary HarFileSystem shows incorrect replication numbers HarFileSystem shows incorrect replication numbers and permissions
          Description In the har dir, the replication # of part-0 is 3.
          {noformat}
          -bash-3.1$ hadoop fs -ls ${DIR}.har
          Found 3 items
          -rw------- 5 tsz users 1141 2010-02-10 18:34 /user/tsz/t20.har/_index
          -rw------- 5 tsz users 24 2010-02-10 18:34 /user/tsz/t20.har/_masterindex
          -rw------- 3 tsz users 15052 2010-02-10 18:34 /user/tsz/t20.har/part-0
          {noformat}
          but the replication # of the individual har:// files is shown as 5.
          {noformat}
          -bash-3.1$ hadoop fs -ls ${HAR_FULL}/${DIR}
          Found 20 items
          -rw------- 5 tsz users 723 2010-02-10 18:34 /user/tsz/t20.har/t20/text-00000000
          -rw------- 5 tsz users 779 2010-02-10 18:34 /user/tsz/t20.har/t20/text-00000001
          -rw------- 5 tsz users 818 2010-02-10 18:34 /user/tsz/t20.har/t20/text-00000002
          -rw------- 5 tsz users 811 2010-02-10 18:34 /user/tsz/t20.har/t20/text-00000003
          ...
          {noformat}
          In the har dir, the replication # of part-0 is 3.
          {noformat}
          -bash-3.1$ hadoop fs -ls ${DIR}.har
          Found 3 items
          -rw------- 5 tsz users 1141 2010-02-10 18:34 /user/tsz/t20.har/_index
          -rw------- 5 tsz users 24 2010-02-10 18:34 /user/tsz/t20.har/_masterindex
          -rw------- 3 tsz users 15052 2010-02-10 18:34 /user/tsz/t20.har/part-0
          {noformat}
          but the replication # of the individual har:// files is shown as 5.
          {noformat}
          -bash-3.1$ hadoop fs -lsr ${HAR_FULL}/
          drw------- - tsz users 0 2010-02-10 18:34 /user/tsz/t20.har/t20
          -rw------- 5 tsz users 723 2010-02-10 18:34 /user/tsz/t20.har/t20/text-00000000
          -rw------- 5 tsz users 779 2010-02-10 18:34 /user/tsz/t20.har/t20/text-00000001
          -rw------- 5 tsz users 818 2010-02-10 18:34 /user/tsz/t20.har/t20/text-00000002
          ...
          {noformat}
          The permission also has similar problem. Clearly, the permission of t20.har/t20 shown above is incorrect.
          Tsz Wo Nicholas Sze created issue -

            People

            • Assignee:
              Tsz Wo Nicholas Sze
              Reporter:
              Tsz Wo Nicholas Sze
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development