Hadoop Common
  1. Hadoop Common
  2. HADOOP-3911

' -blocks ' option not being recognized

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.19.0
    • Component/s: fs, util
    • Labels:
      None
    • Environment:

      branch-0.18

    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Added a check to fsck options to make sure -files is not the first option so as to resolve conflicts with GenericOptionsParser.

      Description

      Somehow depending on the order of options, GenericOptionsParser throws an error.

      This fail.

      [knoguchi@gsgw2001 tmp]$ ~/branch-0.18/bin/hadoop fsck
      Usage: DFSck <path> [-move | -delete | -openforwrite] [-files [-blocks [-locations | -racks]]]
      
      [knoguchi@tmp]$ ~/branch-0.18/bin/hadoop fsck -files -blocks -locations /user/knoguchi
      java.io.FileNotFoundException: File -blocks does not exist.
              at org.apache.hadoop.util.GenericOptionsParser.validateFiles(GenericOptionsParser.java:278)
              at org.apache.hadoop.util.GenericOptionsParser.processGeneralOptions(GenericOptionsParser.java:233)
              at org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:315)
              at org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:134)
              at org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:119)
              at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:59)
              at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
              at org.apache.hadoop.dfs.DFSck.main(DFSck.java:137)
      
      ...Status: HEALTHY
       Total size:    2769 B
       Total dirs:    4
       Total files:   3
       Total blocks (validated):      3 (avg. block size 923 B)
      ...
      

      This works.

      [knoguchi@tmp]$ ~/branch-0.18/bin/hadoop fsck -blocks -locations -files /user/knoguchi
      /user/knoguchi/.Trash <dir>
      /user/knoguchi/hod-logs <dir>
      /user/knoguchi/hod-logs/____ <dir>
      /user/knoguchi/hod-logs/____/aaa.tar.gz 1024 bytes, 1 block(s):  OK
      0. blk_-5724352734215884188_0 len=1024 repl=3 [11.111.11.11:11111, 22.22.22.22:11111, 33.33.33.33:11111]
      ...
      /user/knoguchi/mapredsystem <dir>
      Status: HEALTHY
       Total size:    2769 B
       Total dirs:    4
       Total files:   3
       Total blocks (validated):      3 (avg. block size 923 B)
       
      1. HADOOP-3911.patch
        3 kB
        Lohit Vijayarenu

        Activity

        Nigel Daley made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Robert Chansler made changes -
        Hadoop Flags [Reviewed, Incompatible change] [Incompatible change, Reviewed]
        Release Note Add a check to fsck options to make sure -files is not the first option so as to resolve conflicts with GenericOptionsParser Added a check to fsck options to make sure -files is not the first option so as to resolve conflicts with GenericOptionsParser.
        Hide
        Hudson added a comment -
        Show
        Hudson added a comment - Integrated in Hadoop-trunk #611 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/611/ )
        Lohit Vijayarenu made changes -
        Hadoop Flags [Reviewed, Incompatible change] [Incompatible change, Reviewed]
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Release Note Add a check to fsck options to make sure -files is not the first option so as to resolve conflicts with GenericOptionsParser
        Fix Version/s 0.19.0 [ 12313211 ]
        Hide
        Lohit Vijayarenu added a comment -

        Thanks for review Nicolas,
        I just committed this.

        Show
        Lohit Vijayarenu added a comment - Thanks for review Nicolas, I just committed this.
        Tsz Wo Nicholas Sze made changes -
        Hadoop Flags [Incompatible change, Reviewed]
        Hide
        Tsz Wo Nicholas Sze added a comment -

        +1 patch looks good.

        need release note.

        Show
        Tsz Wo Nicholas Sze added a comment - +1 patch looks good. need release note.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12390048/HADOOP-3911.patch
        against trunk revision 695755.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no tests are needed for this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3270/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3270/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3270/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3270/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12390048/HADOOP-3911.patch against trunk revision 695755. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3270/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3270/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3270/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3270/console This message is automatically generated.
        Lohit Vijayarenu made changes -
        Status Reopened [ 4 ] Patch Available [ 10002 ]
        Lohit Vijayarenu made changes -
        Attachment HADOOP-3911.patch [ 12390048 ]
        Hide
        Lohit Vijayarenu added a comment - - edited

        This was not happenning in 0.18 becuase GenericOptionsParser did not consider -files to be a generic option. What seems to be happening here is that -files which was used by fsck is being treated as an option for GenericOptionsParser. It might not be good idea to change fsck's -files option now. I think making a check that is not the first argument works. Here is a patch for that.

        Show
        Lohit Vijayarenu added a comment - - edited This was not happenning in 0.18 becuase GenericOptionsParser did not consider -files to be a generic option. What seems to be happening here is that -files which was used by fsck is being treated as an option for GenericOptionsParser. It might not be good idea to change fsck's -files option now. I think making a check that is not the first argument works. Here is a patch for that.
        Robert Chansler made changes -
        Assignee Lohit Vijayarenu [ lohit ]
        Hide
        Lohit Vijayarenu added a comment -

        Yes, this has to be fixed. Nothing seems to be changed in terms of options parsing of fsck. Either it was broken from beginning or something changed in GenericOptionsParser.

        Show
        Lohit Vijayarenu added a comment - Yes, this has to be fixed. Nothing seems to be changed in terms of options parsing of fsck. Either it was broken from beginning or something changed in GenericOptionsParser.
        Koji Noguchi made changes -
        Priority Major [ 3 ] Minor [ 4 ]
        Koji Noguchi made changes -
        Status Resolved [ 5 ] Reopened [ 4 ]
        Resolution Invalid [ 6 ]
        Hide
        Koji Noguchi added a comment -

        > But Koji, this is still a bug if a Java stack trace is printed. No command-line commands should spill stack traces.
        >
        True.
        Ok, reopening the Jira.
        I'll change the priority to minor next.

        Show
        Koji Noguchi added a comment - > But Koji, this is still a bug if a Java stack trace is printed. No command-line commands should spill stack traces. > True. Ok, reopening the Jira. I'll change the priority to minor next.
        Hide
        Nigel Daley added a comment -

        But Koji, this is still a bug if a Java stack trace is printed. No command-line commands should spill stack traces.

        Show
        Nigel Daley added a comment - But Koji, this is still a bug if a Java stack trace is printed. No command-line commands should spill stack traces.
        Koji Noguchi made changes -
        Field Original Value New Value
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Invalid [ 6 ]
        Hide
        Koji Noguchi added a comment -

        Lohit told me that path has to come before the options.

        Before version 0.18, it used to be lenient with order of arguments, however, usage clearly states the correct command.

        Closing as invalid.

        Show
        Koji Noguchi added a comment - Lohit told me that path has to come before the options. Before version 0.18, it used to be lenient with order of arguments, however, usage clearly states the correct command. Closing as invalid.
        Koji Noguchi created issue -

          People

          • Assignee:
            Lohit Vijayarenu
            Reporter:
            Koji Noguchi
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development