Pig
  1. Pig
  2. PIG-2441

rmf does not work in local mode

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Invalid
    • Affects Version/s: 0.9.1, 0.9.2, 0.10.0
    • Fix Version/s: None
    • Component/s: grunt
    • Environment:

      Mac OS X 10.6.8

      Description

      $ pig -l /tmp -v -x local
      2011-12-20 18:58:54,074 [main] INFO org.apache.pig.Main - Logging error messages to: /private/tmp/pig_1324436334061.log
      2011-12-20 18:58:54,324 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: file:///
      grunt> rmf '/tmp/mail_pairs.avro'
      grunt> rmf '/tmp/mail_pairs.avro'
      grunt> rm '/tmp/mail_pairs.avro'
      2011-12-20 18:59:02,968 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2997: Encountered IOException. File or directory '/tmp/mail_pairs.avro' does not exist.
      2011-12-20 18:59:02,968 [main] ERROR org.apache.pig.tools.grunt.Grunt - java.io.IOException: File or directory '/tmp/mail_pairs.avro' does not exist.
      at org.apache.pig.tools.grunt.GruntParser.processRemove(GruntParser.java:957)
      at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:430)
      at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:188)
      at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
      at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
      at org.apache.pig.Main.run(Main.java:523)
      at org.apache.pig.Main.main(Main.java:148)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

      Details also at logfile: /private/tmp/pig_1324436334061.log
      grunt> ls /tmp/mail_pairs.avro
      file:/tmp/mail_pairs.avro/part-m-00000.avro<r 1> 0
      file:/tmp/mail_pairs.avro/part-m-00001.avro<r 1> 51024
      file:/tmp/mail_pairs.avro/part-m-00002.avro<r 1> 66431
      file:/tmp/mail_pairs.avro/part-m-00003.avro<r 1> 100130
      file:/tmp/mail_pairs.avro/part-m-00004.avro<r 1> 141016
      file:/tmp/mail_pairs.avro/part-m-00005.avro<r 1> 73522

        Issue Links

          Activity

          Russell Jurney created issue -
          Hide
          Russell Jurney added a comment -

          Note: there is a workaround...

          sh rm -rf '/tmp/mail_pairs.avro'
          sh rm -rf '/tmp/sent_counts.avro'

          Show
          Russell Jurney added a comment - Note: there is a workaround... sh rm -rf '/tmp/mail_pairs.avro' sh rm -rf '/tmp/sent_counts.avro'
          Russell Jurney made changes -
          Field Original Value New Value
          Affects Version/s 0.9.1 [ 12317343 ]
          Affects Version/s 0.9.2 [ 12318248 ]
          Harsh J made changes -
          Link This issue relates to PIG-535 [ PIG-535 ]
          Hide
          Harsh J added a comment -

          Per your comments, looks like even rm/rmr looked at the wrong location, not just rmf.

          Show
          Harsh J added a comment - Per your comments, looks like even rm/rmr looked at the wrong location, not just rmf.
          Russell Jurney made changes -
          Priority Major [ 3 ] Critical [ 2 ]
          Description russell-jurneys-macbook-pro:Collecting-Data peyomp$ pig -l /tmp -v -x local
          2011-12-20 18:58:54,074 [main] INFO org.apache.pig.Main - Logging error messages to: /private/tmp/pig_1324436334061.log
          2011-12-20 18:58:54,324 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: file:///
          grunt> rmf '/tmp/mail_pairs.avro'
          grunt> rmf '/tmp/mail_pairs.avro'
          grunt> rm '/tmp/mail_pairs.avro'
          2011-12-20 18:59:02,968 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2997: Encountered IOException. File or directory '/tmp/mail_pairs.avro' does not exist.
          2011-12-20 18:59:02,968 [main] ERROR org.apache.pig.tools.grunt.Grunt - java.io.IOException: File or directory '/tmp/mail_pairs.avro' does not exist.
          at org.apache.pig.tools.grunt.GruntParser.processRemove(GruntParser.java:957)
          at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:430)
          at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:188)
          at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
          at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
          at org.apache.pig.Main.run(Main.java:523)
          at org.apache.pig.Main.main(Main.java:148)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

          Details also at logfile: /private/tmp/pig_1324436334061.log
          grunt> ls /tmp/mail_pairs.avro
          file:/tmp/mail_pairs.avro/part-m-00000.avro<r 1> 0
          file:/tmp/mail_pairs.avro/part-m-00001.avro<r 1> 51024
          file:/tmp/mail_pairs.avro/part-m-00002.avro<r 1> 66431
          file:/tmp/mail_pairs.avro/part-m-00003.avro<r 1> 100130
          file:/tmp/mail_pairs.avro/part-m-00004.avro<r 1> 141016
          file:/tmp/mail_pairs.avro/part-m-00005.avro<r 1> 73522
          $ pig -l /tmp -v -x local
          2011-12-20 18:58:54,074 [main] INFO org.apache.pig.Main - Logging error messages to: /private/tmp/pig_1324436334061.log
          2011-12-20 18:58:54,324 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: file:///
          grunt> rmf '/tmp/mail_pairs.avro'
          grunt> rmf '/tmp/mail_pairs.avro'
          grunt> rm '/tmp/mail_pairs.avro'
          2011-12-20 18:59:02,968 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2997: Encountered IOException. File or directory '/tmp/mail_pairs.avro' does not exist.
          2011-12-20 18:59:02,968 [main] ERROR org.apache.pig.tools.grunt.Grunt - java.io.IOException: File or directory '/tmp/mail_pairs.avro' does not exist.
          at org.apache.pig.tools.grunt.GruntParser.processRemove(GruntParser.java:957)
          at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:430)
          at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:188)
          at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
          at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
          at org.apache.pig.Main.run(Main.java:523)
          at org.apache.pig.Main.main(Main.java:148)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

          Details also at logfile: /private/tmp/pig_1324436334061.log
          grunt> ls /tmp/mail_pairs.avro
          file:/tmp/mail_pairs.avro/part-m-00000.avro<r 1> 0
          file:/tmp/mail_pairs.avro/part-m-00001.avro<r 1> 51024
          file:/tmp/mail_pairs.avro/part-m-00002.avro<r 1> 66431
          file:/tmp/mail_pairs.avro/part-m-00003.avro<r 1> 100130
          file:/tmp/mail_pairs.avro/part-m-00004.avro<r 1> 141016
          file:/tmp/mail_pairs.avro/part-m-00005.avro<r 1> 73522
          Hide
          Daniel Dai added a comment -

          Seems I have trouble reproduce the issue. Can you tell me which version of Pig you are using? And can you do "pig -secretDebugCmd -x local", and paste the output?

          Show
          Daniel Dai added a comment - Seems I have trouble reproduce the issue. Can you tell me which version of Pig you are using? And can you do "pig -secretDebugCmd -x local", and paste the output?
          Hide
          Russell Jurney added a comment -

          pig -secretDebugCmd -x local
          Find hadoop at /Users/rjurney/hadoop-0.20.2/bin/hadoop
          dry run:
          HADOOP_CLASSPATH: /Users/rjurney/pig/bin/../conf:/System/Library/Frameworks/JavaVM.framework/Versions/1.6/Home/lib/tools.jar:/Users/rjurney/pig/bin/../lib/automaton.jar:/Users/rjurney/pig/bin/../build/ivy/lib/Pig/jython-2.5.0.jar:/Users/rjurney/pig/bin/../pig-withouthadoop.jar:
          HADOOP_OPTS: -Xmx1000m -Dpig.log.dir=/Users/rjurney/pig/bin/../logs -Dpig.log.file=pig.log -Dpig.home.dir=/Users/rjurney/pig/bin/..
          /Users/rjurney/hadoop-0.20.2/bin/hadoop jar /Users/rjurney/pig/bin/../pig-withouthadoop.jar -x local

          Show
          Russell Jurney added a comment - pig -secretDebugCmd -x local Find hadoop at /Users/rjurney/hadoop-0.20.2/bin/hadoop dry run: HADOOP_CLASSPATH: /Users/rjurney/pig/bin/../conf:/System/Library/Frameworks/JavaVM.framework/Versions/1.6/Home/lib/tools.jar:/Users/rjurney/pig/bin/../lib/automaton.jar:/Users/rjurney/pig/bin/../build/ivy/lib/Pig/jython-2.5.0.jar:/Users/rjurney/pig/bin/../pig-withouthadoop.jar: HADOOP_OPTS: -Xmx1000m -Dpig.log.dir=/Users/rjurney/pig/bin/../logs -Dpig.log.file=pig.log -Dpig.home.dir=/Users/rjurney/pig/bin/.. /Users/rjurney/hadoop-0.20.2/bin/hadoop jar /Users/rjurney/pig/bin/../pig-withouthadoop.jar -x local
          Hide
          Russell Jurney added a comment -

          This was actually not working because the path was in single quotes. Once the path is removed from single quotes, the rm works.

          So, should I either close the ticket - or should rmf work with quotes?

          Show
          Russell Jurney added a comment - This was actually not working because the path was in single quotes. Once the path is removed from single quotes, the rm works. So, should I either close the ticket - or should rmf work with quotes?
          Show
          Daniel Dai added a comment - Yes rmf does not work with quotes right now. See http://mail-archives.apache.org/mod_mbox/pig-user/201108.mbox/%3CCAAZU44m9RhagOJu7hJFC-bGAXA8d8tmw+_u2kGeGKk9jqD=w7A@mail.gmail.com%3E
          Hide
          Russell Jurney added a comment -

          Should I edit this ticket to be about quoted paths not working - or close it?

          Show
          Russell Jurney added a comment - Should I edit this ticket to be about quoted paths not working - or close it?
          Hide
          Daniel Dai added a comment -

          You can open a new ticket about quoted paths. This ticket is becoming long and confusing.

          Show
          Daniel Dai added a comment - You can open a new ticket about quoted paths. This ticket is becoming long and confusing.
          Hide
          Dmitriy V. Ryaboy added a comment -

          Please open a separate issue about quotes as per Daniel's request (otherwise it's totally confusing what the problem is, since the stated issue is not the real issue).

          Show
          Dmitriy V. Ryaboy added a comment - Please open a separate issue about quotes as per Daniel's request (otherwise it's totally confusing what the problem is, since the stated issue is not the real issue).
          Dmitriy V. Ryaboy made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Invalid [ 6 ]

            People

            • Assignee:
              Unassigned
              Reporter:
              Russell Jurney
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development