Pig
  1. Pig
  2. PIG-2783

Fix Iterator_1 e2e test for Hadoop 23

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.10.0
    • Fix Version/s: 0.11, 0.10.1
    • Component/s: e2e harness
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Iterator_1 e2e test is failing because the script tries to copy output to a directory that does not exist. From Hadoop 23, the copy commands do not auto-create non-existent paths anymore.

      +Pig.fs("-mkdir :OUTPATH:")
      Pig.fs("-copyFromLocal :TMP:/iterator_output.txt :OUTPATH:/part-m-00000")

      1. PIG-2783.patch
        0.5 kB
        Rohini Palaniswamy

        Activity

        Rohini Palaniswamy created issue -
        Hide
        Rohini Palaniswamy added a comment -

        Comments from the hadoop team on Hadoop 23 copy command behaviour:

        Many commands used to auto-create non-existent paths. This behavior is most commonly relied upon in the copy commands. The target directory path will no longer be created. Please use "mkdir" or "mkdir -p" to create the target directory path prior to the copy. The "-p" flag will avoid an error if the directory already exists. If other commands are still auto-creating paths, do not rely upon that behavior.

        Show
        Rohini Palaniswamy added a comment - Comments from the hadoop team on Hadoop 23 copy command behaviour: Many commands used to auto-create non-existent paths. This behavior is most commonly relied upon in the copy commands. The target directory path will no longer be created. Please use "mkdir" or "mkdir -p" to create the target directory path prior to the copy. The "-p" flag will avoid an error if the directory already exists. If other commands are still auto-creating paths, do not rely upon that behavior.
        Rohini Palaniswamy made changes -
        Field Original Value New Value
        Attachment PIG-2783.patch [ 12534286 ]
        Rohini Palaniswamy made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Daniel Dai added a comment -

        Patch committed to 0.10/trunk. Thanks Rohini!

        Show
        Daniel Dai added a comment - Patch committed to 0.10/trunk. Thanks Rohini!
        Daniel Dai made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags Reviewed [ 10343 ]
        Fix Version/s 0.11 [ 12318878 ]
        Fix Version/s 0.10.1 [ 12320547 ]
        Resolution Fixed [ 1 ]
        Daniel Dai made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        6m 47s 1 Rohini Palaniswamy 02/Jul/12 17:33
        Patch Available Patch Available Resolved Resolved
        7d 2h 17m 1 Daniel Dai 09/Jul/12 19:50
        Resolved Resolved Closed Closed
        181d 5h 6m 1 Daniel Dai 06/Jan/13 23:57

          People

          • Assignee:
            Rohini Palaniswamy
            Reporter:
            Rohini Palaniswamy
          • Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development