Pig
  1. Pig
  2. PIG-2783

Fix Iterator_1 e2e test for Hadoop 23

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.10.0
    • Fix Version/s: 0.11, 0.10.1
    • Component/s: e2e harness
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Iterator_1 e2e test is failing because the script tries to copy output to a directory that does not exist. From Hadoop 23, the copy commands do not auto-create non-existent paths anymore.

      +Pig.fs("-mkdir :OUTPATH:")
      Pig.fs("-copyFromLocal :TMP:/iterator_output.txt :OUTPATH:/part-m-00000")

      1. PIG-2783.patch
        0.5 kB
        Rohini Palaniswamy

        Activity

        Hide
        Rohini Palaniswamy added a comment -

        Comments from the hadoop team on Hadoop 23 copy command behaviour:

        Many commands used to auto-create non-existent paths. This behavior is most commonly relied upon in the copy commands. The target directory path will no longer be created. Please use "mkdir" or "mkdir -p" to create the target directory path prior to the copy. The "-p" flag will avoid an error if the directory already exists. If other commands are still auto-creating paths, do not rely upon that behavior.

        Show
        Rohini Palaniswamy added a comment - Comments from the hadoop team on Hadoop 23 copy command behaviour: Many commands used to auto-create non-existent paths. This behavior is most commonly relied upon in the copy commands. The target directory path will no longer be created. Please use "mkdir" or "mkdir -p" to create the target directory path prior to the copy. The "-p" flag will avoid an error if the directory already exists. If other commands are still auto-creating paths, do not rely upon that behavior.
        Hide
        Daniel Dai added a comment -

        Patch committed to 0.10/trunk. Thanks Rohini!

        Show
        Daniel Dai added a comment - Patch committed to 0.10/trunk. Thanks Rohini!

          People

          • Assignee:
            Rohini Palaniswamy
            Reporter:
            Rohini Palaniswamy
          • Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development