Hadoop Common
  1. Hadoop Common
  2. HADOOP-5889

Allow writing to output directories that exist, as long as they are empty

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Incomplete
    • Affects Version/s: 0.18.3
    • Fix Version/s: None
    • Component/s: fs
    • Labels:
      None

      Description

      The current behavior in FileOutputFormat.checkOutputSpecs is to fail if the path specified by mapred.output.dir exists at the start of the job. This is to protect from accidentally overwriting existing data. There seems no harm then in slightly relaxing this check to allow the case for the output to exist if it is an empty directory.

      At a minimum this would allow outputting to the root of S3N buckets, which is currently impossible (https://issues.apache.org/jira/browse/HADOOP-5805).

        Activity

        Ian Nowland created issue -
        Ian Nowland made changes -
        Field Original Value New Value
        Attachment HADOOP-5889-0.patch [ 12410341 ]
        Ian Nowland made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Fix Version/s 0.21.0 [ 12313563 ]
        Ian Nowland made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Tom White made changes -
        Fix Version/s 0.21.0 [ 12313563 ]
        Allen Wittenauer made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Incomplete [ 4 ]

          People

          • Assignee:
            Unassigned
            Reporter:
            Ian Nowland
          • Votes:
            1 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development