Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-2394

HadoopOutFormat OutputCommitter is default to FileOutputCommiter

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.9.0
    • Fix Version/s: 0.9.1, 0.10.0
    • Component/s: API / DataSet
    • Labels:
      None
    • Flags:
      Important

      Description

      MongoOutputFormat does not write back in collection because the HadoopOutputFormat wrapper does not allow to set the MongoOutputCommiter and is set as default to FileOutputCommitter. Therefore, on close and globalFinalize execution the commit does not happen and mongo collection stays untouched.

      A simple solution would be to:

      1 - create a constructor of HadoopOutputFormatBase and HadoopOutputFormat that gets the OutputCommitter as a parameter
      2 - change the outputCommitter field of HadoopOutputFormatBase to be a generic OutputCommitter
      3 - remove the default assignment in the open() and finalizeGlobal to the outputCommitter to FileOutputCommitter(), or keep it as a default in case of no specific assignment.

        Attachments

          Activity

            People

            • Assignee:
              fhueske Fabian Hueske
              Reporter:
              stefano.bortoli Stefano Bortoli
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: