Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-14170

Support hadoop < 2.7 with StreamingFileSink.BulkFormatBuilder

    XMLWordPrintableJSON

    Details

      Description

      Currently, StreamingFileSink is supported only with Hadoop >= 2.7 irrespective of Row/bulk format builder. This restriction is due to truncate is not supported in  Hadoop < 2.7

      However, BulkFormatBuilder does not use truncate method to restore the file. So the restricting StreamingFileSink.BulkFormatBuilder to be used only with Hadoop >= 2.7 is not necessary.

      So requested improvement is to remove the precondition on HadoopRecoverableWriter and allow  BulkFormatBuilder (Parquet) to be used in Hadoop 2.6 ( Most of the enterprises still on CDH 5.x)

       

       

       

       

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                dasbh Bhagavan
                Reporter:
                dasbh Bhagavan
              • Votes:
                1 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 10m
                  10m