Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-10203

Support truncate method for old Hadoop versions in HadoopRecoverableFsDataOutputStream

    XMLWordPrintableJSON

    Details

      Description

      New StreamingFileSink ( introduced in 1.6 Flink version ) use HadoopRecoverableFsDataOutputStream wrapper to write data in HDFS.

      HadoopRecoverableFsDataOutputStream is a wrapper for FSDataOutputStream to have an ability to restore from a certain point of the file after failure and continue to write data. To achieve this recovery functionality the HadoopRecoverableFsDataOutputStream uses "truncate" method which was introduced only in Hadoop 2.7.
      FLINK-14170 has enabled the usage of StreamingFileSink for OnCheckpointRollingPolicy, but it is still not possible to use StreamingFileSink with DefaultRollingPolicy, which makes writing of the data to HDFS unpractical in scale for HDFS < 2.7.

      Unfortunately, there are a few official Hadoop distributives which latest version still use Hadoop 2.6 (This distributives: Cloudera, Pivotal HD ). As the result Flinks Hadoop connector can't work with this distributives.

      Flink declares that supported Hadoop from version 2.4.0 upwards (https://ci.apache.org/projects/flink/flink-docs-release-1.6/start/building.html#hadoop-versions)

      I guess we should emulate the functionality of "truncate" method for older Hadoop versions.

        Attachments

        1. legacy truncate logic.pdf
          151 kB
          Artsem Semianenka

          Issue Links

            Activity

              People

              • Assignee:
                artsem.semianenka Artsem Semianenka
                Reporter:
                artsem.semianenka Artsem Semianenka
              • Votes:
                4 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 10m
                  10m