Uploaded image for project: 'Camel'
  1. Camel
  2. CAMEL-8542

hdfs & hdfs2 components are merging data locally instead of streaming it

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Abandoned
    • 2.15.0
    • None
    • camel-hdfs
    • None
    • Moderate

    Description

      Here is the conversation

      CAMEL-4555 introduced an ability to merge files from within a single directory.
      The merge operation is done locally, i.e. by means of creating the whole file on the local file system (that may be space and time consuming in case of multi -gigabyte, -terabyte files).

      1. It will be more efficient to stream these files directly from hdfs, for example by wrapping them into SequenceInputStream or something like this MapReducePartInputStreamEnumeration
      2. It will be really great if there will be an ability to switch merging on and off by means of an option or parameter.

      Attachments

        Activity

          People

            Unassigned Unassigned
            szhemzhitsky Sergey Zhemzhitsky
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: