Camel
  1. Camel
  2. CAMEL-7080

Add support of lazy load to csv data format

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.12.3, 2.13.0
    • Component/s: None
    • Labels:
      None
    • Patch Info:
      Patch Available
    • Estimated Complexity:
      Unknown

      Description

      CsvDataFormat uses greedy loading from a stream. This is not applicable for huge files i.e. all rows cannot fit into available memory.

      I added an option to create a row iterator instead list.
      Csv format is not line oriented so split("\n") workaround doesn't work always. This patch adds ability to Camel to process infinite csv files with multiline rows.

      Pull request:
      https://github.com/yaitskov/camel/commits/csv-iterator

      P.S.
      1) I found that Bindy does similar work related with parsing csv file into list of why. And I don't see any clue why it doesn't delegate this job to csv file format. Roughly speaking there is code duplication.

      2) I didn't find camel-csv component on the ticket form.

        Issue Links

          Activity

          Hide
          ASF GitHub Bot added a comment -

          Github user jfarrell closed the pull request at:

          https://github.com/apache/camel/pull/73

          Show
          ASF GitHub Bot added a comment - Github user jfarrell closed the pull request at: https://github.com/apache/camel/pull/73
          Hide
          Willem Jiang added a comment -

          Applied the patch with thanks to Daneel. I also polished the code and fixed some CS errors.

          Show
          Willem Jiang added a comment - Applied the patch with thanks to Daneel. I also polished the code and fixed some CS errors.
          Hide
          Daneel Yaitskov added a comment -

          I did it.

          Show
          Daneel Yaitskov added a comment - I did it.
          Hide
          Willem Jiang added a comment -

          Hi Daneel,

          Can you send a pull request to apache/camel from your forked repo?
          We could have a tracker in the github

          Show
          Willem Jiang added a comment - Hi Daneel, Can you send a pull request to apache/camel from your forked repo? We could have a tracker in the github

            People

            • Assignee:
              Willem Jiang
              Reporter:
              Daneel S. Yaitskov
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development