Uploaded image for project: 'Camel'
  1. Camel
  2. CAMEL-8431

Consume all files in aws S3 bucket where deleteAfterRead = false

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.14.0
    • Fix Version/s: 2.15.4, 2.16.1, 2.17.0
    • Component/s: camel-aws
    • Labels:
      None
    • Estimated Complexity:
      Unknown

      Description

      The current AWS S3Consumer class has a problem if user supply the parameter deleteAfterRead=false. The S3Consumer will always consume the same files over and over again with the size of getMaxMessagesPerPoll().

      After some code chasing, i think, the root cause is because the ListObjectsRequest does not has the previous marker value. Hence, i wonder if we could do something like below:

      private string marker; // new line to define the marker string
      ..
      ..
      ..
      ListObjectsRequest listObjectsRequest = new ListObjectsRequest();
      listObjectsRequest.setBucketName(bucketName);
      listObjectsRequest.setPrefix(getConfiguration().getPrefix());
      listObjectsRequest.setMaxKeys(maxMessagesPerPoll);
      listObjectsRequest.setMarker(marker); // new line to set the marker

      ObjectListing listObjects = getAmazonS3Client().listObjects(listObjectsRequest);
      marker = listObjects.getMarker(); // where marker is track

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                njiang Willem Jiang
                Reporter:
                reusable Yap Poh Soon
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: