Flume
  1. Flume
  2. FLUME-1228

flume-ng fails while writing to S3 sink

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Critical Critical
    • Resolution: Unresolved
    • Affects Version/s: v1.2.0, v1.4.0
    • Fix Version/s: None
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      flume-ng (version 1.2.0) fails while writing to S3 sink since it gets back 404 response code. The files with data is created on S3 though.

      Hadoop version used is 0.20.2-cdh3u4

      Followed all the steps documented in the jira - https://issues.cloudera.org/browse/FLUME-66
      and also I tried swapping out hadoop-core.jar that comes with CDH, with emr-hadoop-core-0.20.jar that comes with EC2 hadoop cluster instance as suggested in the following blog post - http://eric.lubow.org/2011/system-administration/distributed-flume-setup-with-an-s3-sink/ but the issue still remains.

      Following errors are seen in the log:

      2012-05-25 05:04:28,889 WARN httpclient.RestS3Service: Response '/flumedata%2FFlumeData.122585423857995.tmp_%24folder%24' - Unexpected response code 404, expected 200
      2012-05-25 05:04:28,964 INFO s3native.NativeS3FileSystem: OutputStream for key 'flumedata/FlumeData.122585423857995.tmp' writing to tempfile '/tmp/hadoop-root/s3/output-8042215269186280519.tmp'
      2012-05-25 05:04:28,972 INFO s3native.NativeS3FileSystem: OutputStream for key 'flumedata/FlumeData.122585423857995.tmp' closed. Now beginning upload
      2012-05-25 05:04:29,044 INFO s3native.NativeS3FileSystem: OutputStream for key 'flumedata/FlumeData.122585423857995.tmp' upload complete
      2012-05-25 05:04:29,074 INFO hdfs.BucketWriter: Renaming s3n://flume-ng/flumedata/FlumeData.122585423857995.tmp to s3n://flume-ng/flumedata/FlumeData.122585423857995
      2012-05-25 05:04:29,097 WARN httpclient.RestS3Service: Response '/flumedata%2FFlumeData.122585423857995' - Unexpected response code 404, expected 200
      2012-05-25 05:04:29,120 WARN httpclient.RestS3Service: Response '/flumedata%2FFlumeData.122585423857995_%24folder%24' - Unexpected response code 404, expected 200
      2012-05-25 05:04:29,203 WARN httpclient.RestS3Service: Response '/flumedata' - Unexpected response code 404, expected 200
      2012-05-25 05:04:29,224 WARN httpclient.RestS3Service: Response '/flumedata_%24folder%24' - Unexpected response code 404, expected 200
      2012-05-25 05:04:29,608 INFO hdfs.BucketWriter: Creating s3n://flume-ng/flumedata/FlumeData.122585423857996.tmp
      2012-05-25 05:04:29,720 WARN httpclient.RestS3Service: Response '/flumedata%2FFlumeData.122585423857996.tmp' - Unexpected response code 404, expected 200
      2012-05-25 05:04:29,748 WARN httpclient.RestS3Service: Response '/flumedata%2FFlumeData.122585423857996.tmp_%24folder%24' - Unexpected response code 404, expected 200
      2012-05-25 05:04:29,791 INFO s3native.NativeS3FileSystem: OutputStream for key 'flumedata/FlumeData.122585423857996.tmp' writing to tempfile '/tmp/hadoop-root/s3/output-2477068572058013384.tmp'
      2012-05-25 05:04:29,793 INFO s3native.NativeS3FileSystem: OutputStream for key 'flumedata/FlumeData.122585423857996.tmp' closed. Now beginning upload
      2012-05-25 05:04:29,828 INFO s3native.NativeS3FileSystem: OutputStream for key 'flumedata/FlumeData.122585423857996.tmp' upload complete

        Issue Links

          Activity

          Hide
          Rob Slifka added a comment -

          On CDH 5.1, should be there

          Show
          Rob Slifka added a comment - On CDH 5.1, should be there
          Hide
          Dennis Waldron added a comment -

          Are you sure Flume 1.5 ships with an updated version of jets3t? From what I see of the packaging and build configuration, the binary distribution doesn't include the Hadoop libraries and its associated dependencies (of which jets3t is one of them).

          Show
          Dennis Waldron added a comment - Are you sure Flume 1.5 ships with an updated version of jets3t? From what I see of the packaging and build configuration, the binary distribution doesn't include the Hadoop libraries and its associated dependencies (of which jets3t is one of them).
          Hide
          Rob Slifka added a comment -

          As a follow-up, we're moving to Flume 1.5 which ships with an updated jets3t but also because of FLUME-2307. Thanks again!

          Show
          Rob Slifka added a comment - As a follow-up, we're moving to Flume 1.5 which ships with an updated jets3t but also because of FLUME-2307 . Thanks again!
          Hide
          Rob Slifka added a comment -

          Dennis Waldron, we put this single instance under significant load with the new jets3t and did not notice any errors. We modified no other portion of our installation. Thanks again!

          Show
          Rob Slifka added a comment - Dennis Waldron , we put this single instance under significant load with the new jets3t and did not notice any errors. We modified no other portion of our installation. Thanks again!
          Hide
          Rob Slifka added a comment -

          Huge thanks Dennis! Will get back to you after we give this a shot, thanks!

          Show
          Rob Slifka added a comment - Huge thanks Dennis! Will get back to you after we give this a shot, thanks!
          Hide
          Dennis Waldron added a comment -

          Hi Rob, I'm not discussing this topic purely because nobody has asked for additional help.

          I see that you are using CDH 4.4 which maps to Hadoop 2.0.0. As I don't use a full blown Hadoop installation I've made some effort to simulate your problem on an AWS EC2 node using the library versions defined by the CDH project (http://maven-repository.com/artifact/com.cloudera.cdh/cdh-root/4.4.0/pom)

          Using an unmodified Flume 1.4 installation with the following additional libraries:

          -rw-rw-r-- 1 ec2-user ec2-user 1756571 Jul 16 21:34 hadoop-core-2.0.0-mr1-cdh4.4.0.jar
          -rw-rw-r-- 1 ec2-user ec2-user 2284942 Jul 16 21:34 hadoop-common-2.0.0-cdh4.4.0.jar
          -rw-rw-r-- 1 ec2-user ec2-user   46855 Jul 16 21:34 hadoop-auth-2.0.0-cdh4.4.0.jar
          -rw-rw-r-- 1 ec2-user ec2-user  305001 Jul 16 21:34 commons-httpclient-3.1.jar
          -rw-rw-r-- 1 ec2-user ec2-user  298829 Jul 16 21:34 commons-configuration-1.6.jar
          -rw-rw-r-- 1 ec2-user ec2-user  321806 Jul 16 21:34 jets3t-0.6.1.jar
          

          I was able to reproduce the 404 problem:

          2014-07-16 21:36:19,158 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:219)] Creating s3n://S3_BUCKET/flume-debug/FlumeData.1405546579015.tmp
          2014-07-16 21:36:20,291 (hdfs-hdfs-1-call-runner-0) [WARN - org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:393)] Response '/flume-debug%2FFlumeData.1405546579015.tmp' - Unexpected response code 404, expected 200
          2014-07-16 21:36:20,306 (hdfs-hdfs-1-call-runner-0) [WARN - org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:393)] Response '/flume-debug%2FFlumeData.1405546579015.tmp_%24folder%24' - Unexpected response code 404, expected 200
          2014-07-16 21:36:20,336 (hdfs-hdfs-1-call-runner-0) [INFO - org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsOutputStream.<init>(NativeS3FileSystem.java:182)] OutputStream for key 'flume-
          2014-07-16 21:36:20,374 (hdfs-hdfs-1-call-runner-3) [INFO - org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsOutputStream.close(NativeS3FileSystem.java:217)] OutputStream for key 'flume-debug/FlumeData.1405546579015.tmp' closed. Now beginning upload
          2014-07-16 21:36:20,417 (hdfs-hdfs-1-call-runner-3) [INFO - org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsOutputStream.close(NativeS3FileSystem.java:229)] OutputStream for key 'flume-debug/FlumeData.1405546579015.tmp' upload complete
          2014-07-16 21:36:20,435 (hdfs-hdfs-1-call-runner-4) [INFO - org.apache.flume.sink.hdfs.BucketWriter$7.call(BucketWriter.java:487)] Renaming s3n://S3_BUCKET/flume-debug/FlumeData.1405546579015.tmp to s3n://S3_BUCKET/flume-debug/FlumeData.1405546579015
          

          Now by changing the jets3t library from version 0.6.1 to 0.7.1 (as per my previous recommendation):

          -rw-rw-r-- 1 ec2-user ec2-user  377780 Jul 16 21:40 jets3t-0.7.1.jar
          

          The problem is fixed:

          2014-07-16 21:39:36,724 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:219)] Creating s3n://S3_BUCKET/flume-debug/FlumeData.1405546776595.tmp
          2014-07-16 21:39:38,071 (hdfs-hdfs-1-call-runner-0) [INFO - org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsOutputStream.<init>(NativeS3FileSystem.java:182)] OutputStream for key 'flume-debug/FlumeData.1405546776595.tmp' writing to tempfile '/tmp/hadoop-ec2-user/s3/output-3361552510304141410.tmp'
          2014-07-16 21:39:38,110 (hdfs-hdfs-1-call-runner-3) [INFO - org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsOutputStream.close(NativeS3FileSystem.java:217)] OutputStream for key 'flume-debug/FlumeData.1405546776595.tmp' closed. Now beginning upload
          2014-07-16 21:39:38,159 (hdfs-hdfs-1-call-runner-3) [INFO - org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsOutputStream.close(NativeS3FileSystem.java:229)] OutputStream for key 'flume-debug/FlumeData.1405546776595.tmp' upload complete
          2014-07-16 21:39:38,174 (hdfs-hdfs-1-call-runner-4) [INFO - org.apache.flume.sink.hdfs.BucketWriter$7.call(BucketWriter.java:487)] Renaming s3n://S3_BUCKET/flume-debug/FlumeData.1405546776595.tmp to s3n://S3_BUCKET/flume-debug/FlumeData.1405546776595
          

          Have you tried changing the jets3t version?
          Are we talking about the same problem? You make reference to 2 out of 3 nodes having problems, this should be a problem for all your nodes.

          Show
          Dennis Waldron added a comment - Hi Rob, I'm not discussing this topic purely because nobody has asked for additional help. I see that you are using CDH 4.4 which maps to Hadoop 2.0.0. As I don't use a full blown Hadoop installation I've made some effort to simulate your problem on an AWS EC2 node using the library versions defined by the CDH project ( http://maven-repository.com/artifact/com.cloudera.cdh/cdh-root/4.4.0/pom ) Using an unmodified Flume 1.4 installation with the following additional libraries: -rw-rw-r-- 1 ec2-user ec2-user 1756571 Jul 16 21:34 hadoop-core-2.0.0-mr1-cdh4.4.0.jar -rw-rw-r-- 1 ec2-user ec2-user 2284942 Jul 16 21:34 hadoop-common-2.0.0-cdh4.4.0.jar -rw-rw-r-- 1 ec2-user ec2-user 46855 Jul 16 21:34 hadoop-auth-2.0.0-cdh4.4.0.jar -rw-rw-r-- 1 ec2-user ec2-user 305001 Jul 16 21:34 commons-httpclient-3.1.jar -rw-rw-r-- 1 ec2-user ec2-user 298829 Jul 16 21:34 commons-configuration-1.6.jar -rw-rw-r-- 1 ec2-user ec2-user 321806 Jul 16 21:34 jets3t-0.6.1.jar I was able to reproduce the 404 problem: 2014-07-16 21:36:19,158 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:219)] Creating s3n://S3_BUCKET/flume-debug/FlumeData.1405546579015.tmp 2014-07-16 21:36:20,291 (hdfs-hdfs-1-call-runner-0) [WARN - org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:393)] Response '/flume-debug%2FFlumeData.1405546579015.tmp' - Unexpected response code 404, expected 200 2014-07-16 21:36:20,306 (hdfs-hdfs-1-call-runner-0) [WARN - org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:393)] Response '/flume-debug%2FFlumeData.1405546579015.tmp_%24folder%24' - Unexpected response code 404, expected 200 2014-07-16 21:36:20,336 (hdfs-hdfs-1-call-runner-0) [INFO - org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsOutputStream.<init>(NativeS3FileSystem.java:182)] OutputStream for key 'flume- 2014-07-16 21:36:20,374 (hdfs-hdfs-1-call-runner-3) [INFO - org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsOutputStream.close(NativeS3FileSystem.java:217)] OutputStream for key 'flume-debug/FlumeData.1405546579015.tmp' closed. Now beginning upload 2014-07-16 21:36:20,417 (hdfs-hdfs-1-call-runner-3) [INFO - org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsOutputStream.close(NativeS3FileSystem.java:229)] OutputStream for key 'flume-debug/FlumeData.1405546579015.tmp' upload complete 2014-07-16 21:36:20,435 (hdfs-hdfs-1-call-runner-4) [INFO - org.apache.flume.sink.hdfs.BucketWriter$7.call(BucketWriter.java:487)] Renaming s3n://S3_BUCKET/flume-debug/FlumeData.1405546579015.tmp to s3n://S3_BUCKET/flume-debug/FlumeData.1405546579015 Now by changing the jets3t library from version 0.6.1 to 0.7.1 (as per my previous recommendation): -rw-rw-r-- 1 ec2-user ec2-user 377780 Jul 16 21:40 jets3t-0.7.1.jar The problem is fixed: 2014-07-16 21:39:36,724 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:219)] Creating s3n://S3_BUCKET/flume-debug/FlumeData.1405546776595.tmp 2014-07-16 21:39:38,071 (hdfs-hdfs-1-call-runner-0) [INFO - org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsOutputStream.<init>(NativeS3FileSystem.java:182)] OutputStream for key 'flume-debug/FlumeData.1405546776595.tmp' writing to tempfile '/tmp/hadoop-ec2-user/s3/output-3361552510304141410.tmp' 2014-07-16 21:39:38,110 (hdfs-hdfs-1-call-runner-3) [INFO - org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsOutputStream.close(NativeS3FileSystem.java:217)] OutputStream for key 'flume-debug/FlumeData.1405546776595.tmp' closed. Now beginning upload 2014-07-16 21:39:38,159 (hdfs-hdfs-1-call-runner-3) [INFO - org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsOutputStream.close(NativeS3FileSystem.java:229)] OutputStream for key 'flume-debug/FlumeData.1405546776595.tmp' upload complete 2014-07-16 21:39:38,174 (hdfs-hdfs-1-call-runner-4) [INFO - org.apache.flume.sink.hdfs.BucketWriter$7.call(BucketWriter.java:487)] Renaming s3n://S3_BUCKET/flume-debug/FlumeData.1405546776595.tmp to s3n://S3_BUCKET/flume-debug/FlumeData.1405546776595 Have you tried changing the jets3t version? Are we talking about the same problem? You make reference to 2 out of 3 nodes having problems, this should be a problem for all your nodes.
          Hide
          Rob Slifka added a comment - - edited

          This bit us pretty hard today. We're on Flume 1.4 via CDH 4.4. Events began backing up on two of our three nodes with the following logs:

          16 Jul 2014 19:04:18,366 WARN  [hdfs-s3sink-beacons-cold-call-runner-8] (org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest:393)  - Response '/2014-07-16%2F1900%2Fnginx-ip-10-80-54-224.ec2.internal.1405537200166' - Unexpected response code 404, expected 200
          16 Jul 2014 19:04:18,377 WARN  [hdfs-s3sink-beacons-cold-call-runner-8] (org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest:393)  - Response '/2014-07-16%2F1900%2Fnginx-ip-10-80-54-224.ec2.internal.1405537200166_%24folder%24' - Unexpected response code 404, expected 200
          16 Jul 2014 19:04:18,508 WARN  [hdfs-s3sink-beacons-cold-call-runner-8] (org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest:393)  - Response '/2014-07-16%2F1900' - Unexpected response code 404, expected 200
          16 Jul 2014 19:04:18,518 WARN  [hdfs-s3sink-beacons-cold-call-runner-8] (org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest:393)  - Response '/2014-07-16%2F1900_%24folder%24' - Unexpected response code 404, expected 200
          16 Jul 2014 19:04:21,632 INFO  [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.sink.hdfs.BucketWriter.open:219)  - Creating s3n://ACCESS_KEY:SECRET@PRODUCTION_BUCKET/2014-07-16/1900/nginx-ip-10-80-54-224.ec2.internal.1405537200167.tmp
          16 Jul 2014 19:04:21,734 WARN  [hdfs-s3sink-beacons-cold-call-runner-0] (org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest:393)  - Response '/2014-07-16%2F1900%2Fnginx-ip-10-80-54-224.ec2.internal.1405537200167.tmp' - Unexpected response code 404, expected 200
          16 Jul 2014 19:04:21,745 WARN  [hdfs-s3sink-beacons-cold-call-runner-0] (org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest:393)  - Response '/2014-07-16%2F1900%2Fnginx-ip-10-80-54-224.ec2.internal.1405537200167.tmp_%24folder%24' - Unexpected response code 404, expected 200
          16 Jul 2014 19:04:21,761 INFO  [hdfs-s3sink-beacons-cold-call-runner-0] (org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsOutputStream.<init>:182)  - OutputStream for key '2014-07-16/1900/nginx-ip-10-80-54-224.ec2.internal.1405537200167.tmp' writing to tempfile '/tmp/hadoop-root/s3/output-2622317072117705911.tmp'
          16 Jul 2014 19:04:21,761 INFO  [hdfs-s3sink-beacons-cold-call-runner-0] (org.apache.flume.sink.hdfs.AbstractHDFSWriter.reflectGetNumCurrentReplicas:185)  - FileSystem's output stream doesn't support getNumCurrentReplicas; --HDFS-826 not available; fsOut=org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsOutputStream; err=java.lang.NoSuchMethodException: org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsOutputStream.getNumCurrentReplicas()
          16 Jul 2014 19:04:39,254 INFO  [Log-BackgroundWorker-fileChannel] (org.apache.flume.channel.file.EventQueueBackingStoreFile.beginCheckpoint:214)  - Start checkpoint for /mnt/flume/checkpoint/checkpoint, elements to sync = 41091
          16 Jul 2014 19:04:39,261 INFO  [Log-BackgroundWorker-fileChannel] (org.apache.flume.channel.file.EventQueueBackingStoreFile.checkpoint:239)  - Updating checkpoint metadata: logWriteOrderID: 1408437260518, queueSize: 600, queueHead: 705519
          16 Jul 2014 19:04:39,272 INFO  [Log-BackgroundWorker-fileChannel] (org.apache.flume.channel.file.Log.writeCheckpoint:1019)  - Updated checkpoint for file: /mnt/flume/data/log-648 position: 1599320032 logWriteOrderID: 1408437260518
          16 Jul 2014 19:04:39,347 INFO  [Log-BackgroundWorker-hotFileChannel] (org.apache.flume.channel.file.EventQueueBackingStoreFile.beginCheckpoint:214)  - Start checkpoint for /mnt/flume/checkpoint-hot/checkpoint, elements to sync = 40628
          16 Jul 2014 19:04:39,358 INFO  [Log-BackgroundWorker-hotFileChannel] (org.apache.flume.channel.file.EventQueueBackingStoreFile.checkpoint:239)  - Updating checkpoint metadata: logWriteOrderID: 1408437261136, queueSize: 999972, queueHead: 783009
          16 Jul 2014 19:04:39,368 INFO  [Log-BackgroundWorker-hotFileChannel] (org.apache.flume.channel.file.Log.writeCheckpoint:1019)  - Updated checkpoint for file: /mnt/flume/data-hot/log-668 position: 1621862329 logWriteOrderID: 1408437261136
          16 Jul 2014 19:04:39,988 INFO  [pool-5-thread-1] (org.apache.flume.channel.file.Log.roll:944)  - Roll start /mnt/flume/data-hot
          16 Jul 2014 19:04:39,989 INFO  [pool-5-thread-1] (org.apache.flume.channel.file.LogFile$Writer.<init>:188)  - Opened /mnt/flume/data-hot/log-669
          16 Jul 2014 19:04:39,993 INFO  [pool-5-thread-1] (org.apache.flume.channel.file.LogFile$Writer.close:327)  - Closing /mnt/flume/data-hot/log-668
          16 Jul 2014 19:04:39,993 INFO  [pool-5-thread-1] (org.apache.flume.channel.file.Log.roll:959)  - Roll end
          16 Jul 2014 19:04:52,026 INFO  [pool-5-thread-1] (org.apache.flume.channel.file.Log.roll:944)  - Roll start /mnt/flume/data
          16 Jul 2014 19:04:52,026 INFO  [pool-5-thread-1] (org.apache.flume.channel.file.LogFile$Writer.<init>:188)  - Opened /mnt/flume/data/log-649
          16 Jul 2014 19:04:52,030 INFO  [pool-5-thread-1] (org.apache.flume.channel.file.LogFile$Writer.close:327)  - Closing /mnt/flume/data/log-648
          16 Jul 2014 19:04:52,030 INFO  [pool-5-thread-1] (org.apache.flume.channel.file.Log.roll:959)  - Roll end
          16 Jul 2014 19:04:57,917 INFO  [hdfs-s3sink-beacons-cold-call-runner-0] (org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsOutputStream.close:217)  - OutputStream for key '2014-07-16/1900/nginx-ip-10-80-54-224.ec2.internal.1405537200167.tmp' closed. Now beginning upload
          

          When we restarted Flume, things began draining (slowly) though we're still seeing these errors. If a restart solves it or if folks like Dennis have figured it out some other way, I suspect they may not be discussing it. A definitive position or fix would be awesome! We're happy to help if you need something verified.

          Show
          Rob Slifka added a comment - - edited This bit us pretty hard today. We're on Flume 1.4 via CDH 4.4. Events began backing up on two of our three nodes with the following logs: 16 Jul 2014 19:04:18,366 WARN [hdfs-s3sink-beacons-cold-call-runner-8] (org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest:393) - Response '/2014-07-16%2F1900%2Fnginx-ip-10-80-54-224.ec2.internal.1405537200166' - Unexpected response code 404, expected 200 16 Jul 2014 19:04:18,377 WARN [hdfs-s3sink-beacons-cold-call-runner-8] (org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest:393) - Response '/2014-07-16%2F1900%2Fnginx-ip-10-80-54-224.ec2.internal.1405537200166_%24folder%24' - Unexpected response code 404, expected 200 16 Jul 2014 19:04:18,508 WARN [hdfs-s3sink-beacons-cold-call-runner-8] (org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest:393) - Response '/2014-07-16%2F1900' - Unexpected response code 404, expected 200 16 Jul 2014 19:04:18,518 WARN [hdfs-s3sink-beacons-cold-call-runner-8] (org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest:393) - Response '/2014-07-16%2F1900_%24folder%24' - Unexpected response code 404, expected 200 16 Jul 2014 19:04:21,632 INFO [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.sink.hdfs.BucketWriter.open:219) - Creating s3n://ACCESS_KEY:SECRET@PRODUCTION_BUCKET/2014-07-16/1900/nginx-ip-10-80-54-224.ec2.internal.1405537200167.tmp 16 Jul 2014 19:04:21,734 WARN [hdfs-s3sink-beacons-cold-call-runner-0] (org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest:393) - Response '/2014-07-16%2F1900%2Fnginx-ip-10-80-54-224.ec2.internal.1405537200167.tmp' - Unexpected response code 404, expected 200 16 Jul 2014 19:04:21,745 WARN [hdfs-s3sink-beacons-cold-call-runner-0] (org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest:393) - Response '/2014-07-16%2F1900%2Fnginx-ip-10-80-54-224.ec2.internal.1405537200167.tmp_%24folder%24' - Unexpected response code 404, expected 200 16 Jul 2014 19:04:21,761 INFO [hdfs-s3sink-beacons-cold-call-runner-0] (org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsOutputStream.<init>:182) - OutputStream for key '2014-07-16/1900/nginx-ip-10-80-54-224.ec2.internal.1405537200167.tmp' writing to tempfile '/tmp/hadoop-root/s3/output-2622317072117705911.tmp' 16 Jul 2014 19:04:21,761 INFO [hdfs-s3sink-beacons-cold-call-runner-0] (org.apache.flume.sink.hdfs.AbstractHDFSWriter.reflectGetNumCurrentReplicas:185) - FileSystem's output stream doesn't support getNumCurrentReplicas; --HDFS-826 not available; fsOut=org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsOutputStream; err=java.lang.NoSuchMethodException: org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsOutputStream.getNumCurrentReplicas() 16 Jul 2014 19:04:39,254 INFO [Log-BackgroundWorker-fileChannel] (org.apache.flume.channel.file.EventQueueBackingStoreFile.beginCheckpoint:214) - Start checkpoint for /mnt/flume/checkpoint/checkpoint, elements to sync = 41091 16 Jul 2014 19:04:39,261 INFO [Log-BackgroundWorker-fileChannel] (org.apache.flume.channel.file.EventQueueBackingStoreFile.checkpoint:239) - Updating checkpoint metadata: logWriteOrderID: 1408437260518, queueSize: 600, queueHead: 705519 16 Jul 2014 19:04:39,272 INFO [Log-BackgroundWorker-fileChannel] (org.apache.flume.channel.file.Log.writeCheckpoint:1019) - Updated checkpoint for file: /mnt/flume/data/log-648 position: 1599320032 logWriteOrderID: 1408437260518 16 Jul 2014 19:04:39,347 INFO [Log-BackgroundWorker-hotFileChannel] (org.apache.flume.channel.file.EventQueueBackingStoreFile.beginCheckpoint:214) - Start checkpoint for /mnt/flume/checkpoint-hot/checkpoint, elements to sync = 40628 16 Jul 2014 19:04:39,358 INFO [Log-BackgroundWorker-hotFileChannel] (org.apache.flume.channel.file.EventQueueBackingStoreFile.checkpoint:239) - Updating checkpoint metadata: logWriteOrderID: 1408437261136, queueSize: 999972, queueHead: 783009 16 Jul 2014 19:04:39,368 INFO [Log-BackgroundWorker-hotFileChannel] (org.apache.flume.channel.file.Log.writeCheckpoint:1019) - Updated checkpoint for file: /mnt/flume/data-hot/log-668 position: 1621862329 logWriteOrderID: 1408437261136 16 Jul 2014 19:04:39,988 INFO [pool-5-thread-1] (org.apache.flume.channel.file.Log.roll:944) - Roll start /mnt/flume/data-hot 16 Jul 2014 19:04:39,989 INFO [pool-5-thread-1] (org.apache.flume.channel.file.LogFile$Writer.<init>:188) - Opened /mnt/flume/data-hot/log-669 16 Jul 2014 19:04:39,993 INFO [pool-5-thread-1] (org.apache.flume.channel.file.LogFile$Writer.close:327) - Closing /mnt/flume/data-hot/log-668 16 Jul 2014 19:04:39,993 INFO [pool-5-thread-1] (org.apache.flume.channel.file.Log.roll:959) - Roll end 16 Jul 2014 19:04:52,026 INFO [pool-5-thread-1] (org.apache.flume.channel.file.Log.roll:944) - Roll start /mnt/flume/data 16 Jul 2014 19:04:52,026 INFO [pool-5-thread-1] (org.apache.flume.channel.file.LogFile$Writer.<init>:188) - Opened /mnt/flume/data/log-649 16 Jul 2014 19:04:52,030 INFO [pool-5-thread-1] (org.apache.flume.channel.file.LogFile$Writer.close:327) - Closing /mnt/flume/data/log-648 16 Jul 2014 19:04:52,030 INFO [pool-5-thread-1] (org.apache.flume.channel.file.Log.roll:959) - Roll end 16 Jul 2014 19:04:57,917 INFO [hdfs-s3sink-beacons-cold-call-runner-0] (org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsOutputStream.close:217) - OutputStream for key '2014-07-16/1900/nginx-ip-10-80-54-224.ec2.internal.1405537200167.tmp' closed. Now beginning upload When we restarted Flume, things began draining (slowly) though we're still seeing these errors. If a restart solves it or if folks like Dennis have figured it out some other way, I suspect they may not be discussing it. A definitive position or fix would be awesome! We're happy to help if you need something verified.
          Hide
          Ashish Paliwal added a comment -

          Viksit Gaur Can you please try with jets3t 0.7.1 version as suggested by Dennis and let us know if it works. I am a bit tied up coz of my transition, and might take a while to get back on this issue.

          Show
          Ashish Paliwal added a comment - Viksit Gaur Can you please try with jets3t 0.7.1 version as suggested by Dennis and let us know if it works. I am a bit tied up coz of my transition, and might take a while to get back on this issue.
          Hide
          Ashish Paliwal added a comment -

          Thanks Dennis ! I did tried to download 0.7.1, but the site returned 404.

          The test stub that I wrote for append on S3, is writing data and I can see on the same from AWS console. Let me give a try to with new version.

          Show
          Ashish Paliwal added a comment - Thanks Dennis ! I did tried to download 0.7.1, but the site returned 404. The test stub that I wrote for append on S3, is writing data and I can see on the same from AWS console. Let me give a try to with new version.
          Hide
          Dennis Waldron added a comment -

          I've come across this problem before, the solution was to use a Hadoop version with jets3t 0.7.1...

          In our Flume infrastructure at work we use Flume NG 1.4.0 (agents hosted on EC2 with HDFS sink writing to S3) supplemented with the following libraries:

          -rw-r--r-- 1 root root   298829 Aug 19 14:27 commons-configuration-1.6.jar
          -rw-r--r-- 1 root root   279781 Aug 19 14:27 commons-httpclient-3.0.1.jar
          -rw-r--r-- 1 root root  3929148 Aug 19 14:27 hadoop-core-1.0.4.jar
          -rw-r--r-- 1 root root   377780 Aug 19 14:27 jets3t-0.7.1.jar
          

          We've also included hadoop-gpl-compression-0.2.0-dev.jar and the Hadoop native extensions (libgplcompression.so, libhadoop.so and libsnappy.so) to support compression. All libraries can be downloaded from http://mvnrepository.com/artifact/org.apache.hadoop/hadoop-core/1.0.4 excluding hadoop-gpl-compression which can be compiled from source (https://code.google.com/a/apache-extras.org/p/hadoop-gpl-compression/)

          Show
          Dennis Waldron added a comment - I've come across this problem before, the solution was to use a Hadoop version with jets3t 0.7.1... In our Flume infrastructure at work we use Flume NG 1.4.0 (agents hosted on EC2 with HDFS sink writing to S3) supplemented with the following libraries: -rw-r--r-- 1 root root 298829 Aug 19 14:27 commons-configuration-1.6.jar -rw-r--r-- 1 root root 279781 Aug 19 14:27 commons-httpclient-3.0.1.jar -rw-r--r-- 1 root root 3929148 Aug 19 14:27 hadoop-core-1.0.4.jar -rw-r--r-- 1 root root 377780 Aug 19 14:27 jets3t-0.7.1.jar We've also included hadoop-gpl-compression-0.2.0-dev.jar and the Hadoop native extensions (libgplcompression.so, libhadoop.so and libsnappy.so) to support compression. All libraries can be downloaded from http://mvnrepository.com/artifact/org.apache.hadoop/hadoop-core/1.0.4 excluding hadoop-gpl-compression which can be compiled from source ( https://code.google.com/a/apache-extras.org/p/hadoop-gpl-compression/ )
          Hide
          Ashish Paliwal added a comment -

          The error was reproduced using the Agent configured. Created a test case to debug the condition further. The test case works fine and I am able to see the data in my S3 bucket. The test case is copy of doTestTextBatchAppend(), with following changes in context (this is with trunk code)

          context.put("hdfs.path", "s3://ID:KEY@bucketname/events/");
          context.put("hdfs.batchSize", "5");
          context.put("hdfs.callTimeout", "50000");
          

          Will try to dig deeper. Anyone got suggestions on this?

          Show
          Ashish Paliwal added a comment - The error was reproduced using the Agent configured. Created a test case to debug the condition further. The test case works fine and I am able to see the data in my S3 bucket. The test case is copy of doTestTextBatchAppend(), with following changes in context (this is with trunk code) context.put( "hdfs.path" , "s3: //ID:KEY@bucketname/events/" ); context.put( "hdfs.batchSize" , "5" ); context.put( "hdfs.callTimeout" , "50000" ); Will try to dig deeper. Anyone got suggestions on this?
          Hide
          Ashish Paliwal added a comment -

          Have reproduced the error, will debug it whenever free cycles are available.

          Show
          Ashish Paliwal added a comment - Have reproduced the error, will debug it whenever free cycles are available.
          Hide
          Viksit Gaur added a comment -

          Thank you!

          Show
          Viksit Gaur added a comment - Thank you!
          Hide
          Ashish Paliwal added a comment -

          I have not used it, but let me give it a shot. Shall revert back in a day or two. If you don't hear from me, please feel free to ping

          Show
          Ashish Paliwal added a comment - I have not used it, but let me give it a shot. Shall revert back in a day or two. If you don't hear from me, please feel free to ping
          Hide
          Viksit Gaur added a comment -

          Let me also add that using the latest hadoop versions with jets3t 0.9 doesn't work - looks like the S3 implementation is probably relying on a different API (older).

          Show
          Viksit Gaur added a comment - Let me also add that using the latest hadoop versions with jets3t 0.9 doesn't work - looks like the S3 implementation is probably relying on a different API (older).
          Hide
          Viksit Gaur added a comment -

          flume 1.4.0-bin

          Hadoop versions
          rw-rw-r- 1 ubuntu ubuntu 279781 Dec 17 22:24 commons-httpclient-3.0.1.jar
          rw-rw-r- 1 ubuntu ubuntu 58160 Dec 17 22:24 commons-codec-1.4.jar
          rw-rw-r- 1 ubuntu ubuntu 321806 Dec 17 22:24 jets3t-0.6.1.jar
          rw-rw-r- 1 ubuntu ubuntu 4203147 Dec 17 22:24 hadoop-core-1.2.1.jar

          Show
          Viksit Gaur added a comment - flume 1.4.0-bin Hadoop versions rw-rw-r - 1 ubuntu ubuntu 279781 Dec 17 22:24 commons-httpclient-3.0.1.jar rw-rw-r - 1 ubuntu ubuntu 58160 Dec 17 22:24 commons-codec-1.4.jar rw-rw-r - 1 ubuntu ubuntu 321806 Dec 17 22:24 jets3t-0.6.1.jar rw-rw-r - 1 ubuntu ubuntu 4203147 Dec 17 22:24 hadoop-core-1.2.1.jar
          Show
          Viksit Gaur added a comment - Ping: Tom White Gopinathan A Ashish Paliwal Brock Noland
          Hide
          Viksit Gaur added a comment -

          Are there any updates on this?

          Flume is at 1.4, but it looks like no one's really using it to write to S3 anymore? Are we to assume that this functionality is now deprecated and won't be maintained?

          Show
          Viksit Gaur added a comment - Are there any updates on this? Flume is at 1.4, but it looks like no one's really using it to write to S3 anymore? Are we to assume that this functionality is now deprecated and won't be maintained?

            People

            • Assignee:
              Ashish Paliwal
              Reporter:
              Prashanth Jonnalagadda
            • Votes:
              4 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:

                Development