Flume
  1. Flume
  2. FLUME-1487

FileChannel format needs to be extensible

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: v1.3.0
    • Fix Version/s: v1.3.0
    • Component/s: Channel
    • Labels:
      None

      Description

      That is we should be able to add fields without causing users grief.

      1. FLUME-1487-1.patch
        150 kB
        Brock Noland
      2. FLUME-1487-4.patch
        163 kB
        Brock Noland
      3. FLUME-1487-5.patch
        172 kB
        Brock Noland
      4. FLUME-1487-6.patch
        173 kB
        Brock Noland
      5. FLUME-1487-8.patch
        180 kB
        Brock Noland
      6. FLUME-1487-9.patch
        183 kB
        Brock Noland
      7. FLUME-1487-10.patch
        183 kB
        Brock Noland
      8. FLUME-1487-11.patch
        183 kB
        Brock Noland
      9. FLUME-1487-12.patch
        185 kB
        Brock Noland

        Issue Links

          Activity

          Hide
          Brock Noland added a comment -

          This issue should resolve FLUME-1285 as well since we could stop using hadoop writeables.

          Show
          Brock Noland added a comment - This issue should resolve FLUME-1285 as well since we could stop using hadoop writeables.
          Hide
          Brock Noland added a comment -

          Marking patch available as a patch is on RB.

          Show
          Brock Noland added a comment - Marking patch available as a patch is on RB.
          Hide
          Brock Noland added a comment -

          Attached patch from RB.

          Show
          Brock Noland added a comment - Attached patch from RB.
          Hide
          Brock Noland added a comment -

          Latest patch from RB

          Show
          Brock Noland added a comment - Latest patch from RB
          Hide
          Brock Noland added a comment -

          latest from RB

          Show
          Brock Noland added a comment - latest from RB
          Hide
          Brock Noland added a comment -

          patch from RB

          Show
          Brock Noland added a comment - patch from RB
          Hide
          Brock Noland added a comment -

          Latest patch, merged with trunk.

          Show
          Brock Noland added a comment - Latest patch, merged with trunk.
          Hide
          Brock Noland added a comment -

          patch from RB

          Show
          Brock Noland added a comment - patch from RB
          Hide
          Brock Noland added a comment -

          patch from RB

          Show
          Brock Noland added a comment - patch from RB
          Hide
          Brock Noland added a comment -

          patch from RB (removes two pieces of ws)

          Show
          Brock Noland added a comment - patch from RB (removes two pieces of ws)
          Hide
          Brock Noland added a comment -

          latest from RB

          Show
          Brock Noland added a comment - latest from RB
          Hide
          Hari Shreedharan added a comment -

          Patch committed, rev: 680c2d7cd25d6b58d6d4b1a403bcc5ee196c568b. Thanks Brock! Excellent work!

          Show
          Hari Shreedharan added a comment - Patch committed, rev: 680c2d7cd25d6b58d6d4b1a403bcc5ee196c568b. Thanks Brock! Excellent work!
          Hide
          Hudson added a comment -

          Integrated in flume-1.3.0 #8 (See https://builds.apache.org/job/flume-1.3.0/8/)
          FLUME-1487. FileChannel format needs to be extensible. (Revision 810dfe28262eaa882cdc9a251d4978196ac333ea)

          Result = FAILURE
          hshreedharan : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git;a=summary&a=commit&h=810dfe28262eaa882cdc9a251d4978196ac333ea
          Files :

          • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventPointer.java
          • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/EventQueueBackingStoreFactory.java
          • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFileV2.java
          • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Rollback.java
          • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Serialization.java
          • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java
          • flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestEventQueueBackingStoreFactory.java
          • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java
          • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java
          • flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java
          • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogUtils.java
          • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFileV3.java
          • flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecordV2.java
          • flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java
          • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/EventQueueBackingStoreFileV2.java
          • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/EventQueueBackingStoreFile.java
          • flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java
          • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java
          • flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecord.java
          • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java
          • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Take.java
          • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFileFactory.java
          • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/EventQueueBackingStoreFileV3.java
          • flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java
          • flume-ng-channels/flume-file-channel/pom.xml
          • flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java
          • flume-ng-channels/flume-file-channel/src/main/proto/filechannel.proto
          • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Commit.java
          • pom.xml
          • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/CheckpointRebuilder.java
          • flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecordV3.java
          • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java
          • flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogRecord.java
          • flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java
          • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/EventQueueBackingStore.java
          Show
          Hudson added a comment - Integrated in flume-1.3.0 #8 (See https://builds.apache.org/job/flume-1.3.0/8/ ) FLUME-1487 . FileChannel format needs to be extensible. (Revision 810dfe28262eaa882cdc9a251d4978196ac333ea) Result = FAILURE hshreedharan : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git;a=summary&a=commit&h=810dfe28262eaa882cdc9a251d4978196ac333ea Files : flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventPointer.java flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/EventQueueBackingStoreFactory.java flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFileV2.java flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Rollback.java flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Serialization.java flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestEventQueueBackingStoreFactory.java flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogUtils.java flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFileV3.java flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecordV2.java flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/EventQueueBackingStoreFileV2.java flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/EventQueueBackingStoreFile.java flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecord.java flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Take.java flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFileFactory.java flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/EventQueueBackingStoreFileV3.java flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java flume-ng-channels/flume-file-channel/pom.xml flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java flume-ng-channels/flume-file-channel/src/main/proto/filechannel.proto flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Commit.java pom.xml flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/CheckpointRebuilder.java flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecordV3.java flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogRecord.java flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/EventQueueBackingStore.java
          Hide
          Hudson added a comment -

          Integrated in flume-trunk #294 (See https://builds.apache.org/job/flume-trunk/294/)
          FLUME-1487. FileChannel format needs to be extensible. (Revision 680c2d7cd25d6b58d6d4b1a403bcc5ee196c568b)

          Result = FAILURE
          hshreedharan : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git;a=summary&a=commit&h=680c2d7cd25d6b58d6d4b1a403bcc5ee196c568b
          Files :

          • flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java
          • flume-ng-channels/flume-file-channel/pom.xml
          • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFileFactory.java
          • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Serialization.java
          • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/CheckpointRebuilder.java
          • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogUtils.java
          • flume-ng-channels/flume-file-channel/src/main/proto/filechannel.proto
          • flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecord.java
          • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java
          • flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestEventQueueBackingStoreFactory.java
          • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventPointer.java
          • pom.xml
          • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Commit.java
          • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFileV2.java
          • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java
          • flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecordV2.java
          • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Take.java
          • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Rollback.java
          • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/EventQueueBackingStoreFileV3.java
          • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/EventQueueBackingStoreFileV2.java
          • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java
          • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java
          • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/EventQueueBackingStoreFile.java
          • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java
          • flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java
          • flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogRecord.java
          • flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecordV3.java
          • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFileV3.java
          • flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java
          • flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java
          • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java
          • flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java
          • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/EventQueueBackingStoreFactory.java
          • flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java
          • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/EventQueueBackingStore.java
          Show
          Hudson added a comment - Integrated in flume-trunk #294 (See https://builds.apache.org/job/flume-trunk/294/ ) FLUME-1487 . FileChannel format needs to be extensible. (Revision 680c2d7cd25d6b58d6d4b1a403bcc5ee196c568b) Result = FAILURE hshreedharan : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git;a=summary&a=commit&h=680c2d7cd25d6b58d6d4b1a403bcc5ee196c568b Files : flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java flume-ng-channels/flume-file-channel/pom.xml flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFileFactory.java flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Serialization.java flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/CheckpointRebuilder.java flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogUtils.java flume-ng-channels/flume-file-channel/src/main/proto/filechannel.proto flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecord.java flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestEventQueueBackingStoreFactory.java flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventPointer.java pom.xml flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Commit.java flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFileV2.java flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecordV2.java flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Take.java flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Rollback.java flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/EventQueueBackingStoreFileV3.java flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/EventQueueBackingStoreFileV2.java flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/EventQueueBackingStoreFile.java flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogRecord.java flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecordV3.java flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFileV3.java flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/EventQueueBackingStoreFactory.java flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/EventQueueBackingStore.java

            People

            • Assignee:
              Brock Noland
              Reporter:
              Brock Noland
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development