Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2469

DatasetSink should load dataset when needed, not at startup

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.5.0
    • Fix Version/s: 1.6.0
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      The DatasetSink currently loads the target dataset when it starts and fails to initialize if it can't be loaded. This causes minor headaches because it requires an order when setting up a data flow (create the dataset, configure the sink, start flume). The sink should wait until records are sent to load the dataset.

        Activity

        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in flume-trunk #662 (See https://builds.apache.org/job/flume-trunk/662/)
        FLUME-2469. DatasetSink should load dataset when needed, not at startup. (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=ef2b089ab70e04e67b70c23416ef7051e992122b)

        • flume-ng-sinks/flume-dataset-sink/src/main/java/org/apache/flume/sink/kite/DatasetSink.java
        • flume-ng-sinks/flume-dataset-sink/src/test/java/org/apache/flume/sink/kite/TestDatasetSink.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in flume-trunk #662 (See https://builds.apache.org/job/flume-trunk/662/ ) FLUME-2469 . DatasetSink should load dataset when needed, not at startup. (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=ef2b089ab70e04e67b70c23416ef7051e992122b ) flume-ng-sinks/flume-dataset-sink/src/main/java/org/apache/flume/sink/kite/DatasetSink.java flume-ng-sinks/flume-dataset-sink/src/test/java/org/apache/flume/sink/kite/TestDatasetSink.java
        Hide
        hudson Hudson added a comment -

        UNSTABLE: Integrated in Flume-trunk-hbase-98 #22 (See https://builds.apache.org/job/Flume-trunk-hbase-98/22/)
        FLUME-2469. DatasetSink should load dataset when needed, not at startup. (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=ef2b089ab70e04e67b70c23416ef7051e992122b)

        • flume-ng-sinks/flume-dataset-sink/src/test/java/org/apache/flume/sink/kite/TestDatasetSink.java
        • flume-ng-sinks/flume-dataset-sink/src/main/java/org/apache/flume/sink/kite/DatasetSink.java
        Show
        hudson Hudson added a comment - UNSTABLE: Integrated in Flume-trunk-hbase-98 #22 (See https://builds.apache.org/job/Flume-trunk-hbase-98/22/ ) FLUME-2469 . DatasetSink should load dataset when needed, not at startup. (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=ef2b089ab70e04e67b70c23416ef7051e992122b ) flume-ng-sinks/flume-dataset-sink/src/test/java/org/apache/flume/sink/kite/TestDatasetSink.java flume-ng-sinks/flume-dataset-sink/src/main/java/org/apache/flume/sink/kite/DatasetSink.java
        Hide
        hshreedharan Hari Shreedharan added a comment -

        I did remove some unused imports and make some whitespace changes before committing.

        Show
        hshreedharan Hari Shreedharan added a comment - I did remove some unused imports and make some whitespace changes before committing.
        Hide
        hshreedharan Hari Shreedharan added a comment -

        Committed! Thanks Ryan!

        Show
        hshreedharan Hari Shreedharan added a comment - Committed! Thanks Ryan!
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 709534a6514b5fe86890058ecaf94a8b6a6ff44a in flume's branch refs/heads/flume-1.6 from Hari Shreedharan
        [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=709534a ]

        FLUME-2469. DatasetSink should load dataset when needed, not at startup.

        (Ryan Blue via Hari)

        Show
        jira-bot ASF subversion and git services added a comment - Commit 709534a6514b5fe86890058ecaf94a8b6a6ff44a in flume's branch refs/heads/flume-1.6 from Hari Shreedharan [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=709534a ] FLUME-2469 . DatasetSink should load dataset when needed, not at startup. (Ryan Blue via Hari)
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit ef2b089ab70e04e67b70c23416ef7051e992122b in flume's branch refs/heads/trunk from Hari Shreedharan
        [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=ef2b089 ]

        FLUME-2469. DatasetSink should load dataset when needed, not at startup.

        (Ryan Blue via Hari)

        Show
        jira-bot ASF subversion and git services added a comment - Commit ef2b089ab70e04e67b70c23416ef7051e992122b in flume's branch refs/heads/trunk from Hari Shreedharan [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=ef2b089 ] FLUME-2469 . DatasetSink should load dataset when needed, not at startup. (Ryan Blue via Hari)
        Hide
        hshreedharan Hari Shreedharan added a comment -

        +1. Running tests and committing.

        Show
        hshreedharan Hari Shreedharan added a comment - +1. Running tests and committing.
        Hide
        fwiffo Joey Echeverria added a comment -

        +1 (non-binding)

        Show
        fwiffo Joey Echeverria added a comment - +1 (non-binding)
        Hide
        rdblue Ryan Blue added a comment -

        Fixed in the attached patch. Thanks!

        Show
        rdblue Ryan Blue added a comment - Fixed in the attached patch. Thanks!

          People

          • Assignee:
            rdblue Ryan Blue
            Reporter:
            rdblue Ryan Blue
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development