Flume
  1. Flume
  2. FLUME-582

Multiple diskfailover cannot cohabit in the same logical node configuration

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Won't Fix
    • Affects Version/s: v0.9.3
    • Fix Version/s: v0.9.5
    • Component/s: None
    • Labels:
      None
    • Environment:

      Any

      Description

      DiskFailoverDeco relies on a DiskFailoverManager which is associated with the logical node.

      When using configurations such as fanouts with diskfailover and/or agentDFOChain, the multiple instances of DiskFailoverDeco use a single DiskFailoverManager and this causes trouble in the form of IllegalStateExceptions being thrown.

      The same problem might arise with the ackedWriteAhead even though it has provision for forcing the logical node name, this is not used in the configuration generated by agentE2EChain.

      1. ASF.LICENSE.NOT.GRANTED--FLUME-582.patch
        5 kB
        Mathias Herberts
      2. ASF.LICENSE.NOT.GRANTED--FLUME-582-2.patch
        7 kB
        Mathias Herberts
      3. ASF.LICENSE.NOT.GRANTED--FLUME-582-3.patch
        3 kB
        Mathias Herberts

        Activity

        Mathias Herberts created issue -
        Mathias Herberts made changes -
        Field Original Value New Value
        Status Open [ 1 ] Patch Available [ 10000 ]
        Mathias Herberts made changes -
        Status Patch Available [ 10000 ] Open [ 1 ]
        Hide
        Mathias Herberts added a comment -

        The FLUME-582.patch modifies the 'diskfailover' decorator and 'agentDFOChain' so they accept as first argument a subdir suffix that will be added to the logical name node when creating the directory in which the dfo_* dirs will be created.

        In the case of 'agentDFOChain', the provided suffix is propagated to 'diskfailover'.

        This patch does not address the similar problem that might be present in ackedWriteAhead.

        Show
        Mathias Herberts added a comment - The FLUME-582 .patch modifies the 'diskfailover' decorator and 'agentDFOChain' so they accept as first argument a subdir suffix that will be added to the logical name node when creating the directory in which the dfo_* dirs will be created. In the case of 'agentDFOChain', the provided suffix is propagated to 'diskfailover'. This patch does not address the similar problem that might be present in ackedWriteAhead.
        Mathias Herberts made changes -
        Attachment FLUME-582.patch [ 10597 ]
        Hide
        Mathias Herberts added a comment -

        This second version of the patch addresses the issue for agentE2EChain also.

        It also corrects an error in the previous version of the patch (which was incorrectly generated by diffing files from 0.9.3 with files from trunk).

        The patch introduces 'dir_suffix' as first parameter of both agentDFOChain and agentE2EChain. This 'dir_suffix' is appended to the logical node name when creating the target dir for DFO/WAL log files. The 'dir_suffix' is propagated in the translated specs.

        The 'diskfailover' decorator is also modified this way.

        The 'ackedWriteAhead' decorator has its semantics slightly changed. The 'walnode' second arg has been replaced by a 'dir_suffix' first arg which has the semantics explained above. It is therefore no longer possible to SET the 'logical node' associated with a WAL manager, it is only possible to add a suffix to the logical node name.

        Show
        Mathias Herberts added a comment - This second version of the patch addresses the issue for agentE2EChain also. It also corrects an error in the previous version of the patch (which was incorrectly generated by diffing files from 0.9.3 with files from trunk). The patch introduces 'dir_suffix' as first parameter of both agentDFOChain and agentE2EChain. This 'dir_suffix' is appended to the logical node name when creating the target dir for DFO/WAL log files. The 'dir_suffix' is propagated in the translated specs. The 'diskfailover' decorator is also modified this way. The 'ackedWriteAhead' decorator has its semantics slightly changed. The 'walnode' second arg has been replaced by a 'dir_suffix' first arg which has the semantics explained above. It is therefore no longer possible to SET the 'logical node' associated with a WAL manager, it is only possible to add a suffix to the logical node name.
        Mathias Herberts made changes -
        Attachment FLUME-582-2.patch [ 10598 ]
        Hide
        Mathias Herberts added a comment -

        Version 3 of the patch which corrects a bug in DiskFailoverDeco when interpreting numeric params.

        This patch no longer modifies agentDFOChain.

        For the rare cases when there are multiple agentDFOChain/agentE2EChain, their translation can be used directly (thus specifying the DFO/WAL dir suffix).

        Show
        Mathias Herberts added a comment - Version 3 of the patch which corrects a bug in DiskFailoverDeco when interpreting numeric params. This patch no longer modifies agentDFOChain. For the rare cases when there are multiple agentDFOChain/agentE2EChain, their translation can be used directly (thus specifying the DFO/WAL dir suffix).
        Mathias Herberts made changes -
        Attachment FLUME-582-3.patch [ 10599 ]
        Hide
        Jonathan Hsieh added a comment -

        Mathias,

        Can you check this out and follow up https://github.com/cloudera/flume/wiki/HowToContribute so we can proceed? Ideally we'd like the rights grant in the comments and then posting the patch on reviewboard to make reviewing easier.

        Thanks,
        Jon.

        Show
        Jonathan Hsieh added a comment - Mathias, Can you check this out and follow up https://github.com/cloudera/flume/wiki/HowToContribute so we can proceed? Ideally we'd like the rights grant in the comments and then posting the patch on reviewboard to make reviewing easier. Thanks, Jon.
        Jonathan Hsieh made changes -
        Status Open [ 1 ] Patch Available [ 10000 ]
        Hide
        Jonathan Hsieh added a comment -

        Hi Mathias,

        I've created a version of the patch and put it up for review here:

        https://review.cloudera.org/r/1775/diff/#index_header

        I've testing it now against the old tests (may break some code, but not sure). From a quick glance I'm going to say it will need some unit tests to be added.

        Jon.

        Show
        Jonathan Hsieh added a comment - Hi Mathias, I've created a version of the patch and put it up for review here: https://review.cloudera.org/r/1775/diff/#index_header I've testing it now against the old tests (may break some code, but not sure). From a quick glance I'm going to say it will need some unit tests to be added. Jon.
        Jonathan Hsieh made changes -
        Assignee Mathias Herberts [ herberts ]
        Mark Thomas made changes -
        Project Import Tue Aug 02 16:57:12 UTC 2011 [ 1312304232406 ]
        Hide
        Ashish Paliwal added a comment -

        Won't fix. 0.X branch not maintained anymore

        Show
        Ashish Paliwal added a comment - Won't fix. 0.X branch not maintained anymore
        Ashish Paliwal made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Fix Version/s v0.9.5 [ 12317558 ]
        Resolution Won't Fix [ 2 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Blocked Blocked Open Open
        14s 1 Mathias Herberts 26/Mar/11 08:10
        Open Open Blocked Blocked
        59d 12h 11m 2 Jonathan Hsieh 24/May/11 21:21
        Patch Available Patch Available Resolved Resolved
        1260d 13h 47m 1 Ashish Paliwal 05/Nov/14 10:08

          People

          • Assignee:
            Mathias Herberts
            Reporter:
            Mathias Herberts
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development