Flume
  1. Flume
  2. FLUME-582

Multiple diskfailover cannot cohabit in the same logical node configuration

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Won't Fix
    • Affects Version/s: v0.9.3
    • Fix Version/s: v0.9.5
    • Component/s: None
    • Labels:
      None
    • Environment:

      Any

      Description

      DiskFailoverDeco relies on a DiskFailoverManager which is associated with the logical node.

      When using configurations such as fanouts with diskfailover and/or agentDFOChain, the multiple instances of DiskFailoverDeco use a single DiskFailoverManager and this causes trouble in the form of IllegalStateExceptions being thrown.

      The same problem might arise with the ackedWriteAhead even though it has provision for forcing the logical node name, this is not used in the configuration generated by agentE2EChain.

      1. ASF.LICENSE.NOT.GRANTED--FLUME-582-3.patch
        3 kB
        Mathias Herberts
      2. ASF.LICENSE.NOT.GRANTED--FLUME-582-2.patch
        7 kB
        Mathias Herberts
      3. ASF.LICENSE.NOT.GRANTED--FLUME-582.patch
        5 kB
        Mathias Herberts

        Activity

        Hide
        Ashish Paliwal added a comment -

        Won't fix. 0.X branch not maintained anymore

        Show
        Ashish Paliwal added a comment - Won't fix. 0.X branch not maintained anymore
        Hide
        Jonathan Hsieh added a comment -

        Hi Mathias,

        I've created a version of the patch and put it up for review here:

        https://review.cloudera.org/r/1775/diff/#index_header

        I've testing it now against the old tests (may break some code, but not sure). From a quick glance I'm going to say it will need some unit tests to be added.

        Jon.

        Show
        Jonathan Hsieh added a comment - Hi Mathias, I've created a version of the patch and put it up for review here: https://review.cloudera.org/r/1775/diff/#index_header I've testing it now against the old tests (may break some code, but not sure). From a quick glance I'm going to say it will need some unit tests to be added. Jon.
        Hide
        Jonathan Hsieh added a comment -

        Mathias,

        Can you check this out and follow up https://github.com/cloudera/flume/wiki/HowToContribute so we can proceed? Ideally we'd like the rights grant in the comments and then posting the patch on reviewboard to make reviewing easier.

        Thanks,
        Jon.

        Show
        Jonathan Hsieh added a comment - Mathias, Can you check this out and follow up https://github.com/cloudera/flume/wiki/HowToContribute so we can proceed? Ideally we'd like the rights grant in the comments and then posting the patch on reviewboard to make reviewing easier. Thanks, Jon.
        Hide
        Mathias Herberts added a comment -

        Version 3 of the patch which corrects a bug in DiskFailoverDeco when interpreting numeric params.

        This patch no longer modifies agentDFOChain.

        For the rare cases when there are multiple agentDFOChain/agentE2EChain, their translation can be used directly (thus specifying the DFO/WAL dir suffix).

        Show
        Mathias Herberts added a comment - Version 3 of the patch which corrects a bug in DiskFailoverDeco when interpreting numeric params. This patch no longer modifies agentDFOChain. For the rare cases when there are multiple agentDFOChain/agentE2EChain, their translation can be used directly (thus specifying the DFO/WAL dir suffix).
        Hide
        Mathias Herberts added a comment -

        This second version of the patch addresses the issue for agentE2EChain also.

        It also corrects an error in the previous version of the patch (which was incorrectly generated by diffing files from 0.9.3 with files from trunk).

        The patch introduces 'dir_suffix' as first parameter of both agentDFOChain and agentE2EChain. This 'dir_suffix' is appended to the logical node name when creating the target dir for DFO/WAL log files. The 'dir_suffix' is propagated in the translated specs.

        The 'diskfailover' decorator is also modified this way.

        The 'ackedWriteAhead' decorator has its semantics slightly changed. The 'walnode' second arg has been replaced by a 'dir_suffix' first arg which has the semantics explained above. It is therefore no longer possible to SET the 'logical node' associated with a WAL manager, it is only possible to add a suffix to the logical node name.

        Show
        Mathias Herberts added a comment - This second version of the patch addresses the issue for agentE2EChain also. It also corrects an error in the previous version of the patch (which was incorrectly generated by diffing files from 0.9.3 with files from trunk). The patch introduces 'dir_suffix' as first parameter of both agentDFOChain and agentE2EChain. This 'dir_suffix' is appended to the logical node name when creating the target dir for DFO/WAL log files. The 'dir_suffix' is propagated in the translated specs. The 'diskfailover' decorator is also modified this way. The 'ackedWriteAhead' decorator has its semantics slightly changed. The 'walnode' second arg has been replaced by a 'dir_suffix' first arg which has the semantics explained above. It is therefore no longer possible to SET the 'logical node' associated with a WAL manager, it is only possible to add a suffix to the logical node name.
        Hide
        Mathias Herberts added a comment -

        The FLUME-582.patch modifies the 'diskfailover' decorator and 'agentDFOChain' so they accept as first argument a subdir suffix that will be added to the logical name node when creating the directory in which the dfo_* dirs will be created.

        In the case of 'agentDFOChain', the provided suffix is propagated to 'diskfailover'.

        This patch does not address the similar problem that might be present in ackedWriteAhead.

        Show
        Mathias Herberts added a comment - The FLUME-582 .patch modifies the 'diskfailover' decorator and 'agentDFOChain' so they accept as first argument a subdir suffix that will be added to the logical name node when creating the directory in which the dfo_* dirs will be created. In the case of 'agentDFOChain', the provided suffix is propagated to 'diskfailover'. This patch does not address the similar problem that might be present in ackedWriteAhead.

          People

          • Assignee:
            Mathias Herberts
            Reporter:
            Mathias Herberts
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development