Cassandra
  1. Cassandra
  2. CASSANDRA-3197

Separate input and output connection details in ConfigHelper

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Fix Version/s: 1.1.0
    • Component/s: Hadoop
    • Labels:
      None

      Description

      Currently ConfigHelper's getInitialAddress(..) getRpcPort(..) and getPartitioner(..)
      presume CFIF will be using the same cluster as CFOF.

      The latter two are a problem for me as on the same servers i'm running two clusters, one w/ ByteOrderingPartitioner and the other with RP), and i would like to read from the BOP cluster and write to the RP cluster.

        Activity

        Mck SembWever created issue -
        Hide
        Mck SembWever added a comment -

        This does break compatibility. But it's minor and rather obvious to fix.

        Show
        Mck SembWever added a comment - This does break compatibility. But it's minor and rather obvious to fix.
        Mck SembWever made changes -
        Field Original Value New Value
        Attachment CASSANDRA-3197.patch [ 12494215 ]
        Hide
        Mck SembWever added a comment -

        patch for contrib/pig

        Show
        Mck SembWever added a comment - patch for contrib/pig
        Mck SembWever made changes -
        Attachment CASSANDRA-3197-extra.patch [ 12494217 ]
        Hide
        Mck SembWever added a comment -

        good to go

        Show
        Mck SembWever added a comment - good to go
        Mck SembWever made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Mck SembWever made changes -
        Fix Version/s 0.8.6 [ 12317850 ]
        Sylvain Lebresne made changes -
        Fix Version/s 0.8.7 [ 12317966 ]
        Fix Version/s 0.8.6 [ 12317850 ]
        Hide
        Jonathan Ellis added a comment -

        Let's target 1.0 then if it breaks compatibility.

        Show
        Jonathan Ellis added a comment - Let's target 1.0 then if it breaks compatibility.
        Jonathan Ellis made changes -
        Fix Version/s 1.0.0 [ 12316349 ]
        Fix Version/s 0.8.7 [ 12317966 ]
        Affects Version/s 0.7.0 [ 12316026 ]
        Affects Version/s 0.8.5 [ 12317588 ]
        Priority Major [ 3 ] Minor [ 4 ]
        Reviewer brandon.williams
        Hide
        Jonathan Ellis added a comment -

        This doesn't look quite as trivial as I thought initially, so we should respect the 1.0 freeze and put this in 1.1.

        Show
        Jonathan Ellis added a comment - This doesn't look quite as trivial as I thought initially, so we should respect the 1.0 freeze and put this in 1.1.
        Jonathan Ellis made changes -
        Fix Version/s 1.1 [ 12317615 ]
        Fix Version/s 1.0.0 [ 12316349 ]
        Hide
        Mck SembWever added a comment -

        Rebased against trunk

        Show
        Mck SembWever added a comment - Rebased against trunk
        Mck SembWever made changes -
        Attachment CASSANDRA-3197.patch [ 12501444 ]
        Attachment CASSANDRA-3197-extra.patch [ 12501445 ]
        Mck SembWever made changes -
        Description Currently ConfigHelper's getInitialAddress(..) getRpcPort(..) and getPartitioner(..)
        presume CFIF will be using the same cluster as CFOF.

        The latter two are a problem for me as on the same servers i'm running two clusters, one w/ ByteOrderingPartitioner and the other with RP), and i would like to read from the BOP cluster and write to the RP cluster.

        getInitialAddress(..) is of little concern to me.
        Currently ConfigHelper's getInitialAddress(..) getRpcPort(..) and getPartitioner(..)
        presume CFIF will be using the same cluster as CFOF.

        The latter two are a problem for me as on the same servers i'm running two clusters, one w/ ByteOrderingPartitioner and the other with RP), and i would like to read from the BOP cluster and write to the RP cluster.
        Hide
        Brandon Williams added a comment -

        Looks like examples/hadoop_word_count needs to be updated as well.

        Show
        Brandon Williams added a comment - Looks like examples/hadoop_word_count needs to be updated as well.
        Mck SembWever made changes -
        Attachment CASSANDRA-3197-extra.patch [ 12494217 ]
        Mck SembWever made changes -
        Attachment CASSANDRA-3197.patch [ 12494215 ]
        Mck SembWever made changes -
        Attachment CASSANDRA-3197-extra.patch [ 12501445 ]
        Mck SembWever made changes -
        Attachment CASSANDRA-3197.patch [ 12501444 ]
        Hide
        Mck SembWever added a comment -

        new patch including everything (plus examples/hadoop_word_count)

        Show
        Mck SembWever added a comment - new patch including everything (plus examples/hadoop_word_count)
        Mck SembWever made changes -
        Attachment CASSANDRA-3197.patch [ 12505748 ]
        Hide
        Brandon Williams added a comment -

        Unfortunately now it doesn't compile against trunk since the new bulk stuff from CASSANDRA-3045 needs to be updated as well.

        Show
        Brandon Williams added a comment - Unfortunately now it doesn't compile against trunk since the new bulk stuff from CASSANDRA-3045 needs to be updated as well.
        Mck SembWever made changes -
        Attachment CASSANDRA-3197.patch [ 12505748 ]
        Hide
        Mck SembWever added a comment -

        new patch against trunk

        Show
        Mck SembWever added a comment - new patch against trunk
        Mck SembWever made changes -
        Attachment CASSANDRA-3197.patch [ 12506117 ]
        Hide
        Brandon Williams added a comment -

        It looks like the output partitioner/rpcport/initial address aren't being set for pig or the word count so jobs with output NPE at org.apache.cassandra.utils.FBUtilities.newPartitioner.

        Show
        Brandon Williams added a comment - It looks like the output partitioner/rpcport/initial address aren't being set for pig or the word count so jobs with output NPE at org.apache.cassandra.utils.FBUtilities.newPartitioner.
        Hide
        Mck SembWever added a comment -

        That's odd. word count uses a FileOutputFormat.
        But for pig i completely forgot store functionality had been added to it. will fix that.

        Show
        Mck SembWever added a comment - That's odd. word count uses a FileOutputFormat. But for pig i completely forgot store functionality had been added to it. will fix that.
        Hide
        Brandon Williams added a comment -

        That's odd. word count uses a FileOutputFormat.

        Only if the reducer is set to 'filesystem'

        Show
        Brandon Williams added a comment - That's odd. word count uses a FileOutputFormat. Only if the reducer is set to 'filesystem'
        Hide
        Mck SembWever added a comment - - edited

        Quite right. was looking at WordCountCounters :-/

        Show
        Mck SembWever added a comment - - edited Quite right. was looking at WordCountCounters :-/
        Mck SembWever made changes -
        Attachment CASSANDRA-3197.patch [ 12506117 ]
        Hide
        Mck SembWever added a comment -

        new patch

        Show
        Mck SembWever added a comment - new patch
        Mck SembWever made changes -
        Attachment CASSANDRA-3197.patch [ 12506520 ]
        Hide
        Brandon Williams added a comment -

        For pig can we split the env vars into input/output so it can be used with multiple clusters as well, but if the old-style vars are set use those to initialize both input and output for compatibility?

        Show
        Brandon Williams added a comment - For pig can we split the env vars into input/output so it can be used with multiple clusters as well, but if the old-style vars are set use those to initialize both input and output for compatibility?
        Mck SembWever made changes -
        Attachment CASSANDRA-3197.patch [ 12506520 ]
        Hide
        Mck SembWever added a comment -

        Next patch attempt. Provides separate env vars for PIG's input/output. The existing env vars are still in use and used as fallback for both input and output.

        Show
        Mck SembWever added a comment - Next patch attempt. Provides separate env vars for PIG's input/output. The existing env vars are still in use and used as fallback for both input and output.
        Mck SembWever made changes -
        Attachment CASSANDRA-3197.patch [ 12508383 ]
        Hide
        Brandon Williams added a comment -

        Committed.

        Show
        Brandon Williams added a comment - Committed.
        Brandon Williams made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        Brandon Williams added a comment -

        Reverted, this broke BulkRecordWriter.

        Show
        Brandon Williams added a comment - Reverted, this broke BulkRecordWriter.
        Brandon Williams made changes -
        Resolution Fixed [ 1 ]
        Status Resolved [ 5 ] Reopened [ 4 ]
        Hide
        Brandon Williams added a comment -

        Committed again with trivial fixes to BRW.

        Show
        Brandon Williams added a comment - Committed again with trivial fixes to BRW.
        Brandon Williams made changes -
        Status Reopened [ 4 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Gavin made changes -
        Workflow no-reopen-closed, patch-avail [ 12632896 ] patch-available, re-open possible [ 12751601 ]
        Gavin made changes -
        Workflow patch-available, re-open possible [ 12751601 ] reopen-resolved, no closed status, patch-avail, testing [ 12757813 ]

          People

          • Assignee:
            Mck SembWever
            Reporter:
            Mck SembWever
            Reviewer:
            Brandon Williams
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development