Cassandra
  1. Cassandra
  2. CASSANDRA-3197

Separate input and output connection details in ConfigHelper

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Fix Version/s: 1.1.0
    • Component/s: Hadoop
    • Labels:
      None

      Description

      Currently ConfigHelper's getInitialAddress(..) getRpcPort(..) and getPartitioner(..)
      presume CFIF will be using the same cluster as CFOF.

      The latter two are a problem for me as on the same servers i'm running two clusters, one w/ ByteOrderingPartitioner and the other with RP), and i would like to read from the BOP cluster and write to the RP cluster.

        Activity

        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        1h 10m 1 mck 13/Sep/11 13:07
        Patch Available Patch Available Resolved Resolved
        118d 4h 26m 1 Brandon Williams 09/Jan/12 16:33
        Resolved Resolved Reopened Reopened
        1h 20m 1 Brandon Williams 09/Jan/12 17:54
        Reopened Reopened Resolved Resolved
        18m 18s 1 Brandon Williams 09/Jan/12 18:13
        Gavin made changes -
        Workflow patch-available, re-open possible [ 12751601 ] reopen-resolved, no closed status, patch-avail, testing [ 12757813 ]
        Gavin made changes -
        Workflow no-reopen-closed, patch-avail [ 12632896 ] patch-available, re-open possible [ 12751601 ]
        Brandon Williams made changes -
        Status Reopened [ 4 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        Brandon Williams added a comment -

        Committed again with trivial fixes to BRW.

        Show
        Brandon Williams added a comment - Committed again with trivial fixes to BRW.
        Brandon Williams made changes -
        Resolution Fixed [ 1 ]
        Status Resolved [ 5 ] Reopened [ 4 ]
        Hide
        Brandon Williams added a comment -

        Reverted, this broke BulkRecordWriter.

        Show
        Brandon Williams added a comment - Reverted, this broke BulkRecordWriter.
        Brandon Williams made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        Brandon Williams added a comment -

        Committed.

        Show
        Brandon Williams added a comment - Committed.
        mck made changes -
        Attachment CASSANDRA-3197.patch [ 12508383 ]
        Hide
        mck added a comment -

        Next patch attempt. Provides separate env vars for PIG's input/output. The existing env vars are still in use and used as fallback for both input and output.

        Show
        mck added a comment - Next patch attempt. Provides separate env vars for PIG's input/output. The existing env vars are still in use and used as fallback for both input and output.
        mck made changes -
        Attachment CASSANDRA-3197.patch [ 12506520 ]
        Hide
        Brandon Williams added a comment -

        For pig can we split the env vars into input/output so it can be used with multiple clusters as well, but if the old-style vars are set use those to initialize both input and output for compatibility?

        Show
        Brandon Williams added a comment - For pig can we split the env vars into input/output so it can be used with multiple clusters as well, but if the old-style vars are set use those to initialize both input and output for compatibility?
        mck made changes -
        Attachment CASSANDRA-3197.patch [ 12506520 ]
        Hide
        mck added a comment -

        new patch

        Show
        mck added a comment - new patch
        mck made changes -
        Attachment CASSANDRA-3197.patch [ 12506117 ]
        Hide
        mck added a comment - - edited

        Quite right. was looking at WordCountCounters :-/

        Show
        mck added a comment - - edited Quite right. was looking at WordCountCounters :-/
        Hide
        Brandon Williams added a comment -

        That's odd. word count uses a FileOutputFormat.

        Only if the reducer is set to 'filesystem'

        Show
        Brandon Williams added a comment - That's odd. word count uses a FileOutputFormat. Only if the reducer is set to 'filesystem'
        Hide
        mck added a comment -

        That's odd. word count uses a FileOutputFormat.
        But for pig i completely forgot store functionality had been added to it. will fix that.

        Show
        mck added a comment - That's odd. word count uses a FileOutputFormat. But for pig i completely forgot store functionality had been added to it. will fix that.
        Hide
        Brandon Williams added a comment -

        It looks like the output partitioner/rpcport/initial address aren't being set for pig or the word count so jobs with output NPE at org.apache.cassandra.utils.FBUtilities.newPartitioner.

        Show
        Brandon Williams added a comment - It looks like the output partitioner/rpcport/initial address aren't being set for pig or the word count so jobs with output NPE at org.apache.cassandra.utils.FBUtilities.newPartitioner.
        mck made changes -
        Attachment CASSANDRA-3197.patch [ 12506117 ]
        Hide
        mck added a comment -

        new patch against trunk

        Show
        mck added a comment - new patch against trunk
        mck made changes -
        Attachment CASSANDRA-3197.patch [ 12505748 ]
        Hide
        Brandon Williams added a comment -

        Unfortunately now it doesn't compile against trunk since the new bulk stuff from CASSANDRA-3045 needs to be updated as well.

        Show
        Brandon Williams added a comment - Unfortunately now it doesn't compile against trunk since the new bulk stuff from CASSANDRA-3045 needs to be updated as well.
        mck made changes -
        Attachment CASSANDRA-3197.patch [ 12505748 ]
        Hide
        mck added a comment -

        new patch including everything (plus examples/hadoop_word_count)

        Show
        mck added a comment - new patch including everything (plus examples/hadoop_word_count)
        mck made changes -
        Attachment CASSANDRA-3197.patch [ 12501444 ]
        mck made changes -
        Attachment CASSANDRA-3197-extra.patch [ 12501445 ]
        mck made changes -
        Attachment CASSANDRA-3197.patch [ 12494215 ]
        mck made changes -
        Attachment CASSANDRA-3197-extra.patch [ 12494217 ]
        Hide
        Brandon Williams added a comment -

        Looks like examples/hadoop_word_count needs to be updated as well.

        Show
        Brandon Williams added a comment - Looks like examples/hadoop_word_count needs to be updated as well.
        mck made changes -
        Description Currently ConfigHelper's getInitialAddress(..) getRpcPort(..) and getPartitioner(..)
        presume CFIF will be using the same cluster as CFOF.

        The latter two are a problem for me as on the same servers i'm running two clusters, one w/ ByteOrderingPartitioner and the other with RP), and i would like to read from the BOP cluster and write to the RP cluster.

        getInitialAddress(..) is of little concern to me.
        Currently ConfigHelper's getInitialAddress(..) getRpcPort(..) and getPartitioner(..)
        presume CFIF will be using the same cluster as CFOF.

        The latter two are a problem for me as on the same servers i'm running two clusters, one w/ ByteOrderingPartitioner and the other with RP), and i would like to read from the BOP cluster and write to the RP cluster.
        mck made changes -
        Attachment CASSANDRA-3197.patch [ 12501444 ]
        Attachment CASSANDRA-3197-extra.patch [ 12501445 ]
        Hide
        mck added a comment -

        Rebased against trunk

        Show
        mck added a comment - Rebased against trunk
        Jonathan Ellis made changes -
        Fix Version/s 1.1 [ 12317615 ]
        Fix Version/s 1.0.0 [ 12316349 ]
        Hide
        Jonathan Ellis added a comment -

        This doesn't look quite as trivial as I thought initially, so we should respect the 1.0 freeze and put this in 1.1.

        Show
        Jonathan Ellis added a comment - This doesn't look quite as trivial as I thought initially, so we should respect the 1.0 freeze and put this in 1.1.
        Jonathan Ellis made changes -
        Fix Version/s 1.0.0 [ 12316349 ]
        Fix Version/s 0.8.7 [ 12317966 ]
        Affects Version/s 0.7.0 [ 12316026 ]
        Affects Version/s 0.8.5 [ 12317588 ]
        Priority Major [ 3 ] Minor [ 4 ]
        Reviewer brandon.williams
        Hide
        Jonathan Ellis added a comment -

        Let's target 1.0 then if it breaks compatibility.

        Show
        Jonathan Ellis added a comment - Let's target 1.0 then if it breaks compatibility.
        Sylvain Lebresne made changes -
        Fix Version/s 0.8.7 [ 12317966 ]
        Fix Version/s 0.8.6 [ 12317850 ]
        mck made changes -
        Fix Version/s 0.8.6 [ 12317850 ]
        mck made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        mck added a comment -

        good to go

        Show
        mck added a comment - good to go
        mck made changes -
        Attachment CASSANDRA-3197-extra.patch [ 12494217 ]
        Hide
        mck added a comment -

        patch for contrib/pig

        Show
        mck added a comment - patch for contrib/pig
        mck made changes -
        Field Original Value New Value
        Attachment CASSANDRA-3197.patch [ 12494215 ]
        Hide
        mck added a comment -

        This does break compatibility. But it's minor and rather obvious to fix.

        Show
        mck added a comment - This does break compatibility. But it's minor and rather obvious to fix.
        mck created issue -

          People

          • Assignee:
            mck
            Reporter:
            mck
            Reviewer:
            Brandon Williams
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development