Cassandra
  1. Cassandra
  2. CASSANDRA-3197

Separate input and output connection details in ConfigHelper

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Fix Version/s: 1.1.0
    • Component/s: Hadoop
    • Labels:
      None

      Description

      Currently ConfigHelper's getInitialAddress(..) getRpcPort(..) and getPartitioner(..)
      presume CFIF will be using the same cluster as CFOF.

      The latter two are a problem for me as on the same servers i'm running two clusters, one w/ ByteOrderingPartitioner and the other with RP), and i would like to read from the BOP cluster and write to the RP cluster.

        Activity

        Hide
        Mck SembWever added a comment -

        This does break compatibility. But it's minor and rather obvious to fix.

        Show
        Mck SembWever added a comment - This does break compatibility. But it's minor and rather obvious to fix.
        Hide
        Mck SembWever added a comment -

        patch for contrib/pig

        Show
        Mck SembWever added a comment - patch for contrib/pig
        Hide
        Mck SembWever added a comment -

        good to go

        Show
        Mck SembWever added a comment - good to go
        Hide
        Jonathan Ellis added a comment -

        Let's target 1.0 then if it breaks compatibility.

        Show
        Jonathan Ellis added a comment - Let's target 1.0 then if it breaks compatibility.
        Hide
        Jonathan Ellis added a comment -

        This doesn't look quite as trivial as I thought initially, so we should respect the 1.0 freeze and put this in 1.1.

        Show
        Jonathan Ellis added a comment - This doesn't look quite as trivial as I thought initially, so we should respect the 1.0 freeze and put this in 1.1.
        Hide
        Mck SembWever added a comment -

        Rebased against trunk

        Show
        Mck SembWever added a comment - Rebased against trunk
        Hide
        Brandon Williams added a comment -

        Looks like examples/hadoop_word_count needs to be updated as well.

        Show
        Brandon Williams added a comment - Looks like examples/hadoop_word_count needs to be updated as well.
        Hide
        Mck SembWever added a comment -

        new patch including everything (plus examples/hadoop_word_count)

        Show
        Mck SembWever added a comment - new patch including everything (plus examples/hadoop_word_count)
        Hide
        Brandon Williams added a comment -

        Unfortunately now it doesn't compile against trunk since the new bulk stuff from CASSANDRA-3045 needs to be updated as well.

        Show
        Brandon Williams added a comment - Unfortunately now it doesn't compile against trunk since the new bulk stuff from CASSANDRA-3045 needs to be updated as well.
        Hide
        Mck SembWever added a comment -

        new patch against trunk

        Show
        Mck SembWever added a comment - new patch against trunk
        Hide
        Brandon Williams added a comment -

        It looks like the output partitioner/rpcport/initial address aren't being set for pig or the word count so jobs with output NPE at org.apache.cassandra.utils.FBUtilities.newPartitioner.

        Show
        Brandon Williams added a comment - It looks like the output partitioner/rpcport/initial address aren't being set for pig or the word count so jobs with output NPE at org.apache.cassandra.utils.FBUtilities.newPartitioner.
        Hide
        Mck SembWever added a comment -

        That's odd. word count uses a FileOutputFormat.
        But for pig i completely forgot store functionality had been added to it. will fix that.

        Show
        Mck SembWever added a comment - That's odd. word count uses a FileOutputFormat. But for pig i completely forgot store functionality had been added to it. will fix that.
        Hide
        Brandon Williams added a comment -

        That's odd. word count uses a FileOutputFormat.

        Only if the reducer is set to 'filesystem'

        Show
        Brandon Williams added a comment - That's odd. word count uses a FileOutputFormat. Only if the reducer is set to 'filesystem'
        Hide
        Mck SembWever added a comment - - edited

        Quite right. was looking at WordCountCounters :-/

        Show
        Mck SembWever added a comment - - edited Quite right. was looking at WordCountCounters :-/
        Hide
        Mck SembWever added a comment -

        new patch

        Show
        Mck SembWever added a comment - new patch
        Hide
        Brandon Williams added a comment -

        For pig can we split the env vars into input/output so it can be used with multiple clusters as well, but if the old-style vars are set use those to initialize both input and output for compatibility?

        Show
        Brandon Williams added a comment - For pig can we split the env vars into input/output so it can be used with multiple clusters as well, but if the old-style vars are set use those to initialize both input and output for compatibility?
        Hide
        Mck SembWever added a comment -

        Next patch attempt. Provides separate env vars for PIG's input/output. The existing env vars are still in use and used as fallback for both input and output.

        Show
        Mck SembWever added a comment - Next patch attempt. Provides separate env vars for PIG's input/output. The existing env vars are still in use and used as fallback for both input and output.
        Hide
        Brandon Williams added a comment -

        Committed.

        Show
        Brandon Williams added a comment - Committed.
        Hide
        Brandon Williams added a comment -

        Reverted, this broke BulkRecordWriter.

        Show
        Brandon Williams added a comment - Reverted, this broke BulkRecordWriter.
        Hide
        Brandon Williams added a comment -

        Committed again with trivial fixes to BRW.

        Show
        Brandon Williams added a comment - Committed again with trivial fixes to BRW.

          People

          • Assignee:
            Mck SembWever
            Reporter:
            Mck SembWever
            Reviewer:
            Brandon Williams
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development