Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-775

Add input/output formatters for Vertica clustered ADBMS.

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: contrib/vertica
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Add native and streaming support for Vertica as an input or output format taking advantage of parallel read and write properties of the DBMS.
    • Tags:
      vertica, db, formatter

      Description

      Add native support for Vertica as an input or output format taking advantage of parallel read and write properties of the DBMS.

      On the input side allow for parametrized queries (a la prepared statements) and create a split for each combination of parameters. Also support the parameter list to be generated from a sql statement. For example - return metrics for all dimensions that meet criteria X with one input split for each dimension. Divide the read among any number of hosts in the Vertica cluster.

      On the output side, support Vertica streaming load to any number of hosts in the Vertica cluster. Output may be to a different cluster than input.

      Also includes Input and Output formatters that support streaming interface.

      Code has been tested and run on live systems under 19 and 20. Patch for 21 with new API will be ready end of this week.

      1. MAPREDUCE-775.2.patch
        120 kB
        Omer Trajman
      2. MAPREDUCE-775.3.patch
        120 kB
        Omer Trajman
      3. MAPREDUCE-775.4.patch
        121 kB
        Omer Trajman
      4. MAPREDUCE-775.patch
        115 kB
        Omer Trajman

        Activity

        Omer Trajman created issue -
        Omer Trajman made changes -
        Field Original Value New Value
        Attachment MAPREDUCE-775.patch [ 12414581 ]
        Omer Trajman made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Arun C Murthy made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Omer Trajman made changes -
        Attachment MAPREDUCE-775.patch [ 12415514 ]
        Omer Trajman made changes -
        Attachment MAPREDUCE-775.patch [ 12414581 ]
        Omer Trajman made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Omer Trajman made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Omer Trajman made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Tom White made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Assignee Omer Trajman [ otrajman ]
        Omer Trajman made changes -
        Attachment MAPREDUCE-775.2.patch [ 12419368 ]
        Omer Trajman made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Omer Trajman made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Omer Trajman made changes -
        Attachment MAPREDUCE-775.2.patch [ 12419368 ]
        Omer Trajman made changes -
        Attachment MAPREDUCE-775.2.patch [ 12419380 ]
        Omer Trajman made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Omer Trajman made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Omer Trajman made changes -
        Attachment MAPREDUCE-775.3.patch [ 12419401 ]
        Omer Trajman made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Omer Trajman made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Omer Trajman made changes -
        Attachment MAPREDUCE-775.4.patch [ 12419420 ]
        Omer Trajman made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Devaraj Das made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags [Reviewed]
        Resolution Fixed [ 1 ]
        Chris Douglas made changes -
        Component/s contrib/vertica [ 12313270 ]
        Tom White made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Omer Trajman
            Reporter:
            Omer Trajman
          • Votes:
            0 Vote for this issue
            Watchers:
            15 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development