Hive
  1. Hive
  2. HIVE-809

Create a copier to copy data from scribe hdfs cluster to main DW cluster

    Details

    • Type: New Feature New Feature
    • Status: Open
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Currently we have scribe hdfs, which write scribe data directly to HDFS cluster. But in most cases this cluster will not be used for accessing the data.
      This data needs to copied to cluster from which you can access this scribe using hive or some other tool.
      This copier should be able to copy large amounts of data on a new realtime bases.

      1. patch_809_1.txt
        52 kB
        Suresh Antony

        Activity

        Suresh Antony created issue -
        Hide
        Suresh Antony added a comment -

        patch for scribe data copier.

        Show
        Suresh Antony added a comment - patch for scribe data copier.
        Suresh Antony made changes -
        Field Original Value New Value
        Attachment patch_809_1.txt [ 12418408 ]
        Hide
        Suresh Antony added a comment -

        Submitted patch for scribehdfs to main hdfs copier.

        Show
        Suresh Antony added a comment - Submitted patch for scribehdfs to main hdfs copier.
        Suresh Antony made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Johan Oskarsson made changes -
        Fix Version/s 0.5.0 [ 12314156 ]
        Fix Version/s 0.4.0 [ 12313714 ]
        Hide
        Namit Jain added a comment -

        It contains authors names in a few places.
        Contains system.out.println() in a few places.
        No unit test - can you use this to copy from 1 dir to another and add a test for the same.

        Show
        Namit Jain added a comment - It contains authors names in a few places. Contains system.out.println() in a few places. No unit test - can you use this to copy from 1 dir to another and add a test for the same.
        Hide
        Namit Jain added a comment -

        The default copier has some facebook location

        Show
        Namit Jain added a comment - The default copier has some facebook location
        Namit Jain made changes -
        Fix Version/s 0.5.0 [ 12314156 ]
        Hide
        Suresh Antony added a comment -

        I am on vacation from 12/11/09 to 1/5/10.

        Show
        Suresh Antony added a comment - I am on vacation from 12/11/09 to 1/5/10.
        Hide
        Zheng Shao added a comment -

        There are some syntactic improvements. See http://java.sun.com/docs/codeconv/ for details.

        1. Variable names should NOT contain "_"
        2. Import some classes so we don't need to refer to the full name again and again: org.apache.hadoop.record.meta.TypeID

        Also it will be great to add javadocs for all public classes/methods.

        Show
        Zheng Shao added a comment - There are some syntactic improvements. See http://java.sun.com/docs/codeconv/ for details. 1. Variable names should NOT contain "_" 2. Import some classes so we don't need to refer to the full name again and again: org.apache.hadoop.record.meta.TypeID Also it will be great to add javadocs for all public classes/methods.
        Zheng Shao made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]

          People

          • Assignee:
            Suresh Antony
            Reporter:
            Suresh Antony
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:

              Development