Details

    • Type: Sub-task Sub-task
    • Status: Patch Available
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: v0.9.4
    • Fix Version/s: v0.9.5
    • Component/s: None
    • Labels:
      None

      Description

      Add functionality that imports the data inside HIVE once its written to HDFS.

        Activity

        Hide
        Anurag Phadke added a comment - - edited

        csteinbach and aphadke worked on couple of approaches.
        Approach #1: Use Local Metastore
        While this is a good approach and obviates the use of additional service, a ton of hadoop / hive / flume jar & config files are involved leading to a configuration nightmare. We won't be pursing it.

        Approach #2: Use Remote Metastore
        We will need to run hive service (hive --service hiveserver) but setting this up on FLUME side is fairly simple. Only the server hostname + port is needed which can be easily added to .xml files. Integrating rest of the create & append partition calls via THRIFT should be straight-forward.
        For now, we are moving ahead with Approach #2.

        Show
        Anurag Phadke added a comment - - edited csteinbach and aphadke worked on couple of approaches. Approach #1: Use Local Metastore While this is a good approach and obviates the use of additional service, a ton of hadoop / hive / flume jar & config files are involved leading to a configuration nightmare. We won't be pursing it. Approach #2: Use Remote Metastore We will need to run hive service (hive --service hiveserver) but setting this up on FLUME side is fairly simple. Only the server hostname + port is needed which can be easily added to .xml files. Integrating rest of the create & append partition calls via THRIFT should be straight-forward. For now, we are moving ahead with Approach #2.
        Hide
        Jonathan Hsieh added a comment -

        Would like to review code befor making patch available, and would like to have link to connector before we mark closed.

        Show
        Jonathan Hsieh added a comment - Would like to review code befor making patch available, and would like to have link to connector before we mark closed.
        Hide
        Jonathan Hsieh added a comment -

        If these exist, please upload patch or point to repo to pull from!

        Show
        Jonathan Hsieh added a comment - If these exist, please upload patch or point to repo to pull from!

          People

          • Assignee:
            Anurag Phadke
            Reporter:
            Anurag Phadke
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:

              Development