Hive
  1. Hive
  2. HIVE-2126

Hive's symlink text input format should be able to work with ComineHiveInputFormat

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      at compile time, if a partition's file format is SymlinkTextInputFormat, will replace the symlink path with paths in the symlink file. This way, it will work with Hive's HiveCombineFileInputFormat.

      The reason we are doing it at compile time is because:
      1) At run time, the input path is not only used to get record reader, but also used for hive to get aliases and thus operator tree. But the CombineHiveInputFormat can have multiple paths for each split, and when switching paths, it also set the job with new input file name. So it always require a real input path name. Can not fake it.
      2) if write a new input format, it will require a lot of duplication work with existing CombineHiveInputFormat.

      1. HIVE-2126.1.patch
        18 kB
        He Yongqiang
      2. HIVE-2126.2.patch
        21 kB
        He Yongqiang

        Activity

        Carl Steinbach made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Carl Steinbach made changes -
        Fix Version/s 0.8.0 [ 12316178 ]
        Namit Jain made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags [Reviewed]
        Resolution Fixed [ 1 ]
        He Yongqiang made changes -
        Attachment HIVE-2126.2.patch [ 12477334 ]
        He Yongqiang made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        He Yongqiang made changes -
        Field Original Value New Value
        Attachment HIVE-2126.1.patch [ 12477283 ]
        He Yongqiang created issue -

          People

          • Assignee:
            He Yongqiang
            Reporter:
            He Yongqiang
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development