Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-12427 HiveServer2: Improve HiveServer2 JDBC/ODBC ResultSet performance - part1
  3. HIVE-12049

HiveServer2: Provide an option to write serialized thrift objects in final tasks

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0
    • Fix Version/s: 2.1.0
    • Component/s: HiveServer2, JDBC
    • Labels:
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      For each fetch request to HiveServer2, we pay the penalty of deserializing the row objects and translating them into a different representation suitable for the RPC transfer. In a moderate to high concurrency scenarios, this can result in significant CPU and memory wastage. By having each task write the appropriate thrift objects to the output files, HiveServer2 can simply stream a batch of rows on the wire without incurring any of the additional cost of deserialization and translation.
      This can be implemented by writing a new SerDe, which the FileSinkOperator can use to write thrift formatted row batches to the output file. Using the pluggable property of the hive.query.result.fileformat, we can set it to use SequenceFile and write a batch of thrift formatted rows as a value blob. The FetchTask can now simply read the blob and send it over the wire. On the client side, the *DBC driver can read the blob and since it is already formatted in the way it expects, it can continue building the ResultSet the way it does in the current implementation.

        Attachments

        1. HIVE-12049.1.patch
          71 kB
          Vaibhav Gumashta
        2. HIVE-12049.11.patch
          155 kB
          Rohit Dholakia
        3. HIVE-12049.12.patch
          155 kB
          Rohit Dholakia
        4. HIVE-12049.13.patch
          155 kB
          Rohit Dholakia
        5. HIVE-12049.14.patch
          155 kB
          Rohit Dholakia
        6. HIVE-12049.15.patch
          156 kB
          Rohit Dholakia
        7. HIVE-12049.16.patch
          116 kB
          Rohit Dholakia
        8. HIVE-12049.17.patch
          158 kB
          Rohit Dholakia
        9. HIVE-12049.18.patch
          160 kB
          Rohit Dholakia
        10. HIVE-12049.19.patch
          166 kB
          Rohit Dholakia
        11. HIVE-12049.2.patch
          91 kB
          Vaibhav Gumashta
        12. HIVE-12049.25.patch
          163 kB
          Rohit Dholakia
        13. HIVE-12049.26.patch
          163 kB
          Vaibhav Gumashta
        14. HIVE-12049.3.patch
          101 kB
          Vaibhav Gumashta
        15. HIVE-12049.4.patch
          192 kB
          Vaibhav Gumashta
        16. HIVE-12049.5.patch
          192 kB
          Vaibhav Gumashta
        17. HIVE-12049.6.patch
          192 kB
          Rohit Dholakia
        18. HIVE-12049.7.patch
          195 kB
          Rohit Dholakia
        19. HIVE-12049.9.patch
          191 kB
          Vaibhav Gumashta
        20. new-driver-profiles.png
          153 kB
          Gopal Vijayaraghavan
        21. old-driver-profiles.png
          150 kB
          Gopal Vijayaraghavan

          Issue Links

            Activity

              People

              • Assignee:
                rohitdholakia Rohit Dholakia
                Reporter:
                rohitdholakia Rohit Dholakia
              • Votes:
                0 Vote for this issue
                Watchers:
                12 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: