Uploaded image for project: 'Apache Hudi'
  1. Apache Hudi
  2. HUDI-6712

Implement optimized keyed lookup on parquet files

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • 1.1.0
    • None

    Description

      Parquet performs poorly when performing a lookup of specific records, based on a single key lookup column. 

      e.g: select * from parquet where key in ("a","b", "c) (SQL)
      e.g: List<Records> lookup(parquetFile, Set<String> keys) (code) 

      Let's implement a reader, that is optimized for this pattern, by scanning least amount of data. 

      Requirements: 
      1. Need to support multiple values for same key. 
      2. Can assume the file is sorted by the key/lookup field. 
      3. Should handle non-existence of keys.
      4. Should leverage parquet metadata (bloom filters, column index, ... ) to minimize read read. 
      5. Must to the minimum about of RPC calls to cloud storage.

      Attachments

        Issue Links

          Blocked

          New Feature - A new feature of the product, which has yet to be developed. HUDI-6772 Handle missing index metadata for keyed lookup reader

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          New Feature - A new feature of the product, which has yet to be developed. HUDI-6770 Improve on Key Lookup Reader

          • Major - Major loss of function.
          • Closed

          New Feature - A new feature of the product, which has yet to be developed. HUDI-6771 Support Bloom Filter in Keyed Lookup Reader

          • Major - Major loss of function.
          • Patch Available
          blocks

          New Feature - A new feature of the product, which has yet to be developed. HUDI-6769 Integration test on Keyed Lookup Reader

          • Major - Major loss of function.
          • Open

          New Feature - A new feature of the product, which has yet to be developed. HUDI-6783 Add property interface support for key lookup reader

          • Major - Major loss of function.
          • Open
          links to

          Web Link GitHub Pull Request #9564

          Activity

            People

              linliu Lin Liu
              vinoth Vinoth Chandar
              Vinoth Chandar
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated: