Uploaded image for project: 'Ignite'
  1. Ignite
  2. IGNITE-14743

Support Row with large values.

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0-alpha3
    • Component/s: None
    • Labels:
    • Ignite Flags:
      Docs Required, Release Notes Required

      Description

      Motivation.

      For now, TupleAssembler writes offsets for varlen columns as 2-byte {{short }}type.
      This implicitly restricts key/value sizes down to 64 kB in total.
      On another side, for small rows that columns can be addressed with byte type, we will waste few bytes.

      Description.

      Let's

      1. allow 4 byte types (byte, short, int) for offsets.
      2. implement and benchmark different approaches that allow us to write rows in the most compact way.
      3. then choose and merge the best one.

      We can introduce several formats for writing Vartable (using byte/short/int offsets).
      Additional information about Vartable format can be coded into chunk flags.

      The first approach is to precalculate chunk total size, then choose the most compact format and write a chunk.
      The second approach is to write a chunk with the widest format then convert the chunk into the most compact format in place.

        Attachments

        1. Byte array columns only benchmark.txt
          4 kB
          Andrey Mashenkov
        2. Fixlen cols only benchmark.txt
          3 kB
          Andrey Mashenkov
        3. Latin1 string columns benchmark.txt
          5 kB
          Andrey Mashenkov
        4. Non-latin string columns benchmark.txt
          3 kB
          Andrey Mashenkov
        5. String marshalling comparison
          7 kB
          Andrey Mashenkov

          Issue Links

            Activity

              People

              • Assignee:
                amashenkov Andrey Mashenkov
                Reporter:
                amashenkov Andrey Mashenkov
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - 168h
                  168h
                  Remaining:
                  Time Spent - 2.5h Remaining Estimate - 165.5h
                  165.5h
                  Logged:
                  Time Spent - 2.5h Remaining Estimate - 165.5h
                  2.5h