Uploaded image for project: 'Ignite'
  1. Ignite
  2. IGNITE-14743

Support Row with large values.

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 3.0.0-alpha3
    • None
    • Docs Required, Release Notes Required

    Description

      Motivation.

      For now, TupleAssembler writes offsets for varlen columns as 2-byte {{short }}type.
      This implicitly restricts key/value sizes down to 64 kB in total.
      On another side, for small rows that columns can be addressed with byte type, we will waste few bytes.

      Description.

      Let's

      1. allow 4 byte types (byte, short, int) for offsets.
      2. implement and benchmark different approaches that allow us to write rows in the most compact way.
      3. then choose and merge the best one.

      We can introduce several formats for writing Vartable (using byte/short/int offsets).
      Additional information about Vartable format can be coded into chunk flags.

      The first approach is to precalculate chunk total size, then choose the most compact format and write a chunk.
      The second approach is to write a chunk with the widest format then convert the chunk into the most compact format in place.

      Attachments

        1. Fixlen cols only benchmark.txt
          3 kB
          Andrey Mashenkov
        2. Latin1 string columns benchmark.txt
          5 kB
          Andrey Mashenkov
        3. Non-latin string columns benchmark.txt
          3 kB
          Andrey Mashenkov
        4. Byte array columns only benchmark.txt
          4 kB
          Andrey Mashenkov
        5. String marshalling comparison
          7 kB
          Andrey Mashenkov

        Issue Links

          Activity

            People

              amashenkov Andrey Mashenkov
              amashenkov Andrey Mashenkov
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - 168h
                  168h
                  Remaining:
                  Time Spent - 2.5h Remaining Estimate - 165.5h
                  165.5h
                  Logged:
                  Time Spent - 2.5h Remaining Estimate - 165.5h
                  2.5h