Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-11260

[C++][Dataset] Don't require dictionaries for reading dataset with schema-based Partitioning

    XMLWordPrintableJSON

Details

    Description

      As a follow-up on ARROW-10247 (see also https://github.com/apache/arrow/pull/9130#issuecomment-760801124). We currently require the user to pass manually specified dictionary values when reading a dataset with a Partitioning based on a schema with dictionary typed fields.

      In practice that means that the user for example needs to parse the file paths to get all the possible values the partition field can take, while Arrow will then afterwards again do the same to construct the dataset object.
      Naively, it seems that it should be possible to let Arrow infer the dictionary values, even when providing an explicit schema with a dictionary field for the Partitioning (i.e. so when not letting the partitioning schema itself be inferred from the file paths).

      An example use case is when you have a Partitioning schema with both dictionary and non-dictionary fields. When discovering the schema, you can only have all or nothing (all dictionary fields or no dictionary fields).

      cc bkietz

      Attachments

        Issue Links

          Activity

            People

              lidavidm David Li
              jorisvandenbossche Joris Van den Bossche
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 2h
                  2h