Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-4248

MRReaderMapred should propagate properties based on config

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 0.10.1, 0.9.3
    • None
    • None

    Description

      Based on a new config, MRReaderMapred should pass properites into its incremental conf object.

      The config name: tez.mrreader.config.update.properties

      Example use case: in Hive, we would like to push down the projection properties (i.e. which columns are selected) set during MRInput initialization, in order to have this information available when constructing the object inspector (i.e. to create an object inspector which reads only the selected columns) during the map operation. To achieve that, in Hive we would set the following property: conf.set("tez.mrreader.config.update.properties", "hive.io.file.readcolumn.names,hive.io.file.readcolumn.ids"). This would ensure we get these two property values propagated into the main conf object.

      Attachments

        1. TEZ-4248.4.patch
          6 kB
          Marton Bod
        2. TEZ-4248.2.patch
          7 kB
          Marton Bod
        3. TEZ-4248.1.patch
          1 kB
          Marton Bod

        Activity

          People

            Marton Bod Marton Bod
            Marton Bod Marton Bod
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: