Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-2661

[Python/C++] Allow passing HDFS Config values via map/dict instead of needing an hdfs-site.xml file

    XMLWordPrintableJSON

    Details

    • Flags:
      Patch

      Description

      Currently in order to pass HDFS configurations of values down to the underlying libhdfs driver you have to either set HADOOP_HOME (using libhdfs) or LIBHDFS3_CONF (for libhdfs3) to point to a location of an hdfs-site.xml file to use. However the API provided by them allows for calling hdfsBuilderConfSetStr to set arbitrary configuration values on the hdfsBuilder object. This would allow for consumers to programmatically set any arbitrary configurations they want including allowing for encryption and other complex hdfs configurations without needing to provide an hdfs-site.xml file.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                zeroshade Matt Topol
                Reporter:
                zeroshade Matt Topol
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h 20m
                  1h 20m