Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-430

Introducing gcpTempLocation that default to tempLocation

    Details

      Description

      Currently, DataflowPipelineOptions.stagingLocation default to tempLocation. And, it requires tempLocation to be a gcs path.
      Another case is BigQueryIO uses tempLocation and also requires it to be on gcs.
      So, users cannot set tempLocation to a non-gcs path with DataflowRunner or BigQueryIO.

      However, tempLocation could be on any file system. For example, WordCount defaults to output to tempLocation.

      The proposal is to add gcpTempLocation. And, it defaults to tempLocation if tempLocation is a gcs path.
      StagingLocation and BigQueryIO will use gcpTempLocation by default.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                peihe0@gmail.com Pei He
                Reporter:
                peihe0@gmail.com Pei He
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: