Uploaded image for project: 'Apache Gobblin'
  1. Apache Gobblin
  2. GOBBLIN-379

Submit an event when DistCp job resource requirements exceed a hard bound.

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 0.11.0, 0.12.0, 0.13.0
    • 0.12.0
    • distcp-ng
    • None

    Description

      When the resource requirements imposed by a distcp dataset exceed those specified by properties gobblin.copy.prioritization.maxCopy.copyEntities & gobblin.copy.prioritization.maxCopy.size (default: 20000 and 10TB respectively), the dataset is skipped with a WARN message in the log.  In a production setting, this will not suffice and it would be nice to submit an event to Kafka for downstream processing and alerting. 

       

      Attachments

        Activity

          People

            ibuenros Issac Buenrostro
            sv2000 Sudarshan Vasudevan
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: