Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-9016

DocumentStore as a backend for ATSv2

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 3.3.0
    • ATSv2
    • None
    • Reviewed

    Description

      Document Store for ATSv2

                     The Document Store for ATSv2 is a framework for plugging in any Document Store Vendor as a backend for ATSv2 i.e Azure CosmosDB , MongoDB, ElasticSearch etc.

      • Supports multiple Document Store Vendors like CosmosDB, ElasticSearch, MongoDB etc by just adding new configurations properties and writing Document Store reader and writer clients.
      • Currently has support for CosmosDB.
      • All writes are Async and buffered, latest document would be flushed to the store either if the document buffer gets full or periodically at every flush interval in background without adding any additional latency to the running jobs..
      • All the REST API's of Timeline Reader Server are supported.

      How to enable?

      Add the flowing properties under yarn-site.xml

      <!-- config required for ATSv2 to use DocumentStore-->
      <property>
       <name>yarn.timeline-service.writer.class </name>
       <value>org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreTimelineWriterImpl</value>
      </property>
      
      <property>
         <name>yarn.timeline-service.reader.class </name>             <value>org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreTimelineReaderImpl</value>
      </property>
      
      <property> 
         <name>yarn.timeline-service.document-store.db-name</name>       
         <value>YOUR_DATABASE_NAME</value> <!-- default is timeline_service --> 
      </property>

      Creating DB and Collections for storing documents

                            The following config needs to be set inside yarn-site.xml for creating the database and collections for storing documents.

      <!-- Using schema creator class for DocumentStore-->
      <property>
       <name>yarn.timeline-service.schema-creator.class </name>
       <value>org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator</value>
      </property>

                  Running the schema creator tool to create the necessary collections.

      bin/hadoop org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator

       Azure CosmosDB 

             To use Azure CosmosDB as a DocumentStore for ATSv2, the additional properties under yarn-site.xml is required..

      <!-- config required for using Azure CosmosDB as a DocumentStore for ATSv2 -->
      <property>  
         <name>yarn.timeline-service.document-store-type</name>  
         <value>COSMOS_DB</value>
      </property>
      
      <property>
         <name>yarn.timeline-service.document-store.cosmos-db.endpoint</name>
         <value>http://YOUR_AZURE_COSMOS_DB_URL:443/</value>
      </property>
      
      <property>
         <name>yarn.timeline-service.document-store.cosmos-db.masterkey</name>
         <value>YOUR_AZURE_COSMOS_DB_MASTER_KEY_CREDENTIAL</value>
      </property>
      

       
        Testing locally
                    In order to test the Azure CosmosDB as a DocumentStore locally, install the emulator from here and start it locally. Set the endpoint and master key under yarn-site.xml as mentioned above and run any example job like DistributedShell etc. Later you can check the data explorer UI of Azure CosmosDB locally to query the documents or even launch the TimelineReader locally to fetch/query the data from REST API's.   
                        

            

      Attachments

        1. YARN-9016.001.patch
          231 kB
          Sushil Ks
        2. YARN-9016.002.patch
          234 kB
          Sushil Ks
        3. YARN-9016.003.patch
          235 kB
          Sushil Ks
        4. YARN-9016.004.patch
          235 kB
          Sushil Ks

        Issue Links

          Activity

            People

              Sushil-K-S Sushil Ks
              Sushil-K-S Sushil Ks
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: