Details

    Description

      Preconditions:

      1. RStudio/Zeppelin is in running status

      Steps to reproduce:

      1. Create Data Engine Service (Dataproc) on RStudio/Zeppelin

      Actual result:

      1. Data Engine Service creation fails
      [chan 1] Opened sftp connection (server version 3)
      Warning: Permanently added '172.31.16.4' (ECDSA) to the list of known hosts.
      Traceback (most recent call last):
        File "/usr/local/bin/create_configs.py", line 26, in <module>
          from datalab.actions_lib import *
        File "/usr/lib/python3.8/datalab/actions_lib.py", line 24, in <module>
          import datalab.common_lib
        File "/usr/lib/python3.8/datalab/common_lib.py", line 27, in <module>
          from datalab.logger import logging
        File "/usr/lib/python3.8/datalab/logger.py", line 32, in <module>
          local_log_filename = "{}_{}.log".format(os.environ['conf_resource'], os.environ['request_id'])
        File "/usr/lib/python3.8/os.py", line 675, in __getitem__
          raise KeyError(key) from None
      KeyError: 'request_id'
      

      Expected result:

      1. Data Engine Service creation is successful

       

      Attachments

        Activity

          lfrolov Leonid Frolov added a comment -

          fixed request_id error, but got new error

          usage: create_configs.py [-h] [--bucket BUCKET] [--cluster_name CLUSTER_NAME]
                                   [--dry_run DRY_RUN]
                                   [--dataproc_version DATAPROC_VERSION]
                                   [--spark_version SPARK_VERSION]
                                   [--hadoop_version HADOOP_VERSION] [--region REGION]
                                   [--user_name USER_NAME] [--os_user OS_USER]
                                   [--pip_mirror PIP_MIRROR] [--application APPLICATION]
          create_configs.py: error: argument --pip_mirror: expected one argument
          sudo -S -p '[sudo] password: ' \cp /tmp/create_configs.py /usr/local/bin/create_configs.py
          sudo -S -p '[sudo] password: ' chmod 755 /usr/local/bin/create_configs.py
          sudo -S -p '[sudo] password: ' mkdir -p /usr/lib/python3.8/datalab/
          mkdir -p /home/datalab-user/datalab_libs/
          scp -i /root/keys/DLab-Dev-key.pem /usr/lib/python3.8/datalab/*.py datalab-user@172.31.16.2:/home/datalab-user/datalab_libs/
          chmod a+x /home/datalab-user/datalab_libs/*
          sudo -S -p '[sudo] password: ' mv /home/datalab-user/datalab_libs/* /usr/lib/python3.8/datalab/
          sudo -S -p '[sudo] password: ' rm -rf /home/datalab-user/datalab_libs/
          sudo -S -p '[sudo] password: ' mkdir -p /usr/lib64/python3.8
          sudo -S -p '[sudo] password: ' ln -fs /usr/lib/python3.8/datalab /usr/lib64/python3.8/datalab
          sudo -S -p '[sudo] password: ' bash -l -c 'echo "[global]" > /etc/pip.conf; echo "proxy = $(cat /etc/profile | grep proxy | head -n1 | cut -f2 -d=)" >> /etc/pip.conf'
          sudo -S -p '[sudo] password: ' bash -l -c 'echo "use_proxy=yes" > ~/.wgetrc; proxy=$(cat /etc/profile | grep proxy | head -n1 | cut -f2 -d=); echo "http_proxy=$proxy" >> ~/.wgetrc; echo "https_proxy=$proxy" >> ~/.wgetrc'
          sudo S -p '[sudo] password: ' bash -l -c 'unset http_proxy https_proxy; export gcp_project_id="or2-msq-epmc-dlab-t1iylu"; export conf_resource="dataengine-service"; /usr/bin/python3 /usr/local/bin/create_configs.py --bucket lfr-gc-23-11-proj1-local-bucket --cluster_name lfr-gc-23-11-proj1-local-des-dp1 --dataproc_version 2.0.0-RC22-ubuntu18 --spark_version 3.0.1 --hadoop_version 3.2.1 --region us-west1 --user_name proj1 --os_user datalab-user --pip_mirror  -application rstudio'
          Traceback (most recent call last):
            File "/root/scripts/rstudio_install_dataengine-service_kernels.py", line 75, in <module>
              conn.sudo('''bash -l -c 'unset http_proxy https_proxy; export gcp_project_id="{0}"; export conf_resource="{1}"; /usr/bin/python3 /usr/local/bin/create_configs.py --bucket {2} --cluster_name {3} --dataproc_version {4} --spark_version {5} --hadoop_version {6} --region {7} --user_name {8} --os_user {9} --pip_mirror {10} --application {11}' '''
            File "<decorator-gen-4>", line 2, in sudo
            File "/usr/local/lib/python3.8/dist-packages/fabric/connection.py", line 30, in opens
              return method(self, *args, **kwargs)
            File "/usr/local/lib/python3.8/dist-packages/fabric/connection.py", line 737, in sudo
              return self._sudo(self._remote_runner(), command, **kwargs)
            File "/usr/local/lib/python3.8/dist-packages/invoke/context.py", line 213, in _sudo
              return runner.run(cmd_str, watchers=watchers, **kwargs)
            File "/usr/local/lib/python3.8/dist-packages/invoke/runners.py", line 380, in run
              return self._run_body(command, **kwargs)
            File "/usr/local/lib/python3.8/dist-packages/invoke/runners.py", line 442, in _run_body
              return self.make_promise() if self._asynchronous else self._finish()
            File "/usr/local/lib/python3.8/dist-packages/invoke/runners.py", line 509, in _finish
              raise UnexpectedExit(result)
          invoke.exceptions.UnexpectedExit: Encountered a bad command exit code!

          Command: 'sudo S -p \'[sudo] password: \' bash -l -c \'unset http_proxy https_proxy; export gcp_project_id="or2-msq-epmc-dlab-t1iylu"; export conf_resource="dataengine-service"; /usr/bin/python3 /usr/local/bin/create_configs.py --bucket lfr-gc-23-11-proj1-local-bucket --cluster_name lfr-gc-23-11-proj1-local-des-dp1 --dataproc_version 2.0.0-RC22-ubuntu18 --spark_version 3.0.1 --hadoop_version 3.2.1 --region us-west1 --user_name proj1 --os_user datalab-user --pip_mirror  -application rstudio\' '

          Exit code: 2

           

          lfrolov Leonid Frolov added a comment - fixed request_id  error, but got new error usage: create_configs.py [-h] [--bucket BUCKET] [--cluster_name CLUSTER_NAME]                           [--dry_run DRY_RUN]                           [--dataproc_version DATAPROC_VERSION]                           [--spark_version SPARK_VERSION]                           [--hadoop_version HADOOP_VERSION] [--region REGION]                           [--user_name USER_NAME] [--os_user OS_USER]                           [--pip_mirror PIP_MIRROR] [--application APPLICATION] create_configs.py: error: argument --pip_mirror: expected one argument sudo -S -p ' [sudo] password: ' \cp /tmp/create_configs.py /usr/local/bin/create_configs.py sudo -S -p ' [sudo] password: ' chmod 755 /usr/local/bin/create_configs.py sudo -S -p ' [sudo] password: ' mkdir -p /usr/lib/python3.8/datalab/ mkdir -p /home/datalab-user/datalab_libs/ scp -i /root/keys/DLab-Dev-key.pem /usr/lib/python3.8/datalab/*.py datalab-user@172.31.16.2:/home/datalab-user/datalab_libs/ chmod a+x /home/datalab-user/datalab_libs/* sudo -S -p ' [sudo] password: ' mv /home/datalab-user/datalab_libs/* /usr/lib/python3.8/datalab/ sudo -S -p ' [sudo] password: ' rm -rf /home/datalab-user/datalab_libs/ sudo -S -p ' [sudo] password: ' mkdir -p /usr/lib64/python3.8 sudo -S -p ' [sudo] password: ' ln -fs /usr/lib/python3.8/datalab /usr/lib64/python3.8/datalab sudo -S -p ' [sudo] password: ' bash -l -c 'echo " [global] " > /etc/pip.conf; echo "proxy = $(cat /etc/profile | grep proxy | head -n1 | cut -f2 -d=)" >> /etc/pip.conf' sudo -S -p ' [sudo] password: ' bash -l -c 'echo "use_proxy=yes" > ~/.wgetrc; proxy=$(cat /etc/profile | grep proxy | head -n1 | cut -f2 -d=); echo "http_proxy=$proxy" >> ~/.wgetrc; echo "https_proxy=$proxy" >> ~/.wgetrc' sudo S -p ' [sudo] password: ' bash -l -c 'unset http_proxy https_proxy; export gcp_project_id="or2-msq-epmc-dlab-t1iylu"; export conf_resource="dataengine-service"; /usr/bin/python3 /usr/local/bin/create_configs.py --bucket lfr-gc-23-11-proj1-local-bucket --cluster_name lfr-gc-23-11-proj1-local-des-dp1 --dataproc_version 2.0.0-RC22-ubuntu18 --spark_version 3.0.1 --hadoop_version 3.2.1 --region us-west1 --user_name proj1 --os_user datalab-user --pip_mirror   -application rstudio' Traceback (most recent call last):   File "/root/scripts/rstudio_install_dataengine-service_kernels.py", line 75, in <module>     conn.sudo('''bash -l -c 'unset http_proxy https_proxy; export gcp_project_id="{0}"; export conf_resource="{1}"; /usr/bin/python3 /usr/local/bin/create_configs.py --bucket {2} --cluster_name {3} --dataproc_version {4} --spark_version {5} --hadoop_version {6} --region {7} --user_name {8} --os_user {9} --pip_mirror {10} --application {11}' '''   File "<decorator-gen-4>", line 2, in sudo   File "/usr/local/lib/python3.8/dist-packages/fabric/connection.py", line 30, in opens     return method(self, *args, **kwargs)   File "/usr/local/lib/python3.8/dist-packages/fabric/connection.py", line 737, in sudo     return self._sudo(self._remote_runner(), command, **kwargs)   File "/usr/local/lib/python3.8/dist-packages/invoke/context.py", line 213, in _sudo     return runner.run(cmd_str, watchers=watchers, **kwargs)   File "/usr/local/lib/python3.8/dist-packages/invoke/runners.py", line 380, in run     return self._run_body(command, **kwargs)   File "/usr/local/lib/python3.8/dist-packages/invoke/runners.py", line 442, in _run_body     return self.make_promise() if self._asynchronous else self._finish()   File "/usr/local/lib/python3.8/dist-packages/invoke/runners.py", line 509, in _finish     raise UnexpectedExit(result) invoke.exceptions.UnexpectedExit: Encountered a bad command exit code! Command: 'sudo S -p \' [sudo] password: \' bash -l -c \'unset http_proxy https_proxy; export gcp_project_id="or2-msq-epmc-dlab-t1iylu"; export conf_resource="dataengine-service"; /usr/bin/python3 /usr/local/bin/create_configs.py --bucket lfr-gc-23-11-proj1-local-bucket --cluster_name lfr-gc-23-11-proj1-local-des-dp1 --dataproc_version 2.0.0-RC22-ubuntu18 --spark_version 3.0.1 --hadoop_version 3.2.1 --region us-west1 --user_name proj1 --os_user datalab-user --pip_mirror   -application rstudio\' ' Exit code: 2  
          lfrolov Leonid Frolov added a comment -

          tested for rstudio. should also work for zeppellin if there are no other bug

          lfrolov Leonid Frolov added a comment - tested for rstudio. should also work for zeppellin if there are no other bug
          vvitanska Vira Vitanska added a comment -

          Closed:
          Commit ID 47d5c395725cc9634350b701a9eaad1464125403.

          vvitanska Vira Vitanska added a comment - Closed: Commit ID 47d5c395725cc9634350b701a9eaad1464125403.

          People

            lfrolov Leonid Frolov
            vvitanska Vira Vitanska
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: