Support

Troubleshooting Launcher in RStudio Server Pro

Follow
Launcher is a new feature of RStudio Server Pro v1.2 that is only available under named user licensing. RStudio Server Pro version 1.2 without Launcher is available under existing server based licensing. For questions about using Launcher with RStudio Server Pro, please contact sales@rstudio.com.

 

Verifying RStudio Server Pro with Launcher and Kubernetes

Run the following command to test the installation and configuration of RStudio Server Pro with Launcher and Kubernetes:

sudo rstudio-server stop
sudo rstudio-server verify-installation --verify-user=<USER>
sudo rstudio-server start

Replace <USER> with a valid username of a user that is setup to run RStudio Server Pro in your installation.

Refer to the Troubleshooting section in the RStudio Server Pro Administration Guide for more information on using the Launcher verification tool.

 

Troubleshooting Kubernetes API connectivity issues

If you encounter errors in RStudio Server Pro and Launcher such as:

29 Apr 2019 09:34:05 [rserver] ERROR system error 111 (Connection refused);
OCCURRED AT: void rstudio::core::http::TcpIpAsyncConnector::handleConnect(const
rstudio_boost::system::error_code&,
rstudio_boost::asio::ip::basic_resolver<rstudio_boost::asio::ip::tcp>::iterator)
/var/lib/jenkins/workspace/IDE_pro-pipeline_v1.2/src/cpp/core/include/core/http/TcpIpAsyncConnector.hpp:197;
LOGGED FROM: int main(int, char* const*)
/var/lib/jenkins/workspace/IDE_pro-pipeline_v1.2/src/cpp/server/ServerMain.cpp:578

or

May 30 18:34:51 ip-10-20-0-91 rstudio-kubernetes-launcher[2166]:
ERROR asio.ssl error 336134278 (certificate verify failed);
OCCURRED AT:
void rstudio::core::http::TcpIpAsyncClientSsl::handleHandshake(const
rstudio_boost::system::error_code&)
/var/lib/jenkins/workspace/IDE_pro-pipeline_v1.2/src/cpp/core/include/core/http/TcpIpAsyncClientSsl.hpp:121;
LOGGED FROM: int
rstudio::job_launcher::impls::entry_point::run(rstudio_boost::shared_ptr<rstudio::job_launcher::impls::ApiBase>,
rstudio_boost::shared_ptr<rstudio::job_launcher::impls::FrameworkCommunicator>)
/var/lib/jenkins/workspace/IDE_pro-pipeline_v1.2/src/cpp/job_launcher/impls/EntryPoint.cpp:134

This might be due to a misconfigured Kubernetes CA certificate on the machine where RStudio Server Pro and Launcher are running.

You can try the following steps to troubleshoot this scenario:

1) Ensure that the Kubernetes CA certificate is not base64 encoded and that it starts with -----BEGIN CERTIFICATE-----.

2) Ensure that you've added the Kubernetes CA certificate to your system certificate chain by placing the certificate file in /usr/share/pki/ca-trust-source/anchors/cert-eks-cluster.crt (on RHEL/CentOS) or /usr/local/share/ca-certificates (on Ubuntu). Then run sudo update-ca-trust (on RHEL/CentOS) or sudo update-ca-certificates (on Ubuntu).

3) Skip the CA certificate verification by adding verify-ssl-certs=0 to /etc/rstudio/launcher.kubernetes.conf, then restart rstudio-server and rstudio-launcher and start a new session. Note that for production use, you should always have this enabled, but it can be disabled for testing purposes.

 

Troubleshooting NFS mount issues

If you encounter errors in RStudio Server Pro and Launcher when starting a new session related to NFS mounts, or access to /home directories, then this might be due to a misconfigured NFS server or /home directory export.

You can try the following steps to troubleshoot this scenario:

1) Ensure that your NFS server is exporting the /home directory.

2) Ensure that the server with RStudio Server Pro and Launcher has mounted the /home directory that was exported from the NFS share.

3) Ensure that user IDs and group IDs match across the NFS server and machine with RStudio Server Pro and Launcher.

4) Mount the NFS home dir from another machine to verify connectivity, access, and permissions are configured correctly in the NFS share.

 

Troubleshooting Kubernetes runtime / worker node issues

If you encounter errors in RStudio Server Pro and Launcher when starting a new session related to insufficient CPU/memory, no worker nodes, or no resources available to schedule pods, then this might be due to a lack of Kubernetes worker nodes or a lack of adequate CPU/RAM on the Kubernetes worker nodes.

For example, when you create a Kubernetes cluster in Amazon EKS, this will only provision the Kubernetes control plane. You need to perform additional steps to provision EC2 instances and join them as worker nodes on the Amazon EKS cluster.

You can try the following steps to troubleshoot this scenario:

1) Run the command kubectl get nodes and ensure that you have worker nodes with adequate resources to run containers/pods (e.g., 4 CPUs, 16 GB RAM).

2) Add worker nodes with additional CPU/RAM to your Kubernetes cluster and start a new R session.

 

General troubleshooting

The logs for RStudio Server Pro and Launcher can be found at:

  • /var/lib/rstudio-server/monitor/log/rstudio-server.log
  • /var/lib/rstudio-launcher/rstudio-launcher.log
  • /var/lib/rstudio-launcher/Kubernetes/rstudio-kubernetes-launcher.log

Symptom: The user receives no prompt to choose a Cluster before starting an interactive session. Or, the button in the Jobs pane says Start Local Job, instead of Start Launcher Job

Confirm that the RStudio License currently being used is authorized for use with the Launcher service. Run the following command:

sudo rstudio-server license-manager status

The output should include a line that says Enable-Launcher, and it should be set to 1. If it is not, please contact the Sales team at: sales@rstudio.com

Here is a sample of what the output of license-manager status looks like, and where approximately the configuration line to look for is located:

RStudio License Manager 1.2.1287-1
-- Local license status --
Status: Activated
Product-Key: [LICENSE KEY]
Has-Key: Yes
Has-Trial: No
Enable-Launcher: 1
Users: 0
Sessions: 0
Expiration: 2025-12-31 00:00:00
Days-Left: 2508
License-Scope: System
-- Floating license status --
License server not in use.

Kubernetes 

Symptom: New R sessions will not start. They either go to Killed status, or get stuck on Pending status.

Try manually running the Docker image in the machine where the image was built. There are at least two things to check for:

- A running R installation

- The necessary parts of RStudio Server Pro

There are several ways to check for this.  As a Docker user, the R admin may have preferences on how to verify these two things.  Here is a suggestion for a very simple way to check:

1. Start an interactive Docker session with the image

sudo docker run -it [NAME OF YOUR IMAGE] bash

 

2. Confirm that the RStudio Server files are installed by listing the files under /usr/lib/rstudio-server

ls /usr/lib/rstudio-server

Here is an example of what the contents should look like:

root@4a59013e6c7d:~# ls /usr/lib/rstudio-server

bin extras NOTICE R UPSTREAM_VERSION www www-launcher www-workspaces
conf LICENSE NOTICE-SUPPLEMENT resources VERSION www-admin www-symbolmaps

3. Confirm that there is at least one working version of R installed by running the R command and starting an interactive session.

Here is an example of the output, use q() to exit the R session.

root@4a59013e6c7d:~# R

R version 3.4.4 (2018-03-15) -- "Someone to Lean On"
Copyright (C) 2018 The R Foundation for Statistical Computing
...
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.
> q()
Save workspace image? [y/n/c]: n

4. Leave the interactive session with the following keyboard shortcuts: CTRL+p and then CTRL+q

Symptom: New R sessions start, but all jobs started via the Start Launcher Job button Fail. And the output message says: /bin/R not found

The most likely cause is that even though there are one or more versions of R installed in the image, there is no one version setup as default.  The way to do this is by creating a link to /bin/R. Here is an example of how to accomplish that inside the Docker file:

 RUN ln -s /opt/R/R-3.5.1/bin/R /bin/R

 

 

Comments