Verifying RStudio Server Pro with Launcher and Kubernetes
Run the following command to test the installation and configuration of RStudio Server Pro with Launcher and Kubernetes:
sudo rstudio-server stop
sudo rstudio-server verify-installation --verify-user=<USER>
sudo rstudio-server start
Replace <USER> with a valid username of a user that is setup to run RStudio Server Pro in your installation.
Refer to the Troubleshooting section in the RStudio Server Pro Administration Guide for more information on using the Launcher verification tool.
General troubleshooting
The logs for RStudio Server Pro and Launcher can be found at:
/var/lib/rstudio-server/monitor/log/rstudio-server.log/var/lib/rstudio-launcher/rstudio-launcher.log/var/lib/rstudio-launcher/Kubernetes/rstudio-kubernetes-launcher.log
Symptom: The user receives no prompt to choose a Cluster before starting an interactive session. Or, the button in the Jobs pane says Start Local Job, instead of Start Launcher Job
Confirm that the RStudio License currently being used is authorized for use with the Launcher service. Run the following command:
sudo rstudio-server license-manager status
The output should include a line that says Enable-Launcher, and it should be set to 1. If it is not, please contact the Sales team at: sales@rstudio.com
Here is a sample of what the output of license-manager status looks like, and where approximately the configuration line to look for is located:
RStudio License Manager 1.2.1287-1
-- Local license status --
Status: Activated
Product-Key: [LICENSE KEY]
Has-Key: Yes
Has-Trial: No
Enable-Launcher: 1
Users: 0
Sessions: 0
Expiration: 2025-12-31 00:00:00
Days-Left: 2508
License-Scope: System
-- Floating license status --
License server not in use.
Troubleshooting Kubernetes API connectivity issues
If you encounter errors in RStudio Server Pro and Launcher such as:
29 Apr 2019 09:34:05 [rserver] ERROR system error 111 (Connection refused);
OCCURRED AT: void rstudio::core::http::TcpIpAsyncConnector::handleConnect(const
rstudio_boost::system::error_code&,
rstudio_boost::asio::ip::basic_resolver<rstudio_boost::asio::ip::tcp>::iterator)
/var/lib/jenkins/workspace/IDE_pro-pipeline_v1.2/src/cpp/core/include/core/http/TcpIpAsyncConnector.hpp:197;
LOGGED FROM: int main(int, char* const*)
/var/lib/jenkins/workspace/IDE_pro-pipeline_v1.2/src/cpp/server/ServerMain.cpp:578
or
May 30 18:34:51 ip-10-20-0-91 rstudio-kubernetes-launcher[2166]:
ERROR asio.ssl error 336134278 (certificate verify failed);
OCCURRED AT:
void rstudio::core::http::TcpIpAsyncClientSsl::handleHandshake(const
rstudio_boost::system::error_code&)
/var/lib/jenkins/workspace/IDE_pro-pipeline_v1.2/src/cpp/core/include/core/http/TcpIpAsyncClientSsl.hpp:121;
LOGGED FROM: int
rstudio::job_launcher::impls::entry_point::run(rstudio_boost::shared_ptr<rstudio::job_launcher::impls::ApiBase>,
rstudio_boost::shared_ptr<rstudio::job_launcher::impls::FrameworkCommunicator>)
/var/lib/jenkins/workspace/IDE_pro-pipeline_v1.2/src/cpp/job_launcher/impls/EntryPoint.cpp:134
This might be due to a misconfigured Kubernetes CA certificate on the machine where RStudio Server Pro and Launcher are running.
You can try the following steps to troubleshoot this scenario:
1) Ensure that the Kubernetes CA certificate is not base64 encoded and that it starts with -----BEGIN CERTIFICATE-----.
2) Ensure that you've added the Kubernetes CA certificate to your system certificate chain by placing the certificate file in /usr/share/pki/ca-trust-source/anchors/cert-eks-cluster.crt (on RHEL/CentOS) or /usr/local/share/ca-certificates (on Ubuntu). Then run sudo
update-ca-trust (on RHEL/CentOS) or sudo update-ca-certificates (on Ubuntu).
3) Skip the CA certificate verification by adding verify-ssl-certs=0 to /etc/rstudio/launcher.kubernetes.conf, then restart rstudio-server and rstudio-launcher and start a new session. Note that for production use, you should always have this enabled, but it can be disabled for testing purposes.
Troubleshooting NFS mount issues
If you encounter errors in RStudio Server Pro and Launcher when starting a new session related to NFS mounts, or access to /home directories, then this might be due to a misconfigured NFS server or /home directory export.
You can try the following steps to troubleshoot this scenario:
1) Ensure that your NFS server is exporting the /home directory.
2) Ensure that the server with RStudio Server Pro and Launcher has mounted the /home directory that was exported from the NFS share.
3) Ensure that user IDs and group IDs match across the NFS server and machine with RStudio Server Pro and Launcher.
4) Mount the NFS home dir from another machine to verify connectivity, access, and permissions are configured correctly in the NFS share.
Troubleshooting Kubernetes runtime / worker node issues
If you encounter errors in RStudio Server Pro and Launcher when starting a new session related to insufficient CPU/memory, no worker nodes, or no resources available to schedule pods, then this might be due to a lack of Kubernetes worker nodes or a lack of adequate CPU/RAM on the Kubernetes worker nodes.
For example, when you create a Kubernetes cluster in Amazon EKS, this will only provision the Kubernetes control plane. You need to perform additional steps to provision EC2 instances and join them as worker nodes on the Amazon EKS cluster.
You can try the following steps to troubleshoot this scenario:
1) Run the command kubectl get nodes and ensure that you have worker nodes with adequate resources to run containers/pods (e.g., 4 CPUs, 16 GB RAM).
2) Add worker nodes with additional CPU/RAM to your Kubernetes cluster and start a new R session.
New sessions are not starting in Kubernetes
Symptom: New R sessions will not start. They either go to Killed status, or get stuck on Pending status.
Try manually running the Docker image in the machine where the image was built. There are at least two things to check for:
- A running R installation
- The necessary parts of RStudio Server Pro
There are several ways to check for this. As a Docker user, the R admin may have preferences on how to verify these two things. Here is a suggestion for a very simple way to check:
1. Start an interactive Docker session with the image
sudo docker run -it [NAME OF YOUR IMAGE] bash
2. Confirm that the RStudio Server files are installed by listing the files under /usr/lib/rstudio-server
ls /usr/lib/rstudio-server
Here is an example of what the contents should look like:
root@4a59013e6c7d:~# ls /usr/lib/rstudio-server
bin extras NOTICE R UPSTREAM_VERSION www www-launcher www-workspaces
conf LICENSE NOTICE-SUPPLEMENT resources VERSION www-admin www-symbolmaps
3. Confirm that there is at least one working version of R installed by running the R command and starting an interactive session.
Here is an example of the output, use q() to exit the R session.
root@4a59013e6c7d:~# R
R version 3.4.4 (2018-03-15) -- "Someone to Lean On"
Copyright (C) 2018 The R Foundation for Statistical Computing
...
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.
> q()
Save workspace image? [y/n/c]: n
4. Leave the interactive session with the following keyboard shortcuts: CTRL+p and then CTRL+q
Symptom: New R sessions start, but all jobs started via the Start Launcher Job button Fail. And the output message says: /bin/R not found
The most likely cause is that even though there are one or more versions of R installed in the image, there is no one version setup as default. The way to do this is by creating a link to /bin/R. Here is an example of how to accomplish that inside the Docker file:
RUN ln -s /opt/R/R-3.5.1/bin/R /bin/R
Comments