Support

Update Kubernetes Launcher Templates

Follow

Launcher 2.4.0 (released with RStudio Workbench 2022.02.0) added the capability to use templates to configure Launcher jobs on Kubernetes. Launcher 2.5.0 (released with Workbench 2022.07.0) and later versions include updates to the templates to provide new features and bug fixes.

This article will help you upgrade your existing templates to the latest version of the templates supported by the Launcher version you have installed.

Kubernetes Templates

Launcher can be configured to use YAML templates for jobs submitted to Kubernetes. This is enabled if the following setting is present.

/etc/rstudio/launcher.kubernetes.conf

use-templating=1

With this setting enabled Launcher will automatically generate templates the first time it is run unless the templates already exist.

Environment Setup

Note: This article assumes that all executables run in the following steps is in your $PATH

The below examples use a common set of shell variables. Set these shell variables to the appropriate scratch path and cluster name.

# Set scratch path to the location matching the current Launcher configuration
SCRATCH_PATH=/var/lib/rstudio-launcher
# Use the Cluster Name from launcher.conf
CLUSTER=MyCluster

 

Update Base (Unmodified) Templates

Important: If you have made any modifications to customize the templates, skip to the next section for more detailed steps.

If no modifications have been made to the base templates shipped with Launcher, preparation for the updated templates can be made before Launcher is upgraded. This can be performed while Launcher is still running since the Launcher process holds the templates in memory.

Note: This will need to be repeated for each Kubernetes cluster defined in /etc/rstudio/launcher.conf

# Backup current templates
mv ${SCRATCH_PATH}/${CLUSTER}/job.tpl ${SCRATCH_PATH}/${CLUSTER}/job.tpl.bak
mv ${SCRATCH_PATH}/${CLUSTER}/service.tpl ${SCRATCH_PATH}/${CLUSTER}/service.tpl.bak
Launcher will generate upgraded templates the next time it is restarted. To restart the Launcher:
sudo rstudio-launcher restart

 

Update Modified Templates used with Workbench

The following steps outline a process to manually update to the latest version of the Launcher Kubernetes templates and maintain any modifications.

In order to

1. Stop the Launcher

sudo rstudio-launcher stop

2. Create a backup of the existing templates and examine the differences:

Note: Steps 2-4 will need to be repeated for each Kuberentes cluster defined in /etc/rstudio/launcher.conf

# Backup current templates
mv ${SCRATCH_PATH}/${CLUSTER}/job.tpl ${SCRATCH_PATH}/${CLUSTER}/job.tpl.bak
mv ${SCRATCH_PATH}/${CLUSTER}/service.tpl ${SCRATCWorkbench 2021H_PATH}/${CLUSTER}/service.tpl.bak

# Re-generate the templates
# This requires the new version of Launcher to be installed
sudo rstudio-kubernetes-launcher --generate-templates --plugin-name ${CLUSTER}

# View differences
diff -U 1 ${SCRATCH_PATH}/${CLUSTER}/job.tpl.bak ${SCRATCH_PATH}/${CLUSTER}/job.tpl
diff -U 1 ${SCRATCH_PATH}/${CLUSTER}/service.tpl.bak ${SCRATCH_PATH}/${CLUSTER}/service.tpl

The diff output will show additions/changes from new template versions as well as any customization in the existing templates.

The following example shows the diff output for a template that has been modified to use Workload Identities on Google Cloud with the existing customization highlighted in bold for emphasis.

@@ -1,2 +1,2 @@
-# Version: 1003
.
...+0# Version: 2.1.0
apiVersion: batch/v1
@@ -4,2 +4,14 @@
metadata:
+ annotations:
+ {{- with .Job.metadata.job.annotations }}
+ {{- range $key, $val := . }}
+ {{ $key }}: {{ toYaml $val | indent 4 | trimPrefix (repeat 4 " ") }}
+ {{- end }}
+ {{- end }}
+ labels:
+ {{- with .Job.metadata.job.labels }}
+ {{- range $key, $val := . }}
+ {{ $key }}: {{ toYaml $val | indent 4 | trimPrefix (repeat 4 " ") }}
+ {{- end }}
+ {{- end }}
generateName: {{ toYaml .Job.generateName }}
(... omitted for brevity ...)
@@ -36,3 +62,2 @@
nodeSelector:
- iam.gke.io/gke-metadata-server-enabled: "true"
{{- range .Job.placementConstraints }}
(.. omited for brevity ..)

3. Apply the modifications to the new template.

4. Ensure that the final template is valid:

sudo rstudio-kubernetes-launcher --validate-templates --plugin-name ${CLUSTER}

 

Further Details: Template Validation and Versions

Each time Launcher starts it will examine any pre-existing templates to ensure they are compatible with running version of Launcher. If they are incompatible Launcher will fail to start and produce log output similar to:

{
 "time": "2022-06-17T15:45:34.935240Z",
 "service": "rstudio-kubernetes-launcher",
 "level": "ERROR",
 "error": {
   "code": 5,
   "location": {
     "file": "src/cpp/job_launcher/impls/kubernetes/KubernetesTemplating.cpp",
    "function": "rstudio::core::Error rstudio::job_launcher::impls::kubernetes::templating::validateTemplateVersions(bool)",
     "lineNumber": 440
   },
   "message": "Input/output error",
   "type": "system",
   "properties": {
    "description": "Error validating template version: template file /var/lib/rstudio-launcher/MyCluster/job.tpl version 1 is incompatible with expected version 2.1.0"
   }
 },
 "location": {
   "file": "src/cpp/job_launcher/impls/kubernetes/KubernetesLauncherMain.cpp",
   "function": "int main(int, char**)",
   "lineNumber": 286
 },
 "logSource": "MyCluster"
}

 

Run the following command to validate the templates currently on disk:

sudo rstudio-kubernetes-launcher --validate-templates --plugin-name ${CLUSTER}

Template Versions

Launcher uses Semantic Versioning for its Kubernetes templates. This allows the installed version of Launcher to be upgraded without having to update pre-existing templates with the same MAJOR version.

Templates must use the same MAJOR version that Launcher expects. Assuming Launcher expects template version 2.1.2, a mismatch will produce template validation output similar to:

Error validating template version: template file /var/lib/rstudio-launcher/MyCluster/job.tpl
version 1 is incompatible with expected version 2.1.2

A mismatch in the MINOR version will produce template validation output similar to:

job.tpl validation warnings occurred
  * template file /var/lib/rstudio-launcher/MyCluster/job.tpl version 2.0.0
   
does not include new features available in version 2.1.2

A mismatch in the PATCH version will produce template validation output similar to:

service.tpl validation warnings occurred
* template file /var/lib/rstudio-launcher/MyCluster/service.tpl version 2.1.0
 
does not include bug fixes available in version 2.1.2

Similar messages will also be reported as a WARNING in the Launcher Logs

{
 "time": "2022-06-17T15:47:09.697129Z",
 "service": "rstudio-kubernetes-launcher",
 "level": "WARNING",
"message": "template file /var/lib/rstudio-launcher/MyCluster/job.tpl version 2.0.0 does not include new features available in version 2.1.2",
 "location": {
   "file": "src/cpp/job_launcher/impls/kubernetes/KubernetesTemplating.cpp",
  "function": "rstudio::core::Error rstudio::job_launcher::impls::kubernetes::templating::validateTemplateVersions(bool)",
   "lineNumber": 431
 },
 "logSource": "MyCluster"
}
{
 "time": "2022-06-17T15:47:09.697422Z",
 "service": "rstudio-kubernetes-launcher",
 "level": "WARNING",
"message": "template file /var/lib/rstudio-launcher/MyCluster/service.tpl version 2.1.0 does not include bug fixes available in version 2.1.2",
 "location": {
   "file": "src/cpp/job_launcher/impls/kubernetes/KubernetesTemplating.cpp",
  "function": "rstudio::core::Error rstudio::job_launcher::impls::kubernetes::templating::validateTemplateVersions(bool)",
   "lineNumber": 431
 },
 "logSource": "MyCluster"
}

Comments