Support

Using Job Json Overrides with RStudio Workbench / RStudio Server Pro and Kubernetes

Follow

RStudio Workbench (previously RStudio Server Pro), when configured with the RStudio Job Launcher, can start user sessions (RStudio, Jupyter, Jupyter Lab) on Kubernetes running inside of containers.

When it launches user sessions, it creates a "job" on Kubernetes. If you need to customize that "job spec" that gets delivered to Kubernetes, then you can use "job json overrides" to alter the job spec before it gets sent.

A more robust option would be to use Kubernetes Operators , but that is outside the scope of this article. Instead, we will explore a handful of use cases for job-json-overrides below.

Formal documentation on this topic is available in the RStudio Job Launcher Admin Guide.

Kubernetes Profiles

The job-json-overrides specification lives within the "Launcher Kubernetes Profiles" convention. This allows specifying different job execution parameters by linux username or linux group name. These specifics are additive, so it is important to understand how they operate. You can read more about this configuration file here. By convention, it lives at /etc/rstudio/launcher.kubernetes.profiles.conf.

Warning

It is important to be aware that changing the job spec for RStudio Workbench Launcher Jobs can break RStudio Workbench's successful launching of jobs in many different ways. For instance:

- If you remove or alter important attributes that RStudio Workbench depends on

- If you create an invalid job spec

Also, it is worth noting that today we do not have any utilities to make this modification process testable. As a result, we recommend using job-json-overrides sparingly, and only to add functionality to the job spec (instead of overwriting or removing).

Important Note

Each option below is illustrated as a standalone example. You can have multiple job-json-overrides specified for the same user / group by comma-delimiting them.

job-json-overrides="/spec/path":"/file/path","/spec/path2":"/file/path2"

Setting for Particular Users or Groups

Any of the examples below can be set for everyone on the server or for particular users or groups.

Each block of job-json-overrides specification belongs to the user or group specified in the header of that block. Users are specified by their username, and groups are specified by the group name prefixed with the @ symbol.

You can have as many block as users or groups you want to configure.

For example,

/etc/rstudio/launcher.kubernetes.profiles.conf

[*]
job-json-overrides="/spec/template/spec/serviceAccountName":"/etc/rstudio/job-json-overrides/defaultServiceAccount.json"

[@group]
job-json-overrides="/spec/template/spec/serviceAccountName":"/etc/rstudio/job-json-overrides/groupServiceAccount.json"

[user]
job-json-overrides="/spec/template/spec/serviceAccountName":"/etc/rstudio/job-json-overrides/userServiceAccount.json"

Examples

To get started with our examples, create the /etc/rstudio/job-json-overrides directory. This is the directory that we will store configuration within

# mkdir -p /etc/rstudio/job-json-overrides

Run the pod as "privileged"

Configure Job Json Overrides

In certain cases (debugging, some low level system library is in use, etc.), it is important that the pod is running with elevated privileges. To set this up, add the following job-json-overrides line to the appropriate group in your launcher kubernetes profiles configuration. For our example, we will set this for all users on the server

/etc/rstudio/launcher.kubernetes.profiles.conf

 [*]
allow-unknown-images=1
job-json-overrides="/spec/template/spec/containers/0/securityContext":"/etc/rstudio/job-json-overrides/securityContext"

Create an Override File

Now, create the file /etc/rstudio/job-json-overrides/securityContext with the following contents:

/etc/rstudio/job-json-overrides/securityContext

{
"privileged": true
}

 

Verify

Now restart the rstudio-launcher service, and new jobs should be launched as privileged containers! You can confirm using kubectl once a job has been launched

# kubectl -n rstudio get pod mypodname -o yaml | grep securityContext -A 1
securityContext:
privileged: true
--
securityContext: {}
serviceAccount: default

Attach a ConfigMap

Config Maps can be useful for including configuration into pods. This is particularly useful for the rsession.conf file and other related configuration that might be helpful for users (without building that configuration into the image itself).

Create a Config Map

First, you must create a config map with the appropriate input that you want to mount into your containers.

Given a file at session-config.yaml with the following contents:

session-config.yaml

apiVersion: v1
data:
rsession.conf: |
default-rsconnect-server=https://my.connect.example.com
kind: ConfigMap
metadata:
name: session-config
namespace: rstudio

You can create the configmap with:

kubectl apply -f session-config.yaml

 

Configure Job Json Overrides

Create an appropriate line for job-json-overrides in your Launcher Kubernetes Profiles configuration. In this case, we need two job-json-overrides: one for the volume, and one to mount the volume to the container.

/etc/rstudio/launcher.kubernetes.profiles.conf

[*]
allow-unknown-images=1
job-json-overrides="/spec/template/spec/volumes/-":"/etc/rstudio/job-json-overrides/volumes","/spec/template/spec/containers/0/volumeMounts/-":"/etc/rstudio/job-json-overrides/volumeMounts"

 

Create Each Override File

Now create the following two files in /etc/rstudio/job-json-overrides (making any modifications as needed for the files within your configmap):

/etc/rstudio/job-json-overrides/volumes

{
"configMap": {
"name": "session-config",
"items": [
{
"path": "rsession.conf",
"key": "rsession.conf"
}
]
},
"name": "session-config"
}

/etc/rstudio/job-json-overrides/volumeMounts

{
"mountPath": "/etc/rstudio/rsession.conf",
"name": "session-config",
"subPath": "rsession.conf"
}

 

Verify

That should do it! Now restart the rstudio-launcher service and launch a new job. You should be able to see the rsession.conf file from your configmap at the path specified.

 

Change the imagePullPolicy

imagePullPolicy can be useful, especially during setup or iteration on building a session image. By default, we set imagePullPolicy to "IfNotPresent," which will not re-pull an image if it is cached on a host. For debugging purposes (or some security purposes), imagePullPolicy: "Always" can be preferred.

 

You can set this for all jobs, certain users, or certain groups by using the following pattern.

 

Create the Override File

 

/etc/rstudio/job-json-overrides/imagePullPolicy.json

"Always"

 

Configure JobJsonOverrides

 

/etc/rstudio/launcher.kubernetes.profiles.conf

[*]
job-json-overrides="/spec/template/spec/containers/0/imagePullPolicy":"/etc/rstudio/job-json-overrides/imagePullPolicy.json"

 

Verify

That should do it! Now restart the rstudio-launcher service and launch a new job. You should be able to see the imagePullPolicy set to "Always".

 

Set imagePullSecrets

For pulling restricted images, setting imagePullSecrets is often required. You can also manage such access by giving the service account associated with a job the appropriate pull secret.

 

If you create a secret called "my-pull-secret" in the same namespace as RStudio's launcher jobs, then you can set the job's imagePullSecrets as follows:

 

Create the Overrides File

Note: this replaces all imagePullSecrets. You can also append using the patterns illustrated above (see /-)

/etc/rstudio/job-json-overrides/imagePullSecrets.json

[
{"name": "my-pull-secret"}
]

 

Configure JobJsonOverrides

/etc/rstudio/launcher.kubernetes.profiles.conf

[*]
job-json-overrides="/spec/template/spec/imagePullSecrets":"/etc/rstudio/job-json-overrides/imagePullSecrets.json"

 

Verify

That should do it! Now restart the rstudio-launcher service and launch a new job. You should be able to see the imagePullSecrets set to include the "my-pull-secret" secret, and any authenticated image repositories should be accessible for your job.

 

 

Set the serviceAccountName for the job

Setting the service account per user, per group, or for default jobs can be useful for accessing other restricted resources within your Kubernetes cluster or other infrastructure. Below, we will set the serviceAccount for all jobs to "superServiceAccount."


This can be another pattern for pulling restricted images, as some service accounts have a pull secret associated with them.

 

Create the Override File

 

/etc/rstudio/job-json-overrides/myServiceAccount.json

"superServiceAccount"

 

Configure JobJsonOverrides

 

/etc/rstudio/launcher.kubernetes.profiles.conf

[*]
job-json-overrides="/spec/template/spec/serviceAccountName":"/etc/rstudio/job-json-overrides/myServiceAccount.json"

Verify

That should do it! Now restart the rstudio-launcher service and launch a new job. You should be able to see the serviceAccountName for the job set to "superServiceAccount"

 

Set an annotation

Setting an annotation on a pod can be helpful for monitoring resources or integrating with other tools. Unfortunately, Job Json Overrides is static configuration, so it is not possible to generate dynamic annotation values (i.e. username or project name) via this configuration.

Create the Override File

/etc/rstudio/job-json-overrides/annotationValue.json

"my-annotation-value"

Configure JobJsonOverrides

An important tip: "~1" is a subsitution for "/" in the JSON pointer RFC. This is often used for the key of an annotation.

/etc/rstudio/launcher.kubernetes.profiles.conf

[*]
job-json-overrides="/spec/template/metadata/annotations/my.company.com~1key":"/etc/rstudio/job-json-overrides/annotationValue.json"

Verify

That should do it! Save your changes and restart the RStudio Launcher service. When new jobs are launched, you should see the following annotation on your jobs:

annotations:
my.company.com/key: my-annotation-value

 

Set NodeAffinity

Setting node affinity for certain users and groups can be helpful for putting certain types of workloads into a different part of the Kubernetes cluster, if disruption on those nodes is less frequent or other features of the nodes are better set up for data science work.

 

Create the Override File

Node affinities allow you to use any annotation or other attributes to select relevant nodes. You can read about options in this linked article and build your own selectors or use the example below, which selects based on hostname.

 

/etc/rstudio/job-json-overrides/nodeAffinity.json

{
"nodeAffinity": {
"requiredDuringSchedulingIgnoredDuringExecution": {
"nodeSelectorTerms": [
{
"matchExpressions": [
{
"key": "kubernetes.io/hostname",
"operator": "In",
"values": [
"node-hostname"
]
}
]
}
]
}
}
}

 

Configure JobJsonOverrides

 

/etc/rstudio/launcher.kubernetes.profiles.conf

[*]
job-json-overrides="/spec/template/spec/affinity":"/etc/rstudio/job-json-overrides/nodeAffinity.json"

 

Verify

That should do it! Save your changes and restart the RStudio Launcher service. If you launch several jobs now, you should be able to see jobs for your selected user/group (in the example above, everyone) launching with the appropriate nodeAffinity and being scheduled on the appropriate nodes.

 

Tips

  • The JSON pointer RFC is a useful (if highly technical) reference
  • Test on a staging server first!
  • When appending to an array, you must either specify an index (i.e. [0] for the first item) or a "-" to specify "append to the array." You can see this in use for the "Create a Config Map" example. It is also possible to overwrite the entire array, but this is usually discouraged, lest you overwrite something else that is important
  • If you are creating an annotation or label or other JSON key with a slash in it, you can use "~1" to mask the "/" character. I.e. "some~1key" will become "some/key." The "~0" combination becomes a "~". This is described in the JSON pointer RFC.
  • To combine many of these approaches, combine several "key":"value" pairs with commas, i.e. 
job-json-overrides="key":"value","key1":"value1","key2":"value2"
  • We would love to hear about your use case for this feature! Please reach out to your account representative, our support team, or our community forum to chat with us!

Comments