Building an ML Platform on Kubernetes: Yandex Cloud, JupyterHub, Dask and S3”

Hi! I’m Alisa, a DevOps Engineer. This is my translation to English from my original article.

In this article, I want to tell you about one of our recent projects, where I built infrastructure for a team of data engineers and analysts. I want to mention from the beginning that this was not a platform for production model inference. It was mainly an environment for research and experiments.

So, I want to share our practical experience of building scalable infrastructure with autoscaling. If this topic is relevant to you, keep reading.

Table of Contents

• Task

• General Architecture

• GPU and Operator

• JupyterHub as an Entry Point

• Data Access (S3 via CSI)

• Main Problem: S3 Cold Start

• Distributed Computing (Dask)

• Conclusion

Task

The project was created for a team of data analysts. Their main use case was analyzing large amounts of data stored in S3-compatible storage.

The requirements were:

• interactive work through Jupyter;

• access to large datasets without copying them from the bucket to the user’s machine;

• the ability to use GPUs when needed, but not all the time;

• distributed computing for heavy tasks;

• a multi-user environment with 5–6 regular users and some occasional users;

• a wide choice of resources without paying for them when they are not in use.

The company already had a Kubernetes cluster, but I decided to move the ML workloads to a separate cluster. This allowed us to isolate resource-intensive analytics tasks and prevent them from affecting other services that were not related to data processing.

General Architecture

The architecture was quite simple, but it covered all our requirements:

• Managed Kubernetes cluster;

• node groups:

  • CPU nodes of different sizes;

  • GPU nodes (V100);

  • Dask nodes (worker/scheduler);

• JupyterHub as the user entry point;

• NVIDIA GPU Operator for GPU management;

• Dask for distributed computing;

• S3 (Object Storage) + CSI for data access.

The basic workflow is:

User → JupyterHub → pod → required node group → data access through S3 → connection to Dask if needed.

The Kubernetes master node does not have a public IP address, so access to the API is limited to private networks available through the VPC.

The access flow looks like this:

1. The user connects through VPN.

2. There is a load balancer with an IP address from the subnet where the cluster is located. JupyterHub and Dask are available only through internal DNS names, which resolve to the load balancer IP.

3. The load balancer forwards traffic to the NodePorts of the cluster nodes. Then the traffic reaches the ingress controller, which routes it to the required services.

The compute layer of the cluster is divided into several node groups. Each group is used for a specific type of workload:

• small infrastructure nodes for system services, such as monitoring and certificate management;

• CPU nodes of different sizes, such as mini, small, and large, for user notebooks;

• GPU nodes for tasks that need hardware acceleration;

• separate node groups for Dask, so distributed computing pods do not compete for resources with user or infrastructure pods.

This separation allows us to isolate different types of workloads and makes the system behavior more predictable under load.

Both user and system pods are placed using standard Kubernetes mechanisms:

• labels for marking nodes;

• taints and tolerations for controlling pod placement.

resource "yandex_kubernetes_node_group" "k8s-ml-large-v100" {  cluster_id  = yandex_kubernetes_cluster.k8s-ml.id  name        = "k8s-ml-large-v100"  version     = "1.32"  node_labels = { "node.kubernetes.io/role" = "jupyter", "inst_type" = "large-v100" }  node_taints = ["jupyter=true:NoSchedule"]  instance_template {    name        = "k8s-ml-large-v100-8-48-gpu-{instance.short_id}"    platform_id = "gpu-standard-v2"    resources {      memory        = 48      cores         = 8      core_fraction = 100      gpus          = 1    }    metadata = {      "serial-port-enable" = "1"      ssh-keys             = "admin:${local.ssh-keys.admin} admin"    }    gpu_settings {      gpu_environment = "runc"    }  }  scale_policy {    auto_scale {      min     = 0      max     = 2      initial = 0    }  }  allocation_policy {    location {      zone = yandex_vpc_subnet.k8s-nodes-ml-a.zone    }  }}

When a user starts Jupyter, Kubernetes creates a pod. If there is no suitable node, the autoscaler starts creating one. When the user finishes their work, the pod is deleted. If the node is no longer used, it is also automatically deleted.

This means that resources are billed only when they are actually needed. This is especially important for expensive GPUs and large instances.

GPU and Operator

I use NVIDIA GPU Operator to work with GPUs.

I have a prepared Jupyter Docker image with libraries and tools for analysts already installed. However, problems can appear when some components need to be updated, especially CUDA.
For example, the driver version preinstalled on GPU nodes in Yandex Cloud may be incompatible with the required CUDA version. Because of this, I decided to use GPU Operator. Yandex also recommends this approach.

It allowed us to move driver management into Kubernetes and install the required driver version. It also helps us keep the driver version compatible with CUDA in user images and makes controlled updates easier.

For GPU Operator to manage the node correctly, I need to create a Yandex Cloud node without preinstalled drivers. This means adding the following configuration in Terraform:

gpu_settings {  gpu_environment = "runc"}

You can read more about this in my article about NVIDIA GPU drivers in Yandex Managed Kubernetes.

The operator is installed in the standard way using the official Helm chart.
The configuration is connected to GPU nodes through nodeSelector:

driver:  enabled: true  nodeSelector:    inst_type: large-v100

JupyterHub as an Entry Point

JupyterHub is used as the main user interface.

Users open the JupyterHub web interface through GitLab OAuth and can choose from a list of available instance types.

Access to JupyterHub is limited to users who belong to specific GitLab groups.

There are two JupyterHub installations in the cluster. One has ReadWrite access to the buckets, while the other has ReadOnly access. This allows us to separate access levels to buckets from the Jupyter environment.

#  ReadOnlyOAuthenticator:  gitlab_url: <https://gitlab.example.ru>  allowed_gitlab_groups:    - "jupyter-ro" # GitLab group, whos members will have RO access

User profiles are configured so that a pod requests almost all resources of a node.

This means that even if two users choose the same instance type, separate nodes will be created for them. Users who start the same instance type do not have to share node resources. Each user gets the amount of compute resources defined in the profile.

This is an important detail.

Without this configuration, if one user had already started a node but was not using all its resources, and another user requested the same instance type, both pods could be placed on the same node.

The Kubernetes scheduler would not start a new node if both pods could fit on the existing one.

singleuser:  profileList:    - display_name: "4CPU 32RAM"      description: "jupyter-small"      default: true      kubespawner_override:        node_selector:          inst_type: "small"        cpu_guarantee: 3        cpu_limit: 3        mem_guarantee: "26G" #        mem_limit: "26G"

Each user has access to an S3 bucket in Yandex Object Storage from their Jupyter instance.
Each user also has their own PVC and PV with an SSD disk in the cloud for their home directory.

storage:  capacity: 100Gi # Home dir size  dynamic:    storageClass: yc-network-ssd  # extra volumes for mounting into pod (S3)  extraVolumes:    - name: ml-data-1      persistentVolumeClaim:        claimName: csi-s3-ml-data-1    - name: ml-data-2      persistentVolumeClaim:        claimName: csi-s3-ml-data-2

JupyterHub is deployed using the official Helm chart.

In addition JupyterHub to providing a convenient interface that allows analysts to work within a corporate environment without direct access to the node.

One more explaining why I decided to use JupyterHub profiles instead of nodeAffinity.

nodeAffinity only defines preferred or required pod placement on nodes with specific labels. It does not prevent Kubernetes from placing several pods on the same node if there are enough free resources.

This could lead to a situation where two users share the same instance, which makes performance less predictable.

Data Access (S3 via CSI)

The data is stored in Object Storage and mounted through CSI S3.

CSI S3 itself is installed from the official GitHub repository.

I use the ru.yandex.s3.csi driver and GeeseFS as the mounter.

The S3 access flow looks like this:

Pod → PVC → PV → StorageClass → CSI driver → GeeseFS → S3 bucket.

It works in the following way.
I define a separate StorageClass with provisioner:ru.yandex.s3.csi

This tells Kubernetes that if a PersistentVolumeClaim requests this storage class, the volume should be created and managed by the Yandex S3 CSI driver.
The StorageClass also contains parameters required to connect to the bucket: the bucket name, access secrets used by CSI, the mounter type, and other settings.

I use GeeseFS as the mounter.

GeeseFS is a FUSE-based file system for S3. Users can work with the data as if they were normal files, for example using ls or cat. In the background, GeeseFS sends requests to the S3 API.

kind: StorageClassapiVersion: storage.k8s.io/v1metadata:  name: csi-s3provisioner: ru.yandex.s3.csiparameters:  mounter: geesefs  options: "--memory-limit=1000 --dir-mode=0777 --file-mode=0666"  bucket: <name_of_existing_bucket>  csi.storage.k8s.io/provisioner-secret-name: csi-s3-secret  csi.storage.k8s.io/provisioner-secret-namespace: kube-system  csi.storage.k8s.io/controller-publish-secret-name: csi-s3-secret  csi.storage.k8s.io/controller-publish-secret-namespace: kube-system  csi.storage.k8s.io/node-stage-secret-name: csi-s3-secret  csi.storage.k8s.io/node-stage-secret-namespace: kube-system  csi.storage.k8s.io/node-publish-secret-name: csi-s3-secret  csi.storage.k8s.io/node-publish-secret-namespace: kube-system

Usually, this approach is used for dynamic PV provisioning and creating new buckets.

In our case, this functionality is mostly optional. Our analysts already store their data in existing buckets, and they need to connect specifically to these buckets.

For dynamic provisioning, I would have to create a separate StorageClass with the bucket field configured for every bucket.

I decided not to create many StorageClass objects and instead use static provisioning

To avoid creating and maintaining multiple StorageClass objects, I decided to use static provisioning instead. This allowed me to define PV and PVC resources for each existing bucket and specify the connection parameters directly in the PV configuration.

For this setup, the storageClassName field in the PVC should be empty.

In the PV, I explicitly define which PVC it belongs to and manually configure the connection parameters that would normally be passed through the StorageClass during dynamic provisioning.

apiVersion: v1kind: PersistentVolumeClaimmetadata:  name: csi-s3-ml-data-1  namespace: jupyterhubspec:  accessModes:    - ReadWriteMany  resoapiVersion: v1kind: PersistentVolumeClaimmetadata:  name: csi-s3-ml-data-1  namespace: jupyterhubspec:  accessModes:    - ReadWriteMany  resources:    requests:      storage: 4200Gi  storageClassName: ""---apiVersion: v1kind: PersistentVolumemetadata:  name: s3-ml-data-1spec:  storageClassName: csi-s3 # optional  capacity:    storage: 4200Gi  accessModes:    - ReadWriteMany  claimRef:    namespace: jupyterhub    name: csi-s3-ml-data-1  csi:    driver: ru.yandex.s3.csi    volumeHandle: ml-data-1 # name of existing S3    controllerPublishSecretRef:      name: csi-s3-secret      namespace: kube-system    nodePublishSecretRef:      name: csi-s3-secret      namespace: kube-system    nodeStageSecretRef:      name: csi-s3-secret      namespace: kube-system    volumeAttributes:      capacity: 4200Gi      mounter: geesefs      options: "--memory-limit=1000 --dir-mode=0777 --file-mode=0666 --uid=1001"urces:    requests:      storage: 4200Gi  storageClassName: ""---apiVersion: v1kind: PersistentVolumemetadata:  name: s3-ml-data-1spec:  storageClassName: csi-s3 # optional  capacity:    storage: 4200Gi  accessModes:    - ReadWriteMany  claimRef:    namespace: jupyterhub    name: csi-s3-ml-data-1  csi:    driver: ru.yandex.s3.csi    volumeHandle: ml-data-1 #  name of existing S3    controllerPublishSecretRef:      name: csi-s3-secret      namespace: kube-system    nodePublishSecretRef:      name: csi-s3-secret      namespace: kube-system    nodeStageSecretRef:      name: csi-s3-secret      namespace: kube-system    volumeAttributes:      capacity: 4200Gi      mounter: geesefs      options: "--memory-limit=1000 --dir-mode=0777 --file-mode=0666 --uid=1001"

This is how I connected all buckets required by the users.

I used the same approach to mount buckets into Dask pods.

The data is mounted directly into the user’s file system, for example, in /home/ml-data-1 and /home/ml-data-2.

Main Problem: S3 Cold Start

If we want to keep the system cost-efficient and shut down user nodes when they are not working with Jupyter, there is a trade-off.

For example, it can take between 10 and 20 minutes from the moment a user starts an instance until the Jupyter instance is ready to use.

Based on our observations, startup without S3 takes about 1–4 minutes, while startup with S3 takes around 15 minutes.

The reason is that node groups scale down to zero by default.

Nodes appear only when a user requests them by starting a Jupyter pod.

After that, several things need to happen:

• the autoscaler needs to trigger node creation;

• a virtual machine needs to be created in Compute Cloud;

• the node needs to join the Kubernetes cluster;

• the CSI S3 driver needs to be installed on the node.

While this is happening, the Jupyter pod can show errors like:

driver name ru.yandex.s3.csi not found

To prevent JupyterHub from deleting the pod too early, it is better to configure a long enough timeout for the required infrastructure to become ready.

Otherwise, the hub may delete the pod before it has a chance to start correctly with all required mounts.

singleuser:  profileList:    startTimeout: 1800 # 30 min

At first, it may seem that an Init Container could solve this problem by checking whether the S3 mount is available before starting the main container.

However, this does not solve the main problem: the waiting time. Init Containers start only after the pod has been successfully scheduled to a node.

In our case, the delay happens much earlier, when the node itself is being created and the CSI S3 driver is being installed. Until the node appears in the cluster and the driver finishes initialization, the pod cannot start correctly.

This means that the Init Container cannot even start running.

Distributed Computing (Dask)

I use Dask for distributed computing.

It is installed using the official Helm chart.

Additional apt, pip, and CUDA packages can also be installed through the following variables:

• EXTRA_APT_PACKAGES

• EXTRA_CONDA_PACKAGES

• EXTRA_PIP_PACKAGES

scheduler:  name: dask-scheduler  enabled: true  replicas: 1  serviceType: "ClusterIP"  servicePort: 8786  # selector to roolout on exactly scheduler-ноде  nodeSelector:    dask-component: scheduler    dask-cluster: dask-k8s    workload: dask-scheduler  tolerations:    - key: "dask"      operator: "Equal"      value: "scheduler"      effect: "NoSchedule"  env:    - name: EXTRA_APT_PACKAGES      value: "build-essential pkg-config libssl-dev libsnappy-dev cmake llvm clang"    - name: EXTRA_CONDA_PACKAGES      value: "-c conda-forge rust=1.85.* numpy==1.24.4 pandas==2.1.1"    - name: EXTRA_PIP_PACKAGES      value: "polars==1.32.2 cloudpickle==3.0.0 lz4==4.3.2 msgpack==1.0.6 tornado==6.3.3 prometheus-client python-snappy dask==2025.5.1 distributed==2025.5.1 fsspec==2025.9.0 s3fs==2025.9.0 --upgrade"webUI:  name: dask-webui  servicePort: 80  ingress:    enabled: true    ingressClassName: internal    pathType: Prefix    tls: true    secretName: my-secret-name    hostname: my-dask-dashboard.example.ru    annotations: {}worker:  name: dask-worker  strategy:    type: RollingUpdate  nodeSelector:    dask-component: worker    dask-cluster: dask-k8s    workload: dask-worker  tolerations:    - key: "dask"      operator: "Equal"      value: "worker"      effect: "NoSchedule"  # resourses for worker-ноды (16GB RAM, 4 CPU)  resources:    limits:      cpu: "1.5"      memory: "6Gi"    requests:      cpu: "1"      memory: "5Gi"  env:    - name: EXTRA_APT_PACKAGES      value: "build-essential s3fs pkg-config libssl-dev libsnappy-dev cmake llvm clang"    - name: EXTRA_CONDA_PACKAGES      value: "-c conda-forge rust=1.85.* numpy==1.24.4 pandas==2.1.1"    - name: EXTRA_PIP_PACKAGES      value: "polars==1.32.2 cloudpickle==3.0.0 lz4==4.3.2 msgpack==1.0.6 tornado==6.3.3 s3fs pyarrow==14.0.1 prometheus-client python-snappy dask==2025.5.1 distributed==2025.5.1 --upgrade"  # mounting CSI S3 volume for workers  mounts:    volumes:      - name: csi-ml-data-1        persistentVolumeClaim:          claimName: csi-ml-data-1    volumeMounts:      - name: csi-ml-data-1        mountPath: /data/csi-ml-data-1        readOnly: false  portDashboard: 8790jupyter:  name: jupyter  enabled: false # off Jupyter — use our ouwn (not from chrat)

The main idea of Dask is to split a task into many smaller parts and process them in parallel.

Dask has a scheduler, which is the central component of the system. It receives tasks from the client, creates computation graphs, decides the execution order, distributes tasks between worker pods, and manages the rest of the coordination.

The scheduler does not need many resources. Because of this, I created a separate small node group with one node only for the scheduler. No other workloads are placed on this node.

This prevents heavy worker pods, which perform the main resource-intensive work, from affecting the scheduler. For worker pods, I use a node group with autoscaling from 1 to 3 nodes.

Each node can run two worker pods. This is controlled by the resource requests of the pods.

Pod autoscaling is configured with the standard Kubernetes HPA:

metrics:  - type: Resource    resource:      name: cpu      target:        type: Utilization        averageUtilization: 70  - type: Resource    resource:      name: memory      target:        type: Utilization        averageUtilization: 70

A connection to the cluster from a notebook looks like this:

client = Client('dask-cluster-scheduler.dask-system.svc.cluster.local:8786')

There is also a NodePort service for accessing the cluster from outside Kubernetes, but it is available only from internal networks.

Conclusion

This solution works well for data analysis, model training, and distributed data processing.

It is useful when users need a shared computing environment where they can start their own Jupyter instances and work with data interactively.

At the same time, this solution is not designed for production inference and does not cover the full MLOps lifecycle.

Its main purpose is to provide a convenient research and computing environment for analysts and ML engineers, rather than a complete production platform for running models at every stage of their lifecycle.

As a result, I built a minimal ML platform that includes GPUs, JupyterHub, data access, distributed computing, and scalability.

At the same time, the platform does not have unnecessary complexity, and give an opportunity have full control over the infrastructure.

Thank you for reading!