Docker: Dev and Prod Workflows with Runhouse

Docker Cluster Setup

Because we are pulling the Docker image from AWS ECR, we need to provide the corresponding credentials in order to properly pull and setup the image on the cluster. This can be done through a Runhouse Docker secret, or by setting environment variables. Please refer to <Guide: Docker Cluster Setup> for more details.

import subprocess
import runhouse as rh

docker_ecr_creds = {
    "username": "AWS",
    "password": subprocess.run("aws ecr get-login-password --region us-west-1", shell=True, capture_output=True).stdout.strip().decode("utf-8"),
    "server": "763104351884.dkr.ecr.us-west-1.amazonaws.com",
}
docker_secret = rh.provider_secret("docker", values=docker_ecr_creds)

Next, construct a Runhouse image, passing in the docker image ID and secret. Feed this image into the OnDemand cluster factory, and up the cluster.

base_image = rh.Image("docker_image").from_docker(
    "djl-inference:0.27.0-deepspeed0.12.6-cu121", docker_secret=docker_secret
)

cluster = rh.ondemand_cluster(
    name="diffusers_docker",
    image=base_image,
    instance_type="g5.8xlarge",
    provider="aws",
)
cluster.up_if_not()

⠴ Preparing SkyPilot runtime (3/3 - runtime)  View logs at: ~/sky_logs/sky-2024-12-23-13-56-48-619803/provision.log

✓ Cluster launched: diffusers_docker.  View logs at: ~/sky_logs/sky-2024-12-23-13-56-48-619803/provision.log

INFO | 2024-12-23 14:03:39 | runhouse.resources.hardware.launcher_utils:391 | Starting Runhouse server on cluster
INFO | 2024-12-23 14:03:39 | runhouse.resources.hardware.cluster:1247 | Restarting Runhouse API server on diffusers_docker.

INFO:     Started server process [2929]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:32300 (Press CTRL+C to quit)

<runhouse.resources.hardware.on_demand_cluster.OnDemandCluster at 0x127ea7730>

Sample Function

The function we’ll be using in our demo is is_transformers_available from diffusers.utils. We’ll first show how to use the base version of this function, which was installed on the box through the cluster setup (e.g. a production setting). Then, we’ll show how to propogate up local changes and run them on the cluster, if your local version differs from the one in the Docker container (e.g. different package version, or locally edited).

from diffusers.utils import is_transformers_available

Production Workflow

The core of the production workflow is that the Docker image already contains the exact packages and versions we want, probably published into the registry in CI/CD. We don’t want to perform any installs or code changes within the image throughout execution so we can preserve exact reproducibility.

NOTE: By default, Ray and Runhouse are installed on the ondemand cluster during setup time (generally attempting to match the versions you have locally), unless we detect that they’re already present. To make sure that no installs occur in production, please make sure that you have Runhouse and Ray installed in your docker image.

prod_fn = rh.function(is_transformers_available).to(cluster, sync_local=False)

INFO | 2024-12-23 14:04:57 | runhouse.resources.hardware.ssh_tunnel:91 | Running forwarding command: ssh -T -L 32300:localhost:32300 -i ~/.ssh/sky-key -o Port=10022 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ConnectTimeout=30s -o ForwardAgent=yes -o ProxyCommand='ssh -i ~/.ssh/sky-key -o Port=22 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ConnectTimeout=30s -o ForwardAgent=yes -W %h:%p ubuntu@52.24.239.151' root@localhost
INFO | 2024-12-23 14:05:00 | runhouse.resources.module:511 | Sending module is_transformers_available of type <class 'runhouse.resources.functions.function.Function'> to diffusers_docker

prod_fn()

INFO | 2024-12-23 14:05:01 | runhouse.servers.http.http_client:439 | Calling is_transformers_available.call
INFO | 2024-12-23 14:05:06 | runhouse.servers.http.http_client:504 | Time to call is_transformers_available.call: 4.86 seconds

True

Local Development

Now for the local development and experimentation case. Let’s say we have the HuggingFace diffusers repository cloned and installed as a local editable package, and are making changes to it that we want reflected when we run it on the cluster. We also have a different version of the transformers package installed.

def is_transformers_available():
    try:
        import transformers
        return transformers.__version__
    except ImportError:
        return False

from diffusers.utils import is_transformers_available

is_transformers_available()

'4.44.2'

cluster.install_packages(["transformers", "diffusers"], force_sync_local=True)

dev_fn = rh.function(is_transformers_available).to(cluster)

INFO | 2024-12-23 14:11:05 | runhouse.resources.module:511 | Sending module is_transformers_available of type <class 'runhouse.resources.functions.function.Function'> to diffusers_docker

dev_fn()

INFO | 2024-12-23 14:11:19 | runhouse.servers.http.http_client:439 | Calling is_transformers_available.call
INFO | 2024-12-23 14:11:21 | runhouse.servers.http.http_client:504 | Time to call is_transformers_available.call: 2.48 seconds

'4.44.2'

Summary - Setting Up Your Code

Here, we implement the above as a script to demonstrate the difference between dev and prod. The script can easily be adapted and shared between teammates developing and working with the same repos, with a flag or variable flip to differentiate between experimentation and production branches.

from diffusers.utils import is_transformers_available

if __name__ == "__main__":
    cluster = rh.ondemand_cluster(...)
    cluster.up_if_not()

    if prod:
        remote_fn = rh.function(is_transformers_available).to(cluster, sync_local=False)
    else:
        cluster.install_packages(["transformers", "diffusers"], )
        remote_fn = rh.function(is_transformers_available).to(cluster)

    remote_fn()
    cluster.teardown()