How to Use Runhouse

Quick Start

Before reviewing this detailed guide, we recommend you start with the Quick Start guide.

• Install Runhouse with pip install runhouse

• Optionally install with a specific cloud like pip install "runhouse[aws]" or with SkyPilot for elastic compute pip install "runhouse[sky]

• Optionally create an account on the Runhouse website or with runhouse login --sync-secrets to enable saving, reloading, and centralized authentication / secrets management.

Access to a Pool of Compute

In order to use Runhouse, you must be able to access compute resources, which can take any form (e.g. VMs, elastic compute, Kubernetes). You should think about all the compute resources you have as a single pool, from which Runhouse allows you to launch ephemeral clusters to execute your code.

• Kubernetes: All you need is a kubeconfig

• Elastic Compute: We use Skypilot under the hood to launch elastic compute, and support most clouds. You can run sky check in CLI after installing Runhouse to confirm you have access to the cloud.

• Existing Clusters: Runhouse supports a variety of authentication methods to access existing clusters, including SSH with keys or passwords.

For initial projects and getting started quickly, launching from local credentials is possible. In this setting, you already unlock serverless execution for your Python ML code, but you cannot take advantage of advanced usage patterns that are unlocked through compute saving and reuse.

For production settings, we recommend that users load cloud secrets, Kubeconfig, and available compute resources into Runhouse Den and authenticate from all launch environments using only the Runhouse token. Platform teams gain centralized observability over utilization, including insights into who is launching clusters, how often they are launched, and the resources or tasks executed on them. Access management becomes much simpler, especially in multi-cloud or multi-cluster environments. To get started with Den enabled, simply run runhouse login --sync-secrets in the CLI.

Start Your Project

Once you have established access to compute, you can start developing a new ML project. The following steps will provide the details of how to use Runhouse, starting from a blank page in your IDE.

1. Define Compute

Runhouse allows you to define compute requirements in code, and launch ephemeral clusters from the compute pool we described in the prior section. Here, you can define the required CPU, GPU, memory, and disk requirements (or name a specific cluster) to use.

For instance, to create a cluster on AWS with an A10 GPU attached using an arbitrary Docker image, you can write:

import runhouse as rh

cluster = rh.ondemand_cluster(
    name="rh-cluster", # This cluster can be saved and reused by name. We will prefix your username when saved, e.g. /my_username/rh-cluster
    instance_type="A10G:1", # There are a number of options available for instance_type, check out the docs to see them all
    provider="aws", # Specify a cloud provider
    image_id="docker:nvcr.io/nvidia/pytorch:23.10-py3", # Use a Docker image
    autostop_mins=90, # Remember to set autostop_mins to avoid leaving clusters running indefinitely.
    launcher="den", # Launch the cluster with Runhouse; use 'local' for local credentials
).up_if_not()

Copied

You can run CLI commands on the cluster using cluster.run() to layer on setup steps beyond the underlying image; for instance, installing other packages.

cluster.run(['pip install numpy'])

Copied

You can find full documentation about the Runhouse cluster API in the Cluster docs.

def add_two_numbers(a,b):
      return a+b

remote_add = rh.function(add_two_numbers).to(cluster)

class TorchTrainer:
   def __init__(self):
      ..

   def train(self, X, y):
      ..

   def test(self, X, y):
      ..

if __name__ == "__main__":
   my_env = rh.env(reqs=["torch"], name="my-env") # Define the need for PyTorch
   RemoteTrainer = rh.module(TorchTrainer).to(cluster, env=my_env) # Send to cluster
   trainer = RemoteTrainer(name='remote-instance-of-trainer') # Instantiate remote object

3. Execute Your Code Remotely

It’s now possible to use your remote objects as if they were local. From here on, you can think of Runhouse as facilitating regular object-oriented programming but with the objects living remotely, maybe in a different cluster, region, or cloud than the local code. Python behavior like async, exceptions, printing, and logging are all preserved across remote calls but can be disabled or controlled if desired.

result = remote_add(1,2)
print(result)

X, y = ...  # Load data
trainer.train(X,y)

Copied

As noted above, you should be iteratively dispatching and executing code. If you make updates to the add_two_numbers function or the TorchTrainer class, you can simply re-run .to(), and it should take <2 seconds to redeploy. The underlying cluster is persisted and stateful until you choose to down it, so you can take advantage of the remote file system and memory during interactive development as well.

These remote objects are accessible from anywhere you are authenticated with Runhouse, so you and your team can make multi-threaded calls against them. Calling microservices is actually a familiar pattern in programming; however, no team would ever manually split their ML pipeline into multiple applications due to the DevOps overhead.

# Save to Den
remote_add.save(name="my_function")

# Reload the function and invoke it remotely on the cluster
my_func = rh.function(name="/my_username/my_function")

# Share the function with another user, giving them access to call or modify the resource
my_func.share("user_a@gmail.com", access_level="write")

cluster.teardown()

Moving to Production

A key advantage of using Runhouse is that the code developed locally has already been executing production-like on remote compute the entire time. This means research-to-production is an abstract checkpoint in development rather than an actual task to rewrite pipelines for production over different hardware/data.

If your code is for a non-recurring task, then great, check your code into version control and you are already done. If you are deploying a recurring job like recurring training, then simply move the Runhouse launching code into the orchestrator or scheduler of your choice. You should not repackage ML code into orchestrator nodes and make orchestrators your runtime. Instead, you should use orchestrators as minimal systems to schedule and observe your jobs, but the jobs themselves will continue to be executed serverlessly with Runhouse from each node. This saves considerable time upfront as setting up the first orchestrator run less than an hour (compared to multiple weeks in traditional ML research-to-production).

As an example, you might want to make the first task of your orchestrator pipeline simply bringing up the cluster and dispatching code to the new cluster. You can see that we are using the same underlying code (directly importing it from a source file), and then reusing the object and cluster by name across steps.

@task()
def up_and_dispatch():
      cluster = rh.ondemand_cluster(
            name="rh-cluster",
            instance_type="A10G:1",
            provider="aws",
            image_id="docker:nvcr.io/nvidia/pytorch:23.10-py3",
      ).up_if_not()

      from my_code import TorchTrainer
      my_env = rh.env(reqs=["torch"], name="my-env")
      RemoteTrainer = rh.module(TorchTrainer).to(cluster, env=my_env)
      trainer = RemoteTrainer(name='remote-instance-of-trainer')

@task()
def embed():
      cluster = rh.cluster(name="rh-cluster")
      trainer = cluster.get(name='remote-instance-of-trainer')
      X, y = ...  # Load data
      trainer.train(X,y)

Runhouse recommends creating a Docker container which fixes the environment, dependencies, and program code for production pipelines. There are significant benefits to containerization, rather than, for instance, worrying about new breaking changes from package installation with PyPi. This is actually still unproblematic for additional future iteration or debug, since you still easily interactively layer on changes to the environment from local, even when you launch with the container.

My Pipeline is in Production, What’s Next?

Once in production, your ML pipelines will eventually experience some failures you need to debug. With Runhouse engineers can easily reproduce production runs on local, make changes to the underlying code, and simply push a change to the codebase. There is no debugging through the orchestrator, and no need to rebuild and resubmit. However, we find that deploying with Runhouse has fewer errors to begin with, as the code has already been developed in a production-like environment.

This also makes production-to-research a seamless process. Many teams are loathe to revisit the research-to-production process again, so when code is deployed to production, there is little appetite to make small incremental improvements to the pipeline. With Runhouse, the pipeline is already running serverlessly, so incremental changes that are merged to the team codebase are automatically reflected in the production pipeline once tested via normal development processes.

There are other benefits to using Runhouse in production as you scale up usage. A few are included here:

• Shared services: You may want to deploy shared services like an embeddings endpoint, and have all pipelines call it by name as a live service or import the code

from the underlying team repository and stand it up separately in each pipeline. Either way, if you every update or improve this shared service, all pipelines will receive the downstream updates without any changes to the pipeline code. * Compute abstraction: As you add new resources to your pool, get credits from new clouds, or get new quota, if all users are using Runhouse to allocate ephemeral compute, there is no need to update any code or configuration files at the user level. The new resources are added by the platform team, and then automatically adopted by the full team. * Infrastructure Migrations: With Runhouse, your application code is entirely undecorated Python and the dispatch happens to arbitrary compute. If you ever choose to abandon your existing orchestrator, cloud provider, or any other tool, you simply have to move a small amount of dispatch code and infrastructure code configuration. * Adopting Distributed Frameworks: Runhouse is a perfect complement to distributed frameworks, with some built-in abstractions that let you scale to multiple clusters or start using Ray clusters easily.