Tutorial: Import a custom model with the Python SDK

This tutorial takes you from pip install graphn to running OpenAI-compatible chat completions against a model you brought — from HuggingFace or your own S3 bucket — in about ten minutes. No GPUs to provision, no inference server to operate.

By the end you will have:

Imported a model into your workspace from one of three sources (HuggingFace, S3 presigned URL, or S3 with an IAM role)
Waited for the deployment to become ready
Run streaming and non-streaming chat completions against it
Understood how the SDK transparently handles scale-to-zero cold starts so the first request after an idle period "just works"

Note: The same flow is available from the Models page in the GraphN UI. This tutorial is the programmatic equivalent — useful for CI pipelines, MLOps automation, or just keeping your imports versioned in a script.

Prerequisites

A GraphN workspace and an API key (gn_...). Get one from the GraphN dashboard → workspace settings → API Keys.
Your workspace id (ws_...), visible in the workspace switcher.
Python 3.10 or newer.
For S3 imports: a model directory in S3 with the standard HuggingFace layout (config.json, tokenizer files, weight shards).

Step 1: Install and configure the SDK

bash

pip install graphn

The SDK reads credentials from the environment by default:

bash

export GRAPHN_API_KEY=gn_...
export GRAPHN_WORKSPACE_ID=ws_...

Verify the install:

python

import graphn

with graphn.Client() as c:
    for m in c.models.list():
        print(m.id)

This lists every model your workspace can address through the inference gateway — built-in models plus anything you have already imported. If this works, your credentials are good and you can move on.

Tip: Prefer not to use environment variables? Pass the values explicitly: graphn.Client(api_key="gn_...", workspace_id="ws_..."). The same arguments work for graphn.AsyncClient.

Step 2: Pick an import source

Custom-model import supports three weight sources. Pick whichever matches where your weights live:

Scroll horizontally to compare

Source	`weight_source`	When to use
HuggingFace	`huggingface`	Public or gated HF repos. Easiest path.
S3 presigned URL	`s3_presigned`	Private S3 weights, you don't want to share AWS credentials with GraphN.
S3 + IAM role	`s3_assume_role`	Long-lived imports from buckets you control; GraphN re-assumes a role you trust.

The lifecycle past the create call is identical regardless of source — wait, chat, delete. The source only matters at import time.

Path A — Import from HuggingFace

python

import graphn

with graphn.Client() as c:
    model = c.custom_models.create(
        name="my-llama",
        weight_source="huggingface",
        huggingface_model_id="Qwen/Qwen3-0.6B",
    )
    print(f"created {model.id} (status={model.status})")

For gated repos (Llama, Mistral, etc.), first store your HF token as a workspace secret:

python

secret = c.secrets.create(name="hf-token", value="hf_...")
model = c.custom_models.create(
    name="my-llama",
    weight_source="huggingface",
    huggingface_model_id="meta-llama/Llama-3.1-8B-Instruct",
    hf_token_secret_id=secret.id,
)

Path B — Import from S3 (presigned URL)

Required: S3 weights must be a single .tar.gz archive (not a directory or prefix), and you must pass huggingface_model_id. It's the canonical model identifier the inference endpoint advertises and the value you pass in model for chat completions, so without it your chat completion will 404 even after the import succeeds. The SDK raises ValidationError if you omit it for any S3 weight source.

Package the weights, upload them, and generate a presigned URL pointing at the archive. Using the AWS CLI:

bash

tar -czf llama-3.1-8b.tar.gz -C ./llama-3.1-8b .
aws s3 cp llama-3.1-8b.tar.gz s3://my-bucket/llama-3.1-8b.tar.gz
aws s3 presign s3://my-bucket/llama-3.1-8b.tar.gz --expires-in 3600

Then create the model:

python

model = c.custom_models.create(
    name="my-finetune",
    weight_source="s3_presigned",
    s3_url="https://my-bucket.s3.amazonaws.com/llama-3.1-8b.tar.gz?X-Amz-Algorithm=...",
    huggingface_model_id="meta-llama/Llama-3.1-8B-Instruct",
    gpu_count=1,
)

The URL only needs to be valid for the import window (a few minutes), not for the model's lifetime.

Path C — Import from S3 (IAM role assumption)

If you'd rather not deal with presigned URLs, GraphN can assume an IAM role you control. Unlike Path B, the AssumeRole path syncs the weights directly from a bucket prefix, so the packaging rules differ:

s3_url must be a directory prefix ending in / that holds the raw HuggingFace layout (config.json, tokenizer files, and the safetensors shards) — not a .tar.gz archive. The importer runs aws s3 sync, so a single archive object would never be unpacked, and the API rejects a non-/-terminated URL with a 400.
You must pass both s3_role_arn and s3_external_id. The role name must start with graphn-byom-, and the ExternalId (16–256 chars) must match the role's trust policy.

Set up the IAM role — platform principal, trust policy, and ExternalId — by following BYOM S3 setup, which documents the platform role and the CloudFormation flow end-to-end. The role's permissions policy needs s3:GetObject and s3:ListBucket scoped to your prefix.

python

model = c.custom_models.create(
    name="my-finetune",
    weight_source="s3_assume_role",
    s3_url="s3://my-bucket/path/to/model/",  # directory prefix — must end with "/"
    s3_role_arn="arn:aws:iam::123456789012:role/graphn-byom-s3-reader",
    s3_external_id="your-16-to-256-char-external-id",
    huggingface_model_id="meta-llama/Llama-3.1-8B-Instruct",
    gpu_count=1,
)

GraphN re-assumes the role on every import or refresh, so you can rotate credentials underneath without breaking anything.

Tip: Set gpu_count based on the model size. As a rough rule of thumb: 7B → 1× H100, 70B → 4×, 405B → 8×. The control plane will reject obviously-undersized configurations.

Path D — Import a LoRA adapter

GraphN auto-detects LoRA adapters at import time, so you don't have to think about a separate weight_source enum value. Just point at the adapter the same way you would a full model:

From HuggingFace — the validator probes adapter_config.json and routes the import down the LoRA path when it finds one. The base model id, LoRA rank, and the routing name are all inferred from the adapter repo:

python

model = c.custom_models.create(
    name="my-llama-lora",
    huggingface_model_id="my-org/my-llama-3.1-8b-lora",
    # weight_source defaults to "huggingface" -- no need to set it
    gpu_count=1,
)

That's it. GraphN reads adapter_config.json, validates the underlying base model against the LoRA allowlist (call c.custom_models.list_supported_architectures() for the supported set), pulls the base into a shared cluster cache the first time it's needed, and serves the adapter on top.

From S3 — adapter_config.json isn't reachable across a presigned URL or STS-fronted bucket ahead of deploy, so you have to tell GraphN which base to load. The presence of base_model_id is the signal that flips the import into LoRA mode:

python

model = c.custom_models.create(
    name="my-internal-lora",
    weight_source="s3_presigned",
    s3_url="https://my-bucket.s3.amazonaws.com/lora.tar.gz?X-Amz-...",
    huggingface_model_id="my-org/my-internal-lora",
    base_model_id="meta-llama/Llama-3.1-8B-Instruct",  # <-- the LoRA signal
    gpu_count=1,
)

If you call c.custom_models.validate(...) ahead of time, the response carries artifact_type ("lora" or "base"), detected_base_model_id, and lora_rank — useful for surfacing what's about to be deployed in a UI or CI report before committing.

LoRA note: Disaggregated PD and speculative decoding are mutually exclusive with LoRA imports; the SDK will reject the combination at request time.

Step 3: Wait for the deployment to be ready

create returns immediately, but the model isn't servable until GraphN has pulled weights, built a container, and scheduled the deployment. Block on it:

python

model = c.custom_models.wait_until_ready(model.id, timeout=1800)
print(f"ready: status={model.status}, endpoint={model.endpoint}")

wait_until_ready polls the status field and returns once the model reaches ready (or raises TimeoutError after the budget runs out). Status progresses pending → deploying → ready. Large models (70B+) can take 10–20 minutes on first import because of the weight download.

If something goes wrong, the model lands in failed and model.error_message will tell you why (most often: bad HF token, missing files in the S3 prefix, or unsupported architecture).

Step 4: Run a chat completion

Pass the model.id you got back from create() straight to chat completions — the SDK handles the gateway's routing prefix internally:

python

resp = c.chat.completions.create(
    model=model.id,
    messages=[{"role": "user", "content": "In one sentence, what is a Markov chain?"}],
    max_tokens=120,
    temperature=0.0,
)
print(resp.choices[0].message.content)

The chat path is OpenAI-compatible all the way down — the SDK delegates to the official openai Python SDK under the hood. Tools, structured outputs, function calling, and multi-modal inputs all work without any extra wrappers.

Streaming

Pass stream=True for token-by-token output:

python

stream = c.chat.completions.create(
    model=model.id,
    messages=[{"role": "user", "content": "Count to ten."}],
    stream=True,
)
for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)
print()

Step 5: Understand cold starts and `auto_wake`

Custom models default to scale-to-zero: a model with no traffic for cooldown_seconds (default 600) is descheduled, freeing GPUs for other workloads. The first request after a cold period would normally return:

text

503 Service Unavailable: Model is scaled to zero and is now warming up.

The SDK detects this, calls POST /custom-models/{id}/wake to nudge the autoscaler, and retries with exponential backoff until the model serves or wake_timeout elapses (default 180 seconds). You don't have to do anything — the snippets in Step 4 already handle it.

The knobs are there if you need them:

python

# Give the warm-up more headroom for very large models.
resp = c.chat.completions.create(
    model=model.id,
    messages=[...],
    wake_timeout=900,
)

# Disable auto-wake — you'd handle the 503 yourself (e.g., to surface
# a "model warming up" UI to your end users).
resp = c.chat.completions.create(
    model=model.id,
    messages=[...],
    auto_wake=False,
)

Tip: If a model needs to stay hot 24/7, set min_replicas=1 at create time. You'll pay for the always-on GPU but eliminate cold-start latency entirely. Check your usage with c.custom_models.gpu_hours(model.id).

You can also resize an already-deployed model in place — no redeploy, no rolling restart. Useful for pinning a hot model online before a traffic spike, or widening the autoscaling ceiling without a full re-import:

python

# Pin one replica online (disable scale-to-zero)
c.custom_models.update(model.id, min_replicas=1, max_replicas=1)

# Raise the ceiling for a known burst
c.custom_models.update(model.id, max_replicas=3)

The change propagates to the live KServe InferenceService and is applied to KEDA's maxReplicaCount on the next reconcile. Validation (e.g. min_replicas <= max_replicas) is enforced server-side; bad combinations come back as a 4xx with a readable message.

Step 6: Clean up

When you're done experimenting, delete the model so it doesn't accrue idle compute charges (scale-to-zero means the GPU isn't running, but the workspace record still exists and may incur small storage / scheduling fees):

python

c.custom_models.delete(model.id)

To list everything you currently have imported:

python

for m in c.custom_models.list():
    print(m.id, m.name, m.status, m.weight_source)

Putting it all together

Here's the full HuggingFace flow as one runnable script. The S3 variants are identical except for the create call:

python

import graphn

with graphn.Client() as c:
    model = c.custom_models.create(
        name="tutorial-demo",
        weight_source="huggingface",
        huggingface_model_id="Qwen/Qwen3-0.6B",
    )
    model = c.custom_models.wait_until_ready(model.id, timeout=1800)

    resp = c.chat.completions.create(
        model=model.id,
        messages=[{"role": "user", "content": "Say hi in three words."}],
        wake_timeout=600,
    )
    print(resp.choices[0].message.content)

    c.custom_models.delete(model.id)

Async

Everything above has an async equivalent under graphn.AsyncClient — same method names, same arguments, same return types:

python

import asyncio
import graphn

async def main() -> None:
    async with graphn.AsyncClient() as c:
        model = await c.custom_models.create(
            name="tutorial-demo-async",
            weight_source="huggingface",
            huggingface_model_id="Qwen/Qwen3-0.6B",
        )
        model = await c.custom_models.wait_until_ready(model.id, timeout=1800)

        resp = await c.chat.completions.create(
            model=model.id,
            messages=[{"role": "user", "content": "Hello!"}],
        )
        print(resp.choices[0].message.content)

        await c.custom_models.delete(model.id)

asyncio.run(main())

Next steps

Models reference — UI flow, the imported-models (BYO endpoint) variant, and the full model picker landscape.
API explorer — interactive Redoc rendering of the OpenAPI spec; every operation the SDK exposes lives here too.
Python SDK on PyPI — install, full API surface, changelog.
SDK source on GitHub — examples directory (import_and_chat.py, import_from_s3.py, streaming.py, async_client.py, openai_compat.py), cold-starts deep-dive (docs/cold-starts.md), and issue tracker.
OpenAPI spec on GitHub — generate clients in any language (TypeScript, Go, Java). Python is the only first-party wrapped SDK (pip install graphn).

Note: SDK v0.2.x covers the full CLI-reachable GraphN API: workflows, agents, functions, MCP servers, knowledge bases, executions, triggers, storage, organizations, and OpenAI-compatible inference. Evals, datasets, guardrails, and voice agents are not yet first-class SDK resources — those endpoints are still on the API and in the public spec. This tutorial stays focused on custom-model import.

Documentation

Getting Started

CLI & IDE

Concepts

Reference

Tutorials

Tutorial: Import a custom model with the Python SDK

Prerequisites

Step 1: Install and configure the SDK

Step 2: Pick an import source

Path A — Import from HuggingFace

Path B — Import from S3 (presigned URL)

Path C — Import from S3 (IAM role assumption)

Path D — Import a LoRA adapter

Step 3: Wait for the deployment to be ready

Step 4: Run a chat completion

Streaming

Step 5: Understand cold starts and `auto_wake`

Step 6: Clean up

Putting it all together

Async

Next steps

Documentation

Getting Started

CLI & IDE

Concepts

Reference

Tutorials

Tutorial: Import a custom model with the Python SDK

Prerequisites

Step 1: Install and configure the SDK

Step 2: Pick an import source

Path A — Import from HuggingFace

Path B — Import from S3 (presigned URL)

Path C — Import from S3 (IAM role assumption)

Path D — Import a LoRA adapter

Step 3: Wait for the deployment to be ready

Step 4: Run a chat completion

Streaming

Step 5: Understand cold starts and auto_wake

Step 6: Clean up

Putting it all together

Async

Next steps

Step 5: Understand cold starts and `auto_wake`