# Tutorial: Import a custom model with the Python SDK

This tutorial takes you from `pip install graphn` to running OpenAI-compatible chat completions against a model **you brought** — from HuggingFace or your own S3 bucket — in about ten minutes. No GPUs to provision, no inference server to operate.

By the end you will have:

- Imported a model into your workspace from one of three sources (HuggingFace, S3 presigned URL, or S3 with an IAM role)
- Waited for the deployment to become ready
- Run streaming and non-streaming chat completions against it
- Understood how the SDK transparently handles **scale-to-zero** cold starts so the first request after an idle period "just works"

> **Note:** The same flow is available from the [Models page](./models.md#custom-model-import) in the GraphN UI. This tutorial is the **programmatic** equivalent — useful for CI pipelines, MLOps automation, or just keeping your imports versioned in a script.

## Prerequisites

- A GraphN workspace and an API key (`gn_...`). Get one from the [GraphN dashboard](https://graphn.ai) → workspace settings → **API Keys**.
- Your **workspace id** (`ws_...`), visible in the workspace switcher.
- Python **3.10 or newer**.
- For S3 imports: a model directory in S3 with the standard HuggingFace layout (`config.json`, tokenizer files, weight shards).

## Step 1: Install and configure the SDK

```bash
pip install graphn
```

The SDK reads credentials from the environment by default:

```bash
export GRAPHN_API_KEY=gn_...
export GRAPHN_WORKSPACE_ID=ws_...
```

Verify the install:

```python
import graphn

with graphn.Client() as c:
    for m in c.models.list():
        print(m.id)
```

This lists every model your workspace can address through the inference gateway — built-in models plus anything you have already imported. If this works, your credentials are good and you can move on.

> **Tip:** Prefer not to use environment variables? Pass the values explicitly: `graphn.Client(api_key="gn_...", workspace_id="ws_...")`. The same arguments work for `graphn.AsyncClient`.

## Step 2: Pick an import source

Custom-model import supports three weight sources. Pick whichever matches where your weights live:

| Source | `weight_source` | When to use |
| --- | --- | --- |
| HuggingFace | `huggingface` | Public or gated HF repos. Easiest path. |
| S3 presigned URL | `s3_presigned` | Private S3 weights, you don't want to share AWS credentials with GraphN. |
| S3 + IAM role | `s3_assume_role` | Long-lived imports from buckets you control; GraphN re-assumes a role you trust. |

The lifecycle past the `create` call is **identical** regardless of source — wait, chat, delete. The source only matters at import time.

### Path A — Import from HuggingFace

```python
import graphn

with graphn.Client() as c:
    model = c.custom_models.create(
        name="my-llama",
        weight_source="huggingface",
        huggingface_model_id="Qwen/Qwen3-0.6B",
    )
    print(f"created {model.id} (status={model.status})")
```

For **gated** repos (Llama, Mistral, etc.), first store your HF token as a workspace secret:

```python
secret = c.secrets.create(name="hf-token", value="hf_...")
model = c.custom_models.create(
    name="my-llama",
    weight_source="huggingface",
    huggingface_model_id="meta-llama/Llama-3.1-8B-Instruct",
    hf_token_secret_id=secret.id,
)
```

### Path B — Import from S3 (presigned URL)

> **Required:** S3 weights must be a single `.tar.gz` archive (not a directory or prefix), and you **must** pass `huggingface_model_id`. It's the canonical model identifier the inference endpoint advertises and the value you pass in `model` for chat completions, so without it your chat completion will 404 even after the import succeeds. The SDK raises `ValidationError` if you omit it for any S3 weight source.

Package the weights, upload them, and generate a presigned URL pointing at the archive. Using the AWS CLI:

```bash
tar -czf llama-3.1-8b.tar.gz -C ./llama-3.1-8b .
aws s3 cp llama-3.1-8b.tar.gz s3://my-bucket/llama-3.1-8b.tar.gz
aws s3 presign s3://my-bucket/llama-3.1-8b.tar.gz --expires-in 3600
```

Then create the model:

```python
model = c.custom_models.create(
    name="my-finetune",
    weight_source="s3_presigned",
    s3_url="https://my-bucket.s3.amazonaws.com/llama-3.1-8b.tar.gz?X-Amz-Algorithm=...",
    huggingface_model_id="meta-llama/Llama-3.1-8B-Instruct",
    gpu_count=1,
)
```

The URL only needs to be valid for the import window (a few minutes), not for the model's lifetime.

### Path C — Import from S3 (IAM role assumption)

If you'd rather not deal with presigned URLs, GraphN can assume an IAM role you control. Unlike Path B, the AssumeRole path syncs the weights directly from a bucket **prefix**, so the packaging rules differ:

- `s3_url` must be a **directory prefix ending in `/`** that holds the raw HuggingFace layout (`config.json`, tokenizer files, and the `safetensors` shards) — **not** a `.tar.gz` archive. The importer runs `aws s3 sync`, so a single archive object would never be unpacked, and the API rejects a non-`/`-terminated URL with a `400`.
- You must pass both `s3_role_arn` and `s3_external_id`. The role name must start with `graphn-byom-`, and the `ExternalId` (16–256 chars) must match the role's trust policy.

Set up the IAM role — platform principal, trust policy, and `ExternalId` — by following [BYOM S3 setup](./byom-s3-setup.md), which documents the platform role and the CloudFormation flow end-to-end. The role's permissions policy needs `s3:GetObject` and `s3:ListBucket` scoped to your prefix.

```python
model = c.custom_models.create(
    name="my-finetune",
    weight_source="s3_assume_role",
    s3_url="s3://my-bucket/path/to/model/",  # directory prefix — must end with "/"
    s3_role_arn="arn:aws:iam::123456789012:role/graphn-byom-s3-reader",
    s3_external_id="your-16-to-256-char-external-id",
    huggingface_model_id="meta-llama/Llama-3.1-8B-Instruct",
    gpu_count=1,
)
```

GraphN re-assumes the role on every import or refresh, so you can rotate credentials underneath without breaking anything.

> **Tip:** Set `gpu_count` based on the model size. As a rough rule of thumb: 7B → 1× H100, 70B → 4×, 405B → 8×. The control plane will reject obviously-undersized configurations.

### Path D — Import a LoRA adapter

GraphN auto-detects LoRA adapters at import time, so you don't have to think about a separate `weight_source` enum value. Just point at the adapter the same way you would a full model:

**From HuggingFace** — the validator probes `adapter_config.json` and routes the import down the LoRA path when it finds one. The base model id, LoRA rank, and the routing name are all inferred from the adapter repo:

```python
model = c.custom_models.create(
    name="my-llama-lora",
    huggingface_model_id="my-org/my-llama-3.1-8b-lora",
    # weight_source defaults to "huggingface" -- no need to set it
    gpu_count=1,
)
```

That's it. GraphN reads `adapter_config.json`, validates the underlying base model against the LoRA allowlist (call `c.custom_models.list_supported_architectures()` for the supported set), pulls the base into a shared cluster cache the first time it's needed, and serves the adapter on top.

**From S3** — `adapter_config.json` isn't reachable across a presigned URL or STS-fronted bucket ahead of deploy, so you have to tell GraphN which base to load. The presence of `base_model_id` is the signal that flips the import into LoRA mode:

```python
model = c.custom_models.create(
    name="my-internal-lora",
    weight_source="s3_presigned",
    s3_url="https://my-bucket.s3.amazonaws.com/lora.tar.gz?X-Amz-...",
    huggingface_model_id="my-org/my-internal-lora",
    base_model_id="meta-llama/Llama-3.1-8B-Instruct",  # <-- the LoRA signal
    gpu_count=1,
)
```

If you call `c.custom_models.validate(...)` ahead of time, the response carries `artifact_type` (`"lora"` or `"base"`), `detected_base_model_id`, and `lora_rank` — useful for surfacing what's about to be deployed in a UI or CI report before committing.

> **LoRA note:** Disaggregated PD and speculative decoding are mutually exclusive with LoRA imports; the SDK will reject the combination at request time.

## Step 3: Wait for the deployment to be ready

`create` returns immediately, but the model isn't servable until GraphN has pulled weights, built a container, and scheduled the deployment. Block on it:

```python
model = c.custom_models.wait_until_ready(model.id, timeout=1800)
print(f"ready: status={model.status}, endpoint={model.endpoint}")
```

`wait_until_ready` polls the status field and returns once the model reaches `ready` (or raises `TimeoutError` after the budget runs out). Status progresses `pending` → `deploying` → `ready`. Large models (70B+) can take 10–20 minutes on first import because of the weight download.

If something goes wrong, the model lands in `failed` and `model.error_message` will tell you why (most often: bad HF token, missing files in the S3 prefix, or unsupported architecture).

## Step 4: Run a chat completion

Pass the `model.id` you got back from `create()` straight to chat completions — the SDK handles the gateway's routing prefix internally:

```python
resp = c.chat.completions.create(
    model=model.id,
    messages=[{"role": "user", "content": "In one sentence, what is a Markov chain?"}],
    max_tokens=120,
    temperature=0.0,
)
print(resp.choices[0].message.content)
```

The chat path is **OpenAI-compatible all the way down** — the SDK delegates to the official `openai` Python SDK under the hood. Tools, structured outputs, function calling, and multi-modal inputs all work without any extra wrappers.

### Streaming

Pass `stream=True` for token-by-token output:

```python
stream = c.chat.completions.create(
    model=model.id,
    messages=[{"role": "user", "content": "Count to ten."}],
    stream=True,
)
for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)
print()
```

## Step 5: Understand cold starts and `auto_wake`

Custom models default to **scale-to-zero**: a model with no traffic for `cooldown_seconds` (default 600) is descheduled, freeing GPUs for other workloads. The first request after a cold period would normally return:

```text
503 Service Unavailable: Model is scaled to zero and is now warming up.
```

The SDK detects this, calls `POST /custom-models/{id}/wake` to nudge the autoscaler, and retries with exponential backoff until the model serves or `wake_timeout` elapses (default 180 seconds). **You don't have to do anything** — the snippets in Step 4 already handle it.

The knobs are there if you need them:

```python
# Give the warm-up more headroom for very large models.
resp = c.chat.completions.create(
    model=model.id,
    messages=[...],
    wake_timeout=900,
)

# Disable auto-wake — you'd handle the 503 yourself (e.g., to surface
# a "model warming up" UI to your end users).
resp = c.chat.completions.create(
    model=model.id,
    messages=[...],
    auto_wake=False,
)
```

> **Tip:** If a model needs to stay hot 24/7, set `min_replicas=1` at create time. You'll pay for the always-on GPU but eliminate cold-start latency entirely. Check your usage with `c.custom_models.gpu_hours(model.id)`.

You can also resize an already-deployed model in place — no redeploy, no rolling restart. Useful for pinning a hot model online before a traffic spike, or widening the autoscaling ceiling without a full re-import:

```python
# Pin one replica online (disable scale-to-zero)
c.custom_models.update(model.id, min_replicas=1, max_replicas=1)

# Raise the ceiling for a known burst
c.custom_models.update(model.id, max_replicas=3)
```

The change propagates to the live KServe `InferenceService` and is applied to KEDA's `maxReplicaCount` on the next reconcile. Validation (e.g. `min_replicas <= max_replicas`) is enforced server-side; bad combinations come back as a 4xx with a readable message.

## Step 6: Clean up

When you're done experimenting, delete the model so it doesn't accrue idle compute charges (scale-to-zero means the GPU isn't running, but the workspace record still exists and may incur small storage / scheduling fees):

```python
c.custom_models.delete(model.id)
```

To list everything you currently have imported:

```python
for m in c.custom_models.list():
    print(m.id, m.name, m.status, m.weight_source)
```

## Putting it all together

Here's the full HuggingFace flow as one runnable script. The S3 variants are identical except for the `create` call:

```python
import graphn

with graphn.Client() as c:
    model = c.custom_models.create(
        name="tutorial-demo",
        weight_source="huggingface",
        huggingface_model_id="Qwen/Qwen3-0.6B",
    )
    model = c.custom_models.wait_until_ready(model.id, timeout=1800)

    resp = c.chat.completions.create(
        model=model.id,
        messages=[{"role": "user", "content": "Say hi in three words."}],
        wake_timeout=600,
    )
    print(resp.choices[0].message.content)

    c.custom_models.delete(model.id)
```

## Async

Everything above has an `async` equivalent under `graphn.AsyncClient` — same method names, same arguments, same return types:

```python
import asyncio
import graphn

async def main() -> None:
    async with graphn.AsyncClient() as c:
        model = await c.custom_models.create(
            name="tutorial-demo-async",
            weight_source="huggingface",
            huggingface_model_id="Qwen/Qwen3-0.6B",
        )
        model = await c.custom_models.wait_until_ready(model.id, timeout=1800)

        resp = await c.chat.completions.create(
            model=model.id,
            messages=[{"role": "user", "content": "Hello!"}],
        )
        print(resp.choices[0].message.content)

        await c.custom_models.delete(model.id)

asyncio.run(main())
```

## Next steps

- [Models reference](./models.md) — UI flow, the imported-models (BYO endpoint) variant, and the full model picker landscape.
- [API explorer](/api) — interactive Redoc rendering of the OpenAPI spec; every operation the SDK exposes lives here too.
- [Python SDK on PyPI](https://pypi.org/project/graphn/) — install, full API surface, changelog.
- [SDK source on GitHub](https://github.com/voltagepark/graphn-sdk-python) — examples directory (`import_and_chat.py`, `import_from_s3.py`, `streaming.py`, `async_client.py`, `openai_compat.py`), cold-starts deep-dive (`docs/cold-starts.md`), and issue tracker.
- [OpenAPI spec on GitHub](https://github.com/voltagepark/graphn-openapi) — generate clients in any language (TypeScript, Go, Java). Python is the only first-party wrapped SDK (`pip install graphn`).

> **Note:** SDK v0.2.x covers the full CLI-reachable GraphN API: workflows, agents, functions, MCP servers, knowledge bases, executions, triggers, storage, organizations, and OpenAI-compatible inference. Evals, datasets, guardrails, and voice agents are not yet first-class SDK resources — those endpoints are still on the [API](./api-reference.md) and in the public spec. This tutorial stays focused on custom-model import.