Example Deployment

This is a minimal end-to-end setup on the LLM Inference Server. vLLM serves the model locally, and a LiteLLM proxy in front of it provides the Bearer-key authentication and the model alias that Aether expects. The example assumes that the file scanned by Aether is named llama-3.2-3b.safetensors, so Aether requests the model as llama-3.2-3b.

Step 1: Start vLLM. On the host that holds the model (typically the GPU machine), bound to an address LiteLLM can reach. If vLLM and LiteLLM run on the same host, --host 127.0.0.1 is enough; if they run on separate hosts, bind vLLM to a reachable interface and allow the port from the LiteLLM host:

Step 2: Create the LiteLLM configuration file (litellm-config.yaml). The model_name alias is the name that Aether requests, so it must equal the scanned file's name without the extension, with the exact same casing:

Step 3: Start LiteLLM, bound to an address that Aether can reach:

Step 4: Configure Aether with the base URL http://<harness-host>:4000 and the master key as the API key. Aether sends "model": "llama-3.2-3b", and LiteLLM authenticates the request and routes it to vLLM.

Notes

• The name after the hosted_vllm/ prefix must match the name that vLLM itself reports at its own /v1/modelsendpoint. By default, this is the model path or Hugging Face ID that vLLM was started with.

• LiteLLM does not reload its configuration automatically. Restart the proxy after editing the YAML file. vLLM can keep running, so the model stays loaded.

• Verify the setup with, the two checks in Verifying the configuration, using port 4000 and the Bearer key.