GLM-5.2 Inference Endpoint

PREREQUISITES

1. Your account must have an EC2 service quota for P5/P6 On-Demand instances ("Running On-Demand P instances", measured in vCPUs). p5en.48xlarge and p6-b200.48xlarge each require 192 vCPUs. New accounts default to 0. Request an increase at Service Quotas > Amazon EC2 > Running On-Demand P instances well before launch; approval can take several business days. See https://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html 

2. These instance families have constrained capacity. If launch fails with InsufficientInstanceCapacity, try another Availability Zone, or use a Capacity Reservation or ML Capacity Block.

LAUNCH

1. Subscribe to this product and choose Launch through EC2 (or Launch from Website).
2. Select p5en.48xlarge (recommended), p6-b200.48xlarge, or p5e.48xlarge. No other instance types are supported.
3. Assign a security group. Restrict port 22 to your administrative CIDR and port 8000 to the CIDR of the clients that will call the API. Do not open port 8000 to 0.0.0.0/0.
4. Ensure the subnet provides outbound internet access (internet gateway, or NAT gateway for private subnets). The instance must reach Amazon S3 to download the model weights.
5. Launch.

WAIT FOR THE SERVER TO BECOME READY

*** Initial startup takes approximately 40-45 minutes. ***

The instance downloads approximately 750 GB of model weights and loads them into GPU memory. The API returns HTTP 503 until this completes. Do not assume a failure before 30 minutes have elapsed.

Poll for readiness:

RETRIEVE YOUR API KEY

A unique random API key is generated on the first boot of each instance. It is not baked into the image and is different for every instance you launch.

Via the console (no SSH required):

1. EC2 console > Instances > select your instance.
2. Actions > Monitor and troubleshoot > Get system log.
3. Find the API Key and Endpoint Url. (The system log can take a few minutes to populate after boot.)

Example: Status : READY Model name : glm-5.2-fp8 Endpoint URL : http://<public-ip>:8000/v1 API key : sk-glm-xxxxxxxx...

Via SSH: ssh -i <your-key.pem> ubuntu@<INSTANCE_PUBLIC_IP> sudo cat /opt/holbox/api_key

CALL THE ENDPOINT

Base URL: http://<INSTANCE_PUBLIC_IP>:8000/v1 Replace <INSTANCE_PUBLIC_IP> with the public IP shown in the EC2 console.

Any OpenAI-compatible client works:

client = OpenAI(base_url="http://<public-ip>:8000/v1", api_key="sk-glm-xxxx...") print(client.chat.completions.create( model="glm-5.2-fp8", messages=[{"role": "user", "content": "Hello from GLM 5.2!"}], ).choices[0].message.content)

HEALTH AND MONITORING

1. EC2 console > Instances > select your instance > Status checks tab. Both system and instance checks must pass.
2. Service status: glm-status

If /health does not return 200 after 35 minutes, check journalctl for weight download errors (usually a missing route to the internet) or CUDA OOM (usually an unsupported instance type).

There are no additional AWS resource charges: the product uses only the EC2 instance and its EBS root volume. Instances are billed while running regardless of whether the endpoint is being called; stop or terminate instances when idle. Note that stopping and restarting triggers the model load again (though not the download).