Friendli Container

Step 1: Add a GPU Node Group to your EKS Cluster

Step 2: Install Friendli Container EKS add-on

Step 3: Configure IAM roles for service accounts (IRSA) so that the Kubernetes ServiceAccount you want to run your inference pods with can exercise AWSMarketplaceMeteringFullAccess policy on your behalf.

Step 4 (Optional): To deploy a private or gated model in the HuggingFace model hub, create a Kubernetes Secret with the token with read permission for the model.

Step 5. Create a FriendliDeployment object on the Kubernetes Cluster with the model you want to run, the name of the Node Group you created in Step 1, and optionally the name of the Secret you created in Step 4.

Check https://github.com/friendliai/examples/tree/main/aws/eks-addon  for the detailed instructions and examples, or read on.

Step 1: Add a GPU Node Group to your EKS Cluster

Open Amazon EKS console and choose the cluster that you want to create a node group in, and select the "Compute" tab and click "Add node group". Friendli Container supports NVIDIA L4, A10G, L40S, A100, H100, H200, and B200 instances.

Step 2. Install Friendli Container EKS add-on

Open Amazon EKS console and choose the cluster that you want to configure, and Select the "Add-ons" tab and click "Get more add-ons". Scroll down and under the section "AWS Marketplace add-ons", search and check "Friendli Container".

We recommend you to consider also installing the following EKS add-ons from AWS: Amazon VPC CNI, CoreDNS, kube-proxy, and Amazon EKS Pod Identity Agent.

Step 3: Configure IAM roles for service accounts (IRSA)

This is needed to allow the Kubernetes ServiceAccount to contact AWS Marketplace for license validation. The Kubernetes ServiceAccount you want to run your inference pods with should be able to exercise AWSMarketplaceMeteringFullAccess policy on your behalf.

First, create an IAM OIDC provider for the clusterUsing eksctl, execute

eksctl utils associate-iam-oidc-provider --region <REGION> --cluster <CLUSTER> --approve

The steps are detailed on https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html 

Second, assign IAM role to the Kubernetes Service Account.

If you want to use the "default" ServiceAccount under the "default" namespace to run the inference pods, execute

eksctl create iamserviceaccount --region <REGION> --cluster <CLUSTER> --namespace default --name default --role-name AWSMarketplaceMeteringAccessForFriendliContainer --attach-policy-arn arn:aws:iam::aws:policy/AWSMarketplaceMeteringFullAccess --approve --override-existing-serviceaccounts

The steps are detailed on https://docs.aws.amazon.com/eks/latest/userguide/associate-service-account-role.html 

Step 4 (Optional): Create a secret with a HuggingFace token.

To deploy a private or gated model from HuggingFace, you need to create an access token with "read" permission on the model. Assuming you want to create a secret named "hf-secret", execute

kubectl create secret generic hf-secret --from-literal token=<YOUR_TOKEN_HERE>

Step 5. Create a FriendliDeployment object on the Kubernetes Cluster.

Write a FriendliDeployment specification. An example can be found at https://github.com/friendliai/examples/tree/main/aws/eks-addon 

Then create the object using the kubectl tool: kubectl apply -f friendli-deployment.yaml

After the deployment is done, you can port-forward to the Kubernetes service created to execute inference requests.