Shield On-premises AWS ECS Deployment Guide

Pre-requisites

Site Reliability Engineers (SREs)

To ensure the seamless deployment and ongoing management of a highly available Shield instance in a production
environment, it is imperative to have a dedicated team of skilled SREs. They should possess in-depth expertise in
your on-premises infrastructure and be equipped with the necessary permissions to execute their tasks effectively.

Arthur Platform License

The Arthur team will provide a license for the products and features you have

Azure OpenAI

An OpenAI GPT model with at least one endpoint. Shield supports Azure and OpenAI as the LLM service provider.
A secure network route between your AWS infrastructure and the OpenAI endpoint(s) with OpenAI endpoint credentials made available
Token limits, configured appropriately for your use cases

DNS URLs with SSL certs

Certificates signed by a well-known trusted authority:

For Shield (e.g. https://shield.mycompany.com)
For Arthur Auth service (e.g. https://shield-auth.mycompany.com) - not required for API-only install

NOTE: The SSL certificates may NOT be self-signed.

AWS Environment

AWS credentials for the CloudFormation template install with the permissions to manage IAM, security groups, secrets
in the Secrets Manager, load balancer, RDS, ECS, and CloudWatch (IAM and RDS can use existing)
A VPC with 3 private subnets and 2-3 public (client-facing) subnets
An AWS ARN of the TLS certificate for the Shield application DNS URL created/imported in the AWS Certificate Manager
(Skip for API-only install) An AWS ARN of the TLS certificate for the Auth service DNS URL created in the AWS
Certificate Manager
An AWS ARN for an SNS topic for CloudWatch platform alarm notifications

Arthur Container Registry Access

Arthur will provide credentials for pulling Docker containers from our public Container Repository
There must be a network route available to connect to our Arthur Repository via repository.arthur.ai and
docker.arthur.ai

Private Container Registry (Optional but highly recommended)

Arthur suggests hosting the Shield container images on your private container registry, like AWS ECR, for enhanced
speed and reliability during each deployment and scaling-out process. Below is an example of how you can pull them.

docker login docker.arthur.ai -u <username>
docker pull docker.arthur.ai/arthur/auth:1.0.33
docker pull docker.arthur.ai/arthur/shield:<shield_version_number>-gpu # GPU version
docker pull docker.arthur.ai/arthur/shield:<shield_version_number> # CPU version

How to install Shield using AWS CloudFormation template

Log in to the AWS account that has the VPC and the subnets you want to deploy to
Go to the CloudFormation page on the AWS Console
From the "Stacks" page, select "Create stack" -> "With new resources (standard)"
Populate the "Create stack" page and click "Next"
1. Select "Template is ready" and "Amazon S3 URL"
2. Provide the HTTP link to the CloudFormation template that maps to the latest version of Shield found in
  the Shield doc site
  1. Shield full-stack with GPU (includes Shield Admin UI and optionally Chat):
    
    https://arthur-cft.s3.us-east-2.amazonaws.com/templates/<shield_version_number>/root-arthur-shield-gpu.yml
  2. Shield full-stack with CPU (includes Shield Admin UI and optionally Chat):
    
    https://arthur-cft.s3.us-east-2.amazonaws.com/templates/<shield_version_number>/root-arthur-shield-cpu.yml
  3. Shield API-only with CPU:
    
    https://arthur-cft.s3.us-east-2.amazonaws.com/templates/<shield_version_number>/root-arthur-shield-cpu-api-only.yml
Populate the "Specify stack details" page and click "Next"
Populate the "Configure stack options" page and click "Next"
1. Behavior on provisioning failure: "Roll back all stack resources" (Do not use --disable-rollback if you're using AWS CLI)
2. Delete newly created resources during a rollback: "Use deletion policy"
Submit the "Review and create" page
(Skip if you provided Route 53 parameter values) Follow the below steps that must be performed outside CloudFormation
1. (Skip for API-only install) Once the AuthLBStack deployment is complete, create an A record that routes the
  Arthur Auth service ingress DNS URL to the Auth ALB created by the CloudFormation template. The Shield ECS service
  tasks will not start until this step is complete.
2. Once the ShieldLBStack deployment is complete, create an A record that routes the Arthur Shield service
  ingress DNS URL to the Shield ALB created by the CloudFormation template

How to get started with using Shield

API

To get started with Shield API endpoints, open your browser and go to the interactive API documentation on your
deployed Shield instance at /docs. Authenticate with your admin key to create your first API key according to
these instructions. The admin key is stored in AWS Secrets Manager as ${ArthurResourceNamespace}_shield_admin_key_secret${ArthurResourceNameSuffix}.

UI

To access the Shield Admin UI, a Shield admin user must be created with the right role according to
the instructions here.

Architecture

FAQs

What are the private and public subnets used for and what are their requirements?

The private subnets are where the application (app) and the database (DB) are deployed. The app and the DB should not be directly accessible by external clients for best practices. The public subnets should be the entry point to Shield. The Shield app load balancer is deployed in the public subnets and routes the Shield traffic to the Shield app
instances in the private subnets. Public subnets do not necessarily but typically have a route to the Internet via an
Internet Gateway (IGW), which can optionally reside behind a Virtual Private Network (VPN). Private subnets do not
have a direct route to the Internet. Instances in a private subnet are accessed via a Network Address Translation(NAT) gateway in the public gateway. Make sure the proper routes exist between the private and the public subnets. Please remember that the public subnets do not have to be exposed to the Internet. They are the entry point for the client LLM applications to talk to Shield.

Depending on the architecture of your infrastructure, you can choose the same set of subnets for both the private and the public subnets of a Shield deployment. Make sure the app and the DB are secured, and they have access to the AWS Secrets Manager.

How do I configure my bring-your-own IAM roles?

Please refer to the IAM details described in the CloudFormation templates:

https://arthur-cft.s3.us-east-2.amazonaws.com/templates/<shield_version_number>/arthur-shield-iam.yml
(Skip for API-only install) https://arthur-cft.s3.us-east-2.amazonaws.com/templates/<shield_version_number>/arthur-auth-iam.yml

How do I configure my bring-your-own security groups?

Please refer to the security group details described in the CloudFormation templates:

https://arthur-cft.s3.us-east-2.amazonaws.com/templates/<shield_version_number>/arthur-core-security-groups.yml
(Skip for API-only install) https://arthur-cft.s3.us-east-2.amazonaws.com/templates/<shield_version_number>/arthur-auth-security-groups.yml
https://arthur-cft.s3.us-east-2.amazonaws.com/templates/<shield_version_number>/arthur-shield-security-groups.yml

Is there any specific requirements for bring-your-own Postgres?

Arthur recommends using the Postgres deployed by our CFT. However, if you need to provide your own PostgreSQL instance, please use the latest stable version accessible to you. To ensure optimal performance and reliability, use a managed PostgreSQL instance with disk encryption, autoscaling, auto minor version upgrades, and automated backups at a minimum.

Also, ensure the PostgreSQL instance can install pgvector, an open-source vector similarity search capability
extension on PostgreSQL. The extension must be available and the Shield application needs the credentials with
permission to run CREATE EXTENSION IF NOT EXISTS vector.

The usage of my Azure OpenAI endpoint is going beyond my quota. What do I do?

Azure OpenAI has a quota called Tokens-per-Minute (TPM). It limits the number of tokens that a single model can process within a minute in the region the model is deployed. To get a larger quota for Shield, you can deploy additional models in other regions and have Arthur Shield round-robin against multiple Azure OpenAI endpoints. In addition, you can request and get approved for a model quota increase in the desired regions by Azure.