AWS Bedrock

Configure and use AWS Bedrock models through TrueFoundry’s AI Gateway with cross-region support

Adding Models

This section explains the steps to add AWS Bedrock models and configure the required access controls.

1. Navigate to AWS Bedrock Models in AI Gateway

   From the TrueFoundry dashboard, navigate to AI Gateway > Models and select AWS Bedrock.

   

   Navigate to AWS Bedrock Models
2. Add AWS Bedrock Account Name and Collaborators

   Give a unique name for the bedrock account which will be used to refer later in the models. The models in the account will be referred to as @providername/@modelname. Add collaborators to your account. You can decide which users/teams have access to the models in the account (User Role) and who can add/edit/remove models in this account (Manager Role). You can read more about access control here.

   

   AWS Bedrock Model Account Form
3. Add Region and Authentication

   Select the default AWS region for the models in this account. The account-level region serves as the default for all models unless explicitly overridden at the model level. Provide the authentication details on how the gateway can access the Bedrock models. Truefoundry supports both AWS Access Key/Secret Key and Assume Role based authentication. You can read below on how to generate the access/secret keys or roles.

   Get AWS Authentication Details

   Required IAM Policy

   First, create the IAM policy that grants permission to invoke Bedrock models. This policy can be attached to either an IAM user (for access key authentication) or an IAM role (for assumed role authentication).

   The following policy grants permission to invoke all models in your available regions (To check the list of available regions for different models, refer to AWS Bedrock):

   JSON

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Sid": "InvokeAllModels",
         "Action": [
           "bedrock:InvokeModel",
           "bedrock:InvokeModelWithResponseStream"
         ],
         "Resource": [
           "arn:aws:bedrock:*::foundation-model/*",
           "arn:aws:bedrock:*:<aws-account-id>:inference-profile/*",
           "arn:aws:bedrock:*:<aws-account-id>:application-inference-profile/*"
         ]
       }
     ]
   }
   

   Using AWS Access Key and Secret

   1. Create an IAM user (or choose an existing IAM user) following these steps.
   2. Attach the IAM policy created above to this user.
   3. Create an access key for this user as per this doc.
   4. Use this access key and secret while adding the provider account to authenticate requests to the Bedrock model.

   Using Assumed Role

   The gateway role assumes your role, which in turn accesses Bedrock models.

   1. Create an IAM role in your AWS account that has access to Bedrock. Attach the IAM policy with Bedrock permissions (shown above) to this role.
   2. Configure the trust policy for this role to allow the gateway role to assume it. Use the appropriate role ARN based on your deployment:

   For SAAS deployments:

   • Gateway role ARN: arn:aws:iam::416964291864:role/tfy-ctl-production-ai-gateway-deps

   For on-prem deployments:

   • Your gateway role ARN will look like: arn:aws:iam::<your-aws-account-id>:role/<account-prefix>-truefoundry-deps

   

   Trust Policy

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Sid": "Statement1",
         "Effect": "Allow",
         "Principal": {
           // for SAAS deployments:
           "AWS": "arn:aws:iam::416964291864:role/tfy-ctl-production-ai-gateway-deps"
           // or for on-prem deployments:
           // "AWS": "arn:aws:iam::<your-aws-account-id>:role/<account-prefix>-truefoundry-deps"
         },
         "Action": "sts:AssumeRole",
         // (Optional) For additional security use external ID.
         "Condition": {
           "StringEquals": {
             "sts:ExternalId": "your-external-id"
           }
         }
       }
     ]
   }
   

   Replace the Principal AWS ARN in the trust policy with the appropriate gateway role ARN based on your deployment type (SAAS or on-prem).

   You can optionally configure an external ID in the trust policy (as shown in the example above) for additional security. If you use an external ID, make sure to provide the same external ID when creating the Bedrock model integration in TrueFoundry.

   3. Read more about how assumed roles work here.
4. Add Models

   Select the models from the list that you want to add. You can use Select All to select all the models.

   If the model you are looking for is not present in the options, you can add it using + Add Model at the end of list.

   TrueFoundry AI Gateway supports all text and image models in Bedrock.The complete list of models supported by Bedrock can be found here.

Inference

After adding the models, you can perform inference using an OpenAI-compatible API via the Playground or integrate with your own application.

Infer Model in Playground or Get Code Snippet to integrate in your application

FAQ:

How to override the default cost of models?

In case you have custom pricing for your models, you can override the default cost by clicking on Edit Model button and then choosing the Private Cost Metric option.

Edit Model

Set custom cost metric

Can I add models from different regions in a single bedrock integration?

Yes, you can add models from different regions. You can provide a top level default region for the account and also override it at the model level.

How to integrate Bedrock cross-region inference model?

What is Cross-Region Inference?

Cross-Region Inference dynamically routes your inference requests across multiple AWS regions to optimize performance and handle traffic bursts. Bedrock selects the best region based on load, latency, and availability. Learn more in the AWS Bedrock Cross-Region Inference documentation.

Key Difference: Inference Profile ID vs Model ID

To use cross-region inference, you must use an Inference Profile ID instead of a regular model ID. Inference profiles define the foundation model and the AWS regions where requests can be routed.

• Regular Model ID: anthropic.claude-3-5-sonnet-20240620-v1:0 (single region)
• Inference Profile ID: us.anthropic.claude-3-5-sonnet-20240620-v1:0 (cross-region routing)

Inference profile IDs can be:

• System-defined geographic profiles: Use geographic prefixes (us., eu., apac.) followed by the model ID (e.g., us.anthropic.claude-3-5-sonnet-20240620-v1:0). The prefix indicates routing within that geography.
• Custom inference profiles: Use full ARN format (e.g., arn:aws:bedrock:us-east-1:123456789012:inference-profile/my-profile)

How to Use It

1. Add the inference profile ID as a model: When adding a model in TrueFoundry, use the inference profile ID (e.g., us.anthropic.claude-3-5-sonnet-20240620-v1:0) instead of the regular model ID. If it’s not in the dropdown, use + Add Model and enter it manually.

AWS Bedrock cross-region inference configuration interface

2. Configure IAM permissions for ALL destination regions: This is critical. When Bedrock routes to a different region, your IAM role/access key must have permissions in that region. You must grant permissions for both the inference profile and the foundation model in all destination regions.

Update your IAM policy to use * for the region to allow access across all regions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["bedrock:InvokeModel", "bedrock:InvokeModelWithResponseStream"],
      "Resource": ["arn:aws:bedrock:*::foundation-model/*", "arn:aws:bedrock:*:YOUR-AWS-ACCOUNT-ID:inference-profile/*"]
    }
  ]
}

Replace YOUR-AWS-ACCOUNT-ID in the policy above with your actual AWS account ID. The * in the region position allows access across all regions.

Most Common Mistake: Users grant permissions only in their default region. If Bedrock routes to a different region without permissions, requests will fail with “Access Denied”. Always grant permissions in ALL potential destination regions. For geographic profiles, ensure permissions in both source and destination regions.

3. Check Service Control Policies (SCPs): If your organization uses SCPs to restrict region access, ensure they allow access to all destination regions in your inference profile. Blocking any destination region will prevent cross-region inference from working.

Troubleshooting

Request fails with 'Access Denied' error

Cause: Missing IAM permissions in the destination region where Bedrock routed the request.

Solution:

• Ensure your IAM policy grants Bedrock permissions across all regions (use * in the region part of the ARN)
• For geographic profiles, grant permissions in both source and destination regions
• Check if Service Control Policies (SCPs) are blocking access to certain regions

Requests always go to the same region

Cause: You’re using a regular model ID instead of an inference profile ID.

Solution: Use the inference profile ID format (e.g., us.anthropic.claude-3-5-sonnet-20240620-v1:0) instead of the regular model ID.

Learn More

For detailed AWS documentation, see:

• Cross-Region inference overview - How cross-region inference works
• Geographic cross-Region inference - Geographic boundary routing and IAM policy requirements
• Using inference profiles - How to use inference profiles in API calls
• Supported models and regions - List of models that support cross-region inference