Model distillation in Amazon Bedrock: improving efficiency for AI agents
Image generated by the Inferama Editorial Team
01

The cost of inference in AI agents

AI agents often rely on large foundation models to reason through complex problems, use external tools and coordinate multiple systems. This architecture can provide strong capabilities, but it can also bring high inference costs and significant latency at every step.

A substantial share of an agent’s work may involve routine tasks, such as checking calendars or finding documents, that do not require the most capable model. Using frontier models for all these tasks can mean wasted resources and slower responses than the operation’s complexity calls for.

02

What model distillation is

Model distillation is a customization process that transfers knowledge from a more capable model, called a teacher model, to a smaller, less expensive student model. The aim is to improve the student for a particular use case so that it approaches the teacher’s performance while offering a different balance of speed and cost.

In Amazon Bedrock, the workflow automates generating teacher responses and using them to fine-tune the student. Rather than requiring customers to create every prompt-and-response pair manually, distillation can start with prompts alone; the platform generates synthetic responses for training. The resulting behavior still needs to be evaluated for the intended application.

03

Workflow and supported models

A Bedrock distillation workflow starts by selecting a teacher model and a student model. AWS lists supported combinations including Amazon Nova Premier as a teacher and Nova Pro as a student, as well as Claude 3.5 Sonnet v2 and Llama 3.3 70B as teachers, and Llama 3.2 1B or 3B as students. Model availability can depend on the current service configuration and region.

The user provides training data, typically in JSONL format, or grants access to invocation logs stored in Amazon S3. Bedrock may also use data synthesis to expand the training set, for example by generating similar prompts or responses based on supplied examples. This process is intended to help the student behave more like the teacher for the target use case; it does not establish that the student will match the teacher on unrelated tasks.

Models listed for Bedrock distillation

AWS lists the following teacher and student models for its generally available distillation service.

RoleModels listed
TeacherAmazon Nova Premier, Claude 3.5 Sonnet v2, Llama 3.3 70B
StudentAmazon Nova Pro, Llama 3.2 1B, Llama 3.2 3B
04

Performance claims and use cases

AWS says distilled models can be up to 500% faster and 75% less expensive than their original models, with less than 2% accuracy loss for some use cases, including retrieval-augmented generation (RAG). These are provider-reported maximums and a use-case-specific accuracy figure, not a guarantee for every model pair, workload or deployment. Actual comparisons depend on the task, prompts, data, hardware, configuration and measurement method.

AWS also identifies function-call prediction as a use case for distillation in agents. A smaller student may be trained to identify when to call a particular function and what parameters to provide. Whether it does so reliably must be assessed in the application’s own evaluation, including cases where tools should not be called. Faster or less expensive model inference alone does not establish that an agent completes tasks correctly or uses tools safely.

05

Implementation and technical considerations

Customers can start a distillation job from the Amazon Bedrock console or through the CreateModelCustomizationJob API. The job requires a suitable IAM service role that allows Bedrock to read training data and write output files. Users can also configure encryption with AWS Key Management Service (KMS) keys and, where applicable, a virtual private cloud (VPC) to control access to data and resources.

Bedrock can use invocation logs stored in Amazon S3 as a data source. Customers can select prompts alone or prompt-response pairs, and can filter records using request metadata. When using existing responses, the teacher model specified for the job must match the model that produced those responses; otherwise, the logged responses are not used. Using existing pairs avoids regenerating teacher responses, but the resulting training data still depends on the relevance and quality of the selected logs.

Submitting a distillation job

A basic outline of the steps for starting a model-distillation job in Amazon Bedrock.

  1. 01Select a teacher model and a student model supported by the service.
  2. 02Prepare training prompts in JSONL format or configure access to invocation logs in Amazon S3.
  3. 03Specify the S3 locations for input data and job outputs.
  4. 04Set up the required IAM role and choose any applicable encryption or VPC settings.
  5. 05Submit the job, then monitor its status and evaluate the resulting model for the intended task.
06

Keep exploring

Sources consulted

03

Corrections and transparency

If you spot incorrect or outdated information, send us a correction with the page and source we should review.

Submit a correction