I’m completely happy to share that you could now consider, evaluate, and choose the perfect basis fashions (FMs) in your use case in Amazon Bedrock. Mannequin Analysis on Amazon Bedrock is obtainable in the present day in preview.

Amazon Bedrock gives a selection of automated analysis and human analysis. You should use automated analysis with predefined metrics similar to accuracy, robustness, and toxicity. For subjective or customized metrics, similar to friendliness, fashion, and alignment to model voice, you possibly can arrange human analysis workflows with only a few clicks.

Mannequin evaluations are essential in any respect phases of growth. As a developer, you now have analysis instruments obtainable for constructing generative synthetic intelligence (AI) purposes. You can begin by experimenting with totally different fashions within the playground surroundings. To iterate sooner, add automated evaluations of the fashions. Then, while you put together for an preliminary launch or restricted launch, you possibly can incorporate human critiques to assist guarantee high quality.

Let me provide you with a fast tour of Mannequin Analysis on Amazon Bedrock.

Computerized mannequin analysis
With automated mannequin analysis, you possibly can carry your personal knowledge or use built-in, curated datasets and pre-defined metrics for particular duties similar to content material summarization, query and answering, textual content classification, and textual content technology. This takes away the heavy lifting of designing and operating your personal mannequin analysis benchmarks.

To get began, navigate to the Amazon Bedrock console, then choose Mannequin analysis below Evaluation & deployment within the left menu. Create a brand new mannequin analysis and select Computerized.

Subsequent, comply with the setup dialog to decide on the FM you wish to consider and the kind of job, for instance, textual content summarization. Choose the analysis metrics and specify a dataset—both built-in or your personal.

If you happen to carry your personal dataset, ensure it’s in JSON Traces format, and every line incorporates all the key-value pairs that you just wish to consider your mannequin with for the mannequin dimension that you just wish to consider. For instance, if you wish to consider the mannequin on a question-answer job, you’ll format your knowledge as follows (with class being elective):

{"referenceResponse":"Cantal","class":"Capitals","immediate":"Aurillac is the capital of"}
{"referenceResponse":"Bamiyan Province","class":"Capitals","immediate":"Bamiyan metropolis is the capital of"}
{"referenceResponse":"Abkhazia","class":"Capitals","immediate":"Sokhumi is the capital of"}
...

Then, create and run the analysis job to know the mannequin’s task-specific efficiency. As soon as the analysis job is full, you possibly can assessment the ends in the mannequin analysis report.

Human mannequin analysis
For human analysis, you possibly can have Amazon Bedrock arrange human assessment workflows with a couple of clicks. You possibly can carry your personal datasets and outline customized analysis metrics, similar to relevance, fashion, or alignment to model voice. You even have the selection to both leverage your personal inside groups as reviewers or interact an AWS managed workforce. This takes away the tedious effort of constructing and working human analysis workflows.

To get began, create a brand new mannequin analysis and choose Human: Deliver your personal workforce or Human: AWS managed workforce.

If you happen to select an AWS managed workforce for human analysis, describe your mannequin analysis wants, together with job kind, experience of the work workforce, and the approximate variety of prompts, alongside together with your contact data. Within the subsequent step, an AWS skilled will attain out to debate your mannequin analysis venture necessities in additional element. Upon assessment, the workforce will share a customized quote and venture timeline.

If you happen to select to carry your personal workforce, comply with the setup dialog to decide on the FMs you wish to consider and the kind of job, for instance, textual content summarization. Then, choose the analysis metrics, add your take a look at dataset, and arrange the work workforce.

For human analysis, you’ll format the instance knowledge proven earlier than once more in JSON Traces format like this (with class and referenceResponse being elective):

{"immediate":"Aurillac is the capital of","referenceResponse":"Cantal","class":"Capitals"}
{"immediate":"Bamiyan metropolis is the capital of","referenceResponse":"Bamiyan Province","class":"Capitals"}
{"immediate":"Senftenberg is the capital of","referenceResponse":"Oberspreewald-Lausitz","class":"Capitals"}

As soon as the human analysis is accomplished, Amazon Bedrock generates an analysis report with the mannequin’s efficiency towards your chosen metrics.

Issues to know
Listed below are a few essential issues to know:

Mannequin assist – Throughout preview, you possibly can consider and evaluate text-based giant language fashions (LLMs) obtainable on Amazon Bedrock. Throughout preview, you possibly can choose one mannequin for every automated analysis job and as much as two fashions for every human analysis job utilizing your personal workforce. For human analysis utilizing an AWS managed workforce, you possibly can specify customized venture necessities.

Pricing – Throughout preview, AWS solely fees for the mannequin inference wanted to carry out the analysis (processed enter and output tokens for on-demand pricing). There will probably be no separate fees for human analysis or automated analysis. Amazon Bedrock Pricing has all the main points.

Be a part of the preview
Computerized analysis and human analysis utilizing your personal work workforce can be found in the present day in public preview in AWS Areas US East (N. Virginia) and US West (Oregon). Human analysis utilizing an AWS managed workforce is obtainable in public preview in AWS Area US East (N. Virginia). To study extra, go to the Amazon Bedrock Developer Expertise net web page and take a look at the Person Information.

Get began
Log in to the AWS Administration Console and begin exploring mannequin analysis in Amazon Bedrock in the present day!

— Antje