Gen AI Evaluation Toolkit on AWS
v2.0.0
Copyright © 2026 Amazon Web Services, Inc. and/or its affiliates. All rights reserved. Amazon’s trademarks and trade dress may not be used in connection with any product or service that is not Amazon’s, in any manner that is likely to cause confusion among customers, or in any manner that disparages or discredits Amazon. All other trademarks not owned by Amazon are the property of their respective owners, who may or may not be affiliated with, connected to, or sponsored by Amazon.
Table of contents
Python client reference¶
Usage examples¶
Creating test cases with structured data¶
The expected and context fields support both simple strings (for backward compatibility) and structured data (for agentic applications).
from awssolutions_gen_ai_evaluation_toolkit_python_client import ApiClient, Configuration
from awssolutions_gen_ai_evaluation_toolkit_python_client.api import DefaultApi
config = Configuration(host="https://your-api-endpoint")
client = DefaultApi(ApiClient(config))
# Simple string format (backward compatible)
client.create_test_case(
dataset_name="my-dataset",
test_case_data={
"input": "What is Amazon S3?",
"expected": "Amazon S3 is an object storage service.",
"context": ["Amazon S3 is an object storage service that offers..."]
}
)
# Structured data for agentic applications
client.create_test_case(
dataset_name="my-dataset",
test_case_data={
"input": {"prompt": "Book a flight to Seattle for next Monday"},
"expected": {
"toolCalls": [
{"name": "search_flights", "arguments": {"destination": "Seattle"}}
],
"expectedTopics": ["flight booking", "travel"]
},
"context": {
"requiresHumanReview": False,
"evaluationHints": {"checkToolSequence": True}
}
}
)
API client¶
awssolutions_gen_ai_evaluation_toolkit_python_client.api.default_api.DefaultApi ¶
NOTE: This class is auto generated by OpenAPI Generator Ref: https://openapi-generator.tech
Do not edit the class manually.
archive_annotation_schema ¶
archive_annotation_schema(annotation_name, archive_annotation_schema_request_content=None, _request_timeout=None, _request_auth=None, _content_type=None, _headers=None, _host_index=0)
archive_annotation_schema
Mark or unmark an annotation schema as archived to prevent or allow further use
:param annotation_name: Name of the annotation schema (required) :type annotation_name: str :param archive_annotation_schema_request_content: :type archive_annotation_schema_request_content: ArchiveAnnotationSchemaRequestContent :param _request_timeout: timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of (connection, read) timeouts. :type _request_timeout: int, tuple(int, int), optional :param _request_auth: set to override the auth_settings for an a single request; this effectively ignores the authentication in the spec for a single request. :type _request_auth: dict, optional :param _content_type: force content-type for the request. :type _content_type: str, Optional :param _headers: set to override the headers for a single request; this effectively ignores the headers in the spec for a single request. :type _headers: dict, optional :param _host_index: set to override the host_index for a single request; this effectively ignores the host_index in the spec for a single request. :type _host_index: int, optional :return: Returns the result object.
batch_create_results ¶
batch_create_results(report_id, batch_create_results_request_content, _request_timeout=None, _request_auth=None, _content_type=None, _headers=None, _host_index=0)
batch_create_results
Create multiple test results in a batch. Returns 200 with separate success and error lists even if some items fail.
:param report_id: ID of the report to add results to (required) :type report_id: str :param batch_create_results_request_content: (required) :type batch_create_results_request_content: BatchCreateResultsRequestContent :param _request_timeout: timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of (connection, read) timeouts. :type _request_timeout: int, tuple(int, int), optional :param _request_auth: set to override the auth_settings for an a single request; this effectively ignores the authentication in the spec for a single request. :type _request_auth: dict, optional :param _content_type: force content-type for the request. :type _content_type: str, Optional :param _headers: set to override the headers for a single request; this effectively ignores the headers in the spec for a single request. :type _headers: dict, optional :param _host_index: set to override the host_index for a single request; this effectively ignores the host_index in the spec for a single request. :type _host_index: int, optional :return: Returns the result object.
batch_delete_results ¶
batch_delete_results(report_id, batch_delete_results_request_content, _request_timeout=None, _request_auth=None, _content_type=None, _headers=None, _host_index=0)
batch_delete_results
Delete multiple test results by ID. Returns 200 with separate success count and error lists even if some items fail.
:param report_id: ID of the report containing the results (required) :type report_id: str :param batch_delete_results_request_content: (required) :type batch_delete_results_request_content: BatchDeleteResultsRequestContent :param _request_timeout: timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of (connection, read) timeouts. :type _request_timeout: int, tuple(int, int), optional :param _request_auth: set to override the auth_settings for an a single request; this effectively ignores the authentication in the spec for a single request. :type _request_auth: dict, optional :param _content_type: force content-type for the request. :type _content_type: str, Optional :param _headers: set to override the headers for a single request; this effectively ignores the headers in the spec for a single request. :type _headers: dict, optional :param _host_index: set to override the host_index for a single request; this effectively ignores the host_index in the spec for a single request. :type _host_index: int, optional :return: Returns the result object.
batch_update_scores ¶
batch_update_scores(report_id, batch_update_scores_request_content, _request_timeout=None, _request_auth=None, _content_type=None, _headers=None, _host_index=0)
batch_update_scores
Add scores to multiple results in a batch. Returns 200 with separate success and error lists even if some items fail.
:param report_id: ID of the report containing the results (required) :type report_id: str :param batch_update_scores_request_content: (required) :type batch_update_scores_request_content: BatchUpdateScoresRequestContent :param _request_timeout: timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of (connection, read) timeouts. :type _request_timeout: int, tuple(int, int), optional :param _request_auth: set to override the auth_settings for an a single request; this effectively ignores the authentication in the spec for a single request. :type _request_auth: dict, optional :param _content_type: force content-type for the request. :type _content_type: str, Optional :param _headers: set to override the headers for a single request; this effectively ignores the headers in the spec for a single request. :type _headers: dict, optional :param _host_index: set to override the host_index for a single request; this effectively ignores the host_index in the spec for a single request. :type _host_index: int, optional :return: Returns the result object.
create_annotation_schema ¶
create_annotation_schema(create_annotation_schema_request_content, _request_timeout=None, _request_auth=None, _content_type=None, _headers=None, _host_index=0)
create_annotation_schema
Creates a new annotation schema
:param create_annotation_schema_request_content: (required) :type create_annotation_schema_request_content: CreateAnnotationSchemaRequestContent :param _request_timeout: timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of (connection, read) timeouts. :type _request_timeout: int, tuple(int, int), optional :param _request_auth: set to override the auth_settings for an a single request; this effectively ignores the authentication in the spec for a single request. :type _request_auth: dict, optional :param _content_type: force content-type for the request. :type _content_type: str, Optional :param _headers: set to override the headers for a single request; this effectively ignores the headers in the spec for a single request. :type _headers: dict, optional :param _host_index: set to override the host_index for a single request; this effectively ignores the host_index in the spec for a single request. :type _host_index: int, optional :return: Returns the result object.
create_configuration ¶
create_configuration(create_configuration_request_content, _request_timeout=None, _request_auth=None, _content_type=None, _headers=None, _host_index=0)
create_configuration
Creates a new workflow configuration using the supplied workflow details.
:param create_configuration_request_content: (required) :type create_configuration_request_content: CreateConfigurationRequestContent :param _request_timeout: timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of (connection, read) timeouts. :type _request_timeout: int, tuple(int, int), optional :param _request_auth: set to override the auth_settings for an a single request; this effectively ignores the authentication in the spec for a single request. :type _request_auth: dict, optional :param _content_type: force content-type for the request. :type _content_type: str, Optional :param _headers: set to override the headers for a single request; this effectively ignores the headers in the spec for a single request. :type _headers: dict, optional :param _host_index: set to override the host_index for a single request; this effectively ignores the host_index in the spec for a single request. :type _host_index: int, optional :return: Returns the result object.
create_dataset ¶
create_dataset(create_dataset_request_content, _request_timeout=None, _request_auth=None, _content_type=None, _headers=None, _host_index=0)
create_dataset
Creates a new, empty dataset, after which test cases can be added.
:param create_dataset_request_content: (required) :type create_dataset_request_content: CreateDatasetRequestContent :param _request_timeout: timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of (connection, read) timeouts. :type _request_timeout: int, tuple(int, int), optional :param _request_auth: set to override the auth_settings for an a single request; this effectively ignores the authentication in the spec for a single request. :type _request_auth: dict, optional :param _content_type: force content-type for the request. :type _content_type: str, Optional :param _headers: set to override the headers for a single request; this effectively ignores the headers in the spec for a single request. :type _headers: dict, optional :param _host_index: set to override the host_index for a single request; this effectively ignores the host_index in the spec for a single request. :type _host_index: int, optional :return: Returns the result object.
create_dataset_version ¶
create_dataset_version(dataset_id, _request_timeout=None, _request_auth=None, _content_type=None, _headers=None, _host_index=0)
create_dataset_version
Creates a new dataset version using all current test cases
:param dataset_id: Unique identifier of the dataset (required) :type dataset_id: str :param _request_timeout: timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of (connection, read) timeouts. :type _request_timeout: int, tuple(int, int), optional :param _request_auth: set to override the auth_settings for an a single request; this effectively ignores the authentication in the spec for a single request. :type _request_auth: dict, optional :param _content_type: force content-type for the request. :type _content_type: str, Optional :param _headers: set to override the headers for a single request; this effectively ignores the headers in the spec for a single request. :type _headers: dict, optional :param _host_index: set to override the host_index for a single request; this effectively ignores the host_index in the spec for a single request. :type _host_index: int, optional :return: Returns the result object.
create_experiment ¶
create_experiment(create_experiment_request_content, _request_timeout=None, _request_auth=None, _content_type=None, _headers=None, _host_index=0)
create_experiment
Creates a new experiment
:param create_experiment_request_content: (required) :type create_experiment_request_content: CreateExperimentRequestContent :param _request_timeout: timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of (connection, read) timeouts. :type _request_timeout: int, tuple(int, int), optional :param _request_auth: set to override the auth_settings for an a single request; this effectively ignores the authentication in the spec for a single request. :type _request_auth: dict, optional :param _content_type: force content-type for the request. :type _content_type: str, Optional :param _headers: set to override the headers for a single request; this effectively ignores the headers in the spec for a single request. :type _headers: dict, optional :param _host_index: set to override the host_index for a single request; this effectively ignores the host_index in the spec for a single request. :type _host_index: int, optional :return: Returns the result object.
create_job_scheduler ¶
create_job_scheduler(create_job_scheduler_request_content, _request_timeout=None, _request_auth=None, _content_type=None, _headers=None, _host_index=0)
create_job_scheduler
Creates an evaluation job schedule using the supplied job template. The schedule will immediately become active and create new jobs on the supplied cron expression.
:param create_job_scheduler_request_content: (required) :type create_job_scheduler_request_content: CreateJobSchedulerRequestContent :param _request_timeout: timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of (connection, read) timeouts. :type _request_timeout: int, tuple(int, int), optional :param _request_auth: set to override the auth_settings for an a single request; this effectively ignores the authentication in the spec for a single request. :type _request_auth: dict, optional :param _content_type: force content-type for the request. :type _content_type: str, Optional :param _headers: set to override the headers for a single request; this effectively ignores the headers in the spec for a single request. :type _headers: dict, optional :param _host_index: set to override the host_index for a single request; this effectively ignores the host_index in the spec for a single request. :type _host_index: int, optional :return: Returns the result object.
create_report ¶
create_report(create_report_request_content, _request_timeout=None, _request_auth=None, _content_type=None, _headers=None, _host_index=0)
create_report
Create a new report to store test results
:param create_report_request_content: (required) :type create_report_request_content: CreateReportRequestContent :param _request_timeout: timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of (connection, read) timeouts. :type _request_timeout: int, tuple(int, int), optional :param _request_auth: set to override the auth_settings for an a single request; this effectively ignores the authentication in the spec for a single request. :type _request_auth: dict, optional :param _content_type: force content-type for the request. :type _content_type: str, Optional :param _headers: set to override the headers for a single request; this effectively ignores the headers in the spec for a single request. :type _headers: dict, optional :param _host_index: set to override the host_index for a single request; this effectively ignores the host_index in the spec for a single request. :type _host_index: int, optional :return: Returns the result object.
create_test_case ¶
create_test_case(dataset_id, create_test_case_request_content=None, _request_timeout=None, _request_auth=None, _content_type=None, _headers=None, _host_index=0)
create_test_case
Creates a new test case
:param dataset_id: Unique identifier of the dataset (required) :type dataset_id: str :param create_test_case_request_content: :type create_test_case_request_content: CreateTestCaseRequestContent :param _request_timeout: timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of (connection, read) timeouts. :type _request_timeout: int, tuple(int, int), optional :param _request_auth: set to override the auth_settings for an a single request; this effectively ignores the authentication in the spec for a single request. :type _request_auth: dict, optional :param _content_type: force content-type for the request. :type _content_type: str, Optional :param _headers: set to override the headers for a single request; this effectively ignores the headers in the spec for a single request. :type _headers: dict, optional :param _host_index: set to override the host_index for a single request; this effectively ignores the host_index in the spec for a single request. :type _host_index: int, optional :return: Returns the result object.
delete_dataset ¶
delete_dataset(dataset_id, _request_timeout=None, _request_auth=None, _content_type=None, _headers=None, _host_index=0)
delete_dataset
Deletes the dataset
:param dataset_id: Unique identifier of the dataset (required) :type dataset_id: str :param _request_timeout: timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of (connection, read) timeouts. :type _request_timeout: int, tuple(int, int), optional :param _request_auth: set to override the auth_settings for an a single request; this effectively ignores the authentication in the spec for a single request. :type _request_auth: dict, optional :param _content_type: force content-type for the request. :type _content_type: str, Optional :param _headers: set to override the headers for a single request; this effectively ignores the headers in the spec for a single request. :type _headers: dict, optional :param _host_index: set to override the host_index for a single request; this effectively ignores the host_index in the spec for a single request. :type _host_index: int, optional :return: Returns the result object.
delete_dataset_version_annotation ¶
delete_dataset_version_annotation(dataset_id, version, annotation_name, _request_timeout=None, _request_auth=None, _content_type=None, _headers=None, _host_index=0)
delete_dataset_version_annotation
Removes the requested annotation from the given dataset version.
:param dataset_id: Unique identifier of the dataset (required) :type dataset_id: str :param version: Version number (required) :type version: float :param annotation_name: Name of the annotation schema (required) :type annotation_name: str :param _request_timeout: timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of (connection, read) timeouts. :type _request_timeout: int, tuple(int, int), optional :param _request_auth: set to override the auth_settings for an a single request; this effectively ignores the authentication in the spec for a single request. :type _request_auth: dict, optional :param _content_type: force content-type for the request. :type _content_type: str, Optional :param _headers: set to override the headers for a single request; this effectively ignores the headers in the spec for a single request. :type _headers: dict, optional :param _host_index: set to override the host_index for a single request; this effectively ignores the host_index in the spec for a single request. :type _host_index: int, optional :return: Returns the result object.
delete_experiment ¶
delete_experiment(name, _request_timeout=None, _request_auth=None, _content_type=None, _headers=None, _host_index=0)
delete_experiment
Deletes an experiment
:param name: The unique identifier of the experiment (required) :type name: str :param _request_timeout: timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of (connection, read) timeouts. :type _request_timeout: int, tuple(int, int), optional :param _request_auth: set to override the auth_settings for an a single request; this effectively ignores the authentication in the spec for a single request. :type _request_auth: dict, optional :param _content_type: force content-type for the request. :type _content_type: str, Optional :param _headers: set to override the headers for a single request; this effectively ignores the headers in the spec for a single request. :type _headers: dict, optional :param _host_index: set to override the host_index for a single request; this effectively ignores the host_index in the spec for a single request. :type _host_index: int, optional :return: Returns the result object.
delete_job_scheduler ¶
delete_job_scheduler(name, _request_timeout=None, _request_auth=None, _content_type=None, _headers=None, _host_index=0)
delete_job_scheduler
Deletes evaluation job scheduler (existing jobs unaffected)
:param name: Unique name of the job scheduler (required) :type name: str :param _request_timeout: timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of (connection, read) timeouts. :type _request_timeout: int, tuple(int, int), optional :param _request_auth: set to override the auth_settings for an a single request; this effectively ignores the authentication in the spec for a single request. :type _request_auth: dict, optional :param _content_type: force content-type for the request. :type _content_type: str, Optional :param _headers: set to override the headers for a single request; this effectively ignores the headers in the spec for a single request. :type _headers: dict, optional :param _host_index: set to override the host_index for a single request; this effectively ignores the host_index in the spec for a single request. :type _host_index: int, optional :return: Returns the result object.
delete_report ¶
delete_report(report_id, _request_timeout=None, _request_auth=None, _content_type=None, _headers=None, _host_index=0)
delete_report
Delete a report and all its results
:param report_id: ID of the report to delete (required) :type report_id: str :param _request_timeout: timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of (connection, read) timeouts. :type _request_timeout: int, tuple(int, int), optional :param _request_auth: set to override the auth_settings for an a single request; this effectively ignores the authentication in the spec for a single request. :type _request_auth: dict, optional :param _content_type: force content-type for the request. :type _content_type: str, Optional :param _headers: set to override the headers for a single request; this effectively ignores the headers in the spec for a single request. :type _headers: dict, optional :param _host_index: set to override the host_index for a single request; this effectively ignores the host_index in the spec for a single request. :type _host_index: int, optional :return: Returns the result object.
delete_test_case ¶
delete_test_case(dataset_id, test_case_id, _request_timeout=None, _request_auth=None, _content_type=None, _headers=None, _host_index=0)
delete_test_case
Deletes test case from dataset
:param dataset_id: Unique identifier of the dataset (required) :type dataset_id: str :param test_case_id: Unique identifier of the testcase (required) :type test_case_id: str :param _request_timeout: timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of (connection, read) timeouts. :type _request_timeout: int, tuple(int, int), optional :param _request_auth: set to override the auth_settings for an a single request; this effectively ignores the authentication in the spec for a single request. :type _request_auth: dict, optional :param _content_type: force content-type for the request. :type _content_type: str, Optional :param _headers: set to override the headers for a single request; this effectively ignores the headers in the spec for a single request. :type _headers: dict, optional :param _host_index: set to override the host_index for a single request; this effectively ignores the host_index in the spec for a single request. :type _host_index: int, optional :return: Returns the result object.
delete_test_case_annotation ¶
delete_test_case_annotation(dataset_id, test_case_id, annotation_name, _request_timeout=None, _request_auth=None, _content_type=None, _headers=None, _host_index=0)
delete_test_case_annotation
Removes the requested annotation from the given testcase.
:param dataset_id: Unique identifier of the dataset (required) :type dataset_id: str :param test_case_id: Unique identifier of the testcase (required) :type test_case_id: str :param annotation_name: Name of the annotation schema (required) :type annotation_name: str :param _request_timeout: timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of (connection, read) timeouts. :type _request_timeout: int, tuple(int, int), optional :param _request_auth: set to override the auth_settings for an a single request; this effectively ignores the authentication in the spec for a single request. :type _request_auth: dict, optional :param _content_type: force content-type for the request. :type _content_type: str, Optional :param _headers: set to override the headers for a single request; this effectively ignores the headers in the spec for a single request. :type _headers: dict, optional :param _host_index: set to override the host_index for a single request; this effectively ignores the host_index in the spec for a single request. :type _host_index: int, optional :return: Returns the result object.
get_annotation_schema ¶
get_annotation_schema(annotation_name, _request_timeout=None, _request_auth=None, _content_type=None, _headers=None, _host_index=0)
get_annotation_schema
Gets annotation schema details
:param annotation_name: Name of the annotation schema (required) :type annotation_name: str :param _request_timeout: timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of (connection, read) timeouts. :type _request_timeout: int, tuple(int, int), optional :param _request_auth: set to override the auth_settings for an a single request; this effectively ignores the authentication in the spec for a single request. :type _request_auth: dict, optional :param _content_type: force content-type for the request. :type _content_type: str, Optional :param _headers: set to override the headers for a single request; this effectively ignores the headers in the spec for a single request. :type _headers: dict, optional :param _host_index: set to override the host_index for a single request; this effectively ignores the host_index in the spec for a single request. :type _host_index: int, optional :return: Returns the result object.
get_configuration ¶
get_configuration(name, version=None, _request_timeout=None, _request_auth=None, _content_type=None, _headers=None, _host_index=0)
get_configuration
Gets the details of the requested workflow configuration.
:param name: Name of the configuration (required) :type name: str :param version: Version of the configuration :type version: float :param _request_timeout: timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of (connection, read) timeouts. :type _request_timeout: int, tuple(int, int), optional :param _request_auth: set to override the auth_settings for an a single request; this effectively ignores the authentication in the spec for a single request. :type _request_auth: dict, optional :param _content_type: force content-type for the request. :type _content_type: str, Optional :param _headers: set to override the headers for a single request; this effectively ignores the headers in the spec for a single request. :type _headers: dict, optional :param _host_index: set to override the host_index for a single request; this effectively ignores the host_index in the spec for a single request. :type _host_index: int, optional :return: Returns the result object.
get_dataset ¶
get_dataset(dataset_id, _request_timeout=None, _request_auth=None, _content_type=None, _headers=None, _host_index=0)
get_dataset
Gets dataset details
:param dataset_id: Unique identifier of the dataset (required) :type dataset_id: str :param _request_timeout: timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of (connection, read) timeouts. :type _request_timeout: int, tuple(int, int), optional :param _request_auth: set to override the auth_settings for an a single request; this effectively ignores the authentication in the spec for a single request. :type _request_auth: dict, optional :param _content_type: force content-type for the request. :type _content_type: str, Optional :param _headers: set to override the headers for a single request; this effectively ignores the headers in the spec for a single request. :type _headers: dict, optional :param _host_index: set to override the host_index for a single request; this effectively ignores the host_index in the spec for a single request. :type _host_index: int, optional :return: Returns the result object.
get_dataset_version ¶
get_dataset_version(dataset_id, version, _request_timeout=None, _request_auth=None, _content_type=None, _headers=None, _host_index=0)
get_dataset_version
Gets version details
:param dataset_id: Unique identifier of the dataset (required) :type dataset_id: str :param version: Version number or tag (required) :type version: str :param _request_timeout: timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of (connection, read) timeouts. :type _request_timeout: int, tuple(int, int), optional :param _request_auth: set to override the auth_settings for an a single request; this effectively ignores the authentication in the spec for a single request. :type _request_auth: dict, optional :param _content_type: force content-type for the request. :type _content_type: str, Optional :param _headers: set to override the headers for a single request; this effectively ignores the headers in the spec for a single request. :type _headers: dict, optional :param _host_index: set to override the host_index for a single request; this effectively ignores the host_index in the spec for a single request. :type _host_index: int, optional :return: Returns the result object.
get_evaluator_job ¶
get_evaluator_job(job_id, _request_timeout=None, _request_auth=None, _content_type=None, _headers=None, _host_index=0)
get_evaluator_job
Get details of an evaluation job (v2)
:param job_id: ID of the job (required) :type job_id: str :param _request_timeout: timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of (connection, read) timeouts. :type _request_timeout: int, tuple(int, int), optional :param _request_auth: set to override the auth_settings for an a single request; this effectively ignores the authentication in the spec for a single request. :type _request_auth: dict, optional :param _content_type: force content-type for the request. :type _content_type: str, Optional :param _headers: set to override the headers for a single request; this effectively ignores the headers in the spec for a single request. :type _headers: dict, optional :param _host_index: set to override the host_index for a single request; this effectively ignores the host_index in the spec for a single request. :type _host_index: int, optional :return: Returns the result object.
get_experiment ¶
get_experiment(name, _request_timeout=None, _request_auth=None, _content_type=None, _headers=None, _host_index=0)
get_experiment
Gets experiment details
:param name: The unique identifier of the experiment (required) :type name: str :param _request_timeout: timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of (connection, read) timeouts. :type _request_timeout: int, tuple(int, int), optional :param _request_auth: set to override the auth_settings for an a single request; this effectively ignores the authentication in the spec for a single request. :type _request_auth: dict, optional :param _content_type: force content-type for the request. :type _content_type: str, Optional :param _headers: set to override the headers for a single request; this effectively ignores the headers in the spec for a single request. :type _headers: dict, optional :param _host_index: set to override the host_index for a single request; this effectively ignores the host_index in the spec for a single request. :type _host_index: int, optional :return: Returns the result object.
get_generation ¶
get_generation(id, _request_timeout=None, _request_auth=None, _content_type=None, _headers=None, _host_index=0)
get_generation
Gets generation job details
:param id: Unique identifier of the job (required) :type id: str :param _request_timeout: timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of (connection, read) timeouts. :type _request_timeout: int, tuple(int, int), optional :param _request_auth: set to override the auth_settings for an a single request; this effectively ignores the authentication in the spec for a single request. :type _request_auth: dict, optional :param _content_type: force content-type for the request. :type _content_type: str, Optional :param _headers: set to override the headers for a single request; this effectively ignores the headers in the spec for a single request. :type _headers: dict, optional :param _host_index: set to override the host_index for a single request; this effectively ignores the host_index in the spec for a single request. :type _host_index: int, optional :return: Returns the result object.
get_job_scheduler ¶
get_job_scheduler(name, _request_timeout=None, _request_auth=None, _content_type=None, _headers=None, _host_index=0)
get_job_scheduler
Gets evaluation job scheduler details
:param name: Unique name of the job scheduler (required) :type name: str :param _request_timeout: timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of (connection, read) timeouts. :type _request_timeout: int, tuple(int, int), optional :param _request_auth: set to override the auth_settings for an a single request; this effectively ignores the authentication in the spec for a single request. :type _request_auth: dict, optional :param _content_type: force content-type for the request. :type _content_type: str, Optional :param _headers: set to override the headers for a single request; this effectively ignores the headers in the spec for a single request. :type _headers: dict, optional :param _host_index: set to override the host_index for a single request; this effectively ignores the host_index in the spec for a single request. :type _host_index: int, optional :return: Returns the result object.
get_report ¶
get_report(report_id, _request_timeout=None, _request_auth=None, _content_type=None, _headers=None, _host_index=0)
get_report
Get a report by ID
:param report_id: ID of the report to retrieve (required) :type report_id: str :param _request_timeout: timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of (connection, read) timeouts. :type _request_timeout: int, tuple(int, int), optional :param _request_auth: set to override the auth_settings for an a single request; this effectively ignores the authentication in the spec for a single request. :type _request_auth: dict, optional :param _content_type: force content-type for the request. :type _content_type: str, Optional :param _headers: set to override the headers for a single request; this effectively ignores the headers in the spec for a single request. :type _headers: dict, optional :param _host_index: set to override the host_index for a single request; this effectively ignores the host_index in the spec for a single request. :type _host_index: int, optional :return: Returns the result object.
get_test_case ¶
get_test_case(dataset_id, version, test_case_id, _request_timeout=None, _request_auth=None, _content_type=None, _headers=None, _host_index=0)
get_test_case
Gets test case for dataset version
:param dataset_id: Unique identifier of the dataset (required) :type dataset_id: str :param version: Version number or tag (required) :type version: str :param test_case_id: Unique identifier of the testcase (required) :type test_case_id: str :param _request_timeout: timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of (connection, read) timeouts. :type _request_timeout: int, tuple(int, int), optional :param _request_auth: set to override the auth_settings for an a single request; this effectively ignores the authentication in the spec for a single request. :type _request_auth: dict, optional :param _content_type: force content-type for the request. :type _content_type: str, Optional :param _headers: set to override the headers for a single request; this effectively ignores the headers in the spec for a single request. :type _headers: dict, optional :param _host_index: set to override the host_index for a single request; this effectively ignores the host_index in the spec for a single request. :type _host_index: int, optional :return: Returns the result object.
get_test_result ¶
get_test_result(report_id, result_id, _request_timeout=None, _request_auth=None, _content_type=None, _headers=None, _host_index=0)
get_test_result
Get a single test result by ID
:param report_id: ID of the report containing the result (required) :type report_id: str :param result_id: ID of the result to retrieve (required) :type result_id: str :param _request_timeout: timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of (connection, read) timeouts. :type _request_timeout: int, tuple(int, int), optional :param _request_auth: set to override the auth_settings for an a single request; this effectively ignores the authentication in the spec for a single request. :type _request_auth: dict, optional :param _content_type: force content-type for the request. :type _content_type: str, Optional :param _headers: set to override the headers for a single request; this effectively ignores the headers in the spec for a single request. :type _headers: dict, optional :param _host_index: set to override the host_index for a single request; this effectively ignores the host_index in the spec for a single request. :type _host_index: int, optional :return: Returns the result object.
list_annotation_schemas ¶
list_annotation_schemas(max_results=None, next_token=None, include_archived=None, _request_timeout=None, _request_auth=None, _content_type=None, _headers=None, _host_index=0)
list_annotation_schemas
Lists available annotation schemas
:param max_results: Maximum number of results to return :type max_results: float :param next_token: Pagination token for next page of results :type next_token: str :param include_archived: Whether to include archived annotation schemas in the results :type include_archived: bool :param _request_timeout: timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of (connection, read) timeouts. :type _request_timeout: int, tuple(int, int), optional :param _request_auth: set to override the auth_settings for an a single request; this effectively ignores the authentication in the spec for a single request. :type _request_auth: dict, optional :param _content_type: force content-type for the request. :type _content_type: str, Optional :param _headers: set to override the headers for a single request; this effectively ignores the headers in the spec for a single request. :type _headers: dict, optional :param _host_index: set to override the host_index for a single request; this effectively ignores the host_index in the spec for a single request. :type _host_index: int, optional :return: Returns the result object.
list_configurations ¶
list_configurations(max_results=None, next_token=None, list_configurations_request_content=None, _request_timeout=None, _request_auth=None, _content_type=None, _headers=None, _host_index=0)
list_configurations
Lists workflow configurations
:param max_results: Maximum number of results to return :type max_results: float :param next_token: Pagination token for next page of results :type next_token: str :param list_configurations_request_content: :type list_configurations_request_content: ListConfigurationsRequestContent :param _request_timeout: timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of (connection, read) timeouts. :type _request_timeout: int, tuple(int, int), optional :param _request_auth: set to override the auth_settings for an a single request; this effectively ignores the authentication in the spec for a single request. :type _request_auth: dict, optional :param _content_type: force content-type for the request. :type _content_type: str, Optional :param _headers: set to override the headers for a single request; this effectively ignores the headers in the spec for a single request. :type _headers: dict, optional :param _host_index: set to override the host_index for a single request; this effectively ignores the host_index in the spec for a single request. :type _host_index: int, optional :return: Returns the result object.
list_dataset_versions ¶
list_dataset_versions(dataset_id, max_results=None, next_token=None, list_dataset_versions_request_content=None, _request_timeout=None, _request_auth=None, _content_type=None, _headers=None, _host_index=0)
list_dataset_versions
Lists available dataset versions for the requested dataset.
:param dataset_id: Unique identifier of the dataset (required) :type dataset_id: str :param max_results: Maximum number of results to return :type max_results: float :param next_token: Pagination token for next page of results :type next_token: str :param list_dataset_versions_request_content: :type list_dataset_versions_request_content: ListDatasetVersionsRequestContent :param _request_timeout: timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of (connection, read) timeouts. :type _request_timeout: int, tuple(int, int), optional :param _request_auth: set to override the auth_settings for an a single request; this effectively ignores the authentication in the spec for a single request. :type _request_auth: dict, optional :param _content_type: force content-type for the request. :type _content_type: str, Optional :param _headers: set to override the headers for a single request; this effectively ignores the headers in the spec for a single request. :type _headers: dict, optional :param _host_index: set to override the host_index for a single request; this effectively ignores the host_index in the spec for a single request. :type _host_index: int, optional :return: Returns the result object.
list_datasets ¶
list_datasets(max_results=None, next_token=None, _request_timeout=None, _request_auth=None, _content_type=None, _headers=None, _host_index=0)
list_datasets
Lists available datasets
:param max_results: Maximum number of results to return :type max_results: float :param next_token: Pagination token for next page of results :type next_token: str :param _request_timeout: timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of (connection, read) timeouts. :type _request_timeout: int, tuple(int, int), optional :param _request_auth: set to override the auth_settings for an a single request; this effectively ignores the authentication in the spec for a single request. :type _request_auth: dict, optional :param _content_type: force content-type for the request. :type _content_type: str, Optional :param _headers: set to override the headers for a single request; this effectively ignores the headers in the spec for a single request. :type _headers: dict, optional :param _host_index: set to override the host_index for a single request; this effectively ignores the host_index in the spec for a single request. :type _host_index: int, optional :return: Returns the result object.
list_experiment_outputs ¶
list_experiment_outputs(name, list_experiment_outputs_request_content, max_results=None, next_token=None, _request_timeout=None, _request_auth=None, _content_type=None, _headers=None, _host_index=0)
list_experiment_outputs
Lists available dataset versions for the requested experiment.
:param name: The unique identifier of the experiment (required) :type name: str :param list_experiment_outputs_request_content: (required) :type list_experiment_outputs_request_content: ListExperimentOutputsRequestContent :param max_results: Maximum number of results to return :type max_results: float :param next_token: Pagination token for next page of results :type next_token: str :param _request_timeout: timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of (connection, read) timeouts. :type _request_timeout: int, tuple(int, int), optional :param _request_auth: set to override the auth_settings for an a single request; this effectively ignores the authentication in the spec for a single request. :type _request_auth: dict, optional :param _content_type: force content-type for the request. :type _content_type: str, Optional :param _headers: set to override the headers for a single request; this effectively ignores the headers in the spec for a single request. :type _headers: dict, optional :param _host_index: set to override the host_index for a single request; this effectively ignores the host_index in the spec for a single request. :type _host_index: int, optional :return: Returns the result object.
list_experiments ¶
list_experiments(max_results=None, next_token=None, list_experiments_request_content=None, _request_timeout=None, _request_auth=None, _content_type=None, _headers=None, _host_index=0)
list_experiments
Lists experiments
:param max_results: Maximum number of results to return :type max_results: float :param next_token: Pagination token for next page of results :type next_token: str :param list_experiments_request_content: :type list_experiments_request_content: ListExperimentsRequestContent :param _request_timeout: timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of (connection, read) timeouts. :type _request_timeout: int, tuple(int, int), optional :param _request_auth: set to override the auth_settings for an a single request; this effectively ignores the authentication in the spec for a single request. :type _request_auth: dict, optional :param _content_type: force content-type for the request. :type _content_type: str, Optional :param _headers: set to override the headers for a single request; this effectively ignores the headers in the spec for a single request. :type _headers: dict, optional :param _host_index: set to override the host_index for a single request; this effectively ignores the host_index in the spec for a single request. :type _host_index: int, optional :return: Returns the result object.
list_generations ¶
list_generations(max_results=None, next_token=None, status_filter=None, _request_timeout=None, _request_auth=None, _content_type=None, _headers=None, _host_index=0)
list_generations
Lists generation jobs with optional filtering
:param max_results: Maximum number of results to return :type max_results: float :param next_token: Pagination token for next page of results :type next_token: str :param status_filter: Filter by matching job status :type status_filter: StepFunctionJobStatus :param _request_timeout: timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of (connection, read) timeouts. :type _request_timeout: int, tuple(int, int), optional :param _request_auth: set to override the auth_settings for an a single request; this effectively ignores the authentication in the spec for a single request. :type _request_auth: dict, optional :param _content_type: force content-type for the request. :type _content_type: str, Optional :param _headers: set to override the headers for a single request; this effectively ignores the headers in the spec for a single request. :type _headers: dict, optional :param _host_index: set to override the host_index for a single request; this effectively ignores the host_index in the spec for a single request. :type _host_index: int, optional :return: Returns the result object.
list_job_schedulers ¶
list_job_schedulers(max_results=None, next_token=None, _request_timeout=None, _request_auth=None, _content_type=None, _headers=None, _host_index=0)
list_job_schedulers
Lists all evaluation job schedules.
:param max_results: Maximum number of results to return :type max_results: float :param next_token: Pagination token for next page of results :type next_token: str :param _request_timeout: timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of (connection, read) timeouts. :type _request_timeout: int, tuple(int, int), optional :param _request_auth: set to override the auth_settings for an a single request; this effectively ignores the authentication in the spec for a single request. :type _request_auth: dict, optional :param _content_type: force content-type for the request. :type _content_type: str, Optional :param _headers: set to override the headers for a single request; this effectively ignores the headers in the spec for a single request. :type _headers: dict, optional :param _host_index: set to override the host_index for a single request; this effectively ignores the host_index in the spec for a single request. :type _host_index: int, optional :return: Returns the result object.
list_jobs ¶
list_jobs(max_results=None, next_token=None, _request_timeout=None, _request_auth=None, _content_type=None, _headers=None, _host_index=0)
list_jobs
List all evaluation jobs
:param max_results: Maximum number of results to return :type max_results: float :param next_token: Pagination token for next page of results :type next_token: str :param _request_timeout: timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of (connection, read) timeouts. :type _request_timeout: int, tuple(int, int), optional :param _request_auth: set to override the auth_settings for an a single request; this effectively ignores the authentication in the spec for a single request. :type _request_auth: dict, optional :param _content_type: force content-type for the request. :type _content_type: str, Optional :param _headers: set to override the headers for a single request; this effectively ignores the headers in the spec for a single request. :type _headers: dict, optional :param _host_index: set to override the host_index for a single request; this effectively ignores the host_index in the spec for a single request. :type _host_index: int, optional :return: Returns the result object.
list_reports ¶
list_reports(max_results=None, next_token=None, _request_timeout=None, _request_auth=None, _content_type=None, _headers=None, _host_index=0)
list_reports
List all reports. Default maxResults is 50 if not specified.
:param max_results: Maximum number of results to return :type max_results: float :param next_token: Pagination token for next page of results :type next_token: str :param _request_timeout: timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of (connection, read) timeouts. :type _request_timeout: int, tuple(int, int), optional :param _request_auth: set to override the auth_settings for an a single request; this effectively ignores the authentication in the spec for a single request. :type _request_auth: dict, optional :param _content_type: force content-type for the request. :type _content_type: str, Optional :param _headers: set to override the headers for a single request; this effectively ignores the headers in the spec for a single request. :type _headers: dict, optional :param _host_index: set to override the host_index for a single request; this effectively ignores the host_index in the spec for a single request. :type _host_index: int, optional :return: Returns the result object.
list_test_cases ¶
list_test_cases(dataset_id, version, max_results=None, next_token=None, list_test_cases_request_content=None, _request_timeout=None, _request_auth=None, _content_type=None, _headers=None, _host_index=0)
list_test_cases
Lists test cases within the requested dataset version.
:param dataset_id: Unique identifier of the dataset (required) :type dataset_id: str :param version: Version number or tag (required) :type version: str :param max_results: Maximum number of results to return :type max_results: float :param next_token: Pagination token for next page of results :type next_token: str :param list_test_cases_request_content: :type list_test_cases_request_content: ListTestCasesRequestContent :param _request_timeout: timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of (connection, read) timeouts. :type _request_timeout: int, tuple(int, int), optional :param _request_auth: set to override the auth_settings for an a single request; this effectively ignores the authentication in the spec for a single request. :type _request_auth: dict, optional :param _content_type: force content-type for the request. :type _content_type: str, Optional :param _headers: set to override the headers for a single request; this effectively ignores the headers in the spec for a single request. :type _headers: dict, optional :param _host_index: set to override the host_index for a single request; this effectively ignores the host_index in the spec for a single request. :type _host_index: int, optional :return: Returns the result object.
list_test_results ¶
list_test_results(report_id, max_results=None, next_token=None, _request_timeout=None, _request_auth=None, _content_type=None, _headers=None, _host_index=0)
list_test_results
List test results within a report. Default maxResults is 50 if not specified.
:param report_id: ID of the report to list results from (required) :type report_id: str :param max_results: Maximum number of results to return :type max_results: float :param next_token: Pagination token for next page of results :type next_token: str :param _request_timeout: timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of (connection, read) timeouts. :type _request_timeout: int, tuple(int, int), optional :param _request_auth: set to override the auth_settings for an a single request; this effectively ignores the authentication in the spec for a single request. :type _request_auth: dict, optional :param _content_type: force content-type for the request. :type _content_type: str, Optional :param _headers: set to override the headers for a single request; this effectively ignores the headers in the spec for a single request. :type _headers: dict, optional :param _host_index: set to override the host_index for a single request; this effectively ignores the host_index in the spec for a single request. :type _host_index: int, optional :return: Returns the result object.
put_dataset_version_annotation ¶
put_dataset_version_annotation(dataset_id, version, put_dataset_version_annotation_request_content, _request_timeout=None, _request_auth=None, _content_type=None, _headers=None, _host_index=0)
put_dataset_version_annotation
Creates or updates an annotation on the given dataset version.
:param dataset_id: Unique identifier of the dataset (required) :type dataset_id: str :param version: Version number (required) :type version: float :param put_dataset_version_annotation_request_content: (required) :type put_dataset_version_annotation_request_content: PutDatasetVersionAnnotationRequestContent :param _request_timeout: timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of (connection, read) timeouts. :type _request_timeout: int, tuple(int, int), optional :param _request_auth: set to override the auth_settings for an a single request; this effectively ignores the authentication in the spec for a single request. :type _request_auth: dict, optional :param _content_type: force content-type for the request. :type _content_type: str, Optional :param _headers: set to override the headers for a single request; this effectively ignores the headers in the spec for a single request. :type _headers: dict, optional :param _host_index: set to override the host_index for a single request; this effectively ignores the host_index in the spec for a single request. :type _host_index: int, optional :return: Returns the result object.
put_test_case_annotation ¶
put_test_case_annotation(dataset_id, test_case_id, put_test_case_annotation_request_content, _request_timeout=None, _request_auth=None, _content_type=None, _headers=None, _host_index=0)
put_test_case_annotation
Creates or updates an annotation on the given test case.
:param dataset_id: Unique identifier of the dataset (required) :type dataset_id: str :param test_case_id: Unique identifier of the testcase (required) :type test_case_id: str :param put_test_case_annotation_request_content: (required) :type put_test_case_annotation_request_content: PutTestCaseAnnotationRequestContent :param _request_timeout: timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of (connection, read) timeouts. :type _request_timeout: int, tuple(int, int), optional :param _request_auth: set to override the auth_settings for an a single request; this effectively ignores the authentication in the spec for a single request. :type _request_auth: dict, optional :param _content_type: force content-type for the request. :type _content_type: str, Optional :param _headers: set to override the headers for a single request; this effectively ignores the headers in the spec for a single request. :type _headers: dict, optional :param _host_index: set to override the host_index for a single request; this effectively ignores the host_index in the spec for a single request. :type _host_index: int, optional :return: Returns the result object.
restore_dataset ¶
restore_dataset(dataset_id, restore_dataset_request_content, _request_timeout=None, _request_auth=None, _content_type=None, _headers=None, _host_index=0)
restore_dataset
Restores a dataset to specified version
:param dataset_id: Unique identifier of the dataset (required) :type dataset_id: str :param restore_dataset_request_content: (required) :type restore_dataset_request_content: RestoreDatasetRequestContent :param _request_timeout: timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of (connection, read) timeouts. :type _request_timeout: int, tuple(int, int), optional :param _request_auth: set to override the auth_settings for an a single request; this effectively ignores the authentication in the spec for a single request. :type _request_auth: dict, optional :param _content_type: force content-type for the request. :type _content_type: str, Optional :param _headers: set to override the headers for a single request; this effectively ignores the headers in the spec for a single request. :type _headers: dict, optional :param _host_index: set to override the host_index for a single request; this effectively ignores the host_index in the spec for a single request. :type _host_index: int, optional :return: Returns the result object.
start_generation ¶
start_generation(start_generation_request_content, _request_timeout=None, _request_auth=None, _content_type=None, _headers=None, _host_index=0)
start_generation
Starts generation job with specified steps
:param start_generation_request_content: (required) :type start_generation_request_content: StartGenerationRequestContent :param _request_timeout: timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of (connection, read) timeouts. :type _request_timeout: int, tuple(int, int), optional :param _request_auth: set to override the auth_settings for an a single request; this effectively ignores the authentication in the spec for a single request. :type _request_auth: dict, optional :param _content_type: force content-type for the request. :type _content_type: str, Optional :param _headers: set to override the headers for a single request; this effectively ignores the headers in the spec for a single request. :type _headers: dict, optional :param _host_index: set to override the host_index for a single request; this effectively ignores the host_index in the spec for a single request. :type _host_index: int, optional :return: Returns the result object.
stop_generation ¶
stop_generation(id, stop_generation_request_content=None, _request_timeout=None, _request_auth=None, _content_type=None, _headers=None, _host_index=0)
stop_generation
Stops the generation job
:param id: Unique identifier of the job (required) :type id: str :param stop_generation_request_content: :type stop_generation_request_content: StopGenerationRequestContent :param _request_timeout: timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of (connection, read) timeouts. :type _request_timeout: int, tuple(int, int), optional :param _request_auth: set to override the auth_settings for an a single request; this effectively ignores the authentication in the spec for a single request. :type _request_auth: dict, optional :param _content_type: force content-type for the request. :type _content_type: str, Optional :param _headers: set to override the headers for a single request; this effectively ignores the headers in the spec for a single request. :type _headers: dict, optional :param _host_index: set to override the host_index for a single request; this effectively ignores the host_index in the spec for a single request. :type _host_index: int, optional :return: Returns the result object.
submit_app_invocation_job ¶
submit_app_invocation_job(evaluator_name, submit_app_invocation_job_request_content, _request_timeout=None, _request_auth=None, _content_type=None, _headers=None, _host_index=0)
submit_app_invocation_job
Submit an app invocation job: fetch test data, invoke app, record results (no scoring)
:param evaluator_name: Name of the evaluator (required) :type evaluator_name: str :param submit_app_invocation_job_request_content: (required) :type submit_app_invocation_job_request_content: SubmitAppInvocationJobRequestContent :param _request_timeout: timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of (connection, read) timeouts. :type _request_timeout: int, tuple(int, int), optional :param _request_auth: set to override the auth_settings for an a single request; this effectively ignores the authentication in the spec for a single request. :type _request_auth: dict, optional :param _content_type: force content-type for the request. :type _content_type: str, Optional :param _headers: set to override the headers for a single request; this effectively ignores the headers in the spec for a single request. :type _headers: dict, optional :param _host_index: set to override the host_index for a single request; this effectively ignores the host_index in the spec for a single request. :type _host_index: int, optional :return: Returns the result object.
submit_evaluation_job ¶
submit_evaluation_job(evaluator_name, submit_evaluation_job_request_content, _request_timeout=None, _request_auth=None, _content_type=None, _headers=None, _host_index=0)
submit_evaluation_job
Submit a full evaluation job: fetch test data, invoke app, and score results
:param evaluator_name: Name of the evaluator (required) :type evaluator_name: str :param submit_evaluation_job_request_content: (required) :type submit_evaluation_job_request_content: SubmitEvaluationJobRequestContent :param _request_timeout: timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of (connection, read) timeouts. :type _request_timeout: int, tuple(int, int), optional :param _request_auth: set to override the auth_settings for an a single request; this effectively ignores the authentication in the spec for a single request. :type _request_auth: dict, optional :param _content_type: force content-type for the request. :type _content_type: str, Optional :param _headers: set to override the headers for a single request; this effectively ignores the headers in the spec for a single request. :type _headers: dict, optional :param _host_index: set to override the host_index for a single request; this effectively ignores the host_index in the spec for a single request. :type _host_index: int, optional :return: Returns the result object.
submit_scoring_job ¶
submit_scoring_job(evaluator_name, submit_scoring_job_request_content=None, _request_timeout=None, _request_auth=None, _content_type=None, _headers=None, _host_index=0)
submit_scoring_job
Submit a scoring-only job: score existing results in a report
:param evaluator_name: Name of the evaluator (required) :type evaluator_name: str :param submit_scoring_job_request_content: :type submit_scoring_job_request_content: SubmitScoringJobRequestContent :param _request_timeout: timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of (connection, read) timeouts. :type _request_timeout: int, tuple(int, int), optional :param _request_auth: set to override the auth_settings for an a single request; this effectively ignores the authentication in the spec for a single request. :type _request_auth: dict, optional :param _content_type: force content-type for the request. :type _content_type: str, Optional :param _headers: set to override the headers for a single request; this effectively ignores the headers in the spec for a single request. :type _headers: dict, optional :param _host_index: set to override the host_index for a single request; this effectively ignores the host_index in the spec for a single request. :type _host_index: int, optional :return: Returns the result object.
update_configuration ¶
update_configuration(update_configuration_request_content, _request_timeout=None, _request_auth=None, _content_type=None, _headers=None, _host_index=0)
update_configuration
Updates the supplied workflow configuration.
:param update_configuration_request_content: (required) :type update_configuration_request_content: UpdateConfigurationRequestContent :param _request_timeout: timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of (connection, read) timeouts. :type _request_timeout: int, tuple(int, int), optional :param _request_auth: set to override the auth_settings for an a single request; this effectively ignores the authentication in the spec for a single request. :type _request_auth: dict, optional :param _content_type: force content-type for the request. :type _content_type: str, Optional :param _headers: set to override the headers for a single request; this effectively ignores the headers in the spec for a single request. :type _headers: dict, optional :param _host_index: set to override the host_index for a single request; this effectively ignores the host_index in the spec for a single request. :type _host_index: int, optional :return: Returns the result object.
update_evaluator_job_status ¶
update_evaluator_job_status(job_id, update_evaluator_job_status_request_content, _request_timeout=None, _request_auth=None, _content_type=None, _headers=None, _host_index=0)
update_evaluator_job_status
Update the status of an evaluation job (v2)
:param job_id: ID of the job (required) :type job_id: str :param update_evaluator_job_status_request_content: (required) :type update_evaluator_job_status_request_content: UpdateEvaluatorJobStatusRequestContent :param _request_timeout: timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of (connection, read) timeouts. :type _request_timeout: int, tuple(int, int), optional :param _request_auth: set to override the auth_settings for an a single request; this effectively ignores the authentication in the spec for a single request. :type _request_auth: dict, optional :param _content_type: force content-type for the request. :type _content_type: str, Optional :param _headers: set to override the headers for a single request; this effectively ignores the headers in the spec for a single request. :type _headers: dict, optional :param _host_index: set to override the host_index for a single request; this effectively ignores the host_index in the spec for a single request. :type _host_index: int, optional :return: Returns the result object.
update_experiment ¶
update_experiment(name, update_experiment_request_content=None, _request_timeout=None, _request_auth=None, _content_type=None, _headers=None, _host_index=0)
update_experiment
Updates an existing experiment
:param name: Unique identifier for the experiment (required) :type name: str :param update_experiment_request_content: :type update_experiment_request_content: UpdateExperimentRequestContent :param _request_timeout: timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of (connection, read) timeouts. :type _request_timeout: int, tuple(int, int), optional :param _request_auth: set to override the auth_settings for an a single request; this effectively ignores the authentication in the spec for a single request. :type _request_auth: dict, optional :param _content_type: force content-type for the request. :type _content_type: str, Optional :param _headers: set to override the headers for a single request; this effectively ignores the headers in the spec for a single request. :type _headers: dict, optional :param _host_index: set to override the host_index for a single request; this effectively ignores the host_index in the spec for a single request. :type _host_index: int, optional :return: Returns the result object.
update_job_scheduler ¶
update_job_scheduler(name, update_job_scheduler_request_content=None, _request_timeout=None, _request_auth=None, _content_type=None, _headers=None, _host_index=0)
update_job_scheduler
Updates evaluation job scheduler (changes take immediate effect)
:param name: Unique name of the job scheduler (required) :type name: str :param update_job_scheduler_request_content: :type update_job_scheduler_request_content: UpdateJobSchedulerRequestContent :param _request_timeout: timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of (connection, read) timeouts. :type _request_timeout: int, tuple(int, int), optional :param _request_auth: set to override the auth_settings for an a single request; this effectively ignores the authentication in the spec for a single request. :type _request_auth: dict, optional :param _content_type: force content-type for the request. :type _content_type: str, Optional :param _headers: set to override the headers for a single request; this effectively ignores the headers in the spec for a single request. :type _headers: dict, optional :param _host_index: set to override the host_index for a single request; this effectively ignores the host_index in the spec for a single request. :type _host_index: int, optional :return: Returns the result object.