@cdklabs/generative-ai-cdk-constructs
@cdklabs/generative-ai-cdk-constructs / bedrock / WebCrawlerDataSourceProps
Interface to create a new standalone data source object.
readonlyoptionalchunkingStrategy:ChunkingStrategy
The chunking stategy to use for splitting your documents or content. The chunks are then converted to embeddings and written to the vector index allowing for similarity search and retrieval of the content.
ChunkingStrategy.DEFAULT
WebCrawlerDataSourceAssociationProps.chunkingStrategy
readonlyoptionalcrawlingRate:number
The max rate at which pages are crawled, up to 300 per minute per host. Higher values will decrease sync time but increase the load on the host.
300
WebCrawlerDataSourceAssociationProps.crawlingRate
readonlyoptionalcrawlingScope:CrawlingScope
The scope of the crawling.
- CrawlingScope.DEFAULT
WebCrawlerDataSourceAssociationProps.crawlingScope
readonlyoptionalcustomTransformation:CustomTransformation
The custom transformation strategy to use.
- No custom transformation is used.
WebCrawlerDataSourceAssociationProps.customTransformation
readonlyoptionaldataDeletionPolicy:DataDeletionPolicy
The data deletion policy to apply to the data source.
- Sets the data deletion policy to the default of the data source type.
WebCrawlerDataSourceAssociationProps.dataDeletionPolicy
readonlyoptionaldataSourceName:string
The name of the data source.
- A new name will be generated.
WebCrawlerDataSourceAssociationProps.dataSourceName
readonlyoptionaldescription:string
A description of the data source.
- No description is provided.
WebCrawlerDataSourceAssociationProps.description
readonlyoptionalfilters:CrawlingFilters
The filters (regular expression patterns) for the crawling. If there’s a conflict, the exclude pattern takes precedence.
None
WebCrawlerDataSourceAssociationProps.filters
readonlyoptionalkmsKey:IKey
The KMS key to use to encrypt the data source.
- Service owned and managed key.
WebCrawlerDataSourceAssociationProps.kmsKey
readonlyknowledgeBase:IKnowledgeBase
The knowledge base to associate with the data source.
readonlyoptionalparsingStrategy:ParsingStategy
The parsing strategy to use.
- No Parsing Stategy is used.
WebCrawlerDataSourceAssociationProps.parsingStrategy
readonlysourceUrls:string[]
The source urls in the format https://www.sitename.com.
Maximum of 100 URLs.