Skip to content

Commit

Permalink
Refactoring of prepdocs for easier integration with user upload featu…
Browse files Browse the repository at this point in the history
…re (#1407)

* Changes to prepare for upload

* All pass except multilang

* Made mypy happy

* Fix the arguments

* Unit tests for code changes

* 3.9 compatibility

* Remove unused args from prepdocs

* Address feedback

* Ignore typing error after reporting to SDK team
  • Loading branch information
pamelafox authored Mar 13, 2024
1 parent 334047d commit 87f2b9d
Show file tree
Hide file tree
Showing 20 changed files with 529 additions and 387 deletions.
10 changes: 2 additions & 8 deletions scripts/prepdocs.ps1
Original file line number Diff line number Diff line change
Expand Up @@ -30,17 +30,11 @@ if ($env:AZURE_SEARCH_ANALYZER_NAME) {
if ($env:AZURE_VISION_ENDPOINT) {
$visionEndpointArg = "--visionendpoint $env:AZURE_VISION_ENDPOINT"
}
if ($env:AZURE_VISION_KEY) {
$visionKeyArg = "--visionkey $env:AZURE_VISION_KEY"
}

# If vision keys are stored in keyvault provide the keyvault name and secret name
# If any keys are stored in keyvault provide the keyvault name and secret name
if ($env:AZURE_KEY_VAULT_NAME) {
$keyVaultName = "--keyvaultname $env:AZURE_KEY_VAULT_NAME"
}
if ($env:VISION_SECRET_NAME) {
$visionSecretNameArg = "--visionsecretname $env:VISION_SECRET_NAME"
}
if ($env:AZURE_SEARCH_SECRET_NAME) {
$searchSecretNameArg = "--searchsecretname $env:AZURE_SEARCH_SECRET_NAME"
}
Expand Down Expand Up @@ -81,7 +75,7 @@ $argumentList = "./scripts/prepdocs.py $dataArg --verbose " + `
"--openaiservice `"$env:AZURE_OPENAI_SERVICE`" --openaideployment `"$env:AZURE_OPENAI_EMB_DEPLOYMENT`" " + `
"--openaikey `"$env:OPENAI_API_KEY`" --openaiorg `"$env:OPENAI_ORGANIZATION`" " + `
"--documentintelligenceservice $env:AZURE_DOCUMENTINTELLIGENCE_SERVICE " + `
"$searchImagesArg $visionEndpointArg $visionKeyArg $visionSecretNameArg " + `
"$searchImagesArg $visionEndpointArg " + `
"$adlsGen2StorageAccountArg $adlsGen2FilesystemArg $adlsGen2FilesystemPathArg " + `
"$tenantArg $aclArg " + `
"$disableVectorsArg $localPdfParserArg $localHtmlParserArg " + `
Expand Down
Loading

0 comments on commit 87f2b9d

Please sign in to comment.