Skip to content

Commit

Permalink
Support parameterization of arrays in CICD script
Browse files Browse the repository at this point in the history
  • Loading branch information
soma-ms committed Dec 29, 2022
1 parent d5d9902 commit 0c60517
Showing 1 changed file with 10 additions and 2 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -228,7 +228,7 @@ function Compare-TriggerPayload {
return $True
}
catch {
Write-Host "##[warning] Unable to compare payload for '$($triggerDeployed.Name)' trigger, this is not a failure. You can post the issue to https://github.com/Azure/Azure-DataFactory/issues to check if this can be addressed."
Write-Host "##[warning] Unable to compare payload for '$($triggerDeployed.Name)' trigger, this is not a failure. You can post the issue to https://github.com/Azure/Azure-DataFactory/issues to check if this is user error or limitation."
Write-Host "##[warning] $_ from Line: $($_.InvocationInfo.ScriptLineNumber)"
return $True;
}
Expand Down Expand Up @@ -467,7 +467,15 @@ function Update-TriggerTemplate {
foreach ($parameterMatch in $parameterMatches) {
$parameterName = $parameterMatch.Value.Substring(13, $parameterMatch.Value.Length - 16)
if ($null -ne $templateParameters.$($parameterName)) {
$templateJson = $templateJson -replace [System.Text.RegularExpressions.Regex]::Escape($parameterMatch.Value), $templateParameters.$($parameterName).value
$parameterType = $templateParameters.$($parameterName).value ? $templateParameters.$($parameterName).value.GetType().Name : $null
if ($parameterType -eq 'Object[]') {
$parameterValue = ConvertTo-Json $templateParameters.$($parameterName).value
$templateJson = $templateJson -replace [System.Text.RegularExpressions.Regex]::Escape("`"$($parameterMatch.Value)`""), $parameterValue
} elseif ($parameterType -eq 'Boolean' -or $parameterType -eq 'Int64') {
$templateJson = $templateJson -replace [System.Text.RegularExpressions.Regex]::Escape("`"$($parameterMatch.Value)`""), $templateParameters.$($parameterName).value
} else {
$templateJson = $templateJson -replace [System.Text.RegularExpressions.Regex]::Escape($parameterMatch.Value), $templateParameters.$($parameterName).value
}
}
}

Expand Down

15 comments on commit 0c60517

@dmenese
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Line 470 doesn't seem to be working -and I see that I was changed in the last commit. Seems to have some issue with the operators or something:

At D:\a\r1\a_main_branch\zzDeploymentScripts\PrePostDeploymentScript.ps1:470 char:74

  • ... parameterType = $templateParameters.$($parameterName).value ? $templa ...
  •                                                             ~
    

Unexpected token '?' in expression or statement.
PowerShell exited with code '1'.

@dmenese
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just confirmed that by replacing lines 470 to 478 with the previous code it works again.

@soma-ms
Copy link
Contributor Author

@soma-ms soma-ms commented on 0c60517 Jan 11, 2023 via email

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dmenese
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you are right. I'm running it using an Agent which I didn't install named "Hosted Windows 2019 with VS2019". Maybe that is hosting an old Powershell version. Thanks for the advice, I'll ask the DevOps teams to check the Powershell version in it.

Thanks!
Diego.

@feverfunk
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're using azure hosted agents with windows-latest. Attempted to use power shell core to run the PrePostDeploymentScript.ps1. Verified that we're on psversion 7.2.8, but now Get-AzDataFactoryV2Trigger isn't found. What are the steps for this to work in an azure hosted agent? thanks

@shunkino
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@feverfunk What task did you try? It worked fine on my pipeline by setting "Use PowerShell Core" optioin of AzurePowerShell@5 on Microsoft-hosted agent with window-latest image.

@ROCrysty
Copy link

@ROCrysty ROCrysty commented on 0c60517 Jan 20, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is how i've solved the issue:
Change the Agent Specification to ubuntu-latest
This fix may be regional. For West Eu it worked.
image

@feverfunk
Copy link

@feverfunk feverfunk commented on 0c60517 Jan 20, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@shukino - Many thanks. We originally backed out the offending changes and it worked. Obviously we didn't look at the doc for AzurePowerShell@5. Thanks so much for your help!

to answer your question, we assumed (incorrectly) that AzurePowerShell@5 was limited to a version lower than version 7. So we tried using a powershell task with pwsh: true. Our team member who setup the original pipelines is no longer with us, so we had a learning curve to overcome.

thanks again!

@BrilliantLC-MSFT
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

^ my team is having the same issue with bool

@vojtakopal
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This just cost me half a day debugging. 👎 How is it possible to have this reported for two weeks without any fix?

@soma-ms
Copy link
Contributor Author

@soma-ms soma-ms commented on 0c60517 Jan 31, 2023 via email

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Mauchy81
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hello @soma-ms,
When I was trying to understand the script, I noticed that the triggers from the Prod environment "Prd - Trigger Main - every 2 hours 0540 - 2040" are also stopped during pre-deployment handling, even if it is not currently running.
2023-02-02T07:25:51.6042857Z Stopping 1 triggers
2023-02-02T07:25:51.6057083Z
2023-02-02T07:25:51.6063827Z Stopping trigger Prd - Trigger Main - alle 2 Stunden 0540 - 2040
2023-02-02T07:25:53.3660300Z True

During post-deployment handling, however, the DEV trigger "Dev - Trigger Main - twice a week Mon and Thu 0900" from the template is started but not the previously stopped trigger from the Prod environment.

2023-02-02T07:26:53.7790200Z Starting 1 triggers
2023-02-02T07:26:53.7816906Z Starting trigger Dev - Trigger Main - twice a week Mo and Thu 0900
2023-02-02T07:26:55.6057564Z True

Is this how the PowerShell script is supposed to work or is this an error? From the description I had understood that after the deplovemt the trigger from the Prod environment remains active. This simplifies the use of different triggers from different environments.

@zdavid820
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we are using the latest version while faield for post depolyment. Any idea why the depth is configured / hardcoded as 10?

2023-02-06T02:46:01.9494897Z ##[error] Conversion from JSON failed with error: The reader's MaxDepth of 10 has been exceeded. Path 'properties.pipelines[0].parameters.cw_items[0].copyActivity.translator.mappings[0]', line 28, position 21. from Line: 680
2023-02-06T02:46:02.0744316Z ##[error]Conversion from JSON failed with error: The reader's MaxDepth of 10 has been exceeded. Path 'properties.pipelines[0].parameters.cw_items[0].copyActivity.translator.mappings[0]', line 28, position 21.
2023-02-06T02:46:02.1936014Z ##[error]PowerShell exited with code '1'.

@soma-ms
Copy link
Contributor Author

@soma-ms soma-ms commented on 0c60517 Feb 9, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hello @soma-ms, When I was trying to understand the script, I noticed that the triggers from the Prod environment "Prd - Trigger Main - every 2 hours 0540 - 2040" are also stopped during pre-deployment handling, even if it is not currently running. 2023-02-02T07:25:51.6042857Z Stopping 1 triggers 2023-02-02T07:25:51.6057083Z 2023-02-02T07:25:51.6063827Z Stopping trigger Prd - Trigger Main - alle 2 Stunden 0540 - 2040 2023-02-02T07:25:53.3660300Z True

During post-deployment handling, however, the DEV trigger "Dev - Trigger Main - twice a week Mon and Thu 0900" from the template is started but not the previously stopped trigger from the Prod environment.

2023-02-02T07:26:53.7790200Z Starting 1 triggers 2023-02-02T07:26:53.7816906Z Starting trigger Dev - Trigger Main - twice a week Mo and Thu 0900 2023-02-02T07:26:55.6057564Z True

Is this how the PowerShell script is supposed to work or is this an error? From the description I had understood that after the deplovemt the trigger from the Prod environment remains active. This simplifies the use of different triggers from different environments.

The script won't stop the triggers that are not active in live mode. It only stops the active triggers and if scripts find trigger payload change. And in rare cases script might error out while comparing trigger payload and treats that as trigger change.

@soma-ms
Copy link
Contributor Author

@soma-ms soma-ms commented on 0c60517 Feb 9, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we are using the latest version while faield for post depolyment. Any idea why the depth is configured / hardcoded as 10?

2023-02-06T02:46:01.9494897Z ##[error] Conversion from JSON failed with error: The reader's MaxDepth of 10 has been exceeded. Path 'properties.pipelines[0].parameters.cw_items[0].copyActivity.translator.mappings[0]', line 28, position 21. from Line: 680 2023-02-06T02:46:02.0744316Z ##[error]Conversion from JSON failed with error: The reader's MaxDepth of 10 has been exceeded. Path 'properties.pipelines[0].parameters.cw_items[0].copyActivity.translator.mappings[0]', line 28, position 21. 2023-02-06T02:46:02.1936014Z ##[error]PowerShell exited with code '1'.

Overlooked that json can go beyond the depth of 10, update that to be configurable in latest script with default depth as 100.

Please sign in to comment.