Idempotent virtual_folders declaration

[nighoa]

Hi Drakkan,

First of all thanks a million for you work on this product SFTPGo and this provider that is in early stage 0.0.8 and do already a lot of good work.

I having a pretty bad issue of idempotency with the "virtual_folders" in the ressource sftpgo_group, by example :

The order is not the same between Terraform and the group in SFTPGo.

The ID seems to be the name in a type list, we should add the name as key of each element in a map (I think).

`
resource "sftpgo_group" "reader_project" {

name = "/Reader/${var.project_name}"
description = "Keycloak /Reader/${var.project_name}"

user_settings = {
max_sessions = 10
filters = {
denied_protocols = ["FTP", "DAV"]
web_client = ["write-disabled", "password-change-disabled"]
start_directory = "/tmp/group"

    # Hide all tree except current folder
    file_patterns = [
      {
          path = "/"
          denied_patterns = ["*"]
          # Set to 1 to hide denied files/directories in directory listing.
          deny_policy = 1
      },
      {
          path = "/tmp"
          denied_patterns = ["*"]
          # Set to 1 to hide denied files/directories in directory listing.
          deny_policy = 1
      },
      {
          path = "/tmp/group"
          denied_patterns = ["*"]
          # Set to 1 to hide denied files/directories in directory listing.
          allow_policy = 1
      }
    ]
  } 

filesystem = {
  provider = 0
}

# Group permissions configuration
permissions = {
  "${local.parent_directory_prod}/prod_${var.project_name}_backup/*" = "list,download",
  "${local.parent_directory_horsprod}/hors_prod_${var.project_name}_backup/*" = "list,download"
}

}

virtual_folders = [
{
name = "prod_bdd_backup_${var.project_name}"
virtual_path = "/tmp/group/prod_bdd_backup_root_folder/${var.project_name}"
quota_size = 0
quota_files = 0
},
{
name = "horsprod_bdd_backup_${var.project_name}"
virtual_path = "/tmp/group/horsprod_bdd_backup_root_folder/${var.project_name}"
quota_size = 0
quota_files = 0
}
]
}`

Terraform output example :

╷ │ Error: Provider produced inconsistent result after apply │ │ When applying changes to module.sftpgo_groups_oidc["project1"].sftpgo_group.reader_project, provider "provider[\"registry.terraform.io/drakkan/sftpgo\"]" produced an unexpected new value: .virtual_folders[1].name: was │ cty.StringVal("horsprod_bdd_backup_project1"), but now cty.StringVal("prod_bdd_backup_project1"). │ │ This is a bug in the provider, which should be reported in the provider's own issue tracker. ╵ ╷ │ Error: Error creating group │ │ with module.sftpgo_groups_oidc["project2"].sftpgo_group.reader_project, │ on ../../../scw-tf-modules/sftpgo-oidc-project-policies/groups_keycloak_project.tf line 9, in resource "sftpgo_group" "reader_project": │ 9: resource "sftpgo_group" "reader_project" { │ │ Could not create group, unexpected error: status: 500, body: {"error":"Error 1213 (40001): Deadlock found when trying to get lock; try restarting transaction","message":""}

I don't know if it's expected, or if I have a bad implementation on my side, but at this time it's impossible to define multiple "Virtual folder" in this ressource, because the idempotency is not respected.

If you already have figured this bug and you have time to address it one day, keep me in touch if I can help you.

Best regards,

Nighoa

[drakkan]

Hello,

thanks for this detailed report.

Unfortunately I'm busy with lots of other things and don't have time to analyze it further in the immediate future.

The same support policy as SFTPGo applies here.

Feel free to investigate the issue further and send a PR (please be patient if I'm slow to respond).
Thank you!

[nighoa]

Hi @drakkan !

The thing is that I can mitigate the issue with this trick :

TF_CLI_ARGS_apply="-parallelism=1" terraform apply

The API seems that it can't handle high parallelism (by default 10 calls in Terraform) when you declare a loop on a list of project in the ressource sftpgo_group.

At this time it's mitigated but not "perfect", you have to apply "twice" terraform to have a working terraform apply.

No problem for your "support policy", it's not a "request" more a experience return from a user that love your product and want to help you improve it ;)

Have fun !

Best regards

[drakkan]

I have not tried to reproduce the problem myself, but if it is related to virtual folder sorting, try the provider with the development version of SFTPGo, the virtual folders are now sorted by name, so as long as you list the virtual folders sorted by name in the tf file, the order should be preserved

[nighoa]

Hi Drakkan,

I do a quick test, and the result is the same with the latest sftpgo:edge Docker image at this time.

│ Error: Provider produced inconsistent result after apply
│
│ When applying changes to module.sftpgo_groups_oidc["testha"].sftpgo_group.appsvc_reader_project, provider "provider[\"registry.terraform.io/drakkan/sftpgo\"]" produced an unexpected new value: .virtual_folders[0].name: was
│ cty.StringVal("scw_prod_bdd_backup_testha"), but now cty.StringVal("scw_horsprod_bdd_backup_testha").
│
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent result after apply
│
│ When applying changes to module.sftpgo_groups_oidc["testha"].sftpgo_group.appsvc_reader_project, provider "provider[\"registry.terraform.io/drakkan/sftpgo\"]" produced an unexpected new value: .virtual_folders[0].virtual_path: was
│ cty.StringVal("/tmp/group/scw_prod_bdd_backup_root_folder/testha"), but now cty.StringVal("/tmp/group/scw_horsprod_bdd_backup_root_folder/testha").
│
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent result after apply
│
│ When applying changes to module.sftpgo_groups_oidc["testha"].sftpgo_group.appsvc_reader_project, provider "provider[\"registry.terraform.io/drakkan/sftpgo\"]" produced an unexpected new value: .virtual_folders[1].name: was
│ cty.StringVal("scw_horsprod_bdd_backup_testha"), but now cty.StringVal("scw_prod_bdd_backup_testha").
│
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent result after apply
│
│ When applying changes to module.sftpgo_groups_oidc["testha"].sftpgo_group.appsvc_reader_project, provider "provider[\"registry.terraform.io/drakkan/sftpgo\"]" produced an unexpected new value: .virtual_folders[1].virtual_path: was
│ cty.StringVal("/tmp/group/scw_horsprod_bdd_backup_root_folder/testha"), but now cty.StringVal("/tmp/group/scw_prod_bdd_backup_root_folder/testha").
│
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.

TF example sftpgo_group :

The name is composed by a constant prefix and a variable project name "arzea", "bazezae", "abaze", "soian", etc.
The virtual_path is more or less the same logical with a constant "prefix" /tmp/group/scw_horsprod_bdd_backup_root_folder/ or /tmp/group/scw_prod_bdd_backup_root_folder/ then a variable for the project name.

resource "sftpgo_group" "appsvc_reader_project" {

  name        = "/APPSVC-Reader/${var.project_name}"
  description = "Keycloak /APPSVC-Reader/${var.project_name}"
  user_settings = {
    max_sessions = 10
    filters = {
      denied_protocols = ["FTP", "DAV"]
      web_client       = ["write-disabled", "password-change-disabled"]
    }
    filesystem = {
      provider = 0
    }

    # Group permissions configuration
    permissions = {
      "/*" = "list,download",
      "/"  = "list,download"
    }
  }

  virtual_folders = [
    {
      name         = "scw_prod_bdd_backup_${var.project_name}"
      virtual_path = "${local.parent_directory_prod}/${var.project_name}"
      quota_size   = 0
      quota_files  = 0
    },
    {
      name         = "scw_horsprod_bdd_backup_${var.project_name}"
      virtual_path = "${local.parent_directory_horsprod}/${var.project_name}"
      quota_size   = 0
      quota_files  = 0
    }
  ]
}

[drakkan]

please try to order your virtual folders by name ascending. In the edge version the order is fixed (.... ORDER BY f.name)

[nasielgarcia]

Hi @drakkan @nighoa
I'm having a similar issue. I am trying to add a new virtual folder to a user who already has two virtual folders set up, and I keep getting the same error.

│ Error: Provider produced inconsistent result after apply
│ 
│ When applying changes to
│ module.sftpgo_configuration.sftpgo_user.user_room_rate["sandboxmscontent"],
│ provider
│ "module.sftpgo_configuration.provider[\"registry.terraform.io/drakkan/sftpgo\"]"
│ produced an unexpected new value: .virtual_folders[2].name: was
│ cty.StringVal("sandbox-property_chain_images_processed"), but now
│ cty.StringVal("sandbox-room_rate_processed").
│ 
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.

I am currently using version 0.0.11 of the provider. I first got the error with version 0.0.8, so I decided to upgrade, but the error still remains.

Any additional suggestions to fix the problem?
Thanks!

[nasielgarcia]

Hi @drakkan @nighoa I'm having a similar issue. I am trying to add a new virtual folder to a user who already has two virtual folders set up, and I keep getting the same error.

│ Error: Provider produced inconsistent result after apply
│ 
│ When applying changes to
│ module.sftpgo_configuration.sftpgo_user.user_room_rate["sandboxmscontent"],
│ provider
│ "module.sftpgo_configuration.provider[\"registry.terraform.io/drakkan/sftpgo\"]"
│ produced an unexpected new value: .virtual_folders[2].name: was
│ cty.StringVal("sandbox-property_chain_images_processed"), but now
│ cty.StringVal("sandbox-room_rate_processed").
│ 
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.

I am currently using version 0.0.11 of the provider. I first got the error with version 0.0.8, so I decided to upgrade, but the error still remains.

Any additional suggestions to fix the problem? Thanks!

Hi again,

I was able to solve the problem by using the workaround of sorting the virtual folders alphabetically, as you suggested @drakkan

Thanks a lot