How do I use cue concurrently?

[uris77]

I'm trying to create a custom terraform provider that uses cue. Terraform seems to do a lot of things concurrently and cue keeps panicking when that happens. Is there a way to use cue in a concurrent environment?

[verdverm]

CUE is not currently safe for concurrency

• https://github.com/cue-lang/cue/issues?q=is%3Aissue+is%3Aopen+concurrent

In particular #652 #460

[verdverm]

do you mean your previous example of calling newCmd from Go code?

I'm referring to something more like wrapping cue in a bash script and using the TF shell provider to call it

[uris77]

Yes

[uris77]

Oh, I see what you mean. Using something like https://github.com/toddnni/terraform-provider-shell that invokes cue cli.... Not ideal because that would mean that users should have cue installed. That might be show stopper.

[verdverm]

you could always install cue in the local dir as part of that step

[verdverm]

another option would be to read/write a file based lock / mutex

[myitcv]

@uris77 can you explain a bit more what you mean by

custom terraform provider that uses cue

I'm assuming you're using the Go API?

If so, #652 and #460 are both relevant. Using a simple sync.Mutex should be sufficient to guard what you are doing.

[myitcv]

But to stress, use of such a mutex is a temporary measure until a proper model for concurrent use of the Go API is established.

[uris77]

I'm trying to create a terraform provider that provisions Grafana dashboards for internal teams. We have some pre-baked dashboards defined with cue and our intention is for teams to easily deploy the ones they want:

data "internal_dashboard" "pubsub" {
	name = "PubSub"
}

data "internal_dashboard" "cloudsql" {
	name = "CloudSql"
}

resource "grafana_dashboard" "cloudsql" {
  config_json = data.internal_dashboard.cloudsql.rendered
  folder      = grafana_folder.collection.id
}

resource "grafana_dashboard" "pubsub" {
  config_json = data.internal_dashboard.pubsub.rendered
  folder      = grafana_folder.collection.id
}

This works fine when we only provision one dashboard, but it fails when we try to provision more than one. This is probably because Terraform uses goroutines to provision multiple data sources.

The terraform provider for this is pretty simple, the main code is:

func exportCueFile(ct *cue.Context, name string) (string, error) {

	cwd, err := os.Getwd()
	if err != nil {
		return "", err
	}
	cueFilesDir := filepath.Join(cwd, "dashboards")
	dirCfg := &load.Config{
		Dir:     cueFilesDir,
		Package: "grafana",
		Tags:    []string{fmt.Sprintf("title=%s", name)},
	}

	bps := load.Instances([]string{
		".",
		fmt.Sprintf("--expression=%s", strings.ToLower(name))},
		dirCfg)
	v := ct.BuildInstance(bps[0])
	byts, err := v.MarshalJSON()
	if err != nil {
		return "", err
	}
	return string(byts), nil
}

func dataSourceCueFileRead(ctx context.Context, d *schema.ResourceData, m interface{}) diag.Diagnostics {
	var diags diag.Diagnostics
	config := m.(*providerConfig)
	name := d.Get("name").(string)
	
	content, err := exportCueFile(config.cueContext, name)
	if err != nil {
		diags = append(diags, diag.Diagnostic{
			Severity: diag.Error,
			Summary:  fmt.Sprintf("Exporting dashboard to json file failed. name: %s", name),
			Detail:   fmt.Sprintln(err),
		})
		return diags
	}

	if err := d.Set("rendered", content); err != nil {
		diags = append(diags, diag.Diagnostic{
			Severity: diag.Error,
			Summary:  "Failure setting dashboard rendered content",
			Detail:   fmt.Sprintln(err),
		})
		return diags
	}

	// I'm not sure what to do here
	d.SetId(strconv.Itoa(hashcode(content)))
	return nil
}

func hashcode(s string) int {
	v := int(crc32.ChecksumIEEE([]byte(s)))
	if v >= 0 {
		return v
	}
	if -v >= 0 {
		return -v
	}
	return 0
}

[myitcv]

This is probably because Terraform uses goroutines to provision multiple data sources.

I definitely don't know the internals of Terraform, but highly likely. If you guard each the entry points from Terraform with a mutex then that should cause the panics to disappear. On the basis the CUE operations should be fast (modulo known performance issues) this hopefully won't create too much overhead.