deleting a tenant (compactor?)

[aep]

hi, i was wondering how to delete a tenant.

i think the way to do it is enabling the overrides-exporter and then in cortex-runtime-config, set

data:
  runtime_config.yaml: |
    overrides:
      tenant123:
        compactor_blocks_retention_period: 1s

which would cause compactor to auto reload that config and then delete the blocks of this tenant.

is this the correct way to approach this?
seems rather odd to have a deleted tenant be in the config which may pile up eventually.