Skip to content

Feat: Added tiered_storage add/list/remove #47

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 7 commits into
base: mdai_cli
Choose a base branch
from

Conversation

shalane-proctor
Copy link

Unable to get the add form to add to configMap. I can see tiered storage on kubectl, but nothing in it.

Comment on lines +98 to +100
"clickhouse",
"druid",
"pinot"},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are not formats and shouldn't be included as options that you would be choosing from in the context of choosing s3 or gcs storage.

This applies to the cold storage option as well.

I believe parquet fits here while store should include options like my-clickhouse.

},
Formats: []string{"7zip",
"arc",
"br",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"br",
"brotli",

Comment on lines +130 to +131
Formats: []string{"7zip",
"arc",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Formats: []string{"7zip",
"arc",
Formats: []string{"7zip",

"br",
"rar",
"tar",
"GZIP",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"GZIP",
"gzip",

"rar",
"tar",
"GZIP",
"zpaq",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had to look this one up. Should it really be included?

"tar",
"GZIP",
"zpaq",
"zst"},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"zst"},
"zstd"},

boldStyle.Render("Performance") + ": Very slow access (hours or days to retrieve).",
boldStyle.Render("Cost") + ": Extremely low cost per GB, ideal for long-term retention.",
},
Formats: []string{"7zip",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think I ever see 7zip mentioned these days.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, doing a quick google search for best text based compression and maybe i'm wrong here, it gets mentioned.

If we want anymore, assuming the ones I think should be removed are, there is also snappy, lzo and lz4.

"arc",
"br",
"rar",
"tar",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"tar",

tar isn't a compression format.

Formats: []string{"7zip",
"arc",
"br",
"rar",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"rar",

@tsloughter
Copy link
Contributor

Really like this interface but had some nitpicks.

A general question is if the like cold storage is meant to assume the use of the same format as previous selections of hot or something. Because you should need to choose both json/csv/parquet and compression protocol (note, list of compression protocols supported by parquet https://parquet.apache.org/docs/file-format/data-pages/compression/). Really any of the steps hot or cold should give the option to select a compression type. Maybe in hot they'd choose a faster but less efficient compression, or none at all, but makes sense to not be just a format on cold.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants