Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support custom host and port, add streaming response support #54

Open
ddimaria opened this issue Sep 12, 2024 · 3 comments
Open

Support custom host and port, add streaming response support #54

ddimaria opened this issue Sep 12, 2024 · 3 comments

Comments

@ddimaria
Copy link

Thanks for creating this crate, it's a time saver! We needed a way to query Snowflake and convert to arrow/parquet, so this is perfect for us. There are some tweaks I would need in order to use this for tests. I'm happy to submit a PR, but wanted to see if you were accepting them?

Currently, the host is hard-coded to go snowflakecomputing.com. There is a way to test snowflake now on LocalStack, but it requires a custom host and port (host: snowflake.localhost.localstack.cloud, port: 4566).

I think this can be added to the builder without affecting the current API in lib.rs (just the signature to request() in connection.rs):

pub struct SnowflakeApi {
    connection: Arc<Connection>,
    session: Session,
    account_identifier: String,
    host: Option<String>, // this is new
}

pub fn with_host(mut self, uri: &str) {
    self.host = Some(uri);
}

Then, in the request() function in connection.rs:

pub async fn request<R: serde::de::DeserializeOwned>(
    &self,
    query_type: QueryType,
    account_identifier: &str,
    extra_get_params: &[(&str, &str)],
    auth: Option<&str>,
    body: impl serde::Serialize,
    host: Option<&str>, // this is new
)

...

let url = host.unwrap_or_else(|| format!(
    "https://{}.snowflakecomputing.com/{}",
    &account_identifier, context.path
));

REFS:
Hard coding:

let url = format!(
"https://{}.snowflakecomputing.com/{}",
&account_identifier, context.path

LocalStack on Snowflake docs: https://snowflake.localstack.cloud/introduction/

@ddimaria
Copy link
Author

I have a working fork here: https://github.com/quadratichq/snowflake-rs, though it still needs some polish. Sadly, hitting LocalStack Snowflake gave an empty body on the session login. After contacting LocalStack, it seems like their product is in public preview, and may be a paid feature in the future.

My goal is to be able to test the snowflake interactions without hitting the production snowflake server. HTTP Mock is relatively new, which offers the ability to record live requests/responses into saved fixtures to use in tests. They require a custom host injection to act as a proxy, so the code in the fork may still be useful. I'll update this issue with my findings.

@ddimaria
Copy link
Author

In that same fork, I implemented a stream response. The original full response is available, but this stream helps avoid taking down a server for overly-large data sets.

@ddimaria ddimaria changed the title Support custom host and port Support custom host and port, add streaming response support Sep 24, 2024
@sgrebnov
Copy link
Contributor

sgrebnov commented Sep 24, 2024

Note, for streaming support there is also the following PR available:
#44

The code has been battle tested for several months as part of the following project
https://github.com/spiceai/spiceai

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants