tests: implement ccm integration #1203

dkropachev · 2025-02-06T04:52:44Z

CCM is cli tool to ease scylla/cassandra/dse cluster management.
Using it in integration suit will help to expand test coverage.

Pre-review checklist

I have split my patch into logically separate commits.
All commit messages clearly explain what they change and why.
I added relevant tests for new features and bug fixes.
All commits compile, pass static checks and pass test.
PR description sums up the changes and reasons why they should be introduced.
I have provided docstrings for the public items that I want to introduce.
I have adjusted the documentation in ./docs/source/.
I added appropriate Fixes: annotations to PR description.

github-actions · 2025-02-06T05:02:05Z

cargo semver-checks found no API-breaking changes in this PR! 🎉🥳
Checked commit: f858370

wprzytula

This is the first round of the review.
There are for sure more issues, but I'm unable to grasp them until I gain the high-level of the approach, which I failed to achieve so far due to no comments nor commit messages nor the cover letter.

scylla/Cargo.toml

scylla/tests/ccm-integration/logged_cli.rs

scylla/tests/ccm-integration/test_example.rs

scylla/tests/ccm-integration/logged_cli.rs

wprzytula · 2025-02-07T12:01:59Z

scylla/tests/ccm-integration/logged_cli.rs

+        stream: T,
+        writer: Arc<Mutex<File>>,
+        prefix: String,
+        mut buffer: Option<&mut Vec<u8>>,


Suggested change

mut buffer: Option<&mut Vec<u8>>,

buffer: Option<&mut Vec<u8>>,

Does not work:

error[E0596]: cannot borrow `buffer.0` as mutable, as `buffer` is not declared as mutable --> scylla/tests/ccm-integration/logged_cli.rs:249:25 | 249 | if let Some(ref mut buffer) = buffer { | ^^^^^^^^^^^^^^ cannot borrow as mutable | help: consider changing this to be mutable | 237 | mut buffer: Option<&mut Vec<u8>>, | +++

If i change ref mut to ref it fails with:

error[E0596]: cannot borrow `**buffer` as mutable, as it is behind a `&` reference --> scylla/tests/ccm-integration/logged_cli.rs:250:17 | 250 | buffer.extend_from_slice(line.as_bytes()); | ^^^^^^ `buffer` is a `&` reference, so the data it refers to cannot be borrowed as mutable | help: consider changing this to be a mutable reference | 249 | if let Some(ref mut buffer) = buffer { | +++

Try experimenting with matching on buffer.as_ref() or buffer.as_mut() instead of matching on plain buffer. Might solve the issue.

as_mut requires buffer to be mut, as_ref goes wrong way, but match buffer worked well, take a look

scylla/tests/ccm-integration/logged_cli.rs

scylla/tests/ccm-integration/cluster.rs

Cleanup unused dependancies

Lorak-mmk

By no means a thorough review, just posting a few things that I noticed during a brief read.

Few other things:

The utils for tests should be put in some folder inside ccm-integration, instead of being on the same level as the test files.
Before merging, we will need to update Makefile and CONTRIBUTING.md.

Lorak-mmk · 2025-02-09T22:26:08Z

scylla/tests/ccm-integration/cluster.rs

+pub(crate) enum NodeStartOption {
+    // Don't wait node to start
+    NoWait,
+    // Wait till other nodes recognize started node
+    WaitOtherNotice,
+    // Wait till started node report that client port is opened and operational
+    WaitForBinaryProto,
+}
+


My first thought was that it is nice that you translated CCM options to enum, because Rust Driver contributor shouldn't have to know anything about CCM, but now I see that it is not that obvious. Each variant corresponds to a command line option, and Node::start accepts Option<&[NodeStartOption]>.
So this requires the user of those utilities to understand how CCM handles repeated / conflicting command line arguments. What happends if I pass Some(&[NodeStartOption::NoWait, NodeStartOption::WaitOtherNotice])? It is not obvious to me after reading ccm's help and briefly even source code.
I think this should be a struct, with fields wait_other_notice: bool and wait_for_binary_proto: bool. This way the user specifices what they want to wait for, and only the ccm wrapper needs to know how to translate this to command line options.

Lorak-mmk · 2025-02-09T22:26:52Z

scylla/tests/ccm-integration/cluster.rs

+pub(crate) enum NodeStopOption {
+    // Don't wait node to properly stop
+    NoWait,
+    // Force-terminate node with `kill -9`
+    Kill,
+}


As above, this would be better expressed as a struct with boolean fields.

Lorak-mmk · 2025-02-09T22:28:40Z

scylla/tests/ccm-integration/test_example.rs

+lazy_static! {
+    static ref CLUSTER_VERSION: String =
+        std::env::var("SCYLLA_TEST_CLUSTER").unwrap_or("release:6.2.2".to_string());
+    static ref TEST_KEEP_CLUSTER_ON_FAILURE: bool = !std::env::var("TEST_KEEP_CLUSTER_ON_FAILURE")
+        .unwrap_or("".to_string())
+        .parse::<bool>()
+        .unwrap_or(false);
+}
+


The files with tests should not interact with environment at all. Cluster version selection (and thus reading the env var) should be done by the ccm utils.

Preferably, all interaction with env variables should be done in one place in the code.

Lorak-mmk · 2025-02-09T22:28:59Z

scylla/tests/ccm-integration/test_example.rs

+async fn run_ccm_test<C, T, CFut, TFut>(cb: C, test_body: T)
+where
+    C: FnOnce() -> CFut,
+    CFut: std::future::Future<Output = Arc<RwLock<Cluster>>>,
+    T: FnOnce(Arc<RwLock<Cluster>>) -> TFut,
+    TFut: std::future::Future<Output = Result<(), Error>>,
+{
+    let cluster_arc = cb().await;
+    {
+        let res = test_body(cluster_arc.clone()).await;
+        if res.is_err() && *TEST_KEEP_CLUSTER_ON_FAILURE {
+            println!("Test failed, keep cluster alive, TEST_KEEP_CLUSTER_ON_FAILURE=true");
+            cluster_arc.write().await.set_keep_on_drop(true);
+        }
+    }
+}
+


This is also something that should not be present in the test code, but rather in a centralized location.

Lorak-mmk · 2025-02-09T22:30:34Z

scylla/tests/ccm-integration/test_example.rs

+async fn get_session(cluster: &Cluster) -> Session {
+    let endpoints = cluster.nodes.get_contact_endpoints().await;
+    SessionBuilder::new()
+        .known_nodes(endpoints)
+        .build()
+        .await
+        .unwrap()
+}


Two issues with this:

This should not be in test file, but rather a method on Cluster

This should return a SessionBuilder so that a test can customize the session further. See

scylla-rust-driver/scylla/tests/integration/utils.rs

Lines 137 to 167 in bceac6a

pub(crate) fn create_new_session_builder() -> GenericSessionBuilder<impl SessionBuilderKind> {

let session_builder = {

#[cfg(not(scylla_cloud_tests))]

{

use scylla::client::session_builder::SessionBuilder;

let uri = std::env::var("SCYLLA_URI").unwrap_or_else(|_| "127.0.0.1:9042".to_string());

SessionBuilder::new().known_node(uri)

}

#[cfg(scylla_cloud_tests)]

{

use scylla::client::session_builder::{CloudMode, CloudSessionBuilder};

use std::path::Path;

std::env::var("CLOUD_CONFIG_PATH")

.map(|config_path| CloudSessionBuilder::new(Path::new(&config_path)))

.expect("Failed to initialize CloudSessionBuilder")

.expect("CLOUD_CONFIG_PATH environment variable is missing")

}

};

// The reason why we enable so long waiting for TracingInfo is... Cassandra. (Yes, again.)

// In Cassandra Java Driver, the wait time for tracing info is 10 seconds, so here we do the same.

// However, as Scylla usually gets TracingInfo ready really fast (our default interval is hence 3ms),

// we stick to a not-so-much-terribly-long interval here.

session_builder

.tracing_info_fetch_attempts(NonZeroU32::new(200).unwrap())

.tracing_info_fetch_interval(Duration::from_millis(50))

}

Lorak-mmk · 2025-02-09T22:39:26Z

scylla/tests/ccm-integration/cluster.rs

+            version: "".to_string(),
+            ip_prefix: NetPrefix::empty(),
+            root_dir: "/tmp/ccm".to_string(),
+            nodes: Vec::new(),
+            smp: DEFAULT_SMP,


Putting it in /tmp/ccm will prevent running multiple instances of tests simultaneously.
Either put in in some folder in the repo (probably the better option), or (my less preferred option) use some lib like https://docs.rs/mktemp/latest/mktemp/ to make a unique folder in tmp.

Lorak-mmk · 2025-02-09T22:49:25Z

scylla/tests/ccm-integration/logged_cmd.rs

+// A simple abstraction to run commands, log command, exit code its stderr, stdout to a file
+//  and optionally to own stderr/stdout
+// It should allow to run multiple commands in parallel
+pub(crate) struct LoggedCmd {
+    file: Arc<Mutex<File>>,
+    run_id: AtomicI32,
+}


The comment explains what does this struct do, but not why.
Why would we need to log ccm output to a file? In other words why is it not enough to log it to normal output, like other stuff in tests do? Note that if you write something to stdoud/stderr it will be properly caught, and printed in the test fails.
To demonstrate, I wrote a tiny integration test:

#[tokio::test] async fn test_output() { setup_tracing(); debug!("Some debug output"); warn!("Some warn output"); println!("Some stdout output"); eprintln!("Some stderr output"); panic!("To see if the test fails"); }

If I run it with cargo test test_output I get:

And if I enable tracing with RUST_LOG=trace cargo test test_output I get the tracing output too:

Is it for some reason not enough to just normally log ccm commands?

When test is complex it is very handy to be able to see a flow of ccm commands it executed.
Even if we go with dumping everything into stderr, stdout, we will still need this wrapper.

When the complex test fails there are many things that are handy to see, I don't see a reason to single-out CCM to be dumped to the file.

The wrapper can still exists to dump CCM commands to the output.
I'd suggest using trace/debug for the actual output, and info for the command, env, and return code.

Lorak-mmk · 2025-02-09T22:50:01Z

scylla/tests/ccm-integration/cluster.rs

+    async fn sniff_ipv4_prefix() -> Result<NetPrefix, Error> {
+        let mut used_ips: HashSet<IpAddr> = HashSet::new();
+        let file = File::open("/proc/net/tcp").await?;
+        let mut lines = BufReader::new(file).lines();
+        while let Some(line) = lines.next_line().await? {
+            let parts: Vec<&str> = line.split_whitespace().collect();
+            if let Some(ip_hex) = parts.get(1) {
+                let ip_port: Vec<&str> = ip_hex.split(':').collect();
+                if let Some(ip_hex) = ip_port.get(0) {
+                    if let Some(ip) = u32::from_str_radix(ip_hex, 16).ok() {
+                        used_ips.insert(
+                            Ipv4Addr::new(ip as u8, (ip >> 8) as u8, (ip >> 16) as u8, 0).into(),
+                        );
+                    }
+                }
+            }
+        }
+
+        for a in 0..=255 {
+            for b in 0..=255 {
+                if a == 0 && b == 0 {
+                    continue;
+                }
+                let ip_prefix: IpAddr = Ipv4Addr::new(127, a, b, 0).into();
+                if !used_ips.contains(&ip_prefix) {
+                    return Ok(ip_prefix.into());
+                }
+            }
+        }
+        Err(Error::msg("All ip ranges are busy"))
+    }


Its absurd that we have to do this, it should be ccm's job to find a free ips for the nodes.

well, it does not do that and we have to dial with it.

Lorak-mmk · 2025-02-09T22:50:59Z

scylla/tests/ccm-integration/cluster.rs

+    // CCM does not allow to have one active cluster within one config directory
+    // To have more than two active CCM cluster at the same time we isolate each cluster into separate
+    // config director, each config directory is created in `root_dir`.
+    // Example: root_dir = `/tmp/ccm`, config directory for cluster `test_cluster_1` is going be `/tmp/ccm/test_cluster_1`
+    //  and cluster directory (that is created and managed  by CCM) for this cluster is going to be `/tmp/ccm/test_cluster_1/test_cluster_1`
+    pub(crate) root_dir: String,


Again, ccm should be changed so that the commands can specify which cluster to operate on.

true, but for now we have to dial with current behavior.

Lorak-mmk · 2025-02-09T22:52:18Z

scylla/tests/ccm-integration/cluster.rs

+    fn get_ccm_env(&self) -> HashMap<String, String> {
+        let mut env: HashMap<String, String> = HashMap::new();
+        env.insert(
+            "SCYLLA_EXT_OPTS".to_string(),
+            format!("--smp={} --memory={}M", self.opts.smp, self.opts.memory),
+        );
+        env
+    }


I am unable to imagine why would someone think it is a good idea to pass some options throught env instead normal arguments.....

muzarski

I think that we should firstly focus on having the CI pass. Currently, the code does not even compile. Once we fix this, it will be much easier to test the code locally (which will be helpful in understanding the logic). I checked out on this PR locally, to play around with new API/run some tests but I was not able to do so, because it does not compile.

I suggest fixing the issues listed by: cargo clippy --verbose --all-features --all-targets -- -D warnings. Or if you want to not only do the checks, but run the tests as well, I suggest using make ci.

scylla/tests/ccm-integration/logged_cli.rs

scylla/tests/ccm-integration/test_example.rs

Implement simple abstraction that allows to run commands logging command, return status, stderr and stdout to a file and optionally to stderr/stdout

In order to programatically manipulate by scylla.yaml and cassandra.yaml We need to have some struct that provides API for that. It also needs to represent it in the way that ccm updateconf understand

CCM is cli tool to ease scylla/cassandra/dse cluster management. Using it in integration suit will help to expand test coverage.

dkropachev · 2025-02-10T01:18:39Z

I think that we should firstly focus on having the CI pass. Currently, the code does not even compile. Once we fix this, it will be much easier to test the code locally (which will be helpful in understanding the logic). I checked out on this PR locally, to play around with new API/run some tests but I was not able to do so, because it does not compile.

I suggest fixing the issues listed by: cargo clippy --verbose --all-features --all-targets -- -D warnings. Or if you want to not only do the checks, but run the tests as well, I suggest using make ci.

Thanks, done

wprzytula · 2025-02-10T09:20:22Z

scylla/tests/ccm-integration/logged_cli.rs

+    fn get_run_options(opts: RunOptions) -> (HashMap<String, String>, bool) {
+        match opts {
+            RunOptions::None => (HashMap::new(), false),
+            RunOptions::Env(new_env) => (new_env, false),
+            RunOptions::AllowFailure => (HashMap::new(), true),
+            RunOptions::AllowFailureWithEnv(new_env) => (new_env, true),
+        }
+    }


⛏️ I case all four combinations are valid, I'd just represent the type as:

pub(crate) struct RunOptions { env: HashMap<String, String>, allow_failure: bool, }

wprzytula · 2025-02-10T09:24:04Z

scylla/tests/ccm-integration/logged_cli.rs

+    async fn process_child_result(
+        status: io::Result<ExitStatus>,
+        allow_failure: bool,
+        run_id: i32,
+        stderr: Vec<u8>,
+        command_with_args: String,
+        writer: Arc<Mutex<File>>,
+    ) -> Result<ExitStatus, Error> {


🔧 As a rule of thumb, only pass ownership to a function if it requires it; default should to to pass heap-allocated objects by reference (mutable reference if needed).

async fn process_child_result( status: io::Result<ExitStatus>, allow_failure: bool, run_id: i32, stderr: &[u8], command_with_args: &str, writer: &Mutex<File>, ) -> Result<ExitStatus, Error> {

wprzytula · 2025-02-10T09:26:24Z

scylla/tests/ccm-integration/logged_cli.rs

+                    let tmp = stderr.deref();
+                    return Err(Error::msg(format!(
+                        "Command `{}` failed: {}, stderr: \n{}",
+                        command_with_args,
+                        status,
+                        std::str::from_utf8(tmp)?
+                    )));


This tmp should be needed; deref is often called automagically. Try putting &stderr into from_utf8().

wprzytula · 2025-02-10T09:29:17Z

scylla/tests/ccm-integration/logged_cli.rs

+        let command_with_args = std::iter::once(command.as_ref().to_string_lossy().into_owned())
+            .chain(
+                args.clone()
+                    .into_iter()
+                    .map(|s| s.as_ref().to_string_lossy().into_owned()),
+            )
+            .collect::<Vec<String>>()
+            .join(" ");


🔧 You allocate this String unconditionally, and its only use is in process_child_result upon error. I believe it should be built lazily, only in the error case, inside process_child_result. Is this possible?

Moving this would also make this function yet shorter, which is good considering its importance.

wprzytula · 2025-02-10T09:33:19Z

scylla/tests/ccm-integration/logged_cli.rs

+        match buffer {
+            Some(buffer) => {
+                while let Some(line) = lines.next_line().await.ok().flatten() {
+                    let _ = writer
+                        .lock()
+                        .await
+                        .write_all(format!("{} {}\n", prefix, line).as_bytes())
+                        .await;
+                    buffer.extend_from_slice(line.as_bytes());
+                }
+            }
+            None => {
+                while let Some(line) = lines.next_line().await.ok().flatten() {
+                    let _ = writer
+                        .lock()
+                        .await
+                        .write_all(format!("{} {}\n", prefix, line).as_bytes())
+                        .await;
+                }
+            }


🔧 The match over buffer should be inside the while let, not outside. This would deduplicate the while let.

wprzytula · 2025-02-10T09:37:35Z

scylla/tests/ccm-integration/node_config.rs

+impl Default for NodeConfig {
+    fn default() -> NodeConfig {
+        let mut val = HashMap::new();
+        val.insert(
+            "endpoint_snitch".to_string(),
+            NodeConfig::String(
+                "org.apache.cassandra.locator.GossipingPropertyFileSnitch".to_owned(),
+            ),
+        );
+        NodeConfig::Map(val)
+    }
+}


Btw is there a reason for making this a dynamic type instead of a struct with fields corresponding to config values, besides the fact that it would probably be more work?

It's arguably less work, as we let serde_yaml do the struct populating and error handling job for us.

wprzytula · 2025-02-10T09:41:06Z

scylla/tests/ccm-integration/node_config.rs

+        let keys: Vec<&str> = path.split('.').collect();
+        let mut current = self;
+
+        for key in keys {
+            match current {


🔧 You don't need to collect an iterator in order to iterate over it. Let's simply remove collect().

dkropachev requested a review from wprzytula February 6, 2025 04:52

dkropachev force-pushed the dk/tests-add-ccm-support branch 2 times, most recently from 386ad62 to a254460 Compare February 6, 2025 04:56

dkropachev force-pushed the dk/tests-add-ccm-support branch 5 times, most recently from 281258a to 2c4c00b Compare February 6, 2025 15:09

dkropachev requested review from Lorak-mmk and muzarski February 6, 2025 17:52

wprzytula requested changes Feb 7, 2025

View reviewed changes

wprzytula assigned dkropachev Feb 7, 2025

wprzytula requested changes Feb 7, 2025

View reviewed changes

github-actions bot added the semver-checks-breaking cargo-semver-checks reports that this PR introduces breaking API changes label Feb 7, 2025

scylla: cleanup Cargo.toml

e936908

Cleanup unused dependancies

dkropachev force-pushed the dk/tests-add-ccm-support branch from f14fa11 to b900f55 Compare February 8, 2025 03:02

github-actions bot removed the semver-checks-breaking cargo-semver-checks reports that this PR introduces breaking API changes label Feb 8, 2025

dkropachev force-pushed the dk/tests-add-ccm-support branch from b900f55 to 0c9ff2a Compare February 8, 2025 03:10

dkropachev requested a review from wprzytula February 8, 2025 03:10

dkropachev force-pushed the dk/tests-add-ccm-support branch 3 times, most recently from 13d7c5e to d97469d Compare February 8, 2025 21:27

Lorak-mmk requested changes Feb 9, 2025

View reviewed changes

muzarski reviewed Feb 9, 2025

View reviewed changes

scylla/tests/ccm-integration/logged_cli.rs Outdated Show resolved Hide resolved

scylla/tests/ccm-integration/test_example.rs Outdated Show resolved Hide resolved

dkropachev added 3 commits February 9, 2025 21:17

scylla/tests: implement logged cli crate

19c9d76

Implement simple abstraction that allows to run commands logging command, return status, stderr and stdout to a file and optionally to stderr/stdout

scylla/tests: implement struct to represent scylla.yaml

2ac777f

In order to programatically manipulate by scylla.yaml and cassandra.yaml We need to have some struct that provides API for that. It also needs to represent it in the way that ccm updateconf understand

tests: implement ccm integration

f858370

CCM is cli tool to ease scylla/cassandra/dse cluster management. Using it in integration suit will help to expand test coverage.

dkropachev force-pushed the dk/tests-add-ccm-support branch from d97469d to f858370 Compare February 10, 2025 01:17

wprzytula reviewed Feb 10, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tests: implement ccm integration #1203

tests: implement ccm integration #1203

dkropachev commented Feb 6, 2025

github-actions bot commented Feb 6, 2025 •

edited

Loading

wprzytula left a comment

wprzytula Feb 7, 2025

dkropachev Feb 8, 2025

wprzytula Feb 8, 2025

dkropachev Feb 8, 2025

Lorak-mmk left a comment

Lorak-mmk Feb 9, 2025

Lorak-mmk Feb 9, 2025

Lorak-mmk Feb 9, 2025

Lorak-mmk Feb 9, 2025

Lorak-mmk Feb 9, 2025

Lorak-mmk Feb 9, 2025

Lorak-mmk Feb 9, 2025

dkropachev Feb 10, 2025

Lorak-mmk Feb 10, 2025

Lorak-mmk Feb 9, 2025

dkropachev Feb 10, 2025

Lorak-mmk Feb 9, 2025

dkropachev Feb 10, 2025

Lorak-mmk Feb 9, 2025

dkropachev Feb 10, 2025

muzarski left a comment

dkropachev commented Feb 10, 2025

wprzytula Feb 10, 2025

wprzytula Feb 10, 2025

wprzytula Feb 10, 2025

wprzytula Feb 10, 2025

wprzytula Feb 10, 2025

wprzytula Feb 10, 2025

wprzytula Feb 10, 2025

wprzytula Feb 10, 2025

wprzytula Feb 10, 2025

	mut buffer: Option<&mut Vec<u8>>,
	buffer: Option<&mut Vec<u8>>,

	pub(crate) fn create_new_session_builder() -> GenericSessionBuilder<impl SessionBuilderKind> {
	let session_builder = {
	#[cfg(not(scylla_cloud_tests))]
	{
	use scylla::client::session_builder::SessionBuilder;

	let uri = std::env::var("SCYLLA_URI").unwrap_or_else(\|_\| "127.0.0.1:9042".to_string());

	SessionBuilder::new().known_node(uri)
	}

	#[cfg(scylla_cloud_tests)]
	{
	use scylla::client::session_builder::{CloudMode, CloudSessionBuilder};
	use std::path::Path;

	std::env::var("CLOUD_CONFIG_PATH")
	.map(\|config_path\| CloudSessionBuilder::new(Path::new(&config_path)))
	.expect("Failed to initialize CloudSessionBuilder")
	.expect("CLOUD_CONFIG_PATH environment variable is missing")
	}
	};

	// The reason why we enable so long waiting for TracingInfo is... Cassandra. (Yes, again.)
	// In Cassandra Java Driver, the wait time for tracing info is 10 seconds, so here we do the same.
	// However, as Scylla usually gets TracingInfo ready really fast (our default interval is hence 3ms),
	// we stick to a not-so-much-terribly-long interval here.
	session_builder
	.tracing_info_fetch_attempts(NonZeroU32::new(200).unwrap())
	.tracing_info_fetch_interval(Duration::from_millis(50))
	}

tests: implement ccm integration #1203

Are you sure you want to change the base?

tests: implement ccm integration #1203

Conversation

dkropachev commented Feb 6, 2025

Pre-review checklist

github-actions bot commented Feb 6, 2025 • edited Loading

wprzytula left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Lorak-mmk left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

muzarski left a comment

Choose a reason for hiding this comment

dkropachev commented Feb 10, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Feb 6, 2025 •

edited

Loading