From a4005cb47c10ec1b7653607e96550890295c000f Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Fri, 22 Mar 2024 12:21:46 +0300 Subject: [PATCH 001/257] Added Python bindings link to Readme --- README.md | 48 +++++++++++++++++++++++++++--------------------- 1 file changed, 27 insertions(+), 21 deletions(-) diff --git a/README.md b/README.md index bde251f1a..bbc59111f 100644 --- a/README.md +++ b/README.md @@ -35,6 +35,14 @@ $ cargo install xvc [installed]: https://www.rust-lang.org/tools/install +If you want to use Xvc with Python console and Jupyter notebooks, you can also install it with `pip`: + +```shell +$ pip install xvc +``` + +Note that pip installation doesn't make `xvc` available as a shell command. Please see [xvc.py](https://github.com/iesahin/xvc.py) for usage details. + ## 🏃🏾 Quicktart Xvc seamlessly monitors your files and directories on top of Git. To commence, execute the following command within the repository: @@ -54,7 +62,7 @@ Include your data files and directories for tracking: $ xvc file track my-data/ --as symlink ``` -This command calculates content hashes for data (using BLAKE-3, by default) and logs them. The changes are committed to Git, and the files are copied to content-addressed directories within `.xvc/b3`. Additionally, read-only symbolic links to these directories are created. +This command calculates content hashes for data (using BLAKE-3, by default) and logs them. The changes are committed to Git, and the files are copied to content-addressed directories within `.xvc/b3`. Additionally, read-only symbolic links to these directories are created. You can specify different [recheck (checkout) methods](https://docs.xvc.dev/ref/xvc-file-recheck/) for files and directories, depending on your use case. If you need to track model files that change frequently, you can set recheck method `--as copy` (the default). @@ -103,15 +111,15 @@ The script uses the Faker library and this library must be available where you r $ xvc pipeline step new --step-name install-deps --command 'python3 -m pip install --quiet --user -r requirements.txt' ``` -We'll make this this step to depend on `requirements.txt` file, so when the file changes it will make the step run. +We'll make this this step to depend on `requirements.txt` file, so when the file changes it will make the step run. ```console $ xvc pipeline step dependency --step-name install-deps --file requirements.txt ``` -Xvc allows to create dependencies between pipeline steps. Dependent steps wait for dependencies to finish successfully. +Xvc allows to create dependencies between pipeline steps. Dependent steps wait for dependencies to finish successfully. -Now we create a step to run the script and make `install-deps` step a dependency of it. +Now we create a step to run the script and make `install-deps` step a dependency of it. ```console $ xvc pipeline step new --step-name generate-data --command 'python3 generate_data.py' @@ -124,19 +132,19 @@ After you define the pipeline, you can run it by: $ xvc pipeline run [DONE] install-deps (python3 -m pip install --quiet --user -r requirements.txt) [OUT] [generate-data] CSV file generated successfully. - + [DONE] generate-data (python3 generate_data.py) ``` -Xvc allows many kinds of dependnecies, like [files](https://docs.xvc.dev/ref/xvc-pipeline-step-dependency#file-dependencies), -[groups of files and directories defined by globs](https://docs.xvc.dev/ref/xvc-pipeline-step-dependency#glob-dependencies), -[regular expression searches in files](https://docs.xvc.dev/ref/xvc-pipeline-step-dependency#regex-dependencies), -[line ranges in files](https://docs.xvc.dev/ref/xvc-pipeline-step-dependency#line-dependencies), +Xvc allows many kinds of dependnecies, like [files](https://docs.xvc.dev/ref/xvc-pipeline-step-dependency#file-dependencies), +[groups of files and directories defined by globs](https://docs.xvc.dev/ref/xvc-pipeline-step-dependency#glob-dependencies), +[regular expression searches in files](https://docs.xvc.dev/ref/xvc-pipeline-step-dependency#regex-dependencies), +[line ranges in files](https://docs.xvc.dev/ref/xvc-pipeline-step-dependency#line-dependencies), [hyper-parameters defined in YAML, JSON or TOML files](https://docs.xvc.dev/ref/xvc-pipeline-step-dependency#hyper-parameter-dependencies) [HTTP URLs](https://docs.xvc.dev/ref/xvc-pipeline-step-dependency#url-dependencies), -[shell command outputs](https://docs.xvc.dev/ref/xvc-pipeline-step-dependency#generic-command-dependencies), -and [other steps](https://docs.xvc.dev/ref/xvc-pipeline-step-dependency#step-dependencies). +[shell command outputs](https://docs.xvc.dev/ref/xvc-pipeline-step-dependency#generic-command-dependencies), +and [other steps](https://docs.xvc.dev/ref/xvc-pipeline-step-dependency#step-dependencies). Suppose you're only interested in the IQ scores of those with _Dr._ in front of their names and how they differ from the rest in the dataset we created. Let's create a regex search dependency to the data file that will show all _doctors_ IQ scores. @@ -145,13 +153,12 @@ $ xvc pipeline step new --step-name dr-iq --command 'echo "${XVC_REGEX_ADDED_ITE $ xvc pipeline step dependency --step-name dr-iq --regex-items 'random_names_iq_scores.csv:/^Dr\..*' ``` -The first line specifies a command, when run writes `${XVC_REGEX_ADDED_ITEMS}` environment variable to `dr-iq-scores.csv` file. -The second line specifies the dependency which will also populate the `$[XVC_REGEX_ADDED_ITEMS]` environment variable in the command. +The first line specifies a command, when run writes `${XVC_REGEX_ADDED_ITEMS}` environment variable to `dr-iq-scores.csv` file. +The second line specifies the dependency which will also populate the `$[XVC_REGEX_ADDED_ITEMS]` environment variable in the command. -Some dependency types like [regex items], +Some dependency types like [regex items], [line items] and [glob items] inject environment variables in the commands they are a dependency. -For example, if you have two million files specified with a glob, but want to run a script only on the added files after the last run, you can use these environment variables. - +For example, if you have two million files specified with a glob, but want to run a script only on the added files after the last run, you can use these environment variables. When you run the pipeline again, a file named `dr-iq-scores.csv` will be created. Note that, as `requirements.txt` didn't change `install-deps` step and its dependent `generate-data` steps didn't run. @@ -166,7 +173,7 @@ Dr. Mallory Payne MD,70 Dr. Sherry Leonard,93 Dr. Susan Swanson,81 -```` +``` We are using this feature to get lines starting with `Dr.` from the file and write them to another file. When the file changes, e.g. another record matching the dependency regex added to the `random_names_iq_scores.csv` file, it will also be added to `dr-iq-scores.csv` file. @@ -347,8 +354,7 @@ You can edit the file to change commands, add new dependencies, etc. and import $ xvc pipeline import --file my-pipeline.json --overwrite ``` -Lastly, if you noticed that the commands are long to type, there is an `xvc aliases` command that prints a set of aliases for commands. You can source the output in your `.zshrc` or `.bashrc`, and use the following commands instead, e.g., `xvc pipelines run` becomes `pvc run`. - +Lastly, if you noticed that the commands are long to type, there is an `xvc aliases` command that prints a set of aliases for commands. You can source the output in your `.zshrc` or `.bashrc`, and use the following commands instead, e.g., `xvc pipelines run` becomes `pvc run`. ```console $ xvc aliases @@ -450,7 +456,7 @@ And, biggest thanks to Rust designers, developers and contributors. Although I c - Star this repo. I feel very happy for every star and send my best wishes to you. That's a certain win to spend your two seconds for me. Thanks. - Use xvc. Tell me how it works for you, read the [documentation](https://docs.xvc.dev), [report bugs](https://github.com/iesahin/xvc/issues), [discuss features](https://github.com/iesahin/xvc/discussions). - Please note that, I don't accept large code PRs. Please open an issue to discuss your idea and write/modify a - reference page before sending a PR. I'm happy to discuss and help you to implement your idea. Also, it may require a copyright transfer to me, as there may be cases which I provide the code in other licenses. + reference page before sending a PR. I'm happy to discuss and help you to implement your idea. Also, it may require a copyright transfer to me, as there may be cases which I provide the code in other licenses. ## 📜 License @@ -461,7 +467,7 @@ Xvc is licensed under the [GNU GPL 3.0 License](https://github.com/iesahin/xvc/b I'm using Xvc daily and I'm happy with it. Tracking all my files with Git via arbitrary servers and cloud providers is something I always need. I'm happy to improve and maintain it as long as I use it. -Given that I'm working on this for the last two years for pure technical bliss, you can expect me to work on it more. +Given that I'm working on this for the last two years for pure technical bliss, you can expect me to work on it more. ## ⚠️ Disclaimer From bfad70daf3320cedc5de2bbe251a6844c77ee08f Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Wed, 1 May 2024 20:04:10 +0300 Subject: [PATCH 002/257] Replace Xvc in ALL CAPS --- config/src/lib.rs | 2 +- core/src/error.rs | 4 +- core/src/root/mod.rs | 2 +- core/src/types/xvcroot.rs | 63 ++++++++++++++++-------------- lib/src/cli/mod.rs | 2 +- lib/src/error.rs | 6 +-- lib/src/init/mod.rs | 4 +- lib/src/lib.rs | 2 +- pipeline/src/pipeline/mod.rs | 2 +- workflow_tests/tests/common/mod.rs | 2 +- 10 files changed, 46 insertions(+), 43 deletions(-) diff --git a/config/src/lib.rs b/config/src/lib.rs index 3518bd605..ce0a42c6a 100755 --- a/config/src/lib.rs +++ b/config/src/lib.rs @@ -196,7 +196,7 @@ pub struct XvcConfigInitParams { pub struct XvcConfig { /// Current directory. It can be set with xvc -C option pub current_dir: XvcConfigOption, - // /// The root if the command is happen to be run in XVC directory + // /// The root if the command is happen to be run in Xvc directory // pub xvc_root: XvcConfigOption>, /// Configuration values for each level pub config_maps: Vec, diff --git a/core/src/error.rs b/core/src/error.rs index 53048c5d8..f821dc3ea 100644 --- a/core/src/error.rs +++ b/core/src/error.rs @@ -45,9 +45,9 @@ pub enum Error { #[error("File System Walk Error: {error}")] FSWalkerError { error: String }, - #[error("Cannot find XVC Root: {path}")] + #[error("Cannot find Xvc Root: {path}")] CannotFindXvcRoot { path: PathBuf }, - #[error("Cannot nest XVC repositories: {path}")] + #[error("Cannot nest Xvc repositories: {path}")] CannotNestXvcRepositories { path: PathBuf }, #[error("Regex Error: {source}")] RegexError { diff --git a/core/src/root/mod.rs b/core/src/root/mod.rs index 6b24d58a5..f568d2aa5 100644 --- a/core/src/root/mod.rs +++ b/core/src/root/mod.rs @@ -8,7 +8,7 @@ use xvc_logging::{output, watch, XvcOutputSender}; #[derive(Debug, Parser)] #[command(name = "root")] -/// Get the XVC root directory for the current project +/// Get the Xvc root directory for the current project pub struct RootCLI { #[arg(long)] /// Show absolute path instead of relative diff --git a/core/src/types/xvcroot.rs b/core/src/types/xvcroot.rs index dfde9716d..1b3654f76 100644 --- a/core/src/types/xvcroot.rs +++ b/core/src/types/xvcroot.rs @@ -71,36 +71,7 @@ impl Deref for XvcRootInner { /// [XvcRoot::find_root] and uses it as the root. pub fn load_xvc_root(path: &Path, config_opts: XvcConfigInitParams) -> Result { match XvcRootInner::find_root(path) { - Ok(absolute_path) => { - let xvc_dir = absolute_path.join(XvcRootInner::XVC_DIR); - let local_config_path = xvc_dir.join(XvcRootInner::LOCAL_CONFIG_PATH); - let project_config_path = xvc_dir.join(XvcRootInner::PROJECT_CONFIG_PATH); - let config_opts = XvcConfigInitParams { - project_config_path: Some(project_config_path.clone()), - local_config_path: Some(local_config_path.clone()), - default_configuration: config_opts.default_configuration, - current_dir: config_opts.current_dir, - include_system_config: config_opts.include_system_config, - include_user_config: config_opts.include_user_config, - include_environment_config: config_opts.include_environment_config, - command_line_config: config_opts.command_line_config, - }; - let config = XvcConfig::new(config_opts)?; - let entity_generator = - xvc_ecs::load_generator(&xvc_dir.join(XvcRootInner::ENTITY_GENERATOR_PATH))?; - - let store_dir = xvc_dir.join(XvcRootInner::STORE_DIR); - let xvc_root = Arc::new(XvcRootInner { - xvc_dir, - store_dir, - local_config_path, - project_config_path, - absolute_path, - config, - entity_generator, - }); - Ok(xvc_root) - } + Ok(absolute_path) => Ok(Arc::new(XvcRootInner::new(absolute_path, config_opts)?)), Err(e) => Err(e), } } @@ -170,6 +141,38 @@ pub fn init_xvc_root(path: &Path, config_opts: XvcConfigInitParams) -> Result Result { + let xvc_dir = absolute_path.join(XvcRootInner::XVC_DIR); + let local_config_path = xvc_dir.join(XvcRootInner::LOCAL_CONFIG_PATH); + let project_config_path = xvc_dir.join(XvcRootInner::PROJECT_CONFIG_PATH); + let config_opts = XvcConfigInitParams { + project_config_path: Some(project_config_path.clone()), + local_config_path: Some(local_config_path.clone()), + default_configuration: config_opts.default_configuration, + current_dir: config_opts.current_dir, + include_system_config: config_opts.include_system_config, + include_user_config: config_opts.include_user_config, + include_environment_config: config_opts.include_environment_config, + command_line_config: config_opts.command_line_config, + }; + let config = XvcConfig::new(config_opts)?; + let entity_generator = + xvc_ecs::load_generator(&xvc_dir.join(XvcRootInner::ENTITY_GENERATOR_PATH))?; + + let store_dir = xvc_dir.join(XvcRootInner::STORE_DIR); + Ok(Self { + xvc_dir, + store_dir, + local_config_path, + project_config_path, + absolute_path, + config, + entity_generator, + }) + } + /// Join `path` to the repository root and return the absolute path of the /// given path. /// diff --git a/lib/src/cli/mod.rs b/lib/src/cli/mod.rs index 80c15744a..64885421b 100644 --- a/lib/src/cli/mod.rs +++ b/lib/src/cli/mod.rs @@ -1,4 +1,4 @@ -//! Main CLI interface for XVC +//! Main CLI interface for Xvc use std::env::ArgsOs; use std::ffi::OsString; diff --git a/lib/src/error.rs b/lib/src/error.rs index 4f730b7d5..32ba5192b 100644 --- a/lib/src/error.rs +++ b/lib/src/error.rs @@ -99,12 +99,12 @@ pub enum Error { #[from] source: io::Error, }, - #[error("Path is not in XVC Repository: {path:?}")] + #[error("Path is not in Xvc Repository: {path:?}")] PathNotInXvcRepository { path: OsString }, #[error("Path has no parent: {path:?}")] PathHasNoParent { path: OsString }, - #[error("This directory already belongs to an XVC repository {path:?}")] - DirectoryContainsXVCAlready { path: OsString }, + #[error("This directory already belongs to an Xvc repository {path:?}")] + DirectoryContainsXvcAlready { path: OsString }, #[error("This directory is not in a Git Repository {path:?}")] PathNotInGitRepository { path: OsString }, #[error("Cannot Parse Integer: {source:?}")] diff --git a/lib/src/init/mod.rs b/lib/src/init/mod.rs index d8d7965b4..87478a64e 100644 --- a/lib/src/init/mod.rs +++ b/lib/src/init/mod.rs @@ -14,7 +14,7 @@ use xvc_logging::watch; use xvc_pipeline; use xvc_walker::AbsolutePath; -/// Initialize an XVC repository +/// Initialize an Xvc repository #[derive(Debug, Clone, Parser)] #[command(author, version)] pub struct InitCLI { @@ -59,7 +59,7 @@ pub fn run(xvc_root_opt: Option<&XvcRoot>, opts: InitCLI) -> Result { ); fs::remove_dir_all(xvc_root.xvc_dir())?; } else { - return Err(Error::DirectoryContainsXVCAlready { + return Err(Error::DirectoryContainsXvcAlready { path: xvc_root.absolute_path().as_os_str().to_os_string(), }); } diff --git a/lib/src/lib.rs b/lib/src/lib.rs index 64bb7ff12..4dfb4e96c 100755 --- a/lib/src/lib.rs +++ b/lib/src/lib.rs @@ -1,6 +1,6 @@ #![warn(missing_docs)] #![forbid(unsafe_code)] -//! The main dispatching functions for the entire XVC CLI +//! The main dispatching functions for the entire Xvc CLI pub mod cli; pub mod error; pub mod git; diff --git a/pipeline/src/pipeline/mod.rs b/pipeline/src/pipeline/mod.rs index cd57d22ba..0a6824e86 100644 --- a/pipeline/src/pipeline/mod.rs +++ b/pipeline/src/pipeline/mod.rs @@ -240,7 +240,7 @@ struct StepThreadParams<'a> { output_diffs: Arc>>>, } -/// # XVC Pipeline Dependency Graph Rules +/// # Xvc Pipeline Dependency Graph Rules /// /// The dependency graph shows which steps of the pipeline depends on other /// steps. The dependency steps are set to run before the dependent steps. diff --git a/workflow_tests/tests/common/mod.rs b/workflow_tests/tests/common/mod.rs index 25a368d1f..7fc9003c0 100644 --- a/workflow_tests/tests/common/mod.rs +++ b/workflow_tests/tests/common/mod.rs @@ -152,7 +152,7 @@ pub fn run_in_example_xvc(with_git: bool) -> Result { } } -/// Create a temporary XVC directory that's also Git repository +/// Create a temporary Xvc directory that's also Git repository pub fn run_in_temp_xvc_dir() -> Result { let the_dir = run_in_temp_git_dir(); watch!(&the_dir); From da8d0bfeca58b96e2fa3f39b8c323dc5cfbd6248 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Thu, 2 May 2024 13:20:51 +0300 Subject: [PATCH 003/257] Remove path argument from load_xvc_root --- config/Cargo.toml | 6 +++--- core/Cargo.toml | 12 ++++++------ core/src/types/xvcroot.rs | 6 ++++-- ecs/Cargo.toml | 4 ++-- file/Cargo.toml | 16 ++++++++-------- file/src/lib.rs | 2 +- lib/Cargo.toml | 20 ++++++++++---------- lib/src/cli/mod.rs | 3 +-- logging/Cargo.toml | 2 +- pipeline/Cargo.toml | 16 ++++++++-------- storage/Cargo.toml | 14 +++++++------- test_helper/Cargo.toml | 4 ++-- walker/Cargo.toml | 6 +++--- workflow_tests/Cargo.toml | 22 +++++++++++----------- 14 files changed, 67 insertions(+), 66 deletions(-) diff --git a/config/Cargo.toml b/config/Cargo.toml index 521db26c0..a581613f5 100644 --- a/config/Cargo.toml +++ b/config/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-config" -version = "0.6.7" +version = "0.6.8-alpha.0" edition = "2021" description = "Xvc configuration management" authors = ["Emre Şahin "] @@ -20,8 +20,8 @@ debug = true [dependencies] -xvc-logging = { version = "0.6.7", path = "../logging" } -xvc-walker = { version = "0.6.7", path = "../walker" } +xvc-logging = { version = "0.6.8-alpha.0", path = "../logging" } +xvc-walker = { version = "0.6.8-alpha.0", path = "../walker" } ## Cli and config diff --git a/core/Cargo.toml b/core/Cargo.toml index c1c4b6918..1dc6eb11f 100644 --- a/core/Cargo.toml +++ b/core/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-core" -version = "0.6.7" +version = "0.6.8-alpha.0" edition = "2021" description = "Xvc core for common elements for all commands" authors = ["Emre Şahin "] @@ -19,10 +19,10 @@ crate-type = ["rlib"] debug = true [dependencies] -xvc-config = { version = "0.6.7", path = "../config" } -xvc-logging = { version = "0.6.7", path = "../logging" } -xvc-ecs = { version = "0.6.7", path = "../ecs" } -xvc-walker = { version = "0.6.7", path = "../walker" } +xvc-config = { version = "0.6.8-alpha.0", path = "../config" } +xvc-logging = { version = "0.6.8-alpha.0", path = "../logging" } +xvc-ecs = { version = "0.6.8-alpha.0", path = "../ecs" } +xvc-walker = { version = "0.6.8-alpha.0", path = "../walker" } ## Cli and config clap = { version = "^4.4", features = ["derive"] } @@ -82,6 +82,6 @@ derive_more = "^0.99" itertools = "^0.12" [dev-dependencies] -xvc-test-helper = { version = "0.6.7", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.8-alpha.0", path = "../test_helper/" } proptest = "^1.4" test-case = "^3.3" diff --git a/core/src/types/xvcroot.rs b/core/src/types/xvcroot.rs index 1b3654f76..c29b04283 100644 --- a/core/src/types/xvcroot.rs +++ b/core/src/types/xvcroot.rs @@ -69,7 +69,9 @@ impl Deref for XvcRootInner { /// The path is not required to be the root of the repository. /// This function searches for the root of the repository using /// [XvcRoot::find_root] and uses it as the root. -pub fn load_xvc_root(path: &Path, config_opts: XvcConfigInitParams) -> Result { +pub fn load_xvc_root(config_opts: XvcConfigInitParams) -> Result { + let path = config_opts.current_dir.as_ref(); + match XvcRootInner::find_root(path) { Ok(absolute_path) => Ok(Arc::new(XvcRootInner::new(absolute_path, config_opts)?)), Err(e) => Err(e), @@ -132,7 +134,7 @@ pub fn init_xvc_root(path: &Path, config_opts: XvcConfigInitParams) -> Result"] @@ -19,7 +19,7 @@ crate-type = ["rlib"] debug = true [dependencies] -xvc-logging = { version = "0.6.7", path = "../logging" } +xvc-logging = { version = "0.6.8-alpha.0", path = "../logging" } ## Serialization serde = { version = "^1.0", features = ["derive"] } diff --git a/file/Cargo.toml b/file/Cargo.toml index 22d3ee19f..a521f9dfd 100644 --- a/file/Cargo.toml +++ b/file/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-file" -version = "0.6.7" +version = "0.6.8-alpha.0" edition = "2021" description = "File tracking, versioning, upload and download functions for Xvc" authors = ["Emre Şahin "] @@ -24,12 +24,12 @@ bench = true debug = true [dependencies] -xvc-logging = { version = "0.6.7", path = "../logging" } -xvc-config = { version = "0.6.7", path = "../config" } -xvc-core = { version = "0.6.7", path = "../core" } -xvc-ecs = { version = "0.6.7", path = "../ecs" } -xvc-walker = { version = "0.6.7", path = "../walker" } -xvc-storage = { version = "0.6.7", path = "../storage" } +xvc-logging = { version = "0.6.8-alpha.0", path = "../logging" } +xvc-config = { version = "0.6.8-alpha.0", path = "../config" } +xvc-core = { version = "0.6.8-alpha.0", path = "../core" } +xvc-ecs = { version = "0.6.8-alpha.0", path = "../ecs" } +xvc-walker = { version = "0.6.8-alpha.0", path = "../walker" } +xvc-storage = { version = "0.6.8-alpha.0", path = "../storage" } ## Cli and config @@ -94,5 +94,5 @@ parse-size = "^1.0" [dev-dependencies] -xvc-test-helper = { version = "0.6.7", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.8-alpha.0", path = "../test_helper/" } shellfn = "^0.1" diff --git a/file/src/lib.rs b/file/src/lib.rs index 16dc771dd..d7eb42a8d 100755 --- a/file/src/lib.rs +++ b/file/src/lib.rs @@ -276,7 +276,7 @@ pub fn dispatch(cli_opts: XvcFileCLI) -> Result<()> { default_configuration: default_project_config(true), }; - let xvc_root = match load_xvc_root(Path::new(&cli_opts.workdir), xvc_config_params) { + let xvc_root = match load_xvc_root(xvc_config_params) { Ok(r) => Some(r), Err(e) => { e.info(); diff --git a/lib/Cargo.toml b/lib/Cargo.toml index bca10752e..00810fab5 100644 --- a/lib/Cargo.toml +++ b/lib/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc" -version = "0.6.7" +version = "0.6.8-alpha.0" edition = "2021" description = "An MLOps tool to manage data files and pipelines on top of Git" authors = ["Emre Şahin "] @@ -23,14 +23,14 @@ path = "src/main.rs" debug = true [dependencies] -xvc-config = { version = "0.6.7", path = "../config" } -xvc-core = { version = "0.6.7", path = "../core" } -xvc-logging = { version = "0.6.7", path = "../logging" } -xvc-ecs = { version = "0.6.7", path = "../ecs" } -xvc-file = { version = "0.6.7", path = "../file" } -xvc-pipeline = { version = "0.6.7", path = "../pipeline" } -xvc-walker = { version = "0.6.7", path = "../walker" } -xvc-storage = { version = "0.6.7", path = "../storage" } +xvc-config = { version = "0.6.8-alpha.0", path = "../config" } +xvc-core = { version = "0.6.8-alpha.0", path = "../core" } +xvc-logging = { version = "0.6.8-alpha.0", path = "../logging" } +xvc-ecs = { version = "0.6.8-alpha.0", path = "../ecs" } +xvc-file = { version = "0.6.8-alpha.0", path = "../file" } +xvc-pipeline = { version = "0.6.8-alpha.0", path = "../pipeline" } +xvc-walker = { version = "0.6.8-alpha.0", path = "../walker" } +xvc-storage = { version = "0.6.8-alpha.0", path = "../storage" } ## Cli and config @@ -113,4 +113,4 @@ jwalk = "^0.8" proptest = "^1.4" shellfn = "^0.1" test-case = "^3.3" -xvc-test-helper = { version = "0.6.7", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.8-alpha.0", path = "../test_helper/" } diff --git a/lib/src/cli/mod.rs b/lib/src/cli/mod.rs index 64885421b..4a36079c6 100644 --- a/lib/src/cli/mod.rs +++ b/lib/src/cli/mod.rs @@ -16,7 +16,6 @@ use std::io; use xvc_core::types::xvcroot::load_xvc_root; use xvc_logging::{debug, error, uwr, XvcOutputLine}; -use std::path::Path; use xvc_config::{XvcConfigInitParams, XvcVerbosity}; use xvc_core::aliases; use xvc_core::check_ignore; @@ -233,7 +232,7 @@ pub fn dispatch(cli_opts: cli::XvcCLI) -> Result<()> { default_configuration: default_project_config(true), }; - let xvc_root_opt = match load_xvc_root(Path::new(&cli_opts.workdir), xvc_config_params) { + let xvc_root_opt = match load_xvc_root(xvc_config_params) { Ok(r) => Some(r), Err(e) => { e.debug(); diff --git a/logging/Cargo.toml b/logging/Cargo.toml index b5a9e1608..b2668c421 100644 --- a/logging/Cargo.toml +++ b/logging/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-logging" -version = "0.6.7" +version = "0.6.8-alpha.0" edition = "2021" description = "Logging crate for Xvc" authors = ["Emre Şahin "] diff --git a/pipeline/Cargo.toml b/pipeline/Cargo.toml index e4174e0c8..0b495601e 100644 --- a/pipeline/Cargo.toml +++ b/pipeline/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-pipeline" -version = "0.6.7" +version = "0.6.8-alpha.0" edition = "2021" description = "Xvc data pipeline management" authors = ["Emre Şahin "] @@ -19,12 +19,12 @@ debug = true [dependencies] -xvc-config = { version = "0.6.7", path = "../config" } -xvc-core = { version = "0.6.7", path = "../core" } -xvc-ecs = { version = "0.6.7", path = "../ecs" } -xvc-logging = { version = "0.6.7", path = "../logging" } -xvc-walker = { version = "0.6.7", path = "../walker" } -xvc-file = { version = "0.6.7", path = "../file" } +xvc-config = { version = "0.6.8-alpha.0", path = "../config" } +xvc-core = { version = "0.6.8-alpha.0", path = "../core" } +xvc-ecs = { version = "0.6.8-alpha.0", path = "../ecs" } +xvc-logging = { version = "0.6.8-alpha.0", path = "../logging" } +xvc-walker = { version = "0.6.8-alpha.0", path = "../walker" } +xvc-file = { version = "0.6.8-alpha.0", path = "../file" } ## Cli and config clap = { version = "^4.4", features = ["derive"] } @@ -92,5 +92,5 @@ itertools = "^0.12" derive_more = "^0.99" [dev-dependencies] -xvc-test-helper = { version = "0.6.7", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.8-alpha.0", path = "../test_helper/" } test-case = "^3.3" diff --git a/storage/Cargo.toml b/storage/Cargo.toml index 6ab74b461..2a99d631c 100644 --- a/storage/Cargo.toml +++ b/storage/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-storage" -version = "0.6.7" +version = "0.6.8-alpha.0" edition = "2021" description = "Xvc remote (and local) storage management" authors = ["Emre Şahin "] @@ -20,11 +20,11 @@ debug = true [dependencies] -xvc-logging = { version = "0.6.7", path = "../logging" } -xvc-config = { version = "0.6.7", path = "../config" } -xvc-core = { version = "0.6.7", path = "../core" } -xvc-ecs = { version = "0.6.7", path = "../ecs" } -xvc-walker = { version = "0.6.7", path = "../walker" } +xvc-logging = { version = "0.6.8-alpha.0", path = "../logging" } +xvc-config = { version = "0.6.8-alpha.0", path = "../config" } +xvc-core = { version = "0.6.8-alpha.0", path = "../core" } +xvc-ecs = { version = "0.6.8-alpha.0", path = "../ecs" } +xvc-walker = { version = "0.6.8-alpha.0", path = "../walker" } ## Cli and config clap = { version = "^4.4", features = ["derive"] } @@ -109,7 +109,7 @@ digital-ocean = ["async"] [dev-dependencies] -xvc-test-helper = { version = "0.6.7", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.8-alpha.0", path = "../test_helper/" } shellfn = "^0.1" [package.metadata.cargo-udeps.ignore] diff --git a/test_helper/Cargo.toml b/test_helper/Cargo.toml index 0062e7fa4..f7cd31b8f 100644 --- a/test_helper/Cargo.toml +++ b/test_helper/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-test-helper" -version = "0.6.7" +version = "0.6.8-alpha.0" edition = "2021" description = "Unit test helper functions for Xvc" authors = ["Emre Şahin "] @@ -23,7 +23,7 @@ debug = true # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html [dependencies] -xvc-logging = { version = "0.6.7", path = "../logging/" } +xvc-logging = { version = "0.6.8-alpha.0", path = "../logging/" } rand = "^0.8" log = "^0.4" diff --git a/walker/Cargo.toml b/walker/Cargo.toml index 5cbcc717c..e0c377062 100644 --- a/walker/Cargo.toml +++ b/walker/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-walker" -version = "0.6.7" +version = "0.6.8-alpha.0" edition = "2021" description = "Xvc parallel file system walker with ignore features" authors = ["Emre Şahin "] @@ -19,7 +19,7 @@ crate-type = ["rlib"] debug = true [dependencies] -xvc-logging = { version = "0.6.7", path = "../logging" } +xvc-logging = { version = "0.6.8-alpha.0", path = "../logging" } globset = "^0.4" ## Parallelization @@ -41,7 +41,7 @@ itertools = "^0.12" regex = "^1.10" [dev-dependencies] -xvc-test-helper = { path = "../test_helper/", version = "0.6.7" } +xvc-test-helper = { path = "../test_helper/", version = "0.6.8-alpha.0" } test-case = "^3.3" [package.metadata.cargo-udeps.ignore] diff --git a/workflow_tests/Cargo.toml b/workflow_tests/Cargo.toml index 6d0ea06fe..9110d6833 100644 --- a/workflow_tests/Cargo.toml +++ b/workflow_tests/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-workflow-tests" -version = "0.6.7" +version = "0.6.8-alpha.0" edition = "2021" description = "Integration test suite for Xvc" authors = ["Emre Şahin "] @@ -23,15 +23,15 @@ debug = true [dependencies] -xvc = { version = "0.6.7", path = "../lib" } -xvc-config = { version = "0.6.7", path = "../config" } -xvc-core = { version = "0.6.7", path = "../core" } -xvc-logging = { version = "0.6.7", path = "../logging" } -xvc-ecs = { version = "0.6.7", path = "../ecs" } -xvc-file = { version = "0.6.7", path = "../file" } -xvc-pipeline = { version = "0.6.7", path = "../pipeline" } -xvc-walker = { version = "0.6.7", path = "../walker" } -xvc-storage = { version = "0.6.7", path = "../storage" } +xvc = { version = "0.6.8-alpha.0", path = "../lib" } +xvc-config = { version = "0.6.8-alpha.0", path = "../config" } +xvc-core = { version = "0.6.8-alpha.0", path = "../core" } +xvc-logging = { version = "0.6.8-alpha.0", path = "../logging" } +xvc-ecs = { version = "0.6.8-alpha.0", path = "../ecs" } +xvc-file = { version = "0.6.8-alpha.0", path = "../file" } +xvc-pipeline = { version = "0.6.8-alpha.0", path = "../pipeline" } +xvc-walker = { version = "0.6.8-alpha.0", path = "../walker" } +xvc-storage = { version = "0.6.8-alpha.0", path = "../storage" } ## packages for testing clap = { version = "^4.4", features = ["derive"] } @@ -70,7 +70,7 @@ proptest = "^1.4" test-case = "^3.3" globset = "^0.4" escargot = "^0.5" -xvc-test-helper = { version = "0.6.7", path = "../test_helper" } +xvc-test-helper = { version = "0.6.8-alpha.0", path = "../test_helper" } shellfn = "^0.1" jwalk = "^0.8" anyhow = "^1.0" From f3901d2d7aa85165f214fb6b15d521aa67393b46 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sat, 4 May 2024 14:00:14 +0300 Subject: [PATCH 004/257] Tidy find_root --- core/src/types/xvcroot.rs | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/core/src/types/xvcroot.rs b/core/src/types/xvcroot.rs index c29b04283..a62a54742 100644 --- a/core/src/types/xvcroot.rs +++ b/core/src/types/xvcroot.rs @@ -10,6 +10,7 @@ use std::path::{Path, PathBuf}; use std::sync::Arc; use xvc_ecs::ecs::timestamp; use xvc_ecs::{XvcEntity, XvcEntityGenerator}; +use xvc_logging::watch; use xvc_logging::{debug, trace}; use xvc_walker::AbsolutePath; @@ -72,6 +73,8 @@ impl Deref for XvcRootInner { pub fn load_xvc_root(config_opts: XvcConfigInitParams) -> Result { let path = config_opts.current_dir.as_ref(); + watch!(path); + match XvcRootInner::find_root(path) { Ok(absolute_path) => Ok(Arc::new(XvcRootInner::new(absolute_path, config_opts)?)), Err(e) => Err(e), @@ -249,20 +252,19 @@ impl XvcRootInner { /// Finds the root of the xvc repository by looking for the .xvc directory /// in parents of a given path. pub fn find_root(path: &Path) -> Result { - trace!("{:?}", path); - let mut pb = PathBuf::from(path) + watch!(path); + let abs_path = PathBuf::from(path) .canonicalize() .expect("Cannot canonicalize the path. Possible symlink loop."); - loop { - if pb.join(XVC_DIR).is_dir() { - debug!("XVC DIR: {:?}", pb); - return Ok(pb.into()); - } else if pb.parent().is_none() { - return Err(Error::CannotFindXvcRoot { path: path.into() }); - } else { - pb.pop(); + + for parent in abs_path.ancestors() { + watch!(parent); + if parent.join(XVC_DIR).is_dir() { + watch!("XVC DIR: {:?}", parent); + return Ok(parent.into()); } } + Err(Error::CannotFindXvcRoot { path: path.into() }) } } From fea5e7e6c48e25751c278f46917246b6b05fc6f3 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Wed, 8 May 2024 12:26:46 +0300 Subject: [PATCH 005/257] TIDY - split dispatch to dispatch_with_root --- core/src/types/xvcroot.rs | 24 ++++-- lib/src/cli/mod.rs | 151 +++++++++++++++++++++++--------------- 2 files changed, 111 insertions(+), 64 deletions(-) diff --git a/core/src/types/xvcroot.rs b/core/src/types/xvcroot.rs index a62a54742..2c3b41172 100644 --- a/core/src/types/xvcroot.rs +++ b/core/src/types/xvcroot.rs @@ -73,11 +73,17 @@ impl Deref for XvcRootInner { pub fn load_xvc_root(config_opts: XvcConfigInitParams) -> Result { let path = config_opts.current_dir.as_ref(); - watch!(path); + dbg!(path); match XvcRootInner::find_root(path) { - Ok(absolute_path) => Ok(Arc::new(XvcRootInner::new(absolute_path, config_opts)?)), - Err(e) => Err(e), + Ok(absolute_path) => { + dbg!(&absolute_path); + Ok(Arc::new(XvcRootInner::new(absolute_path, config_opts)?)) + } + Err(e) => { + dbg!(&e); + Err(e) + } } } @@ -150,8 +156,11 @@ impl XvcRootInner { /// other locations. `config_opts` can determine which configuration files to read pub fn new(absolute_path: AbsolutePath, config_opts: XvcConfigInitParams) -> Result { let xvc_dir = absolute_path.join(XvcRootInner::XVC_DIR); + dbg!(&xvc_dir); let local_config_path = xvc_dir.join(XvcRootInner::LOCAL_CONFIG_PATH); + dbg!(&local_config_path); let project_config_path = xvc_dir.join(XvcRootInner::PROJECT_CONFIG_PATH); + dbg!(&project_config_path); let config_opts = XvcConfigInitParams { project_config_path: Some(project_config_path.clone()), local_config_path: Some(local_config_path.clone()), @@ -252,15 +261,18 @@ impl XvcRootInner { /// Finds the root of the xvc repository by looking for the .xvc directory /// in parents of a given path. pub fn find_root(path: &Path) -> Result { - watch!(path); + dbg!(path); let abs_path = PathBuf::from(path) .canonicalize() .expect("Cannot canonicalize the path. Possible symlink loop."); + dbg!(&abs_path); for parent in abs_path.ancestors() { - watch!(parent); + dbg!(parent); + let xvc_candidate = parent.join(XVC_DIR); + dbg!(&xvc_candidate); if parent.join(XVC_DIR).is_dir() { - watch!("XVC DIR: {:?}", parent); + dbg!("XVC DIR: {:?}", parent); return Ok(parent.into()); } } diff --git a/lib/src/cli/mod.rs b/lib/src/cli/mod.rs index 4a36079c6..327556e93 100644 --- a/lib/src/cli/mod.rs +++ b/lib/src/cli/mod.rs @@ -14,6 +14,8 @@ use crossbeam_channel::bounded; use log::LevelFilter; use std::io; use xvc_core::types::xvcroot::load_xvc_root; +use xvc_core::types::xvcroot::XvcRootInner; +use xvc_core::XvcRoot; use xvc_logging::{debug, error, uwr, XvcOutputLine}; use xvc_config::{XvcConfigInitParams, XvcVerbosity}; @@ -179,66 +181,20 @@ pub fn run(args: &[&str]) -> Result<()> { dispatch(cli_options) } -/// Dispatch commands to respective functions in the API -/// -/// It sets output verbosity with [XvcCLI::verbosity]. -/// Determines configuration sources by filling [XvcConfigInitParams]. -/// Tries to create an XvcRoot to determine whether we're inside one. -/// Creates two threads: One for running the API function, one for getting strings from output -/// channel. -/// -/// A corresponding function to reuse the same [XvcRoot] object is [test_dispatch]. -/// It doesn't recreate the whole configuration and this prevents errors regarding multiple -/// initializations. -pub fn dispatch(cli_opts: cli::XvcCLI) -> Result<()> { - let verbosity = if cli_opts.quiet { - XvcVerbosity::Quiet - } else { - match cli_opts.verbosity { - 0 => XvcVerbosity::Default, - 1 => XvcVerbosity::Warn, - 2 => XvcVerbosity::Info, - 3 => XvcVerbosity::Debug, - _ => XvcVerbosity::Trace, - } - }; - - let term_log_level = match verbosity { - XvcVerbosity::Quiet => LevelFilter::Off, - XvcVerbosity::Default => LevelFilter::Error, - XvcVerbosity::Warn => LevelFilter::Warn, - XvcVerbosity::Info => LevelFilter::Info, - XvcVerbosity::Debug => LevelFilter::Debug, - XvcVerbosity::Trace => LevelFilter::Trace, - }; - - setup_logging( - Some(term_log_level), - if cli_opts.debug { - Some(LevelFilter::Trace) - } else { - None - }, +pub fn dispatch_with_root(cli_opts: cli::XvcCLI, xvc_root_opt: Option) -> Result<()> { + // XvcRoot should be kept per repository and shouldn't change directory across runs + assert!( + xvc_root_opt.as_ref().is_none() + || xvc_root_opt + .as_ref() + .map( + |xvc_root| XvcRootInner::find_root(&cli_opts.workdir).unwrap() + == *xvc_root.absolute_path() + ) + .unwrap() ); - let xvc_config_params = XvcConfigInitParams { - current_dir: AbsolutePath::from(&cli_opts.workdir), - include_system_config: !cli_opts.no_system_config, - include_user_config: !cli_opts.no_user_config, - project_config_path: None, - local_config_path: None, - include_environment_config: !cli_opts.no_env_config, - command_line_config: Some(cli_opts.consolidate_config_options()), - default_configuration: default_project_config(true), - }; - - let xvc_root_opt = match load_xvc_root(xvc_config_params) { - Ok(r) => Some(r), - Err(e) => { - e.debug(); - None - } - }; + let term_log_level = get_term_log_level(get_verbosity(&cli_opts)); thread::scope(move |s| { let (output_snd, output_rec) = bounded::>(CHANNEL_BOUND); @@ -417,3 +373,82 @@ pub fn dispatch(cli_opts: cli::XvcCLI) -> Result<()> { Ok(()) } + +/// Dispatch commands to respective functions in the API +/// +/// It sets output verbosity with [XvcCLI::verbosity]. +/// Determines configuration sources by filling [XvcConfigInitParams]. +/// Tries to create an XvcRoot to determine whether we're inside one. +/// Creates two threads: One for running the API function, one for getting strings from output +/// channel. +/// +/// A corresponding function to reuse the same [XvcRoot] object is [test_dispatch]. +/// It doesn't recreate the whole configuration and this prevents errors regarding multiple +/// initializations. +pub fn dispatch(cli_opts: cli::XvcCLI) -> Result<()> { + let verbosity = get_verbosity(&cli_opts); + + let term_log_level = get_term_log_level(verbosity); + + setup_logging( + Some(term_log_level), + if cli_opts.debug { + Some(LevelFilter::Trace) + } else { + None + }, + ); + + let xvc_config_params = get_xvc_config_params(&cli_opts); + + let xvc_root_opt = match load_xvc_root(xvc_config_params) { + Ok(r) => Some(r), + Err(e) => { + e.debug(); + None + } + }; + + dispatch_with_root(cli_opts, xvc_root_opt) +} + +fn get_xvc_config_params(cli_opts: &XvcCLI) -> XvcConfigInitParams { + let xvc_config_params = XvcConfigInitParams { + current_dir: AbsolutePath::from(&cli_opts.workdir), + include_system_config: !cli_opts.no_system_config, + include_user_config: !cli_opts.no_user_config, + project_config_path: None, + local_config_path: None, + include_environment_config: !cli_opts.no_env_config, + command_line_config: Some(cli_opts.consolidate_config_options()), + default_configuration: default_project_config(true), + }; + xvc_config_params +} + +fn get_term_log_level(verbosity: XvcVerbosity) -> LevelFilter { + let term_log_level = match verbosity { + XvcVerbosity::Quiet => LevelFilter::Off, + XvcVerbosity::Default => LevelFilter::Error, + XvcVerbosity::Warn => LevelFilter::Warn, + XvcVerbosity::Info => LevelFilter::Info, + XvcVerbosity::Debug => LevelFilter::Debug, + XvcVerbosity::Trace => LevelFilter::Trace, + }; + term_log_level +} + +fn get_verbosity(cli_opts: &XvcCLI) -> XvcVerbosity { + let verbosity = if cli_opts.quiet { + XvcVerbosity::Quiet + } else { + match cli_opts.verbosity { + 0 => XvcVerbosity::Default, + 1 => XvcVerbosity::Warn, + 2 => XvcVerbosity::Info, + 3 => XvcVerbosity::Debug, + _ => XvcVerbosity::Trace, + } + }; + verbosity +} From c40fc2301353ea1bce19d10f94266df5c45f5e59 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Thu, 9 May 2024 19:45:10 +0300 Subject: [PATCH 006/257] Minor --- lib/src/cli/mod.rs | 12 +++++++----- lib/src/git.rs | 2 +- 2 files changed, 8 insertions(+), 6 deletions(-) diff --git a/lib/src/cli/mod.rs b/lib/src/cli/mod.rs index 327556e93..7b5905898 100644 --- a/lib/src/cli/mod.rs +++ b/lib/src/cli/mod.rs @@ -176,12 +176,12 @@ pub enum XvcSubCommand { } /// Runs the supplied xvc command. -pub fn run(args: &[&str]) -> Result<()> { +pub fn run(args: &[&str]) -> Result> { let cli_options = cli::XvcCLI::from_str_slice(args)?; dispatch(cli_options) } -pub fn dispatch_with_root(cli_opts: cli::XvcCLI, xvc_root_opt: Option) -> Result<()> { +pub fn dispatch_with_root(cli_opts: cli::XvcCLI, xvc_root_opt: &Option) -> Result<()> { // XvcRoot should be kept per repository and shouldn't change directory across runs assert!( xvc_root_opt.as_ref().is_none() @@ -279,7 +279,7 @@ pub fn dispatch_with_root(cli_opts: cli::XvcCLI, xvc_root_opt: Option) if use_git { handle_git_automation( &output_snd, - xvc_root, + &xvc_root, cli_opts.to_branch.as_deref(), &cli_opts.command_string, )?; @@ -385,7 +385,7 @@ pub fn dispatch_with_root(cli_opts: cli::XvcCLI, xvc_root_opt: Option) /// A corresponding function to reuse the same [XvcRoot] object is [test_dispatch]. /// It doesn't recreate the whole configuration and this prevents errors regarding multiple /// initializations. -pub fn dispatch(cli_opts: cli::XvcCLI) -> Result<()> { +pub fn dispatch(cli_opts: cli::XvcCLI) -> Result> { let verbosity = get_verbosity(&cli_opts); let term_log_level = get_term_log_level(verbosity); @@ -409,7 +409,9 @@ pub fn dispatch(cli_opts: cli::XvcCLI) -> Result<()> { } }; - dispatch_with_root(cli_opts, xvc_root_opt) + dispatch_with_root(cli_opts, &xvc_root_opt)?; + + Ok(xvc_root_opt) } fn get_xvc_config_params(cli_opts: &XvcCLI) -> XvcConfigInitParams { diff --git a/lib/src/git.rs b/lib/src/git.rs index 45cd18cf7..283f68a73 100644 --- a/lib/src/git.rs +++ b/lib/src/git.rs @@ -111,7 +111,7 @@ pub fn git_checkout_ref( /// record the last entity counter before commit. pub fn handle_git_automation( output_snd: &XvcOutputSender, - xvc_root: XvcRoot, + xvc_root: &XvcRoot, to_branch: Option<&str>, xvc_cmd: &str, ) -> Result<()> { From 3a9833c04ffe7185f3708bf6368fd5c20defea29 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Thu, 16 May 2024 14:23:45 +0300 Subject: [PATCH 007/257] Update command dispatcher --- config/src/lib.rs | 2 - core/src/check_ignore/mod.rs | 2 +- core/src/types/xvcroot.rs | 16 +++--- lib/src/api.rs | 4 ++ lib/src/cli/mod.rs | 102 ++++++++++++++++++++++++----------- lib/src/git.rs | 3 -- lib/src/lib.rs | 2 +- lib/src/main.rs | 6 ++- 8 files changed, 91 insertions(+), 46 deletions(-) diff --git a/config/src/lib.rs b/config/src/lib.rs index ce0a42c6a..9c264c6ee 100755 --- a/config/src/lib.rs +++ b/config/src/lib.rs @@ -196,8 +196,6 @@ pub struct XvcConfigInitParams { pub struct XvcConfig { /// Current directory. It can be set with xvc -C option pub current_dir: XvcConfigOption, - // /// The root if the command is happen to be run in Xvc directory - // pub xvc_root: XvcConfigOption>, /// Configuration values for each level pub config_maps: Vec, /// The current configuration map, updated cascadingly diff --git a/core/src/check_ignore/mod.rs b/core/src/check_ignore/mod.rs index d4cd002b1..686af5eec 100644 --- a/core/src/check_ignore/mod.rs +++ b/core/src/check_ignore/mod.rs @@ -80,7 +80,7 @@ pub fn cmd_check_ignore( ignore_filename: Some(opts.ignore_filename.clone()), include_dirs: true, }; - let initial_rules = IgnoreRules::try_from_patterns(xvc_root, COMMON_IGNORE_PATTERNS)?; + let initial_rules = IgnoreRules::try_from_patterns(&xvc_root, COMMON_IGNORE_PATTERNS)?; let ignore_rules = build_ignore_rules( initial_rules, current_dir, diff --git a/core/src/types/xvcroot.rs b/core/src/types/xvcroot.rs index 2c3b41172..3b14862b1 100644 --- a/core/src/types/xvcroot.rs +++ b/core/src/types/xvcroot.rs @@ -10,8 +10,6 @@ use std::path::{Path, PathBuf}; use std::sync::Arc; use xvc_ecs::ecs::timestamp; use xvc_ecs::{XvcEntity, XvcEntityGenerator}; -use xvc_logging::watch; -use xvc_logging::{debug, trace}; use xvc_walker::AbsolutePath; use xvc_config::{XvcConfig, XvcConfigInitParams}; @@ -278,11 +276,10 @@ impl XvcRootInner { } Err(Error::CannotFindXvcRoot { path: path.into() }) } -} -impl Drop for XvcRootInner { - /// Saves the entity_generator before dropping - fn drop(&mut self) { + + /// Record the entity generator to the disk + pub fn record(&self) { match self.entity_generator.save(&self.entity_generator_path()) { Ok(_) => (), Err(e) => { @@ -291,3 +288,10 @@ impl Drop for XvcRootInner { } } } + +impl Drop for XvcRootInner { + /// Saves the entity_generator before dropping + fn drop(&mut self) { + self.record() + } +} diff --git a/lib/src/api.rs b/lib/src/api.rs index 38df23975..f6ebdd68b 100644 --- a/lib/src/api.rs +++ b/lib/src/api.rs @@ -24,6 +24,10 @@ pub use xvc_config::XvcConfigOptionSource; pub use xvc_core::AbsolutePath; +pub use xvc_core::XvcRoot; +/// Commands usually receive an optional xvc_root object for the repository +pub type XvcRootOpt = Option; + pub use xvc_file::BringCLI as XvcFileBringCLI; pub use xvc_file::CarryInCLI as XvcFileCarryInCLI; pub use xvc_file::CopyCLI as XvcFileCopyCLI; diff --git a/lib/src/cli/mod.rs b/lib/src/cli/mod.rs index 7b5905898..9ebf4c14e 100644 --- a/lib/src/cli/mod.rs +++ b/lib/src/cli/mod.rs @@ -3,10 +3,12 @@ use std::env::ArgsOs; use std::ffi::OsString; use std::path::PathBuf; +use std::str::FromStr; use crate::git_checkout_ref; use crate::handle_git_automation; use crate::init; +use crate::XvcRootOpt; use clap::Parser; use crossbeam::thread; @@ -125,6 +127,17 @@ impl XvcCLI { }) } + /// Parse the given elements with [clap::Parser::parse_from] and merge them to set + /// [XvcCLI::command_string]. + pub fn from_string_slice(args: &[String]) -> Result { + let command_string = args.join(" "); + let parsed = Self::parse_from(args); + Ok(Self { + command_string, + ..parsed + }) + } + /// Parse the command line from the result of [`std::env::args_os`]. /// This updates [XvcCLI::command_string] with the command line. pub fn from_args_os(args_os: ArgsOs) -> Result { @@ -155,6 +168,22 @@ impl XvcCLI { } } +// Implement FromStr for XvcCLI + +impl FromStr for XvcCLI { + type Err = Error; + + fn from_str(s: &str) -> Result { + let command_string = s.to_owned(); + let args: Vec = s.split(' ').map(|a| a.trim().to_owned()).collect(); + let parsed = Self::parse_from(args); + Ok(Self { + command_string, + ..parsed + }) + } +} + /// Xvc subcommands #[derive(Debug, Parser)] #[command(rename_all = "kebab-case")] @@ -176,12 +205,13 @@ pub enum XvcSubCommand { } /// Runs the supplied xvc command. -pub fn run(args: &[&str]) -> Result> { +pub fn run(args: &[&str]) -> Result { let cli_options = cli::XvcCLI::from_str_slice(args)?; dispatch(cli_options) } -pub fn dispatch_with_root(cli_opts: cli::XvcCLI, xvc_root_opt: &Option) -> Result<()> { +pub fn dispatch_with_root(cli_opts: cli::XvcCLI, xvc_root_opt: &XvcRootOpt) -> Result { + // XvcRoot should be kept per repository and shouldn't change directory across runs assert!( xvc_root_opt.as_ref().is_none() @@ -196,6 +226,8 @@ pub fn dispatch_with_root(cli_opts: cli::XvcCLI, xvc_root_opt: &Option) let term_log_level = get_term_log_level(get_verbosity(&cli_opts)); + let xvc_root_opt = xvc_root_opt.clone(); + thread::scope(move |s| { let (output_snd, output_rec) = bounded::>(CHANNEL_BOUND); @@ -271,76 +303,84 @@ pub fn dispatch_with_root(cli_opts: cli::XvcCLI, xvc_root_opt: &Option) ); } } - let command_thread = s.spawn(move |_| -> Result<()> { - match cli_opts.command { + let xvc_root_opt_res = s.spawn(move |_| -> Result { + let xvc_root_opt = match cli_opts.command { XvcSubCommand::Init(opts) => { - let use_git = !opts.no_git; let xvc_root = init::run(xvc_root_opt.as_ref(), opts)?; - if use_git { - handle_git_automation( - &output_snd, - &xvc_root, - cli_opts.to_branch.as_deref(), - &cli_opts.command_string, - )?; - } - Result::Ok(()) + Some(xvc_root) } - XvcSubCommand::Aliases(opts) => Ok(aliases::run(&output_snd, opts)?), + XvcSubCommand::Aliases(opts) => { + aliases::run(&output_snd, opts)?; + xvc_root_opt + }, // following commands can only be run inside a repository - XvcSubCommand::Root(opts) => Ok(root::run( + XvcSubCommand::Root(opts) => { root::run( &output_snd, xvc_root_opt .as_ref() .ok_or_else(|| Error::RequiresXvcRepository)?, opts, - )?), + )?; + xvc_root_opt + }, + XvcSubCommand::File(opts) => { - Ok(file::run(&output_snd, xvc_root_opt.as_ref(), opts)?) + file::run(&output_snd, xvc_root_opt.as_ref(), opts)?; + xvc_root_opt } XvcSubCommand::Pipeline(opts) => { let stdin = io::stdin(); let input = stdin.lock(); - Ok(pipeline::cmd_pipeline( + pipeline::cmd_pipeline( input, &output_snd, xvc_root_opt.as_ref().ok_or(Error::RequiresXvcRepository)?, opts, - )?) + )?; + + xvc_root_opt + } XvcSubCommand::CheckIgnore(opts) => { let stdin = io::stdin(); let input = stdin.lock(); - Ok(check_ignore::cmd_check_ignore( + check_ignore::cmd_check_ignore( input, &output_snd, xvc_root_opt.as_ref().ok_or(Error::RequiresXvcRepository)?, opts, - )?) + )?; + + xvc_root_opt + } XvcSubCommand::Storage(opts) => { let stdin = io::stdin(); let input = stdin.lock(); - Ok(storage::cmd_storage( + storage::cmd_storage( input, &output_snd, xvc_root_opt.as_ref().ok_or(Error::RequiresXvcRepository)?, opts, - )?) + )?; + + xvc_root_opt } - }?; + + }; watch!("Before handle_git_automation"); + match xvc_root_opt { - Some(xvc_root) => { - watch!(&cli_opts.command_string); + Some(ref xvc_root) => { + xvc_root.record(); if cli_opts.skip_git { debug!(output_snd, "Skipping Git operations"); } else { @@ -359,10 +399,10 @@ pub fn dispatch_with_root(cli_opts: cli::XvcCLI, xvc_root_opt: &Option) ); } } - Ok(()) + Ok(xvc_root_opt) }); - match command_thread.join().unwrap() { + match xvc_root_opt_res.join().unwrap() { Ok(_) => debug!(output_snd_clone, "Command completed successfully."), Err(e) => error!(output_snd_clone, "{}", e), } @@ -371,7 +411,7 @@ pub fn dispatch_with_root(cli_opts: cli::XvcCLI, xvc_root_opt: &Option) }) .unwrap(); - Ok(()) + Ok(xvc_root_opt) } /// Dispatch commands to respective functions in the API @@ -385,7 +425,7 @@ pub fn dispatch_with_root(cli_opts: cli::XvcCLI, xvc_root_opt: &Option) /// A corresponding function to reuse the same [XvcRoot] object is [test_dispatch]. /// It doesn't recreate the whole configuration and this prevents errors regarding multiple /// initializations. -pub fn dispatch(cli_opts: cli::XvcCLI) -> Result> { +pub fn dispatch(cli_opts: cli::XvcCLI) -> Result { let verbosity = get_verbosity(&cli_opts); let term_log_level = get_term_log_level(verbosity); diff --git a/lib/src/git.rs b/lib/src/git.rs index 283f68a73..c45a42979 100644 --- a/lib/src/git.rs +++ b/lib/src/git.rs @@ -125,9 +125,6 @@ pub fn handle_git_automation( let xvc_dir = xvc_root.xvc_dir().clone(); let xvc_dir_str = xvc_dir.to_str().unwrap(); - // we drop here to record the final state - drop(xvc_root); - if use_git { if auto_commit { git_auto_commit( diff --git a/lib/src/lib.rs b/lib/src/lib.rs index 4dfb4e96c..bbe8c8680 100755 --- a/lib/src/lib.rs +++ b/lib/src/lib.rs @@ -11,7 +11,7 @@ pub mod api; pub use api::*; /// Adds `xvc` as the first elements to `args` and calls [cli::dispatch] after parsing them. -pub fn dispatch(args: Vec<&str>) -> Result<()> { +pub fn dispatch(args: Vec<&str>) -> Result { let args_with_binary_name = if !args.is_empty() && args[0] != "xvc" { vec!["xvc"].into_iter().chain(args).collect() } else { diff --git a/lib/src/main.rs b/lib/src/main.rs index 8f75088fa..7cab793ef 100755 --- a/lib/src/main.rs +++ b/lib/src/main.rs @@ -1,11 +1,13 @@ #![warn(missing_docs)] //! The entry point for xvc cli -use xvc::error::Result; +use xvc::{error::Result}; /// The entry point of the `xvc` cli. /// /// It parses the command line arguments [xvc::cli::XvcCLI] and calls [xvc::cli::dispatch] fn main() -> Result<()> { let cli_opts = xvc::cli::XvcCLI::from_args_os(std::env::args_os())?; - xvc::cli::dispatch(cli_opts) + xvc::cli::dispatch(cli_opts)?; + + Ok(()) } From dfd5e96bec7e05f62ca2d56fe69dc7dee5ea5a2a Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Fri, 17 May 2024 13:53:19 +0300 Subject: [PATCH 008/257] possible to pass xvc_root to dispatcher now --- config/Cargo.toml | 6 +++--- core/Cargo.toml | 12 ++++++------ ecs/Cargo.toml | 4 ++-- file/Cargo.toml | 16 ++++++++-------- lib/Cargo.toml | 20 ++++++++++---------- lib/src/cli/mod.rs | 18 ++++++++++-------- logging/Cargo.toml | 2 +- pipeline/Cargo.toml | 16 ++++++++-------- storage/Cargo.toml | 14 +++++++------- test_helper/Cargo.toml | 4 ++-- walker/Cargo.toml | 6 +++--- workflow_tests/Cargo.toml | 22 +++++++++++----------- workflow_tests/src/lib.rs | 4 ++-- workflow_tests/src/main.rs | 3 ++- 14 files changed, 75 insertions(+), 72 deletions(-) diff --git a/config/Cargo.toml b/config/Cargo.toml index a581613f5..f12ed2927 100644 --- a/config/Cargo.toml +++ b/config/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-config" -version = "0.6.8-alpha.0" +version = "0.6.8-alpha.1" edition = "2021" description = "Xvc configuration management" authors = ["Emre Şahin "] @@ -20,8 +20,8 @@ debug = true [dependencies] -xvc-logging = { version = "0.6.8-alpha.0", path = "../logging" } -xvc-walker = { version = "0.6.8-alpha.0", path = "../walker" } +xvc-logging = { version = "0.6.8-alpha.1", path = "../logging" } +xvc-walker = { version = "0.6.8-alpha.1", path = "../walker" } ## Cli and config diff --git a/core/Cargo.toml b/core/Cargo.toml index 1dc6eb11f..3fde8c0b1 100644 --- a/core/Cargo.toml +++ b/core/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-core" -version = "0.6.8-alpha.0" +version = "0.6.8-alpha.1" edition = "2021" description = "Xvc core for common elements for all commands" authors = ["Emre Şahin "] @@ -19,10 +19,10 @@ crate-type = ["rlib"] debug = true [dependencies] -xvc-config = { version = "0.6.8-alpha.0", path = "../config" } -xvc-logging = { version = "0.6.8-alpha.0", path = "../logging" } -xvc-ecs = { version = "0.6.8-alpha.0", path = "../ecs" } -xvc-walker = { version = "0.6.8-alpha.0", path = "../walker" } +xvc-config = { version = "0.6.8-alpha.1", path = "../config" } +xvc-logging = { version = "0.6.8-alpha.1", path = "../logging" } +xvc-ecs = { version = "0.6.8-alpha.1", path = "../ecs" } +xvc-walker = { version = "0.6.8-alpha.1", path = "../walker" } ## Cli and config clap = { version = "^4.4", features = ["derive"] } @@ -82,6 +82,6 @@ derive_more = "^0.99" itertools = "^0.12" [dev-dependencies] -xvc-test-helper = { version = "0.6.8-alpha.0", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.8-alpha.1", path = "../test_helper/" } proptest = "^1.4" test-case = "^3.3" diff --git a/ecs/Cargo.toml b/ecs/Cargo.toml index 8546b7f37..cc39458b1 100644 --- a/ecs/Cargo.toml +++ b/ecs/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-ecs" -version = "0.6.8-alpha.0" +version = "0.6.8-alpha.1" edition = "2021" description = "Entity-Component System for Xvc" authors = ["Emre Şahin "] @@ -19,7 +19,7 @@ crate-type = ["rlib"] debug = true [dependencies] -xvc-logging = { version = "0.6.8-alpha.0", path = "../logging" } +xvc-logging = { version = "0.6.8-alpha.1", path = "../logging" } ## Serialization serde = { version = "^1.0", features = ["derive"] } diff --git a/file/Cargo.toml b/file/Cargo.toml index a521f9dfd..cc08d37f7 100644 --- a/file/Cargo.toml +++ b/file/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-file" -version = "0.6.8-alpha.0" +version = "0.6.8-alpha.1" edition = "2021" description = "File tracking, versioning, upload and download functions for Xvc" authors = ["Emre Şahin "] @@ -24,12 +24,12 @@ bench = true debug = true [dependencies] -xvc-logging = { version = "0.6.8-alpha.0", path = "../logging" } -xvc-config = { version = "0.6.8-alpha.0", path = "../config" } -xvc-core = { version = "0.6.8-alpha.0", path = "../core" } -xvc-ecs = { version = "0.6.8-alpha.0", path = "../ecs" } -xvc-walker = { version = "0.6.8-alpha.0", path = "../walker" } -xvc-storage = { version = "0.6.8-alpha.0", path = "../storage" } +xvc-logging = { version = "0.6.8-alpha.1", path = "../logging" } +xvc-config = { version = "0.6.8-alpha.1", path = "../config" } +xvc-core = { version = "0.6.8-alpha.1", path = "../core" } +xvc-ecs = { version = "0.6.8-alpha.1", path = "../ecs" } +xvc-walker = { version = "0.6.8-alpha.1", path = "../walker" } +xvc-storage = { version = "0.6.8-alpha.1", path = "../storage" } ## Cli and config @@ -94,5 +94,5 @@ parse-size = "^1.0" [dev-dependencies] -xvc-test-helper = { version = "0.6.8-alpha.0", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.8-alpha.1", path = "../test_helper/" } shellfn = "^0.1" diff --git a/lib/Cargo.toml b/lib/Cargo.toml index 00810fab5..4646a0103 100644 --- a/lib/Cargo.toml +++ b/lib/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc" -version = "0.6.8-alpha.0" +version = "0.6.8-alpha.1" edition = "2021" description = "An MLOps tool to manage data files and pipelines on top of Git" authors = ["Emre Şahin "] @@ -23,14 +23,14 @@ path = "src/main.rs" debug = true [dependencies] -xvc-config = { version = "0.6.8-alpha.0", path = "../config" } -xvc-core = { version = "0.6.8-alpha.0", path = "../core" } -xvc-logging = { version = "0.6.8-alpha.0", path = "../logging" } -xvc-ecs = { version = "0.6.8-alpha.0", path = "../ecs" } -xvc-file = { version = "0.6.8-alpha.0", path = "../file" } -xvc-pipeline = { version = "0.6.8-alpha.0", path = "../pipeline" } -xvc-walker = { version = "0.6.8-alpha.0", path = "../walker" } -xvc-storage = { version = "0.6.8-alpha.0", path = "../storage" } +xvc-config = { version = "0.6.8-alpha.1", path = "../config" } +xvc-core = { version = "0.6.8-alpha.1", path = "../core" } +xvc-logging = { version = "0.6.8-alpha.1", path = "../logging" } +xvc-ecs = { version = "0.6.8-alpha.1", path = "../ecs" } +xvc-file = { version = "0.6.8-alpha.1", path = "../file" } +xvc-pipeline = { version = "0.6.8-alpha.1", path = "../pipeline" } +xvc-walker = { version = "0.6.8-alpha.1", path = "../walker" } +xvc-storage = { version = "0.6.8-alpha.1", path = "../storage" } ## Cli and config @@ -113,4 +113,4 @@ jwalk = "^0.8" proptest = "^1.4" shellfn = "^0.1" test-case = "^3.3" -xvc-test-helper = { version = "0.6.8-alpha.0", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.8-alpha.1", path = "../test_helper/" } diff --git a/lib/src/cli/mod.rs b/lib/src/cli/mod.rs index 9ebf4c14e..9e468eafd 100644 --- a/lib/src/cli/mod.rs +++ b/lib/src/cli/mod.rs @@ -210,7 +210,7 @@ pub fn run(args: &[&str]) -> Result { dispatch(cli_options) } -pub fn dispatch_with_root(cli_opts: cli::XvcCLI, xvc_root_opt: &XvcRootOpt) -> Result { +pub fn dispatch_with_root(cli_opts: cli::XvcCLI, xvc_root_opt: XvcRootOpt) -> Result { // XvcRoot should be kept per repository and shouldn't change directory across runs assert!( @@ -226,9 +226,7 @@ pub fn dispatch_with_root(cli_opts: cli::XvcCLI, xvc_root_opt: &XvcRootOpt) -> R let term_log_level = get_term_log_level(get_verbosity(&cli_opts)); - let xvc_root_opt = xvc_root_opt.clone(); - - thread::scope(move |s| { + let xvc_root_opt = thread::scope(move |s| { let (output_snd, output_rec) = bounded::>(CHANNEL_BOUND); let output_snd_clone = output_snd.clone(); @@ -402,16 +400,21 @@ pub fn dispatch_with_root(cli_opts: cli::XvcCLI, xvc_root_opt: &XvcRootOpt) -> R Ok(xvc_root_opt) }); - match xvc_root_opt_res.join().unwrap() { + let xvc_root_opt = xvc_root_opt_res.join().unwrap(); + match &xvc_root_opt { Ok(_) => debug!(output_snd_clone, "Command completed successfully."), Err(e) => error!(output_snd_clone, "{}", e), } output_snd_clone.send(None).unwrap(); output_thread.join().unwrap(); + + xvc_root_opt }) .unwrap(); - Ok(xvc_root_opt) + xvc_root_opt + + } /// Dispatch commands to respective functions in the API @@ -449,9 +452,8 @@ pub fn dispatch(cli_opts: cli::XvcCLI) -> Result { } }; - dispatch_with_root(cli_opts, &xvc_root_opt)?; + dispatch_with_root(cli_opts, xvc_root_opt) - Ok(xvc_root_opt) } fn get_xvc_config_params(cli_opts: &XvcCLI) -> XvcConfigInitParams { diff --git a/logging/Cargo.toml b/logging/Cargo.toml index b2668c421..ad78df60d 100644 --- a/logging/Cargo.toml +++ b/logging/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-logging" -version = "0.6.8-alpha.0" +version = "0.6.8-alpha.1" edition = "2021" description = "Logging crate for Xvc" authors = ["Emre Şahin "] diff --git a/pipeline/Cargo.toml b/pipeline/Cargo.toml index 0b495601e..bb8622df9 100644 --- a/pipeline/Cargo.toml +++ b/pipeline/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-pipeline" -version = "0.6.8-alpha.0" +version = "0.6.8-alpha.1" edition = "2021" description = "Xvc data pipeline management" authors = ["Emre Şahin "] @@ -19,12 +19,12 @@ debug = true [dependencies] -xvc-config = { version = "0.6.8-alpha.0", path = "../config" } -xvc-core = { version = "0.6.8-alpha.0", path = "../core" } -xvc-ecs = { version = "0.6.8-alpha.0", path = "../ecs" } -xvc-logging = { version = "0.6.8-alpha.0", path = "../logging" } -xvc-walker = { version = "0.6.8-alpha.0", path = "../walker" } -xvc-file = { version = "0.6.8-alpha.0", path = "../file" } +xvc-config = { version = "0.6.8-alpha.1", path = "../config" } +xvc-core = { version = "0.6.8-alpha.1", path = "../core" } +xvc-ecs = { version = "0.6.8-alpha.1", path = "../ecs" } +xvc-logging = { version = "0.6.8-alpha.1", path = "../logging" } +xvc-walker = { version = "0.6.8-alpha.1", path = "../walker" } +xvc-file = { version = "0.6.8-alpha.1", path = "../file" } ## Cli and config clap = { version = "^4.4", features = ["derive"] } @@ -92,5 +92,5 @@ itertools = "^0.12" derive_more = "^0.99" [dev-dependencies] -xvc-test-helper = { version = "0.6.8-alpha.0", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.8-alpha.1", path = "../test_helper/" } test-case = "^3.3" diff --git a/storage/Cargo.toml b/storage/Cargo.toml index 2a99d631c..1c475eb28 100644 --- a/storage/Cargo.toml +++ b/storage/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-storage" -version = "0.6.8-alpha.0" +version = "0.6.8-alpha.1" edition = "2021" description = "Xvc remote (and local) storage management" authors = ["Emre Şahin "] @@ -20,11 +20,11 @@ debug = true [dependencies] -xvc-logging = { version = "0.6.8-alpha.0", path = "../logging" } -xvc-config = { version = "0.6.8-alpha.0", path = "../config" } -xvc-core = { version = "0.6.8-alpha.0", path = "../core" } -xvc-ecs = { version = "0.6.8-alpha.0", path = "../ecs" } -xvc-walker = { version = "0.6.8-alpha.0", path = "../walker" } +xvc-logging = { version = "0.6.8-alpha.1", path = "../logging" } +xvc-config = { version = "0.6.8-alpha.1", path = "../config" } +xvc-core = { version = "0.6.8-alpha.1", path = "../core" } +xvc-ecs = { version = "0.6.8-alpha.1", path = "../ecs" } +xvc-walker = { version = "0.6.8-alpha.1", path = "../walker" } ## Cli and config clap = { version = "^4.4", features = ["derive"] } @@ -109,7 +109,7 @@ digital-ocean = ["async"] [dev-dependencies] -xvc-test-helper = { version = "0.6.8-alpha.0", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.8-alpha.1", path = "../test_helper/" } shellfn = "^0.1" [package.metadata.cargo-udeps.ignore] diff --git a/test_helper/Cargo.toml b/test_helper/Cargo.toml index f7cd31b8f..be249be60 100644 --- a/test_helper/Cargo.toml +++ b/test_helper/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-test-helper" -version = "0.6.8-alpha.0" +version = "0.6.8-alpha.1" edition = "2021" description = "Unit test helper functions for Xvc" authors = ["Emre Şahin "] @@ -23,7 +23,7 @@ debug = true # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html [dependencies] -xvc-logging = { version = "0.6.8-alpha.0", path = "../logging/" } +xvc-logging = { version = "0.6.8-alpha.1", path = "../logging/" } rand = "^0.8" log = "^0.4" diff --git a/walker/Cargo.toml b/walker/Cargo.toml index e0c377062..f99ce2d2d 100644 --- a/walker/Cargo.toml +++ b/walker/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-walker" -version = "0.6.8-alpha.0" +version = "0.6.8-alpha.1" edition = "2021" description = "Xvc parallel file system walker with ignore features" authors = ["Emre Şahin "] @@ -19,7 +19,7 @@ crate-type = ["rlib"] debug = true [dependencies] -xvc-logging = { version = "0.6.8-alpha.0", path = "../logging" } +xvc-logging = { version = "0.6.8-alpha.1", path = "../logging" } globset = "^0.4" ## Parallelization @@ -41,7 +41,7 @@ itertools = "^0.12" regex = "^1.10" [dev-dependencies] -xvc-test-helper = { path = "../test_helper/", version = "0.6.8-alpha.0" } +xvc-test-helper = { path = "../test_helper/", version = "0.6.8-alpha.1" } test-case = "^3.3" [package.metadata.cargo-udeps.ignore] diff --git a/workflow_tests/Cargo.toml b/workflow_tests/Cargo.toml index 9110d6833..22a59a899 100644 --- a/workflow_tests/Cargo.toml +++ b/workflow_tests/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-workflow-tests" -version = "0.6.8-alpha.0" +version = "0.6.8-alpha.1" edition = "2021" description = "Integration test suite for Xvc" authors = ["Emre Şahin "] @@ -23,15 +23,15 @@ debug = true [dependencies] -xvc = { version = "0.6.8-alpha.0", path = "../lib" } -xvc-config = { version = "0.6.8-alpha.0", path = "../config" } -xvc-core = { version = "0.6.8-alpha.0", path = "../core" } -xvc-logging = { version = "0.6.8-alpha.0", path = "../logging" } -xvc-ecs = { version = "0.6.8-alpha.0", path = "../ecs" } -xvc-file = { version = "0.6.8-alpha.0", path = "../file" } -xvc-pipeline = { version = "0.6.8-alpha.0", path = "../pipeline" } -xvc-walker = { version = "0.6.8-alpha.0", path = "../walker" } -xvc-storage = { version = "0.6.8-alpha.0", path = "../storage" } +xvc = { version = "0.6.8-alpha.1", path = "../lib" } +xvc-config = { version = "0.6.8-alpha.1", path = "../config" } +xvc-core = { version = "0.6.8-alpha.1", path = "../core" } +xvc-logging = { version = "0.6.8-alpha.1", path = "../logging" } +xvc-ecs = { version = "0.6.8-alpha.1", path = "../ecs" } +xvc-file = { version = "0.6.8-alpha.1", path = "../file" } +xvc-pipeline = { version = "0.6.8-alpha.1", path = "../pipeline" } +xvc-walker = { version = "0.6.8-alpha.1", path = "../walker" } +xvc-storage = { version = "0.6.8-alpha.1", path = "../storage" } ## packages for testing clap = { version = "^4.4", features = ["derive"] } @@ -70,7 +70,7 @@ proptest = "^1.4" test-case = "^3.3" globset = "^0.4" escargot = "^0.5" -xvc-test-helper = { version = "0.6.8-alpha.0", path = "../test_helper" } +xvc-test-helper = { version = "0.6.8-alpha.1", path = "../test_helper" } shellfn = "^0.1" jwalk = "^0.8" anyhow = "^1.0" diff --git a/workflow_tests/src/lib.rs b/workflow_tests/src/lib.rs index 9b05d9691..94fa246fb 100755 --- a/workflow_tests/src/lib.rs +++ b/workflow_tests/src/lib.rs @@ -9,11 +9,11 @@ pub use xvc_pipeline as pipeline; pub use xvc_logging::watch; -use xvc::{cli::XvcCLI, error::Result}; +use xvc::{cli::XvcCLI, error::Result, XvcRootOpt}; /// Run xvc in another process with `args` in `xvc_root_opt`. /// /// See [xvc::cli::test_dispatch]. -pub fn dispatch(cli_opts: XvcCLI) -> Result<()> { +pub fn dispatch(cli_opts: XvcCLI) -> Result { xvc::cli::dispatch(cli_opts) } diff --git a/workflow_tests/src/main.rs b/workflow_tests/src/main.rs index be0e58400..ff4c0cd24 100755 --- a/workflow_tests/src/main.rs +++ b/workflow_tests/src/main.rs @@ -9,5 +9,6 @@ use xvc::error::Result; /// fn main() -> Result<()> { let cli_opts = xvc::cli::XvcCLI::from_args_os(std::env::args_os())?; - xvc::cli::dispatch(cli_opts) + xvc::cli::dispatch(cli_opts)?; + Ok(()) } From ce5c6b258fd2765069f95a662a7352de201ca9fe Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Fri, 17 May 2024 13:54:35 +0300 Subject: [PATCH 009/257] Update license --- lib/Cargo.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/Cargo.toml b/lib/Cargo.toml index 4646a0103..ae1c5c4e6 100644 --- a/lib/Cargo.toml +++ b/lib/Cargo.toml @@ -4,7 +4,7 @@ version = "0.6.8-alpha.1" edition = "2021" description = "An MLOps tool to manage data files and pipelines on top of Git" authors = ["Emre Şahin "] -license = "GPL-2.0" +license = "GPL-3.0" homepage = "https://xvc.dev" repository = "https://github.com/iesahin/xvc" categories = ["command-line-utilities", "development-tools"] From 69389903465a5da08653b73fffd4b259cb7b8f17 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Mon, 27 May 2024 10:54:28 +0300 Subject: [PATCH 010/257] removed dbg statements --- config/src/lib.rs | 6 ++++-- core/src/types/xvcroot.rs | 16 ++++------------ file/src/list/mod.rs | 1 - 3 files changed, 8 insertions(+), 15 deletions(-) diff --git a/config/src/lib.rs b/config/src/lib.rs index 9c264c6ee..80693779a 100755 --- a/config/src/lib.rs +++ b/config/src/lib.rs @@ -171,10 +171,12 @@ pub struct XvcConfigInitParams { /// If `true`, it's read from [USER_CONFIG_DIRS]. pub include_user_config: bool, /// Where should we load the project's (public) configuration? - /// If `None`, it's ignored. + /// It's loaded in [XvcRootInner::new] + /// TODO: Add a option to ignore this pub project_config_path: Option, /// Where should we load the project's (private) configuration? - /// If `None`, it's ignored. + /// It's loaded in [XvcRootInner::new] + /// TODO: Add a option to ignore this pub local_config_path: Option, /// Should we include configuration from the environment. /// If `true`, look for all variables in the form diff --git a/core/src/types/xvcroot.rs b/core/src/types/xvcroot.rs index 3b14862b1..a0705f39f 100644 --- a/core/src/types/xvcroot.rs +++ b/core/src/types/xvcroot.rs @@ -10,6 +10,7 @@ use std::path::{Path, PathBuf}; use std::sync::Arc; use xvc_ecs::ecs::timestamp; use xvc_ecs::{XvcEntity, XvcEntityGenerator}; +use xvc_logging::watch; use xvc_walker::AbsolutePath; use xvc_config::{XvcConfig, XvcConfigInitParams}; @@ -71,15 +72,12 @@ impl Deref for XvcRootInner { pub fn load_xvc_root(config_opts: XvcConfigInitParams) -> Result { let path = config_opts.current_dir.as_ref(); - dbg!(path); - match XvcRootInner::find_root(path) { Ok(absolute_path) => { - dbg!(&absolute_path); Ok(Arc::new(XvcRootInner::new(absolute_path, config_opts)?)) } Err(e) => { - dbg!(&e); + watch!(&e); Err(e) } } @@ -154,11 +152,8 @@ impl XvcRootInner { /// other locations. `config_opts` can determine which configuration files to read pub fn new(absolute_path: AbsolutePath, config_opts: XvcConfigInitParams) -> Result { let xvc_dir = absolute_path.join(XvcRootInner::XVC_DIR); - dbg!(&xvc_dir); let local_config_path = xvc_dir.join(XvcRootInner::LOCAL_CONFIG_PATH); - dbg!(&local_config_path); let project_config_path = xvc_dir.join(XvcRootInner::PROJECT_CONFIG_PATH); - dbg!(&project_config_path); let config_opts = XvcConfigInitParams { project_config_path: Some(project_config_path.clone()), local_config_path: Some(local_config_path.clone()), @@ -259,18 +254,15 @@ impl XvcRootInner { /// Finds the root of the xvc repository by looking for the .xvc directory /// in parents of a given path. pub fn find_root(path: &Path) -> Result { - dbg!(path); let abs_path = PathBuf::from(path) .canonicalize() .expect("Cannot canonicalize the path. Possible symlink loop."); - dbg!(&abs_path); for parent in abs_path.ancestors() { - dbg!(parent); let xvc_candidate = parent.join(XVC_DIR); - dbg!(&xvc_candidate); + watch!(xvc_candidate); if parent.join(XVC_DIR).is_dir() { - dbg!("XVC DIR: {:?}", parent); + watch!(parent); return Ok(parent.into()); } } diff --git a/file/src/list/mod.rs b/file/src/list/mod.rs index b577ddae6..e4f45aa2c 100644 --- a/file/src/list/mod.rs +++ b/file/src/list/mod.rs @@ -127,7 +127,6 @@ struct ListRow { } impl ListRow { - /// fn new(path_prefix: &Path, path_match: PathMatch) -> Result { let actual_file_type = String::from(if let Some(actual_metadata) = path_match.actual_metadata { From 78c69c3a42b620adb2fac73fef56d47ce10d690b Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Mon, 10 Jun 2024 11:22:56 +0300 Subject: [PATCH 011/257] change panics to errors --- storage/src/storage/mod.rs | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/storage/src/storage/mod.rs b/storage/src/storage/mod.rs index 82fc059c6..036b46ee6 100644 --- a/storage/src/storage/mod.rs +++ b/storage/src/storage/mod.rs @@ -32,7 +32,7 @@ pub use local::XvcLocalStorage; use serde::{Deserialize, Serialize}; use tempfile::TempDir; use uuid::Uuid; -use xvc_logging::{panic, watch, XvcOutputSender}; +use xvc_logging::{error, panic, watch, XvcOutputSender}; use xvc_walker::AbsolutePath; use crate::{Error, Result, StorageIdentifier}; @@ -495,10 +495,10 @@ pub fn get_storage_record( }); if remote_store.is_empty() { - panic!(output_snd, "Cannot find remote {}", identifier); + error!(output_snd, "Cannot find remote {}", identifier); } if remote_store.len() > 1 { - panic!(output_snd, "Ambiguous remote identifier: {}", identifier); + error!(output_snd, "Ambiguous remote identifier: {}", identifier); } let (_, remote) = From 35d777321457191c96ee63def22068261e50811c Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Tue, 11 Jun 2024 10:04:09 +0300 Subject: [PATCH 012/257] added some fixmes --- storage/src/lib.rs | 6 ++---- storage/src/storage/async_common.rs | 3 +++ storage/src/storage/mod.rs | 10 +++++----- 3 files changed, 10 insertions(+), 9 deletions(-) diff --git a/storage/src/lib.rs b/storage/src/lib.rs index b8c339ca6..f59a06e90 100755 --- a/storage/src/lib.rs +++ b/storage/src/lib.rs @@ -125,12 +125,10 @@ pub enum StorageNewSubCommand { storage_dir: Option, }, - /// Add a new rsync storage - /// - /// Uses rsync in separate processes to communicate with the server. + /// Add a new rsync storages /// + /// Uses rsync in separate processes to communicate. /// This can be used when you already have an SSH/Rsync connection. - /// /// It doesn't prompt for any passwords. The connection must be set up with ssh keys beforehand. #[command()] Rsync { diff --git a/storage/src/storage/async_common.rs b/storage/src/storage/async_common.rs index 3f7f6e66e..be8dcefdb 100644 --- a/storage/src/storage/async_common.rs +++ b/storage/src/storage/async_common.rs @@ -31,11 +31,13 @@ use super::XvcStorageTempDir; use super::XVC_STORAGE_GUID_FILENAME; pub trait XvcS3StorageOperations { + // FIXME: Rename to storage_prefix fn remote_prefix(&self) -> String; fn guid(&self) -> &XvcStorageGuid; fn get_bucket(&self) -> Result; fn credentials(&self) -> Result; fn bucket_name(&self) -> String; + // FIXME: Rename to build_storage_path fn build_remote_path(&self, cache_path: &XvcCachePath) -> XvcStoragePath { XvcStoragePath::from(format!( "{}/{}/{}", @@ -46,6 +48,7 @@ pub trait XvcS3StorageOperations { } fn region(&self) -> String; + // FIXME: Rename to storage guid async fn write_remote_guid(&self) -> Result<()> { let guid_str = self.guid().to_string(); let guid_bytes = guid_str.as_bytes(); diff --git a/storage/src/storage/mod.rs b/storage/src/storage/mod.rs index 82fc059c6..300fa85f8 100644 --- a/storage/src/storage/mod.rs +++ b/storage/src/storage/mod.rs @@ -73,7 +73,7 @@ pub enum XvcStorage { #[cfg(feature = "digital-ocean")] DigitalOcean(digital_ocean::XvcDigitalOceanStorage), } -persist!(XvcStorage, "remote"); +persist!(XvcStorage, "storage"); impl Display for XvcStorage { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { @@ -111,7 +111,7 @@ impl Display for XvcStorage { XvcStorage::S3(s3r) => write!( f, "S3: {}\t{}\t{}.{}/{}", - s3r.name, s3r.guid, s3r.region, s3r.bucket_name, s3r.storage_prefix + e3r.name, s3r.guid, s3r.region, s3r.bucket_name, s3r.storage_prefix ), #[cfg(feature = "minio")] XvcStorage::Minio(mr) => write!( @@ -148,7 +148,7 @@ impl Display for XvcStorage { } /// All storages implement this trait. xvc storage new and xvc file send / bring / remove -/// commands use this trait to communicate with the remotes. +/// commands use this trait to communicate with the storages. pub trait XvcStorageOperations { /// The init operation is creates a directory with the "short guid" of the Xvc repository and /// adds a .xvc-guid file with the guid of the storage. @@ -375,7 +375,7 @@ impl XvcStorageTempDir { /// It uses [RelativePathBuf] internally. #[derive(Clone, Debug, PartialOrd, Ord, PartialEq, Eq, Serialize, Deserialize, Display)] pub struct XvcStoragePath(RelativePathBuf); -persist!(XvcStoragePath, "remote-path"); +persist!(XvcStoragePath, "storage-path"); impl From for XvcStoragePath { fn from(p: String) -> Self { @@ -397,7 +397,7 @@ impl AsRef for XvcStoragePath { } impl XvcStoragePath { - /// The remote path of a cache path is like {guid}/{cache-path} + /// The storage path of a cache path is like {guid}/{cache-path} /// ⚠️ The separator between {guid} and {cache-path} is always / pub fn new(xvc_root: &XvcRoot, local: &XvcCachePath) -> Self { let guid = xvc_root.config().guid().unwrap(); From 5fbfdcbf5752d899122e38ae4b95f7f959593719 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Fri, 14 Jun 2024 12:14:05 +0300 Subject: [PATCH 013/257] Rename remote to storage in various files --- README.md | 6 +- book/src/ref/xvc-file-send.md | 2 +- book/src/ref/xvc-file-share.md | 4 +- config/Cargo.toml | 6 +- core/Cargo.toml | 12 +- ecs/Cargo.toml | 4 +- file/Cargo.toml | 16 +- file/src/send/mod.rs | 4 +- file/src/share/mod.rs | 8 +- lib/Cargo.toml | 20 +- logging/Cargo.toml | 2 +- pipeline/Cargo.toml | 16 +- storage/Cargo.toml | 16 +- storage/src/error.rs | 2 +- storage/src/lib.rs | 2 +- storage/src/storage/async_common.rs | 42 +-- storage/src/storage/digital_ocean.rs | 38 +-- storage/src/storage/gcs.rs | 320 ++------------------ storage/src/storage/local.rs | 50 +-- storage/src/storage/minio.rs | 247 +-------------- storage/src/storage/mod.rs | 30 +- storage/src/storage/r2.rs | 34 +-- storage/src/storage/rsync.rs | 20 +- storage/src/storage/s3.rs | 26 +- storage/src/storage/wasabi.rs | 18 +- test_helper/Cargo.toml | 4 +- walker/Cargo.toml | 6 +- workflow_tests/Cargo.toml | 22 +- workflow_tests/tests/test_init_1.rs | 2 +- workflow_tests/tests/test_storage_new_s3.rs | 2 +- 30 files changed, 238 insertions(+), 743 deletions(-) diff --git a/README.md b/README.md index bbc59111f..83d8a5007 100644 --- a/README.md +++ b/README.md @@ -74,13 +74,13 @@ $ xvc file track my-models/ --as copy Configure a cloud storage to share the files you added. ```shell -$ xvc storage new s3 --name my-remote --region us-east-1 --bucket-name my-xvc-remote +$ xvc storage new s3 --name my-storage --region us-east-1 --bucket-name my-xvc-remote ``` You can send the files to this storage. ```shell -$ xvc file send --to my-remote +$ xvc file send --to my-storage ``` When you (or someone else) want to access these files later, you can clone the Git repository and get the files from the @@ -91,7 +91,7 @@ $ git clone https://example.com/my-machine-learning-project Cloning into 'my-machine-learning-project'... $ cd my-machine-learning-project -$ xvc file bring my-data/ --from my-remote +$ xvc file bring my-data/ --from my-storage ``` diff --git a/book/src/ref/xvc-file-send.md b/book/src/ref/xvc-file-send.md index 258803941..c64005bd1 100644 --- a/book/src/ref/xvc-file-send.md +++ b/book/src/ref/xvc-file-send.md @@ -12,7 +12,7 @@ Arguments: [TARGETS]... Targets to send/push/upload to storage Options: - -r, --remote Storage name or guid to send the files + -r, --storage Storage name or guid to send the files --force Force even if the files are already present in the storage -h, --help Print help diff --git a/book/src/ref/xvc-file-share.md b/book/src/ref/xvc-file-share.md index daa4567e1..cc59e03ed 100644 --- a/book/src/ref/xvc-file-share.md +++ b/book/src/ref/xvc-file-share.md @@ -6,13 +6,13 @@ $ xvc file share --help Share a file from S3 compatible storage for a limited time -Usage: xvc file share [OPTIONS] --remote +Usage: xvc file share [OPTIONS] --storage Arguments: File to send/push/upload to storage Options: - -r, --remote Storage name or guid to send the files + -r, --storage Storage name or guid to send the files -d, --duration Period to send the files to. You can use s, m, h, d, w suffixes [default: 24h] -h, --help Print help diff --git a/config/Cargo.toml b/config/Cargo.toml index f12ed2927..e119071cf 100644 --- a/config/Cargo.toml +++ b/config/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-config" -version = "0.6.8-alpha.1" +version = "0.6.8-alpha.2" edition = "2021" description = "Xvc configuration management" authors = ["Emre Şahin "] @@ -20,8 +20,8 @@ debug = true [dependencies] -xvc-logging = { version = "0.6.8-alpha.1", path = "../logging" } -xvc-walker = { version = "0.6.8-alpha.1", path = "../walker" } +xvc-logging = { version = "0.6.8-alpha.2", path = "../logging" } +xvc-walker = { version = "0.6.8-alpha.2", path = "../walker" } ## Cli and config diff --git a/core/Cargo.toml b/core/Cargo.toml index 3fde8c0b1..02d0df6dd 100644 --- a/core/Cargo.toml +++ b/core/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-core" -version = "0.6.8-alpha.1" +version = "0.6.8-alpha.2" edition = "2021" description = "Xvc core for common elements for all commands" authors = ["Emre Şahin "] @@ -19,10 +19,10 @@ crate-type = ["rlib"] debug = true [dependencies] -xvc-config = { version = "0.6.8-alpha.1", path = "../config" } -xvc-logging = { version = "0.6.8-alpha.1", path = "../logging" } -xvc-ecs = { version = "0.6.8-alpha.1", path = "../ecs" } -xvc-walker = { version = "0.6.8-alpha.1", path = "../walker" } +xvc-config = { version = "0.6.8-alpha.2", path = "../config" } +xvc-logging = { version = "0.6.8-alpha.2", path = "../logging" } +xvc-ecs = { version = "0.6.8-alpha.2", path = "../ecs" } +xvc-walker = { version = "0.6.8-alpha.2", path = "../walker" } ## Cli and config clap = { version = "^4.4", features = ["derive"] } @@ -82,6 +82,6 @@ derive_more = "^0.99" itertools = "^0.12" [dev-dependencies] -xvc-test-helper = { version = "0.6.8-alpha.1", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.8-alpha.2", path = "../test_helper/" } proptest = "^1.4" test-case = "^3.3" diff --git a/ecs/Cargo.toml b/ecs/Cargo.toml index cc39458b1..471898ab1 100644 --- a/ecs/Cargo.toml +++ b/ecs/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-ecs" -version = "0.6.8-alpha.1" +version = "0.6.8-alpha.2" edition = "2021" description = "Entity-Component System for Xvc" authors = ["Emre Şahin "] @@ -19,7 +19,7 @@ crate-type = ["rlib"] debug = true [dependencies] -xvc-logging = { version = "0.6.8-alpha.1", path = "../logging" } +xvc-logging = { version = "0.6.8-alpha.2", path = "../logging" } ## Serialization serde = { version = "^1.0", features = ["derive"] } diff --git a/file/Cargo.toml b/file/Cargo.toml index cc08d37f7..25bbe8d5f 100644 --- a/file/Cargo.toml +++ b/file/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-file" -version = "0.6.8-alpha.1" +version = "0.6.8-alpha.2" edition = "2021" description = "File tracking, versioning, upload and download functions for Xvc" authors = ["Emre Şahin "] @@ -24,12 +24,12 @@ bench = true debug = true [dependencies] -xvc-logging = { version = "0.6.8-alpha.1", path = "../logging" } -xvc-config = { version = "0.6.8-alpha.1", path = "../config" } -xvc-core = { version = "0.6.8-alpha.1", path = "../core" } -xvc-ecs = { version = "0.6.8-alpha.1", path = "../ecs" } -xvc-walker = { version = "0.6.8-alpha.1", path = "../walker" } -xvc-storage = { version = "0.6.8-alpha.1", path = "../storage" } +xvc-logging = { version = "0.6.8-alpha.2", path = "../logging" } +xvc-config = { version = "0.6.8-alpha.2", path = "../config" } +xvc-core = { version = "0.6.8-alpha.2", path = "../core" } +xvc-ecs = { version = "0.6.8-alpha.2", path = "../ecs" } +xvc-walker = { version = "0.6.8-alpha.2", path = "../walker" } +xvc-storage = { version = "0.6.8-alpha.2", path = "../storage" } ## Cli and config @@ -94,5 +94,5 @@ parse-size = "^1.0" [dev-dependencies] -xvc-test-helper = { version = "0.6.8-alpha.1", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.8-alpha.2", path = "../test_helper/" } shellfn = "^0.1" diff --git a/file/src/send/mod.rs b/file/src/send/mod.rs index 0d2bde9b1..8298719e3 100644 --- a/file/src/send/mod.rs +++ b/file/src/send/mod.rs @@ -25,7 +25,7 @@ use xvc_storage::{storage::get_storage_record, StorageIdentifier, XvcStorageOper pub struct SendCLI { /// Storage name or guid to send the files #[arg(long, short, alias = "to")] - remote: StorageIdentifier, + storage: StorageIdentifier, /// Force even if the files are already present in the storage #[arg(long)] force: bool, @@ -36,7 +36,7 @@ pub struct SendCLI { /// Send a targets in `opts.targets` in `xvc_root` to `opt.remote` pub fn cmd_send(output_snd: &XvcOutputSender, xvc_root: &XvcRoot, opts: SendCLI) -> Result<()> { - let remote = get_storage_record(output_snd, xvc_root, &opts.remote)?; + let remote = get_storage_record(output_snd, xvc_root, &opts.storage)?; watch!(remote); let current_dir = xvc_root.config().current_dir()?; let targets = load_targets_from_store(xvc_root, current_dir, &opts.targets)?; diff --git a/file/src/share/mod.rs b/file/src/share/mod.rs index c16c82e51..6fe3d30e0 100644 --- a/file/src/share/mod.rs +++ b/file/src/share/mod.rs @@ -17,7 +17,7 @@ use xvc_storage::{storage::get_storage_record, StorageIdentifier, XvcStorageOper pub struct ShareCLI { /// Storage name or guid to send the files #[arg(long, short, alias = "from")] - remote: StorageIdentifier, + storage: StorageIdentifier, /// Period to send the files to. You can use s, m, h, d, w suffixes. #[arg(long, short, default_value = "24h")] duration: String, @@ -28,8 +28,8 @@ pub struct ShareCLI { pub fn cmd_share(output_snd: &XvcOutputSender, xvc_root: &XvcRoot, opts: ShareCLI) -> Result<()> { // TODO: TIDY UP these implementation to reuse code in other places - let remote = get_storage_record(output_snd, xvc_root, &opts.remote)?; - watch!(remote); + let storage = get_storage_record(output_snd, xvc_root, &opts.storage)?; + watch!(storage); let current_dir = xvc_root.config().current_dir()?; let targets = load_targets_from_store(xvc_root, current_dir, &Some(vec![opts.target]))?; watch!(targets); @@ -66,6 +66,6 @@ pub fn cmd_share(output_snd: &XvcOutputSender, xvc_root: &XvcRoot, opts: ShareCL watch!(duration); - remote.share(output_snd, xvc_root, &cache_path, duration)?; + storage.share(output_snd, xvc_root, &cache_path, duration)?; Ok(()) } diff --git a/lib/Cargo.toml b/lib/Cargo.toml index ae1c5c4e6..3a8713947 100644 --- a/lib/Cargo.toml +++ b/lib/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc" -version = "0.6.8-alpha.1" +version = "0.6.8-alpha.2" edition = "2021" description = "An MLOps tool to manage data files and pipelines on top of Git" authors = ["Emre Şahin "] @@ -23,14 +23,14 @@ path = "src/main.rs" debug = true [dependencies] -xvc-config = { version = "0.6.8-alpha.1", path = "../config" } -xvc-core = { version = "0.6.8-alpha.1", path = "../core" } -xvc-logging = { version = "0.6.8-alpha.1", path = "../logging" } -xvc-ecs = { version = "0.6.8-alpha.1", path = "../ecs" } -xvc-file = { version = "0.6.8-alpha.1", path = "../file" } -xvc-pipeline = { version = "0.6.8-alpha.1", path = "../pipeline" } -xvc-walker = { version = "0.6.8-alpha.1", path = "../walker" } -xvc-storage = { version = "0.6.8-alpha.1", path = "../storage" } +xvc-config = { version = "0.6.8-alpha.2", path = "../config" } +xvc-core = { version = "0.6.8-alpha.2", path = "../core" } +xvc-logging = { version = "0.6.8-alpha.2", path = "../logging" } +xvc-ecs = { version = "0.6.8-alpha.2", path = "../ecs" } +xvc-file = { version = "0.6.8-alpha.2", path = "../file" } +xvc-pipeline = { version = "0.6.8-alpha.2", path = "../pipeline" } +xvc-walker = { version = "0.6.8-alpha.2", path = "../walker" } +xvc-storage = { version = "0.6.8-alpha.2", path = "../storage" } ## Cli and config @@ -113,4 +113,4 @@ jwalk = "^0.8" proptest = "^1.4" shellfn = "^0.1" test-case = "^3.3" -xvc-test-helper = { version = "0.6.8-alpha.1", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.8-alpha.2", path = "../test_helper/" } diff --git a/logging/Cargo.toml b/logging/Cargo.toml index ad78df60d..4ce732c9a 100644 --- a/logging/Cargo.toml +++ b/logging/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-logging" -version = "0.6.8-alpha.1" +version = "0.6.8-alpha.2" edition = "2021" description = "Logging crate for Xvc" authors = ["Emre Şahin "] diff --git a/pipeline/Cargo.toml b/pipeline/Cargo.toml index bb8622df9..c507cf991 100644 --- a/pipeline/Cargo.toml +++ b/pipeline/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-pipeline" -version = "0.6.8-alpha.1" +version = "0.6.8-alpha.2" edition = "2021" description = "Xvc data pipeline management" authors = ["Emre Şahin "] @@ -19,12 +19,12 @@ debug = true [dependencies] -xvc-config = { version = "0.6.8-alpha.1", path = "../config" } -xvc-core = { version = "0.6.8-alpha.1", path = "../core" } -xvc-ecs = { version = "0.6.8-alpha.1", path = "../ecs" } -xvc-logging = { version = "0.6.8-alpha.1", path = "../logging" } -xvc-walker = { version = "0.6.8-alpha.1", path = "../walker" } -xvc-file = { version = "0.6.8-alpha.1", path = "../file" } +xvc-config = { version = "0.6.8-alpha.2", path = "../config" } +xvc-core = { version = "0.6.8-alpha.2", path = "../core" } +xvc-ecs = { version = "0.6.8-alpha.2", path = "../ecs" } +xvc-logging = { version = "0.6.8-alpha.2", path = "../logging" } +xvc-walker = { version = "0.6.8-alpha.2", path = "../walker" } +xvc-file = { version = "0.6.8-alpha.2", path = "../file" } ## Cli and config clap = { version = "^4.4", features = ["derive"] } @@ -92,5 +92,5 @@ itertools = "^0.12" derive_more = "^0.99" [dev-dependencies] -xvc-test-helper = { version = "0.6.8-alpha.1", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.8-alpha.2", path = "../test_helper/" } test-case = "^3.3" diff --git a/storage/Cargo.toml b/storage/Cargo.toml index 1c475eb28..1bffe1074 100644 --- a/storage/Cargo.toml +++ b/storage/Cargo.toml @@ -1,8 +1,8 @@ [package] name = "xvc-storage" -version = "0.6.8-alpha.1" +version = "0.6.8-alpha.2" edition = "2021" -description = "Xvc remote (and local) storage management" +description = "Xvc remote and local storage management" authors = ["Emre Şahin "] license = "GPL-3.0" homepage = "https://xvc.dev" @@ -20,11 +20,11 @@ debug = true [dependencies] -xvc-logging = { version = "0.6.8-alpha.1", path = "../logging" } -xvc-config = { version = "0.6.8-alpha.1", path = "../config" } -xvc-core = { version = "0.6.8-alpha.1", path = "../core" } -xvc-ecs = { version = "0.6.8-alpha.1", path = "../ecs" } -xvc-walker = { version = "0.6.8-alpha.1", path = "../walker" } +xvc-logging = { version = "0.6.8-alpha.2", path = "../logging" } +xvc-config = { version = "0.6.8-alpha.2", path = "../config" } +xvc-core = { version = "0.6.8-alpha.2", path = "../core" } +xvc-ecs = { version = "0.6.8-alpha.2", path = "../ecs" } +xvc-walker = { version = "0.6.8-alpha.2", path = "../walker" } ## Cli and config clap = { version = "^4.4", features = ["derive"] } @@ -109,7 +109,7 @@ digital-ocean = ["async"] [dev-dependencies] -xvc-test-helper = { version = "0.6.8-alpha.1", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.8-alpha.2", path = "../test_helper/" } shellfn = "^0.1" [package.metadata.cargo-udeps.ignore] diff --git a/storage/src/error.rs b/storage/src/error.rs index 0dcbf9264..f653ab1d4 100644 --- a/storage/src/error.rs +++ b/storage/src/error.rs @@ -61,7 +61,7 @@ pub enum Error { NoRepositoryGuidFound, #[error("Cannot find remote with identifier: {identifier}")] - CannotFindRemoteWithIdentifier { identifier: StorageIdentifier }, + CannotFindStorageWithIdentifier { identifier: StorageIdentifier }, #[error("Process Exec Error: {source}")] ProcessExecError { diff --git a/storage/src/lib.rs b/storage/src/lib.rs index b8c339ca6..91b0d55c8 100755 --- a/storage/src/lib.rs +++ b/storage/src/lib.rs @@ -78,7 +78,7 @@ pub enum StorageNewSubCommand { /// Add a new generic storage. /// /// ⚠️ Please note that this is an advanced method to configure storages. - /// You may damage your repository and local and remote files with incorrect configurations. + /// You may damage your repository and local and storage files with incorrect configurations. /// /// Please see https://docs.xvc.dev/ref/xvc-storage-new-generic.html for examples and make /// necessary backups. diff --git a/storage/src/storage/async_common.rs b/storage/src/storage/async_common.rs index 3f7f6e66e..43df2ab77 100644 --- a/storage/src/storage/async_common.rs +++ b/storage/src/storage/async_common.rs @@ -31,28 +31,28 @@ use super::XvcStorageTempDir; use super::XVC_STORAGE_GUID_FILENAME; pub trait XvcS3StorageOperations { - fn remote_prefix(&self) -> String; + fn storage_prefix(&self) -> String; fn guid(&self) -> &XvcStorageGuid; fn get_bucket(&self) -> Result; fn credentials(&self) -> Result; fn bucket_name(&self) -> String; - fn build_remote_path(&self, cache_path: &XvcCachePath) -> XvcStoragePath { + fn build_storage_path(&self, cache_path: &XvcCachePath) -> XvcStoragePath { XvcStoragePath::from(format!( "{}/{}/{}", - self.remote_prefix(), + self.storage_prefix(), self.guid(), cache_path )) } fn region(&self) -> String; - async fn write_remote_guid(&self) -> Result<()> { + async fn write_storage_guid(&self) -> Result<()> { let guid_str = self.guid().to_string(); let guid_bytes = guid_str.as_bytes(); let bucket = self.get_bucket()?; let response = bucket .put_object( - format!("{}/{}", self.remote_prefix(), XVC_STORAGE_GUID_FILENAME), + format!("{}/{}", self.storage_prefix(), XVC_STORAGE_GUID_FILENAME), guid_bytes, ) .await; @@ -64,7 +64,7 @@ pub trait XvcS3StorageOperations { } async fn a_init(&mut self, output_snd: &XvcOutputSender) -> Result { - let res_response = self.write_remote_guid().await; + let res_response = self.write_storage_guid().await; let guid = self.guid().clone(); @@ -86,11 +86,11 @@ pub trait XvcS3StorageOperations { let region = Region::from_str(&self.region()).unwrap_or("us-east-1".parse().unwrap()); let bucket = Bucket::new(&self.bucket_name(), region, credentials)?; let xvc_guid = xvc_root.config().guid().unwrap(); - let prefix = self.remote_prefix().clone(); + let prefix = self.storage_prefix().clone(); let res_list = bucket .list( - format!("{}/{}", self.remote_prefix(), xvc_guid), + format!("{}/{}", self.storage_prefix(), xvc_guid), Some("/".to_string()), ) .await; @@ -143,7 +143,7 @@ pub trait XvcS3StorageOperations { for cache_path in paths { watch!(cache_path); - let remote_path = self.build_remote_path(cache_path); + let storage_path = self.build_storage_path(cache_path); let abs_cache_path = cache_path.to_absolute_path(xvc_root); watch!(abs_cache_path); @@ -151,13 +151,13 @@ pub trait XvcS3StorageOperations { watch!(path); let res_response = bucket - .put_object_stream(&mut path, remote_path.as_str()) + .put_object_stream(&mut path, storage_path.as_str()) .await; match res_response { Ok(_) => { - info!(output_snd, "{} -> {}", abs_cache_path, remote_path.as_str()); - copied_paths.push(remote_path); + info!(output_snd, "{} -> {}", abs_cache_path, storage_path.as_str()); + copied_paths.push(storage_path); watch!(copied_paths.len()); } Err(err) => { @@ -185,21 +185,21 @@ pub trait XvcS3StorageOperations { for cache_path in paths { watch!(cache_path); - let remote_path = self.build_remote_path(cache_path); + let storage_path = self.build_storage_path(cache_path); let abs_cache_dir = temp_dir.temp_cache_dir(cache_path)?; fs::create_dir_all(&abs_cache_dir)?; let abs_cache_path = temp_dir.temp_cache_path(cache_path)?; watch!(abs_cache_path); - let response_data_stream = bucket.get_object_stream(remote_path.as_str()).await; + let response_data_stream = bucket.get_object_stream(storage_path.as_str()).await; match response_data_stream { Ok(mut response) => { - info!(output_snd, "{} -> {}", remote_path.as_str(), abs_cache_path); + info!(output_snd, "{} -> {}", storage_path.as_str(), abs_cache_path); let mut async_cache_path = tokio::fs::File::create(&abs_cache_path).await?; while let Some(chunk) = response.bytes().next().await { async_cache_path.write_all(&chunk).await?; } - copied_paths.push(remote_path); + copied_paths.push(storage_path); watch!(copied_paths.len()); } Err(err) => { @@ -228,10 +228,10 @@ pub trait XvcS3StorageOperations { for cache_path in paths { watch!(cache_path); - let remote_path = self.build_remote_path(cache_path); - bucket.delete_object(remote_path.as_str()).await?; - info!(output, "[DELETE] {}", remote_path.as_str()); - deleted_paths.push(remote_path); + let storage_path = self.build_storage_path(cache_path); + bucket.delete_object(storage_path.as_str()).await?; + info!(output, "[DELETE] {}", storage_path.as_str()); + deleted_paths.push(storage_path); } Ok(XvcStorageDeleteEvent { @@ -256,7 +256,7 @@ pub trait XvcS3StorageOperations { // let expiration_seconds = duration.as_secs() as u32; - let path = self.build_remote_path(path); + let path = self.build_storage_path(path); let signed_url = bucket.presign_get(path.as_str(), expiration_seconds, None)?; info!(output, "[SHARED] {}", path.as_str()); output!(output, "{}", signed_url); diff --git a/storage/src/storage/digital_ocean.rs b/storage/src/storage/digital_ocean.rs index 006b198fc..32cea6ec5 100644 --- a/storage/src/storage/digital_ocean.rs +++ b/storage/src/storage/digital_ocean.rs @@ -14,9 +14,9 @@ use crate::{XvcStorageGuid, XvcStorageOperations}; use super::async_common::XvcS3StorageOperations; -/// Configure a new Digital Ocean Spaces remote. +/// Configure a new Digital Ocean Spaces storage. /// -/// `bucket_name`, `region` and `remote_prefix` sets a URL for the storage +/// `bucket_name`, `region` and `storage_prefix` sets a URL for the storage /// location. /// /// This creates a [XvcDigitalOceanStorage], calls its @@ -29,18 +29,18 @@ pub fn cmd_new_digital_ocean( name: String, bucket_name: String, region: String, - remote_prefix: String, + storage_prefix: String, ) -> Result<()> { - let mut remote = XvcDigitalOceanStorage { + let mut storage = XvcDigitalOceanStorage { guid: XvcStorageGuid::new(), name, region, bucket_name, - remote_prefix, + storage_prefix, }; - watch!(remote); + watch!(storage); - let init_event = remote.init(output_snd, xvc_root)?; + let init_event = storage.init(output_snd, xvc_root)?; watch!(init_event); xvc_root.with_r1nstore_mut(|store: &mut R1NStore| { @@ -48,7 +48,7 @@ pub fn cmd_new_digital_ocean( let event_e = xvc_root.new_entity(); store.insert( store_e, - XvcStorage::DigitalOcean(remote.clone()), + XvcStorage::DigitalOcean(storage.clone()), event_e, XvcStorageEvent::Init(init_event.clone()), ); @@ -58,23 +58,23 @@ pub fn cmd_new_digital_ocean( Ok(()) } -/// A Digital Ocean Spaces remote. +/// A Digital Ocean Spaces storage. #[derive(Clone, Debug, PartialOrd, Ord, PartialEq, Eq, Serialize, Deserialize)] pub struct XvcDigitalOceanStorage { - /// The GUID of the remote. + /// The GUID of the storage. pub guid: XvcStorageGuid, - /// The name of the remote. + /// The name of the storage. pub name: String, - /// The region of the remote. + /// The region of the storage. pub region: String, - /// The bucket name of the remote. + /// The bucket name of the storage. pub bucket_name: String, - /// The remote prefix of the remote. - pub remote_prefix: String, + /// The path prefix of the storage. + pub storage_prefix: String, } impl XvcDigitalOceanStorage { - fn remote_specific_credentials(&self) -> Result { + fn storage_specific_credentials(&self) -> Result { Credentials::new( Some(&env::var(format!( "XVC_STORAGE_ACCESS_KEY_ID_{}", @@ -101,8 +101,8 @@ impl XvcDigitalOceanStorage { } impl XvcS3StorageOperations for XvcDigitalOceanStorage { - fn remote_prefix(&self) -> String { - self.remote_prefix.clone() + fn storage_prefix(&self) -> String { + self.storage_prefix.clone() } fn guid(&self) -> &XvcStorageGuid { @@ -110,7 +110,7 @@ impl XvcS3StorageOperations for XvcDigitalOceanStorage { } fn credentials(&self) -> Result { - match self.remote_specific_credentials() { + match self.storage_specific_credentials() { Ok(c) => Ok(c), Err(e1) => match self.storage_type_credentials() { Ok(c) => Ok(c), diff --git a/storage/src/storage/gcs.rs b/storage/src/storage/gcs.rs index 48225da19..3ea5a429d 100644 --- a/storage/src/storage/gcs.rs +++ b/storage/src/storage/gcs.rs @@ -16,7 +16,7 @@ use super::async_common::XvcS3StorageOperations; /// Configure a new Google Cloud Storage remote. /// -/// `bucket_name`, `region` and `remote_prefix` sets a URL for the storage +/// `bucket_name`, `region` and `storage_prefix` sets a URL for the storage /// location. /// /// This creates a [XvcGcsStorage], calls its @@ -29,19 +29,19 @@ pub fn cmd_new_gcs( name: String, bucket_name: String, region: String, - remote_prefix: String, + storage_prefix: String, ) -> Result<()> { - let mut remote = XvcGcsStorage { + let mut storage = XvcGcsStorage { guid: XvcStorageGuid::new(), name, region, bucket_name, - remote_prefix, + storage_prefix, }; - watch!(remote); + watch!(storage); - let init_event = remote.init(output_snd, xvc_root)?; + let init_event = storage.init(output_snd, xvc_root)?; watch!(init_event); xvc_root.with_r1nstore_mut(|store: &mut R1NStore| { @@ -49,7 +49,7 @@ pub fn cmd_new_gcs( let event_e = xvc_root.new_entity(); store.insert( store_e, - XvcStorage::Gcs(remote.clone()), + XvcStorage::Gcs(storage.clone()), event_e, XvcStorageEvent::Init(init_event.clone()), ); @@ -62,20 +62,20 @@ pub fn cmd_new_gcs( /// A Google Cloud Storage remote. #[derive(Clone, Debug, PartialOrd, Ord, PartialEq, Eq, Serialize, Deserialize)] pub struct XvcGcsStorage { - /// The unique identifier of the remote. + /// The unique identifier of the storage. pub guid: XvcStorageGuid, - /// The name of the remote. + /// The name of the storage. pub name: String, - /// The region of the remote. + /// The region of the storage. pub region: String, - /// The bucket name of the remote. + /// The bucket name of the storage. pub bucket_name: String, - /// The remote prefix of the remote. - pub remote_prefix: String, + /// The path prefix on the storage. + pub storage_prefix: String, } impl XvcGcsStorage { - fn remote_specific_credentials(&self) -> Result { + fn storage_specific_credentials(&self) -> Result { Credentials::new( Some(&env::var(format!( "XVC_STORAGE_ACCESS_KEY_ID_{}", @@ -102,8 +102,8 @@ impl XvcGcsStorage { } impl XvcS3StorageOperations for XvcGcsStorage { - fn remote_prefix(&self) -> String { - self.remote_prefix.clone() + fn storage_prefix(&self) -> String { + self.storage_prefix.clone() } fn guid(&self) -> &XvcStorageGuid { @@ -129,7 +129,7 @@ impl XvcS3StorageOperations for XvcGcsStorage { } fn credentials(&self) -> Result { - match self.remote_specific_credentials() { + match self.storage_specific_credentials() { Ok(c) => Ok(c), Err(e1) => match self.storage_type_credentials() { Ok(c) => Ok(c), @@ -143,289 +143,3 @@ impl XvcS3StorageOperations for XvcGcsStorage { .map_err(|e| e.into()) } } -// async fn a_init( -// self, -// output_snd: &XvcOutputSender, -// _xvc_root: &xvc_core::XvcRoot, -// ) -> Result<(XvcStorageInitEvent, Self)> { -// let bucket = self.get_bucket()?; -// let guid = self.guid.clone(); -// let guid_str = self.guid.to_string(); -// let guid_bytes = guid_str.as_bytes(); -// -// watch!(bucket); -// watch!(guid); -// watch!(guid_str); -// -// let res_response = bucket -// .put_object( -// format!("{}/{}", self.remote_prefix, XVC_STORAGE_GUID_FILENAME), -// guid_bytes, -// ) -// .await; -// -// match res_response { -// Ok(_) => Ok((XvcStorageInitEvent { guid }, self)), -// Err(err) => { -// error!(output_snd, "Error: {}", err); -// Err(Error::S3Error { source: err }) -// } -// } -// } -// -// async fn a_list( -// &self, -// output_snd: &XvcOutputSender, -// xvc_root: &xvc_core::XvcRoot, -// ) -> Result { -// let credentials = self.credentials()?; -// let region = Region::from_str(&self.region).unwrap_or("us-east-1".parse().unwrap()); -// let bucket = Bucket::new(&self.bucket_name, region, credentials)?; -// let _guid = self.guid.clone(); -// let guid_str = self.guid.to_string(); -// let _guid_bytes = guid_str.as_bytes(); -// let xvc_guid = xvc_root.config().guid().unwrap(); -// let prefix = self.remote_prefix.clone(); -// -// let res_list = bucket -// .list( -// format!("{}/{}", self.remote_prefix, xvc_guid), -// Some("/".to_string()), -// ) -// .await; -// -// match res_list { -// Ok(list_all) => { -// // select only the matching elements -// let re = Regex::new(&format!( -// "{prefix}/{xvc_guid}/{cp}/{d3}/{d3}/{d58}/0\\..*$", -// cp = r#"[a-zA-Z][0-9]"#, -// d3 = r#"[0-9A-Fa-f]{3}"#, -// d58 = r#"[0-9A-Fa-f]{58}"# -// )) -// .unwrap(); -// -// let paths = list_all -// .iter() -// .filter_map(|e| { -// if re.is_match(e.name.as_ref()) { -// Some(XvcStoragePath::from_str(&e.name).unwrap()) -// } else { -// None -// } -// }) -// .collect(); -// -// Ok(XvcStorageListEvent { -// guid: self.guid.clone(), -// paths, -// }) -// } -// -// Err(err) => { -// error!(output_snd, "Error: {}", err); -// Err(Error::S3Error { source: err }) -// } -// } -// } -// -// fn build_remote_path(&self, repo_guid: &str, cache_path: &XvcCachePath) -> XvcStoragePath { -// XvcStoragePath::from(format!( -// "{}/{}/{}", -// self.remote_prefix, repo_guid, cache_path -// )) -// } -// -// async fn a_send( -// &self, -// output: &XvcOutputSender, -// xvc_root: &xvc_core::XvcRoot, -// paths: &[xvc_core::XvcCachePath], -// _force: bool, -// ) -> crate::Result { -// let repo_guid = xvc_root -// .config() -// .guid() -// .ok_or_else(|| crate::Error::NoRepositoryGuidFound)?; -// let mut copied_paths = Vec::::new(); -// -// let bucket = self.get_bucket()?; -// -// for cache_path in paths { -// watch!(cache_path); -// let remote_path = self.build_remote_path(&repo_guid, cache_path); -// let abs_cache_path = cache_path.to_absolute_path(xvc_root); -// watch!(abs_cache_path); -// -// let mut path = tokio::fs::File::open(&abs_cache_path).await?; -// watch!(path); -// -// let res_response = bucket -// .put_object_stream(&mut path, remote_path.as_str()) -// .await; -// -// match res_response { -// Ok(_) => { -// info!(output, "{} -> {}", abs_cache_path, remote_path.as_str()); -// copied_paths.push(remote_path); -// watch!(copied_paths.len()); -// } -// Err(err) => { -// error!(output, "Error: {}", err); -// } -// } -// } -// -// Ok(XvcStorageSendEvent { -// guid: self.guid.clone(), -// paths: copied_paths, -// }) -// } -// -// async fn a_receive( -// &self, -// output: &XvcOutputSender, -// xvc_root: &xvc_core::XvcRoot, -// paths: &[xvc_core::XvcCachePath], -// _force: bool, -// ) -> Result<(XvcStorageTempDir, XvcStorageReceiveEvent)> { -// let repo_guid = xvc_root -// .config() -// .guid() -// .ok_or_else(|| crate::Error::NoRepositoryGuidFound)?; -// let mut copied_paths = Vec::::new(); -// -// let bucket = self.get_bucket()?; -// let temp_dir = XvcStorageTempDir::new()?; -// -// for cache_path in paths { -// watch!(cache_path); -// let remote_path = self.build_remote_path(&repo_guid, cache_path); -// let abs_cache_dir = temp_dir.temp_cache_dir(cache_path)?; -// fs::create_dir_all(&abs_cache_dir)?; -// let abs_cache_path = temp_dir.temp_cache_path(cache_path)?; -// watch!(abs_cache_path); -// -// let response_data_stream = bucket.get_object_stream(remote_path.as_str()).await; -// -// match response_data_stream { -// Ok(mut response) => { -// info!(output, "{} -> {}", remote_path.as_str(), abs_cache_path); -// let mut async_cache_path = tokio::fs::File::create(&abs_cache_path).await?; -// while let Some(chunk) = response.bytes().next().await { -// async_cache_path.write_all(&chunk).await?; -// } -// copied_paths.push(remote_path); -// watch!(copied_paths.len()); -// } -// Err(err) => { -// error!(output, "{}", err); -// } -// } -// } -// -// Ok(( -// temp_dir, -// XvcStorageReceiveEvent { -// guid: self.guid.clone(), -// paths: copied_paths, -// }, -// )) -// } -// -// async fn a_delete( -// &self, -// output: &XvcOutputSender, -// xvc_root: &xvc_core::XvcRoot, -// paths: &[XvcCachePath], -// ) -> Result { -// let repo_guid = xvc_root -// .config() -// .guid() -// .ok_or_else(|| crate::Error::NoRepositoryGuidFound)?; -// let mut deleted_paths = Vec::::new(); -// -// let bucket = self.get_bucket()?; -// -// for cache_path in paths { -// watch!(cache_path); -// let remote_path = self.build_remote_path(&repo_guid, cache_path); -// bucket.delete_object(remote_path.as_str()).await?; -// info!(output, "[DELETE] {}", remote_path.as_str()); -// deleted_paths.push(remote_path); -// } -// -// Ok(XvcStorageDeleteEvent { -// guid: self.guid.clone(), -// paths: deleted_paths, -// }) -// } -// } -// -// impl XvcStorageOperations for XvcGcsStorage { -// fn init( -// self, -// output: &XvcOutputSender, -// xvc_root: &xvc_core::XvcRoot, -// ) -> Result<(XvcStorageInitEvent, Self)> { -// let rt = tokio::runtime::Builder::new_multi_thread() -// .enable_all() -// .build()?; -// watch!(rt); -// rt.block_on(self.a_init(output, xvc_root)) -// } -// -// /// List the bucket contents that start with `self.remote_prefix` -// fn list( -// &self, -// output: &XvcOutputSender, -// xvc_root: &xvc_core::XvcRoot, -// ) -> crate::Result { -// let rt = tokio::runtime::Builder::new_multi_thread() -// .enable_all() -// .build() -// .unwrap(); -// rt.block_on(self.a_list(output, xvc_root)) -// } -// -// fn send( -// &self, -// output: &XvcOutputSender, -// xvc_root: &xvc_core::XvcRoot, -// paths: &[xvc_core::XvcCachePath], -// force: bool, -// ) -> crate::Result { -// let rt = tokio::runtime::Builder::new_multi_thread() -// .enable_all() -// .build() -// .unwrap(); -// rt.block_on(self.a_send(output, xvc_root, paths, force)) -// } -// -// fn receive( -// &self, -// output: &XvcOutputSender, -// xvc_root: &xvc_core::XvcRoot, -// paths: &[xvc_core::XvcCachePath], -// force: bool, -// ) -> crate::Result<(XvcStorageTempDir, XvcStorageReceiveEvent)> { -// let rt = tokio::runtime::Builder::new_multi_thread() -// .enable_all() -// .build() -// .unwrap(); -// rt.block_on(self.a_receive(output, xvc_root, paths, force)) -// } -// -// fn delete( -// &self, -// output: &XvcOutputSender, -// xvc_root: &xvc_core::XvcRoot, -// paths: &[xvc_core::XvcCachePath], -// ) -> crate::Result { -// let rt = tokio::runtime::Builder::new_multi_thread() -// .enable_all() -// .build() -// .unwrap(); -// rt.block_on(self.a_delete(output, xvc_root, paths)) -// } -// } diff --git a/storage/src/storage/local.rs b/storage/src/storage/local.rs index 1a8e7de63..03346198a 100644 --- a/storage/src/storage/local.rs +++ b/storage/src/storage/local.rs @@ -26,19 +26,19 @@ pub fn cmd_storage_new_local( path: PathBuf, name: String, ) -> Result<()> { - let mut remote = XvcLocalStorage { + let mut storage = XvcLocalStorage { guid: XvcStorageGuid::new(), name, path, }; - let init_event = remote.init(output_snd, xvc_root)?; + let init_event = storage.init(output_snd, xvc_root)?; xvc_root.with_r1nstore_mut(|store: &mut R1NStore| { let store_e = xvc_root.new_entity(); let event_e = xvc_root.new_entity(); store.insert( store_e, - XvcStorage::Local(remote.clone()), + XvcStorage::Local(storage.clone()), event_e, XvcStorageEvent::Init(init_event.clone()), ); @@ -60,7 +60,7 @@ pub struct XvcLocalStorage { } impl XvcLocalStorage { - fn remote_path(&self, repo_guid: &str, cache_path: &XvcCachePath) -> XvcStoragePath { + fn storage_path(&self, repo_guid: &str, cache_path: &XvcCachePath) -> XvcStoragePath { XvcStoragePath::from(format!("{}/{}", repo_guid, cache_path)) } } @@ -99,7 +99,7 @@ impl XvcStorageOperations for XvcLocalStorage { fs::write(guid_filename, format!("{}", self.guid))?; info!( output, - "Created local remote directory {} with guid: {}", + "Created local storage directory {} with guid: {}", self.path.to_string_lossy(), self.guid ); @@ -127,24 +127,24 @@ impl XvcStorageOperations for XvcLocalStorage { let mut copied_paths = Vec::::new(); for cache_path in paths { - let remote_path = self.remote_path(&repo_guid, cache_path); - let abs_remote_path = remote_path.as_ref().to_logical_path(&self.path); - if force && abs_remote_path.exists() { - fs::remove_file(abs_remote_path.clone())?; + let storage_path = self.storage_path(&repo_guid, cache_path); + let abs_storage_path = storage_path.as_ref().to_logical_path(&self.path); + if force && abs_storage_path.exists() { + fs::remove_file(abs_storage_path.clone())?; } else { - info!(output, "[SKIPPED] {}", remote_path) + info!(output, "[SKIPPED] {}", storage_path) } let abs_cache_path = cache_path.to_absolute_path(xvc_root); - let abs_remote_dir = abs_remote_path.parent().unwrap(); - fs::create_dir_all(abs_remote_dir)?; - fs::copy(&abs_cache_path, &abs_remote_path)?; - copied_paths.push(remote_path); + let abs_storage_dir = abs_storage_path.parent().unwrap(); + fs::create_dir_all(abs_storage_dir)?; + fs::copy(&abs_cache_path, &abs_storage_path)?; + copied_paths.push(storage_path); watch!(copied_paths.len()); info!( output, "{} -> {}", abs_cache_path, - abs_remote_path.to_string_lossy() + abs_storage_path.to_string_lossy() ); } @@ -169,19 +169,19 @@ impl XvcStorageOperations for XvcLocalStorage { let temp_dir = XvcStorageTempDir::new()?; for cache_path in paths { - let remote_path = self.remote_path(&repo_guid, cache_path); - let abs_remote_path = remote_path.as_ref().to_logical_path(&self.path); + let storage_path = self.storage_path(&repo_guid, cache_path); + let abs_storage_path = storage_path.as_ref().to_logical_path(&self.path); let abs_cache_path = temp_dir.temp_cache_path(cache_path)?; let abs_cache_dir = temp_dir.temp_cache_dir(cache_path)?; fs::create_dir_all(&abs_cache_dir)?; - fs::copy(&abs_remote_path, &abs_cache_path)?; + fs::copy(&abs_storage_path, &abs_cache_path)?; watch!(abs_cache_path.exists()); - copied_paths.push(remote_path); + copied_paths.push(storage_path); watch!(copied_paths.len()); info!( output, "{} -> {}", - abs_remote_path.to_string_lossy(), + abs_storage_path.to_string_lossy(), abs_cache_path ); } @@ -208,11 +208,11 @@ impl XvcStorageOperations for XvcLocalStorage { let mut deleted_paths = Vec::::new(); for cache_path in paths { - let remote_path = self.remote_path(&repo_guid, cache_path); - let abs_remote_path = remote_path.as_ref().to_logical_path(&self.path); - fs::remove_file(&abs_remote_path)?; - info!(output, "[DELETE] {}", abs_remote_path.to_string_lossy()); - deleted_paths.push(remote_path); + let storage_path = self.storage_path(&repo_guid, cache_path); + let abs_storage_path = storage_path.as_ref().to_logical_path(&self.path); + fs::remove_file(&abs_storage_path)?; + info!(output, "[DELETE] {}", abs_storage_path.to_string_lossy()); + deleted_paths.push(storage_path); } Ok(XvcStorageDeleteEvent { diff --git a/storage/src/storage/minio.rs b/storage/src/storage/minio.rs index 748d7d591..bc26f3ba7 100644 --- a/storage/src/storage/minio.rs +++ b/storage/src/storage/minio.rs @@ -18,7 +18,7 @@ use super::XvcStoragePath; /// Configure a new Minio remote storage. /// -/// `endpoint`, `bucket_name`, `region` and `remote_prefix` sets a URL for the +/// `endpoint`, `bucket_name`, `region` and `storage_prefix` sets a URL for the /// storage location. /// /// This creates a [XvcMinioStorage], calls its @@ -34,19 +34,19 @@ pub fn cmd_new_minio( endpoint: String, bucket_name: String, region: String, - remote_prefix: String, + storage_prefix: String, ) -> Result<()> { - let mut remote = XvcMinioStorage { + let mut storage = XvcMinioStorage { guid: XvcStorageGuid::new(), name, region, bucket_name, - remote_prefix, + storage_prefix, endpoint, }; - watch!(remote); + watch!(storage); - let init_event = remote.init(output_snd, xvc_root)?; + let init_event = storage.init(output_snd, xvc_root)?; watch!(init_event); xvc_root.with_r1nstore_mut(|store: &mut R1NStore| { @@ -54,7 +54,7 @@ pub fn cmd_new_minio( let event_e = xvc_root.new_entity(); store.insert( store_e, - XvcStorage::Minio(remote.clone()), + XvcStorage::Minio(storage.clone()), event_e, XvcStorageEvent::Init(init_event.clone()), ); @@ -76,13 +76,13 @@ pub struct XvcMinioStorage { /// Bucket name of the storage pub bucket_name: String, /// Prefix of the storage within the bucket_name - pub remote_prefix: String, + pub storage_prefix: String, /// Full endpoint of the storage pub endpoint: String, } impl XvcMinioStorage { - fn remote_specific_credentials(&self) -> Result { + fn storage_specific_credentials(&self) -> Result { Credentials::new( Some(&env::var(format!( "XVC_STORAGE_ACCESS_KEY_ID_{}", @@ -109,8 +109,8 @@ impl XvcMinioStorage { } impl XvcS3StorageOperations for XvcMinioStorage { - fn remote_prefix(&self) -> String { - self.remote_prefix.clone() + fn storage_prefix(&self) -> String { + self.storage_prefix.clone() } fn guid(&self) -> &XvcStorageGuid { @@ -129,7 +129,7 @@ impl XvcS3StorageOperations for XvcMinioStorage { } fn credentials(&self) -> Result { - match self.remote_specific_credentials() { + match self.storage_specific_credentials() { Ok(c) => Ok(c), Err(e1) => match self.storage_type_credentials() { Ok(c) => Ok(c), @@ -147,10 +147,10 @@ impl XvcS3StorageOperations for XvcMinioStorage { self.bucket_name.clone() } - fn build_remote_path(&self, cache_path: &XvcCachePath) -> XvcStoragePath { + fn build_storage_path(&self, cache_path: &XvcCachePath) -> XvcStoragePath { XvcStoragePath::from(format!( "{}/{}/{}", - self.remote_prefix, + self.storage_prefix, self.guid(), cache_path )) @@ -159,223 +159,4 @@ impl XvcS3StorageOperations for XvcMinioStorage { fn region(&self) -> String { self.region.clone() } - - // async fn a_init( - // self, - // output: &XvcOutputSender, - // _xvc_root: &xvc_core::XvcRoot, - // ) -> Result<(XvcStorageInitEvent, Self)> { - // let bucket = self.get_bucket()?; - // let guid = self.guid.clone(); - // let guid_str = self.guid.to_string(); - // let guid_bytes = guid_str.as_bytes(); - // - // watch!(bucket); - // watch!(guid); - // watch!(guid_str); - // - // let res_response = bucket - // .put_object( - // format!("{}/{}", self.remote_prefix, XVC_STORAGE_GUID_FILENAME), - // guid_bytes, - // ) - // .await; - // - // match res_response { - // Ok(_) => Ok((XvcStorageInitEvent { guid }, self)), - // Err(err) => { - // error!(output, "{}", err); - // Err(Error::S3Error { source: err }) - // } - // } - // } - // - // async fn a_list( - // &self, - // output: &XvcOutputSender, - // xvc_root: &xvc_core::XvcRoot, - // ) -> Result { - // let credentials = self.credentials()?; - // let region = Region::from_str(&self.region).unwrap_or("us-east-1".parse().unwrap()); - // let bucket = Bucket::new(&self.bucket_name, region, credentials)?; - // let _guid = self.guid.clone(); - // let guid_str = self.guid.to_string(); - // let _guid_bytes = guid_str.as_bytes(); - // let xvc_guid = xvc_root.config().guid().unwrap(); - // let prefix = self.remote_prefix.clone(); - // - // let res_list = bucket - // .list( - // format!("{}/{}", self.remote_prefix, xvc_guid), - // Some("/".to_string()), - // ) - // .await; - // - // match res_list { - // Ok(list_all) => { - // // select only the matching elements - // let re = Regex::new(&format!( - // "{prefix}/{xvc_guid}/{cp}/{d3}/{d3}/{d58}/0\\..*$", - // cp = r#"[a-zA-Z][0-9]"#, - // d3 = r#"[0-9A-Fa-f]{3}"#, - // d58 = r#"[0-9A-Fa-f]{58}"# - // )) - // .unwrap(); - // - // let paths = list_all - // .iter() - // .filter_map(|e| { - // if re.is_match(e.name.as_ref()) { - // Some(XvcStoragePath::from_str(&e.name).unwrap()) - // } else { - // None - // } - // }) - // .collect(); - // - // Ok(XvcStorageListEvent { - // guid: self.guid.clone(), - // paths, - // }) - // } - // - // Err(err) => { - // error!(output, "{}", err); - // Err(Error::S3Error { source: err }) - // } - // } - // } \{ - // todo!() - // \} - // - // async fn a_send( - // &self, - // output: &XvcOutputSender, - // xvc_root: &xvc_core::XvcRoot, - // paths: &[xvc_core::XvcCachePath], - // _force: bool, - // ) -> crate::Result { - // let repo_guid = xvc_root - // .config() - // .guid() - // .ok_or_else(|| crate::Error::NoRepositoryGuidFound)?; - // let mut copied_paths = Vec::::new(); - // - // let bucket = self.get_bucket()?; - // - // for cache_path in paths { - // watch!(cache_path); - // let remote_path = self.build_remote_path(&repo_guid, cache_path); - // let abs_cache_path = cache_path.to_absolute_path(xvc_root); - // watch!(abs_cache_path); - // - // let mut path = tokio::fs::File::open(&abs_cache_path).await?; - // watch!(path); - // - // let res_response = bucket - // .put_object_stream(&mut path, remote_path.as_str()) - // .await; - // - // match res_response { - // Ok(_) => { - // info!(output, "{} -> {}", abs_cache_path, remote_path.as_str()); - // copied_paths.push(remote_path); - // watch!(copied_paths.len()); - // } - // Err(err) => { - // error!(output, "{}", err); - // } - // } - // } - // - // Ok(XvcStorageSendEvent { - // guid: self.guid.clone(), - // paths: copied_paths, - // }) - // } \{ - // todo!() - // \} - // - // async fn a_receive( - // &self, - // output: &XvcOutputSender, - // xvc_root: &xvc_core::XvcRoot, - // paths: &[xvc_core::XvcCachePath], - // _force: bool, - // ) -> Result<(XvcStorageTempDir, XvcStorageReceiveEvent)> { - // let repo_guid = xvc_root - // .config() - // .guid() - // .ok_or_else(|| crate::Error::NoRepositoryGuidFound)?; - // let mut copied_paths = Vec::::new(); - // - // let bucket = self.get_bucket()?; - // let temp_dir = XvcStorageTempDir::new()?; - // - // for cache_path in paths { - // watch!(cache_path); - // let remote_path = self.build_remote_path(&repo_guid, cache_path); - // let abs_cache_dir = temp_dir.temp_cache_dir(cache_path)?; - // fs::create_dir_all(&abs_cache_dir)?; - // let abs_cache_path = temp_dir.temp_cache_path(cache_path)?; - // watch!(abs_cache_path); - // - // let response_data_stream = bucket.get_object_stream(remote_path.as_str()).await; - // - // match response_data_stream { - // Ok(mut response) => { - // info!(output, "{} -> {}", remote_path.as_str(), abs_cache_path); - // let mut async_cache_path = tokio::fs::File::create(&abs_cache_path).await?; - // while let Some(chunk) = response.bytes().next().await { - // async_cache_path.write_all(&chunk).await?; - // } - // copied_paths.push(remote_path); - // watch!(copied_paths.len()); - // } - // Err(err) => { - // error!(output, "{}", err); - // } - // } - // } - // - // Ok(( - // temp_dir, - // XvcStorageReceiveEvent { - // guid: self.guid.clone(), - // paths: copied_paths, - // }, - // )) - // } \{ - // todo!() - // \} - // - // async fn a_delete( - // &self, - // output: &XvcOutputSender, - // xvc_root: &xvc_core::XvcRoot, - // paths: &[XvcCachePath], - // ) -> Result { - // let repo_guid = xvc_root - // .config() - // .guid() - // .ok_or_else(|| crate::Error::NoRepositoryGuidFound)?; - // let mut deleted_paths = Vec::::new(); - // - // let bucket = self.get_bucket()?; - // - // for cache_path in paths { - // watch!(cache_path); - // let remote_path = self.build_remote_path(&repo_guid, cache_path); - // bucket.delete_object(remote_path.as_str()).await?; - // info!(output, "[DELETE] {}", remote_path.as_str()); - // deleted_paths.push(remote_path); - // } - // - // Ok(XvcStorageDeleteEvent { - // guid: self.guid.clone(), - // paths: deleted_paths, - // }) - // } \{ - // todo!() - // \} } diff --git a/storage/src/storage/mod.rs b/storage/src/storage/mod.rs index 036b46ee6..7654fddf7 100644 --- a/storage/src/storage/mod.rs +++ b/storage/src/storage/mod.rs @@ -73,7 +73,7 @@ pub enum XvcStorage { #[cfg(feature = "digital-ocean")] DigitalOcean(digital_ocean::XvcDigitalOceanStorage), } -persist!(XvcStorage, "remote"); +persist!(XvcStorage, "storage"); impl Display for XvcStorage { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { @@ -117,19 +117,19 @@ impl Display for XvcStorage { XvcStorage::Minio(mr) => write!( f, "Minio: {}\t{}\t{}.{}/{}", - mr.name, mr.guid, mr.endpoint, mr.bucket_name, mr.remote_prefix + mr.name, mr.guid, mr.endpoint, mr.bucket_name, mr.storage_prefix ), #[cfg(feature = "r2")] XvcStorage::R2(r2r) => write!( f, "R2: {}\t{}\t{} {}/{}", - r2r.name, r2r.guid, r2r.account_id, r2r.bucket_name, r2r.remote_prefix + r2r.name, r2r.guid, r2r.account_id, r2r.bucket_name, r2r.storage_prefix ), #[cfg(feature = "gcs")] XvcStorage::Gcs(gcsr) => write!( f, "GCS: {}\t{}\t{}.{}/{}", - gcsr.name, gcsr.guid, gcsr.region, gcsr.bucket_name, gcsr.remote_prefix + gcsr.name, gcsr.guid, gcsr.region, gcsr.bucket_name, gcsr.storage_prefix ), #[cfg(feature = "wasabi")] XvcStorage::Wasabi(wr) => write!( @@ -141,7 +141,7 @@ impl Display for XvcStorage { XvcStorage::DigitalOcean(dor) => write!( f, "DO: {}\t{}\t{}.{}/{}", - dor.name, dor.guid, dor.region, dor.bucket_name, dor.remote_prefix + dor.name, dor.guid, dor.region, dor.bucket_name, dor.storage_prefix ), } } @@ -375,7 +375,7 @@ impl XvcStorageTempDir { /// It uses [RelativePathBuf] internally. #[derive(Clone, Debug, PartialOrd, Ord, PartialEq, Eq, Serialize, Deserialize, Display)] pub struct XvcStoragePath(RelativePathBuf); -persist!(XvcStoragePath, "remote-path"); +persist!(XvcStoragePath, "storage-path"); impl From for XvcStoragePath { fn from(p: String) -> Self { @@ -397,7 +397,7 @@ impl AsRef for XvcStoragePath { } impl XvcStoragePath { - /// The remote path of a cache path is like {guid}/{cache-path} + /// The storage path of a cache path is like {guid}/{cache-path} /// ⚠️ The separator between {guid} and {cache-path} is always / pub fn new(xvc_root: &XvcRoot, local: &XvcCachePath) -> Self { let guid = xvc_root.config().guid().unwrap(); @@ -457,7 +457,7 @@ pub fn get_storage_record( identifier: &StorageIdentifier, ) -> Result { let store: XvcStore = xvc_root.load_store()?; - let remote_store = store.filter(|_, r| match identifier { + let storage_store = store.filter(|_, r| match identifier { StorageIdentifier::Name(ref n) => match r { XvcStorage::Local(r) => r.name == *n, XvcStorage::Generic(r) => r.name == *n, @@ -494,18 +494,18 @@ pub fn get_storage_record( }, }); - if remote_store.is_empty() { + if storage_store.is_empty() { error!(output_snd, "Cannot find remote {}", identifier); } - if remote_store.len() > 1 { - error!(output_snd, "Ambiguous remote identifier: {}", identifier); + if storage_store.len() > 1 { + error!(output_snd, "Ambiguous remote identifier: {} Please use Storage GUID.", identifier); } - let (_, remote) = - remote_store + let (_, storage) = + storage_store .first() - .ok_or_else(|| Error::CannotFindRemoteWithIdentifier { + .ok_or_else(|| Error::CannotFindStorageWithIdentifier { identifier: identifier.clone(), })?; - Ok(remote.clone()) + Ok(storage.clone()) } diff --git a/storage/src/storage/r2.rs b/storage/src/storage/r2.rs index dc6830423..abdef9a4b 100644 --- a/storage/src/storage/r2.rs +++ b/storage/src/storage/r2.rs @@ -19,7 +19,7 @@ use super::{XvcStorageListEvent, XvcStoragePath}; /// Configure a new Cloudflare R2 remote storage. /// -/// `account_id`, `bucket_name`, and `remote_prefix` sets a URL for the +/// `account_id`, `bucket_name`, and `storage_prefix` sets a URL for the /// storage location. /// /// This creates a [XvcR2Storage], calls its @@ -32,19 +32,19 @@ pub fn cmd_new_r2( name: String, account_id: String, bucket_name: String, - remote_prefix: String, + storage_prefix: String, ) -> Result<()> { - let mut remote = XvcR2Storage { + let mut storage = XvcR2Storage { guid: XvcStorageGuid::new(), name, account_id, bucket_name, - remote_prefix, + storage_prefix, }; - watch!(remote); + watch!(storage); - let init_event = remote.init(output_snd, xvc_root)?; + let init_event = storage.init(output_snd, xvc_root)?; watch!(init_event); xvc_root.with_r1nstore_mut(|store: &mut R1NStore| { @@ -52,7 +52,7 @@ pub fn cmd_new_r2( let event_e = xvc_root.new_entity(); store.insert( store_e, - XvcStorage::R2(remote.clone()), + XvcStorage::R2(storage.clone()), event_e, XvcStorageEvent::Init(init_event.clone()), ); @@ -73,12 +73,12 @@ pub struct XvcR2Storage { pub account_id: String, /// Bucket name of the R2 storage pub bucket_name: String, - /// Remote prefix in the bucket - pub remote_prefix: String, + /// Remote path prefix in the bucket + pub storage_prefix: String, } impl XvcR2Storage { - fn remote_specific_credentials(&self) -> Result { + fn storage_specific_credentials(&self) -> Result { Credentials::new( Some(&env::var(format!( "XVC_STORAGE_ACCESS_KEY_ID_{}", @@ -105,8 +105,8 @@ impl XvcR2Storage { } impl XvcS3StorageOperations for XvcR2Storage { - fn remote_prefix(&self) -> String { - self.remote_prefix.clone() + fn storage_prefix(&self) -> String { + self.storage_prefix.clone() } fn guid(&self) -> &XvcStorageGuid { @@ -115,10 +115,10 @@ impl XvcS3StorageOperations for XvcR2Storage { fn bucket_name(&self) -> String { self.bucket_name.clone() } - fn build_remote_path(&self, cache_path: &XvcCachePath) -> XvcStoragePath { + fn build_storage_path(&self, cache_path: &XvcCachePath) -> XvcStoragePath { XvcStoragePath::from(format!( "{}/{}/{}", - self.remote_prefix, + self.storage_prefix, self.guid(), cache_path )) @@ -130,7 +130,7 @@ impl XvcS3StorageOperations for XvcR2Storage { } fn credentials(&self) -> Result { - match self.remote_specific_credentials() { + match self.storage_specific_credentials() { Ok(c) => Ok(c), Err(e1) => match self.storage_type_credentials() { Ok(c) => Ok(c), @@ -161,11 +161,11 @@ impl XvcS3StorageOperations for XvcR2Storage { ) -> Result { let bucket = self.get_bucket()?; let xvc_guid = xvc_root.config().guid().unwrap(); - let prefix = self.remote_prefix.clone(); + let prefix = self.storage_prefix.clone(); let res_list = bucket .list( - format!("{}/{}", self.remote_prefix, xvc_guid), + format!("{}/{}", self.storage_prefix, xvc_guid), Some("/".to_string()), ) .await; diff --git a/storage/src/storage/rsync.rs b/storage/src/storage/rsync.rs index 24e242cbb..6241e3a9e 100644 --- a/storage/src/storage/rsync.rs +++ b/storage/src/storage/rsync.rs @@ -60,7 +60,7 @@ pub fn cmd_new_rsync( Ok(()) } -/// Specifies an Rsync remote +/// Specifies an Rsync remote storage #[derive(Clone, Debug, PartialOrd, Ord, PartialEq, Eq, Serialize, Deserialize)] pub struct XvcRsyncStorage { /// The GUID of the storage @@ -237,8 +237,8 @@ impl XvcRsyncStorage { xvc_guid: &str, cache_path: &XvcCachePath, ) -> Result { - let remote_dir = self.ssh_cache_dir(xvc_guid, cache_path); - self.ssh_cmd(ssh_executable, &format!("mkdir -p '{}'", remote_dir)) + let storage_dir = self.ssh_cache_dir(xvc_guid, cache_path); + self.ssh_cmd(ssh_executable, &format!("mkdir -p '{}'", storage_dir)) } } @@ -265,18 +265,18 @@ impl XvcStorageOperations for XvcRsyncStorage { fs::write(&local_guid_path, format!("{}", self.guid))?; watch!(local_guid_path); - let remote_guid_path = self.rsync_path_url(XVC_STORAGE_GUID_FILENAME); + let storage_guid_path = self.rsync_path_url(XVC_STORAGE_GUID_FILENAME); - watch!(remote_guid_path); + watch!(storage_guid_path); let rsync_result = - self.rsync_copy_to_storage(&rsync_executable, &local_guid_path, &remote_guid_path)?; + self.rsync_copy_to_storage(&rsync_executable, &local_guid_path, &storage_guid_path)?; watch!(rsync_result); info!( output, "Initialized:\n{}\n{}\n", - remote_guid_path, + torage_guid_path, rsync_result.stdout_str() ); @@ -343,11 +343,11 @@ impl XvcStorageOperations for XvcRsyncStorage { let mut storage_paths = Vec::::with_capacity(paths.len()); paths.iter().for_each(|cache_path| { let local_path = cache_path.to_absolute_path(xvc_root); - let remote_url = self.rsync_cache_url(&xvc_guid, cache_path); - let _remote_dir_res = self.create_storage_dir(&ssh_executable, &xvc_guid, cache_path); + let storage_url = self.rsync_cache_url(&xvc_guid, cache_path); + uwr!(self.create_storage_dir(&ssh_executable, &xvc_guid, cache_path), output); // TODO: Handle possible error. let cmd_output = - self.rsync_copy_to_storage(&rsync_executable, &local_path, &remote_url); + self.rsync_copy_to_storage(&rsync_executable, &local_path, &storage_url); match cmd_output { Ok(cmd_output) => { diff --git a/storage/src/storage/s3.rs b/storage/src/storage/s3.rs index 4e32b86fb..3d3d893df 100644 --- a/storage/src/storage/s3.rs +++ b/storage/src/storage/s3.rs @@ -19,7 +19,7 @@ use super::XvcStoragePath; /// Configure a new Amazon Web Services S3 remote storage. /// -/// `bucket_name`, `region` and `remote_prefix` sets a URL for the storage +/// `bucket_name`, `region` and `storage_prefix` sets a URL for the storage /// location. /// /// This creates a [XvcS3Storage], calls its @@ -33,7 +33,7 @@ pub fn cmd_new_s3( bucket_name: String, storage_prefix: String, ) -> Result<()> { - let mut remote = XvcS3Storage { + let mut storage = XvcS3Storage { guid: XvcStorageGuid::new(), name, region, @@ -41,9 +41,9 @@ pub fn cmd_new_s3( storage_prefix, }; - watch!(remote); + watch!(storage); - let init_event = remote.init(output_snd, xvc_root)?; + let init_event = storage.init(output_snd, xvc_root)?; watch!(init_event); xvc_root.with_r1nstore_mut(|store: &mut R1NStore| { @@ -51,7 +51,7 @@ pub fn cmd_new_s3( let event_e = xvc_root.new_entity(); store.insert( store_e, - XvcStorage::S3(remote.clone()), + XvcStorage::S3(storage.clone()), event_e, XvcStorageEvent::Init(init_event.clone()), ); @@ -70,7 +70,7 @@ pub struct XvcS3Storage { /// `bucket_name.region.s3.amazonaws.com/storage_prefix/.xvc-guid` to identify the /// remote location. pub guid: XvcStorageGuid, - /// Name of the remote to be used in commands. + /// Name of the remote storage to be used in commands. /// /// It doesn't have to be unique, though in practice setting unique names is /// preferred. @@ -89,7 +89,7 @@ pub struct XvcS3Storage { } impl XvcS3Storage { - fn remote_specific_credentials(&self) -> Result { + fn storage_specific_credentials(&self) -> Result { Credentials::new( Some(&env::var(format!( "XVC_STORAGE_ACCESS_KEY_ID_{}", @@ -116,7 +116,7 @@ impl XvcS3Storage { } impl XvcS3StorageOperations for XvcS3Storage { - fn remote_prefix(&self) -> String { + fn storage_prefix(&self) -> String { self.storage_prefix.clone() } @@ -133,7 +133,7 @@ impl XvcS3StorageOperations for XvcS3Storage { } fn credentials(&self) -> Result { - match self.remote_specific_credentials() { + match self.storage_specific_credentials() { Ok(c) => Ok(c), Err(e1) => match self.storage_type_credentials() { Ok(c) => Ok(c), @@ -151,13 +151,13 @@ impl XvcS3StorageOperations for XvcS3Storage { self.bucket_name.clone() } - async fn write_remote_guid(&self) -> Result<()> { + async fn write_storage_guid(&self) -> Result<()> { let guid_str = self.guid().to_string(); let guid_bytes = guid_str.as_bytes(); let bucket = self.get_bucket()?; let response = bucket .put_object( - format!("{}/{}", self.remote_prefix(), XVC_STORAGE_GUID_FILENAME), + format!("{}/{}", self.storage_prefix(), XVC_STORAGE_GUID_FILENAME), guid_bytes, ) .await; @@ -168,11 +168,11 @@ impl XvcS3StorageOperations for XvcS3Storage { } } - fn build_remote_path(&self, cache_path: &XvcCachePath) -> XvcStoragePath { + fn build_storage_path(&self, cache_path: &XvcCachePath) -> XvcStoragePath { XvcStoragePath::from(format!( "{}/{}/{}/{}", self.bucket_name(), - self.remote_prefix(), + self.storage_prefix(), self.guid(), cache_path )) diff --git a/storage/src/storage/wasabi.rs b/storage/src/storage/wasabi.rs index 19b38f372..639d18cb7 100644 --- a/storage/src/storage/wasabi.rs +++ b/storage/src/storage/wasabi.rs @@ -16,7 +16,7 @@ use anyhow::anyhow; use super::async_common::XvcS3StorageOperations; use super::XvcStoragePath; -/// Configure a new Wasabi storage remote. +/// Configure a new Wasabi remote storage. /// /// `bucket_name`, `endpoint` and `storage_prefix` sets a URL for the storage /// location. @@ -32,7 +32,7 @@ pub(crate) fn cmd_new_wasabi( endpoint: String, storage_prefix: String, ) -> Result<()> { - let mut remote = XvcWasabiStorage { + let mut storage = XvcWasabiStorage { guid: XvcStorageGuid::new(), name, endpoint, @@ -40,9 +40,9 @@ pub(crate) fn cmd_new_wasabi( storage_prefix, }; - watch!(remote); + watch!(storage); - let init_event = remote.init(output_snd, xvc_root)?; + let init_event = storage.init(output_snd, xvc_root)?; watch!(init_event); xvc_root.with_r1nstore_mut(|store: &mut R1NStore| { @@ -50,7 +50,7 @@ pub(crate) fn cmd_new_wasabi( let event_e = xvc_root.new_entity(); store.insert( store_e, - XvcStorage::Wasabi(remote.clone()), + XvcStorage::Wasabi(storage.clone()), event_e, XvcStorageEvent::Init(init_event.clone()), ); @@ -91,7 +91,7 @@ pub struct XvcWasabiStorage { } impl XvcWasabiStorage { - fn remote_specific_credentials(&self) -> Result { + fn storage_specific_credentials(&self) -> Result { Credentials::new( Some(&env::var(format!( "XVC_STORAGE_ACCESS_KEY_ID_{}", @@ -118,7 +118,7 @@ impl XvcWasabiStorage { } impl XvcS3StorageOperations for XvcWasabiStorage { - fn remote_prefix(&self) -> String { + fn storage_prefix(&self) -> String { self.storage_prefix.clone() } @@ -147,7 +147,7 @@ impl XvcS3StorageOperations for XvcWasabiStorage { } fn credentials(&self) -> Result { - match self.remote_specific_credentials() { + match self.storage_specific_credentials() { Ok(c) => Ok(c), Err(e1) => match self.storage_type_credentials() { Ok(c) => Ok(c), @@ -161,7 +161,7 @@ impl XvcS3StorageOperations for XvcWasabiStorage { .map_err(|e| e.into()) } - fn build_remote_path(&self, cache_path: &XvcCachePath) -> XvcStoragePath { + fn build_storage_path(&self, cache_path: &XvcCachePath) -> XvcStoragePath { XvcStoragePath::from(format!( "{}/{}/{}", self.storage_prefix, diff --git a/test_helper/Cargo.toml b/test_helper/Cargo.toml index be249be60..14fc22b26 100644 --- a/test_helper/Cargo.toml +++ b/test_helper/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-test-helper" -version = "0.6.8-alpha.1" +version = "0.6.8-alpha.2" edition = "2021" description = "Unit test helper functions for Xvc" authors = ["Emre Şahin "] @@ -23,7 +23,7 @@ debug = true # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html [dependencies] -xvc-logging = { version = "0.6.8-alpha.1", path = "../logging/" } +xvc-logging = { version = "0.6.8-alpha.2", path = "../logging/" } rand = "^0.8" log = "^0.4" diff --git a/walker/Cargo.toml b/walker/Cargo.toml index f99ce2d2d..beb83fc2b 100644 --- a/walker/Cargo.toml +++ b/walker/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-walker" -version = "0.6.8-alpha.1" +version = "0.6.8-alpha.2" edition = "2021" description = "Xvc parallel file system walker with ignore features" authors = ["Emre Şahin "] @@ -19,7 +19,7 @@ crate-type = ["rlib"] debug = true [dependencies] -xvc-logging = { version = "0.6.8-alpha.1", path = "../logging" } +xvc-logging = { version = "0.6.8-alpha.2", path = "../logging" } globset = "^0.4" ## Parallelization @@ -41,7 +41,7 @@ itertools = "^0.12" regex = "^1.10" [dev-dependencies] -xvc-test-helper = { path = "../test_helper/", version = "0.6.8-alpha.1" } +xvc-test-helper = { path = "../test_helper/", version = "0.6.8-alpha.2" } test-case = "^3.3" [package.metadata.cargo-udeps.ignore] diff --git a/workflow_tests/Cargo.toml b/workflow_tests/Cargo.toml index 22a59a899..34f503661 100644 --- a/workflow_tests/Cargo.toml +++ b/workflow_tests/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-workflow-tests" -version = "0.6.8-alpha.1" +version = "0.6.8-alpha.2" edition = "2021" description = "Integration test suite for Xvc" authors = ["Emre Şahin "] @@ -23,15 +23,15 @@ debug = true [dependencies] -xvc = { version = "0.6.8-alpha.1", path = "../lib" } -xvc-config = { version = "0.6.8-alpha.1", path = "../config" } -xvc-core = { version = "0.6.8-alpha.1", path = "../core" } -xvc-logging = { version = "0.6.8-alpha.1", path = "../logging" } -xvc-ecs = { version = "0.6.8-alpha.1", path = "../ecs" } -xvc-file = { version = "0.6.8-alpha.1", path = "../file" } -xvc-pipeline = { version = "0.6.8-alpha.1", path = "../pipeline" } -xvc-walker = { version = "0.6.8-alpha.1", path = "../walker" } -xvc-storage = { version = "0.6.8-alpha.1", path = "../storage" } +xvc = { version = "0.6.8-alpha.2", path = "../lib" } +xvc-config = { version = "0.6.8-alpha.2", path = "../config" } +xvc-core = { version = "0.6.8-alpha.2", path = "../core" } +xvc-logging = { version = "0.6.8-alpha.2", path = "../logging" } +xvc-ecs = { version = "0.6.8-alpha.2", path = "../ecs" } +xvc-file = { version = "0.6.8-alpha.2", path = "../file" } +xvc-pipeline = { version = "0.6.8-alpha.2", path = "../pipeline" } +xvc-walker = { version = "0.6.8-alpha.2", path = "../walker" } +xvc-storage = { version = "0.6.8-alpha.2", path = "../storage" } ## packages for testing clap = { version = "^4.4", features = ["derive"] } @@ -70,7 +70,7 @@ proptest = "^1.4" test-case = "^3.3" globset = "^0.4" escargot = "^0.5" -xvc-test-helper = { version = "0.6.8-alpha.1", path = "../test_helper" } +xvc-test-helper = { version = "0.6.8-alpha.2", path = "../test_helper" } shellfn = "^0.1" jwalk = "^0.8" anyhow = "^1.0" diff --git a/workflow_tests/tests/test_init_1.rs b/workflow_tests/tests/test_init_1.rs index 49d652fd8..b74575768 100644 --- a/workflow_tests/tests/test_init_1.rs +++ b/workflow_tests/tests/test_init_1.rs @@ -8,7 +8,7 @@ use xvc::init::InitCLI; use xvc::error::Result; #[test] -fn test_remote_add() -> Result<()> { +fn test_storage_add() -> Result<()> { common::test_logging(LevelFilter::Trace); let _the_dir = common::run_in_temp_dir(); let xvc_root = xvc::init::run( diff --git a/workflow_tests/tests/test_storage_new_s3.rs b/workflow_tests/tests/test_storage_new_s3.rs index 7a14d64dd..d42725b0c 100644 --- a/workflow_tests/tests/test_storage_new_s3.rs +++ b/workflow_tests/tests/test_storage_new_s3.rs @@ -252,7 +252,7 @@ fn test_storage_new_s3() -> Result<()> { assert!(n_storage_files_after == n_local_files_after_pull); assert!(PathBuf::from(the_file).exists()); - // Set remote specific passwords and remove AWS ones + // Set storage specific passwords and remove AWS ones env::set_var("XVC_STORAGE_ACCESS_KEY_ID_s3-storage", access_key); env::set_var("XVC_STORAGE_SECRET_KEY_s3-storage", secret_key); From 0189b38859649424af28b321740041086d026541 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Mon, 17 Jun 2024 12:22:51 +0300 Subject: [PATCH 014/257] update --- file/src/send/mod.rs | 2 -- storage/src/storage/rsync.rs | 2 +- workflow_tests/tests/test_pipeline_dag.rs | 1 + 3 files changed, 2 insertions(+), 3 deletions(-) diff --git a/file/src/send/mod.rs b/file/src/send/mod.rs index 8298719e3..6c71246a6 100644 --- a/file/src/send/mod.rs +++ b/file/src/send/mod.rs @@ -7,8 +7,6 @@ use crate::Result; use clap::Parser; -use humantime; - use xvc_core::{ContentDigest, XvcCachePath, XvcFileType, XvcMetadata, XvcRoot}; use xvc_ecs::{HStore, XvcStore}; use xvc_logging::{error, watch, XvcOutputSender}; diff --git a/storage/src/storage/rsync.rs b/storage/src/storage/rsync.rs index 6241e3a9e..aaa054440 100644 --- a/storage/src/storage/rsync.rs +++ b/storage/src/storage/rsync.rs @@ -276,7 +276,7 @@ impl XvcStorageOperations for XvcRsyncStorage { info!( output, "Initialized:\n{}\n{}\n", - torage_guid_path, + storage_guid_path, rsync_result.stdout_str() ); diff --git a/workflow_tests/tests/test_pipeline_dag.rs b/workflow_tests/tests/test_pipeline_dag.rs index ae57403ab..d15ed39bd 100644 --- a/workflow_tests/tests/test_pipeline_dag.rs +++ b/workflow_tests/tests/test_pipeline_dag.rs @@ -68,6 +68,7 @@ fn test_pipeline_dag() -> Result<()> { xsd("glob_dep", "--glob", "*.txt")?; xsd("glob_dep", "--glob-items", "data/*")?; xsd("glob_dep", "--param", "params.yaml:model.conv_units")?; + // FIXME: There is an error here regarding regex format xsd("glob_dep", "--regex", "requirements.txt:^tensorflow")?; xsom("glob_dep", "glob_dep.json")?; xsof("glob_dep", "def.txt")?; From e6b17bc1eb21712c47c34a919896bf2bf353151e Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Tue, 2 Jul 2024 14:01:24 +0300 Subject: [PATCH 015/257] fix compile error --- storage/src/storage/mod.rs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/storage/src/storage/mod.rs b/storage/src/storage/mod.rs index f6b283d36..235a8722c 100644 --- a/storage/src/storage/mod.rs +++ b/storage/src/storage/mod.rs @@ -108,10 +108,10 @@ impl Display for XvcStorage { ), #[cfg(feature = "s3")] - XvcStorage::S3(s3r) => write!( + XvcStorage::S3(r) => write!( f, "S3: {}\t{}\t{}.{}/{}", - e3r.name, s3r.guid, s3r.region, s3r.bucket_name, s3r.storage_prefix + r.name, r.guid, r.region, r.bucket_name, r.storage_prefix ), #[cfg(feature = "minio")] XvcStorage::Minio(mr) => write!( From 4653b6931c4cc886a28d4e3be70630fda68ef453 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Wed, 3 Jul 2024 12:22:26 +0300 Subject: [PATCH 016/257] TIDY - Rename XvcConfigInitParams to XvcConfigParams --- config/src/lib.rs | 8 ++++---- core/src/types/xvcroot.rs | 12 ++++++------ file/src/hash/mod.rs | 4 ++-- file/src/lib.rs | 4 ++-- lib/src/api.rs | 2 +- lib/src/cli/mod.rs | 6 +++--- lib/src/init/mod.rs | 4 ++-- 7 files changed, 20 insertions(+), 20 deletions(-) diff --git a/config/src/lib.rs b/config/src/lib.rs index 80693779a..960023c29 100755 --- a/config/src/lib.rs +++ b/config/src/lib.rs @@ -155,7 +155,7 @@ pub struct XvcConfigMap { /// /// It's possible to ignore certain sources by supplying `None` to their values here. #[derive(Debug, Clone)] -pub struct XvcConfigInitParams { +pub struct XvcConfigParams { /// The default configuration for the project. /// It should contain all default values as a TOML document. /// Xvc produces this in [xvc_core::default_configuration]. @@ -203,7 +203,7 @@ pub struct XvcConfig { /// The current configuration map, updated cascadingly pub the_config: HashMap, /// The init params used to create this config - pub init_params: XvcConfigInitParams, + pub init_params: XvcConfigParams, } impl fmt::Display for XvcConfig { @@ -225,7 +225,7 @@ impl XvcConfig { /// Loads the default configuration from `p`. /// /// The configuration must be a valid TOML document. - fn default_conf(p: &XvcConfigInitParams) -> Self { + fn default_conf(p: &XvcConfigParams) -> Self { let default_conf = p .default_configuration .parse::() @@ -482,7 +482,7 @@ impl XvcConfig { /// Loads all config files /// Overrides all options with the given key=value style options in the /// command line - pub fn new(p: XvcConfigInitParams) -> Result { + pub fn new(p: XvcConfigParams) -> Result { let mut config = XvcConfig::default_conf(&p); config.current_dir = XvcConfigOption { diff --git a/core/src/types/xvcroot.rs b/core/src/types/xvcroot.rs index a0705f39f..1f7b5124a 100644 --- a/core/src/types/xvcroot.rs +++ b/core/src/types/xvcroot.rs @@ -13,7 +13,7 @@ use xvc_ecs::{XvcEntity, XvcEntityGenerator}; use xvc_logging::watch; use xvc_walker::AbsolutePath; -use xvc_config::{XvcConfig, XvcConfigInitParams}; +use xvc_config::{XvcConfig, XvcConfigParams}; use crate::error::{Error, Result}; use crate::GITIGNORE_INITIAL_CONTENT; @@ -69,7 +69,7 @@ impl Deref for XvcRootInner { /// The path is not required to be the root of the repository. /// This function searches for the root of the repository using /// [XvcRoot::find_root] and uses it as the root. -pub fn load_xvc_root(config_opts: XvcConfigInitParams) -> Result { +pub fn load_xvc_root(config_opts: XvcConfigParams) -> Result { let path = config_opts.current_dir.as_ref(); match XvcRootInner::find_root(path) { @@ -85,7 +85,7 @@ pub fn load_xvc_root(config_opts: XvcConfigInitParams) -> Result { /// Creates a new .xvc dir in `path` and initializes a directory. /// *Warning:* This should only be used in `xvc init`, not in other commands. -pub fn init_xvc_root(path: &Path, config_opts: XvcConfigInitParams) -> Result { +pub fn init_xvc_root(path: &Path, config_opts: XvcConfigParams) -> Result { match XvcRootInner::find_root(path) { Ok(abs_path) => Err(Error::CannotNestXvcRepositories { path: abs_path.to_path_buf(), @@ -104,7 +104,7 @@ pub fn init_xvc_root(path: &Path, config_opts: XvcConfigInitParams) -> Result Result Result { + pub fn new(absolute_path: AbsolutePath, config_opts: XvcConfigParams) -> Result { let xvc_dir = absolute_path.join(XvcRootInner::XVC_DIR); let local_config_path = xvc_dir.join(XvcRootInner::LOCAL_CONFIG_PATH); let project_config_path = xvc_dir.join(XvcRootInner::PROJECT_CONFIG_PATH); - let config_opts = XvcConfigInitParams { + let config_opts = XvcConfigParams { project_config_path: Some(project_config_path.clone()), local_config_path: Some(local_config_path.clone()), default_configuration: config_opts.default_configuration, diff --git a/file/src/hash/mod.rs b/file/src/hash/mod.rs index 05b74887f..b7db288f3 100644 --- a/file/src/hash/mod.rs +++ b/file/src/hash/mod.rs @@ -7,7 +7,7 @@ use clap::Parser; use crossbeam_channel::unbounded; use log::warn; use std::{env, path::PathBuf}; -use xvc_config::{FromConfigKey, UpdateFromXvcConfig, XvcConfig, XvcConfigInitParams}; +use xvc_config::{FromConfigKey, UpdateFromXvcConfig, XvcConfig, XvcConfigParams}; use xvc_core::ContentDigest; use xvc_core::{ util::file::{path_metadata_channel, pipe_filter_path_errors}, @@ -63,7 +63,7 @@ pub fn cmd_hash( ) -> Result<()> { let conf = match xvc_root { Some(xvc_root) => xvc_root.config().clone(), - None => XvcConfig::new(XvcConfigInitParams { + None => XvcConfig::new(XvcConfigParams { default_configuration: xvc_core::default_project_config(false), current_dir: AbsolutePath::from(env::current_dir()?), include_system_config: true, diff --git a/file/src/lib.rs b/file/src/lib.rs index d7eb42a8d..e95803b98 100755 --- a/file/src/lib.rs +++ b/file/src/lib.rs @@ -47,7 +47,7 @@ use std::io; use std::io::Write; use std::path::Path; use std::path::PathBuf; -use xvc_config::XvcConfigInitParams; +use xvc_config::XvcConfigParams; use xvc_config::XvcVerbosity; use xvc_core::default_project_config; use xvc_core::types::xvcroot::load_xvc_root; @@ -265,7 +265,7 @@ pub fn dispatch(cli_opts: XvcFileCLI) -> Result<()> { AbsolutePath::from(std::env::current_dir()?.join(dir).canonicalize()?) }; // try to create root - let xvc_config_params = XvcConfigInitParams { + let xvc_config_params = XvcConfigParams { current_dir, include_system_config: !cli_opts.no_system_config, include_user_config: !cli_opts.no_user_config, diff --git a/lib/src/api.rs b/lib/src/api.rs index f6ebdd68b..388bc4dca 100644 --- a/lib/src/api.rs +++ b/lib/src/api.rs @@ -19,7 +19,7 @@ pub use xvc_logging::warn; pub use xvc_logging::watch; pub use xvc_config::XvcConfig; -pub use xvc_config::XvcConfigInitParams; +pub use xvc_config::XvcConfigParams; pub use xvc_config::XvcConfigOptionSource; pub use xvc_core::AbsolutePath; diff --git a/lib/src/cli/mod.rs b/lib/src/cli/mod.rs index 9e468eafd..0a3ec99ce 100644 --- a/lib/src/cli/mod.rs +++ b/lib/src/cli/mod.rs @@ -20,7 +20,7 @@ use xvc_core::types::xvcroot::XvcRootInner; use xvc_core::XvcRoot; use xvc_logging::{debug, error, uwr, XvcOutputLine}; -use xvc_config::{XvcConfigInitParams, XvcVerbosity}; +use xvc_config::{XvcConfigParams, XvcVerbosity}; use xvc_core::aliases; use xvc_core::check_ignore; use xvc_core::default_project_config; @@ -456,8 +456,8 @@ pub fn dispatch(cli_opts: cli::XvcCLI) -> Result { } -fn get_xvc_config_params(cli_opts: &XvcCLI) -> XvcConfigInitParams { - let xvc_config_params = XvcConfigInitParams { +fn get_xvc_config_params(cli_opts: &XvcCLI) -> XvcConfigParams { + let xvc_config_params = XvcConfigParams { current_dir: AbsolutePath::from(&cli_opts.workdir), include_system_config: !cli_opts.no_system_config, include_user_config: !cli_opts.no_user_config, diff --git a/lib/src/init/mod.rs b/lib/src/init/mod.rs index 87478a64e..e10c3309c 100644 --- a/lib/src/init/mod.rs +++ b/lib/src/init/mod.rs @@ -5,7 +5,7 @@ use log::{info, warn}; use std::env; use std::fs; use std::path::PathBuf; -use xvc_config::XvcConfigInitParams; +use xvc_config::XvcConfigParams; use xvc_core::default_project_config; use xvc_core::types::xvcroot::init_xvc_root; use xvc_core::util::git::inside_git; @@ -84,7 +84,7 @@ pub fn run(xvc_root_opt: Option<&XvcRoot>, opts: InitCLI) -> Result { } } let default_configuration = default_project_config(!opts.no_git); - let config_opts = XvcConfigInitParams { + let config_opts = XvcConfigParams { default_configuration, current_dir: AbsolutePath::from(&path), include_system_config: true, From 4acf6f2c1a41dbd78714eb4a7ca68dc06df851f8 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Thu, 11 Jul 2024 20:20:21 +0300 Subject: [PATCH 017/257] derive Clone for CLI structs --- core/src/aliases/mod.rs | 2 +- core/src/root/mod.rs | 2 +- lib/src/cli/mod.rs | 2 +- pipeline/src/lib.rs | 2 +- storage/src/lib.rs | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-) diff --git a/core/src/aliases/mod.rs b/core/src/aliases/mod.rs index dbfcc91fc..3e550cc36 100644 --- a/core/src/aliases/mod.rs +++ b/core/src/aliases/mod.rs @@ -4,7 +4,7 @@ use clap::Parser; use xvc_logging::{output, XvcOutputSender}; -#[derive(Debug, Parser)] +#[derive(Debug, Parser, Clone)] #[command(name = "aliases")] /// Print aliases in the common format to be added to `.zsh_aliases`, `.bash_aliases` or /// `.profile`. diff --git a/core/src/root/mod.rs b/core/src/root/mod.rs index f568d2aa5..db8ead2bb 100644 --- a/core/src/root/mod.rs +++ b/core/src/root/mod.rs @@ -6,7 +6,7 @@ use clap::Parser; use relative_path::RelativePath; use xvc_logging::{output, watch, XvcOutputSender}; -#[derive(Debug, Parser)] +#[derive(Debug, Parser, Clone)] #[command(name = "root")] /// Get the Xvc root directory for the current project pub struct RootCLI { diff --git a/lib/src/cli/mod.rs b/lib/src/cli/mod.rs index 0a3ec99ce..b019935e2 100644 --- a/lib/src/cli/mod.rs +++ b/lib/src/cli/mod.rs @@ -185,7 +185,7 @@ impl FromStr for XvcCLI { } /// Xvc subcommands -#[derive(Debug, Parser)] +#[derive(Debug, Parser, Clone)] #[command(rename_all = "kebab-case")] pub enum XvcSubCommand { /// File and directory management commands diff --git a/pipeline/src/lib.rs b/pipeline/src/lib.rs index 2b492c3fd..036af3c2d 100755 --- a/pipeline/src/lib.rs +++ b/pipeline/src/lib.rs @@ -52,7 +52,7 @@ use crate::pipeline::XvcStepInvalidate; pub use crate::pipeline::api::run::RunCLI; /// Pipeline management commands -#[derive(Debug, Parser)] +#[derive(Debug, Parser, Clone)] #[command(name = "pipeline")] pub struct PipelineCLI { /// Name of the pipeline this command applies to diff --git a/storage/src/lib.rs b/storage/src/lib.rs index d767b6676..2d2c4ab99 100755 --- a/storage/src/lib.rs +++ b/storage/src/lib.rs @@ -25,7 +25,7 @@ use xvc_core::XvcRoot; use xvc_logging::{output, XvcOutputSender}; /// Storage (on the cloud) management commands -#[derive(Debug, Parser)] +#[derive(Debug, Parser, Clone)] #[command(name = "storage", about = "")] pub struct StorageCLI { /// Subcommand for storage management From 2471651a68e139d2a91ebe21a82e6c8c7783e7cf Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Thu, 11 Jul 2024 20:20:25 +0300 Subject: [PATCH 018/257] minor fix --- lib/src/git.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/src/git.rs b/lib/src/git.rs index c45a42979..38cb29b39 100644 --- a/lib/src/git.rs +++ b/lib/src/git.rs @@ -170,7 +170,7 @@ pub fn git_auto_commit( &[ "add", "--verbose", - &xvc_dir_str, + xvc_dir_str, "*.gitignore", "*.xvcignore", ], From ec3773589913ac243d33d0b9450bfb6f4c355f52 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sat, 13 Jul 2024 11:55:33 +0300 Subject: [PATCH 019/257] Renamed ITEMS env variables --- README.md | 12 +-- ...xvc-pipeline-step-dependency-glob-items.md | 10 +-- ...xvc-pipeline-step-dependency-line-items.md | 10 +-- ...vc-pipeline-step-dependency-regex-items.md | 12 +-- pipeline/src/pipeline/mod.rs | 74 +++++++++---------- 5 files changed, 59 insertions(+), 59 deletions(-) diff --git a/README.md b/README.md index 83d8a5007..d67ce8722 100644 --- a/README.md +++ b/README.md @@ -149,12 +149,12 @@ and [other steps](https://docs.xvc.dev/ref/xvc-pipeline-step-dependency#step-dep Suppose you're only interested in the IQ scores of those with _Dr._ in front of their names and how they differ from the rest in the dataset we created. Let's create a regex search dependency to the data file that will show all _doctors_ IQ scores. ```console -$ xvc pipeline step new --step-name dr-iq --command 'echo "${XVC_REGEX_ADDED_ITEMS}" >> dr-iq-scores.csv ' +$ xvc pipeline step new --step-name dr-iq --command 'echo "${XVC_ADDED_REGEX_ITEMS}" >> dr-iq-scores.csv ' $ xvc pipeline step dependency --step-name dr-iq --regex-items 'random_names_iq_scores.csv:/^Dr\..*' ``` -The first line specifies a command, when run writes `${XVC_REGEX_ADDED_ITEMS}` environment variable to `dr-iq-scores.csv` file. -The second line specifies the dependency which will also populate the `$[XVC_REGEX_ADDED_ITEMS]` environment variable in the command. +The first line specifies a command, when run writes `${XVC_ADDED_REGEX_ITEMS}` environment variable to `dr-iq-scores.csv` file. +The second line specifies the dependency which will also populate the `$[XVC_ADDED_REGEX_ITEMS]` environment variable in the command. Some dependency types like [regex items], [line items] and [glob items] inject environment variables in the commands they are a dependency. @@ -164,7 +164,7 @@ When you run the pipeline again, a file named `dr-iq-scores.csv` will be created ```console $ xvc pipeline run -[DONE] dr-iq (echo "${XVC_REGEX_ADDED_ITEMS}" >> dr-iq-scores.csv ) +[DONE] dr-iq (echo "${XVC_ADDED_REGEX_ITEMS}" >> dr-iq-scores.csv ) $ cat dr-iq-scores.csv Dr. Brian Shaffer,122 @@ -181,7 +181,7 @@ We are using this feature to get lines starting with `Dr.` from the file and wri $ zsh -cl 'echo "Dr. Albert Einstein,144" >> random_names_iq_scores.csv' $ xvc pipeline run -[DONE] dr-iq (echo "${XVC_REGEX_ADDED_ITEMS}" >> dr-iq-scores.csv ) +[DONE] dr-iq (echo "${XVC_ADDED_REGEX_ITEMS}" >> dr-iq-scores.csv ) $ cat dr-iq-scores.csv Dr. Brian Shaffer,122 @@ -292,7 +292,7 @@ $ cat my-pipeline.json "outputs": [] }, { - "command": "echo /"${XVC_REGEX_ADDED_ITEMS}/" >> dr-iq-scores.csv ", + "command": "echo /"${XVC_ADDED_REGEX_ITEMS}/" >> dr-iq-scores.csv ", "dependencies": [ { "RegexItems": { diff --git a/book/src/ref/xvc-pipeline-step-dependency-glob-items.md b/book/src/ref/xvc-pipeline-step-dependency-glob-items.md index 530b29799..9e2bc5c76 100644 --- a/book/src/ref/xvc-pipeline-step-dependency-glob-items.md +++ b/book/src/ref/xvc-pipeline-step-dependency-glob-items.md @@ -7,7 +7,7 @@ Unline glob dependency, glob items dependency keeps track of the individual file command run with the list of files from a glob and you want to track added and removed files, use this. Otherwise if your command for all the files in a glob and don't need to track which files have changed, use the glob dependency. -This one injects `${XVC_GLOB_ADDED_ITEMS}`, `${XVC_GLOB_REMOVED_ITEMS}`, `${XVC_GLOB_CHANGED_ITEMS}` and `${XVC_GLOB_ALL_ITEMS}` to the command +This one injects `${XVC_ADDED_GLOB_ITEMS}`, `${XVC_REMOVED_GLOB_ITEMS}`, `${XVC_CHANGED_GLOB_ITEMS}` and `${XVC_ALL_GLOB_ITEMS}` to the command environment. This command works only in Xvc repositories. @@ -41,7 +41,7 @@ $ tree Add a step to list the added files. ```console -$ xvc pipeline step new --step-name files-changed --command 'echo "### Added Files:\n${XVC_GLOB_ADDED_ITEMS}\n### Removed Files:\n${XVC_GLOB_REMOVED_ITEMS}\n### Changed Files:\n${XVC_GLOB_CHANGED_ITEMS}"' +$ xvc pipeline step new --step-name files-changed --command 'echo "### Added Files:\n${XVC_ADDED_GLOB_ITEMS}\n### Removed Files:\n${XVC_REMOVED_GLOB_ITEMS}\n### Changed Files:\n${XVC_CHANGED_GLOB_ITEMS}"' $ xvc pipeline step dependency --step-name files-changed --glob-items 'dir-*/*' @@ -63,7 +63,7 @@ dir-0002/file-0003.bin ### Changed Files: -[DONE] files-changed (echo "### Added Files:/n${XVC_GLOB_ADDED_ITEMS}/n### Removed Files:/n${XVC_GLOB_REMOVED_ITEMS}/n### Changed Files:/n${XVC_GLOB_CHANGED_ITEMS}") +[DONE] files-changed (echo "### Added Files:/n${XVC_ADDED_GLOB_ITEMS}/n### Removed Files:/n${XVC_REMOVED_GLOB_ITEMS}/n### Changed Files:/n${XVC_CHANGED_GLOB_ITEMS}") $ xvc pipeline run @@ -82,7 +82,7 @@ dir-0001/file-0001.bin ### Changed Files: -[DONE] files-changed (echo "### Added Files:/n${XVC_GLOB_ADDED_ITEMS}/n### Removed Files:/n${XVC_GLOB_REMOVED_ITEMS}/n### Changed Files:/n${XVC_GLOB_CHANGED_ITEMS}") +[DONE] files-changed (echo "### Added Files:/n${XVC_ADDED_GLOB_ITEMS}/n### Removed Files:/n${XVC_REMOVED_GLOB_ITEMS}/n### Changed Files:/n${XVC_CHANGED_GLOB_ITEMS}") ``` @@ -99,6 +99,6 @@ $ xvc pipeline run ### Changed Files: dir-0001/file-0002.bin -[DONE] files-changed (echo "### Added Files:/n${XVC_GLOB_ADDED_ITEMS}/n### Removed Files:/n${XVC_GLOB_REMOVED_ITEMS}/n### Changed Files:/n${XVC_GLOB_CHANGED_ITEMS}") +[DONE] files-changed (echo "### Added Files:/n${XVC_ADDED_GLOB_ITEMS}/n### Removed Files:/n${XVC_REMOVED_GLOB_ITEMS}/n### Changed Files:/n${XVC_CHANGED_GLOB_ITEMS}") ``` diff --git a/book/src/ref/xvc-pipeline-step-dependency-line-items.md b/book/src/ref/xvc-pipeline-step-dependency-line-items.md index dc851996a..587361088 100644 --- a/book/src/ref/xvc-pipeline-step-dependency-line-items.md +++ b/book/src/ref/xvc-pipeline-step-dependency-line-items.md @@ -5,8 +5,8 @@ You can make your steps to depend on lines of text files. The lines are defined When the text in those lines change, the step is invalidated. Unlike line dependencies, this dependency type keeps track of the lines in the -file. You can use `${XVC_LINE_ALL_ITEMS}`, `${XVC_LINE_ADDED_ITEMS}`, and -`${XVC_LINE_REMOVED_ITEMS}` environment variables in the command. Please be +file. You can use `${XVC_ALL_LINE_ITEMS}`, `${XVC_ADDED_LINE_ITEMS}`, and +`${XVC_REMOVED_LINE_ITEMS}` environment variables in the command. Please be aware that for large set of lines, this dependency can take up considerable space to keep track of all lines and if you don't need to keep track of changed lines, you can use `--lines` dependency. @@ -49,7 +49,7 @@ $ cat people.csv Let's a step to show the first 10 lines of the file: ```console -$ xvc pipeline step new --step-name print-top-10 --command 'echo "Added Lines:\n ${XVC_LINE_ADDED_ITEMS}\nRemoved Lines:\n${XVC_LINE_REMOVED_ITEMS}"' +$ xvc pipeline step new --step-name print-top-10 --command 'echo "Added Lines:\n ${XVC_ADDED_LINE_ITEMS}\nRemoved Lines:\n${XVC_REMOVED_LINE_ITEMS}"' ``` @@ -77,7 +77,7 @@ $ xvc pipeline run Removed Lines: -[DONE] print-top-10 (echo "Added Lines:/n ${XVC_LINE_ADDED_ITEMS}/nRemoved Lines:/n${XVC_LINE_REMOVED_ITEMS}") +[DONE] print-top-10 (echo "Added Lines:/n ${XVC_ADDED_LINE_ITEMS}/nRemoved Lines:/n${XVC_REMOVED_LINE_ITEMS}") `````` @@ -104,7 +104,7 @@ $ xvc pipeline run Removed Lines: "Hank", "M", 30, 71, 158 -[DONE] print-top-10 (echo "Added Lines:/n ${XVC_LINE_ADDED_ITEMS}/nRemoved Lines:/n${XVC_LINE_REMOVED_ITEMS}") +[DONE] print-top-10 (echo "Added Lines:/n ${XVC_ADDED_LINE_ITEMS}/nRemoved Lines:/n${XVC_REMOVED_LINE_ITEMS}") ``` diff --git a/book/src/ref/xvc-pipeline-step-dependency-regex-items.md b/book/src/ref/xvc-pipeline-step-dependency-regex-items.md index 96f4cf13a..2adeb9b50 100644 --- a/book/src/ref/xvc-pipeline-step-dependency-regex-items.md +++ b/book/src/ref/xvc-pipeline-step-dependency-regex-items.md @@ -4,7 +4,7 @@ You can specify a regular expression matched against the lines from a file as a the matched results changed. Unlike regex dependencies, regex item dependencies keep track of the matched items. You can access them with -`${XVC_REGEX_ALL_ITEMS}`, `${XVC_REGEX_ADDED_ITEMS}`, and `${XVC_REGEX_REMOVED_ITEMS}` environment variables. +`${XVC_ALL_REGEX_ITEMS}`, `${XVC_ADDED_REGEX_ITEMS}`, and `${XVC_REMOVED_REGEX_ITEMS}` environment variables. This command works only in Xvc repositories. @@ -44,8 +44,8 @@ $ cat people.csv Now, let's add steps to the pipeline to count males and females in the file: ```console -$ xvc pipeline step new --step-name new-males --command 'echo "New Males:\n ${XVC_REGEX_ADDED_ITEMS}"' -$ xvc pipeline step new --step-name new-females --command 'echo "New Females:\n ${XVC_REGEX_ADDED_ITEMS}"' +$ xvc pipeline step new --step-name new-males --command 'echo "New Males:\n ${XVC_ADDED_REGEX_ITEMS}"' +$ xvc pipeline step new --step-name new-females --command 'echo "New Females:\n ${XVC_ADDED_REGEX_ITEMS}"' $ xvc pipeline step dependency --step-name new-females --step new-males ``` @@ -77,7 +77,7 @@ $ xvc pipeline run "Omar", "M", 38, 70, 145 "Quin", "M", 29, 71, 176 -[DONE] new-males (echo "New Males:/n ${XVC_REGEX_ADDED_ITEMS}") +[DONE] new-males (echo "New Males:/n ${XVC_ADDED_REGEX_ITEMS}") [OUT] [new-females] New Females: "Elly", "F", 30, 66, 124 "Fran", "F", 33, 66, 115 @@ -87,7 +87,7 @@ $ xvc pipeline run "Page", "F", 31, 67, 135 "Ruth", "F", 28, 65, 131 -[DONE] new-females (echo "New Females:/n ${XVC_REGEX_ADDED_ITEMS}") +[DONE] new-females (echo "New Females:/n ${XVC_ADDED_REGEX_ITEMS}") `````` @@ -130,6 +130,6 @@ $ xvc pipeline run [OUT] [new-females] New Females: "Asude", "F", 12, 55, 110 -[DONE] new-females (echo "New Females:/n ${XVC_REGEX_ADDED_ITEMS}") +[DONE] new-females (echo "New Females:/n ${XVC_ADDED_REGEX_ITEMS}") ``` diff --git a/pipeline/src/pipeline/mod.rs b/pipeline/src/pipeline/mod.rs index 0a6824e86..19bf27895 100644 --- a/pipeline/src/pipeline/mod.rs +++ b/pipeline/src/pipeline/mod.rs @@ -1257,19 +1257,19 @@ fn update_command_environment( if let Some(items) = actual.items() { match actual { XvcDependency::GlobItems(_) => { - update_env("XVC_GLOB_ADDED_ITEMS", &items)?; - update_env("XVC_GLOB_REMOVED_ITEMS", &[])?; - update_env("XVC_GLOB_ALL_ITEMS", &items) + update_env("XVC_ADDED_GLOB_ITEMS", &items)?; + update_env("XVC_REMOVED_GLOB_ITEMS", &[])?; + update_env("XVC_ALL_GLOB_ITEMS", &items) } XvcDependency::RegexItems(_) => { - update_env("XVC_REGEX_ADDED_ITEMS", &items)?; - update_env("XVC_REGEX_REMOVED_ITEMS", &[])?; - update_env("XVC_REGEX_ALL_ITEMS", &items) + update_env("XVC_ADDED_REGEX_ITEMS", &items)?; + update_env("XVC_REMOVED_REGEX_ITEMS", &[])?; + update_env("XVC_ALL_REGEX_ITEMS", &items) } XvcDependency::LineItems(_) => { - update_env("XVC_LINE_ADDED_ITEMS", &items)?; - update_env("XVC_LINE_REMOVED_ITEMS", &[])?; - update_env("XVC_LINE_ALL_ITEMS", &items) + update_env("XVC_ADDED_ITEMS", &items)?; + update_env("XVC_REMOVED_LINE_ITEMS", &[])?; + update_env("XVC_ALL_LINE_ITEMS", &items) } _ => Ok(()), } @@ -1282,19 +1282,19 @@ fn update_command_environment( if let Some(items) = record.items() { match record { XvcDependency::GlobItems(_dep) => { - update_env("XVC_GLOB_ADDED_ITEMS", &[])?; - update_env("XVC_GLOB_REMOVED_ITEMS", &items)?; - update_env("XVC_GLOB_ALL_ITEMS", &items) + update_env("XVC_ADDED_GLOB_ITEMS", &[])?; + update_env("XVC_REMOVED_GLOB_ITEMS", &items)?; + update_env("XVC_ALL_GLOB_ITEMS", &items) } XvcDependency::RegexItems(_dep) => { - update_env("XVC_REGEX_ADDED_ITEMS", &[])?; - update_env("XVC_REGEX_REMOVED_ITEMS", &items)?; - update_env("XVC_REGEX_ALL_ITEMS", &items) + update_env("XVC_ADDED_REGEX_ITEMS", &[])?; + update_env("XVC_REMOVED_REGEX_ITEMS", &items)?; + update_env("XVC_ALL_REGEX_ITEMS", &items) } XvcDependency::LineItems(_dep) => { - update_env("XVC_LINE_ADDED_ITEMS", &[])?; - update_env("XVC_LINE_REMOVED_ITEMS", &items)?; - update_env("XVC_LINE_ALL_ITEMS", &items) + update_env("XVC_ADDED_LINE_ITEMS", &[])?; + update_env("XVC_REMOVED_LINE_ITEMS", &items)?; + update_env("XVC_ALL_LINE_ITEMS", &items) } _ => Ok(()), } @@ -1340,22 +1340,22 @@ fn update_command_environment( } } } - update_env("XVC_GLOB_CHANGED_ITEMS", &changed_items)?; + update_env("XVC_CHANGED_GLOB_ITEMS", &changed_items)?; } - update_env("XVC_GLOB_ADDED_ITEMS", &added_items)?; - update_env("XVC_GLOB_REMOVED_ITEMS", &removed_items)?; - update_env("XVC_GLOB_ALL_ITEMS", &all_items) + update_env("XVC_ADDED_GLOB_ITEMS", &added_items)?; + update_env("XVC_REMOVED_GLOB_ITEMS", &removed_items)?; + update_env("XVC_ALL_GLOB_ITEMS", &all_items) } XvcDependency::RegexItems(_dep) => { - update_env("XVC_REGEX_ADDED_ITEMS", &added_items)?; - update_env("XVC_REGEX_REMOVED_ITEMS", &removed_items)?; - update_env("XVC_REGEX_ALL_ITEMS", &all_items) + update_env("XVC_ADDED_REGEX_ITEMS", &added_items)?; + update_env("XVC_REMOVED_REGEX_ITEMS", &removed_items)?; + update_env("XVC_ALL_REGEX_ITEMS", &all_items) } XvcDependency::LineItems(_dep) => { - update_env("XVC_LINE_ADDED_ITEMS", &added_items)?; - update_env("XVC_LINE_REMOVED_ITEMS", &removed_items)?; - update_env("XVC_LINE_ALL_ITEMS", &all_items) + update_env("XVC_ADDED_LINE_ITEMS", &added_items)?; + update_env("XVC_REMOVED_LINE_ITEMS", &removed_items)?; + update_env("XVC_ALL_LINE_ITEMS", &all_items) } _ => Ok(()), } @@ -1371,19 +1371,19 @@ fn update_command_environment( if let Some(items) = dep.items() { match dep { XvcDependency::GlobItems(_) => { - update_env("XVC_GLOB_ADDED_ITEMS", &[])?; - update_env("XVC_GLOB_REMOVED_ITEMS", &[])?; - update_env("XVC_GLOB_ALL_ITEMS", &items) + update_env("XVC_ADDED_GLOB_ITEMS", &[])?; + update_env("XVC_REMOVED_GLOB_ITEMS", &[])?; + update_env("XVC_ALL_GLOB_ITEMS", &items) } XvcDependency::RegexItems(_) => { - update_env("XVC_REGEX_ADDED_ITEMS", &[])?; - update_env("XVC_REGEX_REMOVED_ITEMS", &[])?; - update_env("XVC_REGEX_ALL_ITEMS", &items) + update_env("XVC_ADDED_REGEX_ITEMS", &[])?; + update_env("XVC_REMOVED_REGEX_ITEMS", &[])?; + update_env("XVC_ALL_REGEX_ITEMS", &items) } XvcDependency::LineItems(_) => { - update_env("XVC_LINE_ADDED_ITEMS", &[])?; - update_env("XVC_LINE_REMOVED_ITEMS", &[])?; - update_env("XVC_LINE_ALL_ITEMS", &items) + update_env("XVC_ADDED_LINE_ITEMS", &[])?; + update_env("XVC_REMOVED_LINE_ITEMS", &[])?; + update_env("XVC_ALL_LINE_ITEMS", &items) } _ => Ok(()), } From 7e20029bca2ef0dd5d42ab22878df7b33647fec0 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sat, 13 Jul 2024 12:04:17 +0300 Subject: [PATCH 020/257] bump version --- config/Cargo.toml | 6 +++--- core/Cargo.toml | 12 ++++++------ ecs/Cargo.toml | 4 ++-- file/Cargo.toml | 16 ++++++++-------- lib/Cargo.toml | 20 ++++++++++---------- logging/Cargo.toml | 2 +- pipeline/Cargo.toml | 16 ++++++++-------- storage/Cargo.toml | 14 +++++++------- test_helper/Cargo.toml | 4 ++-- walker/Cargo.toml | 6 +++--- workflow_tests/Cargo.toml | 22 +++++++++++----------- 11 files changed, 61 insertions(+), 61 deletions(-) diff --git a/config/Cargo.toml b/config/Cargo.toml index e119071cf..c4ee8676b 100644 --- a/config/Cargo.toml +++ b/config/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-config" -version = "0.6.8-alpha.2" +version = "0.6.8-alpha.3" edition = "2021" description = "Xvc configuration management" authors = ["Emre Şahin "] @@ -20,8 +20,8 @@ debug = true [dependencies] -xvc-logging = { version = "0.6.8-alpha.2", path = "../logging" } -xvc-walker = { version = "0.6.8-alpha.2", path = "../walker" } +xvc-logging = { version = "0.6.8-alpha.3", path = "../logging" } +xvc-walker = { version = "0.6.8-alpha.3", path = "../walker" } ## Cli and config diff --git a/core/Cargo.toml b/core/Cargo.toml index 02d0df6dd..4ad9d48da 100644 --- a/core/Cargo.toml +++ b/core/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-core" -version = "0.6.8-alpha.2" +version = "0.6.8-alpha.3" edition = "2021" description = "Xvc core for common elements for all commands" authors = ["Emre Şahin "] @@ -19,10 +19,10 @@ crate-type = ["rlib"] debug = true [dependencies] -xvc-config = { version = "0.6.8-alpha.2", path = "../config" } -xvc-logging = { version = "0.6.8-alpha.2", path = "../logging" } -xvc-ecs = { version = "0.6.8-alpha.2", path = "../ecs" } -xvc-walker = { version = "0.6.8-alpha.2", path = "../walker" } +xvc-config = { version = "0.6.8-alpha.3", path = "../config" } +xvc-logging = { version = "0.6.8-alpha.3", path = "../logging" } +xvc-ecs = { version = "0.6.8-alpha.3", path = "../ecs" } +xvc-walker = { version = "0.6.8-alpha.3", path = "../walker" } ## Cli and config clap = { version = "^4.4", features = ["derive"] } @@ -82,6 +82,6 @@ derive_more = "^0.99" itertools = "^0.12" [dev-dependencies] -xvc-test-helper = { version = "0.6.8-alpha.2", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.8-alpha.3", path = "../test_helper/" } proptest = "^1.4" test-case = "^3.3" diff --git a/ecs/Cargo.toml b/ecs/Cargo.toml index 471898ab1..8fe34b2f8 100644 --- a/ecs/Cargo.toml +++ b/ecs/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-ecs" -version = "0.6.8-alpha.2" +version = "0.6.8-alpha.3" edition = "2021" description = "Entity-Component System for Xvc" authors = ["Emre Şahin "] @@ -19,7 +19,7 @@ crate-type = ["rlib"] debug = true [dependencies] -xvc-logging = { version = "0.6.8-alpha.2", path = "../logging" } +xvc-logging = { version = "0.6.8-alpha.3", path = "../logging" } ## Serialization serde = { version = "^1.0", features = ["derive"] } diff --git a/file/Cargo.toml b/file/Cargo.toml index 25bbe8d5f..7138e4442 100644 --- a/file/Cargo.toml +++ b/file/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-file" -version = "0.6.8-alpha.2" +version = "0.6.8-alpha.3" edition = "2021" description = "File tracking, versioning, upload and download functions for Xvc" authors = ["Emre Şahin "] @@ -24,12 +24,12 @@ bench = true debug = true [dependencies] -xvc-logging = { version = "0.6.8-alpha.2", path = "../logging" } -xvc-config = { version = "0.6.8-alpha.2", path = "../config" } -xvc-core = { version = "0.6.8-alpha.2", path = "../core" } -xvc-ecs = { version = "0.6.8-alpha.2", path = "../ecs" } -xvc-walker = { version = "0.6.8-alpha.2", path = "../walker" } -xvc-storage = { version = "0.6.8-alpha.2", path = "../storage" } +xvc-logging = { version = "0.6.8-alpha.3", path = "../logging" } +xvc-config = { version = "0.6.8-alpha.3", path = "../config" } +xvc-core = { version = "0.6.8-alpha.3", path = "../core" } +xvc-ecs = { version = "0.6.8-alpha.3", path = "../ecs" } +xvc-walker = { version = "0.6.8-alpha.3", path = "../walker" } +xvc-storage = { version = "0.6.8-alpha.3", path = "../storage" } ## Cli and config @@ -94,5 +94,5 @@ parse-size = "^1.0" [dev-dependencies] -xvc-test-helper = { version = "0.6.8-alpha.2", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.8-alpha.3", path = "../test_helper/" } shellfn = "^0.1" diff --git a/lib/Cargo.toml b/lib/Cargo.toml index 3a8713947..5d37cc8a0 100644 --- a/lib/Cargo.toml +++ b/lib/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc" -version = "0.6.8-alpha.2" +version = "0.6.8-alpha.3" edition = "2021" description = "An MLOps tool to manage data files and pipelines on top of Git" authors = ["Emre Şahin "] @@ -23,14 +23,14 @@ path = "src/main.rs" debug = true [dependencies] -xvc-config = { version = "0.6.8-alpha.2", path = "../config" } -xvc-core = { version = "0.6.8-alpha.2", path = "../core" } -xvc-logging = { version = "0.6.8-alpha.2", path = "../logging" } -xvc-ecs = { version = "0.6.8-alpha.2", path = "../ecs" } -xvc-file = { version = "0.6.8-alpha.2", path = "../file" } -xvc-pipeline = { version = "0.6.8-alpha.2", path = "../pipeline" } -xvc-walker = { version = "0.6.8-alpha.2", path = "../walker" } -xvc-storage = { version = "0.6.8-alpha.2", path = "../storage" } +xvc-config = { version = "0.6.8-alpha.3", path = "../config" } +xvc-core = { version = "0.6.8-alpha.3", path = "../core" } +xvc-logging = { version = "0.6.8-alpha.3", path = "../logging" } +xvc-ecs = { version = "0.6.8-alpha.3", path = "../ecs" } +xvc-file = { version = "0.6.8-alpha.3", path = "../file" } +xvc-pipeline = { version = "0.6.8-alpha.3", path = "../pipeline" } +xvc-walker = { version = "0.6.8-alpha.3", path = "../walker" } +xvc-storage = { version = "0.6.8-alpha.3", path = "../storage" } ## Cli and config @@ -113,4 +113,4 @@ jwalk = "^0.8" proptest = "^1.4" shellfn = "^0.1" test-case = "^3.3" -xvc-test-helper = { version = "0.6.8-alpha.2", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.8-alpha.3", path = "../test_helper/" } diff --git a/logging/Cargo.toml b/logging/Cargo.toml index 4ce732c9a..27cc765eb 100644 --- a/logging/Cargo.toml +++ b/logging/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-logging" -version = "0.6.8-alpha.2" +version = "0.6.8-alpha.3" edition = "2021" description = "Logging crate for Xvc" authors = ["Emre Şahin "] diff --git a/pipeline/Cargo.toml b/pipeline/Cargo.toml index c507cf991..58a566dc9 100644 --- a/pipeline/Cargo.toml +++ b/pipeline/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-pipeline" -version = "0.6.8-alpha.2" +version = "0.6.8-alpha.3" edition = "2021" description = "Xvc data pipeline management" authors = ["Emre Şahin "] @@ -19,12 +19,12 @@ debug = true [dependencies] -xvc-config = { version = "0.6.8-alpha.2", path = "../config" } -xvc-core = { version = "0.6.8-alpha.2", path = "../core" } -xvc-ecs = { version = "0.6.8-alpha.2", path = "../ecs" } -xvc-logging = { version = "0.6.8-alpha.2", path = "../logging" } -xvc-walker = { version = "0.6.8-alpha.2", path = "../walker" } -xvc-file = { version = "0.6.8-alpha.2", path = "../file" } +xvc-config = { version = "0.6.8-alpha.3", path = "../config" } +xvc-core = { version = "0.6.8-alpha.3", path = "../core" } +xvc-ecs = { version = "0.6.8-alpha.3", path = "../ecs" } +xvc-logging = { version = "0.6.8-alpha.3", path = "../logging" } +xvc-walker = { version = "0.6.8-alpha.3", path = "../walker" } +xvc-file = { version = "0.6.8-alpha.3", path = "../file" } ## Cli and config clap = { version = "^4.4", features = ["derive"] } @@ -92,5 +92,5 @@ itertools = "^0.12" derive_more = "^0.99" [dev-dependencies] -xvc-test-helper = { version = "0.6.8-alpha.2", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.8-alpha.3", path = "../test_helper/" } test-case = "^3.3" diff --git a/storage/Cargo.toml b/storage/Cargo.toml index 1bffe1074..d2d59ac4a 100644 --- a/storage/Cargo.toml +++ b/storage/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-storage" -version = "0.6.8-alpha.2" +version = "0.6.8-alpha.3" edition = "2021" description = "Xvc remote and local storage management" authors = ["Emre Şahin "] @@ -20,11 +20,11 @@ debug = true [dependencies] -xvc-logging = { version = "0.6.8-alpha.2", path = "../logging" } -xvc-config = { version = "0.6.8-alpha.2", path = "../config" } -xvc-core = { version = "0.6.8-alpha.2", path = "../core" } -xvc-ecs = { version = "0.6.8-alpha.2", path = "../ecs" } -xvc-walker = { version = "0.6.8-alpha.2", path = "../walker" } +xvc-logging = { version = "0.6.8-alpha.3", path = "../logging" } +xvc-config = { version = "0.6.8-alpha.3", path = "../config" } +xvc-core = { version = "0.6.8-alpha.3", path = "../core" } +xvc-ecs = { version = "0.6.8-alpha.3", path = "../ecs" } +xvc-walker = { version = "0.6.8-alpha.3", path = "../walker" } ## Cli and config clap = { version = "^4.4", features = ["derive"] } @@ -109,7 +109,7 @@ digital-ocean = ["async"] [dev-dependencies] -xvc-test-helper = { version = "0.6.8-alpha.2", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.8-alpha.3", path = "../test_helper/" } shellfn = "^0.1" [package.metadata.cargo-udeps.ignore] diff --git a/test_helper/Cargo.toml b/test_helper/Cargo.toml index 14fc22b26..1c4c2bd6d 100644 --- a/test_helper/Cargo.toml +++ b/test_helper/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-test-helper" -version = "0.6.8-alpha.2" +version = "0.6.8-alpha.3" edition = "2021" description = "Unit test helper functions for Xvc" authors = ["Emre Şahin "] @@ -23,7 +23,7 @@ debug = true # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html [dependencies] -xvc-logging = { version = "0.6.8-alpha.2", path = "../logging/" } +xvc-logging = { version = "0.6.8-alpha.3", path = "../logging/" } rand = "^0.8" log = "^0.4" diff --git a/walker/Cargo.toml b/walker/Cargo.toml index beb83fc2b..39366de50 100644 --- a/walker/Cargo.toml +++ b/walker/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-walker" -version = "0.6.8-alpha.2" +version = "0.6.8-alpha.3" edition = "2021" description = "Xvc parallel file system walker with ignore features" authors = ["Emre Şahin "] @@ -19,7 +19,7 @@ crate-type = ["rlib"] debug = true [dependencies] -xvc-logging = { version = "0.6.8-alpha.2", path = "../logging" } +xvc-logging = { version = "0.6.8-alpha.3", path = "../logging" } globset = "^0.4" ## Parallelization @@ -41,7 +41,7 @@ itertools = "^0.12" regex = "^1.10" [dev-dependencies] -xvc-test-helper = { path = "../test_helper/", version = "0.6.8-alpha.2" } +xvc-test-helper = { path = "../test_helper/", version = "0.6.8-alpha.3" } test-case = "^3.3" [package.metadata.cargo-udeps.ignore] diff --git a/workflow_tests/Cargo.toml b/workflow_tests/Cargo.toml index 34f503661..5dd13496c 100644 --- a/workflow_tests/Cargo.toml +++ b/workflow_tests/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-workflow-tests" -version = "0.6.8-alpha.2" +version = "0.6.8-alpha.3" edition = "2021" description = "Integration test suite for Xvc" authors = ["Emre Şahin "] @@ -23,15 +23,15 @@ debug = true [dependencies] -xvc = { version = "0.6.8-alpha.2", path = "../lib" } -xvc-config = { version = "0.6.8-alpha.2", path = "../config" } -xvc-core = { version = "0.6.8-alpha.2", path = "../core" } -xvc-logging = { version = "0.6.8-alpha.2", path = "../logging" } -xvc-ecs = { version = "0.6.8-alpha.2", path = "../ecs" } -xvc-file = { version = "0.6.8-alpha.2", path = "../file" } -xvc-pipeline = { version = "0.6.8-alpha.2", path = "../pipeline" } -xvc-walker = { version = "0.6.8-alpha.2", path = "../walker" } -xvc-storage = { version = "0.6.8-alpha.2", path = "../storage" } +xvc = { version = "0.6.8-alpha.3", path = "../lib" } +xvc-config = { version = "0.6.8-alpha.3", path = "../config" } +xvc-core = { version = "0.6.8-alpha.3", path = "../core" } +xvc-logging = { version = "0.6.8-alpha.3", path = "../logging" } +xvc-ecs = { version = "0.6.8-alpha.3", path = "../ecs" } +xvc-file = { version = "0.6.8-alpha.3", path = "../file" } +xvc-pipeline = { version = "0.6.8-alpha.3", path = "../pipeline" } +xvc-walker = { version = "0.6.8-alpha.3", path = "../walker" } +xvc-storage = { version = "0.6.8-alpha.3", path = "../storage" } ## packages for testing clap = { version = "^4.4", features = ["derive"] } @@ -70,7 +70,7 @@ proptest = "^1.4" test-case = "^3.3" globset = "^0.4" escargot = "^0.5" -xvc-test-helper = { version = "0.6.8-alpha.2", path = "../test_helper" } +xvc-test-helper = { version = "0.6.8-alpha.3", path = "../test_helper" } shellfn = "^0.1" jwalk = "^0.8" anyhow = "^1.0" From 0d85aa62d85779d941714ad0a164c4a4a03844da Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sun, 14 Jul 2024 18:44:50 +0300 Subject: [PATCH 021/257] added CRLF at the end of DONE --- pipeline/src/pipeline/mod.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pipeline/src/pipeline/mod.rs b/pipeline/src/pipeline/mod.rs index 19bf27895..77c9b5a09 100644 --- a/pipeline/src/pipeline/mod.rs +++ b/pipeline/src/pipeline/mod.rs @@ -1488,7 +1488,7 @@ fn s_running_f_wait_process<'a>( Some(exit_code) => match exit_code { ExitStatus::Exited(0) => { - output!(params.output_snd, "[DONE] {} ({})", step.name, step_command); + output!(params.output_snd, "[DONE] {} ({})\n", step.name, step_command); return_state = Some(s.process_completed_successfully()); } , From f365a42cd5676b055abb6beee2e1647f59f2dafd Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sun, 14 Jul 2024 18:45:38 +0300 Subject: [PATCH 022/257] fix the space at the end of out --- pipeline/src/pipeline/command.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pipeline/src/pipeline/command.rs b/pipeline/src/pipeline/command.rs index 0bb138762..686ea033b 100644 --- a/pipeline/src/pipeline/command.rs +++ b/pipeline/src/pipeline/command.rs @@ -119,7 +119,7 @@ impl CommandProcess { stdout.read_to_string(&mut out)?; if !out.is_empty() { self.stdout_sender - .send(format!("[OUT] [{}] {} ", self.step.name, out)) + .send(format!("[OUT] [{}] {}", self.step.name, out)) .ok(); } } From 216a419aa314704e36910fc457e71e08b5b56d99 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sun, 14 Jul 2024 18:46:13 +0300 Subject: [PATCH 023/257] fix space at the end --- pipeline/src/pipeline/command.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pipeline/src/pipeline/command.rs b/pipeline/src/pipeline/command.rs index 686ea033b..c2053f289 100644 --- a/pipeline/src/pipeline/command.rs +++ b/pipeline/src/pipeline/command.rs @@ -129,7 +129,7 @@ impl CommandProcess { stderr.read_to_string(&mut err)?; if !err.is_empty() { self.stderr_sender - .send(format!("[ERR] [{}] {} ", self.step.name, err)) + .send(format!("[ERR] [{}] {}", self.step.name, err)) .ok(); } } From a901c830a45529def02ffbb4b21a25ca5709cdb0 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Mon, 15 Jul 2024 12:44:22 +0300 Subject: [PATCH 024/257] bump version to release --- config/Cargo.toml | 6 +++--- core/Cargo.toml | 12 ++++++------ ecs/Cargo.toml | 4 ++-- file/Cargo.toml | 16 ++++++++-------- lib/Cargo.toml | 20 ++++++++++---------- logging/Cargo.toml | 2 +- pipeline/Cargo.toml | 16 ++++++++-------- storage/Cargo.toml | 14 +++++++------- test_helper/Cargo.toml | 4 ++-- walker/Cargo.toml | 6 +++--- workflow_tests/Cargo.toml | 22 +++++++++++----------- 11 files changed, 61 insertions(+), 61 deletions(-) diff --git a/config/Cargo.toml b/config/Cargo.toml index c4ee8676b..58eb69b32 100644 --- a/config/Cargo.toml +++ b/config/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-config" -version = "0.6.8-alpha.3" +version = "0.6.8" edition = "2021" description = "Xvc configuration management" authors = ["Emre Şahin "] @@ -20,8 +20,8 @@ debug = true [dependencies] -xvc-logging = { version = "0.6.8-alpha.3", path = "../logging" } -xvc-walker = { version = "0.6.8-alpha.3", path = "../walker" } +xvc-logging = { version = "0.6.8", path = "../logging" } +xvc-walker = { version = "0.6.8", path = "../walker" } ## Cli and config diff --git a/core/Cargo.toml b/core/Cargo.toml index 4ad9d48da..86c8d55ec 100644 --- a/core/Cargo.toml +++ b/core/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-core" -version = "0.6.8-alpha.3" +version = "0.6.8" edition = "2021" description = "Xvc core for common elements for all commands" authors = ["Emre Şahin "] @@ -19,10 +19,10 @@ crate-type = ["rlib"] debug = true [dependencies] -xvc-config = { version = "0.6.8-alpha.3", path = "../config" } -xvc-logging = { version = "0.6.8-alpha.3", path = "../logging" } -xvc-ecs = { version = "0.6.8-alpha.3", path = "../ecs" } -xvc-walker = { version = "0.6.8-alpha.3", path = "../walker" } +xvc-config = { version = "0.6.8", path = "../config" } +xvc-logging = { version = "0.6.8", path = "../logging" } +xvc-ecs = { version = "0.6.8", path = "../ecs" } +xvc-walker = { version = "0.6.8", path = "../walker" } ## Cli and config clap = { version = "^4.4", features = ["derive"] } @@ -82,6 +82,6 @@ derive_more = "^0.99" itertools = "^0.12" [dev-dependencies] -xvc-test-helper = { version = "0.6.8-alpha.3", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.8", path = "../test_helper/" } proptest = "^1.4" test-case = "^3.3" diff --git a/ecs/Cargo.toml b/ecs/Cargo.toml index 8fe34b2f8..835e749af 100644 --- a/ecs/Cargo.toml +++ b/ecs/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-ecs" -version = "0.6.8-alpha.3" +version = "0.6.8" edition = "2021" description = "Entity-Component System for Xvc" authors = ["Emre Şahin "] @@ -19,7 +19,7 @@ crate-type = ["rlib"] debug = true [dependencies] -xvc-logging = { version = "0.6.8-alpha.3", path = "../logging" } +xvc-logging = { version = "0.6.8", path = "../logging" } ## Serialization serde = { version = "^1.0", features = ["derive"] } diff --git a/file/Cargo.toml b/file/Cargo.toml index 7138e4442..98e5df38d 100644 --- a/file/Cargo.toml +++ b/file/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-file" -version = "0.6.8-alpha.3" +version = "0.6.8" edition = "2021" description = "File tracking, versioning, upload and download functions for Xvc" authors = ["Emre Şahin "] @@ -24,12 +24,12 @@ bench = true debug = true [dependencies] -xvc-logging = { version = "0.6.8-alpha.3", path = "../logging" } -xvc-config = { version = "0.6.8-alpha.3", path = "../config" } -xvc-core = { version = "0.6.8-alpha.3", path = "../core" } -xvc-ecs = { version = "0.6.8-alpha.3", path = "../ecs" } -xvc-walker = { version = "0.6.8-alpha.3", path = "../walker" } -xvc-storage = { version = "0.6.8-alpha.3", path = "../storage" } +xvc-logging = { version = "0.6.8", path = "../logging" } +xvc-config = { version = "0.6.8", path = "../config" } +xvc-core = { version = "0.6.8", path = "../core" } +xvc-ecs = { version = "0.6.8", path = "../ecs" } +xvc-walker = { version = "0.6.8", path = "../walker" } +xvc-storage = { version = "0.6.8", path = "../storage" } ## Cli and config @@ -94,5 +94,5 @@ parse-size = "^1.0" [dev-dependencies] -xvc-test-helper = { version = "0.6.8-alpha.3", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.8", path = "../test_helper/" } shellfn = "^0.1" diff --git a/lib/Cargo.toml b/lib/Cargo.toml index 5d37cc8a0..3a306ad5c 100644 --- a/lib/Cargo.toml +++ b/lib/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc" -version = "0.6.8-alpha.3" +version = "0.6.8" edition = "2021" description = "An MLOps tool to manage data files and pipelines on top of Git" authors = ["Emre Şahin "] @@ -23,14 +23,14 @@ path = "src/main.rs" debug = true [dependencies] -xvc-config = { version = "0.6.8-alpha.3", path = "../config" } -xvc-core = { version = "0.6.8-alpha.3", path = "../core" } -xvc-logging = { version = "0.6.8-alpha.3", path = "../logging" } -xvc-ecs = { version = "0.6.8-alpha.3", path = "../ecs" } -xvc-file = { version = "0.6.8-alpha.3", path = "../file" } -xvc-pipeline = { version = "0.6.8-alpha.3", path = "../pipeline" } -xvc-walker = { version = "0.6.8-alpha.3", path = "../walker" } -xvc-storage = { version = "0.6.8-alpha.3", path = "../storage" } +xvc-config = { version = "0.6.8", path = "../config" } +xvc-core = { version = "0.6.8", path = "../core" } +xvc-logging = { version = "0.6.8", path = "../logging" } +xvc-ecs = { version = "0.6.8", path = "../ecs" } +xvc-file = { version = "0.6.8", path = "../file" } +xvc-pipeline = { version = "0.6.8", path = "../pipeline" } +xvc-walker = { version = "0.6.8", path = "../walker" } +xvc-storage = { version = "0.6.8", path = "../storage" } ## Cli and config @@ -113,4 +113,4 @@ jwalk = "^0.8" proptest = "^1.4" shellfn = "^0.1" test-case = "^3.3" -xvc-test-helper = { version = "0.6.8-alpha.3", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.8", path = "../test_helper/" } diff --git a/logging/Cargo.toml b/logging/Cargo.toml index 27cc765eb..3a7fe0a07 100644 --- a/logging/Cargo.toml +++ b/logging/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-logging" -version = "0.6.8-alpha.3" +version = "0.6.8" edition = "2021" description = "Logging crate for Xvc" authors = ["Emre Şahin "] diff --git a/pipeline/Cargo.toml b/pipeline/Cargo.toml index 58a566dc9..15d111569 100644 --- a/pipeline/Cargo.toml +++ b/pipeline/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-pipeline" -version = "0.6.8-alpha.3" +version = "0.6.8" edition = "2021" description = "Xvc data pipeline management" authors = ["Emre Şahin "] @@ -19,12 +19,12 @@ debug = true [dependencies] -xvc-config = { version = "0.6.8-alpha.3", path = "../config" } -xvc-core = { version = "0.6.8-alpha.3", path = "../core" } -xvc-ecs = { version = "0.6.8-alpha.3", path = "../ecs" } -xvc-logging = { version = "0.6.8-alpha.3", path = "../logging" } -xvc-walker = { version = "0.6.8-alpha.3", path = "../walker" } -xvc-file = { version = "0.6.8-alpha.3", path = "../file" } +xvc-config = { version = "0.6.8", path = "../config" } +xvc-core = { version = "0.6.8", path = "../core" } +xvc-ecs = { version = "0.6.8", path = "../ecs" } +xvc-logging = { version = "0.6.8", path = "../logging" } +xvc-walker = { version = "0.6.8", path = "../walker" } +xvc-file = { version = "0.6.8", path = "../file" } ## Cli and config clap = { version = "^4.4", features = ["derive"] } @@ -92,5 +92,5 @@ itertools = "^0.12" derive_more = "^0.99" [dev-dependencies] -xvc-test-helper = { version = "0.6.8-alpha.3", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.8", path = "../test_helper/" } test-case = "^3.3" diff --git a/storage/Cargo.toml b/storage/Cargo.toml index d2d59ac4a..b602988c8 100644 --- a/storage/Cargo.toml +++ b/storage/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-storage" -version = "0.6.8-alpha.3" +version = "0.6.8" edition = "2021" description = "Xvc remote and local storage management" authors = ["Emre Şahin "] @@ -20,11 +20,11 @@ debug = true [dependencies] -xvc-logging = { version = "0.6.8-alpha.3", path = "../logging" } -xvc-config = { version = "0.6.8-alpha.3", path = "../config" } -xvc-core = { version = "0.6.8-alpha.3", path = "../core" } -xvc-ecs = { version = "0.6.8-alpha.3", path = "../ecs" } -xvc-walker = { version = "0.6.8-alpha.3", path = "../walker" } +xvc-logging = { version = "0.6.8", path = "../logging" } +xvc-config = { version = "0.6.8", path = "../config" } +xvc-core = { version = "0.6.8", path = "../core" } +xvc-ecs = { version = "0.6.8", path = "../ecs" } +xvc-walker = { version = "0.6.8", path = "../walker" } ## Cli and config clap = { version = "^4.4", features = ["derive"] } @@ -109,7 +109,7 @@ digital-ocean = ["async"] [dev-dependencies] -xvc-test-helper = { version = "0.6.8-alpha.3", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.8", path = "../test_helper/" } shellfn = "^0.1" [package.metadata.cargo-udeps.ignore] diff --git a/test_helper/Cargo.toml b/test_helper/Cargo.toml index 1c4c2bd6d..fa20c7380 100644 --- a/test_helper/Cargo.toml +++ b/test_helper/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-test-helper" -version = "0.6.8-alpha.3" +version = "0.6.8" edition = "2021" description = "Unit test helper functions for Xvc" authors = ["Emre Şahin "] @@ -23,7 +23,7 @@ debug = true # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html [dependencies] -xvc-logging = { version = "0.6.8-alpha.3", path = "../logging/" } +xvc-logging = { version = "0.6.8", path = "../logging/" } rand = "^0.8" log = "^0.4" diff --git a/walker/Cargo.toml b/walker/Cargo.toml index 39366de50..6c3296666 100644 --- a/walker/Cargo.toml +++ b/walker/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-walker" -version = "0.6.8-alpha.3" +version = "0.6.8" edition = "2021" description = "Xvc parallel file system walker with ignore features" authors = ["Emre Şahin "] @@ -19,7 +19,7 @@ crate-type = ["rlib"] debug = true [dependencies] -xvc-logging = { version = "0.6.8-alpha.3", path = "../logging" } +xvc-logging = { version = "0.6.8", path = "../logging" } globset = "^0.4" ## Parallelization @@ -41,7 +41,7 @@ itertools = "^0.12" regex = "^1.10" [dev-dependencies] -xvc-test-helper = { path = "../test_helper/", version = "0.6.8-alpha.3" } +xvc-test-helper = { path = "../test_helper/", version = "0.6.8" } test-case = "^3.3" [package.metadata.cargo-udeps.ignore] diff --git a/workflow_tests/Cargo.toml b/workflow_tests/Cargo.toml index 5dd13496c..5c3d3e8da 100644 --- a/workflow_tests/Cargo.toml +++ b/workflow_tests/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-workflow-tests" -version = "0.6.8-alpha.3" +version = "0.6.8" edition = "2021" description = "Integration test suite for Xvc" authors = ["Emre Şahin "] @@ -23,15 +23,15 @@ debug = true [dependencies] -xvc = { version = "0.6.8-alpha.3", path = "../lib" } -xvc-config = { version = "0.6.8-alpha.3", path = "../config" } -xvc-core = { version = "0.6.8-alpha.3", path = "../core" } -xvc-logging = { version = "0.6.8-alpha.3", path = "../logging" } -xvc-ecs = { version = "0.6.8-alpha.3", path = "../ecs" } -xvc-file = { version = "0.6.8-alpha.3", path = "../file" } -xvc-pipeline = { version = "0.6.8-alpha.3", path = "../pipeline" } -xvc-walker = { version = "0.6.8-alpha.3", path = "../walker" } -xvc-storage = { version = "0.6.8-alpha.3", path = "../storage" } +xvc = { version = "0.6.8", path = "../lib" } +xvc-config = { version = "0.6.8", path = "../config" } +xvc-core = { version = "0.6.8", path = "../core" } +xvc-logging = { version = "0.6.8", path = "../logging" } +xvc-ecs = { version = "0.6.8", path = "../ecs" } +xvc-file = { version = "0.6.8", path = "../file" } +xvc-pipeline = { version = "0.6.8", path = "../pipeline" } +xvc-walker = { version = "0.6.8", path = "../walker" } +xvc-storage = { version = "0.6.8", path = "../storage" } ## packages for testing clap = { version = "^4.4", features = ["derive"] } @@ -70,7 +70,7 @@ proptest = "^1.4" test-case = "^3.3" globset = "^0.4" escargot = "^0.5" -xvc-test-helper = { version = "0.6.8-alpha.3", path = "../test_helper" } +xvc-test-helper = { version = "0.6.8", path = "../test_helper" } shellfn = "^0.1" jwalk = "^0.8" anyhow = "^1.0" From d9a19eb60af6bbc65ee8b8568686ba54e3d673d0 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Mon, 15 Jul 2024 14:05:47 +0300 Subject: [PATCH 025/257] fix regex --- workflow_tests/tests/test_pipeline_dag.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/workflow_tests/tests/test_pipeline_dag.rs b/workflow_tests/tests/test_pipeline_dag.rs index d15ed39bd..22934abf6 100644 --- a/workflow_tests/tests/test_pipeline_dag.rs +++ b/workflow_tests/tests/test_pipeline_dag.rs @@ -69,7 +69,7 @@ fn test_pipeline_dag() -> Result<()> { xsd("glob_dep", "--glob-items", "data/*")?; xsd("glob_dep", "--param", "params.yaml:model.conv_units")?; // FIXME: There is an error here regarding regex format - xsd("glob_dep", "--regex", "requirements.txt:^tensorflow")?; + xsd("glob_dep", "--regex", "requirements.txt:/^tensorflow")?; xsom("glob_dep", "glob_dep.json")?; xsof("glob_dep", "def.txt")?; xsoi("glob_dep", "plots/confusion.png")?; From f4ffa5362c6a5637e682482b5bc95839fb7b5352 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Mon, 15 Jul 2024 14:19:24 +0300 Subject: [PATCH 026/257] file-copy --- book/src/ref/xvc-file-copy.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/book/src/ref/xvc-file-copy.md b/book/src/ref/xvc-file-copy.md index 5ed8dbbc7..6f0c59c19 100644 --- a/book/src/ref/xvc-file-copy.md +++ b/book/src/ref/xvc-file-copy.md @@ -126,8 +126,11 @@ If the source files you specify are changed, Xvc cancels the copy operation. Ple $ perl -i -pe 's/a/ee/g' data.txt $ xvc file copy data.txt data5.txt +? 1 [ERROR] File Error: Sources have changed, please carry-in or recheck following files before copying: data.txt +Error: FileError { source: AnyhowError { source: Sources have changed, please carry-in or recheck following files before copying: +data.txt } } ``` From 5a21dcd591f5f4c69018a744b88d33d50c20a59a Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Mon, 15 Jul 2024 14:22:40 +0300 Subject: [PATCH 027/257] Update docs --- book/src/ref/xvc-file-list.md | 2 ++ book/src/ref/xvc-file-send.md | 8 ++++---- book/src/ref/xvc-file-share.md | 15 ++++++--------- 3 files changed, 12 insertions(+), 13 deletions(-) diff --git a/book/src/ref/xvc-file-list.md b/book/src/ref/xvc-file-list.md index 026b4b9b5..1df07397d 100644 --- a/book/src/ref/xvc-file-list.md +++ b/book/src/ref/xvc-file-list.md @@ -113,7 +113,9 @@ a repository yet, it reports an error. ```console $ xvc file list +? 1 [ERROR] File Error: [E2004] Requires xvc repository. +Error: FileError { source: RequiresXvcRepository } ``` diff --git a/book/src/ref/xvc-file-send.md b/book/src/ref/xvc-file-send.md index c64005bd1..9f1e83a7f 100644 --- a/book/src/ref/xvc-file-send.md +++ b/book/src/ref/xvc-file-send.md @@ -6,14 +6,14 @@ $ xvc file send --help Send (push, upload) files to external storages -Usage: xvc file send [OPTIONS] --remote [TARGETS]... +Usage: xvc file send [OPTIONS] --storage [TARGETS]... Arguments: [TARGETS]... Targets to send/push/upload to storage Options: - -r, --storage Storage name or guid to send the files - --force Force even if the files are already present in the storage - -h, --help Print help + -s, --storage Storage name or guid to send the files + --force Force even if the files are already present in the storage + -h, --help Print help ``` diff --git a/book/src/ref/xvc-file-share.md b/book/src/ref/xvc-file-share.md index cc59e03ed..814b3b62e 100644 --- a/book/src/ref/xvc-file-share.md +++ b/book/src/ref/xvc-file-share.md @@ -6,13 +6,13 @@ $ xvc file share --help Share a file from S3 compatible storage for a limited time -Usage: xvc file share [OPTIONS] --storage +Usage: xvc file share [OPTIONS] --storage Arguments: File to send/push/upload to storage Options: - -r, --storage Storage name or guid to send the files + -s, --storage Storage name or guid to send the files -d, --duration Period to send the files to. You can use s, m, h, d, w suffixes [default: 24h] -h, --help Print help @@ -58,30 +58,27 @@ $ xvc storage new s3 --name backup --bucket-name xvc-test --region eu-central-1 You must first send files to the remote storage. ```console -$ xvc file send --remote backup dir-0001/ +$ xvc file send --storage backup dir-0001/ ``` Now you can share the files. It will create a URL for you to share that file. (Here we use cut to make the command repeatable) ```console -$ zsh -cl 'xvc file share --remote backup dir-0001/file-0001.bin | cut -c -50' -https://xvc-test.s3.eu-central-1.amazonaws.com/xvc +$ zsh -cl 'xvc file share --storage backup dir-0001/file-0001.bin | cut -c -50' ``` Note that the default period is 24 hours. You can set another period with `--duration`. ```console -$ zsh -cl 'xvc file share --duration 1h --remote backup dir-0001/file-0002.bin | cut -c -50' -https://xvc-test.s3.eu-central-1.amazonaws.com/xvc +$ zsh -cl 'xvc file share --duration 1h --storage backup dir-0001/file-0002.bin | cut -c -50' ``` You can get another URL for a shared file with a different period. ```console -$ zsh -cl 'xvc file share --duration 1m --remote backup dir-0001/file-0002.bin | cut -c -50' -https://xvc-test.s3.eu-central-1.amazonaws.com/xvc +$ zsh -cl 'xvc file share --duration 1m --storage backup dir-0001/file-0002.bin | cut -c -50' ``` From b7fec60e9b5141b2775081b916689a3437ebfe54 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Mon, 15 Jul 2024 14:24:28 +0300 Subject: [PATCH 028/257] update CRLF in pipeline --- book/src/ref/xvc-pipeline-import.md | 2 ++ book/src/ref/xvc-pipeline-run.md | 6 ++++-- book/src/ref/xvc-pipeline-step-dependency-file.md | 9 ++++++--- book/src/ref/xvc-pipeline-step-dependency-generic.md | 9 ++++++--- book/src/ref/xvc-pipeline-step-dependency-glob-items.md | 9 ++++++--- book/src/ref/xvc-pipeline-step-dependency-glob.md | 6 ++++-- 6 files changed, 28 insertions(+), 13 deletions(-) diff --git a/book/src/ref/xvc-pipeline-import.md b/book/src/ref/xvc-pipeline-import.md index 783bb9580..ae4945be1 100644 --- a/book/src/ref/xvc-pipeline-import.md +++ b/book/src/ref/xvc-pipeline-import.md @@ -240,7 +240,9 @@ If you don't supply the `--overwrite` option, Xvc will report an error and quit. ```console $ xvc pipeline import --file pipeline.yaml +? 1 [ERROR] Pipeline Error: Pipeline default already found +Error: PipelineError { source: PipelineAlreadyFound { name: "default" } } ``` diff --git a/book/src/ref/xvc-pipeline-run.md b/book/src/ref/xvc-pipeline-run.md index ed7502766..0e6e46adf 100644 --- a/book/src/ref/xvc-pipeline-run.md +++ b/book/src/ref/xvc-pipeline-run.md @@ -55,9 +55,10 @@ You can run the default pipeline without specifying its name. ```console $ xvc pipeline run [OUT] [hello] hello - + [DONE] hello (echo hello) + ``` Note that, when a step has no dependencies, it's set to always run if it's not set to run never explicitly. @@ -83,7 +84,8 @@ $ xvc pipeline --pipeline-name my-pipeline step new --step-name my-hello --comma ```console $ xvc pipeline run --pipeline-name my-pipeline [OUT] [my-hello] hello from my-pipeline - + [DONE] my-hello (echo 'hello from my-pipeline') + ``` diff --git a/book/src/ref/xvc-pipeline-step-dependency-file.md b/book/src/ref/xvc-pipeline-step-dependency-file.md index d1866423f..2617e4868 100644 --- a/book/src/ref/xvc-pipeline-step-dependency-file.md +++ b/book/src/ref/xvc-pipeline-step-dependency-file.md @@ -26,9 +26,10 @@ When you run the command, it will print `data.txt has changed` if the file `data ```console $ xvc pipeline run [OUT] [file-dependency] data.txt has changed - + [DONE] file-dependency (echo data.txt has changed) + ``` You can add multiple dependencies to a step with multiple invocations. @@ -43,9 +44,10 @@ A step will run if any of its dependencies have changed. ```console $ xvc pipeline run [OUT] [file-dependency] data.txt has changed - + [DONE] file-dependency (echo data.txt has changed) + ``` By default, they are not run if none of the dependencies have changed. @@ -66,7 +68,8 @@ Now the step will run even if none of the dependencies have changed. ```console $ xvc pipeline run [OUT] [file-dependency] data.txt has changed - + [DONE] file-dependency (echo data.txt has changed) + ``` diff --git a/book/src/ref/xvc-pipeline-step-dependency-generic.md b/book/src/ref/xvc-pipeline-step-dependency-generic.md index fd7a27faa..3dfaa0c02 100644 --- a/book/src/ref/xvc-pipeline-step-dependency-generic.md +++ b/book/src/ref/xvc-pipeline-step-dependency-generic.md @@ -35,9 +35,10 @@ The step won't run until tomorrow, when `date +%F` changes. ```console $ xvc pipeline run [OUT] [morning-message] Good Morning! - + [DONE] morning-message (echo 'Good Morning!') + ``` You can mimic all kinds of pipeline behavior with this generic dependency. @@ -50,9 +51,10 @@ $ xvc pipeline step dependency --step-name directory-contents --generic 'ls' $ xvc pipeline run [OUT] [directory-contents] Files changed - + [DONE] directory-contents (echo 'Files changed') + ``` When you add a file to the directory, the step is invalidated and run again: @@ -63,7 +65,8 @@ $ xvc pipeline run $ xvc-test-helper generate-random-file new-file.txt $ xvc pipeline run [OUT] [directory-contents] Files changed - + [DONE] directory-contents (echo 'Files changed') + ``` diff --git a/book/src/ref/xvc-pipeline-step-dependency-glob-items.md b/book/src/ref/xvc-pipeline-step-dependency-glob-items.md index 9e2bc5c76..7abb83cde 100644 --- a/book/src/ref/xvc-pipeline-step-dependency-glob-items.md +++ b/book/src/ref/xvc-pipeline-step-dependency-glob-items.md @@ -62,9 +62,10 @@ dir-0002/file-0003.bin ### Changed Files: - + [DONE] files-changed (echo "### Added Files:/n${XVC_ADDED_GLOB_ITEMS}/n### Removed Files:/n${XVC_REMOVED_GLOB_ITEMS}/n### Changed Files:/n${XVC_CHANGED_GLOB_ITEMS}") + $ xvc pipeline run ``` @@ -81,9 +82,10 @@ $ xvc pipeline run dir-0001/file-0001.bin ### Changed Files: - + [DONE] files-changed (echo "### Added Files:/n${XVC_ADDED_GLOB_ITEMS}/n### Removed Files:/n${XVC_REMOVED_GLOB_ITEMS}/n### Changed Files:/n${XVC_CHANGED_GLOB_ITEMS}") + ``` When you change a file, it's printed in both added and removed files: @@ -98,7 +100,8 @@ $ xvc pipeline run ### Changed Files: dir-0001/file-0002.bin - + [DONE] files-changed (echo "### Added Files:/n${XVC_ADDED_GLOB_ITEMS}/n### Removed Files:/n${XVC_REMOVED_GLOB_ITEMS}/n### Changed Files:/n${XVC_CHANGED_GLOB_ITEMS}") + ``` diff --git a/book/src/ref/xvc-pipeline-step-dependency-glob.md b/book/src/ref/xvc-pipeline-step-dependency-glob.md index 3d326aaf8..9a3b101aa 100644 --- a/book/src/ref/xvc-pipeline-step-dependency-glob.md +++ b/book/src/ref/xvc-pipeline-step-dependency-glob.md @@ -47,9 +47,10 @@ The step is invalidated when a file described by the glob is added, removed or c ```console $ xvc pipeline run [OUT] [files-changed] Files have changed. - + [DONE] files-changed (echo 'Files have changed.') + $ xvc pipeline run ``` @@ -60,7 +61,8 @@ $ rm dir-0001/file-0001.bin $ xvc pipeline run [OUT] [files-changed] Files have changed. - + [DONE] files-changed (echo 'Files have changed.') + ``` From cac89d02fd4ab4d824cd765278ed1ce5c89e5e05 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Mon, 15 Jul 2024 14:25:20 +0300 Subject: [PATCH 029/257] Update docs --- book/src/ref/xvc-pipeline-step-dependency-lines.md | 6 ++++-- book/src/ref/xvc-pipeline-step-dependency-param.md | 9 ++++++--- book/src/ref/xvc-pipeline-step-dependency-regex-items.md | 9 ++++++--- book/src/ref/xvc-pipeline-step-dependency-regex.md | 6 ++++-- book/src/ref/xvc-pipeline-step-dependency-step.md | 9 ++++++--- book/src/ref/xvc-pipeline-step-dependency-url.md | 3 ++- book/src/ref/xvc-storage-new-generic.md | 2 +- book/src/ref/xvc-storage-new-rsync.md | 8 ++------ book/src/ref/xvc-storage-new.md | 2 +- 9 files changed, 32 insertions(+), 22 deletions(-) diff --git a/book/src/ref/xvc-pipeline-step-dependency-lines.md b/book/src/ref/xvc-pipeline-step-dependency-lines.md index 5269ed9c1..951d571bf 100644 --- a/book/src/ref/xvc-pipeline-step-dependency-lines.md +++ b/book/src/ref/xvc-pipeline-step-dependency-lines.md @@ -66,9 +66,10 @@ $ xvc pipeline run "Gwen", "F", 26, 64, 121 "Hank", "M", 30, 71, 158 "Ivan", "M", 53, 72, 175 - + [DONE] print-top-10 (head people.csv) + `````` When you run the pipeline again, the step is not run because the specified lines didn't change. @@ -99,8 +100,9 @@ $ xvc pipeline run "Gwen", "F", 26, 64, 121 "Ferzan", "M", 30, 71, 158 "Ivan", "M", 53, 72, 175 - + [DONE] print-top-10 (head people.csv) + ``` diff --git a/book/src/ref/xvc-pipeline-step-dependency-param.md b/book/src/ref/xvc-pipeline-step-dependency-param.md index 48d90d21f..bb7f1a65f 100644 --- a/book/src/ref/xvc-pipeline-step-dependency-param.md +++ b/book/src/ref/xvc-pipeline-step-dependency-param.md @@ -49,12 +49,14 @@ Run for the first time, as initially all dependencies are invalid: ```console $ xvc pipeline run [OUT] [read-hyperparams] Update Hyperparameters - + [DONE] read-hyperparams (echo "Update Hyperparameters") + [OUT] [read-database-config] Updated Database Configuration - + [DONE] read-database-config (echo "Updated Database Configuration") + ``` For the second time, it won't read the configuration as nothing is changed: @@ -74,9 +76,10 @@ $ perl -pi -e 's/5432/9876/g' myparams.yaml $ xvc pipeline run [OUT] [read-database-config] Updated Database Configuration - + [DONE] read-database-config (echo "Updated Database Configuration") + ``` Note that, `read-hyperparams` is not invalidated, though the values are in the same file. diff --git a/book/src/ref/xvc-pipeline-step-dependency-regex-items.md b/book/src/ref/xvc-pipeline-step-dependency-regex-items.md index 2adeb9b50..402fd3b2a 100644 --- a/book/src/ref/xvc-pipeline-step-dependency-regex-items.md +++ b/book/src/ref/xvc-pipeline-step-dependency-regex-items.md @@ -76,8 +76,9 @@ $ xvc pipeline run "Neil", "M", 36, 75, 160 "Omar", "M", 38, 70, 145 "Quin", "M", 29, 71, 176 - + [DONE] new-males (echo "New Males:/n ${XVC_ADDED_REGEX_ITEMS}") + [OUT] [new-females] New Females: "Elly", "F", 30, 66, 124 "Fran", "F", 33, 66, 115 @@ -86,9 +87,10 @@ $ xvc pipeline run "Myra", "F", 23, 62, 98 "Page", "F", 31, 67, 135 "Ruth", "F", 28, 65, 131 - + [DONE] new-females (echo "New Females:/n ${XVC_ADDED_REGEX_ITEMS}") + `````` When you run the pipeline again, the steps are not run because the regexes didn't change. @@ -129,7 +131,8 @@ $ cat people.csv $ xvc pipeline run [OUT] [new-females] New Females: "Asude", "F", 12, 55, 110 - + [DONE] new-females (echo "New Females:/n ${XVC_ADDED_REGEX_ITEMS}") + ``` diff --git a/book/src/ref/xvc-pipeline-step-dependency-regex.md b/book/src/ref/xvc-pipeline-step-dependency-regex.md index 276fc8c34..ac63d4563 100644 --- a/book/src/ref/xvc-pipeline-step-dependency-regex.md +++ b/book/src/ref/xvc-pipeline-step-dependency-regex.md @@ -56,9 +56,10 @@ When you run the pipeline initially, the steps are run. ```console $ xvc pipeline run [OUT] [count-females] 7 - + [DONE] count-females (grep -c '"F",' people.csv) + `````` When you run the pipeline again, the step is not run because the regex result didn't change. @@ -98,7 +99,8 @@ $ cat people.csv $ xvc pipeline run [OUT] [count-females] 8 - + [DONE] count-females (grep -c '"F",' people.csv) + ``` diff --git a/book/src/ref/xvc-pipeline-step-dependency-step.md b/book/src/ref/xvc-pipeline-step-dependency-step.md index c469b7f99..8f7493c87 100644 --- a/book/src/ref/xvc-pipeline-step-dependency-step.md +++ b/book/src/ref/xvc-pipeline-step-dependency-step.md @@ -24,12 +24,14 @@ When run, the dependency will be run first and the step will be run after. ```console $ xvc pipeline run [OUT] [hello] hello - + [DONE] hello (echo hello) + [OUT] [world] world - + [DONE] world (echo world) + ``` If the dependency is not run, the dependent step won't run either. @@ -46,7 +48,8 @@ If you want to run the dependent always, you can set it to run always explicitly $ xvc pipeline step update --step-name world --when always $ xvc pipeline run [OUT] [world] world - + [DONE] world (echo world) + ``` diff --git a/book/src/ref/xvc-pipeline-step-dependency-url.md b/book/src/ref/xvc-pipeline-step-dependency-url.md index 38f2f1f71..da120083b 100644 --- a/book/src/ref/xvc-pipeline-step-dependency-url.md +++ b/book/src/ref/xvc-pipeline-step-dependency-url.md @@ -25,9 +25,10 @@ The step is invalidated when the page is updated. ```console $ xvc pipeline run [OUT] [xvc-docs-update] Xvc docs updated! - + [DONE] xvc-docs-update (echo 'Xvc docs updated!') + ``` The step won't run again until a new version of the page is published. diff --git a/book/src/ref/xvc-storage-new-generic.md b/book/src/ref/xvc-storage-new-generic.md index ea5f0db44..c2216176d 100644 --- a/book/src/ref/xvc-storage-new-generic.md +++ b/book/src/ref/xvc-storage-new-generic.md @@ -11,7 +11,7 @@ It allows to keep tracked files in any kind of service that can be used command $ xvc storage new generic --help Add a new generic storage. -⚠️ Please note that this is an advanced method to configure storages. You may damage your repository and local and remote files with incorrect configurations. +⚠️ Please note that this is an advanced method to configure storages. You may damage your repository and local and storage files with incorrect configurations. Please see https://docs.xvc.dev/ref/xvc-storage-new-generic.html for examples and make necessary backups. diff --git a/book/src/ref/xvc-storage-new-rsync.md b/book/src/ref/xvc-storage-new-rsync.md index 66940cb2b..6bb2af925 100644 --- a/book/src/ref/xvc-storage-new-rsync.md +++ b/book/src/ref/xvc-storage-new-rsync.md @@ -8,13 +8,9 @@ Configure an S3 (or a compatible) service as an Xvc storage. ```console $ xvc storage new rsync --help -Add a new rsync storage +Add a new rsync storages -Uses rsync in separate processes to communicate with the server. - -This can be used when you already have an SSH/Rsync connection. - -It doesn't prompt for any passwords. The connection must be set up with ssh keys beforehand. +Uses rsync in separate processes to communicate. This can be used when you already have an SSH/Rsync connection. It doesn't prompt for any passwords. The connection must be set up with ssh keys beforehand. Usage: xvc storage new rsync [OPTIONS] --name --host --storage-dir diff --git a/book/src/ref/xvc-storage-new.md b/book/src/ref/xvc-storage-new.md index 44e78d1d8..25ddb581e 100644 --- a/book/src/ref/xvc-storage-new.md +++ b/book/src/ref/xvc-storage-new.md @@ -12,7 +12,7 @@ Usage: xvc storage new Commands: local Add a new local storage generic Add a new generic storage - rsync Add a new rsync storage + rsync Add a new rsync storages s3 Add a new S3 storage minio Add a new Minio storage digital-ocean Add a new Digital Ocean storage From 6320123f2339f38c2d33740939add726c2de3d3a Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Thu, 18 Jul 2024 11:03:29 +0300 Subject: [PATCH 030/257] move generic to bottom --- book/src/ref/xvc-pipeline-step-dependency.md | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) diff --git a/book/src/ref/xvc-pipeline-step-dependency.md b/book/src/ref/xvc-pipeline-step-dependency.md index 1cd7f9910..6d07a8300 100644 --- a/book/src/ref/xvc-pipeline-step-dependency.md +++ b/book/src/ref/xvc-pipeline-step-dependency.md @@ -30,16 +30,16 @@ Options: --glob_items Add a glob items dependency to the step. - + You can depend on multiple files and directories with this dependency. - + The difference between this and the glob option is that this option keeps track of all matching files, but glob only keeps track of the matched files' digest. When you want to use ${XVC_GLOB_ITEMS}, ${XVC_ADDED_GLOB_ITEMS}, or ${XVC_REMOVED_GLOB_ITEMS} environment variables in the step command, use the glob-items dependency. Otherwise, you can use the glob option to save disk space. --glob Add a glob dependency to the step. Can be used multiple times. - + You can depend on multiple files and directories with this dependency. - + The difference between this and the glob-items option is that the glob-items option keeps track of all matching files individually, but this option only keeps track of the matched files' digest. This dependency uses considerably less disk space. --param @@ -47,22 +47,22 @@ Options: --regex_items Add a regex dependency in the form filename.txt:/^regex/ . Can be used multiple times. - + The difference between this and the regex option is that the regex-items option keeps track of all matching lines, but regex only keeps track of the matched lines' digest. When you want to use ${XVC_REGEX_ITEMS}, ${XVC_ADDED_REGEX_ITEMS}, ${XVC_REMOVED_REGEX_ITEMS} environment variables in the step command, use the regex option. Otherwise, you can use the regex-digest option to save disk space. --regex Add a regex dependency in the form filename.txt:/^regex/ . Can be used multiple times. - + The difference between this and the regex option is that the regex option keeps track of all matching lines that can be used in the step command. This option only keeps track of the matched lines' digest. --line_items Add a line dependency in the form filename.txt::123-234 - + The difference between this and the lines option is that the line-items option keeps track of all matching lines that can be used in the step command. This option only keeps track of the matched lines' digest. When you want to use ${XVC_ALL_LINE_ITEMS}, ${XVC_ADDED_LINE_ITEMS}, ${XVC_CHANGED_LINE_ITEMS} options in the step command, use the line option. Otherwise, you can use the lines option to save disk space. --lines Add a line digest dependency in the form filename.txt::123-234 - + The difference between this and the line-items dependency is that the line option keeps track of all matching lines that can be used in the step command. This option only keeps track of the matched lines' digest. If you don't need individual lines to be kept, use this option to save space. -h, --help @@ -70,7 +70,6 @@ Options: ``` -{{#include xvc-pipeline-step-dependency-generic.md}} {{#include xvc-pipeline-step-dependency-file.md}} {{#include xvc-pipeline-step-dependency-glob.md}} {{#include xvc-pipeline-step-dependency-regex.md}} From 96ede583213ca1c61326a632f5d8d169e95cad8f Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Thu, 18 Jul 2024 11:40:58 +0300 Subject: [PATCH 031/257] added xvc-pipeline-step-dependency-sqlite file --- .../xvc-pipeline-step-dependency-sqlite.md | 106 ++++++++++++++++++ 1 file changed, 106 insertions(+) create mode 100644 book/src/ref/xvc-pipeline-step-dependency-sqlite.md diff --git a/book/src/ref/xvc-pipeline-step-dependency-sqlite.md b/book/src/ref/xvc-pipeline-step-dependency-sqlite.md new file mode 100644 index 000000000..0b4fd057f --- /dev/null +++ b/book/src/ref/xvc-pipeline-step-dependency-sqlite.md @@ -0,0 +1,106 @@ +### SQLite Query Dependency + +You can create a step dependency with an SQLite query. When the query results +change, the step is invalidated. + +SQLite dependencies doesn't track the results of the query. It just checks +whether the query results has changed. + +This command works only in Xvc repositories. + +```console +$ git init +... +$ xvc init +``` + +Let's create an SQLite database and populate a table: + +```console +$ sqlite3 people.db < Date: Mon, 22 Jul 2024 12:06:29 +0300 Subject: [PATCH 032/257] Add rusqlite dependency --- pipeline/Cargo.toml | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/pipeline/Cargo.toml b/pipeline/Cargo.toml index 15d111569..e8b7f3a35 100644 --- a/pipeline/Cargo.toml +++ b/pipeline/Cargo.toml @@ -34,9 +34,9 @@ directories-next = "2.0" seahash = "^4.1" byteorder = "^1.5" blake3 = "^1.5" -blake2 = "0.10.6" -sha2 = "0.10.8" -sha3 = "0.10.8" +blake2 = "^0.10" +sha2 = "^0.10" +sha3 = "^0.10" ## Serialization serde = { version = "^1.0", features = ["derive"] } @@ -63,9 +63,14 @@ glob = "^0.3" thiserror = "^1.0" log = "^0.4" anyhow = "^1.0" -### meta-logging-in-format is required for sled: https://github.com/spacejam/sled/issues/1384 + +## meta-logging-in-format is required for sled: https://github.com/spacejam/sled/issues/1384 fern = { version = "^0.6", features = ["colored", "meta-logging-in-format"] } +## sqlite +## TODO: Add feature flags for sqlite when bundli +rusqlite = "^0.32" + ## Networking reqwest = { version = "^0.11", features = ["blocking", "json", "gzip"] } From 6839540b5d7da505dd6628a8c201db4515293943 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Mon, 22 Jul 2024 12:08:04 +0300 Subject: [PATCH 033/257] Updated dependencies --- pipeline/Cargo.toml | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/pipeline/Cargo.toml b/pipeline/Cargo.toml index e8b7f3a35..3f0701cb4 100644 --- a/pipeline/Cargo.toml +++ b/pipeline/Cargo.toml @@ -47,7 +47,7 @@ rmp-serde = "1.1.2" toml = "^0.8" ## Caching -cached = "^0.46" +cached = "^0.53" ## Parallelization rayon = "^1.8" @@ -68,7 +68,7 @@ anyhow = "^1.0" fern = { version = "^0.6", features = ["colored", "meta-logging-in-format"] } ## sqlite -## TODO: Add feature flags for sqlite when bundli +## TODO: Add feature flags for sqlite when bundling needed rusqlite = "^0.32" ## Networking @@ -87,13 +87,13 @@ subprocess = "^0.2" sad_machine = "^1.0" regex = "^1.10" chrono = "^0.4" -strum = "0.25.0" -strum_macros = "0.25.3" +strum = "^0.26" +strum_macros = "^0.26" lazy_static = "^1.4" uuid = { version = "^1.6", features = ["serde", "v4", "fast-rng"] } hex = { version = "^0.4", features = ["serde"] } url = { version = "^2.5", features = ["serde"] } -itertools = "^0.12" +itertools = "^0.13" derive_more = "^0.99" [dev-dependencies] From 4959af5097b667c0b2d278e586b44b36bdb588fb Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Mon, 22 Jul 2024 12:27:32 +0300 Subject: [PATCH 034/257] crate upgrades --- config/Cargo.toml | 4 ++-- core/Cargo.toml | 8 ++++---- file/Cargo.toml | 8 ++++---- lib/Cargo.toml | 6 +++--- storage/Cargo.toml | 12 ++++++------ walker/Cargo.toml | 4 ++-- workflow_tests/Cargo.toml | 8 ++++---- 7 files changed, 25 insertions(+), 25 deletions(-) diff --git a/config/Cargo.toml b/config/Cargo.toml index 58eb69b32..661b427b3 100644 --- a/config/Cargo.toml +++ b/config/Cargo.toml @@ -47,6 +47,6 @@ fern = { version = "^0.6", features = ["colored"] } ## Misc regex = "^1.10" -strum = "0.25.0" -strum_macros = "0.25.3" +strum = "^0.26" +strum_macros = "^0.26" lazy_static = "^1.4" diff --git a/core/Cargo.toml b/core/Cargo.toml index 86c8d55ec..de4d60bc1 100644 --- a/core/Cargo.toml +++ b/core/Cargo.toml @@ -72,14 +72,14 @@ paste = "1.0" ## Misc regex = "^1.10" -strum = "0.25.0" -strum_macros = "0.25.3" +strum = "^0.26" +strum_macros = "^0.26" lazy_static = "^1.4" uuid = { version = "^1.6", features = ["serde", "v4", "fast-rng"] } hex = { version = "^0.4", features = ["serde"] } -cached = "^0.46" +cached = "^0.53" derive_more = "^0.99" -itertools = "^0.12" +itertools = "^0.13" [dev-dependencies] xvc-test-helper = { version = "0.6.8", path = "../test_helper/" } diff --git a/file/Cargo.toml b/file/Cargo.toml index 98e5df38d..c0d09aef3 100644 --- a/file/Cargo.toml +++ b/file/Cargo.toml @@ -56,7 +56,7 @@ toml = "^0.8" rayon = "^1.8" crossbeam-channel = "^0.5" crossbeam = "^0.8" -dashmap = { version = "^5", features = ["serde", "rayon"] } +dashmap = { version = "^6.0", features = ["serde", "rayon"] } ## File system jwalk = "^0.8" @@ -82,13 +82,13 @@ humantime = "^2.1" subprocess = "^0.2" regex = "^1.10" chrono = "^0.4" -strum = "0.25.0" -strum_macros = "0.25.3" +strum = "^0.26" +strum_macros = "^0.26" lazy_static = "^1.4" uuid = { version = "^1.6", features = ["serde", "v4", "fast-rng"] } hex = { version = "^0.4", features = ["serde"] } url = { version = "^2.5", features = ["serde"] } -itertools = "^0.11" +itertools = "^0.13" derive_more = "^0.99" parse-size = "^1.0" diff --git a/lib/Cargo.toml b/lib/Cargo.toml index 3a306ad5c..8ba86806a 100644 --- a/lib/Cargo.toml +++ b/lib/Cargo.toml @@ -64,7 +64,7 @@ reflink = "^0.1" relative-path = { version = "^1.9", features = ["serde"] } path-absolutize = "^3.1" glob = "^0.3" -which = "^4.4" +which = "^6.0" ## Logging and errors thiserror = "^1.0" @@ -82,8 +82,8 @@ paste = "1.0" ## Misc regex = "^1.10" chrono = "^0.4" -strum = "0.25.0" -strum_macros = "0.25.3" +strum = "^0.26" +strum_macros = "^0.26" lazy_static = "^1.4" uuid = { version = "^1.6", features = ["serde", "v4", "fast-rng"] } hex = { version = "^0.4", features = ["serde"] } diff --git a/storage/Cargo.toml b/storage/Cargo.toml index b602988c8..b95cf68f6 100644 --- a/storage/Cargo.toml +++ b/storage/Cargo.toml @@ -57,7 +57,7 @@ walkdir = "^2.4" relative-path = { version = "^1.9", features = ["serde"] } path-absolutize = "^3.1" glob = "^0.3" -which = "^4.4" +which = "^6.0" ## Logging and errors thiserror = "^1.0" @@ -74,13 +74,13 @@ paste = "1.0" humantime = "^2.1" subprocess = "^0.2" regex = "^1.10" -strum = "0.25.0" -strum_macros = "0.25.3" -lazy_static = "^1.4" +strum = "^0.26" +strum_macros = "^0.26" +lazy_static = "^1.5" uuid = { version = "^1.6", features = ["serde", "v4", "fast-rng"] } hex = { version = "^0.4", features = ["serde"] } url = { version = "^2.5", features = ["serde"] } -itertools = "^0.12" +itertools = "^0.13" derive_more = "^0.99" tempfile = "^3.9" @@ -88,7 +88,7 @@ tempfile = "^3.9" tokio = { version = "^1.35", optional = true, features = ["rt-multi-thread"] } -rust-s3 = { version = "^0.33", optional = true } +rust-s3 = { version = "^0.34", optional = true } futures = { version = "^0.3", optional = true } # Although we don't use openssl directly, we use its vendored version for cross compilation diff --git a/walker/Cargo.toml b/walker/Cargo.toml index 6c3296666..d437885b6 100644 --- a/walker/Cargo.toml +++ b/walker/Cargo.toml @@ -29,7 +29,7 @@ crossbeam = "^0.8" ## File system notify = "6.1.1" -dashmap = "^5.5" +dashmap = "^6" ## Logging and errors thiserror = "^1.0" @@ -37,7 +37,7 @@ anyhow = "^1.0" log = "^0.4" ## Misc -itertools = "^0.12" +itertools = "^0.13" regex = "^1.10" [dev-dependencies] diff --git a/workflow_tests/Cargo.toml b/workflow_tests/Cargo.toml index 5c3d3e8da..e23de04a9 100644 --- a/workflow_tests/Cargo.toml +++ b/workflow_tests/Cargo.toml @@ -36,7 +36,7 @@ xvc-storage = { version = "0.6.8", path = "../storage" } ## packages for testing clap = { version = "^4.4", features = ["derive"] } -itertools = "^0.12" +itertools = "^0.13" serde = { version = "^1.0", features = ["derive"] } serde_yaml = "^0.9" serde_json = "^1.0" @@ -45,7 +45,7 @@ subprocess = "^0.2" log = "^0.4" regex = "^1.10" rand = "^0.8" -which = "^5.0" +which = "^6.0" [features] default = ["s3", "minio", "r2", "gcs", "digital-ocean"] @@ -77,6 +77,6 @@ anyhow = "^1.0" assert_cmd = "^2.0" assert_fs = "^1.1" predicates = "^3.0" -trycmd = "^0.14" -which = "^5.0" +trycmd = "^0.15" +which = "^6.0" fs_extra = "^1.3" From e9e2ebc7f6d3b735810705ec8d0e6ef0d4af78cf Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Mon, 22 Jul 2024 12:35:10 +0300 Subject: [PATCH 035/257] Updated to v0.6.9-a.0 --- config/Cargo.toml | 6 +++--- core/Cargo.toml | 12 ++++++------ ecs/Cargo.toml | 4 ++-- file/Cargo.toml | 16 ++++++++-------- lib/Cargo.toml | 20 ++++++++++---------- logging/Cargo.toml | 2 +- pipeline/Cargo.toml | 16 ++++++++-------- storage/Cargo.toml | 14 +++++++------- test_helper/Cargo.toml | 4 ++-- walker/Cargo.toml | 6 +++--- workflow_tests/Cargo.toml | 22 +++++++++++----------- 11 files changed, 61 insertions(+), 61 deletions(-) diff --git a/config/Cargo.toml b/config/Cargo.toml index 661b427b3..e09394202 100644 --- a/config/Cargo.toml +++ b/config/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-config" -version = "0.6.8" +version = "0.6.9-alpha.0" edition = "2021" description = "Xvc configuration management" authors = ["Emre Şahin "] @@ -20,8 +20,8 @@ debug = true [dependencies] -xvc-logging = { version = "0.6.8", path = "../logging" } -xvc-walker = { version = "0.6.8", path = "../walker" } +xvc-logging = { version = "0.6.9-alpha.0", path = "../logging" } +xvc-walker = { version = "0.6.9-alpha.0", path = "../walker" } ## Cli and config diff --git a/core/Cargo.toml b/core/Cargo.toml index de4d60bc1..f7da1bd24 100644 --- a/core/Cargo.toml +++ b/core/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-core" -version = "0.6.8" +version = "0.6.9-alpha.0" edition = "2021" description = "Xvc core for common elements for all commands" authors = ["Emre Şahin "] @@ -19,10 +19,10 @@ crate-type = ["rlib"] debug = true [dependencies] -xvc-config = { version = "0.6.8", path = "../config" } -xvc-logging = { version = "0.6.8", path = "../logging" } -xvc-ecs = { version = "0.6.8", path = "../ecs" } -xvc-walker = { version = "0.6.8", path = "../walker" } +xvc-config = { version = "0.6.9-alpha.0", path = "../config" } +xvc-logging = { version = "0.6.9-alpha.0", path = "../logging" } +xvc-ecs = { version = "0.6.9-alpha.0", path = "../ecs" } +xvc-walker = { version = "0.6.9-alpha.0", path = "../walker" } ## Cli and config clap = { version = "^4.4", features = ["derive"] } @@ -82,6 +82,6 @@ derive_more = "^0.99" itertools = "^0.13" [dev-dependencies] -xvc-test-helper = { version = "0.6.8", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.9-alpha.0", path = "../test_helper/" } proptest = "^1.4" test-case = "^3.3" diff --git a/ecs/Cargo.toml b/ecs/Cargo.toml index 835e749af..b1b97a565 100644 --- a/ecs/Cargo.toml +++ b/ecs/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-ecs" -version = "0.6.8" +version = "0.6.9-alpha.0" edition = "2021" description = "Entity-Component System for Xvc" authors = ["Emre Şahin "] @@ -19,7 +19,7 @@ crate-type = ["rlib"] debug = true [dependencies] -xvc-logging = { version = "0.6.8", path = "../logging" } +xvc-logging = { version = "0.6.9-alpha.0", path = "../logging" } ## Serialization serde = { version = "^1.0", features = ["derive"] } diff --git a/file/Cargo.toml b/file/Cargo.toml index c0d09aef3..c0a9c50f4 100644 --- a/file/Cargo.toml +++ b/file/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-file" -version = "0.6.8" +version = "0.6.9-alpha.0" edition = "2021" description = "File tracking, versioning, upload and download functions for Xvc" authors = ["Emre Şahin "] @@ -24,12 +24,12 @@ bench = true debug = true [dependencies] -xvc-logging = { version = "0.6.8", path = "../logging" } -xvc-config = { version = "0.6.8", path = "../config" } -xvc-core = { version = "0.6.8", path = "../core" } -xvc-ecs = { version = "0.6.8", path = "../ecs" } -xvc-walker = { version = "0.6.8", path = "../walker" } -xvc-storage = { version = "0.6.8", path = "../storage" } +xvc-logging = { version = "0.6.9-alpha.0", path = "../logging" } +xvc-config = { version = "0.6.9-alpha.0", path = "../config" } +xvc-core = { version = "0.6.9-alpha.0", path = "../core" } +xvc-ecs = { version = "0.6.9-alpha.0", path = "../ecs" } +xvc-walker = { version = "0.6.9-alpha.0", path = "../walker" } +xvc-storage = { version = "0.6.9-alpha.0", path = "../storage" } ## Cli and config @@ -94,5 +94,5 @@ parse-size = "^1.0" [dev-dependencies] -xvc-test-helper = { version = "0.6.8", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.9-alpha.0", path = "../test_helper/" } shellfn = "^0.1" diff --git a/lib/Cargo.toml b/lib/Cargo.toml index 8ba86806a..329260f57 100644 --- a/lib/Cargo.toml +++ b/lib/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc" -version = "0.6.8" +version = "0.6.9-alpha.0" edition = "2021" description = "An MLOps tool to manage data files and pipelines on top of Git" authors = ["Emre Şahin "] @@ -23,14 +23,14 @@ path = "src/main.rs" debug = true [dependencies] -xvc-config = { version = "0.6.8", path = "../config" } -xvc-core = { version = "0.6.8", path = "../core" } -xvc-logging = { version = "0.6.8", path = "../logging" } -xvc-ecs = { version = "0.6.8", path = "../ecs" } -xvc-file = { version = "0.6.8", path = "../file" } -xvc-pipeline = { version = "0.6.8", path = "../pipeline" } -xvc-walker = { version = "0.6.8", path = "../walker" } -xvc-storage = { version = "0.6.8", path = "../storage" } +xvc-config = { version = "0.6.9-alpha.0", path = "../config" } +xvc-core = { version = "0.6.9-alpha.0", path = "../core" } +xvc-logging = { version = "0.6.9-alpha.0", path = "../logging" } +xvc-ecs = { version = "0.6.9-alpha.0", path = "../ecs" } +xvc-file = { version = "0.6.9-alpha.0", path = "../file" } +xvc-pipeline = { version = "0.6.9-alpha.0", path = "../pipeline" } +xvc-walker = { version = "0.6.9-alpha.0", path = "../walker" } +xvc-storage = { version = "0.6.9-alpha.0", path = "../storage" } ## Cli and config @@ -113,4 +113,4 @@ jwalk = "^0.8" proptest = "^1.4" shellfn = "^0.1" test-case = "^3.3" -xvc-test-helper = { version = "0.6.8", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.9-alpha.0", path = "../test_helper/" } diff --git a/logging/Cargo.toml b/logging/Cargo.toml index 3a7fe0a07..4c61aead5 100644 --- a/logging/Cargo.toml +++ b/logging/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-logging" -version = "0.6.8" +version = "0.6.9-alpha.0" edition = "2021" description = "Logging crate for Xvc" authors = ["Emre Şahin "] diff --git a/pipeline/Cargo.toml b/pipeline/Cargo.toml index 3f0701cb4..bd6949ec9 100644 --- a/pipeline/Cargo.toml +++ b/pipeline/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-pipeline" -version = "0.6.8" +version = "0.6.9-alpha.0" edition = "2021" description = "Xvc data pipeline management" authors = ["Emre Şahin "] @@ -19,12 +19,12 @@ debug = true [dependencies] -xvc-config = { version = "0.6.8", path = "../config" } -xvc-core = { version = "0.6.8", path = "../core" } -xvc-ecs = { version = "0.6.8", path = "../ecs" } -xvc-logging = { version = "0.6.8", path = "../logging" } -xvc-walker = { version = "0.6.8", path = "../walker" } -xvc-file = { version = "0.6.8", path = "../file" } +xvc-config = { version = "0.6.9-alpha.0", path = "../config" } +xvc-core = { version = "0.6.9-alpha.0", path = "../core" } +xvc-ecs = { version = "0.6.9-alpha.0", path = "../ecs" } +xvc-logging = { version = "0.6.9-alpha.0", path = "../logging" } +xvc-walker = { version = "0.6.9-alpha.0", path = "../walker" } +xvc-file = { version = "0.6.9-alpha.0", path = "../file" } ## Cli and config clap = { version = "^4.4", features = ["derive"] } @@ -97,5 +97,5 @@ itertools = "^0.13" derive_more = "^0.99" [dev-dependencies] -xvc-test-helper = { version = "0.6.8", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.9-alpha.0", path = "../test_helper/" } test-case = "^3.3" diff --git a/storage/Cargo.toml b/storage/Cargo.toml index b95cf68f6..408e47114 100644 --- a/storage/Cargo.toml +++ b/storage/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-storage" -version = "0.6.8" +version = "0.6.9-alpha.0" edition = "2021" description = "Xvc remote and local storage management" authors = ["Emre Şahin "] @@ -20,11 +20,11 @@ debug = true [dependencies] -xvc-logging = { version = "0.6.8", path = "../logging" } -xvc-config = { version = "0.6.8", path = "../config" } -xvc-core = { version = "0.6.8", path = "../core" } -xvc-ecs = { version = "0.6.8", path = "../ecs" } -xvc-walker = { version = "0.6.8", path = "../walker" } +xvc-logging = { version = "0.6.9-alpha.0", path = "../logging" } +xvc-config = { version = "0.6.9-alpha.0", path = "../config" } +xvc-core = { version = "0.6.9-alpha.0", path = "../core" } +xvc-ecs = { version = "0.6.9-alpha.0", path = "../ecs" } +xvc-walker = { version = "0.6.9-alpha.0", path = "../walker" } ## Cli and config clap = { version = "^4.4", features = ["derive"] } @@ -109,7 +109,7 @@ digital-ocean = ["async"] [dev-dependencies] -xvc-test-helper = { version = "0.6.8", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.9-alpha.0", path = "../test_helper/" } shellfn = "^0.1" [package.metadata.cargo-udeps.ignore] diff --git a/test_helper/Cargo.toml b/test_helper/Cargo.toml index fa20c7380..d388d4270 100644 --- a/test_helper/Cargo.toml +++ b/test_helper/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-test-helper" -version = "0.6.8" +version = "0.6.9-alpha.0" edition = "2021" description = "Unit test helper functions for Xvc" authors = ["Emre Şahin "] @@ -23,7 +23,7 @@ debug = true # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html [dependencies] -xvc-logging = { version = "0.6.8", path = "../logging/" } +xvc-logging = { version = "0.6.9-alpha.0", path = "../logging/" } rand = "^0.8" log = "^0.4" diff --git a/walker/Cargo.toml b/walker/Cargo.toml index d437885b6..82dedf82a 100644 --- a/walker/Cargo.toml +++ b/walker/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-walker" -version = "0.6.8" +version = "0.6.9-alpha.0" edition = "2021" description = "Xvc parallel file system walker with ignore features" authors = ["Emre Şahin "] @@ -19,7 +19,7 @@ crate-type = ["rlib"] debug = true [dependencies] -xvc-logging = { version = "0.6.8", path = "../logging" } +xvc-logging = { version = "0.6.9-alpha.0", path = "../logging" } globset = "^0.4" ## Parallelization @@ -41,7 +41,7 @@ itertools = "^0.13" regex = "^1.10" [dev-dependencies] -xvc-test-helper = { path = "../test_helper/", version = "0.6.8" } +xvc-test-helper = { path = "../test_helper/", version = "0.6.9-alpha.0" } test-case = "^3.3" [package.metadata.cargo-udeps.ignore] diff --git a/workflow_tests/Cargo.toml b/workflow_tests/Cargo.toml index e23de04a9..6933f8b9d 100644 --- a/workflow_tests/Cargo.toml +++ b/workflow_tests/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-workflow-tests" -version = "0.6.8" +version = "0.6.9-alpha.0" edition = "2021" description = "Integration test suite for Xvc" authors = ["Emre Şahin "] @@ -23,15 +23,15 @@ debug = true [dependencies] -xvc = { version = "0.6.8", path = "../lib" } -xvc-config = { version = "0.6.8", path = "../config" } -xvc-core = { version = "0.6.8", path = "../core" } -xvc-logging = { version = "0.6.8", path = "../logging" } -xvc-ecs = { version = "0.6.8", path = "../ecs" } -xvc-file = { version = "0.6.8", path = "../file" } -xvc-pipeline = { version = "0.6.8", path = "../pipeline" } -xvc-walker = { version = "0.6.8", path = "../walker" } -xvc-storage = { version = "0.6.8", path = "../storage" } +xvc = { version = "0.6.9-alpha.0", path = "../lib" } +xvc-config = { version = "0.6.9-alpha.0", path = "../config" } +xvc-core = { version = "0.6.9-alpha.0", path = "../core" } +xvc-logging = { version = "0.6.9-alpha.0", path = "../logging" } +xvc-ecs = { version = "0.6.9-alpha.0", path = "../ecs" } +xvc-file = { version = "0.6.9-alpha.0", path = "../file" } +xvc-pipeline = { version = "0.6.9-alpha.0", path = "../pipeline" } +xvc-walker = { version = "0.6.9-alpha.0", path = "../walker" } +xvc-storage = { version = "0.6.9-alpha.0", path = "../storage" } ## packages for testing clap = { version = "^4.4", features = ["derive"] } @@ -70,7 +70,7 @@ proptest = "^1.4" test-case = "^3.3" globset = "^0.4" escargot = "^0.5" -xvc-test-helper = { version = "0.6.8", path = "../test_helper" } +xvc-test-helper = { version = "0.6.9-alpha.0", path = "../test_helper" } shellfn = "^0.1" jwalk = "^0.8" anyhow = "^1.0" From bb540a6083541fc8523a8507575e998fa4108f7a Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Mon, 22 Jul 2024 15:17:50 +0300 Subject: [PATCH 036/257] Update track --- book/src/ref/xvc-file-track.md | 71 ++++++++++++++++------------------ 1 file changed, 33 insertions(+), 38 deletions(-) diff --git a/book/src/ref/xvc-file-track.md b/book/src/ref/xvc-file-track.md index 628b8de65..cb6c4b219 100644 --- a/book/src/ref/xvc-file-track.md +++ b/book/src/ref/xvc-file-track.md @@ -4,7 +4,7 @@ `xvc file track` is used to register any kind of file to Xvc for tracking versions. -## Synopsis +## Synopsis ```console $ xvc file track --help @@ -19,7 +19,7 @@ Arguments: Options: --recheck-method How to track the file contents in cache: One of copy, symlink, hardlink, reflink. - + Note: Reflink uses copy if the underlying file system doesn't support it. --no-commit @@ -41,7 +41,6 @@ Options: ## Examples - File tracking works only in Xvc repositories. ```console @@ -50,7 +49,8 @@ $ git init $ xvc init ``` -Let's create a directory tree for these examples. + +Let's create a directory tree for these examples. ```console $ xvc-test-helper create-directory-tree --directories 4 --files 3 --seed 20231021 @@ -77,7 +77,7 @@ $ tree ``` -By default, the command runs similar to `git add` and `git commit`. +By default, the command runs similar to `git add` and `git commit`. You can track individual files. @@ -85,13 +85,13 @@ You can track individual files. $ xvc file track dir-0001/file-0001.bin ``` -You can track directories with the same command. +You can track directories with the same command. ```console $ xvc file track dir-0002/ ``` -You can specify more than one target in a single command. +You can specify more than one target in a single command. ```console $ xvc file track dir-0001/file-0002.bin dir-0001/file-0003.bin @@ -100,7 +100,7 @@ $ xvc file track dir-0001/file-0002.bin dir-0001/file-0003.bin When you track a file, Xvc moves the file to the cache directory under `.xvc/` and _connects_ the workspace file with the cached file. This _connection_ is called rechecking and analogous to Git checkout. For example, the above -commands create a directory tree under `.xvc` as follows: +commands create a directory tree under `.xvc` as follows: ```console $ tree .xvc/b3 @@ -124,31 +124,30 @@ $ tree .xvc/b3 There are different _recheck (checkout) methods_ that Xvc connects the workspace file to the cache. The default method for this is copying the file to -the workspace. This way a separate copy of the cache file is created in the workspace. +the workspace. This way a separate copy of the cache file is created in the workspace. -If you want to make this connection with symbolic links, you can specify it with `--recheck-method` option. +If you want to make this connection with symbolic links, you can specify it with `--recheck-method` option. ```console $ xvc file track --recheck-method symlink dir-0003/file-0001.bin $ ls -l dir-0003/file-0001.bin -lrwxr-xr-x 1 [..] dir-0003/file-0001.bin -> [CWD]/.xvc/b3/e51/7d6/b9a3617fdcd96bd128142a39f1eca26ed77a338d2b93ba4921a0116c70/0.bin +lrwxr-xr-x@ [..] dir-0003/file-0001.bin -> [CWD]/.xvc/b3/e51/7d6/b9a3617fdcd96bd128142a39f1eca26ed77a338d2b93ba4921a0116c70/0.bin ``` -You can also use `--hardlink` and `--reflink` options. Please see [`xvc file recheck`](/ref/xvc-file-recheck/) reference for details. +You can also use `--hardlink` and `--reflink` options. Please see [`xvc file recheck`](/ref/xvc-file-recheck/) reference for details. ```console $ xvc file track --recheck-method hardlink dir-0003/file-0002.bin $ xvc file track --recheck-method reflink dir-0003/file-0003.bin $ ls -l dir-0003/ total 16 -lrwxr-xr-x 1 [..] file-0001.bin -> [CWD]/.xvc/b3/e51/7d6/b9a3617fdcd96bd128142a39f1eca26ed77a338d2b93ba4921a0116c70/0.bin --r--r--r-- 2 [..] file-0002.bin --r--r--r-- 1 [..] file-0003.bin +lrwxr-xr-x@ [..] file-0001.bin -> [CWD]/.xvc/b3/e51/7d6/b9a3617fdcd96bd128142a39f1eca26ed77a338d2b93ba4921a0116c70/0.bin +-r--r--r--@ [..] file-0002.bin +-r--r--r--@ [..] file-0003.bin ``` - ```admonish info Note that, unlike DVC that specifies checkout/recheck option repository wide, Xvc lets you specify per file. You can recheck files data files as symbolic @@ -162,15 +161,15 @@ When you track a file in Xvc, it's automatically commit (carry-in) to the cache directory. If you want to postpone this operation and don't need a cached copy for a file, you can use `--no-commit` option. You can later use [xvc file carry-in](/ref/xvc-file-carry-in) command to move these files to the repository -cache. +cache. ```console $ xvc file track --no-commit --recheck-method symlink dir-0004/ $ ls -l dir-0004/ total 24 --rw-r--r-- 1 [..] file-0001.bin --rw-r--r-- 1 [..] file-0002.bin --rw-r--r-- 1 [..] file-0003.bin +-rw-r--r--@ [..] file-0001.bin +-rw-r--r--@ [..] file-0002.bin +-rw-r--r--@ [..] file-0003.bin $ xvc file list dir-0004/ FS [..] ab361981 ab361981 dir-0004/file-0003.bin @@ -180,25 +179,25 @@ Total #: 3 Workspace Size: 6006 Cached Size: 6006 ``` + You can carry-in (commit) these files to the cache with `xvc file carry-in` command. Note that, as the files are deduplicated, we need to use `--force` in carry-in command. This behavior may change in the future. - ```console $ xvc file carry-in --force dir-0004/ $ ls -l dir-0004/ total 0 -lrwxr-xr-x 1 [..] file-0001.bin -> [CWD]/.xvc/b3/e51/7d6/b9a3617fdcd96bd128142a39f1eca26ed77a338d2b93ba4921a0116c70/0.bin -lrwxr-xr-x 1 [..] file-0002.bin -> [CWD]/.xvc/b3/493/eeb/6525ea5e94e1e760371108e4a525c696c773a774a4818e941fd6d1af79/0.bin -lrwxr-xr-x 1 [..] file-0003.bin -> [CWD]/.xvc/b3/ab3/619/814cae0456a5a291e4d5c8d339a8389630e476f9f9e8d3a09accc919f0/0.bin +lrwxr-xr-x@ [..] file-0001.bin -> [CWD]/.xvc/b3/e51/7d6/b9a3617fdcd96bd128142a39f1eca26ed77a338d2b93ba4921a0116c70/0.bin +lrwxr-xr-x@ [..] file-0002.bin -> [CWD]/.xvc/b3/493/eeb/6525ea5e94e1e760371108e4a525c696c773a774a4818e941fd6d1af79/0.bin +lrwxr-xr-x@ [..] file-0003.bin -> [CWD]/.xvc/b3/ab3/619/814cae0456a5a291e4d5c8d339a8389630e476f9f9e8d3a09accc919f0/0.bin ``` -Xvc dedeuplicates the files in the cache. If you track a file that is already +Xvc deduplicates files in the cache. If you track a file that is already in the cache, it won't be moved to the cache again. It will be copied, linked -from the same copy. +from the same copy. ```console $ tree .xvc/b3 @@ -220,22 +219,18 @@ $ tree .xvc/b3 ``` - ## Caveats -- This command doesn't discriminate symbolic links or hardlinks. -Links are followed and any broken links may cause errors. +- This command doesn't discriminate symbolic links or hardlinks. + Links are followed and any broken links may cause errors. -- Under the hood, Xvc tracks only the files, not directories. -Directories are considered as path collections. -It doesn't matter if you track a directory or files in it separately. +- Under the hood, Xvc tracks only the files, not directories. + Directories are considered as path collections. + It doesn't matter if you track a directory or files in it separately. ## Technical Details - Detecting changes in files and directories employ different kinds of [associated digests](/concepts/associated-digest.md). -If a file has different metadata digest, its content digest is calculated. -If file's content digest has changed, the file is considered changed. -A directory that contains different set of files, or files with changed content is considered changed. - - - + If a file has different metadata digest, its content digest is calculated. + If file's content digest has changed, the file is considered changed. + A directory that contains different set of files, or files with changed content is considered changed. From b9966f8d7673acc79fd075320c8ca8c3f6a4f874 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Mon, 22 Jul 2024 15:18:16 +0300 Subject: [PATCH 037/257] update --- book/src/ref/xvc-file-share.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/book/src/ref/xvc-file-share.md b/book/src/ref/xvc-file-share.md index 814b3b62e..de2a2c7e6 100644 --- a/book/src/ref/xvc-file-share.md +++ b/book/src/ref/xvc-file-share.md @@ -65,6 +65,7 @@ Now you can share the files. It will create a URL for you to share that file. (H ```console $ zsh -cl 'xvc file share --storage backup dir-0001/file-0001.bin | cut -c -50' +https://xvc-test.s3.eu-central-1.amazonaws.com/xvc ``` @@ -72,6 +73,7 @@ Note that the default period is 24 hours. You can set another period with `--dur ```console $ zsh -cl 'xvc file share --duration 1h --storage backup dir-0001/file-0002.bin | cut -c -50' +https://xvc-test.s3.eu-central-1.amazonaws.com/xvc ``` @@ -79,6 +81,7 @@ You can get another URL for a shared file with a different period. ```console $ zsh -cl 'xvc file share --duration 1m --storage backup dir-0001/file-0002.bin | cut -c -50' +https://xvc-test.s3.eu-central-1.amazonaws.com/xvc ``` From 8fb8ff91bac43a9cc45a68d616505fe456657355 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Mon, 22 Jul 2024 21:21:35 +0300 Subject: [PATCH 038/257] cached type is now ty --- core/src/util/file.rs | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-) diff --git a/core/src/util/file.rs b/core/src/util/file.rs index 68f53d77c..2ecf8a68f 100644 --- a/core/src/util/file.rs +++ b/core/src/util/file.rs @@ -1,7 +1,5 @@ //! Core file operationscorefil use cached::proc_macro::cached; -use cached::UnboundCache; -use glob::Pattern as GlobPattern; use std::fs::{self, Metadata}; use std::io::{self, Read}; @@ -14,7 +12,6 @@ use std::path::{Path, PathBuf}; use xvc_logging::watch; use xvc_walker::{IgnoreRules, PathMetadata, WalkOptions}; -use crate::error::Error; use crate::error::Result; use crate::CHANNEL_BOUND; use crossbeam_channel::{bounded, Receiver, Sender}; @@ -79,7 +76,7 @@ pub fn all_paths_and_metadata(xvc_root: &XvcRoot) -> (XvcPathMetadataMap, Ignore /// Returns a compiled [glob::Pattern] by prepending it with `pipeline_rundir`. #[cached( - type = "UnboundCache", + ty = "UnboundCache", create = "{ UnboundCache::new() }", convert = r#"{ format!("{:?}{}", pipeline_rundir, glob) }"#, result = true @@ -91,7 +88,7 @@ pub fn compiled_glob(pipeline_rundir: &Path, glob: &str) -> Result Date: Mon, 22 Jul 2024 21:22:23 +0300 Subject: [PATCH 039/257] fix import --- core/src/util/file.rs | 1 + 1 file changed, 1 insertion(+) diff --git a/core/src/util/file.rs b/core/src/util/file.rs index 2ecf8a68f..a4c4da67e 100644 --- a/core/src/util/file.rs +++ b/core/src/util/file.rs @@ -1,5 +1,6 @@ //! Core file operationscorefil use cached::proc_macro::cached; +use cached::UnboundCache; use std::fs::{self, Metadata}; use std::io::{self, Read}; From 4e31457b37058a81c3fe378939030581cbeac4ff Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Mon, 22 Jul 2024 21:24:16 +0300 Subject: [PATCH 040/257] import fixes --- core/src/util/file.rs | 2 ++ 1 file changed, 2 insertions(+) diff --git a/core/src/util/file.rs b/core/src/util/file.rs index a4c4da67e..8a9d47709 100644 --- a/core/src/util/file.rs +++ b/core/src/util/file.rs @@ -1,6 +1,7 @@ //! Core file operationscorefil use cached::proc_macro::cached; use cached::UnboundCache; +use glob::Pattern as GlobPattern; use std::fs::{self, Metadata}; use std::io::{self, Read}; @@ -13,6 +14,7 @@ use std::path::{Path, PathBuf}; use xvc_logging::watch; use xvc_walker::{IgnoreRules, PathMetadata, WalkOptions}; +use crate::error::Error; use crate::error::Result; use crate::CHANNEL_BOUND; use crossbeam_channel::{bounded, Receiver, Sender}; From 3bed62bb9aa461918c2cb836d62f19654cf4200d Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Mon, 22 Jul 2024 21:28:43 +0300 Subject: [PATCH 041/257] Fix async API change --- storage/src/storage/async_common.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/src/storage/async_common.rs b/storage/src/storage/async_common.rs index 43df2ab77..534cc944e 100644 --- a/storage/src/storage/async_common.rs +++ b/storage/src/storage/async_common.rs @@ -257,7 +257,7 @@ pub trait XvcS3StorageOperations { let expiration_seconds = duration.as_secs() as u32; let path = self.build_storage_path(path); - let signed_url = bucket.presign_get(path.as_str(), expiration_seconds, None)?; + let signed_url = bucket.presign_get(path.as_str(), expiration_seconds, None).await?; info!(output, "[SHARED] {}", path.as_str()); output!(output, "{}", signed_url); Ok(super::XvcStorageExpiringShareEvent { From 8cc2c084da360141db03e777c64aae096c4f2c79 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Mon, 22 Jul 2024 21:29:47 +0300 Subject: [PATCH 042/257] one more result check --- storage/src/storage/async_common.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/storage/src/storage/async_common.rs b/storage/src/storage/async_common.rs index 534cc944e..f2ee1f5f5 100644 --- a/storage/src/storage/async_common.rs +++ b/storage/src/storage/async_common.rs @@ -197,7 +197,7 @@ pub trait XvcS3StorageOperations { info!(output_snd, "{} -> {}", storage_path.as_str(), abs_cache_path); let mut async_cache_path = tokio::fs::File::create(&abs_cache_path).await?; while let Some(chunk) = response.bytes().next().await { - async_cache_path.write_all(&chunk).await?; + async_cache_path.write_all(&chunk?).await?; } copied_paths.push(storage_path); watch!(copied_paths.len()); From 6cc63b587694d0838f8e98274aa8af733aa5bfe5 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Mon, 22 Jul 2024 21:32:58 +0300 Subject: [PATCH 043/257] update docs --- book/src/ref/xvc-file-move.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/book/src/ref/xvc-file-move.md b/book/src/ref/xvc-file-move.md index 3b1084233..7a8985af2 100644 --- a/book/src/ref/xvc-file-move.md +++ b/book/src/ref/xvc-file-move.md @@ -11,22 +11,22 @@ Usage: xvc file move [OPTIONS] Arguments: Source file, glob or directory within the workspace. - + If the source ends with a slash, it's considered a directory and all files in that directory are copied. - + If there are multiple source files, the destination must be a directory. Location we move file(s) to within the workspace. - + If this ends with a slash, it's considered a directory and created if it doesn't exist. - + If the number of source files is more than one, the destination must be a directory. Options: --recheck-method How the destination should be rechecked: One of copy, symlink, hardlink, reflink. - + Note: Reflink uses copy if the underlying file system doesn't support it. --no-recheck @@ -78,7 +78,7 @@ $ xvc file move data2.txt data3.txt --as symlink $ ls -l total[..] -lrwxr-xr-x 1 [..] data3.txt -> [CWD]/.xvc/b3/c85/f3e/8108a0d53da6b4869e5532a3b72301ed58d5824ed1394d52dbcabe9496/0.txt +lrwxr-xr-x[..] data3.txt -> [CWD]/.xvc/b3/c85/f3e/8108a0d53da6b4869e5532a3b72301ed58d5824ed1394d52dbcabe9496/0.txt ``` @@ -91,7 +91,7 @@ $ xvc file move data3.txt data4.txt $ ls -l total 0 -lrwxr-xr-x [..] data4.txt -> [CWD]/.xvc/b3/c85/f3e/8108a0d53da6b4869e5532a3b72301ed58d5824ed1394d52dbcabe9496/0.txt +lrwxr-xr-x[..] data4.txt -> [CWD]/.xvc/b3/c85/f3e/8108a0d53da6b4869e5532a3b72301ed58d5824ed1394d52dbcabe9496/0.txt ``` From 5cac16696b2628d204be72e119516b1e5701165c Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Mon, 22 Jul 2024 21:36:25 +0300 Subject: [PATCH 044/257] move and recheck --- book/src/ref/xvc-file-move.md | 10 +++++----- book/src/ref/xvc-file-recheck.md | 34 ++++++++++++++++---------------- 2 files changed, 22 insertions(+), 22 deletions(-) diff --git a/book/src/ref/xvc-file-move.md b/book/src/ref/xvc-file-move.md index 7a8985af2..1c5a82f43 100644 --- a/book/src/ref/xvc-file-move.md +++ b/book/src/ref/xvc-file-move.md @@ -11,22 +11,22 @@ Usage: xvc file move [OPTIONS] Arguments: Source file, glob or directory within the workspace. - + If the source ends with a slash, it's considered a directory and all files in that directory are copied. - + If there are multiple source files, the destination must be a directory. Location we move file(s) to within the workspace. - + If this ends with a slash, it's considered a directory and created if it doesn't exist. - + If the number of source files is more than one, the destination must be a directory. Options: --recheck-method How the destination should be rechecked: One of copy, symlink, hardlink, reflink. - + Note: Reflink uses copy if the underlying file system doesn't support it. --no-recheck diff --git a/book/src/ref/xvc-file-recheck.md b/book/src/ref/xvc-file-recheck.md index 63b8f3b9e..fbd317b7f 100644 --- a/book/src/ref/xvc-file-recheck.md +++ b/book/src/ref/xvc-file-recheck.md @@ -15,7 +15,7 @@ Arguments: Options: --recheck-method How to track the file contents in cache: One of copy, symlink, hardlink, reflink. - + Note: Reflink uses copy if the underlying file system doesn't support it. --no-parallel @@ -39,7 +39,7 @@ This command has an alias [`xvc file checkout`](/ref/xvc-file-checkout.md) if yo Rechecking is analogous to [git checkout](https://git-scm.com/docs/git-checkout). It copies or links a cached file to the workspace. -Let's create an example directory hierarchy as a showcase. +Let's create an example directory hierarchy as a showcase. ```console $ xvc-test-helper create-directory-tree --directories 2 --files 3 --seed 231123 @@ -58,7 +58,7 @@ $ tree ``` -Start by tracking files. +Start by tracking files. ```console $ git init @@ -98,16 +98,17 @@ $ rm -rf dir-0002/ $ xvc -v file recheck dir-0002/ $ ls -l dir-0002/ total 24 --rw-rw-rw- [..] file-0001.bin --rw-rw-rw- [..] file-0002.bin --rw-rw-rw- [..] file-0003.bin +-rw-rw-rw- [..] file-0001.bin +-rw-rw-rw- [..] file-0002.bin +-rw-rw-rw- [..] file-0003.bin ``` + You can use glob patterns to recheck files. -```console + ```console $ xvc file track 'dir-*' - +``` You can update the recheck method of a file. Otherwise it will be kept as same before. @@ -116,18 +117,18 @@ $ rm -rf dir-0002/ $ xvc -v file recheck dir-0002/ --as symlink $ ls -l dir-0002/ total 0 -lrwxr-xr-x [..] file-0001.bin -> [CWD]/.xvc/b3/3c9/255/424e13d9c38a37c5ddd376e1070cdd5de66996fbc82194c462f653856d/0.bin -lrwxr-xr-x [..] file-0002.bin -> [CWD]/.xvc/b3/6bc/65f/581e3a03edb127b63b71c5690be176e2fe265266f70abc65f72613f62e/0.bin -lrwxr-xr-x [..] file-0003.bin -> [CWD]/.xvc/b3/804/fb8/edbb122e735facd7f943c1bbe754e939a968f385c12f56b10411a4a015/0.bin +lrwxr-xr-x [..] file-0001.bin -> [CWD]/.xvc/b3/3c9/255/424e13d9c38a37c5ddd376e1070cdd5de66996fbc82194c462f653856d/0.bin +lrwxr-xr-x [..] file-0002.bin -> [CWD]/.xvc/b3/6bc/65f/581e3a03edb127b63b71c5690be176e2fe265266f70abc65f72613f62e/0.bin +lrwxr-xr-x [..] file-0003.bin -> [CWD]/.xvc/b3/804/fb8/edbb122e735facd7f943c1bbe754e939a968f385c12f56b10411a4a015/0.bin $ rm -rf dir-0002/ -$ xvc -v file recheck dir-0002/ +$ xvc -v file recheck dir-0002/ $ ls -l dir-0002/ total 0 -lrwxr-xr-x [..] file-0001.bin -> [CWD]/.xvc/b3/3c9/255/424e13d9c38a37c5ddd376e1070cdd5de66996fbc82194c462f653856d/0.bin -lrwxr-xr-x [..] file-0002.bin -> [CWD]/.xvc/b3/6bc/65f/581e3a03edb127b63b71c5690be176e2fe265266f70abc65f72613f62e/0.bin -lrwxr-xr-x [..] file-0003.bin -> [CWD]/.xvc/b3/804/fb8/edbb122e735facd7f943c1bbe754e939a968f385c12f56b10411a4a015/0.bin +lrwxr-xr-x [..] file-0001.bin -> [CWD]/.xvc/b3/3c9/255/424e13d9c38a37c5ddd376e1070cdd5de66996fbc82194c462f653856d/0.bin +lrwxr-xr-x [..] file-0002.bin -> [CWD]/.xvc/b3/6bc/65f/581e3a03edb127b63b71c5690be176e2fe265266f70abc65f72613f62e/0.bin +lrwxr-xr-x [..] file-0003.bin -> [CWD]/.xvc/b3/804/fb8/edbb122e735facd7f943c1bbe754e939a968f385c12f56b10411a4a015/0.bin ``` @@ -144,6 +145,7 @@ $ xvc file recheck dir-0002/file-0001.bin --as copy $ zsh -c 'echo "120912" >> dir-0002/file-0001.bin' ``` + Note that, as files in the cache are kept read-only, hardlinks and symlinks are also read only. Files rechecked as copy are made read-write explicitly. ```console @@ -166,5 +168,3 @@ $ xvc file recheck data.txt --as reflink ``` The above command will create a read only link in macOS APFS and a copy in ext4 or NTFS file systems. - - From 60a210a7bafed7ecf9ce37fd8337ee3252825e75 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Mon, 22 Jul 2024 21:36:51 +0300 Subject: [PATCH 045/257] ws --- book/src/ref/xvc-file-recheck.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/book/src/ref/xvc-file-recheck.md b/book/src/ref/xvc-file-recheck.md index fbd317b7f..7859955fc 100644 --- a/book/src/ref/xvc-file-recheck.md +++ b/book/src/ref/xvc-file-recheck.md @@ -15,7 +15,7 @@ Arguments: Options: --recheck-method How to track the file contents in cache: One of copy, symlink, hardlink, reflink. - + Note: Reflink uses copy if the underlying file system doesn't support it. --no-parallel From 38fd54d86f7fd495d93ea3dd901985a14758c42d Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Wed, 24 Jul 2024 11:33:56 +0300 Subject: [PATCH 046/257] Added implementation for SqliteDependency --- pipeline/Cargo.toml | 1 + pipeline/src/error.rs | 8 ++ pipeline/src/pipeline/deps/mod.rs | 6 + pipeline/src/pipeline/deps/sqlite_query.rs | 125 +++++++++++++++++++++ 4 files changed, 140 insertions(+) create mode 100644 pipeline/src/pipeline/deps/sqlite_query.rs diff --git a/pipeline/Cargo.toml b/pipeline/Cargo.toml index bd6949ec9..6e7d37b96 100644 --- a/pipeline/Cargo.toml +++ b/pipeline/Cargo.toml @@ -70,6 +70,7 @@ fern = { version = "^0.6", features = ["colored", "meta-logging-in-format"] } ## sqlite ## TODO: Add feature flags for sqlite when bundling needed rusqlite = "^0.32" +fallible-iterator = "^0.3" ## Networking reqwest = { version = "^0.11", features = ["blocking", "json", "gzip"] } diff --git a/pipeline/src/error.rs b/pipeline/src/error.rs index da73be9e0..40b5578aa 100644 --- a/pipeline/src/error.rs +++ b/pipeline/src/error.rs @@ -8,6 +8,7 @@ use std::num::TryFromIntError; use std::path::PathBuf; use std::sync::PoisonError; use thiserror::Error as ThisError; +use rusqlite::Error as SqliteError; #[derive(ThisError, Debug)] /// Error messages for pipelines crate @@ -197,6 +198,12 @@ pub enum Error { #[from] source: TryFromIntError, }, + + #[error("Sqlite Error:")] + SqliteError { + #[from] + source: SqliteError + }, } @@ -259,5 +266,6 @@ impl Error { } } + /// The result type for xvc pipeline crate pub type Result = std::result::Result; diff --git a/pipeline/src/pipeline/deps/mod.rs b/pipeline/src/pipeline/deps/mod.rs index 85e522896..82171778f 100644 --- a/pipeline/src/pipeline/deps/mod.rs +++ b/pipeline/src/pipeline/deps/mod.rs @@ -11,6 +11,7 @@ pub mod regex; pub mod regex_items; pub mod step; pub mod url; +pub mod sqlite_query; use std::fmt::{Display, Formatter}; @@ -68,8 +69,13 @@ pub enum XvcDependency { LineItems(LineItemsDep), /// A dependenci to a set of lines defined by a range. Doesn't keep track of individual lines. Lines(LinesDep), + /// A dependency to a URL's content UrlDigest(UrlDigestDep), + + /// A dependency to an SQLite Query, that invalidates when the query results change + SqliteQueryDigest(SqliteQueryDigest), + // TODO: Slice {path, begin, length} to specify portions of binary files // TODO: DatabaseTable { database, table } to specify particular tables from databases // TODO: DatabaseQuery { database, query } to specify the result of queries diff --git a/pipeline/src/pipeline/deps/sqlite_query.rs b/pipeline/src/pipeline/deps/sqlite_query.rs new file mode 100644 index 000000000..5334da93b --- /dev/null +++ b/pipeline/src/pipeline/deps/sqlite_query.rs @@ -0,0 +1,125 @@ +use serde::{Deserialize, Serialize}; +use crate::XvcDependency; +use rusqlite::{Connection, OpenFlags}; +use fallible_iterator::FallibleIterator; + +use xvc_core::types::diff::Diffable; +use xvc_core::{ContentDigest, Diff, HashAlgorithm, XvcDigest, XvcMetadata, XvcPath, XvcRoot}; +use crate::Result; + +use xvc_ecs::persist; + + +/// When a step depends to a regex search in a text file +#[derive(Debug, PartialOrd, Ord, Clone, Eq, PartialEq, Serialize, Deserialize)] +pub struct SqliteQueryDep { + /// Path of the database file + pub path: XvcPath, + /// The query we run on the database + pub query: String, + /// The digest of the lines that match the regex + pub query_digest: Option, + /// The metadata of the database file + pub xvc_metadata: Option, +} + + +persist!(SqliteQueryDep, "sqlite-query-dependency"); + +impl From for XvcDependency { + fn from(val: SqliteQueryDep) -> Self { + XvcDependency::SqliteQueryDigest(val) + } +} + + +impl SqliteQueryDep { + /// Create a new SqliteQueryDep with empty metadata and digest + pub fn new(path: XvcPath, query: String) -> Self { + Self { + path, + query, + query_digest: None, + xvc_metadata: None, + } + } + /// Update the metadata of the file + pub fn update_metadata(self, xvc_metadata: Option) -> Self { + Self { + xvc_metadata, + ..self + } + } + + /// Update the digest of the file by reading the file and collecting all lines that match the regex + pub fn update_digest(self, xvc_root: &XvcRoot, algorithm: HashAlgorithm) -> Result { + let path = self.path.to_absolute_path(xvc_root); + let flags = OpenFlags::SQLITE_OPEN_READ_ONLY; + let sqlite = Connection::open_with_flags(path, flags)?; + let mut prepared = sqlite.prepare(&self.query)?; + // TODO: Should we allow params in the queries? + let query_res = prepared.raw_query(); + let query_lines = query_res.map(|row| + { + let mut i = 0; + // TODO: Add salting with the repo id here? + let mut els = String::new(); + while let Ok(col) = row.get_ref(i) { + els.push_str(col.as_str()?); + i += 1; + } + Ok(els) + } + ).collect::>()?.join("\n"); + + + let query_digest = Some(XvcDigest::from_content(&query_lines, algorithm).into()); + Ok(Self { + query_digest, + ..self + }) + } + +} + +impl Diffable for SqliteQueryDep { + type Item = Self; + + /// ⚠️ Update the metadata with actual.update_metadata before calling this function + fn diff_superficial(record: &Self::Item, actual: &Self::Item) -> Diff { + assert!(record.path == actual.path); + + match (record.xvc_metadata, actual.xvc_metadata) { + (Some(rec_md), Some(act_md)) => { + if rec_md == act_md { + Diff::Identical + } else { + Diff::Different { + record: record.clone(), + actual: actual.clone(), + } + } + } + (None, Some(_)) => Diff::RecordMissing { + actual: actual.clone(), + }, + (Some(_), None) => Diff::ActualMissing { + record: record.clone(), + }, + (None, None) => unreachable!("Either record or actual should have metadata"), + } + } + + /// ⚠️ Update the metadata and lines with actual.update_digest before calling this function + fn diff_thorough(record: &Self::Item, actual: &Self::Item) -> Diff { + assert!(record.path == actual.path); + if record.query_digest == actual.query_digest { + Diff::Identical + } else { + Diff::Different { + record: record.clone(), + actual: actual.clone(), + } + } + } +} From 7960a60e9a880cce9101afbaf8d46336dd0858eb Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Fri, 26 Jul 2024 11:58:33 +0300 Subject: [PATCH 047/257] added missing branches --- pipeline/src/pipeline/api/dag.rs | 3 ++ pipeline/src/pipeline/deps/compare.rs | 34 ++++++++++++++++++++++ pipeline/src/pipeline/deps/mod.rs | 8 ++++- pipeline/src/pipeline/deps/sqlite_query.rs | 3 ++ 4 files changed, 47 insertions(+), 1 deletion(-) diff --git a/pipeline/src/pipeline/api/dag.rs b/pipeline/src/pipeline/api/dag.rs index d65890a7f..64cf96800 100644 --- a/pipeline/src/pipeline/api/dag.rs +++ b/pipeline/src/pipeline/api/dag.rs @@ -155,6 +155,7 @@ fn dep_identity(dep: &XvcDependency) -> Result { id_from_string(&format!("{}::{}-{}", dep.path, dep.begin, dep.end)) } XvcDependency::UrlDigest(dep) => id_from_string(dep.url.as_ref()), + XvcDependency::SqliteQueryDigest(dep) => id_from_string(&format!("{} {}", dep.path, dep.query)), } } @@ -185,6 +186,7 @@ fn dep_node_attributes(dep: &XvcDependency) -> AttrList { XvcDependency::LineItems(_) => Shape::Component, XvcDependency::Lines(_) => Shape::Component, XvcDependency::UrlDigest(_) => Shape::Invtrapezium, + XvcDependency::SqliteQueryDigest(_) => Shape::Cylinder, }; AttrList::new() @@ -209,6 +211,7 @@ fn dep_label(dep: &XvcDependency) -> String { XvcDependency::LineItems(dep) => format!("{}::{}-{}", dep.path, dep.begin, dep.end), XvcDependency::Lines(dep) => format!("{}::{}-{}", dep.path, dep.begin, dep.end), XvcDependency::UrlDigest(dep) => dep.url.to_string(), + XvcDependency::SqliteQueryDigest(dep) => format!("{} {}", dep.path, dep.query), } } diff --git a/pipeline/src/pipeline/deps/compare.rs b/pipeline/src/pipeline/deps/compare.rs index 36fba7627..6d0ddb725 100644 --- a/pipeline/src/pipeline/deps/compare.rs +++ b/pipeline/src/pipeline/deps/compare.rs @@ -16,6 +16,7 @@ use super::line_items::LineItemsDep; use super::lines::LinesDep; use super::regex::RegexDep; use super::regex_items::RegexItemsDep; +use super::sqlite_query::SqliteQueryDep; use super::step::StepDep; use super::{ParamDep, XvcDependency}; @@ -253,6 +254,7 @@ pub fn thorough_compare_dependency( XvcDependency::Glob(dep) => diff_of_dep(thorough_compare_glob(cmp_params, &dep)?), XvcDependency::Regex(dep) => diff_of_dep(thorough_compare_regex(cmp_params, &dep)?), XvcDependency::Lines(dep) => diff_of_dep(thorough_compare_lines(cmp_params, &dep)?), + XvcDependency::SqliteQueryDigest(dep) => diff_of_dep(thorough_compare_query_digest(cmp_params, &dep)?) }; Ok(diff) @@ -396,6 +398,27 @@ fn thorough_compare_lines( } } +fn thorough_compare_query_digest( + cmp_params: &StepStateParams, + record: &SqliteQueryDep, +) -> Result> { + let actual = SqliteQueryDep::new(record.path.clone(), record.query.clone()) + .update_metadata(cmp_params.pmp.get(&record.path)); + + // Shortcircuit if the metadata is identical + match SqliteQueryDep::diff_superficial(record, &actual) { + Diff::Different { record, actual } => { + let actual = actual.update_digest(cmp_params.xvc_root, cmp_params.algorithm)?; + Ok(SqliteQueryDep::diff_thorough(&record, &actual)) + } + Diff::RecordMissing { actual } => { + let actual = actual.update_digest(cmp_params.xvc_root, cmp_params.algorithm)?; + Ok(Diff::RecordMissing { actual }) + } + diff => Ok(diff), + } +} + /// Compares dependencies with their earlier version _superficially_, meaning the cost of /// comparison is minimized by being optimistic that a dependency is unchanged. /// @@ -448,6 +471,7 @@ pub fn superficial_compare_dependency( XvcDependency::Glob(dep) => diff_of_dep(superficial_compare_glob(cmp_params, dep)?), XvcDependency::Regex(dep) => diff_of_dep(superficial_compare_regex(cmp_params, dep)?), XvcDependency::Lines(dep) => diff_of_dep(superficial_compare_lines(cmp_params, dep)?), + XvcDependency::SqliteQueryDigest(dep) => todo!(), }; Ok(diff) @@ -552,6 +576,16 @@ fn superficial_compare_lines( Ok(LinesDep::diff_superficial(record, &actual)) } + +fn superficial_compare_query_digest( + cmp_params: &StepStateParams, + record: &SqliteQueryDep, +) -> Result> { + let actual = SqliteQueryDep::new(record.path.clone(), record.query.clone()); + + Ok(SqliteQueryDep::diff_superficial(record, &actual)) +} + fn diff_of_dep(dep: Diff) -> Diff where T: Storable + Into, diff --git a/pipeline/src/pipeline/deps/mod.rs b/pipeline/src/pipeline/deps/mod.rs index 82171778f..487543ba4 100644 --- a/pipeline/src/pipeline/deps/mod.rs +++ b/pipeline/src/pipeline/deps/mod.rs @@ -36,6 +36,7 @@ pub use self::regex::RegexDep; pub use self::regex_items::RegexItemsDep; pub use self::step::StepDep; pub use self::url::UrlDigestDep; +pub use self::sqlite_query::SqliteQueryDep; /// Return default name for the params file from the config pub fn conf_params_file(conf: &XvcConfig) -> Result { @@ -74,7 +75,7 @@ pub enum XvcDependency { UrlDigest(UrlDigestDep), /// A dependency to an SQLite Query, that invalidates when the query results change - SqliteQueryDigest(SqliteQueryDigest), + SqliteQueryDigest(SqliteQueryDep), // TODO: Slice {path, begin, length} to specify portions of binary files // TODO: DatabaseTable { database, table } to specify particular tables from databases @@ -109,6 +110,7 @@ impl Display for XvcDependency { write!(f, "lines({}::{}-{})", dep.path, dep.begin, dep.end) } XvcDependency::UrlDigest(dep) => write!(f, "url-digest({})", dep.url), + XvcDependency::SqliteQueryDigest(dep) => write!(f, "sqlite({}:\"{}\")", dep.path, dep.query) } } } @@ -123,6 +125,7 @@ impl XvcDependency { XvcDependency::Param(dep) => Some(dep.path.clone()), XvcDependency::LineItems(dep) => Some(dep.path.clone()), XvcDependency::Lines(dep) => Some(dep.path.clone()), + XvcDependency::SqliteQueryDigest(dep) => Some(dep.path.clone()), XvcDependency::Step(_) => None, XvcDependency::Generic(_) => None, XvcDependency::GlobItems(_) => None, @@ -153,6 +156,7 @@ impl XvcDependency { | XvcDependency::Regex(_) | XvcDependency::Param(_) | XvcDependency::Lines(_) + | XvcDependency::SqliteQueryDigest(_) | XvcDependency::UrlDigest(_) => None, } } @@ -190,6 +194,7 @@ pub fn dependencies_to_path( XvcDependency::Param(dep) => dep.path == *to_path, XvcDependency::LineItems(dep) => dep.path == *to_path, XvcDependency::Lines(dep) => dep.path == *to_path, + XvcDependency::SqliteQueryDigest(dep) => dep.path == *to_path, XvcDependency::Generic(_) | XvcDependency::Step(_) | XvcDependency::UrlDigest(_) => { false } @@ -242,5 +247,6 @@ pub fn dependency_paths( XvcDependency::LineItems(dep) => make_map(&dep.path), XvcDependency::Regex(dep) => make_map(&dep.path), XvcDependency::Lines(dep) => make_map(&dep.path), + XvcDependency::SqliteQueryDigest(dep) => make_map(&dep.path), } } diff --git a/pipeline/src/pipeline/deps/sqlite_query.rs b/pipeline/src/pipeline/deps/sqlite_query.rs index 5334da93b..8bf220034 100644 --- a/pipeline/src/pipeline/deps/sqlite_query.rs +++ b/pipeline/src/pipeline/deps/sqlite_query.rs @@ -1,3 +1,6 @@ +//! Implements dependencies on sqlite databases. [SqliteQueryDep] is a dependency that runs a query +//! and checks whether the result of that query has changed. It doesn't run the query if the +//! metadata of the database file hasn't changed. use serde::{Deserialize, Serialize}; use crate::XvcDependency; use rusqlite::{Connection, OpenFlags}; From 90a85de03ac2b299e835f4569fe99ab91a5e83f6 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Fri, 26 Jul 2024 20:01:30 +0300 Subject: [PATCH 048/257] Fix for args --- pipeline/src/pipeline/api/step_dependency.rs | 18 +++++++++++++++++- pipeline/src/pipeline/step.rs | 7 +++++++ 2 files changed, 24 insertions(+), 1 deletion(-) diff --git a/pipeline/src/pipeline/api/step_dependency.rs b/pipeline/src/pipeline/api/step_dependency.rs index 42ea42e14..064fb4ac8 100644 --- a/pipeline/src/pipeline/api/step_dependency.rs +++ b/pipeline/src/pipeline/api/step_dependency.rs @@ -1,7 +1,7 @@ use std::cell::RefCell; use std::path::PathBuf; -use crate::deps::{LinesDep, RegexDep}; +use crate::deps::{sqlite_query, LinesDep, RegexDep}; use crate::error::{Error, Result}; use crate::pipeline::deps::file::FileDep; use crate::pipeline::deps::generic::GenericDep; @@ -47,6 +47,7 @@ pub fn cmd_step_dependency( regexes, line_items, lines, + sqlite_query, } = cmd_opts { XvcDependencyList::new(output_snd, xvc_root, pipeline_name, &step_name)? @@ -61,6 +62,7 @@ pub fn cmd_step_dependency( .lines(lines)? .line_items(line_items)? .urls(urls)? + .sqlite_query(sqlite_query)? .record() } else { Err(anyhow::anyhow!("This method is only for StepSubCommand::Dependency").into()) @@ -380,6 +382,20 @@ impl<'a> XvcDependencyList<'a> { } Ok(self) } + + pub fn sqlite_query(&mut self, sqlite_query: Option>) -> Result<&mut Self> { + + if let Some(sqlite_query) = sqlite_query { + let mut deps = self.deps.borrow_mut(); + for (path, query) in sqlite_query { + let pathbuf = PathBuf::from(path); + let path = XvcPath::new(self.xvc_root, self.current_dir, &pathbuf)?; + let sqlite_query = sqlite_query::SqliteQueryDep::new(path, query); + deps.push(XvcDependency::SqliteQueryDigest(sqlite_query)); + } + } + Ok(self) + } /// Records dependencies the store, as childs of `self.step`. pub fn record(&self) -> Result<()> { self.xvc_root diff --git a/pipeline/src/pipeline/step.rs b/pipeline/src/pipeline/step.rs index 341776117..f8c6ddc85 100644 --- a/pipeline/src/pipeline/step.rs +++ b/pipeline/src/pipeline/step.rs @@ -179,6 +179,13 @@ pub enum StepSubCommand { aliases = &["line"], )] lines: Option>, + + #[arg( + long = "sqlite-query", + aliases = &["sqlite_query", "sqlite_query_digest", "sqlite-query-digest"], + num_args = 2, + )] + sqlite_query: Option>, }, /// Add an output to a step From 831557459b18418538cdd99518bb736d0dfd30ce Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Fri, 26 Jul 2024 20:06:02 +0300 Subject: [PATCH 049/257] handler update --- pipeline/src/pipeline/api/step_dependency.rs | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/pipeline/src/pipeline/api/step_dependency.rs b/pipeline/src/pipeline/api/step_dependency.rs index 064fb4ac8..52346330b 100644 --- a/pipeline/src/pipeline/api/step_dependency.rs +++ b/pipeline/src/pipeline/api/step_dependency.rs @@ -383,16 +383,16 @@ impl<'a> XvcDependencyList<'a> { Ok(self) } - pub fn sqlite_query(&mut self, sqlite_query: Option>) -> Result<&mut Self> { + pub fn sqlite_query(&mut self, sqlite_query: Option>) -> Result<&mut Self> { if let Some(sqlite_query) = sqlite_query { let mut deps = self.deps.borrow_mut(); - for (path, query) in sqlite_query { - let pathbuf = PathBuf::from(path); - let path = XvcPath::new(self.xvc_root, self.current_dir, &pathbuf)?; - let sqlite_query = sqlite_query::SqliteQueryDep::new(path, query); - deps.push(XvcDependency::SqliteQueryDigest(sqlite_query)); - } + let path = sqlite_query[0].clone(); + let query = sqlite_query[1].clone(); + let pathbuf = PathBuf::from(path); + let xvc_path = XvcPath::new(self.xvc_root, self.current_dir, &pathbuf)?; + let sqlite_query = sqlite_query::SqliteQueryDep::new(xvc_path, query); + deps.push(XvcDependency::SqliteQueryDigest(sqlite_query)); } Ok(self) } From 6bd56867a46715e6a0c0f2e987bde037d589c3fc Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Fri, 26 Jul 2024 20:41:58 +0300 Subject: [PATCH 050/257] unnecessary let --- lib/src/cli/mod.rs | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/lib/src/cli/mod.rs b/lib/src/cli/mod.rs index b019935e2..c7bcf9fca 100644 --- a/lib/src/cli/mod.rs +++ b/lib/src/cli/mod.rs @@ -17,7 +17,6 @@ use log::LevelFilter; use std::io; use xvc_core::types::xvcroot::load_xvc_root; use xvc_core::types::xvcroot::XvcRootInner; -use xvc_core::XvcRoot; use xvc_logging::{debug, error, uwr, XvcOutputLine}; use xvc_config::{XvcConfigParams, XvcVerbosity}; @@ -457,7 +456,7 @@ pub fn dispatch(cli_opts: cli::XvcCLI) -> Result { } fn get_xvc_config_params(cli_opts: &XvcCLI) -> XvcConfigParams { - let xvc_config_params = XvcConfigParams { + XvcConfigParams { current_dir: AbsolutePath::from(&cli_opts.workdir), include_system_config: !cli_opts.no_system_config, include_user_config: !cli_opts.no_user_config, @@ -466,8 +465,7 @@ fn get_xvc_config_params(cli_opts: &XvcCLI) -> XvcConfigParams { include_environment_config: !cli_opts.no_env_config, command_line_config: Some(cli_opts.consolidate_config_options()), default_configuration: default_project_config(true), - }; - xvc_config_params + } } fn get_term_log_level(verbosity: XvcVerbosity) -> LevelFilter { From dea58d373c90a9dc90f3f027e951b95bdf36b946 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Fri, 26 Jul 2024 21:24:33 +0300 Subject: [PATCH 051/257] Fix minor errors --- core/src/check_ignore/mod.rs | 2 +- file/src/carry_in/mod.rs | 2 +- file/src/lib.rs | 1 - file/src/share/mod.rs | 1 + pipeline/src/pipeline/deps/sqlite_query.rs | 3 +-- pipeline/src/pipeline/step.rs | 1 + 6 files changed, 5 insertions(+), 5 deletions(-) diff --git a/core/src/check_ignore/mod.rs b/core/src/check_ignore/mod.rs index 686af5eec..d4cd002b1 100644 --- a/core/src/check_ignore/mod.rs +++ b/core/src/check_ignore/mod.rs @@ -80,7 +80,7 @@ pub fn cmd_check_ignore( ignore_filename: Some(opts.ignore_filename.clone()), include_dirs: true, }; - let initial_rules = IgnoreRules::try_from_patterns(&xvc_root, COMMON_IGNORE_PATTERNS)?; + let initial_rules = IgnoreRules::try_from_patterns(xvc_root, COMMON_IGNORE_PATTERNS)?; let ignore_rules = build_ignore_rules( initial_rules, current_dir, diff --git a/file/src/carry_in/mod.rs b/file/src/carry_in/mod.rs index 780a1b5b6..46ce65e06 100644 --- a/file/src/carry_in/mod.rs +++ b/file/src/carry_in/mod.rs @@ -167,7 +167,7 @@ pub fn cmd_carry_in( let cache_paths_to_carry: HStore = xvc_paths_to_carry .iter() - .filter_map(|(xe, xp)| match content_digest_diff[&xe] { + .filter_map(|(xe, xp)| match content_digest_diff[xe] { Diff::Identical | Diff::Skipped => { // use stored digest for cache path info!(output_snd, "[FORCE] {xp} is identical to cached copy."); diff --git a/file/src/lib.rs b/file/src/lib.rs index e95803b98..bc3a344da 100755 --- a/file/src/lib.rs +++ b/file/src/lib.rs @@ -45,7 +45,6 @@ use crossbeam_channel::bounded; use log::{debug, error, info, warn, LevelFilter}; use std::io; use std::io::Write; -use std::path::Path; use std::path::PathBuf; use xvc_config::XvcConfigParams; use xvc_config::XvcVerbosity; diff --git a/file/src/share/mod.rs b/file/src/share/mod.rs index 6fe3d30e0..64347e629 100644 --- a/file/src/share/mod.rs +++ b/file/src/share/mod.rs @@ -1,3 +1,4 @@ +//! Share files from S3 compatible storages for a limited time use std::path::PathBuf; use crate::{common::load_targets_from_store, error, Result}; diff --git a/pipeline/src/pipeline/deps/sqlite_query.rs b/pipeline/src/pipeline/deps/sqlite_query.rs index 8bf220034..56402d31f 100644 --- a/pipeline/src/pipeline/deps/sqlite_query.rs +++ b/pipeline/src/pipeline/deps/sqlite_query.rs @@ -55,8 +55,7 @@ impl SqliteQueryDep { } /// Update the digest of the file by reading the file and collecting all lines that match the regex - pub fn update_digest(self, xvc_root: &XvcRoot, algorithm: HashAlgorithm) -> Result { - let path = self.path.to_absolute_path(xvc_root); + pub fn update_digest(self, xvc_root: &XvcRoot, algorithm: HashAlgorithm) -> Result { let path = self.path.to_absolute_path(xvc_root); let flags = OpenFlags::SQLITE_OPEN_READ_ONLY; let sqlite = Connection::open_with_flags(path, flags)?; let mut prepared = sqlite.prepare(&self.query)?; diff --git a/pipeline/src/pipeline/step.rs b/pipeline/src/pipeline/step.rs index f8c6ddc85..ab091bbbe 100644 --- a/pipeline/src/pipeline/step.rs +++ b/pipeline/src/pipeline/step.rs @@ -80,6 +80,7 @@ pub enum StepSubCommand { when: Option, }, + /// Add a dependency to a step #[command()] Dependency { From 6e38ce7748643754426c8fd7a5282c02fc7d009c Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Fri, 26 Jul 2024 21:30:18 +0300 Subject: [PATCH 052/257] xvc file track fixes --- book/src/ref/xvc-file-track.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/book/src/ref/xvc-file-track.md b/book/src/ref/xvc-file-track.md index cb6c4b219..a4ca9d87b 100644 --- a/book/src/ref/xvc-file-track.md +++ b/book/src/ref/xvc-file-track.md @@ -131,7 +131,7 @@ If you want to make this connection with symbolic links, you can specify it with ```console $ xvc file track --recheck-method symlink dir-0003/file-0001.bin $ ls -l dir-0003/file-0001.bin -lrwxr-xr-x@ [..] dir-0003/file-0001.bin -> [CWD]/.xvc/b3/e51/7d6/b9a3617fdcd96bd128142a39f1eca26ed77a338d2b93ba4921a0116c70/0.bin +lrwxr-xr-x[..] dir-0003/file-0001.bin -> [CWD]/.xvc/b3/e51/7d6/b9a3617fdcd96bd128142a39f1eca26ed77a338d2b93ba4921a0116c70/0.bin ``` @@ -142,9 +142,9 @@ $ xvc file track --recheck-method hardlink dir-0003/file-0002.bin $ xvc file track --recheck-method reflink dir-0003/file-0003.bin $ ls -l dir-0003/ total 16 -lrwxr-xr-x@ [..] file-0001.bin -> [CWD]/.xvc/b3/e51/7d6/b9a3617fdcd96bd128142a39f1eca26ed77a338d2b93ba4921a0116c70/0.bin --r--r--r--@ [..] file-0002.bin --r--r--r--@ [..] file-0003.bin +l[..] file-0001.bin -> [CWD]/.xvc/b3/e51/7d6/b9a3617fdcd96bd128142a39f1eca26ed77a338d2b93ba4921a0116c70/0.bin +-[..] file-0002.bin +-[..] file-0003.bin ``` @@ -167,9 +167,9 @@ cache. $ xvc file track --no-commit --recheck-method symlink dir-0004/ $ ls -l dir-0004/ total 24 --rw-r--r--@ [..] file-0001.bin --rw-r--r--@ [..] file-0002.bin --rw-r--r--@ [..] file-0003.bin +-rw-r--r--[..] file-0001.bin +-rw-r--r--[..] file-0002.bin +-rw-r--r--[..] file-0003.bin $ xvc file list dir-0004/ FS [..] ab361981 ab361981 dir-0004/file-0003.bin @@ -189,9 +189,9 @@ $ xvc file carry-in --force dir-0004/ $ ls -l dir-0004/ total 0 -lrwxr-xr-x@ [..] file-0001.bin -> [CWD]/.xvc/b3/e51/7d6/b9a3617fdcd96bd128142a39f1eca26ed77a338d2b93ba4921a0116c70/0.bin -lrwxr-xr-x@ [..] file-0002.bin -> [CWD]/.xvc/b3/493/eeb/6525ea5e94e1e760371108e4a525c696c773a774a4818e941fd6d1af79/0.bin -lrwxr-xr-x@ [..] file-0003.bin -> [CWD]/.xvc/b3/ab3/619/814cae0456a5a291e4d5c8d339a8389630e476f9f9e8d3a09accc919f0/0.bin +lrwxr-xr-x[..] file-0001.bin -> [CWD]/.xvc/b3/e51/7d6/b9a3617fdcd96bd128142a39f1eca26ed77a338d2b93ba4921a0116c70/0.bin +lrwxr-xr-x[..] file-0002.bin -> [CWD]/.xvc/b3/493/eeb/6525ea5e94e1e760371108e4a525c696c773a774a4818e941fd6d1af79/0.bin +lrwxr-xr-x[..] file-0003.bin -> [CWD]/.xvc/b3/ab3/619/814cae0456a5a291e4d5c8d339a8389630e476f9f9e8d3a09accc919f0/0.bin ``` From 76a85a4adec09de80c925720f61c2e1bb18038d3 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Fri, 26 Jul 2024 21:34:57 +0300 Subject: [PATCH 053/257] ws updates --- book/src/ref/xvc-file-track.md | 2 +- book/src/ref/xvc-pipeline-step-dependency-line-items.md | 9 ++++----- book/src/ref/xvc-pipeline-step-dependency-regex-items.md | 4 ++-- 3 files changed, 7 insertions(+), 8 deletions(-) diff --git a/book/src/ref/xvc-file-track.md b/book/src/ref/xvc-file-track.md index a4ca9d87b..51df7d9d3 100644 --- a/book/src/ref/xvc-file-track.md +++ b/book/src/ref/xvc-file-track.md @@ -19,7 +19,7 @@ Arguments: Options: --recheck-method How to track the file contents in cache: One of copy, symlink, hardlink, reflink. - + Note: Reflink uses copy if the underlying file system doesn't support it. --no-commit diff --git a/book/src/ref/xvc-pipeline-step-dependency-line-items.md b/book/src/ref/xvc-pipeline-step-dependency-line-items.md index 587361088..2629ba2b2 100644 --- a/book/src/ref/xvc-pipeline-step-dependency-line-items.md +++ b/book/src/ref/xvc-pipeline-step-dependency-line-items.md @@ -76,17 +76,17 @@ $ xvc pipeline run "Ivan", "M", 53, 72, 175 Removed Lines: - + [DONE] print-top-10 (echo "Added Lines:/n ${XVC_ADDED_LINE_ITEMS}/nRemoved Lines:/n${XVC_REMOVED_LINE_ITEMS}") -`````` +``` When you run the pipeline again, the step is not run because the specified lines didn't change. ```console $ xvc pipeline run -`````` +``` When you change a line from the file, the step is invalidated. @@ -103,8 +103,7 @@ $ xvc pipeline run "Ferzan", "M", 30, 71, 158 Removed Lines: "Hank", "M", 30, 71, 158 - + [DONE] print-top-10 (echo "Added Lines:/n ${XVC_ADDED_LINE_ITEMS}/nRemoved Lines:/n${XVC_REMOVED_LINE_ITEMS}") ``` - diff --git a/book/src/ref/xvc-pipeline-step-dependency-regex-items.md b/book/src/ref/xvc-pipeline-step-dependency-regex-items.md index 402fd3b2a..2b0dd1a6c 100644 --- a/book/src/ref/xvc-pipeline-step-dependency-regex-items.md +++ b/book/src/ref/xvc-pipeline-step-dependency-regex-items.md @@ -91,14 +91,14 @@ $ xvc pipeline run [DONE] new-females (echo "New Females:/n ${XVC_ADDED_REGEX_ITEMS}") -`````` +``` When you run the pipeline again, the steps are not run because the regexes didn't change. ```console $ xvc pipeline run -`````` +``` When you add a new female record to the file, only the female count step is run. From bd4688d6750870dd3c004ca7bba889e5476a08d7 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Fri, 26 Jul 2024 21:38:37 +0300 Subject: [PATCH 054/257] update command --- .../ref/xvc-pipeline-step-dependency-line-items.md | 14 ++++---------- 1 file changed, 4 insertions(+), 10 deletions(-) diff --git a/book/src/ref/xvc-pipeline-step-dependency-line-items.md b/book/src/ref/xvc-pipeline-step-dependency-line-items.md index 2629ba2b2..13ecb7f15 100644 --- a/book/src/ref/xvc-pipeline-step-dependency-line-items.md +++ b/book/src/ref/xvc-pipeline-step-dependency-line-items.md @@ -49,7 +49,7 @@ $ cat people.csv Let's a step to show the first 10 lines of the file: ```console -$ xvc pipeline step new --step-name print-top-10 --command 'echo "Added Lines:\n ${XVC_ADDED_LINE_ITEMS}\nRemoved Lines:\n${XVC_REMOVED_LINE_ITEMS}"' +$ xvc pipeline step new --step-name print-top-10 --command 'env' ``` @@ -65,20 +65,13 @@ When you run the pipeline initially, the step is run. ```console $ xvc pipeline run [OUT] [print-top-10] Added Lines: - "Alex", "M", 41, 74, 170 -"Bert", "M", 42, 68, 166 -"Carl", "M", 32, 70, 155 -"Dave", "M", 39, 72, 167 -"Elly", "F", 30, 66, 124 -"Fran", "F", 33, 66, 115 -"Gwen", "F", 26, 64, 121 -"Hank", "M", 30, 71, 158 -"Ivan", "M", 53, 72, 175 + Removed Lines: [DONE] print-top-10 (echo "Added Lines:/n ${XVC_ADDED_LINE_ITEMS}/nRemoved Lines:/n${XVC_REMOVED_LINE_ITEMS}") + ``` When you run the pipeline again, the step is not run because the specified lines didn't change. @@ -106,4 +99,5 @@ Removed Lines: [DONE] print-top-10 (echo "Added Lines:/n ${XVC_ADDED_LINE_ITEMS}/nRemoved Lines:/n${XVC_REMOVED_LINE_ITEMS}") + ``` From 414351fc350d16f61e5413cc83c52efe904fc54b Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Fri, 26 Jul 2024 21:40:31 +0300 Subject: [PATCH 055/257] fix line items --- pipeline/src/pipeline/mod.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pipeline/src/pipeline/mod.rs b/pipeline/src/pipeline/mod.rs index 77c9b5a09..acbd4254a 100644 --- a/pipeline/src/pipeline/mod.rs +++ b/pipeline/src/pipeline/mod.rs @@ -1267,7 +1267,7 @@ fn update_command_environment( update_env("XVC_ALL_REGEX_ITEMS", &items) } XvcDependency::LineItems(_) => { - update_env("XVC_ADDED_ITEMS", &items)?; + update_env("XVC_ADDED_LINE_ITEMS", &items)?; update_env("XVC_REMOVED_LINE_ITEMS", &[])?; update_env("XVC_ALL_LINE_ITEMS", &items) } From 8b4b11f2ab2f16027ea1e6c92abf7442a8eeb873 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Fri, 26 Jul 2024 21:41:02 +0300 Subject: [PATCH 056/257] revert command --- book/src/ref/xvc-pipeline-step-dependency-line-items.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/book/src/ref/xvc-pipeline-step-dependency-line-items.md b/book/src/ref/xvc-pipeline-step-dependency-line-items.md index 13ecb7f15..d850207e6 100644 --- a/book/src/ref/xvc-pipeline-step-dependency-line-items.md +++ b/book/src/ref/xvc-pipeline-step-dependency-line-items.md @@ -49,7 +49,7 @@ $ cat people.csv Let's a step to show the first 10 lines of the file: ```console -$ xvc pipeline step new --step-name print-top-10 --command 'env' +$ xvc pipeline step new --step-name print-top-10 --command 'env' echo "Added Lines:\n ${XVC_ADDED_LINE_ITEMS}\nRemoved Lines:\n${XVC_REMOVED_LINE_ITEMS}"' ``` From 9d59c6b6eea8acd8848acc62c2a26801ad195488 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Fri, 26 Jul 2024 21:45:06 +0300 Subject: [PATCH 057/257] fix command --- ...xvc-pipeline-step-dependency-line-items.md | 25 ++++++++----------- 1 file changed, 10 insertions(+), 15 deletions(-) diff --git a/book/src/ref/xvc-pipeline-step-dependency-line-items.md b/book/src/ref/xvc-pipeline-step-dependency-line-items.md index d850207e6..fe0aa017d 100644 --- a/book/src/ref/xvc-pipeline-step-dependency-line-items.md +++ b/book/src/ref/xvc-pipeline-step-dependency-line-items.md @@ -49,7 +49,13 @@ $ cat people.csv Let's a step to show the first 10 lines of the file: ```console -$ xvc pipeline step new --step-name print-top-10 --command 'env' echo "Added Lines:\n ${XVC_ADDED_LINE_ITEMS}\nRemoved Lines:\n${XVC_REMOVED_LINE_ITEMS}"' +$ xvc pipeline step new --step-name print-top-10 --command 'echo "Added Lines:\n ${XVC_ADDED_LINE_ITEMS}\nRemoved Lines:\n${XVC_REMOVED_LINE_ITEMS}"' +? 2 +error: unexpected argument 'echo' found + +Usage: xvc pipeline step new [OPTIONS] --step-name --command + +For more information, try '--help'. ``` @@ -57,6 +63,9 @@ The command is run only when those lines change. ```console $ xvc pipeline step dependency --step-name print-top-10 --line-items 'people.csv::1-10' +? 1 +[ERROR] Pipeline Error: Step print-top-10 not found in pipeline +Error: PipelineError { source: StepNotFoundInPipeline { step: "print-top-10" } } ``` @@ -64,13 +73,6 @@ When you run the pipeline initially, the step is run. ```console $ xvc pipeline run -[OUT] [print-top-10] Added Lines: - -Removed Lines: - - -[DONE] print-top-10 (echo "Added Lines:/n ${XVC_ADDED_LINE_ITEMS}/nRemoved Lines:/n${XVC_REMOVED_LINE_ITEMS}") - ``` @@ -92,12 +94,5 @@ Now, when you run the pipeline, it will print the changed line, with its new and ``` $ xvc pipeline run -[OUT] [print-top-10] Added Lines: - "Ferzan", "M", 30, 71, 158 -Removed Lines: -"Hank", "M", 30, 71, 158 - -[DONE] print-top-10 (echo "Added Lines:/n ${XVC_ADDED_LINE_ITEMS}/nRemoved Lines:/n${XVC_REMOVED_LINE_ITEMS}") - ``` From 074804477fe60428568efba3926c1529794534c0 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Fri, 26 Jul 2024 21:50:58 +0300 Subject: [PATCH 058/257] Fixed line errors --- ...xvc-pipeline-step-dependency-line-items.md | 31 +++++++++++++------ 1 file changed, 22 insertions(+), 9 deletions(-) diff --git a/book/src/ref/xvc-pipeline-step-dependency-line-items.md b/book/src/ref/xvc-pipeline-step-dependency-line-items.md index fe0aa017d..daab4b15e 100644 --- a/book/src/ref/xvc-pipeline-step-dependency-line-items.md +++ b/book/src/ref/xvc-pipeline-step-dependency-line-items.md @@ -50,12 +50,6 @@ Let's a step to show the first 10 lines of the file: ```console $ xvc pipeline step new --step-name print-top-10 --command 'echo "Added Lines:\n ${XVC_ADDED_LINE_ITEMS}\nRemoved Lines:\n${XVC_REMOVED_LINE_ITEMS}"' -? 2 -error: unexpected argument 'echo' found - -Usage: xvc pipeline step new [OPTIONS] --step-name --command - -For more information, try '--help'. ``` @@ -63,9 +57,6 @@ The command is run only when those lines change. ```console $ xvc pipeline step dependency --step-name print-top-10 --line-items 'people.csv::1-10' -? 1 -[ERROR] Pipeline Error: Step print-top-10 not found in pipeline -Error: PipelineError { source: StepNotFoundInPipeline { step: "print-top-10" } } ``` @@ -73,6 +64,21 @@ When you run the pipeline initially, the step is run. ```console $ xvc pipeline run +[OUT] [print-top-10] Added Lines: + "Alex", "M", 41, 74, 170 +"Bert", "M", 42, 68, 166 +"Carl", "M", 32, 70, 155 +"Dave", "M", 39, 72, 167 +"Elly", "F", 30, 66, 124 +"Fran", "F", 33, 66, 115 +"Gwen", "F", 26, 64, 121 +"Hank", "M", 30, 71, 158 +"Ivan", "M", 53, 72, 175 +Removed Lines: + + +[DONE] print-top-10 (echo "Added Lines:/n ${XVC_ADDED_LINE_ITEMS}/nRemoved Lines:/n${XVC_REMOVED_LINE_ITEMS}") + ``` @@ -94,5 +100,12 @@ Now, when you run the pipeline, it will print the changed line, with its new and ``` $ xvc pipeline run +[OUT] [print-top-10] Added Lines: + "Ferzan", "M", 30, 71, 158 +Removed Lines: +"Hank", "M", 30, 71, 158 + +[DONE] print-top-10 (echo "Added Lines:/n ${XVC_ADDED_LINE_ITEMS}/nRemoved Lines:/n${XVC_REMOVED_LINE_ITEMS}") + ``` From 1752495531bece58e3a4ed08cd0048a0330ad96e Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Fri, 26 Jul 2024 21:57:48 +0300 Subject: [PATCH 059/257] added value names --- pipeline/src/pipeline/step.rs | 1 + 1 file changed, 1 insertion(+) diff --git a/pipeline/src/pipeline/step.rs b/pipeline/src/pipeline/step.rs index ab091bbbe..f1e544de5 100644 --- a/pipeline/src/pipeline/step.rs +++ b/pipeline/src/pipeline/step.rs @@ -185,6 +185,7 @@ pub enum StepSubCommand { long = "sqlite-query", aliases = &["sqlite_query", "sqlite_query_digest", "sqlite-query-digest"], num_args = 2, + value_names = &["SQLITE_FILE", "SQLITE_QUERY"], )] sqlite_query: Option>, }, From 4de345b99c358d355b1c5b0cd2e5c6b8bfe9da53 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Fri, 26 Jul 2024 22:05:38 +0300 Subject: [PATCH 060/257] Added help text for sqlite --- pipeline/src/pipeline/step.rs | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/pipeline/src/pipeline/step.rs b/pipeline/src/pipeline/step.rs index f1e544de5..629525bcb 100644 --- a/pipeline/src/pipeline/step.rs +++ b/pipeline/src/pipeline/step.rs @@ -181,6 +181,11 @@ pub enum StepSubCommand { )] lines: Option>, + /// Add a sqlite query dependency to the step with the file and the query. Can be used + /// once. + /// + /// The step is invalidated when the query run and the result is different from previous + /// runs, e.g. when an aggregate changed or a new row added to a table. #[arg( long = "sqlite-query", aliases = &["sqlite_query", "sqlite_query_digest", "sqlite-query-digest"], From beeaac846f998337fb0a438ae1af3dbb8335e26b Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Fri, 26 Jul 2024 22:07:10 +0300 Subject: [PATCH 061/257] update help text --- book/src/ref/xvc-pipeline-step-dependency.md | 21 ++++++++++++-------- 1 file changed, 13 insertions(+), 8 deletions(-) diff --git a/book/src/ref/xvc-pipeline-step-dependency.md b/book/src/ref/xvc-pipeline-step-dependency.md index 6d07a8300..96ae50217 100644 --- a/book/src/ref/xvc-pipeline-step-dependency.md +++ b/book/src/ref/xvc-pipeline-step-dependency.md @@ -30,16 +30,16 @@ Options: --glob_items Add a glob items dependency to the step. - + You can depend on multiple files and directories with this dependency. - + The difference between this and the glob option is that this option keeps track of all matching files, but glob only keeps track of the matched files' digest. When you want to use ${XVC_GLOB_ITEMS}, ${XVC_ADDED_GLOB_ITEMS}, or ${XVC_REMOVED_GLOB_ITEMS} environment variables in the step command, use the glob-items dependency. Otherwise, you can use the glob option to save disk space. --glob Add a glob dependency to the step. Can be used multiple times. - + You can depend on multiple files and directories with this dependency. - + The difference between this and the glob-items option is that the glob-items option keeps track of all matching files individually, but this option only keeps track of the matched files' digest. This dependency uses considerably less disk space. --param @@ -47,24 +47,29 @@ Options: --regex_items Add a regex dependency in the form filename.txt:/^regex/ . Can be used multiple times. - + The difference between this and the regex option is that the regex-items option keeps track of all matching lines, but regex only keeps track of the matched lines' digest. When you want to use ${XVC_REGEX_ITEMS}, ${XVC_ADDED_REGEX_ITEMS}, ${XVC_REMOVED_REGEX_ITEMS} environment variables in the step command, use the regex option. Otherwise, you can use the regex-digest option to save disk space. --regex Add a regex dependency in the form filename.txt:/^regex/ . Can be used multiple times. - + The difference between this and the regex option is that the regex option keeps track of all matching lines that can be used in the step command. This option only keeps track of the matched lines' digest. --line_items Add a line dependency in the form filename.txt::123-234 - + The difference between this and the lines option is that the line-items option keeps track of all matching lines that can be used in the step command. This option only keeps track of the matched lines' digest. When you want to use ${XVC_ALL_LINE_ITEMS}, ${XVC_ADDED_LINE_ITEMS}, ${XVC_CHANGED_LINE_ITEMS} options in the step command, use the line option. Otherwise, you can use the lines option to save disk space. --lines Add a line digest dependency in the form filename.txt::123-234 - + The difference between this and the line-items dependency is that the line option keeps track of all matching lines that can be used in the step command. This option only keeps track of the matched lines' digest. If you don't need individual lines to be kept, use this option to save space. + --sqlite-query + Add a sqlite query dependency to the step with the file and the query. Can be used once. + + The step is invalidated when the query run and the result is different from previous runs, e.g. when an aggregate changed or a new row added to a table. + -h, --help Print help (see a summary with '-h') From a467f807ab1231386646e9527c83c735fe885db2 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Fri, 26 Jul 2024 22:08:53 +0300 Subject: [PATCH 062/257] fix sql commands --- .../ref/xvc-pipeline-step-dependency-sqlite.md | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/book/src/ref/xvc-pipeline-step-dependency-sqlite.md b/book/src/ref/xvc-pipeline-step-dependency-sqlite.md index 0b4fd057f..11f9f4314 100644 --- a/book/src/ref/xvc-pipeline-step-dependency-sqlite.md +++ b/book/src/ref/xvc-pipeline-step-dependency-sqlite.md @@ -55,19 +55,30 @@ EOF Now, we'll add a step to the pipeline to calculate the average age of these people. ```console -$ xvc pipeline step new --step-name average-age --command 'sqlite3 people.db "SELECT AVG(Age) FROM People"' +$ xvc pipeline step new --step-name average-age --command 'sqlite3 people.db "SELECT AVG(Age) FROM People;"' ``` Let's run the step without a dependency first. ```console $ xvc pipeline run +[ERROR] Step average-age finished UNSUCCESSFULLY with command sqlite3 people.db "SELECT AVG(Age) FROM People" + ``` Now, we'll add a dependency to this step and it will only run the step when the results of that query changes. ```console -$ xvc pipeline step dependency --step-name average-age --sqlite 'people.db "SELECT count(*) FROM People"' +$ xvc pipeline step dependency --step-name average-age --sqlite-query people.db "SELECT count(*) FROM People;"' +? 2 +error: unexpected argument '--sqlite' found + + tip: a similar argument exists: '--sqlite_query' + +Usage: xvc pipeline step dependency <--step-name |--generic |--url |--file |--step |--glob_items |--glob |--param |--regex_items |--regex |--line_items |--lines |--sqlite-query > + +For more information, try '--help'. + ``` ```note @@ -78,6 +89,7 @@ So, when the number of people in the table changes, the step will run. Initially ```console $ xvc pipeline run +[ERROR] Step average-age finished UNSUCCESSFULLY with command sqlite3 people.db "SELECT AVG(Age) FROM People" ``` @@ -85,6 +97,7 @@ But it won't run the step a second time, as the table didn't change. ```console $ xvc pipeline run +[ERROR] Step average-age finished UNSUCCESSFULLY with command sqlite3 people.db "SELECT AVG(Age) FROM People" ``` @@ -98,6 +111,7 @@ This time, the step will run again as the result from dependency query (`SELECT ```console $ xvc pipeline run +[ERROR] Step average-age finished UNSUCCESSFULLY with command sqlite3 people.db "SELECT AVG(Age) FROM People" ``` From 502e8ae8f707f96981a27217392e2c7cb16b7875 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Fri, 26 Jul 2024 22:16:36 +0300 Subject: [PATCH 063/257] update --- .../ref/xvc-pipeline-step-dependency-sqlite.md | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/book/src/ref/xvc-pipeline-step-dependency-sqlite.md b/book/src/ref/xvc-pipeline-step-dependency-sqlite.md index 11f9f4314..6eb248c2f 100644 --- a/book/src/ref/xvc-pipeline-step-dependency-sqlite.md +++ b/book/src/ref/xvc-pipeline-step-dependency-sqlite.md @@ -55,27 +55,25 @@ EOF Now, we'll add a step to the pipeline to calculate the average age of these people. ```console -$ xvc pipeline step new --step-name average-age --command 'sqlite3 people.db "SELECT AVG(Age) FROM People;"' +$ xvc -vvv pipeline step new --step-name average-age --command 'sqlite3 people.db "SELECT AVG(Age) FROM People;"' ``` Let's run the step without a dependency first. ```console $ xvc pipeline run -[ERROR] Step average-age finished UNSUCCESSFULLY with command sqlite3 people.db "SELECT AVG(Age) FROM People" +[ERROR] Step average-age finished UNSUCCESSFULLY with command sqlite3 people.db "SELECT AVG(Age) FROM People;" ``` Now, we'll add a dependency to this step and it will only run the step when the results of that query changes. ```console -$ xvc pipeline step dependency --step-name average-age --sqlite-query people.db "SELECT count(*) FROM People;"' +$ xvc pipeline step dependency --step-name average-age --sqlite-query people.db 'SELECT count(*) FROM People;' ? 2 -error: unexpected argument '--sqlite' found +error: 2 values required for '--sqlite-query ' but 1 was provided - tip: a similar argument exists: '--sqlite_query' - -Usage: xvc pipeline step dependency <--step-name |--generic |--url |--file |--step |--glob_items |--glob |--param |--regex_items |--regex |--line_items |--lines |--sqlite-query > +Usage: xvc pipeline step dependency [OPTIONS] --step-name For more information, try '--help'. @@ -89,7 +87,7 @@ So, when the number of people in the table changes, the step will run. Initially ```console $ xvc pipeline run -[ERROR] Step average-age finished UNSUCCESSFULLY with command sqlite3 people.db "SELECT AVG(Age) FROM People" +[ERROR] Step average-age finished UNSUCCESSFULLY with command sqlite3 people.db "SELECT AVG(Age) FROM People;" ``` @@ -97,7 +95,7 @@ But it won't run the step a second time, as the table didn't change. ```console $ xvc pipeline run -[ERROR] Step average-age finished UNSUCCESSFULLY with command sqlite3 people.db "SELECT AVG(Age) FROM People" +[ERROR] Step average-age finished UNSUCCESSFULLY with command sqlite3 people.db "SELECT AVG(Age) FROM People;" ``` @@ -111,7 +109,7 @@ This time, the step will run again as the result from dependency query (`SELECT ```console $ xvc pipeline run -[ERROR] Step average-age finished UNSUCCESSFULLY with command sqlite3 people.db "SELECT AVG(Age) FROM People" +[ERROR] Step average-age finished UNSUCCESSFULLY with command sqlite3 people.db "SELECT AVG(Age) FROM People;" ``` From 1fe870ffa2b4839fe9bf1bf11e90936709e8a873 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Fri, 26 Jul 2024 22:18:25 +0300 Subject: [PATCH 064/257] debug --- .../xvc-pipeline-step-dependency-sqlite.md | 40 ++++++++++++++----- 1 file changed, 29 insertions(+), 11 deletions(-) diff --git a/book/src/ref/xvc-pipeline-step-dependency-sqlite.md b/book/src/ref/xvc-pipeline-step-dependency-sqlite.md index 6eb248c2f..b53f6b942 100644 --- a/book/src/ref/xvc-pipeline-step-dependency-sqlite.md +++ b/book/src/ref/xvc-pipeline-step-dependency-sqlite.md @@ -55,13 +55,28 @@ EOF Now, we'll add a step to the pipeline to calculate the average age of these people. ```console -$ xvc -vvv pipeline step new --step-name average-age --command 'sqlite3 people.db "SELECT AVG(Age) FROM People;"' +$ xvc pipeline step new --step-name average-age --command 'sqlite3 people.db "SELECT AVG(Age) FROM People;"' +[DEBUG][logging/src/lib.rs::237] Terminal logger enabled with level: Debug +[DEBUG][config/src/error.rs::72] Config source for level "system" not found at "/Users/iex/Library/Application Support/com.emresult.xvc" +[DEBUG][config/src/error.rs::72] Config source for level "global" not found at "/Users/iex/Library/Application Support/xvc" +[DEBUG] Using Git: /opt/homebrew/bin/git +[DEBUG] Committing .xvc/ to git: [main d67183e] Xvc auto-commit after '/Users/iex/github.com/iesahin/xvc/target/debug/xvc -vvv pipeline step new --step-name average-age --command sqlite3 people.db "SELECT AVG(Age) FROM People;"' + 6 files changed, 6 insertions(+) + create mode 100644 .xvc/ec/1722021403030243 + create mode 100644 .xvc/store/xvc-step-command-store/1722021403029764.json + create mode 100644 .xvc/store/xvc-step-invalidate-store/1722021403029598.json + create mode 100644 .xvc/store/xvc-step-store/1722021403029432.json + create mode 100644 .xvc/store/xvc-step-store/1722021403030038.json + create mode 100644 .xvc/store/xvc-step-xvc-pipeline-r1n-store/1722021403030134.json + +[DEBUG] Command completed successfully. + ``` Let's run the step without a dependency first. ```console -$ xvc pipeline run +$ xvc -vvv pipeline run [ERROR] Step average-age finished UNSUCCESSFULLY with command sqlite3 people.db "SELECT AVG(Age) FROM People;" ``` @@ -70,12 +85,6 @@ Now, we'll add a dependency to this step and it will only run the step when the ```console $ xvc pipeline step dependency --step-name average-age --sqlite-query people.db 'SELECT count(*) FROM People;' -? 2 -error: 2 values required for '--sqlite-query ' but 1 was provided - -Usage: xvc pipeline step dependency [OPTIONS] --step-name - -For more information, try '--help'. ``` @@ -87,7 +96,10 @@ So, when the number of people in the table changes, the step will run. Initially ```console $ xvc pipeline run -[ERROR] Step average-age finished UNSUCCESSFULLY with command sqlite3 people.db "SELECT AVG(Age) FROM People;" +thread '' panicked at pipeline/src/pipeline/deps/compare.rs:474:50: +not yet implemented +note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace +[ERROR] Error in step thread: Any { .. } ``` @@ -95,7 +107,10 @@ But it won't run the step a second time, as the table didn't change. ```console $ xvc pipeline run -[ERROR] Step average-age finished UNSUCCESSFULLY with command sqlite3 people.db "SELECT AVG(Age) FROM People;" +thread '' panicked at pipeline/src/pipeline/deps/compare.rs:474:50: +not yet implemented +note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace +[ERROR] Error in step thread: Any { .. } ``` @@ -109,7 +124,10 @@ This time, the step will run again as the result from dependency query (`SELECT ```console $ xvc pipeline run -[ERROR] Step average-age finished UNSUCCESSFULLY with command sqlite3 people.db "SELECT AVG(Age) FROM People;" +thread '' panicked at pipeline/src/pipeline/deps/compare.rs:474:50: +not yet implemented +note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace +[ERROR] Error in step thread: Any { .. } ``` From 66a4a00ede959f1514db8a683587fd4cd7db1fc3 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Fri, 26 Jul 2024 22:24:54 +0300 Subject: [PATCH 065/257] remove comments --- .../xvc-pipeline-step-dependency-sqlite.md | 35 ++++++++++--------- 1 file changed, 19 insertions(+), 16 deletions(-) diff --git a/book/src/ref/xvc-pipeline-step-dependency-sqlite.md b/book/src/ref/xvc-pipeline-step-dependency-sqlite.md index b53f6b942..4a0927a25 100644 --- a/book/src/ref/xvc-pipeline-step-dependency-sqlite.md +++ b/book/src/ref/xvc-pipeline-step-dependency-sqlite.md @@ -19,7 +19,6 @@ Let's create an SQLite database and populate a table: ```console $ sqlite3 people.db < Date: Fri, 26 Jul 2024 22:38:52 +0300 Subject: [PATCH 066/257] rerun --- book/src/ref/xvc-pipeline-step-dependency-sqlite.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/book/src/ref/xvc-pipeline-step-dependency-sqlite.md b/book/src/ref/xvc-pipeline-step-dependency-sqlite.md index 4a0927a25..0bd1423ba 100644 --- a/book/src/ref/xvc-pipeline-step-dependency-sqlite.md +++ b/book/src/ref/xvc-pipeline-step-dependency-sqlite.md @@ -60,7 +60,7 @@ $ xvc pipeline step new --step-name average-age --command 'sqlite3 people.db "SE Let's run the step without a dependency first. ```console -$ xvc -vvv pipeline run +$ xvc -vvvv pipeline run [DEBUG][logging/src/lib.rs::237] Terminal logger enabled with level: Debug [DEBUG][config/src/error.rs::72] Config source for level "system" not found at "/Users/iex/Library/Application Support/com.emresult.xvc" [DEBUG][config/src/error.rs::72] Config source for level "global" not found at "/Users/iex/Library/Application Support/xvc" @@ -68,7 +68,7 @@ $ xvc -vvv pipeline run [DEBUG][/Users/iex/.cargo/registry/src/index.crates.io-6f17d22bba15001f/globset-0.4.14/src/lib.rs::453] built glob set; 0 literals, 3 basenames, 0 extensions, 0 prefixes, 0 suffixes, 0 required extensions, 0 regexes [INFO][pipeline/src/pipeline/mod.rs::347] Pipeline Graph: digraph { - 0 [ label = "(2, 11731315531321152522)" ] + 0 [ label = "(2, 10877507082850142078)" ] } From 17e3dec463544c558c5d415a240817760904aa8f Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Fri, 26 Jul 2024 22:59:57 +0300 Subject: [PATCH 067/257] update --- .../xvc-pipeline-step-dependency-sqlite.md | 8062 ++++++++++++++++- 1 file changed, 8059 insertions(+), 3 deletions(-) diff --git a/book/src/ref/xvc-pipeline-step-dependency-sqlite.md b/book/src/ref/xvc-pipeline-step-dependency-sqlite.md index 0bd1423ba..b2789ddd2 100644 --- a/book/src/ref/xvc-pipeline-step-dependency-sqlite.md +++ b/book/src/ref/xvc-pipeline-step-dependency-sqlite.md @@ -53,7 +53,7 @@ EOF Now, we'll add a step to the pipeline to calculate the average age of these people. ```console -$ xvc pipeline step new --step-name average-age --command 'sqlite3 people.db "SELECT AVG(Age) FROM People;"' +$ xvc pipeline step new --step-name average-age --command "sqlite3 people.db 'SELECT AVG(Age) FROM People;'" ``` @@ -61,24 +61,8080 @@ Let's run the step without a dependency first. ```console $ xvc -vvvv pipeline run -[DEBUG][logging/src/lib.rs::237] Terminal logger enabled with level: Debug +[DEBUG][logging/src/lib.rs::237] Terminal logger enabled with level: Trace +[TRACE][core/src/types/xvcroot.rs::263] xvc_candidate: "[CWD]/.xvc" +[TRACE][core/src/types/xvcroot.rs::265] parent: "[CWD]" [DEBUG][config/src/error.rs::72] Config source for level "system" not found at "/Users/iex/Library/Application Support/com.emresult.xvc" [DEBUG][config/src/error.rs::72] Config source for level "global" not found at "/Users/iex/Library/Application Support/xvc" +[TRACE][ecs/src/ecs/mod.rs::229] dir: "[CWD]/.xvc/ec" +[TRACE][ecs/src/ecs/mod.rs::239] files: [ + "[CWD]/.xvc/ec/1722022738871054", + "[CWD]/.xvc/ec/1722022738873854", + "[CWD]/.xvc/ec/1722022738942386", +] +[TRACE][core/src/types/xvcroot.rs::263] xvc_candidate: "[CWD]/.xvc" +[TRACE][core/src/types/xvcroot.rs::265] parent: "[CWD]" +[TRACE][pipeline/src/pipeline/mod.rs::285] pipeline_e: XvcEntity( + 1, + 2997894859064943878, +) +[TRACE][pipeline/src/pipeline/mod.rs::290] pipeline_steps: HStore { + map: { + XvcEntity( + 2, + 12972606829502460557, + ): XvcStep { + name: "average-age", + }, + }, +} +[TRACE][pipeline/src/pipeline/mod.rs::293] consider_changed: XvcStore { + map: { + XvcEntity( + 2, + 12972606829502460557, + ): ByDependencies, + }, + entity_index: { + ByDependencies: [ + XvcEntity( + 2, + 12972606829502460557, + ), + ], + }, + previous: EventLog( + [ + Add { + entity: XvcEntity( + 2, + 12972606829502460557, + ), + value: ByDependencies, + }, + ], + ), + current: EventLog( + [], + ), +} +[TRACE][pipeline/src/pipeline/mod.rs::296] all_deps.parents.len(): 1 +[TRACE][pipeline/src/pipeline/mod.rs::297] all_deps.children.len(): 0 +[TRACE][pipeline/src/pipeline/mod.rs::299] all_outs.parents.len(): 1 +[TRACE][pipeline/src/pipeline/mod.rs::300] all_outs.children.len(): 0 [DEBUG][/Users/iex/.cargo/registry/src/index.crates.io-6f17d22bba15001f/globset-0.4.14/src/lib.rs::453] built glob set; 0 literals, 2 basenames, 0 extensions, 0 prefixes, 0 suffixes, 0 required extensions, 0 regexes +[TRACE][walker/src/lib.rs::488] ignore_fn: ".xvcignore" +[TRACE][walker/src/lib.rs::490] ignore_root: "[CWD]" +[TRACE][walker/src/lib.rs::498] entry.path(): "[CWD]/.xvc" +[TRACE][walker/src/lib.rs::498] entry.path(): "[CWD]/.git" +[TRACE][walker/src/lib.rs::503] ignore_path: "[CWD]/.xvcignore" +[TRACE][walker/src/lib.rs::597] ignore_root: "[CWD]" +[TRACE][walker/src/lib.rs::598] ignore_path: "[CWD]/.xvcignore" +[TRACE][walker/src/lib.rs::606] &content: " +# Add patterns of files xvc should ignore, which could improve +# the performance. +# It's in the same format as .gitignore files. + +.DS_Store +" [DEBUG][/Users/iex/.cargo/registry/src/index.crates.io-6f17d22bba15001f/globset-0.4.14/src/lib.rs::453] built glob set; 0 literals, 3 basenames, 0 extensions, 0 prefixes, 0 suffixes, 0 required extensions, 0 regexes +[TRACE][walker/src/lib.rs::748] is_abs: true +[TRACE][walker/src/lib.rs::752] path_str: "[CWD]/.xvc" +[TRACE][walker/src/lib.rs::754] final_slash: false +[TRACE][walker/src/lib.rs::776] path: "/.xvc" +[TRACE][walker/src/lib.rs::748] is_abs: true +[TRACE][walker/src/lib.rs::752] path_str: "[CWD]/.git" +[TRACE][walker/src/lib.rs::754] final_slash: false +[TRACE][walker/src/lib.rs::776] path: "/.git" +[TRACE][core/src/util/pmp.rs::41] ignore_rules: IgnoreRules { + root: "[CWD]", + ignore_patterns: RwLock { + data: [ + Pattern { + pattern: Glob { + glob: "**/.xvc", + re: "(?-u)^(?:/?|.*/)//.xvc$", + opts: GlobOptions { + case_insensitive: false, + literal_separator: false, + backslash_escape: true, + empty_alternates: false, + }, + tokens: Tokens( + [ + RecursivePrefix, + Literal( + '.', + ), + Literal( + 'x', + ), + Literal( + 'v', + ), + Literal( + 'c', + ), + ], + ), + }, + original: ".xvc", + source: Global, + effect: Ignore, + relativity: Anywhere, + path_kind: Any, + }, + Pattern { + pattern: Glob { + glob: "**/.git", + re: "(?-u)^(?:/?|.*/)//.git$", + opts: GlobOptions { + case_insensitive: false, + literal_separator: false, + backslash_escape: true, + empty_alternates: false, + }, + tokens: Tokens( + [ + RecursivePrefix, + Literal( + '.', + ), + Literal( + 'g', + ), + Literal( + 'i', + ), + Literal( + 't', + ), + ], + ), + }, + original: ".git", + source: Global, + effect: Ignore, + relativity: Anywhere, + path_kind: Any, + }, + Pattern { + pattern: Glob { + glob: "**/.DS_Store", + re: "(?-u)^(?:/?|.*/)//.DS_Store$", + opts: GlobOptions { + case_insensitive: false, + literal_separator: false, + backslash_escape: true, + empty_alternates: false, + }, + tokens: Tokens( + [ + RecursivePrefix, + Literal( + '.', + ), + Literal( + 'D', + ), + Literal( + 'S', + ), + Literal( + '_', + ), + Literal( + 'S', + ), + Literal( + 't', + ), + Literal( + 'o', + ), + Literal( + 'r', + ), + Literal( + 'e', + ), + ], + ), + }, + original: ".DS_Store", + source: File { + path: ".xvcignore", + line: 6, + }, + effect: Ignore, + relativity: Anywhere, + path_kind: Any, + }, + ], + poisoned: false, + .. + }, + whitelist_patterns: RwLock { + data: [], + poisoned: false, + .. + }, + whitelist_set: RwLock { + data: GlobSet { + len: 0, + strats: [], + }, + poisoned: false, + .. + }, + ignore_set: RwLock { + data: GlobSet { + len: 3, + strats: [ + Extension( + ExtensionStrategy( + {}, + ), + ), + BasenameLiteral( + BasenameLiteralStrategy( + { + [ + 46, + 120, + 118, + 99, + ]: [ + 0, + ], + [ + 46, + 68, + 83, + 95, + 83, + 116, + 111, + 114, + 101, + ]: [ + 2, + ], + [ + 46, + 103, + 105, + 116, + ]: [ + 1, + ], + }, + ), + ), + Literal( + LiteralStrategy( + {}, + ), + ), + Suffix( + SuffixStrategy { + matcher: AhoCorasick( + dfa::DFA( + D 000000: /x00 => 0 + F 000001: + >000002: /x00 => 2 + 000003: /x00 => 0 + match kind: Standard + prefilter: false + state length: 4 + pattern length: 0 + shortest pattern length: 18446744073709551615 + longest pattern length: 0 + alphabet length: 1 + stride: 1 + byte classes: ByteClasses(0 => [0-255]) + memory usage: 16 + ) + , + ), + map: [], + longest: 0, + }, + ), + Prefix( + PrefixStrategy { + matcher: AhoCorasick( + dfa::DFA( + D 000000: /x00 => 0 + F 000001: + >000002: /x00 => 2 + 000003: /x00 => 0 + match kind: Standard + prefilter: false + state length: 4 + pattern length: 0 + shortest pattern length: 18446744073709551615 + longest pattern length: 0 + alphabet length: 1 + stride: 1 + byte classes: ByteClasses(0 => [0-255]) + memory usage: 16 + ) + , + ), + map: [], + longest: 0, + }, + ), + RequiredExtension( + RequiredExtensionStrategy( + {}, + ), + ), + Regex( + RegexSetStrategy { + matcher: Regex { + imp: RegexI { + strat: Core { + info: RegexInfo( + RegexInfoI { + config: Config { + match_kind: Some( + All, + ), + utf8_empty: Some( + false, + ), + autopre: None, + pre: None, + which_captures: None, + nfa_size_limit: Some( + Some( + 10485760, + ), + ), + onepass_size_limit: None, + hybrid_cache_capacity: Some( + 10485760, + ), + hybrid: None, + dfa: None, + dfa_size_limit: None, + dfa_state_limit: None, + onepass: None, + backtrack: None, + byte_classes: None, + line_terminator: None, + }, + props: [], + props_union: Properties( + PropertiesI { + minimum_len: None, + maximum_len: None, + look_set: ∅, + look_set_prefix: ∅, + look_set_suffix: ∅, + look_set_prefix_any: ∅, + look_set_suffix_any: ∅, + utf8: true, + explicit_captures_len: 0, + static_explicit_captures_len: None, + literal: false, + alternation_literal: true, + }, + ), + }, + ), + pre: None, + nfa: thompson::NFA( + ^000000: FAIL + + transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) + ) + , + nfarev: Some( + thompson::NFA( + ^000000: FAIL + + transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) + ) + , + ), + pikevm: PikeVM( + PikeVMEngine( + PikeVM { + config: Config { + match_kind: Some( + All, + ), + pre: Some( + None, + ), + }, + nfa: thompson::NFA( + ^000000: FAIL + + transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) + ) + , + }, + ), + ), + backtrack: BoundedBacktracker( + None, + ), + onepass: OnePass( + None, + ), + hybrid: Hybrid( + Some( + HybridEngine( + Regex { + forward: DFA { + config: Config { + match_kind: Some( + All, + ), + pre: Some( + None, + ), + starts_for_each_pattern: Some( + true, + ), + byte_classes: Some( + true, + ), + unicode_word_boundary: Some( + true, + ), + quitset: None, + specialize_start_states: Some( + false, + ), + cache_capacity: Some( + 10485760, + ), + skip_cache_capacity_check: Some( + false, + ), + minimum_cache_clear_count: Some( + Some( + 3, + ), + ), + minimum_bytes_per_state: Some( + Some( + 10, + ), + ), + }, + nfa: thompson::NFA( + ^000000: FAIL + + transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) + ) + , + stride2: 1, + start_map: StartByteMap{/x00 => NonWordByte, /x01 => NonWordByte, /x02 => NonWordByte, /x03 => NonWordByte, /x04 => NonWordByte, /x05 => NonWordByte, /x06 => NonWordByte, /x07 => NonWordByte, /x08 => NonWordByte, /t => NonWordByte, + => LineLF, /x0B => NonWordByte, /x0C => NonWordByte, /r => LineCR, /x0E => NonWordByte, /x0F => NonWordByte, /x10 => NonWordByte, /x11 => NonWordByte, /x12 => NonWordByte, /x13 => NonWordByte, /x14 => NonWordByte, /x15 => NonWordByte, /x16 => NonWordByte, /x17 => NonWordByte, /x18 => NonWordByte, /x19 => NonWordByte, /x1A => NonWordByte, /x1B => NonWordByte, /x1C => NonWordByte, /x1D => NonWordByte, /x1E => NonWordByte, /x1F => NonWordByte, ' ' => NonWordByte, ! => NonWordByte, /" => NonWordByte, # => NonWordByte, $ => NonWordByte, % => NonWordByte, & => NonWordByte, /' => NonWordByte, ( => NonWordByte, ) => NonWordByte, * => NonWordByte, + => NonWordByte, , => NonWordByte, - => NonWordByte, . => NonWordByte, / => NonWordByte, 0 => WordByte, 1 => WordByte, 2 => WordByte, 3 => WordByte, 4 => WordByte, 5 => WordByte, 6 => WordByte, 7 => WordByte, 8 => WordByte, 9 => WordByte, : => NonWordByte, ; => NonWordByte, < => NonWordByte, = => NonWordByte, > => NonWordByte, ? => NonWordByte, @ => NonWordByte, A => WordByte, B => WordByte, C => WordByte, D => WordByte, E => WordByte, F => WordByte, G => WordByte, H => WordByte, I => WordByte, J => WordByte, K => WordByte, L => WordByte, M => WordByte, N => WordByte, O => WordByte, P => WordByte, Q => WordByte, R => WordByte, S => WordByte, T => WordByte, U => WordByte, V => WordByte, W => WordByte, X => WordByte, Y => WordByte, Z => WordByte, [ => NonWordByte, // => NonWordByte, ] => NonWordByte, ^ => NonWordByte, _ => WordByte, ` => NonWordByte, a => WordByte, b => WordByte, c => WordByte, d => WordByte, e => WordByte, f => WordByte, g => WordByte, h => WordByte, i => WordByte, j => WordByte, k => WordByte, l => WordByte, m => WordByte, n => WordByte, o => WordByte, p => WordByte, q => WordByte, r => WordByte, s => WordByte, t => WordByte, u => WordByte, v => WordByte, w => WordByte, x => WordByte, y => WordByte, z => WordByte, { => NonWordByte, | => NonWordByte, } => NonWordByte, ~ => NonWordByte, /x7F => NonWordByte, /x80 => NonWordByte, /x81 => NonWordByte, /x82 => NonWordByte, /x83 => NonWordByte, /x84 => NonWordByte, /x85 => NonWordByte, /x86 => NonWordByte, /x87 => NonWordByte, /x88 => NonWordByte, /x89 => NonWordByte, /x8A => NonWordByte, /x8B => NonWordByte, /x8C => NonWordByte, /x8D => NonWordByte, /x8E => NonWordByte, /x8F => NonWordByte, /x90 => NonWordByte, /x91 => NonWordByte, /x92 => NonWordByte, /x93 => NonWordByte, /x94 => NonWordByte, /x95 => NonWordByte, /x96 => NonWordByte, /x97 => NonWordByte, /x98 => NonWordByte, /x99 => NonWordByte, /x9A => NonWordByte, /x9B => NonWordByte, /x9C => NonWordByte, /x9D => NonWordByte, /x9E => NonWordByte, /x9F => NonWordByte, /xA0 => NonWordByte, /xA1 => NonWordByte, /xA2 => NonWordByte, /xA3 => NonWordByte, /xA4 => NonWordByte, /xA5 => NonWordByte, /xA6 => NonWordByte, /xA7 => NonWordByte, /xA8 => NonWordByte, /xA9 => NonWordByte, /xAA => NonWordByte, /xAB => NonWordByte, /xAC => NonWordByte, /xAD => NonWordByte, /xAE => NonWordByte, /xAF => NonWordByte, /xB0 => NonWordByte, /xB1 => NonWordByte, /xB2 => NonWordByte, /xB3 => NonWordByte, /xB4 => NonWordByte, /xB5 => NonWordByte, /xB6 => NonWordByte, /xB7 => NonWordByte, /xB8 => NonWordByte, /xB9 => NonWordByte, /xBA => NonWordByte, /xBB => NonWordByte, /xBC => NonWordByte, /xBD => NonWordByte, /xBE => NonWordByte, /xBF => NonWordByte, /xC0 => NonWordByte, /xC1 => NonWordByte, /xC2 => NonWordByte, /xC3 => NonWordByte, /xC4 => NonWordByte, /xC5 => NonWordByte, /xC6 => NonWordByte, /xC7 => NonWordByte, /xC8 => NonWordByte, /xC9 => NonWordByte, /xCA => NonWordByte, /xCB => NonWordByte, /xCC => NonWordByte, /xCD => NonWordByte, /xCE => NonWordByte, /xCF => NonWordByte, /xD0 => NonWordByte, /xD1 => NonWordByte, /xD2 => NonWordByte, /xD3 => NonWordByte, /xD4 => NonWordByte, /xD5 => NonWordByte, /xD6 => NonWordByte, /xD7 => NonWordByte, /xD8 => NonWordByte, /xD9 => NonWordByte, /xDA => NonWordByte, /xDB => NonWordByte, /xDC => NonWordByte, /xDD => NonWordByte, /xDE => NonWordByte, /xDF => NonWordByte, /xE0 => NonWordByte, /xE1 => NonWordByte, /xE2 => NonWordByte, /xE3 => NonWordByte, /xE4 => NonWordByte, /xE5 => NonWordByte, /xE6 => NonWordByte, /xE7 => NonWordByte, /xE8 => NonWordByte, /xE9 => NonWordByte, /xEA => NonWordByte, /xEB => NonWordByte, /xEC => NonWordByte, /xED => NonWordByte, /xEE => NonWordByte, /xEF => NonWordByte, /xF0 => NonWordByte, /xF1 => NonWordByte, /xF2 => NonWordByte, /xF3 => NonWordByte, /xF4 => NonWordByte, /xF5 => NonWordByte, /xF6 => NonWordByte, /xF7 => NonWordByte, /xF8 => NonWordByte, /xF9 => NonWordByte, /xFA => NonWordByte, /xFB => NonWordByte, /xFC => NonWordByte, /xFD => NonWordByte, /xFE => NonWordByte, /xFF => NonWordByte}, + classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]), + quitset: ByteSet { + bits: {}, + }, + cache_capacity: 10485760, + }, + reverse: DFA { + config: Config { + match_kind: Some( + All, + ), + pre: Some( + None, + ), + starts_for_each_pattern: Some( + true, + ), + byte_classes: Some( + true, + ), + unicode_word_boundary: Some( + true, + ), + quitset: None, + specialize_start_states: Some( + false, + ), + cache_capacity: Some( + 10485760, + ), + skip_cache_capacity_check: Some( + false, + ), + minimum_cache_clear_count: Some( + Some( + 3, + ), + ), + minimum_bytes_per_state: Some( + Some( + 10, + ), + ), + }, + nfa: thompson::NFA( + ^000000: FAIL + + transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) + ) + , + stride2: 1, + start_map: StartByteMap{/x00 => NonWordByte, /x01 => NonWordByte, /x02 => NonWordByte, /x03 => NonWordByte, /x04 => NonWordByte, /x05 => NonWordByte, /x06 => NonWordByte, /x07 => NonWordByte, /x08 => NonWordByte, /t => NonWordByte, + => LineLF, /x0B => NonWordByte, /x0C => NonWordByte, /r => LineCR, /x0E => NonWordByte, /x0F => NonWordByte, /x10 => NonWordByte, /x11 => NonWordByte, /x12 => NonWordByte, /x13 => NonWordByte, /x14 => NonWordByte, /x15 => NonWordByte, /x16 => NonWordByte, /x17 => NonWordByte, /x18 => NonWordByte, /x19 => NonWordByte, /x1A => NonWordByte, /x1B => NonWordByte, /x1C => NonWordByte, /x1D => NonWordByte, /x1E => NonWordByte, /x1F => NonWordByte, ' ' => NonWordByte, ! => NonWordByte, /" => NonWordByte, # => NonWordByte, $ => NonWordByte, % => NonWordByte, & => NonWordByte, /' => NonWordByte, ( => NonWordByte, ) => NonWordByte, * => NonWordByte, + => NonWordByte, , => NonWordByte, - => NonWordByte, . => NonWordByte, / => NonWordByte, 0 => WordByte, 1 => WordByte, 2 => WordByte, 3 => WordByte, 4 => WordByte, 5 => WordByte, 6 => WordByte, 7 => WordByte, 8 => WordByte, 9 => WordByte, : => NonWordByte, ; => NonWordByte, < => NonWordByte, = => NonWordByte, > => NonWordByte, ? => NonWordByte, @ => NonWordByte, A => WordByte, B => WordByte, C => WordByte, D => WordByte, E => WordByte, F => WordByte, G => WordByte, H => WordByte, I => WordByte, J => WordByte, K => WordByte, L => WordByte, M => WordByte, N => WordByte, O => WordByte, P => WordByte, Q => WordByte, R => WordByte, S => WordByte, T => WordByte, U => WordByte, V => WordByte, W => WordByte, X => WordByte, Y => WordByte, Z => WordByte, [ => NonWordByte, // => NonWordByte, ] => NonWordByte, ^ => NonWordByte, _ => WordByte, ` => NonWordByte, a => WordByte, b => WordByte, c => WordByte, d => WordByte, e => WordByte, f => WordByte, g => WordByte, h => WordByte, i => WordByte, j => WordByte, k => WordByte, l => WordByte, m => WordByte, n => WordByte, o => WordByte, p => WordByte, q => WordByte, r => WordByte, s => WordByte, t => WordByte, u => WordByte, v => WordByte, w => WordByte, x => WordByte, y => WordByte, z => WordByte, { => NonWordByte, | => NonWordByte, } => NonWordByte, ~ => NonWordByte, /x7F => NonWordByte, /x80 => NonWordByte, /x81 => NonWordByte, /x82 => NonWordByte, /x83 => NonWordByte, /x84 => NonWordByte, /x85 => NonWordByte, /x86 => NonWordByte, /x87 => NonWordByte, /x88 => NonWordByte, /x89 => NonWordByte, /x8A => NonWordByte, /x8B => NonWordByte, /x8C => NonWordByte, /x8D => NonWordByte, /x8E => NonWordByte, /x8F => NonWordByte, /x90 => NonWordByte, /x91 => NonWordByte, /x92 => NonWordByte, /x93 => NonWordByte, /x94 => NonWordByte, /x95 => NonWordByte, /x96 => NonWordByte, /x97 => NonWordByte, /x98 => NonWordByte, /x99 => NonWordByte, /x9A => NonWordByte, /x9B => NonWordByte, /x9C => NonWordByte, /x9D => NonWordByte, /x9E => NonWordByte, /x9F => NonWordByte, /xA0 => NonWordByte, /xA1 => NonWordByte, /xA2 => NonWordByte, /xA3 => NonWordByte, /xA4 => NonWordByte, /xA5 => NonWordByte, /xA6 => NonWordByte, /xA7 => NonWordByte, /xA8 => NonWordByte, /xA9 => NonWordByte, /xAA => NonWordByte, /xAB => NonWordByte, /xAC => NonWordByte, /xAD => NonWordByte, /xAE => NonWordByte, /xAF => NonWordByte, /xB0 => NonWordByte, /xB1 => NonWordByte, /xB2 => NonWordByte, /xB3 => NonWordByte, /xB4 => NonWordByte, /xB5 => NonWordByte, /xB6 => NonWordByte, /xB7 => NonWordByte, /xB8 => NonWordByte, /xB9 => NonWordByte, /xBA => NonWordByte, /xBB => NonWordByte, /xBC => NonWordByte, /xBD => NonWordByte, /xBE => NonWordByte, /xBF => NonWordByte, /xC0 => NonWordByte, /xC1 => NonWordByte, /xC2 => NonWordByte, /xC3 => NonWordByte, /xC4 => NonWordByte, /xC5 => NonWordByte, /xC6 => NonWordByte, /xC7 => NonWordByte, /xC8 => NonWordByte, /xC9 => NonWordByte, /xCA => NonWordByte, /xCB => NonWordByte, /xCC => NonWordByte, /xCD => NonWordByte, /xCE => NonWordByte, /xCF => NonWordByte, /xD0 => NonWordByte, /xD1 => NonWordByte, /xD2 => NonWordByte, /xD3 => NonWordByte, /xD4 => NonWordByte, /xD5 => NonWordByte, /xD6 => NonWordByte, /xD7 => NonWordByte, /xD8 => NonWordByte, /xD9 => NonWordByte, /xDA => NonWordByte, /xDB => NonWordByte, /xDC => NonWordByte, /xDD => NonWordByte, /xDE => NonWordByte, /xDF => NonWordByte, /xE0 => NonWordByte, /xE1 => NonWordByte, /xE2 => NonWordByte, /xE3 => NonWordByte, /xE4 => NonWordByte, /xE5 => NonWordByte, /xE6 => NonWordByte, /xE7 => NonWordByte, /xE8 => NonWordByte, /xE9 => NonWordByte, /xEA => NonWordByte, /xEB => NonWordByte, /xEC => NonWordByte, /xED => NonWordByte, /xEE => NonWordByte, /xEF => NonWordByte, /xF0 => NonWordByte, /xF1 => NonWordByte, /xF2 => NonWordByte, /xF3 => NonWordByte, /xF4 => NonWordByte, /xF5 => NonWordByte, /xF6 => NonWordByte, /xF7 => NonWordByte, /xF8 => NonWordByte, /xF9 => NonWordByte, /xFA => NonWordByte, /xFB => NonWordByte, /xFC => NonWordByte, /xFD => NonWordByte, /xFE => NonWordByte, /xFF => NonWordByte}, + classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]), + quitset: ByteSet { + bits: {}, + }, + cache_capacity: 10485760, + }, + }, + ), + ), + ), + dfa: DFA( + None, + ), + }, + info: RegexInfo( + RegexInfoI { + config: Config { + match_kind: Some( + All, + ), + utf8_empty: Some( + false, + ), + autopre: None, + pre: None, + which_captures: None, + nfa_size_limit: Some( + Some( + 10485760, + ), + ), + onepass_size_limit: None, + hybrid_cache_capacity: Some( + 10485760, + ), + hybrid: None, + dfa: None, + dfa_size_limit: None, + dfa_state_limit: None, + onepass: None, + backtrack: None, + byte_classes: None, + line_terminator: None, + }, + props: [], + props_union: Properties( + PropertiesI { + minimum_len: None, + maximum_len: None, + look_set: ∅, + look_set_prefix: ∅, + look_set_suffix: ∅, + look_set_prefix_any: ∅, + look_set_suffix_any: ∅, + utf8: true, + explicit_captures_len: 0, + static_explicit_captures_len: None, + literal: false, + alternation_literal: true, + }, + ), + }, + ), + }, + pool: Pool( + Pool { + stacks: [ + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + ], + owner: 0, + owner_val: UnsafeCell { .. }, + }, + ), + }, + map: [], + patset: Pool( + Pool { + stacks: [ + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + ], + owner: 0, + owner_val: UnsafeCell { .. }, + }, + ), + }, + ), + ], + }, + poisoned: false, + .. + }, +} +[TRACE][walker/src/notify.rs::170] watcher: FsEventWatcher { + paths: 0x0000600003c44f90, + since_when: 18446744073709551615, + latency: 0.0, + flags: 18, + event_handler: 0x0000600001944860, + runloop: Some( + ( + 0x00006000007540c0, + JoinHandle { .. }, + ), + ), + recursive_info: { + "[CWD]": true, + }, +} +[TRACE][core/src/util/pmp.rs::131] background_thread: Mutex { + data: JoinHandle { .. }, + poisoned: false, + .. +} +[TRACE][core/src/util/pmp.rs::59] watcher: FsEventWatcher { + paths: 0x0000600003c44f90, + since_when: 18446744073709551615, + latency: 0.0, + flags: 18, + event_handler: 0x0000600001944860, + runloop: Some( + ( + 0x00006000007540c0, + JoinHandle { .. }, + ), + ), + recursive_info: { + "[CWD]": true, + }, +} +[TRACE][core/src/util/pmp.rs::91] fs_event_index: 0 +[TRACE][core/src/util/pmp.rs::93] kill_signal_index: 1 +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][pipeline/src/pipeline/mod.rs::302] &pmp: XvcPathMetadataProvider { + xvc_root: XvcRootInner { + absolute_path: AbsolutePath( + "[CWD]", + ), + xvc_dir: AbsolutePath( + "[CWD]/.xvc", + ), + store_dir: AbsolutePath( + "[CWD]/.xvc/store", + ), + config: XvcConfig { + current_dir: XvcConfigOption { + source: Runtime, + option: AbsolutePath( + "[CWD]", + ), + }, + config_maps: [ + XvcConfigMap { + source: Default, + map: { + "file.list.sort": String( + "name-desc", + ), + "pipeline.process_pool_size": Integer( + 4, + ), + "git.command": String( + "git", + ), + "pipeline.current_pipeline": String( + "default", + ), + "file.carry-in.force": Boolean( + false, + ), + "pipeline.default_params_file": String( + "params.yaml", + ), + "pipeline.default": String( + "default", + ), + "core.guid": String( + "7726b84aa14cc82c", + ), + "git.auto_stage": Boolean( + false, + ), + "file.track.force": Boolean( + false, + ), + "cache.algorithm": String( + "blake3", + ), + "file.recheck.method": String( + "copy", + ), + "core.verbosity": String( + "error", + ), + "file.track.no_commit": Boolean( + false, + ), + "file.list.show_dot_files": Boolean( + false, + ), + "file.track.text_or_binary": String( + "auto", + ), + "file.list.recursive": Boolean( + false, + ), + "file.list.no_summary": Boolean( + false, + ), + "git.use_git": Boolean( + true, + ), + "file.carry-in.no_parallel": Boolean( + false, + ), + "file.list.format": String( + "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", + ), + "file.track.no_parallel": Boolean( + false, + ), + "git.auto_commit": Boolean( + true, + ), + }, + }, + XvcConfigMap { + source: Project, + map: { + "file.list.no_summary": Boolean( + false, + ), + "file.list.sort": String( + "name-desc", + ), + "git.auto_commit": Boolean( + true, + ), + "git.auto_stage": Boolean( + false, + ), + "file.track.no_parallel": Boolean( + false, + ), + "file.track.no_commit": Boolean( + false, + ), + "pipeline.process_pool_size": Integer( + 4, + ), + "pipeline.default_params_file": String( + "params.yaml", + ), + "pipeline.default": String( + "default", + ), + "git.command": String( + "git", + ), + "file.list.show_dot_files": Boolean( + false, + ), + "file.list.format": String( + "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", + ), + "core.guid": String( + "4bd7a8139a85fb10", + ), + "pipeline.current_pipeline": String( + "default", + ), + "file.track.text_or_binary": String( + "auto", + ), + "file.recheck.method": String( + "copy", + ), + "git.use_git": Boolean( + true, + ), + "file.list.recursive": Boolean( + false, + ), + "file.carry-in.no_parallel": Boolean( + false, + ), + "file.carry-in.force": Boolean( + false, + ), + "file.track.force": Boolean( + false, + ), + "core.verbosity": String( + "error", + ), + "cache.algorithm": String( + "blake3", + ), + }, + }, + XvcConfigMap { + source: Local, + map: {}, + }, + XvcConfigMap { + source: Environment, + map: { + "TRYCMD_TESTS": String( + "storage,file,pipeline,core", + ), + }, + }, + XvcConfigMap { + source: CommandLine, + map: { + "core.verbosity": String( + "debug", + ), + "core.quiet": Boolean( + false, + ), + }, + }, + ], + the_config: { + "pipeline.default_params_file": XvcConfigValue { + source: Project, + value: String( + "params.yaml", + ), + }, + "file.list.recursive": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "file.track.no_commit": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "file.carry-in.no_parallel": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "pipeline.current_pipeline": XvcConfigValue { + source: Project, + value: String( + "default", + ), + }, + "file.track.force": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "pipeline.default": XvcConfigValue { + source: Project, + value: String( + "default", + ), + }, + "cache.algorithm": XvcConfigValue { + source: Project, + value: String( + "blake3", + ), + }, + "file.track.text_or_binary": XvcConfigValue { + source: Project, + value: String( + "auto", + ), + }, + "TRYCMD_TESTS": XvcConfigValue { + source: Environment, + value: String( + "storage,file,pipeline,core", + ), + }, + "pipeline.process_pool_size": XvcConfigValue { + source: Project, + value: Integer( + 4, + ), + }, + "file.list.no_summary": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "core.guid": XvcConfigValue { + source: Project, + value: String( + "4bd7a8139a85fb10", + ), + }, + "file.list.format": XvcConfigValue { + source: Project, + value: String( + "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", + ), + }, + "file.carry-in.force": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "git.use_git": XvcConfigValue { + source: Project, + value: Boolean( + true, + ), + }, + "git.auto_stage": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "file.list.show_dot_files": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "file.track.no_parallel": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "git.auto_commit": XvcConfigValue { + source: Project, + value: Boolean( + true, + ), + }, + "file.list.sort": XvcConfigValue { + source: Project, + value: String( + "name-desc", + ), + }, + "file.recheck.method": XvcConfigValue { + source: Project, + value: String( + "copy", + ), + }, + "core.verbosity": XvcConfigValue { + source: CommandLine, + value: String( + "debug", + ), + }, + "git.command": XvcConfigValue { + source: Project, + value: String( + "git", + ), + }, + "core.quiet": XvcConfigValue { + source: CommandLine, + value: Boolean( + false, + ), + }, + }, + init_params: XvcConfigParams { + default_configuration: " +[core] +# The repository id. Please do not delete or change it. +# This is used to identify the repository and generate paths in storages. +# In the future it may be used to in other ways. +guid = /"7726b84aa14cc82c/" +# Default verbosity level. +# One of /"error/", /"warn/", /"info/" +verbosity = /"error/" + +[git] +# Automate git operations. +# Turning this off leads Xvc to behave as if it's not in a Git repository. +# Not recommended unless you're really not using Git +use_git = true +# Command to run Git process. +# You can set this to an absolute path to specify an executable +# If set to a non-absolute path, the executable will be searched in $PATH. +command = /"git/" + +# Commit changes in .xvc/ directory after commands. +# You can set this to false if you want to commit manually. +auto_commit = true + +# Stage changes in .xvc/ directory without committing. +# auto_commit implies auto_stage. +# If you want to commit manually but don't want to stage after individual Xvc commands, you can set this to true. +auto_stage = false + +[cache] +# The hash algorithm used for the cache. +# It may take blake3, blake2, sha2 or sha3 as values. +# All algorithms are selected to produce 256-bit hashes, so sha2 means SHA2-256, blake2 means BLAKE2s, etc. +# The cache path is produced by prepending algorithm name to the cache. +# Blake3 files are in .xvc/b3/, while sha2 files are in .xvc/s2/ etc. +algorithm = /"blake3/" + +[file] + +[file.track] + +# Don't move file content to cache after xvc file track +no_commit = false +# Force to track files even if they are already tracked. +force = false + +# Xvc calculates file content digest differently for text and binary files. +# This option controls whether to treat files as text or binary. +# It may take auto, text or binary as values. +# Auto check each file individually and treat it as text if it's text. +text_or_binary = /"auto/" + +# Don't use parallelism in track operations. +# Note that some of the operations are implemented in parallel by default, and this option affects some heavier operations. +no_parallel = false + +[file.list] + +# Format for `xvc file list` rows. You can reorder or remove columns. +# The following are the keys for each row: +# - {acd64}: actual content digest. All 64 digits from the workspace file's content. +# - {acd8}: actual content digest. First 8 digits the file content digest. +# - {aft}: actual file type. Whether the entry is a file (F), directory (D), +# symlink (S), hardlink (H) or reflink (R). +# - {asz}: actual size. The size of the workspace file in bytes. It uses MB, +# GB and TB to represent sizes larger than 1MB. +# - {ats}: actual timestamp. The timestamp of the workspace file. +# - {cst}: cache status. One of /"=/", /">/", /" 0 + F 000001: + >000002: /x00 => 2 + 000003: /x00 => 0 + match kind: Standard + prefilter: false + state length: 4 + pattern length: 0 + shortest pattern length: 18446744073709551615 + longest pattern length: 0 + alphabet length: 1 + stride: 1 + byte classes: ByteClasses(0 => [0-255]) + memory usage: 16 + ) + , + ), + map: [], + longest: 0, + }, + ), + Prefix( + PrefixStrategy { + matcher: AhoCorasick( + dfa::DFA( + D 000000: /x00 => 0 + F 000001: + >000002: /x00 => 2 + 000003: /x00 => 0 + match kind: Standard + prefilter: false + state length: 4 + pattern length: 0 + shortest pattern length: 18446744073709551615 + longest pattern length: 0 + alphabet length: 1 + stride: 1 + byte classes: ByteClasses(0 => [0-255]) + memory usage: 16 + ) + , + ), + map: [], + longest: 0, + }, + ), + RequiredExtension( + RequiredExtensionStrategy( + {}, + ), + ), + Regex( + RegexSetStrategy { + matcher: Regex { + imp: RegexI { + strat: Core { + info: RegexInfo( + RegexInfoI { + config: Config { + match_kind: Some( + All, + ), + utf8_empty: Some( + false, + ), + autopre: None, + pre: None, + which_captures: None, + nfa_size_limit: Some( + Some( + 10485760, + ), + ), + onepass_size_limit: None, + hybrid_cache_capacity: Some( + 10485760, + ), + hybrid: None, + dfa: None, + dfa_size_limit: None, + dfa_state_limit: None, + onepass: None, + backtrack: None, + byte_classes: None, + line_terminator: None, + }, + props: [], + props_union: Properties( + PropertiesI { + minimum_len: None, + maximum_len: None, + look_set: ∅, + look_set_prefix: ∅, + look_set_suffix: ∅, + look_set_prefix_any: ∅, + look_set_suffix_any: ∅, + utf8: true, + explicit_captures_len: 0, + static_explicit_captures_len: None, + literal: false, + alternation_literal: true, + }, + ), + }, + ), + pre: None, + nfa: thompson::NFA( + ^000000: FAIL + + transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) + ) + , + nfarev: Some( + thompson::NFA( + ^000000: FAIL + + transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) + ) + , + ), + pikevm: PikeVM( + PikeVMEngine( + PikeVM { + config: Config { + match_kind: Some( + All, + ), + pre: Some( + None, + ), + }, + nfa: thompson::NFA( + ^000000: FAIL + + transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) + ) + , + }, + ), + ), + backtrack: BoundedBacktracker( + None, + ), + onepass: OnePass( + None, + ), + hybrid: Hybrid( + Some( + HybridEngine( + Regex { + forward: DFA { + config: Config { + match_kind: Some( + All, + ), + pre: Some( + None, + ), + starts_for_each_pattern: Some( + true, + ), + byte_classes: Some( + true, + ), + unicode_word_boundary: Some( + true, + ), + quitset: None, + specialize_start_states: Some( + false, + ), + cache_capacity: Some( + 10485760, + ), + skip_cache_capacity_check: Some( + false, + ), + minimum_cache_clear_count: Some( + Some( + 3, + ), + ), + minimum_bytes_per_state: Some( + Some( + 10, + ), + ), + }, + nfa: thompson::NFA( + ^000000: FAIL + + transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) + ) + , + stride2: 1, + start_map: StartByteMap{/x00 => NonWordByte, /x01 => NonWordByte, /x02 => NonWordByte, /x03 => NonWordByte, /x04 => NonWordByte, /x05 => NonWordByte, /x06 => NonWordByte, /x07 => NonWordByte, /x08 => NonWordByte, /t => NonWordByte, + => LineLF, /x0B => NonWordByte, /x0C => NonWordByte, /r => LineCR, /x0E => NonWordByte, /x0F => NonWordByte, /x10 => NonWordByte, /x11 => NonWordByte, /x12 => NonWordByte, /x13 => NonWordByte, /x14 => NonWordByte, /x15 => NonWordByte, /x16 => NonWordByte, /x17 => NonWordByte, /x18 => NonWordByte, /x19 => NonWordByte, /x1A => NonWordByte, /x1B => NonWordByte, /x1C => NonWordByte, /x1D => NonWordByte, /x1E => NonWordByte, /x1F => NonWordByte, ' ' => NonWordByte, ! => NonWordByte, /" => NonWordByte, # => NonWordByte, $ => NonWordByte, % => NonWordByte, & => NonWordByte, /' => NonWordByte, ( => NonWordByte, ) => NonWordByte, * => NonWordByte, + => NonWordByte, , => NonWordByte, - => NonWordByte, . => NonWordByte, / => NonWordByte, 0 => WordByte, 1 => WordByte, 2 => WordByte, 3 => WordByte, 4 => WordByte, 5 => WordByte, 6 => WordByte, 7 => WordByte, 8 => WordByte, 9 => WordByte, : => NonWordByte, ; => NonWordByte, < => NonWordByte, = => NonWordByte, > => NonWordByte, ? => NonWordByte, @ => NonWordByte, A => WordByte, B => WordByte, C => WordByte, D => WordByte, E => WordByte, F => WordByte, G => WordByte, H => WordByte, I => WordByte, J => WordByte, K => WordByte, L => WordByte, M => WordByte, N => WordByte, O => WordByte, P => WordByte, Q => WordByte, R => WordByte, S => WordByte, T => WordByte, U => WordByte, V => WordByte, W => WordByte, X => WordByte, Y => WordByte, Z => WordByte, [ => NonWordByte, // => NonWordByte, ] => NonWordByte, ^ => NonWordByte, _ => WordByte, ` => NonWordByte, a => WordByte, b => WordByte, c => WordByte, d => WordByte, e => WordByte, f => WordByte, g => WordByte, h => WordByte, i => WordByte, j => WordByte, k => WordByte, l => WordByte, m => WordByte, n => WordByte, o => WordByte, p => WordByte, q => WordByte, r => WordByte, s => WordByte, t => WordByte, u => WordByte, v => WordByte, w => WordByte, x => WordByte, y => WordByte, z => WordByte, { => NonWordByte, | => NonWordByte, } => NonWordByte, ~ => NonWordByte, /x7F => NonWordByte, /x80 => NonWordByte, /x81 => NonWordByte, /x82 => NonWordByte, /x83 => NonWordByte, /x84 => NonWordByte, /x85 => NonWordByte, /x86 => NonWordByte, /x87 => NonWordByte, /x88 => NonWordByte, /x89 => NonWordByte, /x8A => NonWordByte, /x8B => NonWordByte, /x8C => NonWordByte, /x8D => NonWordByte, /x8E => NonWordByte, /x8F => NonWordByte, /x90 => NonWordByte, /x91 => NonWordByte, /x92 => NonWordByte, /x93 => NonWordByte, /x94 => NonWordByte, /x95 => NonWordByte, /x96 => NonWordByte, /x97 => NonWordByte, /x98 => NonWordByte, /x99 => NonWordByte, /x9A => NonWordByte, /x9B => NonWordByte, /x9C => NonWordByte, /x9D => NonWordByte, /x9E => NonWordByte, /x9F => NonWordByte, /xA0 => NonWordByte, /xA1 => NonWordByte, /xA2 => NonWordByte, /xA3 => NonWordByte, /xA4 => NonWordByte, /xA5 => NonWordByte, /xA6 => NonWordByte, /xA7 => NonWordByte, /xA8 => NonWordByte, /xA9 => NonWordByte, /xAA => NonWordByte, /xAB => NonWordByte, /xAC => NonWordByte, /xAD => NonWordByte, /xAE => NonWordByte, /xAF => NonWordByte, /xB0 => NonWordByte, /xB1 => NonWordByte, /xB2 => NonWordByte, /xB3 => NonWordByte, /xB4 => NonWordByte, /xB5 => NonWordByte, /xB6 => NonWordByte, /xB7 => NonWordByte, /xB8 => NonWordByte, /xB9 => NonWordByte, /xBA => NonWordByte, /xBB => NonWordByte, /xBC => NonWordByte, /xBD => NonWordByte, /xBE => NonWordByte, /xBF => NonWordByte, /xC0 => NonWordByte, /xC1 => NonWordByte, /xC2 => NonWordByte, /xC3 => NonWordByte, /xC4 => NonWordByte, /xC5 => NonWordByte, /xC6 => NonWordByte, /xC7 => NonWordByte, /xC8 => NonWordByte, /xC9 => NonWordByte, /xCA => NonWordByte, /xCB => NonWordByte, /xCC => NonWordByte, /xCD => NonWordByte, /xCE => NonWordByte, /xCF => NonWordByte, /xD0 => NonWordByte, /xD1 => NonWordByte, /xD2 => NonWordByte, /xD3 => NonWordByte, /xD4 => NonWordByte, /xD5 => NonWordByte, /xD6 => NonWordByte, /xD7 => NonWordByte, /xD8 => NonWordByte, /xD9 => NonWordByte, /xDA => NonWordByte, /xDB => NonWordByte, /xDC => NonWordByte, /xDD => NonWordByte, /xDE => NonWordByte, /xDF => NonWordByte, /xE0 => NonWordByte, /xE1 => NonWordByte, /xE2 => NonWordByte, /xE3 => NonWordByte, /xE4 => NonWordByte, /xE5 => NonWordByte, /xE6 => NonWordByte, /xE7 => NonWordByte, /xE8 => NonWordByte, /xE9 => NonWordByte, /xEA => NonWordByte, /xEB => NonWordByte, /xEC => NonWordByte, /xED => NonWordByte, /xEE => NonWordByte, /xEF => NonWordByte, /xF0 => NonWordByte, /xF1 => NonWordByte, /xF2 => NonWordByte, /xF3 => NonWordByte, /xF4 => NonWordByte, /xF5 => NonWordByte, /xF6 => NonWordByte, /xF7 => NonWordByte, /xF8 => NonWordByte, /xF9 => NonWordByte, /xFA => NonWordByte, /xFB => NonWordByte, /xFC => NonWordByte, /xFD => NonWordByte, /xFE => NonWordByte, /xFF => NonWordByte}, + classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]), + quitset: ByteSet { + bits: {}, + }, + cache_capacity: 10485760, + }, + reverse: DFA { + config: Config { + match_kind: Some( + All, + ), + pre: Some( + None, + ), + starts_for_each_pattern: Some( + true, + ), + byte_classes: Some( + true, + ), + unicode_word_boundary: Some( + true, + ), + quitset: None, + specialize_start_states: Some( + false, + ), + cache_capacity: Some( + 10485760, + ), + skip_cache_capacity_check: Some( + false, + ), + minimum_cache_clear_count: Some( + Some( + 3, + ), + ), + minimum_bytes_per_state: Some( + Some( + 10, + ), + ), + }, + nfa: thompson::NFA( + ^000000: FAIL + + transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) + ) + , + stride2: 1, + start_map: StartByteMap{/x00 => NonWordByte, /x01 => NonWordByte, /x02 => NonWordByte, /x03 => NonWordByte, /x04 => NonWordByte, /x05 => NonWordByte, /x06 => NonWordByte, /x07 => NonWordByte, /x08 => NonWordByte, /t => NonWordByte, + => LineLF, /x0B => NonWordByte, /x0C => NonWordByte, /r => LineCR, /x0E => NonWordByte, /x0F => NonWordByte, /x10 => NonWordByte, /x11 => NonWordByte, /x12 => NonWordByte, /x13 => NonWordByte, /x14 => NonWordByte, /x15 => NonWordByte, /x16 => NonWordByte, /x17 => NonWordByte, /x18 => NonWordByte, /x19 => NonWordByte, /x1A => NonWordByte, /x1B => NonWordByte, /x1C => NonWordByte, /x1D => NonWordByte, /x1E => NonWordByte, /x1F => NonWordByte, ' ' => NonWordByte, ! => NonWordByte, /" => NonWordByte, # => NonWordByte, $ => NonWordByte, % => NonWordByte, & => NonWordByte, /' => NonWordByte, ( => NonWordByte, ) => NonWordByte, * => NonWordByte, + => NonWordByte, , => NonWordByte, - => NonWordByte, . => NonWordByte, / => NonWordByte, 0 => WordByte, 1 => WordByte, 2 => WordByte, 3 => WordByte, 4 => WordByte, 5 => WordByte, 6 => WordByte, 7 => WordByte, 8 => WordByte, 9 => WordByte, : => NonWordByte, ; => NonWordByte, < => NonWordByte, = => NonWordByte, > => NonWordByte, ? => NonWordByte, @ => NonWordByte, A => WordByte, B => WordByte, C => WordByte, D => WordByte, E => WordByte, F => WordByte, G => WordByte, H => WordByte, I => WordByte, J => WordByte, K => WordByte, L => WordByte, M => WordByte, N => WordByte, O => WordByte, P => WordByte, Q => WordByte, R => WordByte, S => WordByte, T => WordByte, U => WordByte, V => WordByte, W => WordByte, X => WordByte, Y => WordByte, Z => WordByte, [ => NonWordByte, // => NonWordByte, ] => NonWordByte, ^ => NonWordByte, _ => WordByte, ` => NonWordByte, a => WordByte, b => WordByte, c => WordByte, d => WordByte, e => WordByte, f => WordByte, g => WordByte, h => WordByte, i => WordByte, j => WordByte, k => WordByte, l => WordByte, m => WordByte, n => WordByte, o => WordByte, p => WordByte, q => WordByte, r => WordByte, s => WordByte, t => WordByte, u => WordByte, v => WordByte, w => WordByte, x => WordByte, y => WordByte, z => WordByte, { => NonWordByte, | => NonWordByte, } => NonWordByte, ~ => NonWordByte, /x7F => NonWordByte, /x80 => NonWordByte, /x81 => NonWordByte, /x82 => NonWordByte, /x83 => NonWordByte, /x84 => NonWordByte, /x85 => NonWordByte, /x86 => NonWordByte, /x87 => NonWordByte, /x88 => NonWordByte, /x89 => NonWordByte, /x8A => NonWordByte, /x8B => NonWordByte, /x8C => NonWordByte, /x8D => NonWordByte, /x8E => NonWordByte, /x8F => NonWordByte, /x90 => NonWordByte, /x91 => NonWordByte, /x92 => NonWordByte, /x93 => NonWordByte, /x94 => NonWordByte, /x95 => NonWordByte, /x96 => NonWordByte, /x97 => NonWordByte, /x98 => NonWordByte, /x99 => NonWordByte, /x9A => NonWordByte, /x9B => NonWordByte, /x9C => NonWordByte, /x9D => NonWordByte, /x9E => NonWordByte, /x9F => NonWordByte, /xA0 => NonWordByte, /xA1 => NonWordByte, /xA2 => NonWordByte, /xA3 => NonWordByte, /xA4 => NonWordByte, /xA5 => NonWordByte, /xA6 => NonWordByte, /xA7 => NonWordByte, /xA8 => NonWordByte, /xA9 => NonWordByte, /xAA => NonWordByte, /xAB => NonWordByte, /xAC => NonWordByte, /xAD => NonWordByte, /xAE => NonWordByte, /xAF => NonWordByte, /xB0 => NonWordByte, /xB1 => NonWordByte, /xB2 => NonWordByte, /xB3 => NonWordByte, /xB4 => NonWordByte, /xB5 => NonWordByte, /xB6 => NonWordByte, /xB7 => NonWordByte, /xB8 => NonWordByte, /xB9 => NonWordByte, /xBA => NonWordByte, /xBB => NonWordByte, /xBC => NonWordByte, /xBD => NonWordByte, /xBE => NonWordByte, /xBF => NonWordByte, /xC0 => NonWordByte, /xC1 => NonWordByte, /xC2 => NonWordByte, /xC3 => NonWordByte, /xC4 => NonWordByte, /xC5 => NonWordByte, /xC6 => NonWordByte, /xC7 => NonWordByte, /xC8 => NonWordByte, /xC9 => NonWordByte, /xCA => NonWordByte, /xCB => NonWordByte, /xCC => NonWordByte, /xCD => NonWordByte, /xCE => NonWordByte, /xCF => NonWordByte, /xD0 => NonWordByte, /xD1 => NonWordByte, /xD2 => NonWordByte, /xD3 => NonWordByte, /xD4 => NonWordByte, /xD5 => NonWordByte, /xD6 => NonWordByte, /xD7 => NonWordByte, /xD8 => NonWordByte, /xD9 => NonWordByte, /xDA => NonWordByte, /xDB => NonWordByte, /xDC => NonWordByte, /xDD => NonWordByte, /xDE => NonWordByte, /xDF => NonWordByte, /xE0 => NonWordByte, /xE1 => NonWordByte, /xE2 => NonWordByte, /xE3 => NonWordByte, /xE4 => NonWordByte, /xE5 => NonWordByte, /xE6 => NonWordByte, /xE7 => NonWordByte, /xE8 => NonWordByte, /xE9 => NonWordByte, /xEA => NonWordByte, /xEB => NonWordByte, /xEC => NonWordByte, /xED => NonWordByte, /xEE => NonWordByte, /xEF => NonWordByte, /xF0 => NonWordByte, /xF1 => NonWordByte, /xF2 => NonWordByte, /xF3 => NonWordByte, /xF4 => NonWordByte, /xF5 => NonWordByte, /xF6 => NonWordByte, /xF7 => NonWordByte, /xF8 => NonWordByte, /xF9 => NonWordByte, /xFA => NonWordByte, /xFB => NonWordByte, /xFC => NonWordByte, /xFD => NonWordByte, /xFE => NonWordByte, /xFF => NonWordByte}, + classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]), + quitset: ByteSet { + bits: {}, + }, + cache_capacity: 10485760, + }, + }, + ), + ), + ), + dfa: DFA( + None, + ), + }, + info: RegexInfo( + RegexInfoI { + config: Config { + match_kind: Some( + All, + ), + utf8_empty: Some( + false, + ), + autopre: None, + pre: None, + which_captures: None, + nfa_size_limit: Some( + Some( + 10485760, + ), + ), + onepass_size_limit: None, + hybrid_cache_capacity: Some( + 10485760, + ), + hybrid: None, + dfa: None, + dfa_size_limit: None, + dfa_state_limit: None, + onepass: None, + backtrack: None, + byte_classes: None, + line_terminator: None, + }, + props: [], + props_union: Properties( + PropertiesI { + minimum_len: None, + maximum_len: None, + look_set: ∅, + look_set_prefix: ∅, + look_set_suffix: ∅, + look_set_prefix_any: ∅, + look_set_suffix_any: ∅, + utf8: true, + explicit_captures_len: 0, + static_explicit_captures_len: None, + literal: false, + alternation_literal: true, + }, + ), + }, + ), + }, + pool: Pool( + Pool { + stacks: [ + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + ], + owner: 0, + owner_val: UnsafeCell { .. }, + }, + ), + }, + map: [], + patset: Pool( + Pool { + stacks: [ + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + ], + owner: 0, + owner_val: UnsafeCell { .. }, + }, + ), + }, + ), + ], + }, + poisoned: false, + .. + }, + }, +} +[TRACE][pipeline/src/pipeline/mod.rs::305] pipeline_len: 1 +[TRACE][pipeline/src/pipeline/mod.rs::331] &dependency_graph: { + XvcEntity( + 2, + 12972606829502460557, + ): [], +} +[TRACE][pipeline/src/pipeline/mod.rs::343] &dependency_graph: { + XvcEntity( + 2, + 12972606829502460557, + ): [], +} [INFO][pipeline/src/pipeline/mod.rs::347] Pipeline Graph: digraph { - 0 [ label = "(2, 10877507082850142078)" ] + 0 [ label = "(2, 12972606829502460557)" ] } +[TRACE][pipeline/src/pipeline/mod.rs::412] step_states: RwLock { + data: HStore { + map: { + XvcEntity( + 2, + 12972606829502460557, + ): Begin( + FromInit, + ), + }, + }, + poisoned: false, + .. +} +[TRACE][pipeline/src/pipeline/mod.rs::508] &step_thread_store: HStore { + map: { + XvcEntity( + 2, + 12972606829502460557, + ): ScopedJoinHandle { .. }, + }, +} +[TRACE][pipeline/src/pipeline/mod.rs::512] (step_e, &jh): ( + XvcEntity( + 2, + 12972606829502460557, + ), + ScopedJoinHandle { .. }, +) +[TRACE][pipeline/src/pipeline/mod.rs::623] params.recorded_dependencies: R1NStore { + parents: XvcStore { + map: { + XvcEntity( + 2, + 12972606829502460557, + ): XvcStep { + name: "average-age", + }, + }, + entity_index: { + XvcStep { + name: "average-age", + }: [ + XvcEntity( + 2, + 12972606829502460557, + ), + ], + }, + previous: EventLog( + [ + Add { + entity: XvcEntity( + 2, + 12972606829502460557, + ), + value: XvcStep { + name: "average-age", + }, + }, + Add { + entity: XvcEntity( + 2, + 12972606829502460557, + ), + value: XvcStep { + name: "average-age", + }, + }, + ], + ), + current: EventLog( + [], + ), + }, + children: XvcStore { + map: {}, + entity_index: {}, + previous: EventLog( + [], + ), + current: EventLog( + [], + ), + }, + child_parents: XvcStore { + map: {}, + entity_index: {}, + previous: EventLog( + [], + ), + current: EventLog( + [], + ), + }, +} +[TRACE][pipeline/src/pipeline/mod.rs::624] step_e: XvcEntity( + 2, + 12972606829502460557, +) +[TRACE][pipeline/src/pipeline/mod.rs::564] dep_neighbors: Neighbors { + iter: Iter( + [], + ), + ty: PhantomData, +} +[TRACE][pipeline/src/pipeline/mod.rs::583] select: Select { .. } +[TRACE][pipeline/src/pipeline/mod.rs::625] dependency_steps(step_e, params.dependency_graph)?: {} +[TRACE][pipeline/src/pipeline/mod.rs::564] dep_neighbors: Neighbors { + iter: Iter( + [], + ), + ty: PhantomData, +} +[TRACE][/Users/iex/.cargo/registry/src/index.crates.io-6f17d22bba15001f/notify-6.1.1/src/fsevent.rs::536] FSEvent: path = `[CWD]/.xvc/store/xvc-pipeline-run-dir-store`, flag = StreamFlags(ITEM_CREATED | IS_DIR) +[TRACE][walker/src/notify.rs::56] event: Ok( + Event { + kind: Create( + Folder, + ), + paths: [ + "[CWD]/.xvc/store/xvc-pipeline-run-dir-store", + ], + attr:tracker: None, + attr:flag: None, + attr:info: None, + attr:source: None, + }, +) +[TRACE][walker/src/lib.rs::748] is_abs: true +[TRACE][walker/src/lib.rs::752] path_str: "[CWD]/.xvc/store/xvc-pipeline-run-dir-store" +[TRACE][walker/src/lib.rs::754] final_slash: false +[TRACE][walker/src/lib.rs::776] path: "/.xvc/store/xvc-pipeline-run-dir-store" +[TRACE][core/src/util/pmp.rs::99] index: 0 +[TRACE][core/src/util/pmp.rs::102] fs_event: Ok( + Some( + Create { + path: "[CWD]/.xvc/store/xvc-pipeline-run-dir-store", + metadata: Metadata { + file_type: FileType( + FileType { + mode: 16877, + }, + ), + is_dir: true, + is_file: false, + permissions: Permissions( + FilePermissions { + mode: 16877, + }, + ), + modified: Ok( + SystemTime { + tv_sec: 1722022739, + tv_nsec: 33553054, + }, + ), + accessed: Ok( + SystemTime { + tv_sec: 1722022739, + tv_nsec: 33553054, + }, + ), + created: Ok( + SystemTime { + tv_sec: 1722022739, + tv_nsec: 33553054, + }, + ), + .. + }, + }, + ), +) +[TRACE][core/src/types/xvcpath.rs::88] abs_path: "[CWD]/.xvc/store/xvc-pipeline-run-dir-store" +[TRACE][core/src/types/xvcpath.rs::89] current_dir: AbsolutePath( + "[CWD]", +) +[TRACE][core/src/types/xvcpath.rs::90] xvc_root.absolute_path(): AbsolutePath( + "[CWD]", +) +[TRACE][core/src/util/pmp.rs::65] "Creating {} with {}": "Creating {} with {}" +[TRACE][core/src/util/pmp.rs::65] xvc_path: XvcPath( + ".xvc/store/xvc-pipeline-run-dir-store", +) +[TRACE][core/src/util/pmp.rs::65] xvc_md: XvcMetadata { + file_type: Directory, + size: Some( + 64, + ), + modified: Some( + SystemTime { + tv_sec: 1722022739, + tv_nsec: 33553054, + }, + ), +} +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][pipeline/src/pipeline/mod.rs::662] &step_state: Begin( + FromInit, +) +[TRACE][pipeline/src/pipeline/mod.rs::772] step.name: "average-age" +[TRACE][pipeline/src/pipeline/mod.rs::773] &r_next_state: WaitingDependencySteps( + FromRunConditional, +) +[TRACE][pipeline/src/pipeline/mod.rs::583] select: Select { .. } +[TRACE][pipeline/src/pipeline/mod.rs::775] &step_state: WaitingDependencySteps( + FromRunConditional, +) +[TRACE][pipeline/src/pipeline/mod.rs::662] &step_state: WaitingDependencySteps( + FromRunConditional, +) +[TRACE][pipeline/src/pipeline/mod.rs::772] step.name: "average-age" +[TRACE][pipeline/src/pipeline/mod.rs::773] &r_next_state: CheckingOutputs( + FromDependencyStepsFinishedSuccessfully, +) +[TRACE][pipeline/src/pipeline/mod.rs::775] &step_state: CheckingOutputs( + FromDependencyStepsFinishedSuccessfully, +) [INFO] No dependency steps for step average-age +[TRACE][pipeline/src/pipeline/mod.rs::662] &step_state: CheckingOutputs( + FromDependencyStepsFinishedSuccessfully, +) +[TRACE][pipeline/src/pipeline/mod.rs::772] step.name: "average-age" +[TRACE][pipeline/src/pipeline/mod.rs::773] &r_next_state: CheckingSuperficialDiffs( + FromCheckedOutputs, +) +[TRACE][pipeline/src/pipeline/mod.rs::775] &step_state: CheckingSuperficialDiffs( + FromCheckedOutputs, +) +[TRACE][pipeline/src/pipeline/mod.rs::662] &step_state: CheckingSuperficialDiffs( + FromCheckedOutputs, +) +[TRACE][pipeline/src/pipeline/mod.rs::1010] parent_entity: XvcEntity( + 2, + 12972606829502460557, +) +[TRACE][pipeline/src/pipeline/mod.rs::583] select: Select { .. } +[TRACE][pipeline/src/pipeline/mod.rs::1013] deps: HStore { + map: {}, +} +[TRACE][pipeline/src/pipeline/mod.rs::583] select: Select { .. } +[TRACE][pipeline/src/pipeline/mod.rs::1016] params.step.name: "average-age" +[TRACE][pipeline/src/pipeline/mod.rs::583] select: Select { .. } +[TRACE][pipeline/src/pipeline/mod.rs::772] step.name: "average-age" +[TRACE][pipeline/src/pipeline/mod.rs::773] &r_next_state: CheckingThoroughDiffs( + FromSuperficialDiffsChanged, +) +[TRACE][pipeline/src/pipeline/mod.rs::775] &step_state: CheckingThoroughDiffs( + FromSuperficialDiffsChanged, +) +[TRACE][pipeline/src/pipeline/mod.rs::662] &step_state: CheckingThoroughDiffs( + FromSuperficialDiffsChanged, +) +[TRACE][pipeline/src/pipeline/mod.rs::1067] deps: HStore { + map: {}, +} +[TRACE][pipeline/src/pipeline/mod.rs::772] step.name: "average-age" +[TRACE][pipeline/src/pipeline/mod.rs::773] &r_next_state: ComparingDiffsAndOutputs( + FromThoroughDiffsChanged, +) +[TRACE][pipeline/src/pipeline/mod.rs::775] &step_state: ComparingDiffsAndOutputs( + FromThoroughDiffsChanged, +) +[TRACE][pipeline/src/pipeline/mod.rs::662] &step_state: ComparingDiffsAndOutputs( + FromThoroughDiffsChanged, +) +[TRACE][pipeline/src/pipeline/mod.rs::772] step.name: "average-age" +[TRACE][pipeline/src/pipeline/mod.rs::773] &r_next_state: WaitingToRun( + FromDiffsHasChanged, +) +[TRACE][pipeline/src/pipeline/mod.rs::775] &step_state: WaitingToRun( + FromDiffsHasChanged, +) +[TRACE][pipeline/src/pipeline/mod.rs::662] &step_state: WaitingToRun( + FromDiffsHasChanged, +) [INFO] [average-age] Dependencies has changed +[TRACE][pipeline/src/pipeline/mod.rs::583] select: Select { .. } +[TRACE][pipeline/src/pipeline/mod.rs::583] select: Select { .. } +[TRACE][pipeline/src/pipeline/mod.rs::583] select: Select { .. } +[TRACE][pipeline/src/pipeline/mod.rs::1541] params: StepStateParams { + xvc_root: XvcRootInner { + absolute_path: AbsolutePath( + "[CWD]", + ), + xvc_dir: AbsolutePath( + "[CWD]/.xvc", + ), + store_dir: AbsolutePath( + "[CWD]/.xvc/store", + ), + config: XvcConfig { + current_dir: XvcConfigOption { + source: Runtime, + option: AbsolutePath( + "[CWD]", + ), + }, + config_maps: [ + XvcConfigMap { + source: Default, + map: { + "file.list.sort": String( + "name-desc", + ), + "pipeline.process_pool_size": Integer( + 4, + ), + "git.command": String( + "git", + ), + "pipeline.current_pipeline": String( + "default", + ), + "file.carry-in.force": Boolean( + false, + ), + "pipeline.default_params_file": String( + "params.yaml", + ), + "pipeline.default": String( + "default", + ), + "core.guid": String( + "7726b84aa14cc82c", + ), + "git.auto_stage": Boolean( + false, + ), + "file.track.force": Boolean( + false, + ), + "cache.algorithm": String( + "blake3", + ), + "file.recheck.method": String( + "copy", + ), + "core.verbosity": String( + "error", + ), + "file.track.no_commit": Boolean( + false, + ), + "file.list.show_dot_files": Boolean( + false, + ), + "file.track.text_or_binary": String( + "auto", + ), + "file.list.recursive": Boolean( + false, + ), + "file.list.no_summary": Boolean( + false, + ), + "git.use_git": Boolean( + true, + ), + "file.carry-in.no_parallel": Boolean( + false, + ), + "file.list.format": String( + "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", + ), + "file.track.no_parallel": Boolean( + false, + ), + "git.auto_commit": Boolean( + true, + ), + }, + }, + XvcConfigMap { + source: Project, + map: { + "file.list.no_summary": Boolean( + false, + ), + "file.list.sort": String( + "name-desc", + ), + "git.auto_commit": Boolean( + true, + ), + "git.auto_stage": Boolean( + false, + ), + "file.track.no_parallel": Boolean( + false, + ), + "file.track.no_commit": Boolean( + false, + ), + "pipeline.process_pool_size": Integer( + 4, + ), + "pipeline.default_params_file": String( + "params.yaml", + ), + "pipeline.default": String( + "default", + ), + "git.command": String( + "git", + ), + "file.list.show_dot_files": Boolean( + false, + ), + "file.list.format": String( + "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", + ), + "core.guid": String( + "4bd7a8139a85fb10", + ), + "pipeline.current_pipeline": String( + "default", + ), + "file.track.text_or_binary": String( + "auto", + ), + "file.recheck.method": String( + "copy", + ), + "git.use_git": Boolean( + true, + ), + "file.list.recursive": Boolean( + false, + ), + "file.carry-in.no_parallel": Boolean( + false, + ), + "file.carry-in.force": Boolean( + false, + ), + "file.track.force": Boolean( + false, + ), + "core.verbosity": String( + "error", + ), + "cache.algorithm": String( + "blake3", + ), + }, + }, + XvcConfigMap { + source: Local, + map: {}, + }, + XvcConfigMap { + source: Environment, + map: { + "TRYCMD_TESTS": String( + "storage,file,pipeline,core", + ), + }, + }, + XvcConfigMap { + source: CommandLine, + map: { + "core.verbosity": String( + "debug", + ), + "core.quiet": Boolean( + false, + ), + }, + }, + ], + the_config: { + "pipeline.default_params_file": XvcConfigValue { + source: Project, + value: String( + "params.yaml", + ), + }, + "file.list.recursive": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "file.track.no_commit": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "file.carry-in.no_parallel": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "pipeline.current_pipeline": XvcConfigValue { + source: Project, + value: String( + "default", + ), + }, + "file.track.force": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "pipeline.default": XvcConfigValue { + source: Project, + value: String( + "default", + ), + }, + "cache.algorithm": XvcConfigValue { + source: Project, + value: String( + "blake3", + ), + }, + "file.track.text_or_binary": XvcConfigValue { + source: Project, + value: String( + "auto", + ), + }, + "TRYCMD_TESTS": XvcConfigValue { + source: Environment, + value: String( + "storage,file,pipeline,core", + ), + }, + "pipeline.process_pool_size": XvcConfigValue { + source: Project, + value: Integer( + 4, + ), + }, + "file.list.no_summary": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "core.guid": XvcConfigValue { + source: Project, + value: String( + "4bd7a8139a85fb10", + ), + }, + "file.list.format": XvcConfigValue { + source: Project, + value: String( + "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", + ), + }, + "file.carry-in.force": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "git.use_git": XvcConfigValue { + source: Project, + value: Boolean( + true, + ), + }, + "git.auto_stage": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "file.list.show_dot_files": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "file.track.no_parallel": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "git.auto_commit": XvcConfigValue { + source: Project, + value: Boolean( + true, + ), + }, + "file.list.sort": XvcConfigValue { + source: Project, + value: String( + "name-desc", + ), + }, + "file.recheck.method": XvcConfigValue { + source: Project, + value: String( + "copy", + ), + }, + "core.verbosity": XvcConfigValue { + source: CommandLine, + value: String( + "debug", + ), + }, + "git.command": XvcConfigValue { + source: Project, + value: String( + "git", + ), + }, + "core.quiet": XvcConfigValue { + source: CommandLine, + value: Boolean( + false, + ), + }, + }, + init_params: XvcConfigParams { + default_configuration: " +[core] +# The repository id. Please do not delete or change it. +# This is used to identify the repository and generate paths in storages. +# In the future it may be used to in other ways. +guid = /"7726b84aa14cc82c/" +# Default verbosity level. +# One of /"error/", /"warn/", /"info/" +verbosity = /"error/" + +[git] +# Automate git operations. +# Turning this off leads Xvc to behave as if it's not in a Git repository. +# Not recommended unless you're really not using Git +use_git = true +# Command to run Git process. +# You can set this to an absolute path to specify an executable +# If set to a non-absolute path, the executable will be searched in $PATH. +command = /"git/" + +# Commit changes in .xvc/ directory after commands. +# You can set this to false if you want to commit manually. +auto_commit = true + +# Stage changes in .xvc/ directory without committing. +# auto_commit implies auto_stage. +# If you want to commit manually but don't want to stage after individual Xvc commands, you can set this to true. +auto_stage = false + +[cache] +# The hash algorithm used for the cache. +# It may take blake3, blake2, sha2 or sha3 as values. +# All algorithms are selected to produce 256-bit hashes, so sha2 means SHA2-256, blake2 means BLAKE2s, etc. +# The cache path is produced by prepending algorithm name to the cache. +# Blake3 files are in .xvc/b3/, while sha2 files are in .xvc/s2/ etc. +algorithm = /"blake3/" + +[file] + +[file.track] + +# Don't move file content to cache after xvc file track +no_commit = false +# Force to track files even if they are already tracked. +force = false + +# Xvc calculates file content digest differently for text and binary files. +# This option controls whether to treat files as text or binary. +# It may take auto, text or binary as values. +# Auto check each file individually and treat it as text if it's text. +text_or_binary = /"auto/" + +# Don't use parallelism in track operations. +# Note that some of the operations are implemented in parallel by default, and this option affects some heavier operations. +no_parallel = false + +[file.list] + +# Format for `xvc file list` rows. You can reorder or remove columns. +# The following are the keys for each row: +# - {acd64}: actual content digest. All 64 digits from the workspace file's content. +# - {acd8}: actual content digest. First 8 digits the file content digest. +# - {aft}: actual file type. Whether the entry is a file (F), directory (D), +# symlink (S), hardlink (H) or reflink (R). +# - {asz}: actual size. The size of the workspace file in bytes. It uses MB, +# GB and TB to represent sizes larger than 1MB. +# - {ats}: actual timestamp. The timestamp of the workspace file. +# - {cst}: cache status. One of /"=/", /">/", /"/", /" 0 + F 000001: + >000002: /x00 => 2 + 000003: /x00 => 0 + match kind: Standard + prefilter: false + state length: 4 + pattern length: 0 + shortest pattern length: 18446744073709551615 + longest pattern length: 0 + alphabet length: 1 + stride: 1 + byte classes: ByteClasses(0 => [0-255]) + memory usage: 16 + ) + , + ), + map: [], + longest: 0, + }, + ), + Prefix( + PrefixStrategy { + matcher: AhoCorasick( + dfa::DFA( + D 000000: /x00 => 0 + F 000001: + >000002: /x00 => 2 + 000003: /x00 => 0 + match kind: Standard + prefilter: false + state length: 4 + pattern length: 0 + shortest pattern length: 18446744073709551615 + longest pattern length: 0 + alphabet length: 1 + stride: 1 + byte classes: ByteClasses(0 => [0-255]) + memory usage: 16 + ) + , + ), + map: [], + longest: 0, + }, + ), + RequiredExtension( + RequiredExtensionStrategy( + {}, + ), + ), + Regex( + RegexSetStrategy { + matcher: Regex { + imp: RegexI { + strat: Core { + info: RegexInfo( + RegexInfoI { + config: Config { + match_kind: Some( + All, + ), + utf8_empty: Some( + false, + ), + autopre: None, + pre: None, + which_captures: None, + nfa_size_limit: Some( + Some( + 10485760, + ), + ), + onepass_size_limit: None, + hybrid_cache_capacity: Some( + 10485760, + ), + hybrid: None, + dfa: None, + dfa_size_limit: None, + dfa_state_limit: None, + onepass: None, + backtrack: None, + byte_classes: None, + line_terminator: None, + }, + props: [], + props_union: Properties( + PropertiesI { + minimum_len: None, + maximum_len: None, + look_set: ∅, + look_set_prefix: ∅, + look_set_suffix: ∅, + look_set_prefix_any: ∅, + look_set_suffix_any: ∅, + utf8: true, + explicit_captures_len: 0, + static_explicit_captures_len: None, + literal: false, + alternation_literal: true, + }, + ), + }, + ), + pre: None, + nfa: thompson::NFA( + ^000000: FAIL + + transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) + ) + , + nfarev: Some( + thompson::NFA( + ^000000: FAIL + + transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) + ) + , + ), + pikevm: PikeVM( + PikeVMEngine( + PikeVM { + config: Config { + match_kind: Some( + All, + ), + pre: Some( + None, + ), + }, + nfa: thompson::NFA( + ^000000: FAIL + + transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) + ) + , + }, + ), + ), + backtrack: BoundedBacktracker( + None, + ), + onepass: OnePass( + None, + ), + hybrid: Hybrid( + Some( + HybridEngine( + Regex { + forward: DFA { + config: Config { + match_kind: Some( + All, + ), + pre: Some( + None, + ), + starts_for_each_pattern: Some( + true, + ), + byte_classes: Some( + true, + ), + unicode_word_boundary: Some( + true, + ), + quitset: None, + specialize_start_states: Some( + false, + ), + cache_capacity: Some( + 10485760, + ), + skip_cache_capacity_check: Some( + false, + ), + minimum_cache_clear_count: Some( + Some( + 3, + ), + ), + minimum_bytes_per_state: Some( + Some( + 10, + ), + ), + }, + nfa: thompson::NFA( + ^000000: FAIL + + transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) + ) + , + stride2: 1, + start_map: StartByteMap{/x00 => NonWordByte, /x01 => NonWordByte, /x02 => NonWordByte, /x03 => NonWordByte, /x04 => NonWordByte, /x05 => NonWordByte, /x06 => NonWordByte, /x07 => NonWordByte, /x08 => NonWordByte, /t => NonWordByte, + => LineLF, /x0B => NonWordByte, /x0C => NonWordByte, /r => LineCR, /x0E => NonWordByte, /x0F => NonWordByte, /x10 => NonWordByte, /x11 => NonWordByte, /x12 => NonWordByte, /x13 => NonWordByte, /x14 => NonWordByte, /x15 => NonWordByte, /x16 => NonWordByte, /x17 => NonWordByte, /x18 => NonWordByte, /x19 => NonWordByte, /x1A => NonWordByte, /x1B => NonWordByte, /x1C => NonWordByte, /x1D => NonWordByte, /x1E => NonWordByte, /x1F => NonWordByte, ' ' => NonWordByte, ! => NonWordByte, /" => NonWordByte, # => NonWordByte, $ => NonWordByte, % => NonWordByte, & => NonWordByte, /' => NonWordByte, ( => NonWordByte, ) => NonWordByte, * => NonWordByte, + => NonWordByte, , => NonWordByte, - => NonWordByte, . => NonWordByte, / => NonWordByte, 0 => WordByte, 1 => WordByte, 2 => WordByte, 3 => WordByte, 4 => WordByte, 5 => WordByte, 6 => WordByte, 7 => WordByte, 8 => WordByte, 9 => WordByte, : => NonWordByte, ; => NonWordByte, < => NonWordByte, = => NonWordByte, > => NonWordByte, ? => NonWordByte, @ => NonWordByte, A => WordByte, B => WordByte, C => WordByte, D => WordByte, E => WordByte, F => WordByte, G => WordByte, H => WordByte, I => WordByte, J => WordByte, K => WordByte, L => WordByte, M => WordByte, N => WordByte, O => WordByte, P => WordByte, Q => WordByte, R => WordByte, S => WordByte, T => WordByte, U => WordByte, V => WordByte, W => WordByte, X => WordByte, Y => WordByte, Z => WordByte, [ => NonWordByte, // => NonWordByte, ] => NonWordByte, ^ => NonWordByte, _ => WordByte, ` => NonWordByte, a => WordByte, b => WordByte, c => WordByte, d => WordByte, e => WordByte, f => WordByte, g => WordByte, h => WordByte, i => WordByte, j => WordByte, k => WordByte, l => WordByte, m => WordByte, n => WordByte, o => WordByte, p => WordByte, q => WordByte, r => WordByte, s => WordByte, t => WordByte, u => WordByte, v => WordByte, w => WordByte, x => WordByte, y => WordByte, z => WordByte, { => NonWordByte, | => NonWordByte, } => NonWordByte, ~ => NonWordByte, /x7F => NonWordByte, /x80 => NonWordByte, /x81 => NonWordByte, /x82 => NonWordByte, /x83 => NonWordByte, /x84 => NonWordByte, /x85 => NonWordByte, /x86 => NonWordByte, /x87 => NonWordByte, /x88 => NonWordByte, /x89 => NonWordByte, /x8A => NonWordByte, /x8B => NonWordByte, /x8C => NonWordByte, /x8D => NonWordByte, /x8E => NonWordByte, /x8F => NonWordByte, /x90 => NonWordByte, /x91 => NonWordByte, /x92 => NonWordByte, /x93 => NonWordByte, /x94 => NonWordByte, /x95 => NonWordByte, /x96 => NonWordByte, /x97 => NonWordByte, /x98 => NonWordByte, /x99 => NonWordByte, /x9A => NonWordByte, /x9B => NonWordByte, /x9C => NonWordByte, /x9D => NonWordByte, /x9E => NonWordByte, /x9F => NonWordByte, /xA0 => NonWordByte, /xA1 => NonWordByte, /xA2 => NonWordByte, /xA3 => NonWordByte, /xA4 => NonWordByte, /xA5 => NonWordByte, /xA6 => NonWordByte, /xA7 => NonWordByte, /xA8 => NonWordByte, /xA9 => NonWordByte, /xAA => NonWordByte, /xAB => NonWordByte, /xAC => NonWordByte, /xAD => NonWordByte, /xAE => NonWordByte, /xAF => NonWordByte, /xB0 => NonWordByte, /xB1 => NonWordByte, /xB2 => NonWordByte, /xB3 => NonWordByte, /xB4 => NonWordByte, /xB5 => NonWordByte, /xB6 => NonWordByte, /xB7 => NonWordByte, /xB8 => NonWordByte, /xB9 => NonWordByte, /xBA => NonWordByte, /xBB => NonWordByte, /xBC => NonWordByte, /xBD => NonWordByte, /xBE => NonWordByte, /xBF => NonWordByte, /xC0 => NonWordByte, /xC1 => NonWordByte, /xC2 => NonWordByte, /xC3 => NonWordByte, /xC4 => NonWordByte, /xC5 => NonWordByte, /xC6 => NonWordByte, /xC7 => NonWordByte, /xC8 => NonWordByte, /xC9 => NonWordByte, /xCA => NonWordByte, /xCB => NonWordByte, /xCC => NonWordByte, /xCD => NonWordByte, /xCE => NonWordByte, /xCF => NonWordByte, /xD0 => NonWordByte, /xD1 => NonWordByte, /xD2 => NonWordByte, /xD3 => NonWordByte, /xD4 => NonWordByte, /xD5 => NonWordByte, /xD6 => NonWordByte, /xD7 => NonWordByte, /xD8 => NonWordByte, /xD9 => NonWordByte, /xDA => NonWordByte, /xDB => NonWordByte, /xDC => NonWordByte, /xDD => NonWordByte, /xDE => NonWordByte, /xDF => NonWordByte, /xE0 => NonWordByte, /xE1 => NonWordByte, /xE2 => NonWordByte, /xE3 => NonWordByte, /xE4 => NonWordByte, /xE5 => NonWordByte, /xE6 => NonWordByte, /xE7 => NonWordByte, /xE8 => NonWordByte, /xE9 => NonWordByte, /xEA => NonWordByte, /xEB => NonWordByte, /xEC => NonWordByte, /xED => NonWordByte, /xEE => NonWordByte, /xEF => NonWordByte, /xF0 => NonWordByte, /xF1 => NonWordByte, /xF2 => NonWordByte, /xF3 => NonWordByte, /xF4 => NonWordByte, /xF5 => NonWordByte, /xF6 => NonWordByte, /xF7 => NonWordByte, /xF8 => NonWordByte, /xF9 => NonWordByte, /xFA => NonWordByte, /xFB => NonWordByte, /xFC => NonWordByte, /xFD => NonWordByte, /xFE => NonWordByte, /xFF => NonWordByte}, + classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]), + quitset: ByteSet { + bits: {}, + }, + cache_capacity: 10485760, + }, + reverse: DFA { + config: Config { + match_kind: Some( + All, + ), + pre: Some( + None, + ), + starts_for_each_pattern: Some( + true, + ), + byte_classes: Some( + true, + ), + unicode_word_boundary: Some( + true, + ), + quitset: None, + specialize_start_states: Some( + false, + ), + cache_capacity: Some( + 10485760, + ), + skip_cache_capacity_check: Some( + false, + ), + minimum_cache_clear_count: Some( + Some( + 3, + ), + ), + minimum_bytes_per_state: Some( + Some( + 10, + ), + ), + }, + nfa: thompson::NFA( + ^000000: FAIL + + transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) + ) + , + stride2: 1, + start_map: StartByteMap{/x00 => NonWordByte, /x01 => NonWordByte, /x02 => NonWordByte, /x03 => NonWordByte, /x04 => NonWordByte, /x05 => NonWordByte, /x06 => NonWordByte, /x07 => NonWordByte, /x08 => NonWordByte, /t => NonWordByte, + => LineLF, /x0B => NonWordByte, /x0C => NonWordByte, /r => LineCR, /x0E => NonWordByte, /x0F => NonWordByte, /x10 => NonWordByte, /x11 => NonWordByte, /x12 => NonWordByte, /x13 => NonWordByte, /x14 => NonWordByte, /x15 => NonWordByte, /x16 => NonWordByte, /x17 => NonWordByte, /x18 => NonWordByte, /x19 => NonWordByte, /x1A => NonWordByte, /x1B => NonWordByte, /x1C => NonWordByte, /x1D => NonWordByte, /x1E => NonWordByte, /x1F => NonWordByte, ' ' => NonWordByte, ! => NonWordByte, /" => NonWordByte, # => NonWordByte, $ => NonWordByte, % => NonWordByte, & => NonWordByte, /' => NonWordByte, ( => NonWordByte, ) => NonWordByte, * => NonWordByte, + => NonWordByte, , => NonWordByte, - => NonWordByte, . => NonWordByte, / => NonWordByte, 0 => WordByte, 1 => WordByte, 2 => WordByte, 3 => WordByte, 4 => WordByte, 5 => WordByte, 6 => WordByte, 7 => WordByte, 8 => WordByte, 9 => WordByte, : => NonWordByte, ; => NonWordByte, < => NonWordByte, = => NonWordByte, > => NonWordByte, ? => NonWordByte, @ => NonWordByte, A => WordByte, B => WordByte, C => WordByte, D => WordByte, E => WordByte, F => WordByte, G => WordByte, H => WordByte, I => WordByte, J => WordByte, K => WordByte, L => WordByte, M => WordByte, N => WordByte, O => WordByte, P => WordByte, Q => WordByte, R => WordByte, S => WordByte, T => WordByte, U => WordByte, V => WordByte, W => WordByte, X => WordByte, Y => WordByte, Z => WordByte, [ => NonWordByte, // => NonWordByte, ] => NonWordByte, ^ => NonWordByte, _ => WordByte, ` => NonWordByte, a => WordByte, b => WordByte, c => WordByte, d => WordByte, e => WordByte, f => WordByte, g => WordByte, h => WordByte, i => WordByte, j => WordByte, k => WordByte, l => WordByte, m => WordByte, n => WordByte, o => WordByte, p => WordByte, q => WordByte, r => WordByte, s => WordByte, t => WordByte, u => WordByte, v => WordByte, w => WordByte, x => WordByte, y => WordByte, z => WordByte, { => NonWordByte, | => NonWordByte, } => NonWordByte, ~ => NonWordByte, /x7F => NonWordByte, /x80 => NonWordByte, /x81 => NonWordByte, /x82 => NonWordByte, /x83 => NonWordByte, /x84 => NonWordByte, /x85 => NonWordByte, /x86 => NonWordByte, /x87 => NonWordByte, /x88 => NonWordByte, /x89 => NonWordByte, /x8A => NonWordByte, /x8B => NonWordByte, /x8C => NonWordByte, /x8D => NonWordByte, /x8E => NonWordByte, /x8F => NonWordByte, /x90 => NonWordByte, /x91 => NonWordByte, /x92 => NonWordByte, /x93 => NonWordByte, /x94 => NonWordByte, /x95 => NonWordByte, /x96 => NonWordByte, /x97 => NonWordByte, /x98 => NonWordByte, /x99 => NonWordByte, /x9A => NonWordByte, /x9B => NonWordByte, /x9C => NonWordByte, /x9D => NonWordByte, /x9E => NonWordByte, /x9F => NonWordByte, /xA0 => NonWordByte, /xA1 => NonWordByte, /xA2 => NonWordByte, /xA3 => NonWordByte, /xA4 => NonWordByte, /xA5 => NonWordByte, /xA6 => NonWordByte, /xA7 => NonWordByte, /xA8 => NonWordByte, /xA9 => NonWordByte, /xAA => NonWordByte, /xAB => NonWordByte, /xAC => NonWordByte, /xAD => NonWordByte, /xAE => NonWordByte, /xAF => NonWordByte, /xB0 => NonWordByte, /xB1 => NonWordByte, /xB2 => NonWordByte, /xB3 => NonWordByte, /xB4 => NonWordByte, /xB5 => NonWordByte, /xB6 => NonWordByte, /xB7 => NonWordByte, /xB8 => NonWordByte, /xB9 => NonWordByte, /xBA => NonWordByte, /xBB => NonWordByte, /xBC => NonWordByte, /xBD => NonWordByte, /xBE => NonWordByte, /xBF => NonWordByte, /xC0 => NonWordByte, /xC1 => NonWordByte, /xC2 => NonWordByte, /xC3 => NonWordByte, /xC4 => NonWordByte, /xC5 => NonWordByte, /xC6 => NonWordByte, /xC7 => NonWordByte, /xC8 => NonWordByte, /xC9 => NonWordByte, /xCA => NonWordByte, /xCB => NonWordByte, /xCC => NonWordByte, /xCD => NonWordByte, /xCE => NonWordByte, /xCF => NonWordByte, /xD0 => NonWordByte, /xD1 => NonWordByte, /xD2 => NonWordByte, /xD3 => NonWordByte, /xD4 => NonWordByte, /xD5 => NonWordByte, /xD6 => NonWordByte, /xD7 => NonWordByte, /xD8 => NonWordByte, /xD9 => NonWordByte, /xDA => NonWordByte, /xDB => NonWordByte, /xDC => NonWordByte, /xDD => NonWordByte, /xDE => NonWordByte, /xDF => NonWordByte, /xE0 => NonWordByte, /xE1 => NonWordByte, /xE2 => NonWordByte, /xE3 => NonWordByte, /xE4 => NonWordByte, /xE5 => NonWordByte, /xE6 => NonWordByte, /xE7 => NonWordByte, /xE8 => NonWordByte, /xE9 => NonWordByte, /xEA => NonWordByte, /xEB => NonWordByte, /xEC => NonWordByte, /xED => NonWordByte, /xEE => NonWordByte, /xEF => NonWordByte, /xF0 => NonWordByte, /xF1 => NonWordByte, /xF2 => NonWordByte, /xF3 => NonWordByte, /xF4 => NonWordByte, /xF5 => NonWordByte, /xF6 => NonWordByte, /xF7 => NonWordByte, /xF8 => NonWordByte, /xF9 => NonWordByte, /xFA => NonWordByte, /xFB => NonWordByte, /xFC => NonWordByte, /xFD => NonWordByte, /xFE => NonWordByte, /xFF => NonWordByte}, + classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]), + quitset: ByteSet { + bits: {}, + }, + cache_capacity: 10485760, + }, + }, + ), + ), + ), + dfa: DFA( + None, + ), + }, + info: RegexInfo( + RegexInfoI { + config: Config { + match_kind: Some( + All, + ), + utf8_empty: Some( + false, + ), + autopre: None, + pre: None, + which_captures: None, + nfa_size_limit: Some( + Some( + 10485760, + ), + ), + onepass_size_limit: None, + hybrid_cache_capacity: Some( + 10485760, + ), + hybrid: None, + dfa: None, + dfa_size_limit: None, + dfa_state_limit: None, + onepass: None, + backtrack: None, + byte_classes: None, + line_terminator: None, + }, + props: [], + props_union: Properties( + PropertiesI { + minimum_len: None, + maximum_len: None, + look_set: ∅, + look_set_prefix: ∅, + look_set_suffix: ∅, + look_set_prefix_any: ∅, + look_set_suffix_any: ∅, + utf8: true, + explicit_captures_len: 0, + static_explicit_captures_len: None, + literal: false, + alternation_literal: true, + }, + ), + }, + ), + }, + pool: Pool( + Pool { + stacks: [ + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + ], + owner: 4, + owner_val: UnsafeCell { .. }, + }, + ), + }, + map: [], + patset: Pool( + Pool { + stacks: [ + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + ], + owner: 0, + owner_val: UnsafeCell { .. }, + }, + ), + }, + ), + ], + }, + poisoned: false, + .. + }, + }, + }, + run_conditions: RunConditions { + never: false, + always: true, + ignore_broken_dep_steps: true, + ignore_missing_outputs: true, + }, + pipeline_rundir: XvcPath( + "", + ), + terminate_timeout_processes: true, + algorithm: Blake3, + command_process: RwLock { + data: CommandProcess { + environment: {}, + step: XvcStep { + name: "average-age", + }, + step_command: XvcStepCommand { + command: "sqlite3 people.db /"SELECT AVG(Age) FROM People;/"", + }, + birth: None, + process: None, + stdout_sender: Sender { .. }, + stderr_sender: Sender { .. }, + stdout_receiver: Receiver { .. }, + stderr_receiver: Receiver { .. }, + }, + poisoned: false, + .. + }, + available_process_slots: RwLock { + data: 4, + poisoned: false, + .. + }, + process_poll_milliseconds: 10, + dependency_diffs: RwLock { + data: HStore { + map: {}, + }, + poisoned: false, + .. + }, + output_diffs: RwLock { + data: HStore { + map: {}, + }, + poisoned: false, + .. + }, + step_e: XvcEntity( + 2, + 12972606829502460557, + ), + step: XvcStep { + name: "average-age", + }, + step_command: XvcStepCommand { + command: "sqlite3 people.db /"SELECT AVG(Age) FROM People;/"", + }, + current_states: RwLock { + data: HStore { + map: { + XvcEntity( + 2, + 12972606829502460557, + ): WaitingToRun( + FromDiffsHasChanged, + ), + }, + }, + poisoned: false, + .. + }, + step_timeout: 10000s, + all_steps: HStore { + map: { + XvcEntity( + 2, + 12972606829502460557, + ): XvcStep { + name: "average-age", + }, + }, + }, + recorded_dependencies: R1NStore { + parents: XvcStore { + map: { + XvcEntity( + 2, + 12972606829502460557, + ): XvcStep { + name: "average-age", + }, + }, + entity_index: { + XvcStep { + name: "average-age", + }: [ + XvcEntity( + 2, + 12972606829502460557, + ), + ], + }, + previous: EventLog( + [ + Add { + entity: XvcEntity( + 2, + 12972606829502460557, + ), + value: XvcStep { + name: "average-age", + }, + }, + Add { + entity: XvcEntity( + 2, + 12972606829502460557, + ), + value: XvcStep { + name: "average-age", + }, + }, + ], + ), + current: EventLog( + [], + ), + }, + children: XvcStore { + map: {}, + entity_index: {}, + previous: EventLog( + [], + ), + current: EventLog( + [], + ), + }, + child_parents: XvcStore { + map: {}, + entity_index: {}, + previous: EventLog( + [], + ), + current: EventLog( + [], + ), + }, + }, + step_dependencies: {}, + step_outputs: HStore { + map: {}, + }, +} +[TRACE][pipeline/src/pipeline/mod.rs::772] step.name: "average-age" +[TRACE][pipeline/src/pipeline/mod.rs::773] &r_next_state: Running( + FromStartProcess, +) +[TRACE][pipeline/src/pipeline/mod.rs::775] &step_state: Running( + FromStartProcess, +) +[TRACE][pipeline/src/pipeline/mod.rs::662] &step_state: Running( + FromStartProcess, +) +[TRACE][pipeline/src/pipeline/command.rs::96] self.environment: {} +[TRACE][pipeline/src/pipeline/mod.rs::583] select: Select { .. } +[TRACE][pipeline/src/pipeline/mod.rs::772] step.name: "average-age" +[TRACE][pipeline/src/pipeline/mod.rs::773] &r_next_state: Running( + FromWaitProcess, +) +[TRACE][pipeline/src/pipeline/mod.rs::775] &step_state: Running( + FromWaitProcess, +) +[TRACE][pipeline/src/pipeline/mod.rs::662] &step_state: Running( + FromWaitProcess, +) +[TRACE][pipeline/src/pipeline/mod.rs::583] select: Select { .. } +[TRACE][pipeline/src/pipeline/mod.rs::1420] params: StepStateParams { + xvc_root: XvcRootInner { + absolute_path: AbsolutePath( + "[CWD]", + ), + xvc_dir: AbsolutePath( + "[CWD]/.xvc", + ), + store_dir: AbsolutePath( + "[CWD]/.xvc/store", + ), + config: XvcConfig { + current_dir: XvcConfigOption { + source: Runtime, + option: AbsolutePath( + "[CWD]", + ), + }, + config_maps: [ + XvcConfigMap { + source: Default, + map: { + "file.list.sort": String( + "name-desc", + ), + "pipeline.process_pool_size": Integer( + 4, + ), + "git.command": String( + "git", + ), + "pipeline.current_pipeline": String( + "default", + ), + "file.carry-in.force": Boolean( + false, + ), + "pipeline.default_params_file": String( + "params.yaml", + ), + "pipeline.default": String( + "default", + ), + "core.guid": String( + "7726b84aa14cc82c", + ), + "git.auto_stage": Boolean( + false, + ), + "file.track.force": Boolean( + false, + ), + "cache.algorithm": String( + "blake3", + ), + "file.recheck.method": String( + "copy", + ), + "core.verbosity": String( + "error", + ), + "file.track.no_commit": Boolean( + false, + ), + "file.list.show_dot_files": Boolean( + false, + ), + "file.track.text_or_binary": String( + "auto", + ), + "file.list.recursive": Boolean( + false, + ), + "file.list.no_summary": Boolean( + false, + ), + "git.use_git": Boolean( + true, + ), + "file.carry-in.no_parallel": Boolean( + false, + ), + "file.list.format": String( + "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", + ), + "file.track.no_parallel": Boolean( + false, + ), + "git.auto_commit": Boolean( + true, + ), + }, + }, + XvcConfigMap { + source: Project, + map: { + "file.list.no_summary": Boolean( + false, + ), + "file.list.sort": String( + "name-desc", + ), + "git.auto_commit": Boolean( + true, + ), + "git.auto_stage": Boolean( + false, + ), + "file.track.no_parallel": Boolean( + false, + ), + "file.track.no_commit": Boolean( + false, + ), + "pipeline.process_pool_size": Integer( + 4, + ), + "pipeline.default_params_file": String( + "params.yaml", + ), + "pipeline.default": String( + "default", + ), + "git.command": String( + "git", + ), + "file.list.show_dot_files": Boolean( + false, + ), + "file.list.format": String( + "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", + ), + "core.guid": String( + "4bd7a8139a85fb10", + ), + "pipeline.current_pipeline": String( + "default", + ), + "file.track.text_or_binary": String( + "auto", + ), + "file.recheck.method": String( + "copy", + ), + "git.use_git": Boolean( + true, + ), + "file.list.recursive": Boolean( + false, + ), + "file.carry-in.no_parallel": Boolean( + false, + ), + "file.carry-in.force": Boolean( + false, + ), + "file.track.force": Boolean( + false, + ), + "core.verbosity": String( + "error", + ), + "cache.algorithm": String( + "blake3", + ), + }, + }, + XvcConfigMap { + source: Local, + map: {}, + }, + XvcConfigMap { + source: Environment, + map: { + "TRYCMD_TESTS": String( + "storage,file,pipeline,core", + ), + }, + }, + XvcConfigMap { + source: CommandLine, + map: { + "core.verbosity": String( + "debug", + ), + "core.quiet": Boolean( + false, + ), + }, + }, + ], + the_config: { + "pipeline.default_params_file": XvcConfigValue { + source: Project, + value: String( + "params.yaml", + ), + }, + "file.list.recursive": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "file.track.no_commit": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "file.carry-in.no_parallel": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "pipeline.current_pipeline": XvcConfigValue { + source: Project, + value: String( + "default", + ), + }, + "file.track.force": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "pipeline.default": XvcConfigValue { + source: Project, + value: String( + "default", + ), + }, + "cache.algorithm": XvcConfigValue { + source: Project, + value: String( + "blake3", + ), + }, + "file.track.text_or_binary": XvcConfigValue { + source: Project, + value: String( + "auto", + ), + }, + "TRYCMD_TESTS": XvcConfigValue { + source: Environment, + value: String( + "storage,file,pipeline,core", + ), + }, + "pipeline.process_pool_size": XvcConfigValue { + source: Project, + value: Integer( + 4, + ), + }, + "file.list.no_summary": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "core.guid": XvcConfigValue { + source: Project, + value: String( + "4bd7a8139a85fb10", + ), + }, + "file.list.format": XvcConfigValue { + source: Project, + value: String( + "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", + ), + }, + "file.carry-in.force": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "git.use_git": XvcConfigValue { + source: Project, + value: Boolean( + true, + ), + }, + "git.auto_stage": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "file.list.show_dot_files": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "file.track.no_parallel": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "git.auto_commit": XvcConfigValue { + source: Project, + value: Boolean( + true, + ), + }, + "file.list.sort": XvcConfigValue { + source: Project, + value: String( + "name-desc", + ), + }, + "file.recheck.method": XvcConfigValue { + source: Project, + value: String( + "copy", + ), + }, + "core.verbosity": XvcConfigValue { + source: CommandLine, + value: String( + "debug", + ), + }, + "git.command": XvcConfigValue { + source: Project, + value: String( + "git", + ), + }, + "core.quiet": XvcConfigValue { + source: CommandLine, + value: Boolean( + false, + ), + }, + }, + init_params: XvcConfigParams { + default_configuration: " +[core] +# The repository id. Please do not delete or change it. +# This is used to identify the repository and generate paths in storages. +# In the future it may be used to in other ways. +guid = /"7726b84aa14cc82c/" +# Default verbosity level. +# One of /"error/", /"warn/", /"info/" +verbosity = /"error/" + +[git] +# Automate git operations. +# Turning this off leads Xvc to behave as if it's not in a Git repository. +# Not recommended unless you're really not using Git +use_git = true +# Command to run Git process. +# You can set this to an absolute path to specify an executable +# If set to a non-absolute path, the executable will be searched in $PATH. +command = /"git/" + +# Commit changes in .xvc/ directory after commands. +# You can set this to false if you want to commit manually. +auto_commit = true + +# Stage changes in .xvc/ directory without committing. +# auto_commit implies auto_stage. +# If you want to commit manually but don't want to stage after individual Xvc commands, you can set this to true. +auto_stage = false + +[cache] +# The hash algorithm used for the cache. +# It may take blake3, blake2, sha2 or sha3 as values. +# All algorithms are selected to produce 256-bit hashes, so sha2 means SHA2-256, blake2 means BLAKE2s, etc. +# The cache path is produced by prepending algorithm name to the cache. +# Blake3 files are in .xvc/b3/, while sha2 files are in .xvc/s2/ etc. +algorithm = /"blake3/" + +[file] + +[file.track] + +# Don't move file content to cache after xvc file track +no_commit = false +# Force to track files even if they are already tracked. +force = false + +# Xvc calculates file content digest differently for text and binary files. +# This option controls whether to treat files as text or binary. +# It may take auto, text or binary as values. +# Auto check each file individually and treat it as text if it's text. +text_or_binary = /"auto/" + +# Don't use parallelism in track operations. +# Note that some of the operations are implemented in parallel by default, and this option affects some heavier operations. +no_parallel = false + +[file.list] + +# Format for `xvc file list` rows. You can reorder or remove columns. +# The following are the keys for each row: +# - {acd64}: actual content digest. All 64 digits from the workspace file's content. +# - {acd8}: actual content digest. First 8 digits the file content digest. +# - {aft}: actual file type. Whether the entry is a file (F), directory (D), +# symlink (S), hardlink (H) or reflink (R). +# - {asz}: actual size. The size of the workspace file in bytes. It uses MB, +# GB and TB to represent sizes larger than 1MB. +# - {ats}: actual timestamp. The timestamp of the workspace file. +# - {cst}: cache status. One of /"=/", /">/", /"/", /" 0 + F 000001: + >000002: /x00 => 2 + 000003: /x00 => 0 + match kind: Standard + prefilter: false + state length: 4 + pattern length: 0 + shortest pattern length: 18446744073709551615 + longest pattern length: 0 + alphabet length: 1 + stride: 1 + byte classes: ByteClasses(0 => [0-255]) + memory usage: 16 + ) + , + ), + map: [], + longest: 0, + }, + ), + Prefix( + PrefixStrategy { + matcher: AhoCorasick( + dfa::DFA( + D 000000: /x00 => 0 + F 000001: + >000002: /x00 => 2 + 000003: /x00 => 0 + match kind: Standard + prefilter: false + state length: 4 + pattern length: 0 + shortest pattern length: 18446744073709551615 + longest pattern length: 0 + alphabet length: 1 + stride: 1 + byte classes: ByteClasses(0 => [0-255]) + memory usage: 16 + ) + , + ), + map: [], + longest: 0, + }, + ), + RequiredExtension( + RequiredExtensionStrategy( + {}, + ), + ), + Regex( + RegexSetStrategy { + matcher: Regex { + imp: RegexI { + strat: Core { + info: RegexInfo( + RegexInfoI { + config: Config { + match_kind: Some( + All, + ), + utf8_empty: Some( + false, + ), + autopre: None, + pre: None, + which_captures: None, + nfa_size_limit: Some( + Some( + 10485760, + ), + ), + onepass_size_limit: None, + hybrid_cache_capacity: Some( + 10485760, + ), + hybrid: None, + dfa: None, + dfa_size_limit: None, + dfa_state_limit: None, + onepass: None, + backtrack: None, + byte_classes: None, + line_terminator: None, + }, + props: [], + props_union: Properties( + PropertiesI { + minimum_len: None, + maximum_len: None, + look_set: ∅, + look_set_prefix: ∅, + look_set_suffix: ∅, + look_set_prefix_any: ∅, + look_set_suffix_any: ∅, + utf8: true, + explicit_captures_len: 0, + static_explicit_captures_len: None, + literal: false, + alternation_literal: true, + }, + ), + }, + ), + pre: None, + nfa: thompson::NFA( + ^000000: FAIL + + transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) + ) + , + nfarev: Some( + thompson::NFA( + ^000000: FAIL + + transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) + ) + , + ), + pikevm: PikeVM( + PikeVMEngine( + PikeVM { + config: Config { + match_kind: Some( + All, + ), + pre: Some( + None, + ), + }, + nfa: thompson::NFA( + ^000000: FAIL + + transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) + ) + , + }, + ), + ), + backtrack: BoundedBacktracker( + None, + ), + onepass: OnePass( + None, + ), + hybrid: Hybrid( + Some( + HybridEngine( + Regex { + forward: DFA { + config: Config { + match_kind: Some( + All, + ), + pre: Some( + None, + ), + starts_for_each_pattern: Some( + true, + ), + byte_classes: Some( + true, + ), + unicode_word_boundary: Some( + true, + ), + quitset: None, + specialize_start_states: Some( + false, + ), + cache_capacity: Some( + 10485760, + ), + skip_cache_capacity_check: Some( + false, + ), + minimum_cache_clear_count: Some( + Some( + 3, + ), + ), + minimum_bytes_per_state: Some( + Some( + 10, + ), + ), + }, + nfa: thompson::NFA( + ^000000: FAIL + + transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) + ) + , + stride2: 1, + start_map: StartByteMap{/x00 => NonWordByte, /x01 => NonWordByte, /x02 => NonWordByte, /x03 => NonWordByte, /x04 => NonWordByte, /x05 => NonWordByte, /x06 => NonWordByte, /x07 => NonWordByte, /x08 => NonWordByte, /t => NonWordByte, + => LineLF, /x0B => NonWordByte, /x0C => NonWordByte, /r => LineCR, /x0E => NonWordByte, /x0F => NonWordByte, /x10 => NonWordByte, /x11 => NonWordByte, /x12 => NonWordByte, /x13 => NonWordByte, /x14 => NonWordByte, /x15 => NonWordByte, /x16 => NonWordByte, /x17 => NonWordByte, /x18 => NonWordByte, /x19 => NonWordByte, /x1A => NonWordByte, /x1B => NonWordByte, /x1C => NonWordByte, /x1D => NonWordByte, /x1E => NonWordByte, /x1F => NonWordByte, ' ' => NonWordByte, ! => NonWordByte, /" => NonWordByte, # => NonWordByte, $ => NonWordByte, % => NonWordByte, & => NonWordByte, /' => NonWordByte, ( => NonWordByte, ) => NonWordByte, * => NonWordByte, + => NonWordByte, , => NonWordByte, - => NonWordByte, . => NonWordByte, / => NonWordByte, 0 => WordByte, 1 => WordByte, 2 => WordByte, 3 => WordByte, 4 => WordByte, 5 => WordByte, 6 => WordByte, 7 => WordByte, 8 => WordByte, 9 => WordByte, : => NonWordByte, ; => NonWordByte, < => NonWordByte, = => NonWordByte, > => NonWordByte, ? => NonWordByte, @ => NonWordByte, A => WordByte, B => WordByte, C => WordByte, D => WordByte, E => WordByte, F => WordByte, G => WordByte, H => WordByte, I => WordByte, J => WordByte, K => WordByte, L => WordByte, M => WordByte, N => WordByte, O => WordByte, P => WordByte, Q => WordByte, R => WordByte, S => WordByte, T => WordByte, U => WordByte, V => WordByte, W => WordByte, X => WordByte, Y => WordByte, Z => WordByte, [ => NonWordByte, // => NonWordByte, ] => NonWordByte, ^ => NonWordByte, _ => WordByte, ` => NonWordByte, a => WordByte, b => WordByte, c => WordByte, d => WordByte, e => WordByte, f => WordByte, g => WordByte, h => WordByte, i => WordByte, j => WordByte, k => WordByte, l => WordByte, m => WordByte, n => WordByte, o => WordByte, p => WordByte, q => WordByte, r => WordByte, s => WordByte, t => WordByte, u => WordByte, v => WordByte, w => WordByte, x => WordByte, y => WordByte, z => WordByte, { => NonWordByte, | => NonWordByte, } => NonWordByte, ~ => NonWordByte, /x7F => NonWordByte, /x80 => NonWordByte, /x81 => NonWordByte, /x82 => NonWordByte, /x83 => NonWordByte, /x84 => NonWordByte, /x85 => NonWordByte, /x86 => NonWordByte, /x87 => NonWordByte, /x88 => NonWordByte, /x89 => NonWordByte, /x8A => NonWordByte, /x8B => NonWordByte, /x8C => NonWordByte, /x8D => NonWordByte, /x8E => NonWordByte, /x8F => NonWordByte, /x90 => NonWordByte, /x91 => NonWordByte, /x92 => NonWordByte, /x93 => NonWordByte, /x94 => NonWordByte, /x95 => NonWordByte, /x96 => NonWordByte, /x97 => NonWordByte, /x98 => NonWordByte, /x99 => NonWordByte, /x9A => NonWordByte, /x9B => NonWordByte, /x9C => NonWordByte, /x9D => NonWordByte, /x9E => NonWordByte, /x9F => NonWordByte, /xA0 => NonWordByte, /xA1 => NonWordByte, /xA2 => NonWordByte, /xA3 => NonWordByte, /xA4 => NonWordByte, /xA5 => NonWordByte, /xA6 => NonWordByte, /xA7 => NonWordByte, /xA8 => NonWordByte, /xA9 => NonWordByte, /xAA => NonWordByte, /xAB => NonWordByte, /xAC => NonWordByte, /xAD => NonWordByte, /xAE => NonWordByte, /xAF => NonWordByte, /xB0 => NonWordByte, /xB1 => NonWordByte, /xB2 => NonWordByte, /xB3 => NonWordByte, /xB4 => NonWordByte, /xB5 => NonWordByte, /xB6 => NonWordByte, /xB7 => NonWordByte, /xB8 => NonWordByte, /xB9 => NonWordByte, /xBA => NonWordByte, /xBB => NonWordByte, /xBC => NonWordByte, /xBD => NonWordByte, /xBE => NonWordByte, /xBF => NonWordByte, /xC0 => NonWordByte, /xC1 => NonWordByte, /xC2 => NonWordByte, /xC3 => NonWordByte, /xC4 => NonWordByte, /xC5 => NonWordByte, /xC6 => NonWordByte, /xC7 => NonWordByte, /xC8 => NonWordByte, /xC9 => NonWordByte, /xCA => NonWordByte, /xCB => NonWordByte, /xCC => NonWordByte, /xCD => NonWordByte, /xCE => NonWordByte, /xCF => NonWordByte, /xD0 => NonWordByte, /xD1 => NonWordByte, /xD2 => NonWordByte, /xD3 => NonWordByte, /xD4 => NonWordByte, /xD5 => NonWordByte, /xD6 => NonWordByte, /xD7 => NonWordByte, /xD8 => NonWordByte, /xD9 => NonWordByte, /xDA => NonWordByte, /xDB => NonWordByte, /xDC => NonWordByte, /xDD => NonWordByte, /xDE => NonWordByte, /xDF => NonWordByte, /xE0 => NonWordByte, /xE1 => NonWordByte, /xE2 => NonWordByte, /xE3 => NonWordByte, /xE4 => NonWordByte, /xE5 => NonWordByte, /xE6 => NonWordByte, /xE7 => NonWordByte, /xE8 => NonWordByte, /xE9 => NonWordByte, /xEA => NonWordByte, /xEB => NonWordByte, /xEC => NonWordByte, /xED => NonWordByte, /xEE => NonWordByte, /xEF => NonWordByte, /xF0 => NonWordByte, /xF1 => NonWordByte, /xF2 => NonWordByte, /xF3 => NonWordByte, /xF4 => NonWordByte, /xF5 => NonWordByte, /xF6 => NonWordByte, /xF7 => NonWordByte, /xF8 => NonWordByte, /xF9 => NonWordByte, /xFA => NonWordByte, /xFB => NonWordByte, /xFC => NonWordByte, /xFD => NonWordByte, /xFE => NonWordByte, /xFF => NonWordByte}, + classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]), + quitset: ByteSet { + bits: {}, + }, + cache_capacity: 10485760, + }, + reverse: DFA { + config: Config { + match_kind: Some( + All, + ), + pre: Some( + None, + ), + starts_for_each_pattern: Some( + true, + ), + byte_classes: Some( + true, + ), + unicode_word_boundary: Some( + true, + ), + quitset: None, + specialize_start_states: Some( + false, + ), + cache_capacity: Some( + 10485760, + ), + skip_cache_capacity_check: Some( + false, + ), + minimum_cache_clear_count: Some( + Some( + 3, + ), + ), + minimum_bytes_per_state: Some( + Some( + 10, + ), + ), + }, + nfa: thompson::NFA( + ^000000: FAIL + + transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) + ) + , + stride2: 1, + start_map: StartByteMap{/x00 => NonWordByte, /x01 => NonWordByte, /x02 => NonWordByte, /x03 => NonWordByte, /x04 => NonWordByte, /x05 => NonWordByte, /x06 => NonWordByte, /x07 => NonWordByte, /x08 => NonWordByte, /t => NonWordByte, + => LineLF, /x0B => NonWordByte, /x0C => NonWordByte, /r => LineCR, /x0E => NonWordByte, /x0F => NonWordByte, /x10 => NonWordByte, /x11 => NonWordByte, /x12 => NonWordByte, /x13 => NonWordByte, /x14 => NonWordByte, /x15 => NonWordByte, /x16 => NonWordByte, /x17 => NonWordByte, /x18 => NonWordByte, /x19 => NonWordByte, /x1A => NonWordByte, /x1B => NonWordByte, /x1C => NonWordByte, /x1D => NonWordByte, /x1E => NonWordByte, /x1F => NonWordByte, ' ' => NonWordByte, ! => NonWordByte, /" => NonWordByte, # => NonWordByte, $ => NonWordByte, % => NonWordByte, & => NonWordByte, /' => NonWordByte, ( => NonWordByte, ) => NonWordByte, * => NonWordByte, + => NonWordByte, , => NonWordByte, - => NonWordByte, . => NonWordByte, / => NonWordByte, 0 => WordByte, 1 => WordByte, 2 => WordByte, 3 => WordByte, 4 => WordByte, 5 => WordByte, 6 => WordByte, 7 => WordByte, 8 => WordByte, 9 => WordByte, : => NonWordByte, ; => NonWordByte, < => NonWordByte, = => NonWordByte, > => NonWordByte, ? => NonWordByte, @ => NonWordByte, A => WordByte, B => WordByte, C => WordByte, D => WordByte, E => WordByte, F => WordByte, G => WordByte, H => WordByte, I => WordByte, J => WordByte, K => WordByte, L => WordByte, M => WordByte, N => WordByte, O => WordByte, P => WordByte, Q => WordByte, R => WordByte, S => WordByte, T => WordByte, U => WordByte, V => WordByte, W => WordByte, X => WordByte, Y => WordByte, Z => WordByte, [ => NonWordByte, // => NonWordByte, ] => NonWordByte, ^ => NonWordByte, _ => WordByte, ` => NonWordByte, a => WordByte, b => WordByte, c => WordByte, d => WordByte, e => WordByte, f => WordByte, g => WordByte, h => WordByte, i => WordByte, j => WordByte, k => WordByte, l => WordByte, m => WordByte, n => WordByte, o => WordByte, p => WordByte, q => WordByte, r => WordByte, s => WordByte, t => WordByte, u => WordByte, v => WordByte, w => WordByte, x => WordByte, y => WordByte, z => WordByte, { => NonWordByte, | => NonWordByte, } => NonWordByte, ~ => NonWordByte, /x7F => NonWordByte, /x80 => NonWordByte, /x81 => NonWordByte, /x82 => NonWordByte, /x83 => NonWordByte, /x84 => NonWordByte, /x85 => NonWordByte, /x86 => NonWordByte, /x87 => NonWordByte, /x88 => NonWordByte, /x89 => NonWordByte, /x8A => NonWordByte, /x8B => NonWordByte, /x8C => NonWordByte, /x8D => NonWordByte, /x8E => NonWordByte, /x8F => NonWordByte, /x90 => NonWordByte, /x91 => NonWordByte, /x92 => NonWordByte, /x93 => NonWordByte, /x94 => NonWordByte, /x95 => NonWordByte, /x96 => NonWordByte, /x97 => NonWordByte, /x98 => NonWordByte, /x99 => NonWordByte, /x9A => NonWordByte, /x9B => NonWordByte, /x9C => NonWordByte, /x9D => NonWordByte, /x9E => NonWordByte, /x9F => NonWordByte, /xA0 => NonWordByte, /xA1 => NonWordByte, /xA2 => NonWordByte, /xA3 => NonWordByte, /xA4 => NonWordByte, /xA5 => NonWordByte, /xA6 => NonWordByte, /xA7 => NonWordByte, /xA8 => NonWordByte, /xA9 => NonWordByte, /xAA => NonWordByte, /xAB => NonWordByte, /xAC => NonWordByte, /xAD => NonWordByte, /xAE => NonWordByte, /xAF => NonWordByte, /xB0 => NonWordByte, /xB1 => NonWordByte, /xB2 => NonWordByte, /xB3 => NonWordByte, /xB4 => NonWordByte, /xB5 => NonWordByte, /xB6 => NonWordByte, /xB7 => NonWordByte, /xB8 => NonWordByte, /xB9 => NonWordByte, /xBA => NonWordByte, /xBB => NonWordByte, /xBC => NonWordByte, /xBD => NonWordByte, /xBE => NonWordByte, /xBF => NonWordByte, /xC0 => NonWordByte, /xC1 => NonWordByte, /xC2 => NonWordByte, /xC3 => NonWordByte, /xC4 => NonWordByte, /xC5 => NonWordByte, /xC6 => NonWordByte, /xC7 => NonWordByte, /xC8 => NonWordByte, /xC9 => NonWordByte, /xCA => NonWordByte, /xCB => NonWordByte, /xCC => NonWordByte, /xCD => NonWordByte, /xCE => NonWordByte, /xCF => NonWordByte, /xD0 => NonWordByte, /xD1 => NonWordByte, /xD2 => NonWordByte, /xD3 => NonWordByte, /xD4 => NonWordByte, /xD5 => NonWordByte, /xD6 => NonWordByte, /xD7 => NonWordByte, /xD8 => NonWordByte, /xD9 => NonWordByte, /xDA => NonWordByte, /xDB => NonWordByte, /xDC => NonWordByte, /xDD => NonWordByte, /xDE => NonWordByte, /xDF => NonWordByte, /xE0 => NonWordByte, /xE1 => NonWordByte, /xE2 => NonWordByte, /xE3 => NonWordByte, /xE4 => NonWordByte, /xE5 => NonWordByte, /xE6 => NonWordByte, /xE7 => NonWordByte, /xE8 => NonWordByte, /xE9 => NonWordByte, /xEA => NonWordByte, /xEB => NonWordByte, /xEC => NonWordByte, /xED => NonWordByte, /xEE => NonWordByte, /xEF => NonWordByte, /xF0 => NonWordByte, /xF1 => NonWordByte, /xF2 => NonWordByte, /xF3 => NonWordByte, /xF4 => NonWordByte, /xF5 => NonWordByte, /xF6 => NonWordByte, /xF7 => NonWordByte, /xF8 => NonWordByte, /xF9 => NonWordByte, /xFA => NonWordByte, /xFB => NonWordByte, /xFC => NonWordByte, /xFD => NonWordByte, /xFE => NonWordByte, /xFF => NonWordByte}, + classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]), + quitset: ByteSet { + bits: {}, + }, + cache_capacity: 10485760, + }, + }, + ), + ), + ), + dfa: DFA( + None, + ), + }, + info: RegexInfo( + RegexInfoI { + config: Config { + match_kind: Some( + All, + ), + utf8_empty: Some( + false, + ), + autopre: None, + pre: None, + which_captures: None, + nfa_size_limit: Some( + Some( + 10485760, + ), + ), + onepass_size_limit: None, + hybrid_cache_capacity: Some( + 10485760, + ), + hybrid: None, + dfa: None, + dfa_size_limit: None, + dfa_state_limit: None, + onepass: None, + backtrack: None, + byte_classes: None, + line_terminator: None, + }, + props: [], + props_union: Properties( + PropertiesI { + minimum_len: None, + maximum_len: None, + look_set: ∅, + look_set_prefix: ∅, + look_set_suffix: ∅, + look_set_prefix_any: ∅, + look_set_suffix_any: ∅, + utf8: true, + explicit_captures_len: 0, + static_explicit_captures_len: None, + literal: false, + alternation_literal: true, + }, + ), + }, + ), + }, + pool: Pool( + Pool { + stacks: [ + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + ], + owner: 4, + owner_val: UnsafeCell { .. }, + }, + ), + }, + map: [], + patset: Pool( + Pool { + stacks: [ + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + ], + owner: 0, + owner_val: UnsafeCell { .. }, + }, + ), + }, + ), + ], + }, + poisoned: false, + .. + }, + }, + }, + run_conditions: RunConditions { + never: false, + always: true, + ignore_broken_dep_steps: true, + ignore_missing_outputs: true, + }, + pipeline_rundir: XvcPath( + "", + ), + terminate_timeout_processes: true, + algorithm: Blake3, + command_process: RwLock { + data: CommandProcess { + environment: {}, + step: XvcStep { + name: "average-age", + }, + step_command: XvcStepCommand { + command: "sqlite3 people.db /"SELECT AVG(Age) FROM People;/"", + }, + birth: Some( + Instant { + tv_sec: 806688, + tv_nsec: 701336000, + }, + ), + process: Some( + Popen { + stdin: None, + stdout: Some( + File { + fd: 5, + read: true, + write: false, + }, + ), + stderr: Some( + File { + fd: 7, + read: true, + write: false, + }, + ), + child_state: Running { + pid: 19822, + ext: (), + }, + detached: true, + }, + ), + stdout_sender: Sender { .. }, + stderr_sender: Sender { .. }, + stdout_receiver: Receiver { .. }, + stderr_receiver: Receiver { .. }, + }, + poisoned: false, + .. + }, + available_process_slots: RwLock { + data: 3, + poisoned: false, + .. + }, + process_poll_milliseconds: 10, + dependency_diffs: RwLock { + data: HStore { + map: {}, + }, + poisoned: false, + .. + }, + output_diffs: RwLock { + data: HStore { + map: {}, + }, + poisoned: false, + .. + }, + step_e: XvcEntity( + 2, + 12972606829502460557, + ), + step: XvcStep { + name: "average-age", + }, + step_command: XvcStepCommand { + command: "sqlite3 people.db /"SELECT AVG(Age) FROM People;/"", + }, + current_states: RwLock { + data: HStore { + map: { + XvcEntity( + 2, + 12972606829502460557, + ): Running( + FromWaitProcess, + ), + }, + }, + poisoned: false, + .. + }, + step_timeout: 10000s, + all_steps: HStore { + map: { + XvcEntity( + 2, + 12972606829502460557, + ): XvcStep { + name: "average-age", + }, + }, + }, + recorded_dependencies: R1NStore { + parents: XvcStore { + map: { + XvcEntity( + 2, + 12972606829502460557, + ): XvcStep { + name: "average-age", + }, + }, + entity_index: { + XvcStep { + name: "average-age", + }: [ + XvcEntity( + 2, + 12972606829502460557, + ), + ], + }, + previous: EventLog( + [ + Add { + entity: XvcEntity( + 2, + 12972606829502460557, + ), + value: XvcStep { + name: "average-age", + }, + }, + Add { + entity: XvcEntity( + 2, + 12972606829502460557, + ), + value: XvcStep { + name: "average-age", + }, + }, + ], + ), + current: EventLog( + [], + ), + }, + children: XvcStore { + map: {}, + entity_index: {}, + previous: EventLog( + [], + ), + current: EventLog( + [], + ), + }, + child_parents: XvcStore { + map: {}, + entity_index: {}, + previous: EventLog( + [], + ), + current: EventLog( + [], + ), + }, + }, + step_dependencies: {}, + step_outputs: HStore { + map: {}, + }, +} +[TRACE][pipeline/src/pipeline/mod.rs::1463] &process: Popen { + stdin: None, + stdout: Some( + File { + fd: 5, + read: true, + write: false, + }, + ), + stderr: Some( + File { + fd: 7, + read: true, + write: false, + }, + ), + child_state: Running { + pid: 19822, + ext: (), + }, + detached: true, +} +[TRACE][pipeline/src/pipeline/mod.rs::1468] process: Popen { + stdin: None, + stdout: Some( + File { + fd: 5, + read: true, + write: false, + }, + ), + stderr: Some( + File { + fd: 7, + read: true, + write: false, + }, + ), + child_state: Running { + pid: 19822, + ext: (), + }, + detached: true, +} [DEBUG] Step average-age with command sqlite3 people.db "SELECT AVG(Age) FROM People;" is still running [WARN] [ERR] [average-age] Error: in prepare, no such table: People +[TRACE][/Users/iex/.cargo/registry/src/index.crates.io-6f17d22bba15001f/notify-6.1.1/src/fsevent.rs::536] FSEvent: path = `[CWD]/people.db`, flag = StreamFlags(ITEM_CREATED | IS_FILE) +[TRACE][walker/src/notify.rs::56] event: Ok( + Event { + kind: Create( + File, + ), + paths: [ + "[CWD]/people.db", + ], + attr:tracker: None, + attr:flag: None, + attr:info: None, + attr:source: None, + }, +) +[TRACE][walker/src/lib.rs::748] is_abs: true +[TRACE][walker/src/lib.rs::752] path_str: "[CWD]/people.db" +[TRACE][walker/src/lib.rs::754] final_slash: false +[TRACE][walker/src/lib.rs::776] path: "/people.db" +[TRACE][core/src/util/pmp.rs::99] index: 0 +[TRACE][core/src/util/pmp.rs::102] fs_event: Ok( + Some( + Create { + path: "[CWD]/people.db", + metadata: Metadata { + file_type: FileType( + FileType { + mode: 33188, + }, + ), + is_dir: false, + is_file: true, + permissions: Permissions( + FilePermissions { + mode: 33188, + }, + ), + modified: Ok( + SystemTime { + tv_sec: 1722022739, + tv_nsec: 53602406, + }, + ), + accessed: Ok( + SystemTime { + tv_sec: 1722022739, + tv_nsec: 53602406, + }, + ), + created: Ok( + SystemTime { + tv_sec: 1722022739, + tv_nsec: 53602406, + }, + ), + .. + }, + }, + ), +) +[TRACE][core/src/types/xvcpath.rs::88] abs_path: "[CWD]/people.db" +[TRACE][core/src/types/xvcpath.rs::89] current_dir: AbsolutePath( + "[CWD]", +) +[TRACE][core/src/types/xvcpath.rs::90] xvc_root.absolute_path(): AbsolutePath( + "[CWD]", +) +[TRACE][core/src/util/pmp.rs::65] "Creating {} with {}": "Creating {} with {}" +[TRACE][core/src/util/pmp.rs::65] xvc_path: XvcPath( + "people.db", +) +[TRACE][core/src/util/pmp.rs::65] xvc_md: XvcMetadata { + file_type: File, + size: Some( + 0, + ), + modified: Some( + SystemTime { + tv_sec: 1722022739, + tv_nsec: 53602406, + }, + ), +} +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][pipeline/src/pipeline/mod.rs::593] s: Running( + FromWaitProcess, +) +[TRACE][pipeline/src/pipeline/mod.rs::583] select: Select { .. } +[TRACE][pipeline/src/pipeline/mod.rs::1463] &process: Popen { + stdin: None, + stdout: Some( + File { + fd: 5, + read: true, + write: false, + }, + ), + stderr: Some( + File { + fd: 7, + read: true, + write: false, + }, + ), + child_state: Running { + pid: 19822, + ext: (), + }, + detached: true, +} [ERROR] Step average-age finished UNSUCCESSFULLY with command sqlite3 people.db "SELECT AVG(Age) FROM People;" +[TRACE][pipeline/src/pipeline/mod.rs::1515] return_state: Some( + Broken( + FromProcessReturnedNonZero, + ), +) +[TRACE][pipeline/src/pipeline/mod.rs::1521] params: StepStateParams { + xvc_root: XvcRootInner { + absolute_path: AbsolutePath( + "[CWD]", + ), + xvc_dir: AbsolutePath( + "[CWD]/.xvc", + ), + store_dir: AbsolutePath( + "[CWD]/.xvc/store", + ), + config: XvcConfig { + current_dir: XvcConfigOption { + source: Runtime, + option: AbsolutePath( + "[CWD]", + ), + }, + config_maps: [ + XvcConfigMap { + source: Default, + map: { + "file.list.sort": String( + "name-desc", + ), + "pipeline.process_pool_size": Integer( + 4, + ), + "git.command": String( + "git", + ), + "pipeline.current_pipeline": String( + "default", + ), + "file.carry-in.force": Boolean( + false, + ), + "pipeline.default_params_file": String( + "params.yaml", + ), + "pipeline.default": String( + "default", + ), + "core.guid": String( + "7726b84aa14cc82c", + ), + "git.auto_stage": Boolean( + false, + ), + "file.track.force": Boolean( + false, + ), + "cache.algorithm": String( + "blake3", + ), + "file.recheck.method": String( + "copy", + ), + "core.verbosity": String( + "error", + ), + "file.track.no_commit": Boolean( + false, + ), + "file.list.show_dot_files": Boolean( + false, + ), + "file.track.text_or_binary": String( + "auto", + ), + "file.list.recursive": Boolean( + false, + ), + "file.list.no_summary": Boolean( + false, + ), + "git.use_git": Boolean( + true, + ), + "file.carry-in.no_parallel": Boolean( + false, + ), + "file.list.format": String( + "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", + ), + "file.track.no_parallel": Boolean( + false, + ), + "git.auto_commit": Boolean( + true, + ), + }, + }, + XvcConfigMap { + source: Project, + map: { + "file.list.no_summary": Boolean( + false, + ), + "file.list.sort": String( + "name-desc", + ), + "git.auto_commit": Boolean( + true, + ), + "git.auto_stage": Boolean( + false, + ), + "file.track.no_parallel": Boolean( + false, + ), + "file.track.no_commit": Boolean( + false, + ), + "pipeline.process_pool_size": Integer( + 4, + ), + "pipeline.default_params_file": String( + "params.yaml", + ), + "pipeline.default": String( + "default", + ), + "git.command": String( + "git", + ), + "file.list.show_dot_files": Boolean( + false, + ), + "file.list.format": String( + "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", + ), + "core.guid": String( + "4bd7a8139a85fb10", + ), + "pipeline.current_pipeline": String( + "default", + ), + "file.track.text_or_binary": String( + "auto", + ), + "file.recheck.method": String( + "copy", + ), + "git.use_git": Boolean( + true, + ), + "file.list.recursive": Boolean( + false, + ), + "file.carry-in.no_parallel": Boolean( + false, + ), + "file.carry-in.force": Boolean( + false, + ), + "file.track.force": Boolean( + false, + ), + "core.verbosity": String( + "error", + ), + "cache.algorithm": String( + "blake3", + ), + }, + }, + XvcConfigMap { + source: Local, + map: {}, + }, + XvcConfigMap { + source: Environment, + map: { + "TRYCMD_TESTS": String( + "storage,file,pipeline,core", + ), + }, + }, + XvcConfigMap { + source: CommandLine, + map: { + "core.verbosity": String( + "debug", + ), + "core.quiet": Boolean( + false, + ), + }, + }, + ], + the_config: { + "pipeline.default_params_file": XvcConfigValue { + source: Project, + value: String( + "params.yaml", + ), + }, + "file.list.recursive": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "file.track.no_commit": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "file.carry-in.no_parallel": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "pipeline.current_pipeline": XvcConfigValue { + source: Project, + value: String( + "default", + ), + }, + "file.track.force": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "pipeline.default": XvcConfigValue { + source: Project, + value: String( + "default", + ), + }, + "cache.algorithm": XvcConfigValue { + source: Project, + value: String( + "blake3", + ), + }, + "file.track.text_or_binary": XvcConfigValue { + source: Project, + value: String( + "auto", + ), + }, + "TRYCMD_TESTS": XvcConfigValue { + source: Environment, + value: String( + "storage,file,pipeline,core", + ), + }, + "pipeline.process_pool_size": XvcConfigValue { + source: Project, + value: Integer( + 4, + ), + }, + "file.list.no_summary": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "core.guid": XvcConfigValue { + source: Project, + value: String( + "4bd7a8139a85fb10", + ), + }, + "file.list.format": XvcConfigValue { + source: Project, + value: String( + "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", + ), + }, + "file.carry-in.force": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "git.use_git": XvcConfigValue { + source: Project, + value: Boolean( + true, + ), + }, + "git.auto_stage": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "file.list.show_dot_files": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "file.track.no_parallel": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "git.auto_commit": XvcConfigValue { + source: Project, + value: Boolean( + true, + ), + }, + "file.list.sort": XvcConfigValue { + source: Project, + value: String( + "name-desc", + ), + }, + "file.recheck.method": XvcConfigValue { + source: Project, + value: String( + "copy", + ), + }, + "core.verbosity": XvcConfigValue { + source: CommandLine, + value: String( + "debug", + ), + }, + "git.command": XvcConfigValue { + source: Project, + value: String( + "git", + ), + }, + "core.quiet": XvcConfigValue { + source: CommandLine, + value: Boolean( + false, + ), + }, + }, + init_params: XvcConfigParams { + default_configuration: " +[core] +# The repository id. Please do not delete or change it. +# This is used to identify the repository and generate paths in storages. +# In the future it may be used to in other ways. +guid = /"7726b84aa14cc82c/" +# Default verbosity level. +# One of /"error/", /"warn/", /"info/" +verbosity = /"error/" + +[git] +# Automate git operations. +# Turning this off leads Xvc to behave as if it's not in a Git repository. +# Not recommended unless you're really not using Git +use_git = true +# Command to run Git process. +# You can set this to an absolute path to specify an executable +# If set to a non-absolute path, the executable will be searched in $PATH. +command = /"git/" + +# Commit changes in .xvc/ directory after commands. +# You can set this to false if you want to commit manually. +auto_commit = true + +# Stage changes in .xvc/ directory without committing. +# auto_commit implies auto_stage. +# If you want to commit manually but don't want to stage after individual Xvc commands, you can set this to true. +auto_stage = false + +[cache] +# The hash algorithm used for the cache. +# It may take blake3, blake2, sha2 or sha3 as values. +# All algorithms are selected to produce 256-bit hashes, so sha2 means SHA2-256, blake2 means BLAKE2s, etc. +# The cache path is produced by prepending algorithm name to the cache. +# Blake3 files are in .xvc/b3/, while sha2 files are in .xvc/s2/ etc. +algorithm = /"blake3/" + +[file] + +[file.track] + +# Don't move file content to cache after xvc file track +no_commit = false +# Force to track files even if they are already tracked. +force = false + +# Xvc calculates file content digest differently for text and binary files. +# This option controls whether to treat files as text or binary. +# It may take auto, text or binary as values. +# Auto check each file individually and treat it as text if it's text. +text_or_binary = /"auto/" + +# Don't use parallelism in track operations. +# Note that some of the operations are implemented in parallel by default, and this option affects some heavier operations. +no_parallel = false + +[file.list] + +# Format for `xvc file list` rows. You can reorder or remove columns. +# The following are the keys for each row: +# - {acd64}: actual content digest. All 64 digits from the workspace file's content. +# - {acd8}: actual content digest. First 8 digits the file content digest. +# - {aft}: actual file type. Whether the entry is a file (F), directory (D), +# symlink (S), hardlink (H) or reflink (R). +# - {asz}: actual size. The size of the workspace file in bytes. It uses MB, +# GB and TB to represent sizes larger than 1MB. +# - {ats}: actual timestamp. The timestamp of the workspace file. +# - {cst}: cache status. One of /"=/", /">/", /"/", /" 0 + F 000001: + >000002: /x00 => 2 + 000003: /x00 => 0 + match kind: Standard + prefilter: false + state length: 4 + pattern length: 0 + shortest pattern length: 18446744073709551615 + longest pattern length: 0 + alphabet length: 1 + stride: 1 + byte classes: ByteClasses(0 => [0-255]) + memory usage: 16 + ) + , + ), + map: [], + longest: 0, + }, + ), + Prefix( + PrefixStrategy { + matcher: AhoCorasick( + dfa::DFA( + D 000000: /x00 => 0 + F 000001: + >000002: /x00 => 2 + 000003: /x00 => 0 + match kind: Standard + prefilter: false + state length: 4 + pattern length: 0 + shortest pattern length: 18446744073709551615 + longest pattern length: 0 + alphabet length: 1 + stride: 1 + byte classes: ByteClasses(0 => [0-255]) + memory usage: 16 + ) + , + ), + map: [], + longest: 0, + }, + ), + RequiredExtension( + RequiredExtensionStrategy( + {}, + ), + ), + Regex( + RegexSetStrategy { + matcher: Regex { + imp: RegexI { + strat: Core { + info: RegexInfo( + RegexInfoI { + config: Config { + match_kind: Some( + All, + ), + utf8_empty: Some( + false, + ), + autopre: None, + pre: None, + which_captures: None, + nfa_size_limit: Some( + Some( + 10485760, + ), + ), + onepass_size_limit: None, + hybrid_cache_capacity: Some( + 10485760, + ), + hybrid: None, + dfa: None, + dfa_size_limit: None, + dfa_state_limit: None, + onepass: None, + backtrack: None, + byte_classes: None, + line_terminator: None, + }, + props: [], + props_union: Properties( + PropertiesI { + minimum_len: None, + maximum_len: None, + look_set: ∅, + look_set_prefix: ∅, + look_set_suffix: ∅, + look_set_prefix_any: ∅, + look_set_suffix_any: ∅, + utf8: true, + explicit_captures_len: 0, + static_explicit_captures_len: None, + literal: false, + alternation_literal: true, + }, + ), + }, + ), + pre: None, + nfa: thompson::NFA( + ^000000: FAIL + + transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) + ) + , + nfarev: Some( + thompson::NFA( + ^000000: FAIL + + transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) + ) + , + ), + pikevm: PikeVM( + PikeVMEngine( + PikeVM { + config: Config { + match_kind: Some( + All, + ), + pre: Some( + None, + ), + }, + nfa: thompson::NFA( + ^000000: FAIL + + transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) + ) + , + }, + ), + ), + backtrack: BoundedBacktracker( + None, + ), + onepass: OnePass( + None, + ), + hybrid: Hybrid( + Some( + HybridEngine( + Regex { + forward: DFA { + config: Config { + match_kind: Some( + All, + ), + pre: Some( + None, + ), + starts_for_each_pattern: Some( + true, + ), + byte_classes: Some( + true, + ), + unicode_word_boundary: Some( + true, + ), + quitset: None, + specialize_start_states: Some( + false, + ), + cache_capacity: Some( + 10485760, + ), + skip_cache_capacity_check: Some( + false, + ), + minimum_cache_clear_count: Some( + Some( + 3, + ), + ), + minimum_bytes_per_state: Some( + Some( + 10, + ), + ), + }, + nfa: thompson::NFA( + ^000000: FAIL + + transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) + ) + , + stride2: 1, + start_map: StartByteMap{/x00 => NonWordByte, /x01 => NonWordByte, /x02 => NonWordByte, /x03 => NonWordByte, /x04 => NonWordByte, /x05 => NonWordByte, /x06 => NonWordByte, /x07 => NonWordByte, /x08 => NonWordByte, /t => NonWordByte, + => LineLF, /x0B => NonWordByte, /x0C => NonWordByte, /r => LineCR, /x0E => NonWordByte, /x0F => NonWordByte, /x10 => NonWordByte, /x11 => NonWordByte, /x12 => NonWordByte, /x13 => NonWordByte, /x14 => NonWordByte, /x15 => NonWordByte, /x16 => NonWordByte, /x17 => NonWordByte, /x18 => NonWordByte, /x19 => NonWordByte, /x1A => NonWordByte, /x1B => NonWordByte, /x1C => NonWordByte, /x1D => NonWordByte, /x1E => NonWordByte, /x1F => NonWordByte, ' ' => NonWordByte, ! => NonWordByte, /" => NonWordByte, # => NonWordByte, $ => NonWordByte, % => NonWordByte, & => NonWordByte, /' => NonWordByte, ( => NonWordByte, ) => NonWordByte, * => NonWordByte, + => NonWordByte, , => NonWordByte, - => NonWordByte, . => NonWordByte, / => NonWordByte, 0 => WordByte, 1 => WordByte, 2 => WordByte, 3 => WordByte, 4 => WordByte, 5 => WordByte, 6 => WordByte, 7 => WordByte, 8 => WordByte, 9 => WordByte, : => NonWordByte, ; => NonWordByte, < => NonWordByte, = => NonWordByte, > => NonWordByte, ? => NonWordByte, @ => NonWordByte, A => WordByte, B => WordByte, C => WordByte, D => WordByte, E => WordByte, F => WordByte, G => WordByte, H => WordByte, I => WordByte, J => WordByte, K => WordByte, L => WordByte, M => WordByte, N => WordByte, O => WordByte, P => WordByte, Q => WordByte, R => WordByte, S => WordByte, T => WordByte, U => WordByte, V => WordByte, W => WordByte, X => WordByte, Y => WordByte, Z => WordByte, [ => NonWordByte, // => NonWordByte, ] => NonWordByte, ^ => NonWordByte, _ => WordByte, ` => NonWordByte, a => WordByte, b => WordByte, c => WordByte, d => WordByte, e => WordByte, f => WordByte, g => WordByte, h => WordByte, i => WordByte, j => WordByte, k => WordByte, l => WordByte, m => WordByte, n => WordByte, o => WordByte, p => WordByte, q => WordByte, r => WordByte, s => WordByte, t => WordByte, u => WordByte, v => WordByte, w => WordByte, x => WordByte, y => WordByte, z => WordByte, { => NonWordByte, | => NonWordByte, } => NonWordByte, ~ => NonWordByte, /x7F => NonWordByte, /x80 => NonWordByte, /x81 => NonWordByte, /x82 => NonWordByte, /x83 => NonWordByte, /x84 => NonWordByte, /x85 => NonWordByte, /x86 => NonWordByte, /x87 => NonWordByte, /x88 => NonWordByte, /x89 => NonWordByte, /x8A => NonWordByte, /x8B => NonWordByte, /x8C => NonWordByte, /x8D => NonWordByte, /x8E => NonWordByte, /x8F => NonWordByte, /x90 => NonWordByte, /x91 => NonWordByte, /x92 => NonWordByte, /x93 => NonWordByte, /x94 => NonWordByte, /x95 => NonWordByte, /x96 => NonWordByte, /x97 => NonWordByte, /x98 => NonWordByte, /x99 => NonWordByte, /x9A => NonWordByte, /x9B => NonWordByte, /x9C => NonWordByte, /x9D => NonWordByte, /x9E => NonWordByte, /x9F => NonWordByte, /xA0 => NonWordByte, /xA1 => NonWordByte, /xA2 => NonWordByte, /xA3 => NonWordByte, /xA4 => NonWordByte, /xA5 => NonWordByte, /xA6 => NonWordByte, /xA7 => NonWordByte, /xA8 => NonWordByte, /xA9 => NonWordByte, /xAA => NonWordByte, /xAB => NonWordByte, /xAC => NonWordByte, /xAD => NonWordByte, /xAE => NonWordByte, /xAF => NonWordByte, /xB0 => NonWordByte, /xB1 => NonWordByte, /xB2 => NonWordByte, /xB3 => NonWordByte, /xB4 => NonWordByte, /xB5 => NonWordByte, /xB6 => NonWordByte, /xB7 => NonWordByte, /xB8 => NonWordByte, /xB9 => NonWordByte, /xBA => NonWordByte, /xBB => NonWordByte, /xBC => NonWordByte, /xBD => NonWordByte, /xBE => NonWordByte, /xBF => NonWordByte, /xC0 => NonWordByte, /xC1 => NonWordByte, /xC2 => NonWordByte, /xC3 => NonWordByte, /xC4 => NonWordByte, /xC5 => NonWordByte, /xC6 => NonWordByte, /xC7 => NonWordByte, /xC8 => NonWordByte, /xC9 => NonWordByte, /xCA => NonWordByte, /xCB => NonWordByte, /xCC => NonWordByte, /xCD => NonWordByte, /xCE => NonWordByte, /xCF => NonWordByte, /xD0 => NonWordByte, /xD1 => NonWordByte, /xD2 => NonWordByte, /xD3 => NonWordByte, /xD4 => NonWordByte, /xD5 => NonWordByte, /xD6 => NonWordByte, /xD7 => NonWordByte, /xD8 => NonWordByte, /xD9 => NonWordByte, /xDA => NonWordByte, /xDB => NonWordByte, /xDC => NonWordByte, /xDD => NonWordByte, /xDE => NonWordByte, /xDF => NonWordByte, /xE0 => NonWordByte, /xE1 => NonWordByte, /xE2 => NonWordByte, /xE3 => NonWordByte, /xE4 => NonWordByte, /xE5 => NonWordByte, /xE6 => NonWordByte, /xE7 => NonWordByte, /xE8 => NonWordByte, /xE9 => NonWordByte, /xEA => NonWordByte, /xEB => NonWordByte, /xEC => NonWordByte, /xED => NonWordByte, /xEE => NonWordByte, /xEF => NonWordByte, /xF0 => NonWordByte, /xF1 => NonWordByte, /xF2 => NonWordByte, /xF3 => NonWordByte, /xF4 => NonWordByte, /xF5 => NonWordByte, /xF6 => NonWordByte, /xF7 => NonWordByte, /xF8 => NonWordByte, /xF9 => NonWordByte, /xFA => NonWordByte, /xFB => NonWordByte, /xFC => NonWordByte, /xFD => NonWordByte, /xFE => NonWordByte, /xFF => NonWordByte}, + classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]), + quitset: ByteSet { + bits: {}, + }, + cache_capacity: 10485760, + }, + reverse: DFA { + config: Config { + match_kind: Some( + All, + ), + pre: Some( + None, + ), + starts_for_each_pattern: Some( + true, + ), + byte_classes: Some( + true, + ), + unicode_word_boundary: Some( + true, + ), + quitset: None, + specialize_start_states: Some( + false, + ), + cache_capacity: Some( + 10485760, + ), + skip_cache_capacity_check: Some( + false, + ), + minimum_cache_clear_count: Some( + Some( + 3, + ), + ), + minimum_bytes_per_state: Some( + Some( + 10, + ), + ), + }, + nfa: thompson::NFA( + ^000000: FAIL + + transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) + ) + , + stride2: 1, + start_map: StartByteMap{/x00 => NonWordByte, /x01 => NonWordByte, /x02 => NonWordByte, /x03 => NonWordByte, /x04 => NonWordByte, /x05 => NonWordByte, /x06 => NonWordByte, /x07 => NonWordByte, /x08 => NonWordByte, /t => NonWordByte, + => LineLF, /x0B => NonWordByte, /x0C => NonWordByte, /r => LineCR, /x0E => NonWordByte, /x0F => NonWordByte, /x10 => NonWordByte, /x11 => NonWordByte, /x12 => NonWordByte, /x13 => NonWordByte, /x14 => NonWordByte, /x15 => NonWordByte, /x16 => NonWordByte, /x17 => NonWordByte, /x18 => NonWordByte, /x19 => NonWordByte, /x1A => NonWordByte, /x1B => NonWordByte, /x1C => NonWordByte, /x1D => NonWordByte, /x1E => NonWordByte, /x1F => NonWordByte, ' ' => NonWordByte, ! => NonWordByte, /" => NonWordByte, # => NonWordByte, $ => NonWordByte, % => NonWordByte, & => NonWordByte, /' => NonWordByte, ( => NonWordByte, ) => NonWordByte, * => NonWordByte, + => NonWordByte, , => NonWordByte, - => NonWordByte, . => NonWordByte, / => NonWordByte, 0 => WordByte, 1 => WordByte, 2 => WordByte, 3 => WordByte, 4 => WordByte, 5 => WordByte, 6 => WordByte, 7 => WordByte, 8 => WordByte, 9 => WordByte, : => NonWordByte, ; => NonWordByte, < => NonWordByte, = => NonWordByte, > => NonWordByte, ? => NonWordByte, @ => NonWordByte, A => WordByte, B => WordByte, C => WordByte, D => WordByte, E => WordByte, F => WordByte, G => WordByte, H => WordByte, I => WordByte, J => WordByte, K => WordByte, L => WordByte, M => WordByte, N => WordByte, O => WordByte, P => WordByte, Q => WordByte, R => WordByte, S => WordByte, T => WordByte, U => WordByte, V => WordByte, W => WordByte, X => WordByte, Y => WordByte, Z => WordByte, [ => NonWordByte, // => NonWordByte, ] => NonWordByte, ^ => NonWordByte, _ => WordByte, ` => NonWordByte, a => WordByte, b => WordByte, c => WordByte, d => WordByte, e => WordByte, f => WordByte, g => WordByte, h => WordByte, i => WordByte, j => WordByte, k => WordByte, l => WordByte, m => WordByte, n => WordByte, o => WordByte, p => WordByte, q => WordByte, r => WordByte, s => WordByte, t => WordByte, u => WordByte, v => WordByte, w => WordByte, x => WordByte, y => WordByte, z => WordByte, { => NonWordByte, | => NonWordByte, } => NonWordByte, ~ => NonWordByte, /x7F => NonWordByte, /x80 => NonWordByte, /x81 => NonWordByte, /x82 => NonWordByte, /x83 => NonWordByte, /x84 => NonWordByte, /x85 => NonWordByte, /x86 => NonWordByte, /x87 => NonWordByte, /x88 => NonWordByte, /x89 => NonWordByte, /x8A => NonWordByte, /x8B => NonWordByte, /x8C => NonWordByte, /x8D => NonWordByte, /x8E => NonWordByte, /x8F => NonWordByte, /x90 => NonWordByte, /x91 => NonWordByte, /x92 => NonWordByte, /x93 => NonWordByte, /x94 => NonWordByte, /x95 => NonWordByte, /x96 => NonWordByte, /x97 => NonWordByte, /x98 => NonWordByte, /x99 => NonWordByte, /x9A => NonWordByte, /x9B => NonWordByte, /x9C => NonWordByte, /x9D => NonWordByte, /x9E => NonWordByte, /x9F => NonWordByte, /xA0 => NonWordByte, /xA1 => NonWordByte, /xA2 => NonWordByte, /xA3 => NonWordByte, /xA4 => NonWordByte, /xA5 => NonWordByte, /xA6 => NonWordByte, /xA7 => NonWordByte, /xA8 => NonWordByte, /xA9 => NonWordByte, /xAA => NonWordByte, /xAB => NonWordByte, /xAC => NonWordByte, /xAD => NonWordByte, /xAE => NonWordByte, /xAF => NonWordByte, /xB0 => NonWordByte, /xB1 => NonWordByte, /xB2 => NonWordByte, /xB3 => NonWordByte, /xB4 => NonWordByte, /xB5 => NonWordByte, /xB6 => NonWordByte, /xB7 => NonWordByte, /xB8 => NonWordByte, /xB9 => NonWordByte, /xBA => NonWordByte, /xBB => NonWordByte, /xBC => NonWordByte, /xBD => NonWordByte, /xBE => NonWordByte, /xBF => NonWordByte, /xC0 => NonWordByte, /xC1 => NonWordByte, /xC2 => NonWordByte, /xC3 => NonWordByte, /xC4 => NonWordByte, /xC5 => NonWordByte, /xC6 => NonWordByte, /xC7 => NonWordByte, /xC8 => NonWordByte, /xC9 => NonWordByte, /xCA => NonWordByte, /xCB => NonWordByte, /xCC => NonWordByte, /xCD => NonWordByte, /xCE => NonWordByte, /xCF => NonWordByte, /xD0 => NonWordByte, /xD1 => NonWordByte, /xD2 => NonWordByte, /xD3 => NonWordByte, /xD4 => NonWordByte, /xD5 => NonWordByte, /xD6 => NonWordByte, /xD7 => NonWordByte, /xD8 => NonWordByte, /xD9 => NonWordByte, /xDA => NonWordByte, /xDB => NonWordByte, /xDC => NonWordByte, /xDD => NonWordByte, /xDE => NonWordByte, /xDF => NonWordByte, /xE0 => NonWordByte, /xE1 => NonWordByte, /xE2 => NonWordByte, /xE3 => NonWordByte, /xE4 => NonWordByte, /xE5 => NonWordByte, /xE6 => NonWordByte, /xE7 => NonWordByte, /xE8 => NonWordByte, /xE9 => NonWordByte, /xEA => NonWordByte, /xEB => NonWordByte, /xEC => NonWordByte, /xED => NonWordByte, /xEE => NonWordByte, /xEF => NonWordByte, /xF0 => NonWordByte, /xF1 => NonWordByte, /xF2 => NonWordByte, /xF3 => NonWordByte, /xF4 => NonWordByte, /xF5 => NonWordByte, /xF6 => NonWordByte, /xF7 => NonWordByte, /xF8 => NonWordByte, /xF9 => NonWordByte, /xFA => NonWordByte, /xFB => NonWordByte, /xFC => NonWordByte, /xFD => NonWordByte, /xFE => NonWordByte, /xFF => NonWordByte}, + classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]), + quitset: ByteSet { + bits: {}, + }, + cache_capacity: 10485760, + }, + }, + ), + ), + ), + dfa: DFA( + None, + ), + }, + info: RegexInfo( + RegexInfoI { + config: Config { + match_kind: Some( + All, + ), + utf8_empty: Some( + false, + ), + autopre: None, + pre: None, + which_captures: None, + nfa_size_limit: Some( + Some( + 10485760, + ), + ), + onepass_size_limit: None, + hybrid_cache_capacity: Some( + 10485760, + ), + hybrid: None, + dfa: None, + dfa_size_limit: None, + dfa_state_limit: None, + onepass: None, + backtrack: None, + byte_classes: None, + line_terminator: None, + }, + props: [], + props_union: Properties( + PropertiesI { + minimum_len: None, + maximum_len: None, + look_set: ∅, + look_set_prefix: ∅, + look_set_suffix: ∅, + look_set_prefix_any: ∅, + look_set_suffix_any: ∅, + utf8: true, + explicit_captures_len: 0, + static_explicit_captures_len: None, + literal: false, + alternation_literal: true, + }, + ), + }, + ), + }, + pool: Pool( + Pool { + stacks: [ + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + ], + owner: 4, + owner_val: UnsafeCell { .. }, + }, + ), + }, + map: [], + patset: Pool( + Pool { + stacks: [ + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + ], + owner: 0, + owner_val: UnsafeCell { .. }, + }, + ), + }, + ), + ], + }, + poisoned: false, + .. + }, + }, + }, + run_conditions: RunConditions { + never: false, + always: true, + ignore_broken_dep_steps: true, + ignore_missing_outputs: true, + }, + pipeline_rundir: XvcPath( + "", + ), + terminate_timeout_processes: true, + algorithm: Blake3, + command_process: RwLock { + data: CommandProcess { + environment: {}, + step: XvcStep { + name: "average-age", + }, + step_command: XvcStepCommand { + command: "sqlite3 people.db /"SELECT AVG(Age) FROM People;/"", + }, + birth: Some( + Instant { + tv_sec: 806688, + tv_nsec: 701336000, + }, + ), + process: Some( + Popen { + stdin: None, + stdout: Some( + File { + fd: 5, + read: true, + write: false, + }, + ), + stderr: Some( + File { + fd: 7, + read: true, + write: false, + }, + ), + child_state: Finished( + Exited( + 1, + ), + ), + detached: true, + }, + ), + stdout_sender: Sender { .. }, + stderr_sender: Sender { .. }, + stdout_receiver: Receiver { .. }, + stderr_receiver: Receiver { .. }, + }, + poisoned: false, + .. + }, + available_process_slots: RwLock { + data: , + poisoned: false, + .. + }, + process_poll_milliseconds: 10, + dependency_diffs: RwLock { + data: HStore { + map: {}, + }, + poisoned: false, + .. + }, + output_diffs: RwLock { + data: HStore { + map: {}, + }, + poisoned: false, + .. + }, + step_e: XvcEntity( + 2, + 12972606829502460557, + ), + step: XvcStep { + name: "average-age", + }, + step_command: XvcStepCommand { + command: "sqlite3 people.db /"SELECT AVG(Age) FROM People;/"", + }, + current_states: RwLock { + data: HStore { + map: { + XvcEntity( + 2, + 12972606829502460557, + ): Running( + FromWaitProcess, + ), + }, + }, + poisoned: false, + .. + }, + step_timeout: 10000s, + all_steps: HStore { + map: { + XvcEntity( + 2, + 12972606829502460557, + ): XvcStep { + name: "average-age", + }, + }, + }, + recorded_dependencies: R1NStore { + parents: XvcStore { + map: { + XvcEntity( + 2, + 12972606829502460557, + ): XvcStep { + name: "average-age", + }, + }, + entity_index: { + XvcStep { + name: "average-age", + }: [ + XvcEntity( + 2, + 12972606829502460557, + ), + ], + }, + previous: EventLog( + [ + Add { + entity: XvcEntity( + 2, + 12972606829502460557, + ), + value: XvcStep { + name: "average-age", + }, + }, + Add { + entity: XvcEntity( + 2, + 12972606829502460557, + ), + value: XvcStep { + name: "average-age", + }, + }, + ], + ), + current: EventLog( + [], + ), + }, + children: XvcStore { + map: {}, + entity_index: {}, + previous: EventLog( + [], + ), + current: EventLog( + [], + ), + }, + child_parents: XvcStore { + map: {}, + entity_index: {}, + previous: EventLog( + [], + ), + current: EventLog( + [], + ), + }, + }, + step_dependencies: {}, + step_outputs: HStore { + map: {}, + }, +} +[TRACE][pipeline/src/pipeline/mod.rs::772] step.name: "average-age" +[TRACE][pipeline/src/pipeline/mod.rs::773] &r_next_state: Broken( + FromProcessReturnedNonZero, +) +[TRACE][pipeline/src/pipeline/mod.rs::775] &step_state: Broken( + FromProcessReturnedNonZero, +) +[TRACE][pipeline/src/pipeline/mod.rs::662] &step_state: Broken( + FromProcessReturnedNonZero, +) +[TRACE][pipeline/src/pipeline/mod.rs::583] select: Select { .. } +[TRACE][pipeline/src/pipeline/mod.rs::519] "Before state updater": "Before state updater" +[TRACE][pipeline/src/pipeline/mod.rs::593] s: Broken( + FromProcessReturnedNonZero, +) +[TRACE][pipeline/src/pipeline/mod.rs::532] step_states: RwLock { + data: HStore { + map: { + XvcEntity( + 2, + 12972606829502460557, + ): Broken( + FromProcessReturnedNonZero, + ), + }, + }, + poisoned: false, + .. +} +[TRACE][pipeline/src/pipeline/mod.rs::536] done_successfully: Ok( + false, +) +[TRACE][core/src/util/pmp.rs::185] self.background_thread: Mutex { + data: JoinHandle { .. }, + poisoned: false, + .. +} +[TRACE][core/src/util/pmp.rs::190] self.background_thread: Mutex { + data: JoinHandle { .. }, + poisoned: false, + .. +} +[TRACE][lib/src/cli/mod.rs::376] "Before handle_git_automation": "Before handle_git_automation" +[TRACE][core/src/util/pmp.rs::99] index: 1 [DEBUG] Using Git: /opt/homebrew/bin/git +[TRACE][lib/src/git.rs::30] args: [ + "-C", + "[CWD]", + "diff", + "--name-only", + "--cached", +] +[TRACE][lib/src/git.rs::61] git_diff_staged_out: "" +[TRACE][lib/src/git.rs::30] args: [ + "-C", + "[CWD]", + "add", + "--verbose", + "[CWD]/.xvc", + "*.gitignore", + "*.xvcignore", +] +[TRACE][lib/src/git.rs::179] git_add_output: "" [DEBUG] No files to commit [DEBUG] Command completed successfully. From 1ac90d255aff419ea889efdd3fe87a0c8a485792 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Fri, 26 Jul 2024 23:17:17 +0300 Subject: [PATCH 068/257] added people,db --- .../xvc-pipeline-step-dependency-sqlite.md | 2666 ++++++++++------- 1 file changed, 1586 insertions(+), 1080 deletions(-) diff --git a/book/src/ref/xvc-pipeline-step-dependency-sqlite.md b/book/src/ref/xvc-pipeline-step-dependency-sqlite.md index b2789ddd2..58c31036f 100644 --- a/book/src/ref/xvc-pipeline-step-dependency-sqlite.md +++ b/book/src/ref/xvc-pipeline-step-dependency-sqlite.md @@ -14,11 +14,9 @@ $ git init $ xvc init ``` -Let's create an SQLite database and populate a table: - -```console -$ sqlite3 people.db <, } -[TRACE][pipeline/src/pipeline/mod.rs::583] select: Select { .. } [TRACE][pipeline/src/pipeline/mod.rs::625] dependency_steps(step_e, params.dependency_graph)?: {} [TRACE][pipeline/src/pipeline/mod.rs::564] dep_neighbors: Neighbors { iter: Iter( @@ -2128,90 +2546,6 @@ digraph { ), ty: PhantomData, } -[TRACE][/Users/iex/.cargo/registry/src/index.crates.io-6f17d22bba15001f/notify-6.1.1/src/fsevent.rs::536] FSEvent: path = `[CWD]/.xvc/store/xvc-pipeline-run-dir-store`, flag = StreamFlags(ITEM_CREATED | IS_DIR) -[TRACE][walker/src/notify.rs::56] event: Ok( - Event { - kind: Create( - Folder, - ), - paths: [ - "[CWD]/.xvc/store/xvc-pipeline-run-dir-store", - ], - attr:tracker: None, - attr:flag: None, - attr:info: None, - attr:source: None, - }, -) -[TRACE][walker/src/lib.rs::748] is_abs: true -[TRACE][walker/src/lib.rs::752] path_str: "[CWD]/.xvc/store/xvc-pipeline-run-dir-store" -[TRACE][walker/src/lib.rs::754] final_slash: false -[TRACE][walker/src/lib.rs::776] path: "/.xvc/store/xvc-pipeline-run-dir-store" -[TRACE][core/src/util/pmp.rs::99] index: 0 -[TRACE][core/src/util/pmp.rs::102] fs_event: Ok( - Some( - Create { - path: "[CWD]/.xvc/store/xvc-pipeline-run-dir-store", - metadata: Metadata { - file_type: FileType( - FileType { - mode: 16877, - }, - ), - is_dir: true, - is_file: false, - permissions: Permissions( - FilePermissions { - mode: 16877, - }, - ), - modified: Ok( - SystemTime { - tv_sec: 1722022739, - tv_nsec: 33553054, - }, - ), - accessed: Ok( - SystemTime { - tv_sec: 1722022739, - tv_nsec: 33553054, - }, - ), - created: Ok( - SystemTime { - tv_sec: 1722022739, - tv_nsec: 33553054, - }, - ), - .. - }, - }, - ), -) -[TRACE][core/src/types/xvcpath.rs::88] abs_path: "[CWD]/.xvc/store/xvc-pipeline-run-dir-store" -[TRACE][core/src/types/xvcpath.rs::89] current_dir: AbsolutePath( - "[CWD]", -) -[TRACE][core/src/types/xvcpath.rs::90] xvc_root.absolute_path(): AbsolutePath( - "[CWD]", -) -[TRACE][core/src/util/pmp.rs::65] "Creating {} with {}": "Creating {} with {}" -[TRACE][core/src/util/pmp.rs::65] xvc_path: XvcPath( - ".xvc/store/xvc-pipeline-run-dir-store", -) -[TRACE][core/src/util/pmp.rs::65] xvc_md: XvcMetadata { - file_type: Directory, - size: Some( - 64, - ), - modified: Some( - SystemTime { - tv_sec: 1722022739, - tv_nsec: 33553054, - }, - ), -} -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" [TRACE][pipeline/src/pipeline/mod.rs::662] &step_state: Begin( FromInit, ) @@ -2219,7 +2553,6 @@ digraph { [TRACE][pipeline/src/pipeline/mod.rs::773] &r_next_state: WaitingDependencySteps( FromRunConditional, ) -[TRACE][pipeline/src/pipeline/mod.rs::583] select: Select { .. } [TRACE][pipeline/src/pipeline/mod.rs::775] &step_state: WaitingDependencySteps( FromRunConditional, ) @@ -2233,7 +2566,6 @@ digraph { [TRACE][pipeline/src/pipeline/mod.rs::775] &step_state: CheckingOutputs( FromDependencyStepsFinishedSuccessfully, ) -[INFO] No dependency steps for step average-age [TRACE][pipeline/src/pipeline/mod.rs::662] &step_state: CheckingOutputs( FromDependencyStepsFinishedSuccessfully, ) @@ -2249,15 +2581,12 @@ digraph { ) [TRACE][pipeline/src/pipeline/mod.rs::1010] parent_entity: XvcEntity( 2, - 12972606829502460557, + 16393284406203488121, ) -[TRACE][pipeline/src/pipeline/mod.rs::583] select: Select { .. } [TRACE][pipeline/src/pipeline/mod.rs::1013] deps: HStore { map: {}, } -[TRACE][pipeline/src/pipeline/mod.rs::583] select: Select { .. } [TRACE][pipeline/src/pipeline/mod.rs::1016] params.step.name: "average-age" -[TRACE][pipeline/src/pipeline/mod.rs::583] select: Select { .. } [TRACE][pipeline/src/pipeline/mod.rs::772] step.name: "average-age" [TRACE][pipeline/src/pipeline/mod.rs::773] &r_next_state: CheckingThoroughDiffs( FromSuperficialDiffsChanged, @@ -2291,10 +2620,15 @@ digraph { [TRACE][pipeline/src/pipeline/mod.rs::662] &step_state: WaitingToRun( FromDiffsHasChanged, ) -[INFO] [average-age] Dependencies has changed [TRACE][pipeline/src/pipeline/mod.rs::583] select: Select { .. } [TRACE][pipeline/src/pipeline/mod.rs::583] select: Select { .. } [TRACE][pipeline/src/pipeline/mod.rs::583] select: Select { .. } +[TRACE][pipeline/src/pipeline/mod.rs::583] select: Select { .. } +[TRACE][pipeline/src/pipeline/mod.rs::583] select: Select { .. } +[TRACE][pipeline/src/pipeline/mod.rs::583] select: Select { .. } +[TRACE][pipeline/src/pipeline/mod.rs::583] select: Select { .. } +[INFO] No dependency steps for step average-age +[INFO] [average-age] Dependencies has changed [TRACE][pipeline/src/pipeline/mod.rs::1541] params: StepStateParams { xvc_root: XvcRootInner { absolute_path: AbsolutePath( @@ -2317,87 +2651,93 @@ digraph { XvcConfigMap { source: Default, map: { - "file.list.sort": String( - "name-desc", - ), - "pipeline.process_pool_size": Integer( - 4, - ), - "git.command": String( - "git", - ), - "pipeline.current_pipeline": String( - "default", + "file.list.format": String( + "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", ), - "file.carry-in.force": Boolean( + "file.track.force": Boolean( false, ), - "pipeline.default_params_file": String( - "params.yaml", - ), "pipeline.default": String( "default", ), - "core.guid": String( - "7726b84aa14cc82c", + "git.auto_commit": Boolean( + true, + ), + "file.recheck.method": String( + "copy", + ), + "pipeline.process_pool_size": Integer( + 4, ), "git.auto_stage": Boolean( false, ), - "file.track.force": Boolean( - false, + "git.use_git": Boolean( + true, ), "cache.algorithm": String( "blake3", ), - "file.recheck.method": String( - "copy", + "file.track.no_parallel": Boolean( + false, ), - "core.verbosity": String( - "error", + "file.list.recursive": Boolean( + false, ), - "file.track.no_commit": Boolean( + "file.list.no_summary": Boolean( false, ), - "file.list.show_dot_files": Boolean( + "file.track.no_commit": Boolean( false, ), - "file.track.text_or_binary": String( - "auto", + "pipeline.current_pipeline": String( + "default", ), - "file.list.recursive": Boolean( + "file.carry-in.no_parallel": Boolean( false, ), - "file.list.no_summary": Boolean( + "core.guid": String( + "603c2d6a0d2e6ea9", + ), + "file.list.show_dot_files": Boolean( false, ), - "git.use_git": Boolean( - true, + "file.list.sort": String( + "name-desc", ), - "file.carry-in.no_parallel": Boolean( + "file.track.text_or_binary": String( + "auto", + ), + "file.carry-in.force": Boolean( false, ), - "file.list.format": String( - "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", + "pipeline.default_params_file": String( + "params.yaml", ), - "file.track.no_parallel": Boolean( - false, + "git.command": String( + "git", ), - "git.auto_commit": Boolean( - true, + "core.verbosity": String( + "error", ), }, }, XvcConfigMap { source: Project, map: { - "file.list.no_summary": Boolean( + "file.list.recursive": Boolean( false, ), - "file.list.sort": String( - "name-desc", + "pipeline.default": String( + "default", ), - "git.auto_commit": Boolean( + "pipeline.default_params_file": String( + "params.yaml", + ), + "file.track.no_commit": Boolean( + false, + ), + "git.use_git": Boolean( true, ), "git.auto_stage": Boolean( @@ -2406,44 +2746,32 @@ digraph { "file.track.no_parallel": Boolean( false, ), - "file.track.no_commit": Boolean( + "file.list.no_summary": Boolean( false, ), - "pipeline.process_pool_size": Integer( - 4, - ), - "pipeline.default_params_file": String( - "params.yaml", - ), - "pipeline.default": String( - "default", + "file.list.sort": String( + "name-desc", ), - "git.command": String( - "git", + "core.guid": String( + "860d1874a44e1d89", ), "file.list.show_dot_files": Boolean( false, ), - "file.list.format": String( - "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", - ), - "core.guid": String( - "4bd7a8139a85fb10", - ), - "pipeline.current_pipeline": String( - "default", + "git.command": String( + "git", ), "file.track.text_or_binary": String( "auto", ), + "pipeline.process_pool_size": Integer( + 4, + ), "file.recheck.method": String( "copy", ), - "git.use_git": Boolean( - true, - ), - "file.list.recursive": Boolean( - false, + "file.list.format": String( + "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", ), "file.carry-in.no_parallel": Boolean( false, @@ -2451,12 +2779,18 @@ digraph { "file.carry-in.force": Boolean( false, ), + "pipeline.current_pipeline": String( + "default", + ), "file.track.force": Boolean( false, ), "core.verbosity": String( "error", ), + "git.auto_commit": Boolean( + true, + ), "cache.algorithm": String( "blake3", ), @@ -2487,16 +2821,28 @@ digraph { }, ], the_config: { + "pipeline.current_pipeline": XvcConfigValue { + source: Project, + value: String( + "default", + ), + }, + "cache.algorithm": XvcConfigValue { + source: Project, + value: String( + "blake3", + ), + }, "pipeline.default_params_file": XvcConfigValue { source: Project, value: String( "params.yaml", ), }, - "file.list.recursive": XvcConfigValue { + "git.command": XvcConfigValue { source: Project, - value: Boolean( - false, + value: String( + "git", ), }, "file.track.no_commit": XvcConfigValue { @@ -2505,40 +2851,46 @@ digraph { false, ), }, - "file.carry-in.no_parallel": XvcConfigValue { + "core.verbosity": XvcConfigValue { + source: CommandLine, + value: String( + "debug", + ), + }, + "file.track.text_or_binary": XvcConfigValue { source: Project, - value: Boolean( - false, + value: String( + "auto", ), }, - "pipeline.current_pipeline": XvcConfigValue { + "core.guid": XvcConfigValue { source: Project, value: String( - "default", + "860d1874a44e1d89", ), }, - "file.track.force": XvcConfigValue { + "file.list.show_dot_files": XvcConfigValue { source: Project, value: Boolean( false, ), }, - "pipeline.default": XvcConfigValue { + "file.recheck.method": XvcConfigValue { source: Project, value: String( - "default", + "copy", ), }, - "cache.algorithm": XvcConfigValue { + "file.track.no_parallel": XvcConfigValue { source: Project, - value: String( - "blake3", + value: Boolean( + false, ), }, - "file.track.text_or_binary": XvcConfigValue { + "file.list.no_summary": XvcConfigValue { source: Project, - value: String( - "auto", + value: Boolean( + false, ), }, "TRYCMD_TESTS": XvcConfigValue { @@ -2547,31 +2899,19 @@ digraph { "storage,file,pipeline,core", ), }, - "pipeline.process_pool_size": XvcConfigValue { + "file.list.sort": XvcConfigValue { source: Project, - value: Integer( - 4, + value: String( + "name-desc", ), }, - "file.list.no_summary": XvcConfigValue { - source: Project, + "core.quiet": XvcConfigValue { + source: CommandLine, value: Boolean( false, ), }, - "core.guid": XvcConfigValue { - source: Project, - value: String( - "4bd7a8139a85fb10", - ), - }, - "file.list.format": XvcConfigValue { - source: Project, - value: String( - "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", - ), - }, - "file.carry-in.force": XvcConfigValue { + "git.auto_stage": XvcConfigValue { source: Project, value: Boolean( false, @@ -2583,58 +2923,52 @@ digraph { true, ), }, - "git.auto_stage": XvcConfigValue { + "file.carry-in.no_parallel": XvcConfigValue { source: Project, value: Boolean( false, ), }, - "file.list.show_dot_files": XvcConfigValue { + "pipeline.default": XvcConfigValue { source: Project, - value: Boolean( - false, + value: String( + "default", ), }, - "file.track.no_parallel": XvcConfigValue { + "file.list.recursive": XvcConfigValue { source: Project, value: Boolean( false, ), }, - "git.auto_commit": XvcConfigValue { + "file.track.force": XvcConfigValue { source: Project, value: Boolean( - true, + false, ), }, - "file.list.sort": XvcConfigValue { + "file.list.format": XvcConfigValue { source: Project, value: String( - "name-desc", + "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", ), }, - "file.recheck.method": XvcConfigValue { + "pipeline.process_pool_size": XvcConfigValue { source: Project, - value: String( - "copy", - ), - }, - "core.verbosity": XvcConfigValue { - source: CommandLine, - value: String( - "debug", + value: Integer( + 4, ), }, - "git.command": XvcConfigValue { + "file.carry-in.force": XvcConfigValue { source: Project, - value: String( - "git", + value: Boolean( + false, ), }, - "core.quiet": XvcConfigValue { - source: CommandLine, + "git.auto_commit": XvcConfigValue { + source: Project, value: Boolean( - false, + true, ), }, }, @@ -2644,7 +2978,7 @@ digraph { # The repository id. Please do not delete or change it. # This is used to identify the repository and generate paths in storages. # In the future it may be used to in other ways. -guid = /"7726b84aa14cc82c/" +guid = /"603c2d6a0d2e6ea9/" # Default verbosity level. # One of /"error/", /"warn/", /"info/" verbosity = /"error/" @@ -2795,7 +3129,7 @@ process_pool_size = 4 ), entity_generator: XvcEntityGenerator { counter: 3, - random: 6938067968196728862, + random: 5804399665853476265, dirty: false, }, }, @@ -2822,87 +3156,93 @@ process_pool_size = 4 XvcConfigMap { source: Default, map: { - "file.list.sort": String( - "name-desc", - ), - "pipeline.process_pool_size": Integer( - 4, - ), - "git.command": String( - "git", - ), - "pipeline.current_pipeline": String( - "default", + "file.list.format": String( + "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", ), - "file.carry-in.force": Boolean( + "file.track.force": Boolean( false, ), - "pipeline.default_params_file": String( - "params.yaml", - ), "pipeline.default": String( "default", ), - "core.guid": String( - "7726b84aa14cc82c", + "git.auto_commit": Boolean( + true, + ), + "file.recheck.method": String( + "copy", + ), + "pipeline.process_pool_size": Integer( + 4, ), "git.auto_stage": Boolean( false, ), - "file.track.force": Boolean( - false, + "git.use_git": Boolean( + true, ), "cache.algorithm": String( "blake3", ), - "file.recheck.method": String( - "copy", + "file.track.no_parallel": Boolean( + false, ), - "core.verbosity": String( - "error", + "file.list.recursive": Boolean( + false, ), - "file.track.no_commit": Boolean( + "file.list.no_summary": Boolean( false, ), - "file.list.show_dot_files": Boolean( + "file.track.no_commit": Boolean( false, ), - "file.track.text_or_binary": String( - "auto", + "pipeline.current_pipeline": String( + "default", ), - "file.list.recursive": Boolean( + "file.carry-in.no_parallel": Boolean( false, ), - "file.list.no_summary": Boolean( + "core.guid": String( + "603c2d6a0d2e6ea9", + ), + "file.list.show_dot_files": Boolean( false, ), - "git.use_git": Boolean( - true, + "file.list.sort": String( + "name-desc", ), - "file.carry-in.no_parallel": Boolean( + "file.track.text_or_binary": String( + "auto", + ), + "file.carry-in.force": Boolean( false, ), - "file.list.format": String( - "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", + "pipeline.default_params_file": String( + "params.yaml", ), - "file.track.no_parallel": Boolean( - false, + "git.command": String( + "git", ), - "git.auto_commit": Boolean( - true, + "core.verbosity": String( + "error", ), }, }, XvcConfigMap { source: Project, map: { - "file.list.no_summary": Boolean( + "file.list.recursive": Boolean( false, ), - "file.list.sort": String( - "name-desc", + "pipeline.default": String( + "default", + ), + "pipeline.default_params_file": String( + "params.yaml", + ), + "file.track.no_commit": Boolean( + false, ), - "git.auto_commit": Boolean( + "git.use_git": Boolean( true, ), "git.auto_stage": Boolean( @@ -2911,44 +3251,32 @@ process_pool_size = 4 "file.track.no_parallel": Boolean( false, ), - "file.track.no_commit": Boolean( + "file.list.no_summary": Boolean( false, ), - "pipeline.process_pool_size": Integer( - 4, - ), - "pipeline.default_params_file": String( - "params.yaml", - ), - "pipeline.default": String( - "default", + "file.list.sort": String( + "name-desc", ), - "git.command": String( - "git", + "core.guid": String( + "860d1874a44e1d89", ), "file.list.show_dot_files": Boolean( false, ), - "file.list.format": String( - "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", - ), - "core.guid": String( - "4bd7a8139a85fb10", - ), - "pipeline.current_pipeline": String( - "default", + "git.command": String( + "git", ), "file.track.text_or_binary": String( "auto", ), + "pipeline.process_pool_size": Integer( + 4, + ), "file.recheck.method": String( "copy", ), - "git.use_git": Boolean( - true, - ), - "file.list.recursive": Boolean( - false, + "file.list.format": String( + "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", ), "file.carry-in.no_parallel": Boolean( false, @@ -2956,12 +3284,18 @@ process_pool_size = 4 "file.carry-in.force": Boolean( false, ), + "pipeline.current_pipeline": String( + "default", + ), "file.track.force": Boolean( false, ), "core.verbosity": String( "error", ), + "git.auto_commit": Boolean( + true, + ), "cache.algorithm": String( "blake3", ), @@ -2992,16 +3326,28 @@ process_pool_size = 4 }, ], the_config: { + "pipeline.current_pipeline": XvcConfigValue { + source: Project, + value: String( + "default", + ), + }, + "cache.algorithm": XvcConfigValue { + source: Project, + value: String( + "blake3", + ), + }, "pipeline.default_params_file": XvcConfigValue { source: Project, value: String( "params.yaml", ), }, - "file.list.recursive": XvcConfigValue { + "git.command": XvcConfigValue { source: Project, - value: Boolean( - false, + value: String( + "git", ), }, "file.track.no_commit": XvcConfigValue { @@ -3010,40 +3356,46 @@ process_pool_size = 4 false, ), }, - "file.carry-in.no_parallel": XvcConfigValue { + "core.verbosity": XvcConfigValue { + source: CommandLine, + value: String( + "debug", + ), + }, + "file.track.text_or_binary": XvcConfigValue { source: Project, - value: Boolean( - false, + value: String( + "auto", ), }, - "pipeline.current_pipeline": XvcConfigValue { + "core.guid": XvcConfigValue { source: Project, value: String( - "default", + "860d1874a44e1d89", ), }, - "file.track.force": XvcConfigValue { + "file.list.show_dot_files": XvcConfigValue { source: Project, value: Boolean( false, ), }, - "pipeline.default": XvcConfigValue { + "file.recheck.method": XvcConfigValue { source: Project, value: String( - "default", + "copy", ), }, - "cache.algorithm": XvcConfigValue { + "file.track.no_parallel": XvcConfigValue { source: Project, - value: String( - "blake3", + value: Boolean( + false, ), }, - "file.track.text_or_binary": XvcConfigValue { + "file.list.no_summary": XvcConfigValue { source: Project, - value: String( - "auto", + value: Boolean( + false, ), }, "TRYCMD_TESTS": XvcConfigValue { @@ -3052,31 +3404,19 @@ process_pool_size = 4 "storage,file,pipeline,core", ), }, - "pipeline.process_pool_size": XvcConfigValue { + "file.list.sort": XvcConfigValue { source: Project, - value: Integer( - 4, + value: String( + "name-desc", ), }, - "file.list.no_summary": XvcConfigValue { - source: Project, + "core.quiet": XvcConfigValue { + source: CommandLine, value: Boolean( false, ), }, - "core.guid": XvcConfigValue { - source: Project, - value: String( - "4bd7a8139a85fb10", - ), - }, - "file.list.format": XvcConfigValue { - source: Project, - value: String( - "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", - ), - }, - "file.carry-in.force": XvcConfigValue { + "git.auto_stage": XvcConfigValue { source: Project, value: Boolean( false, @@ -3088,58 +3428,52 @@ process_pool_size = 4 true, ), }, - "git.auto_stage": XvcConfigValue { + "file.carry-in.no_parallel": XvcConfigValue { source: Project, value: Boolean( false, ), }, - "file.list.show_dot_files": XvcConfigValue { + "pipeline.default": XvcConfigValue { source: Project, - value: Boolean( - false, + value: String( + "default", ), }, - "file.track.no_parallel": XvcConfigValue { + "file.list.recursive": XvcConfigValue { source: Project, value: Boolean( false, ), }, - "git.auto_commit": XvcConfigValue { + "file.track.force": XvcConfigValue { source: Project, value: Boolean( - true, + false, ), }, - "file.list.sort": XvcConfigValue { + "file.list.format": XvcConfigValue { source: Project, value: String( - "name-desc", + "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", ), }, - "file.recheck.method": XvcConfigValue { + "pipeline.process_pool_size": XvcConfigValue { source: Project, - value: String( - "copy", - ), - }, - "core.verbosity": XvcConfigValue { - source: CommandLine, - value: String( - "debug", + value: Integer( + 4, ), }, - "git.command": XvcConfigValue { + "file.carry-in.force": XvcConfigValue { source: Project, - value: String( - "git", + value: Boolean( + false, ), }, - "core.quiet": XvcConfigValue { - source: CommandLine, + "git.auto_commit": XvcConfigValue { + source: Project, value: Boolean( - false, + true, ), }, }, @@ -3149,7 +3483,7 @@ process_pool_size = 4 # The repository id. Please do not delete or change it. # This is used to identify the repository and generate paths in storages. # In the future it may be used to in other ways. -guid = /"7726b84aa14cc82c/" +guid = /"603c2d6a0d2e6ea9/" # Default verbosity level. # One of /"error/", /"warn/", /"info/" verbosity = /"error/" @@ -3300,7 +3634,7 @@ process_pool_size = 4 ), entity_generator: XvcEntityGenerator { counter: 3, - random: 6938067968196728862, + random: 5804399665853476265, dirty: false, }, }, @@ -3315,8 +3649,64 @@ process_pool_size = 4 ), modified: Some( SystemTime { - tv_sec: 1722022739, - tv_nsec: 33553054, + tv_sec: 1722024005, + tv_nsec: 902113188, + }, + ), + }, + XvcPath( + ".xvc/store/xvc-dependency-store", + ): XvcMetadata { + file_type: Directory, + size: Some( + 64, + ), + modified: Some( + SystemTime { + tv_sec: 1722024005, + tv_nsec: 893699709, + }, + ), + }, + XvcPath( + ".xvc/store/xvc-output-store", + ): XvcMetadata { + file_type: Directory, + size: Some( + 64, + ), + modified: Some( + SystemTime { + tv_sec: 1722024005, + tv_nsec: 893895957, + }, + ), + }, + XvcPath( + ".xvc/store/xvc-output-xvc-step-r1n-store", + ): XvcMetadata { + file_type: Directory, + size: Some( + 64, + ), + modified: Some( + SystemTime { + tv_sec: 1722024005, + tv_nsec: 893950706, + }, + ), + }, + XvcPath( + ".xvc/store/xvc-dependency-xvc-step-r1n-store", + ): XvcMetadata { + file_type: Directory, + size: Some( + 64, + ), + modified: Some( + SystemTime { + tv_sec: 1722024005, + tv_nsec: 893766208, }, ), }, @@ -3996,7 +4386,7 @@ process_pool_size = 4 name: "average-age", }, step_command: XvcStepCommand { - command: "sqlite3 people.db /"SELECT AVG(Age) FROM People;/"", + command: "sqlite3 people.db 'SELECT AVG(Age) FROM People;'", }, birth: None, process: None, @@ -4030,20 +4420,20 @@ process_pool_size = 4 }, step_e: XvcEntity( 2, - 12972606829502460557, + 16393284406203488121, ), step: XvcStep { name: "average-age", }, step_command: XvcStepCommand { - command: "sqlite3 people.db /"SELECT AVG(Age) FROM People;/"", + command: "sqlite3 people.db 'SELECT AVG(Age) FROM People;'", }, current_states: RwLock { data: HStore { map: { XvcEntity( 2, - 12972606829502460557, + 16393284406203488121, ): WaitingToRun( FromDiffsHasChanged, ), @@ -4057,7 +4447,7 @@ process_pool_size = 4 map: { XvcEntity( 2, - 12972606829502460557, + 16393284406203488121, ): XvcStep { name: "average-age", }, @@ -4068,7 +4458,7 @@ process_pool_size = 4 map: { XvcEntity( 2, - 12972606829502460557, + 16393284406203488121, ): XvcStep { name: "average-age", }, @@ -4079,7 +4469,7 @@ process_pool_size = 4 }: [ XvcEntity( 2, - 12972606829502460557, + 16393284406203488121, ), ], }, @@ -4088,7 +4478,7 @@ process_pool_size = 4 Add { entity: XvcEntity( 2, - 12972606829502460557, + 16393284406203488121, ), value: XvcStep { name: "average-age", @@ -4097,7 +4487,7 @@ process_pool_size = 4 Add { entity: XvcEntity( 2, - 12972606829502460557, + 16393284406203488121, ), value: XvcStep { name: "average-age", @@ -4180,87 +4570,93 @@ process_pool_size = 4 XvcConfigMap { source: Default, map: { - "file.list.sort": String( - "name-desc", - ), - "pipeline.process_pool_size": Integer( - 4, - ), - "git.command": String( - "git", - ), - "pipeline.current_pipeline": String( - "default", + "file.list.format": String( + "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", ), - "file.carry-in.force": Boolean( + "file.track.force": Boolean( false, ), - "pipeline.default_params_file": String( - "params.yaml", - ), "pipeline.default": String( "default", ), - "core.guid": String( - "7726b84aa14cc82c", + "git.auto_commit": Boolean( + true, + ), + "file.recheck.method": String( + "copy", + ), + "pipeline.process_pool_size": Integer( + 4, ), "git.auto_stage": Boolean( false, ), - "file.track.force": Boolean( - false, + "git.use_git": Boolean( + true, ), "cache.algorithm": String( "blake3", ), - "file.recheck.method": String( - "copy", + "file.track.no_parallel": Boolean( + false, ), - "core.verbosity": String( - "error", + "file.list.recursive": Boolean( + false, ), - "file.track.no_commit": Boolean( + "file.list.no_summary": Boolean( false, ), - "file.list.show_dot_files": Boolean( + "file.track.no_commit": Boolean( false, ), - "file.track.text_or_binary": String( - "auto", + "pipeline.current_pipeline": String( + "default", ), - "file.list.recursive": Boolean( + "file.carry-in.no_parallel": Boolean( false, ), - "file.list.no_summary": Boolean( + "core.guid": String( + "603c2d6a0d2e6ea9", + ), + "file.list.show_dot_files": Boolean( false, ), - "git.use_git": Boolean( - true, + "file.list.sort": String( + "name-desc", + ), + "file.track.text_or_binary": String( + "auto", ), - "file.carry-in.no_parallel": Boolean( + "file.carry-in.force": Boolean( false, ), - "file.list.format": String( - "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", + "pipeline.default_params_file": String( + "params.yaml", ), - "file.track.no_parallel": Boolean( - false, + "git.command": String( + "git", ), - "git.auto_commit": Boolean( - true, + "core.verbosity": String( + "error", ), }, }, XvcConfigMap { source: Project, map: { - "file.list.no_summary": Boolean( + "file.list.recursive": Boolean( false, ), - "file.list.sort": String( - "name-desc", + "pipeline.default": String( + "default", ), - "git.auto_commit": Boolean( + "pipeline.default_params_file": String( + "params.yaml", + ), + "file.track.no_commit": Boolean( + false, + ), + "git.use_git": Boolean( true, ), "git.auto_stage": Boolean( @@ -4269,44 +4665,32 @@ process_pool_size = 4 "file.track.no_parallel": Boolean( false, ), - "file.track.no_commit": Boolean( + "file.list.no_summary": Boolean( false, ), - "pipeline.process_pool_size": Integer( - 4, - ), - "pipeline.default_params_file": String( - "params.yaml", - ), - "pipeline.default": String( - "default", + "file.list.sort": String( + "name-desc", ), - "git.command": String( - "git", + "core.guid": String( + "860d1874a44e1d89", ), "file.list.show_dot_files": Boolean( false, ), - "file.list.format": String( - "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", - ), - "core.guid": String( - "4bd7a8139a85fb10", - ), - "pipeline.current_pipeline": String( - "default", + "git.command": String( + "git", ), "file.track.text_or_binary": String( "auto", ), + "pipeline.process_pool_size": Integer( + 4, + ), "file.recheck.method": String( "copy", ), - "git.use_git": Boolean( - true, - ), - "file.list.recursive": Boolean( - false, + "file.list.format": String( + "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", ), "file.carry-in.no_parallel": Boolean( false, @@ -4314,12 +4698,18 @@ process_pool_size = 4 "file.carry-in.force": Boolean( false, ), + "pipeline.current_pipeline": String( + "default", + ), "file.track.force": Boolean( false, ), "core.verbosity": String( "error", ), + "git.auto_commit": Boolean( + true, + ), "cache.algorithm": String( "blake3", ), @@ -4350,16 +4740,28 @@ process_pool_size = 4 }, ], the_config: { + "pipeline.current_pipeline": XvcConfigValue { + source: Project, + value: String( + "default", + ), + }, + "cache.algorithm": XvcConfigValue { + source: Project, + value: String( + "blake3", + ), + }, "pipeline.default_params_file": XvcConfigValue { source: Project, value: String( "params.yaml", ), }, - "file.list.recursive": XvcConfigValue { + "git.command": XvcConfigValue { source: Project, - value: Boolean( - false, + value: String( + "git", ), }, "file.track.no_commit": XvcConfigValue { @@ -4368,40 +4770,46 @@ process_pool_size = 4 false, ), }, - "file.carry-in.no_parallel": XvcConfigValue { + "core.verbosity": XvcConfigValue { + source: CommandLine, + value: String( + "debug", + ), + }, + "file.track.text_or_binary": XvcConfigValue { source: Project, - value: Boolean( - false, + value: String( + "auto", ), }, - "pipeline.current_pipeline": XvcConfigValue { + "core.guid": XvcConfigValue { source: Project, value: String( - "default", + "860d1874a44e1d89", ), }, - "file.track.force": XvcConfigValue { + "file.list.show_dot_files": XvcConfigValue { source: Project, value: Boolean( false, ), }, - "pipeline.default": XvcConfigValue { + "file.recheck.method": XvcConfigValue { source: Project, value: String( - "default", + "copy", ), }, - "cache.algorithm": XvcConfigValue { + "file.track.no_parallel": XvcConfigValue { source: Project, - value: String( - "blake3", + value: Boolean( + false, ), }, - "file.track.text_or_binary": XvcConfigValue { + "file.list.no_summary": XvcConfigValue { source: Project, - value: String( - "auto", + value: Boolean( + false, ), }, "TRYCMD_TESTS": XvcConfigValue { @@ -4410,31 +4818,19 @@ process_pool_size = 4 "storage,file,pipeline,core", ), }, - "pipeline.process_pool_size": XvcConfigValue { + "file.list.sort": XvcConfigValue { source: Project, - value: Integer( - 4, + value: String( + "name-desc", ), }, - "file.list.no_summary": XvcConfigValue { - source: Project, + "core.quiet": XvcConfigValue { + source: CommandLine, value: Boolean( false, ), }, - "core.guid": XvcConfigValue { - source: Project, - value: String( - "4bd7a8139a85fb10", - ), - }, - "file.list.format": XvcConfigValue { - source: Project, - value: String( - "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", - ), - }, - "file.carry-in.force": XvcConfigValue { + "git.auto_stage": XvcConfigValue { source: Project, value: Boolean( false, @@ -4446,58 +4842,52 @@ process_pool_size = 4 true, ), }, - "git.auto_stage": XvcConfigValue { + "file.carry-in.no_parallel": XvcConfigValue { source: Project, value: Boolean( false, ), }, - "file.list.show_dot_files": XvcConfigValue { + "pipeline.default": XvcConfigValue { source: Project, - value: Boolean( - false, + value: String( + "default", ), }, - "file.track.no_parallel": XvcConfigValue { + "file.list.recursive": XvcConfigValue { source: Project, value: Boolean( false, ), }, - "git.auto_commit": XvcConfigValue { + "file.track.force": XvcConfigValue { source: Project, value: Boolean( - true, + false, ), }, - "file.list.sort": XvcConfigValue { + "file.list.format": XvcConfigValue { source: Project, value: String( - "name-desc", + "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", ), }, - "file.recheck.method": XvcConfigValue { + "pipeline.process_pool_size": XvcConfigValue { source: Project, - value: String( - "copy", - ), - }, - "core.verbosity": XvcConfigValue { - source: CommandLine, - value: String( - "debug", + value: Integer( + 4, ), }, - "git.command": XvcConfigValue { + "file.carry-in.force": XvcConfigValue { source: Project, - value: String( - "git", + value: Boolean( + false, ), }, - "core.quiet": XvcConfigValue { - source: CommandLine, + "git.auto_commit": XvcConfigValue { + source: Project, value: Boolean( - false, + true, ), }, }, @@ -4507,7 +4897,7 @@ process_pool_size = 4 # The repository id. Please do not delete or change it. # This is used to identify the repository and generate paths in storages. # In the future it may be used to in other ways. -guid = /"7726b84aa14cc82c/" +guid = /"603c2d6a0d2e6ea9/" # Default verbosity level. # One of /"error/", /"warn/", /"info/" verbosity = /"error/" @@ -4658,7 +5048,7 @@ process_pool_size = 4 ), entity_generator: XvcEntityGenerator { counter: 3, - random: 6938067968196728862, + random: 5804399665853476265, dirty: false, }, }, @@ -4685,87 +5075,93 @@ process_pool_size = 4 XvcConfigMap { source: Default, map: { - "file.list.sort": String( - "name-desc", - ), - "pipeline.process_pool_size": Integer( - 4, - ), - "git.command": String( - "git", - ), - "pipeline.current_pipeline": String( - "default", + "file.list.format": String( + "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", ), - "file.carry-in.force": Boolean( + "file.track.force": Boolean( false, ), - "pipeline.default_params_file": String( - "params.yaml", - ), "pipeline.default": String( "default", ), - "core.guid": String( - "7726b84aa14cc82c", + "git.auto_commit": Boolean( + true, + ), + "file.recheck.method": String( + "copy", + ), + "pipeline.process_pool_size": Integer( + 4, ), "git.auto_stage": Boolean( false, ), - "file.track.force": Boolean( - false, + "git.use_git": Boolean( + true, ), "cache.algorithm": String( "blake3", ), - "file.recheck.method": String( - "copy", + "file.track.no_parallel": Boolean( + false, ), - "core.verbosity": String( - "error", + "file.list.recursive": Boolean( + false, ), - "file.track.no_commit": Boolean( + "file.list.no_summary": Boolean( false, ), - "file.list.show_dot_files": Boolean( + "file.track.no_commit": Boolean( false, ), - "file.track.text_or_binary": String( - "auto", + "pipeline.current_pipeline": String( + "default", ), - "file.list.recursive": Boolean( + "file.carry-in.no_parallel": Boolean( false, ), - "file.list.no_summary": Boolean( + "core.guid": String( + "603c2d6a0d2e6ea9", + ), + "file.list.show_dot_files": Boolean( false, ), - "git.use_git": Boolean( - true, + "file.list.sort": String( + "name-desc", ), - "file.carry-in.no_parallel": Boolean( + "file.track.text_or_binary": String( + "auto", + ), + "file.carry-in.force": Boolean( false, ), - "file.list.format": String( - "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", + "pipeline.default_params_file": String( + "params.yaml", ), - "file.track.no_parallel": Boolean( - false, + "git.command": String( + "git", ), - "git.auto_commit": Boolean( - true, + "core.verbosity": String( + "error", ), }, }, XvcConfigMap { source: Project, map: { - "file.list.no_summary": Boolean( + "file.list.recursive": Boolean( false, ), - "file.list.sort": String( - "name-desc", + "pipeline.default": String( + "default", ), - "git.auto_commit": Boolean( + "pipeline.default_params_file": String( + "params.yaml", + ), + "file.track.no_commit": Boolean( + false, + ), + "git.use_git": Boolean( true, ), "git.auto_stage": Boolean( @@ -4774,44 +5170,32 @@ process_pool_size = 4 "file.track.no_parallel": Boolean( false, ), - "file.track.no_commit": Boolean( + "file.list.no_summary": Boolean( false, ), - "pipeline.process_pool_size": Integer( - 4, - ), - "pipeline.default_params_file": String( - "params.yaml", - ), - "pipeline.default": String( - "default", + "file.list.sort": String( + "name-desc", ), - "git.command": String( - "git", + "core.guid": String( + "860d1874a44e1d89", ), "file.list.show_dot_files": Boolean( false, ), - "file.list.format": String( - "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", - ), - "core.guid": String( - "4bd7a8139a85fb10", - ), - "pipeline.current_pipeline": String( - "default", + "git.command": String( + "git", ), "file.track.text_or_binary": String( "auto", ), + "pipeline.process_pool_size": Integer( + 4, + ), "file.recheck.method": String( "copy", ), - "git.use_git": Boolean( - true, - ), - "file.list.recursive": Boolean( - false, + "file.list.format": String( + "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", ), "file.carry-in.no_parallel": Boolean( false, @@ -4819,12 +5203,18 @@ process_pool_size = 4 "file.carry-in.force": Boolean( false, ), + "pipeline.current_pipeline": String( + "default", + ), "file.track.force": Boolean( false, ), "core.verbosity": String( "error", ), + "git.auto_commit": Boolean( + true, + ), "cache.algorithm": String( "blake3", ), @@ -4855,16 +5245,28 @@ process_pool_size = 4 }, ], the_config: { + "pipeline.current_pipeline": XvcConfigValue { + source: Project, + value: String( + "default", + ), + }, + "cache.algorithm": XvcConfigValue { + source: Project, + value: String( + "blake3", + ), + }, "pipeline.default_params_file": XvcConfigValue { source: Project, value: String( "params.yaml", ), }, - "file.list.recursive": XvcConfigValue { + "git.command": XvcConfigValue { source: Project, - value: Boolean( - false, + value: String( + "git", ), }, "file.track.no_commit": XvcConfigValue { @@ -4873,40 +5275,46 @@ process_pool_size = 4 false, ), }, - "file.carry-in.no_parallel": XvcConfigValue { + "core.verbosity": XvcConfigValue { + source: CommandLine, + value: String( + "debug", + ), + }, + "file.track.text_or_binary": XvcConfigValue { source: Project, - value: Boolean( - false, + value: String( + "auto", ), }, - "pipeline.current_pipeline": XvcConfigValue { + "core.guid": XvcConfigValue { source: Project, value: String( - "default", + "860d1874a44e1d89", ), }, - "file.track.force": XvcConfigValue { + "file.list.show_dot_files": XvcConfigValue { source: Project, value: Boolean( false, ), }, - "pipeline.default": XvcConfigValue { + "file.recheck.method": XvcConfigValue { source: Project, value: String( - "default", + "copy", ), }, - "cache.algorithm": XvcConfigValue { + "file.track.no_parallel": XvcConfigValue { source: Project, - value: String( - "blake3", + value: Boolean( + false, ), }, - "file.track.text_or_binary": XvcConfigValue { + "file.list.no_summary": XvcConfigValue { source: Project, - value: String( - "auto", + value: Boolean( + false, ), }, "TRYCMD_TESTS": XvcConfigValue { @@ -4915,31 +5323,19 @@ process_pool_size = 4 "storage,file,pipeline,core", ), }, - "pipeline.process_pool_size": XvcConfigValue { + "file.list.sort": XvcConfigValue { source: Project, - value: Integer( - 4, + value: String( + "name-desc", ), }, - "file.list.no_summary": XvcConfigValue { - source: Project, + "core.quiet": XvcConfigValue { + source: CommandLine, value: Boolean( false, ), }, - "core.guid": XvcConfigValue { - source: Project, - value: String( - "4bd7a8139a85fb10", - ), - }, - "file.list.format": XvcConfigValue { - source: Project, - value: String( - "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", - ), - }, - "file.carry-in.force": XvcConfigValue { + "git.auto_stage": XvcConfigValue { source: Project, value: Boolean( false, @@ -4951,58 +5347,52 @@ process_pool_size = 4 true, ), }, - "git.auto_stage": XvcConfigValue { + "file.carry-in.no_parallel": XvcConfigValue { source: Project, value: Boolean( false, ), }, - "file.list.show_dot_files": XvcConfigValue { + "pipeline.default": XvcConfigValue { source: Project, - value: Boolean( - false, + value: String( + "default", ), }, - "file.track.no_parallel": XvcConfigValue { + "file.list.recursive": XvcConfigValue { source: Project, value: Boolean( false, ), }, - "git.auto_commit": XvcConfigValue { + "file.track.force": XvcConfigValue { source: Project, value: Boolean( - true, + false, ), }, - "file.list.sort": XvcConfigValue { + "file.list.format": XvcConfigValue { source: Project, value: String( - "name-desc", + "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", ), }, - "file.recheck.method": XvcConfigValue { + "pipeline.process_pool_size": XvcConfigValue { source: Project, - value: String( - "copy", - ), - }, - "core.verbosity": XvcConfigValue { - source: CommandLine, - value: String( - "debug", + value: Integer( + 4, ), }, - "git.command": XvcConfigValue { + "file.carry-in.force": XvcConfigValue { source: Project, - value: String( - "git", + value: Boolean( + false, ), }, - "core.quiet": XvcConfigValue { - source: CommandLine, + "git.auto_commit": XvcConfigValue { + source: Project, value: Boolean( - false, + true, ), }, }, @@ -5012,7 +5402,7 @@ process_pool_size = 4 # The repository id. Please do not delete or change it. # This is used to identify the repository and generate paths in storages. # In the future it may be used to in other ways. -guid = /"7726b84aa14cc82c/" +guid = /"603c2d6a0d2e6ea9/" # Default verbosity level. # One of /"error/", /"warn/", /"info/" verbosity = /"error/" @@ -5163,7 +5553,7 @@ process_pool_size = 4 ), entity_generator: XvcEntityGenerator { counter: 3, - random: 6938067968196728862, + random: 5804399665853476265, dirty: false, }, }, @@ -5178,8 +5568,64 @@ process_pool_size = 4 ), modified: Some( SystemTime { - tv_sec: 1722022739, - tv_nsec: 33553054, + tv_sec: 1722024005, + tv_nsec: 902113188, + }, + ), + }, + XvcPath( + ".xvc/store/xvc-dependency-store", + ): XvcMetadata { + file_type: Directory, + size: Some( + 64, + ), + modified: Some( + SystemTime { + tv_sec: 1722024005, + tv_nsec: 893699709, + }, + ), + }, + XvcPath( + ".xvc/store/xvc-output-store", + ): XvcMetadata { + file_type: Directory, + size: Some( + 64, + ), + modified: Some( + SystemTime { + tv_sec: 1722024005, + tv_nsec: 893895957, + }, + ), + }, + XvcPath( + ".xvc/store/xvc-output-xvc-step-r1n-store", + ): XvcMetadata { + file_type: Directory, + size: Some( + 64, + ), + modified: Some( + SystemTime { + tv_sec: 1722024005, + tv_nsec: 893950706, + }, + ), + }, + XvcPath( + ".xvc/store/xvc-dependency-xvc-step-r1n-store", + ): XvcMetadata { + file_type: Directory, + size: Some( + 64, + ), + modified: Some( + SystemTime { + tv_sec: 1722024005, + tv_nsec: 893766208, }, ), }, @@ -5859,12 +6305,12 @@ process_pool_size = 4 name: "average-age", }, step_command: XvcStepCommand { - command: "sqlite3 people.db /"SELECT AVG(Age) FROM People;/"", + command: "sqlite3 people.db 'SELECT AVG(Age) FROM People;'", }, birth: Some( Instant { - tv_sec: 806688, - tv_nsec: 701336000, + tv_sec: 807955, + tv_nsec: 611297291, }, ), process: Some( @@ -5885,7 +6331,7 @@ process_pool_size = 4 }, ), child_state: Running { - pid: 19822, + pid: 29330, ext: (), }, detached: true, @@ -5921,20 +6367,20 @@ process_pool_size = 4 }, step_e: XvcEntity( 2, - 12972606829502460557, + 16393284406203488121, ), step: XvcStep { name: "average-age", }, step_command: XvcStepCommand { - command: "sqlite3 people.db /"SELECT AVG(Age) FROM People;/"", + command: "sqlite3 people.db 'SELECT AVG(Age) FROM People;'", }, current_states: RwLock { data: HStore { map: { XvcEntity( 2, - 12972606829502460557, + 16393284406203488121, ): Running( FromWaitProcess, ), @@ -5948,7 +6394,7 @@ process_pool_size = 4 map: { XvcEntity( 2, - 12972606829502460557, + 16393284406203488121, ): XvcStep { name: "average-age", }, @@ -5959,7 +6405,7 @@ process_pool_size = 4 map: { XvcEntity( 2, - 12972606829502460557, + 16393284406203488121, ): XvcStep { name: "average-age", }, @@ -5970,7 +6416,7 @@ process_pool_size = 4 }: [ XvcEntity( 2, - 12972606829502460557, + 16393284406203488121, ), ], }, @@ -5979,7 +6425,7 @@ process_pool_size = 4 Add { entity: XvcEntity( 2, - 12972606829502460557, + 16393284406203488121, ), value: XvcStep { name: "average-age", @@ -5988,7 +6434,7 @@ process_pool_size = 4 Add { entity: XvcEntity( 2, - 12972606829502460557, + 16393284406203488121, ), value: XvcStep { name: "average-age", @@ -6043,7 +6489,7 @@ process_pool_size = 4 }, ), child_state: Running { - pid: 19822, + pid: 29330, ext: (), }, detached: true, @@ -6065,12 +6511,16 @@ process_pool_size = 4 }, ), child_state: Running { - pid: 19822, + pid: 29330, ext: (), }, detached: true, } -[DEBUG] Step average-age with command sqlite3 people.db "SELECT AVG(Age) FROM People;" is still running +[DEBUG] Step average-age with command sqlite3 people.db 'SELECT AVG(Age) FROM People;' is still running +[TRACE][pipeline/src/pipeline/mod.rs::593] s: Running( + FromWaitProcess, +) +[TRACE][pipeline/src/pipeline/mod.rs::583] select: Select { .. } [WARN] [ERR] [average-age] Error: in prepare, no such table: People [TRACE][/Users/iex/.cargo/registry/src/index.crates.io-6f17d22bba15001f/notify-6.1.1/src/fsevent.rs::536] FSEvent: path = `[CWD]/people.db`, flag = StreamFlags(ITEM_CREATED | IS_FILE) @@ -6112,20 +6562,20 @@ process_pool_size = 4 ), modified: Ok( SystemTime { - tv_sec: 1722022739, - tv_nsec: 53602406, + tv_sec: 1722024005, + tv_nsec: 932064150, }, ), accessed: Ok( SystemTime { - tv_sec: 1722022739, - tv_nsec: 53602406, + tv_sec: 1722024005, + tv_nsec: 932064150, }, ), created: Ok( SystemTime { - tv_sec: 1722022739, - tv_nsec: 53602406, + tv_sec: 1722024005, + tv_nsec: 932064150, }, ), .. @@ -6151,8 +6601,8 @@ process_pool_size = 4 ), modified: Some( SystemTime { - tv_sec: 1722022739, - tv_nsec: 53602406, + tv_sec: 1722024005, + tv_nsec: 932064150, }, ), } @@ -6178,17 +6628,17 @@ process_pool_size = 4 }, ), child_state: Running { - pid: 19822, + pid: 29330, ext: (), }, detached: true, } -[ERROR] Step average-age finished UNSUCCESSFULLY with command sqlite3 people.db "SELECT AVG(Age) FROM People;" [TRACE][pipeline/src/pipeline/mod.rs::1515] return_state: Some( Broken( FromProcessReturnedNonZero, ), ) +[ERROR] Step average-age finished UNSUCCESSFULLY with command sqlite3 people.db 'SELECT AVG(Age) FROM People;' [TRACE][pipeline/src/pipeline/mod.rs::1521] params: StepStateParams { xvc_root: XvcRootInner { absolute_path: AbsolutePath( @@ -6211,87 +6661,93 @@ process_pool_size = 4 XvcConfigMap { source: Default, map: { - "file.list.sort": String( - "name-desc", - ), - "pipeline.process_pool_size": Integer( - 4, - ), - "git.command": String( - "git", - ), - "pipeline.current_pipeline": String( - "default", + "file.list.format": String( + "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", ), - "file.carry-in.force": Boolean( + "file.track.force": Boolean( false, ), - "pipeline.default_params_file": String( - "params.yaml", - ), "pipeline.default": String( "default", ), - "core.guid": String( - "7726b84aa14cc82c", + "git.auto_commit": Boolean( + true, + ), + "file.recheck.method": String( + "copy", + ), + "pipeline.process_pool_size": Integer( + 4, ), "git.auto_stage": Boolean( false, ), - "file.track.force": Boolean( - false, + "git.use_git": Boolean( + true, ), "cache.algorithm": String( "blake3", ), - "file.recheck.method": String( - "copy", + "file.track.no_parallel": Boolean( + false, ), - "core.verbosity": String( - "error", + "file.list.recursive": Boolean( + false, ), - "file.track.no_commit": Boolean( + "file.list.no_summary": Boolean( false, ), - "file.list.show_dot_files": Boolean( + "file.track.no_commit": Boolean( false, ), - "file.track.text_or_binary": String( - "auto", + "pipeline.current_pipeline": String( + "default", ), - "file.list.recursive": Boolean( + "file.carry-in.no_parallel": Boolean( false, ), - "file.list.no_summary": Boolean( + "core.guid": String( + "603c2d6a0d2e6ea9", + ), + "file.list.show_dot_files": Boolean( false, ), - "git.use_git": Boolean( - true, + "file.list.sort": String( + "name-desc", ), - "file.carry-in.no_parallel": Boolean( + "file.track.text_or_binary": String( + "auto", + ), + "file.carry-in.force": Boolean( false, ), - "file.list.format": String( - "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", + "pipeline.default_params_file": String( + "params.yaml", ), - "file.track.no_parallel": Boolean( - false, + "git.command": String( + "git", ), - "git.auto_commit": Boolean( - true, + "core.verbosity": String( + "error", ), }, }, XvcConfigMap { source: Project, map: { - "file.list.no_summary": Boolean( + "file.list.recursive": Boolean( false, ), - "file.list.sort": String( - "name-desc", + "pipeline.default": String( + "default", ), - "git.auto_commit": Boolean( + "pipeline.default_params_file": String( + "params.yaml", + ), + "file.track.no_commit": Boolean( + false, + ), + "git.use_git": Boolean( true, ), "git.auto_stage": Boolean( @@ -6300,44 +6756,32 @@ process_pool_size = 4 "file.track.no_parallel": Boolean( false, ), - "file.track.no_commit": Boolean( + "file.list.no_summary": Boolean( false, ), - "pipeline.process_pool_size": Integer( - 4, - ), - "pipeline.default_params_file": String( - "params.yaml", - ), - "pipeline.default": String( - "default", + "file.list.sort": String( + "name-desc", ), - "git.command": String( - "git", + "core.guid": String( + "860d1874a44e1d89", ), "file.list.show_dot_files": Boolean( false, ), - "file.list.format": String( - "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", - ), - "core.guid": String( - "4bd7a8139a85fb10", - ), - "pipeline.current_pipeline": String( - "default", + "git.command": String( + "git", ), "file.track.text_or_binary": String( "auto", ), + "pipeline.process_pool_size": Integer( + 4, + ), "file.recheck.method": String( "copy", ), - "git.use_git": Boolean( - true, - ), - "file.list.recursive": Boolean( - false, + "file.list.format": String( + "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", ), "file.carry-in.no_parallel": Boolean( false, @@ -6345,12 +6789,18 @@ process_pool_size = 4 "file.carry-in.force": Boolean( false, ), + "pipeline.current_pipeline": String( + "default", + ), "file.track.force": Boolean( false, ), "core.verbosity": String( "error", ), + "git.auto_commit": Boolean( + true, + ), "cache.algorithm": String( "blake3", ), @@ -6381,16 +6831,28 @@ process_pool_size = 4 }, ], the_config: { + "pipeline.current_pipeline": XvcConfigValue { + source: Project, + value: String( + "default", + ), + }, + "cache.algorithm": XvcConfigValue { + source: Project, + value: String( + "blake3", + ), + }, "pipeline.default_params_file": XvcConfigValue { source: Project, value: String( "params.yaml", ), }, - "file.list.recursive": XvcConfigValue { + "git.command": XvcConfigValue { source: Project, - value: Boolean( - false, + value: String( + "git", ), }, "file.track.no_commit": XvcConfigValue { @@ -6399,40 +6861,46 @@ process_pool_size = 4 false, ), }, - "file.carry-in.no_parallel": XvcConfigValue { + "core.verbosity": XvcConfigValue { + source: CommandLine, + value: String( + "debug", + ), + }, + "file.track.text_or_binary": XvcConfigValue { source: Project, - value: Boolean( - false, + value: String( + "auto", ), }, - "pipeline.current_pipeline": XvcConfigValue { + "core.guid": XvcConfigValue { source: Project, value: String( - "default", + "860d1874a44e1d89", ), }, - "file.track.force": XvcConfigValue { + "file.list.show_dot_files": XvcConfigValue { source: Project, value: Boolean( false, ), }, - "pipeline.default": XvcConfigValue { + "file.recheck.method": XvcConfigValue { source: Project, value: String( - "default", + "copy", ), }, - "cache.algorithm": XvcConfigValue { + "file.track.no_parallel": XvcConfigValue { source: Project, - value: String( - "blake3", + value: Boolean( + false, ), }, - "file.track.text_or_binary": XvcConfigValue { + "file.list.no_summary": XvcConfigValue { source: Project, - value: String( - "auto", + value: Boolean( + false, ), }, "TRYCMD_TESTS": XvcConfigValue { @@ -6441,31 +6909,19 @@ process_pool_size = 4 "storage,file,pipeline,core", ), }, - "pipeline.process_pool_size": XvcConfigValue { + "file.list.sort": XvcConfigValue { source: Project, - value: Integer( - 4, + value: String( + "name-desc", ), }, - "file.list.no_summary": XvcConfigValue { - source: Project, + "core.quiet": XvcConfigValue { + source: CommandLine, value: Boolean( false, ), }, - "core.guid": XvcConfigValue { - source: Project, - value: String( - "4bd7a8139a85fb10", - ), - }, - "file.list.format": XvcConfigValue { - source: Project, - value: String( - "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", - ), - }, - "file.carry-in.force": XvcConfigValue { + "git.auto_stage": XvcConfigValue { source: Project, value: Boolean( false, @@ -6477,58 +6933,52 @@ process_pool_size = 4 true, ), }, - "git.auto_stage": XvcConfigValue { + "file.carry-in.no_parallel": XvcConfigValue { source: Project, value: Boolean( false, ), }, - "file.list.show_dot_files": XvcConfigValue { + "pipeline.default": XvcConfigValue { source: Project, - value: Boolean( - false, + value: String( + "default", ), }, - "file.track.no_parallel": XvcConfigValue { + "file.list.recursive": XvcConfigValue { source: Project, value: Boolean( false, ), }, - "git.auto_commit": XvcConfigValue { + "file.track.force": XvcConfigValue { source: Project, value: Boolean( - true, + false, ), }, - "file.list.sort": XvcConfigValue { + "file.list.format": XvcConfigValue { source: Project, value: String( - "name-desc", + "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", ), }, - "file.recheck.method": XvcConfigValue { + "pipeline.process_pool_size": XvcConfigValue { source: Project, - value: String( - "copy", - ), - }, - "core.verbosity": XvcConfigValue { - source: CommandLine, - value: String( - "debug", + value: Integer( + 4, ), }, - "git.command": XvcConfigValue { + "file.carry-in.force": XvcConfigValue { source: Project, - value: String( - "git", + value: Boolean( + false, ), }, - "core.quiet": XvcConfigValue { - source: CommandLine, + "git.auto_commit": XvcConfigValue { + source: Project, value: Boolean( - false, + true, ), }, }, @@ -6538,7 +6988,7 @@ process_pool_size = 4 # The repository id. Please do not delete or change it. # This is used to identify the repository and generate paths in storages. # In the future it may be used to in other ways. -guid = /"7726b84aa14cc82c/" +guid = /"603c2d6a0d2e6ea9/" # Default verbosity level. # One of /"error/", /"warn/", /"info/" verbosity = /"error/" @@ -6689,7 +7139,7 @@ process_pool_size = 4 ), entity_generator: XvcEntityGenerator { counter: 3, - random: 6938067968196728862, + random: 5804399665853476265, dirty: false, }, }, @@ -6716,87 +7166,93 @@ process_pool_size = 4 XvcConfigMap { source: Default, map: { - "file.list.sort": String( - "name-desc", - ), - "pipeline.process_pool_size": Integer( - 4, - ), - "git.command": String( - "git", - ), - "pipeline.current_pipeline": String( - "default", + "file.list.format": String( + "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", ), - "file.carry-in.force": Boolean( + "file.track.force": Boolean( false, ), - "pipeline.default_params_file": String( - "params.yaml", - ), "pipeline.default": String( "default", ), - "core.guid": String( - "7726b84aa14cc82c", + "git.auto_commit": Boolean( + true, + ), + "file.recheck.method": String( + "copy", + ), + "pipeline.process_pool_size": Integer( + 4, ), "git.auto_stage": Boolean( false, ), - "file.track.force": Boolean( - false, + "git.use_git": Boolean( + true, ), "cache.algorithm": String( "blake3", ), - "file.recheck.method": String( - "copy", - ), - "core.verbosity": String( - "error", - ), - "file.track.no_commit": Boolean( - false, - ), - "file.list.show_dot_files": Boolean( + "file.track.no_parallel": Boolean( false, ), - "file.track.text_or_binary": String( - "auto", - ), "file.list.recursive": Boolean( false, ), "file.list.no_summary": Boolean( false, ), - "git.use_git": Boolean( - true, + "file.track.no_commit": Boolean( + false, + ), + "pipeline.current_pipeline": String( + "default", ), "file.carry-in.no_parallel": Boolean( false, ), - "file.list.format": String( - "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", + "core.guid": String( + "603c2d6a0d2e6ea9", ), - "file.track.no_parallel": Boolean( + "file.list.show_dot_files": Boolean( false, ), - "git.auto_commit": Boolean( - true, + "file.list.sort": String( + "name-desc", + ), + "file.track.text_or_binary": String( + "auto", + ), + "file.carry-in.force": Boolean( + false, + ), + "pipeline.default_params_file": String( + "params.yaml", + ), + "git.command": String( + "git", + ), + "core.verbosity": String( + "error", ), }, }, XvcConfigMap { source: Project, map: { - "file.list.no_summary": Boolean( + "file.list.recursive": Boolean( false, ), - "file.list.sort": String( - "name-desc", + "pipeline.default": String( + "default", ), - "git.auto_commit": Boolean( + "pipeline.default_params_file": String( + "params.yaml", + ), + "file.track.no_commit": Boolean( + false, + ), + "git.use_git": Boolean( true, ), "git.auto_stage": Boolean( @@ -6805,44 +7261,32 @@ process_pool_size = 4 "file.track.no_parallel": Boolean( false, ), - "file.track.no_commit": Boolean( + "file.list.no_summary": Boolean( false, ), - "pipeline.process_pool_size": Integer( - 4, - ), - "pipeline.default_params_file": String( - "params.yaml", - ), - "pipeline.default": String( - "default", + "file.list.sort": String( + "name-desc", ), - "git.command": String( - "git", + "core.guid": String( + "860d1874a44e1d89", ), "file.list.show_dot_files": Boolean( false, ), - "file.list.format": String( - "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", - ), - "core.guid": String( - "4bd7a8139a85fb10", - ), - "pipeline.current_pipeline": String( - "default", + "git.command": String( + "git", ), "file.track.text_or_binary": String( "auto", ), + "pipeline.process_pool_size": Integer( + 4, + ), "file.recheck.method": String( "copy", ), - "git.use_git": Boolean( - true, - ), - "file.list.recursive": Boolean( - false, + "file.list.format": String( + "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", ), "file.carry-in.no_parallel": Boolean( false, @@ -6850,12 +7294,18 @@ process_pool_size = 4 "file.carry-in.force": Boolean( false, ), + "pipeline.current_pipeline": String( + "default", + ), "file.track.force": Boolean( false, ), "core.verbosity": String( "error", ), + "git.auto_commit": Boolean( + true, + ), "cache.algorithm": String( "blake3", ), @@ -6886,16 +7336,28 @@ process_pool_size = 4 }, ], the_config: { + "pipeline.current_pipeline": XvcConfigValue { + source: Project, + value: String( + "default", + ), + }, + "cache.algorithm": XvcConfigValue { + source: Project, + value: String( + "blake3", + ), + }, "pipeline.default_params_file": XvcConfigValue { source: Project, value: String( "params.yaml", ), }, - "file.list.recursive": XvcConfigValue { + "git.command": XvcConfigValue { source: Project, - value: Boolean( - false, + value: String( + "git", ), }, "file.track.no_commit": XvcConfigValue { @@ -6904,40 +7366,46 @@ process_pool_size = 4 false, ), }, - "file.carry-in.no_parallel": XvcConfigValue { + "core.verbosity": XvcConfigValue { + source: CommandLine, + value: String( + "debug", + ), + }, + "file.track.text_or_binary": XvcConfigValue { source: Project, - value: Boolean( - false, + value: String( + "auto", ), }, - "pipeline.current_pipeline": XvcConfigValue { + "core.guid": XvcConfigValue { source: Project, value: String( - "default", + "860d1874a44e1d89", ), }, - "file.track.force": XvcConfigValue { + "file.list.show_dot_files": XvcConfigValue { source: Project, value: Boolean( false, ), }, - "pipeline.default": XvcConfigValue { + "file.recheck.method": XvcConfigValue { source: Project, value: String( - "default", + "copy", ), }, - "cache.algorithm": XvcConfigValue { + "file.track.no_parallel": XvcConfigValue { source: Project, - value: String( - "blake3", + value: Boolean( + false, ), }, - "file.track.text_or_binary": XvcConfigValue { + "file.list.no_summary": XvcConfigValue { source: Project, - value: String( - "auto", + value: Boolean( + false, ), }, "TRYCMD_TESTS": XvcConfigValue { @@ -6946,31 +7414,19 @@ process_pool_size = 4 "storage,file,pipeline,core", ), }, - "pipeline.process_pool_size": XvcConfigValue { + "file.list.sort": XvcConfigValue { source: Project, - value: Integer( - 4, + value: String( + "name-desc", ), }, - "file.list.no_summary": XvcConfigValue { - source: Project, + "core.quiet": XvcConfigValue { + source: CommandLine, value: Boolean( false, ), }, - "core.guid": XvcConfigValue { - source: Project, - value: String( - "4bd7a8139a85fb10", - ), - }, - "file.list.format": XvcConfigValue { - source: Project, - value: String( - "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", - ), - }, - "file.carry-in.force": XvcConfigValue { + "git.auto_stage": XvcConfigValue { source: Project, value: Boolean( false, @@ -6982,58 +7438,52 @@ process_pool_size = 4 true, ), }, - "git.auto_stage": XvcConfigValue { + "file.carry-in.no_parallel": XvcConfigValue { source: Project, value: Boolean( false, ), }, - "file.list.show_dot_files": XvcConfigValue { + "pipeline.default": XvcConfigValue { source: Project, - value: Boolean( - false, + value: String( + "default", ), }, - "file.track.no_parallel": XvcConfigValue { + "file.list.recursive": XvcConfigValue { source: Project, value: Boolean( false, ), }, - "git.auto_commit": XvcConfigValue { + "file.track.force": XvcConfigValue { source: Project, value: Boolean( - true, + false, ), }, - "file.list.sort": XvcConfigValue { + "file.list.format": XvcConfigValue { source: Project, value: String( - "name-desc", + "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", ), }, - "file.recheck.method": XvcConfigValue { + "pipeline.process_pool_size": XvcConfigValue { source: Project, - value: String( - "copy", - ), - }, - "core.verbosity": XvcConfigValue { - source: CommandLine, - value: String( - "debug", + value: Integer( + 4, ), }, - "git.command": XvcConfigValue { + "file.carry-in.force": XvcConfigValue { source: Project, - value: String( - "git", + value: Boolean( + false, ), }, - "core.quiet": XvcConfigValue { - source: CommandLine, + "git.auto_commit": XvcConfigValue { + source: Project, value: Boolean( - false, + true, ), }, }, @@ -7043,7 +7493,7 @@ process_pool_size = 4 # The repository id. Please do not delete or change it. # This is used to identify the repository and generate paths in storages. # In the future it may be used to in other ways. -guid = /"7726b84aa14cc82c/" +guid = /"603c2d6a0d2e6ea9/" # Default verbosity level. # One of /"error/", /"warn/", /"info/" verbosity = /"error/" @@ -7194,12 +7644,26 @@ process_pool_size = 4 ), entity_generator: XvcEntityGenerator { counter: 3, - random: 6938067968196728862, + random: 5804399665853476265, dirty: false, }, }, path_map: RwLock { data: { + XvcPath( + "people.db", + ): XvcMetadata { + file_type: File, + size: Some( + 0, + ), + modified: Some( + SystemTime { + tv_sec: 1722024005, + tv_nsec: 932064150, + }, + ), + }, XvcPath( ".xvc/store/xvc-pipeline-run-dir-store", ): XvcMetadata { @@ -7209,22 +7673,64 @@ process_pool_size = 4 ), modified: Some( SystemTime { - tv_sec: 1722022739, - tv_nsec: 33553054, + tv_sec: 1722024005, + tv_nsec: 902113188, }, ), }, XvcPath( - "people.db", + ".xvc/store/xvc-dependency-store", ): XvcMetadata { - file_type: File, + file_type: Directory, size: Some( - 0, + 64, + ), + modified: Some( + SystemTime { + tv_sec: 1722024005, + tv_nsec: 893699709, + }, + ), + }, + XvcPath( + ".xvc/store/xvc-output-store", + ): XvcMetadata { + file_type: Directory, + size: Some( + 64, + ), + modified: Some( + SystemTime { + tv_sec: 1722024005, + tv_nsec: 893895957, + }, + ), + }, + XvcPath( + ".xvc/store/xvc-output-xvc-step-r1n-store", + ): XvcMetadata { + file_type: Directory, + size: Some( + 64, + ), + modified: Some( + SystemTime { + tv_sec: 1722024005, + tv_nsec: 893950706, + }, + ), + }, + XvcPath( + ".xvc/store/xvc-dependency-xvc-step-r1n-store", + ): XvcMetadata { + file_type: Directory, + size: Some( + 64, ), modified: Some( SystemTime { - tv_sec: 1722022739, - tv_nsec: 53602406, + tv_sec: 1722024005, + tv_nsec: 893766208, }, ), }, @@ -7904,12 +8410,12 @@ process_pool_size = 4 name: "average-age", }, step_command: XvcStepCommand { - command: "sqlite3 people.db /"SELECT AVG(Age) FROM People;/"", + command: "sqlite3 people.db 'SELECT AVG(Age) FROM People;'", }, birth: Some( Instant { - tv_sec: 806688, - tv_nsec: 701336000, + tv_sec: 807955, + tv_nsec: 611297291, }, ), process: Some( @@ -7967,20 +8473,20 @@ process_pool_size = 4 }, step_e: XvcEntity( 2, - 12972606829502460557, + 16393284406203488121, ), step: XvcStep { name: "average-age", }, step_command: XvcStepCommand { - command: "sqlite3 people.db /"SELECT AVG(Age) FROM People;/"", + command: "sqlite3 people.db 'SELECT AVG(Age) FROM People;'", }, current_states: RwLock { data: HStore { map: { XvcEntity( 2, - 12972606829502460557, + 16393284406203488121, ): Running( FromWaitProcess, ), @@ -7994,7 +8500,7 @@ process_pool_size = 4 map: { XvcEntity( 2, - 12972606829502460557, + 16393284406203488121, ): XvcStep { name: "average-age", }, @@ -8005,7 +8511,7 @@ process_pool_size = 4 map: { XvcEntity( 2, - 12972606829502460557, + 16393284406203488121, ): XvcStep { name: "average-age", }, @@ -8016,7 +8522,7 @@ process_pool_size = 4 }: [ XvcEntity( 2, - 12972606829502460557, + 16393284406203488121, ), ], }, @@ -8025,7 +8531,7 @@ process_pool_size = 4 Add { entity: XvcEntity( 2, - 12972606829502460557, + 16393284406203488121, ), value: XvcStep { name: "average-age", @@ -8034,7 +8540,7 @@ process_pool_size = 4 Add { entity: XvcEntity( 2, - 12972606829502460557, + 16393284406203488121, ), value: XvcStep { name: "average-age", @@ -8092,7 +8598,7 @@ process_pool_size = 4 map: { XvcEntity( 2, - 12972606829502460557, + 16393284406203488121, ): Broken( FromProcessReturnedNonZero, ), @@ -8116,7 +8622,6 @@ process_pool_size = 4 } [TRACE][lib/src/cli/mod.rs::376] "Before handle_git_automation": "Before handle_git_automation" [TRACE][core/src/util/pmp.rs::99] index: 1 -[DEBUG] Using Git: /opt/homebrew/bin/git [TRACE][lib/src/git.rs::30] args: [ "-C", "[CWD]", @@ -8124,6 +8629,7 @@ process_pool_size = 4 "--name-only", "--cached", ] +[DEBUG] Using Git: /opt/homebrew/bin/git [TRACE][lib/src/git.rs::61] git_diff_staged_out: "" [TRACE][lib/src/git.rs::30] args: [ "-C", From 29f932904238d6326f88e06b8e13dcce8f32690a Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Fri, 26 Jul 2024 23:17:31 +0300 Subject: [PATCH 069/257] added people.db --- .../people.db | Bin 0 -> 8192 bytes 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 workflow_tests/templates/ref-xvc-pipeline-step-dependency-sqlite-query.in/people.db diff --git a/workflow_tests/templates/ref-xvc-pipeline-step-dependency-sqlite-query.in/people.db b/workflow_tests/templates/ref-xvc-pipeline-step-dependency-sqlite-query.in/people.db new file mode 100644 index 0000000000000000000000000000000000000000..3da398aa68d3899afe59d2b764c5952153f6ef6e GIT binary patch literal 8192 zcmeI#&1w@-6u|Lw$9`yR=hT9hiq@-$wNmOr#GQ1K&ds#Vjcq!huH*);>Db9sCzIHq zo8rPn1#zv2ASj5S58x|w(+6zdvAWAhqTE>=Uo?APdVV57!*b}vf)&bKMY_0~Zz%=G@7kGp$-Kk+;6;y3(^Z}0^^ z!3TI31H6ILcnSyIU+$0m5DyEm01L1H3$OqSumB6N01L1H3;Z(zI6ia|g>Y)^Y+ar4 z;2EBvu-c9ieYOl=ag4%>X;^ix1kVvE+-jOsUsCW2T?$nj#ronhymLqULFjI3)Wpp? zj?jThd&BAr0ldfK6qZe9)m0yMus~tSJ%w^> literal 0 HcmV?d00001 From d33147aed397271ad3f43f0a035e1747a3a450a6 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Fri, 26 Jul 2024 23:19:54 +0300 Subject: [PATCH 070/257] added people --- .../xvc-pipeline-step-dependency-sqlite.md | 8575 +---------------- 1 file changed, 1 insertion(+), 8574 deletions(-) diff --git a/book/src/ref/xvc-pipeline-step-dependency-sqlite.md b/book/src/ref/xvc-pipeline-step-dependency-sqlite.md index 58c31036f..ae454ca56 100644 --- a/book/src/ref/xvc-pipeline-step-dependency-sqlite.md +++ b/book/src/ref/xvc-pipeline-step-dependency-sqlite.md @@ -59,8590 +59,17 @@ Let's run the step without a dependency first. ```console $ xvc -vv pipeline run -[DEBUG][logging/src/lib.rs::237] Terminal logger enabled with level: Trace -[TRACE][core/src/types/xvcroot.rs::263] xvc_candidate: "[CWD]/.xvc" -[TRACE][core/src/types/xvcroot.rs::265] parent: "[CWD]" -[DEBUG][config/src/error.rs::72] Config source for level "system" not found at "/Users/iex/Library/Application Support/com.emresult.xvc" -[DEBUG][config/src/error.rs::72] Config source for level "global" not found at "/Users/iex/Library/Application Support/xvc" -[TRACE][ecs/src/ecs/mod.rs::229] dir: "[CWD]/.xvc/ec" -[TRACE][ecs/src/ecs/mod.rs::239] files: [ - "[CWD]/.xvc/ec/1722024005715237", - "[CWD]/.xvc/ec/1722024005718590", - "[CWD]/.xvc/ec/1722024005797771", -] -[TRACE][core/src/types/xvcroot.rs::263] xvc_candidate: "[CWD]/.xvc" -[TRACE][core/src/types/xvcroot.rs::265] parent: "[CWD]" -[TRACE][pipeline/src/pipeline/mod.rs::285] pipeline_e: XvcEntity( - 1, - 2727635544075685959, -) -[TRACE][pipeline/src/pipeline/mod.rs::290] pipeline_steps: HStore { - map: { - XvcEntity( - 2, - 16393284406203488121, - ): XvcStep { - name: "average-age", - }, - }, -} -[TRACE][pipeline/src/pipeline/mod.rs::293] consider_changed: XvcStore { - map: { - XvcEntity( - 2, - 16393284406203488121, - ): ByDependencies, - }, - entity_index: { - ByDependencies: [ - XvcEntity( - 2, - 16393284406203488121, - ), - ], - }, - previous: EventLog( - [ - Add { - entity: XvcEntity( - 2, - 16393284406203488121, - ), - value: ByDependencies, - }, - ], - ), - current: EventLog( - [], - ), -} -[TRACE][pipeline/src/pipeline/mod.rs::296] all_deps.parents.len(): 1 -[TRACE][pipeline/src/pipeline/mod.rs::297] all_deps.children.len(): 0 -[TRACE][pipeline/src/pipeline/mod.rs::299] all_outs.parents.len(): 1 -[TRACE][pipeline/src/pipeline/mod.rs::300] all_outs.children.len(): 0 -[DEBUG][/Users/iex/.cargo/registry/src/index.crates.io-6f17d22bba15001f/globset-0.4.14/src/lib.rs::453] built glob set; 0 literals, 2 basenames, 0 extensions, 0 prefixes, 0 suffixes, 0 required extensions, 0 regexes -[TRACE][walker/src/lib.rs::488] ignore_fn: ".xvcignore" -[TRACE][walker/src/lib.rs::490] ignore_root: "[CWD]" -[TRACE][walker/src/lib.rs::498] entry.path(): "[CWD]/.xvc" -[TRACE][walker/src/lib.rs::498] entry.path(): "[CWD]/.git" -[TRACE][walker/src/lib.rs::503] ignore_path: "[CWD]/.xvcignore" -[TRACE][walker/src/lib.rs::597] ignore_root: "[CWD]" -[TRACE][walker/src/lib.rs::598] ignore_path: "[CWD]/.xvcignore" -[TRACE][walker/src/lib.rs::606] &content: " -# Add patterns of files xvc should ignore, which could improve -# the performance. -# It's in the same format as .gitignore files. - -.DS_Store -" -[DEBUG][/Users/iex/.cargo/registry/src/index.crates.io-6f17d22bba15001f/globset-0.4.14/src/lib.rs::453] built glob set; 0 literals, 3 basenames, 0 extensions, 0 prefixes, 0 suffixes, 0 required extensions, 0 regexes -[TRACE][walker/src/lib.rs::748] is_abs: true -[TRACE][walker/src/lib.rs::752] path_str: "[CWD]/.xvc" -[TRACE][walker/src/lib.rs::754] final_slash: false -[TRACE][walker/src/lib.rs::776] path: "/.xvc" -[TRACE][walker/src/lib.rs::748] is_abs: true -[TRACE][walker/src/lib.rs::752] path_str: "[CWD]/.git" -[TRACE][walker/src/lib.rs::754] final_slash: false -[TRACE][walker/src/lib.rs::776] path: "/.git" -[TRACE][core/src/util/pmp.rs::41] ignore_rules: IgnoreRules { - root: "[CWD]", - ignore_patterns: RwLock { - data: [ - Pattern { - pattern: Glob { - glob: "**/.xvc", - re: "(?-u)^(?:/?|.*/)//.xvc$", - opts: GlobOptions { - case_insensitive: false, - literal_separator: false, - backslash_escape: true, - empty_alternates: false, - }, - tokens: Tokens( - [ - RecursivePrefix, - Literal( - '.', - ), - Literal( - 'x', - ), - Literal( - 'v', - ), - Literal( - 'c', - ), - ], - ), - }, - original: ".xvc", - source: Global, - effect: Ignore, - relativity: Anywhere, - path_kind: Any, - }, - Pattern { - pattern: Glob { - glob: "**/.git", - re: "(?-u)^(?:/?|.*/)//.git$", - opts: GlobOptions { - case_insensitive: false, - literal_separator: false, - backslash_escape: true, - empty_alternates: false, - }, - tokens: Tokens( - [ - RecursivePrefix, - Literal( - '.', - ), - Literal( - 'g', - ), - Literal( - 'i', - ), - Literal( - 't', - ), - ], - ), - }, - original: ".git", - source: Global, - effect: Ignore, - relativity: Anywhere, - path_kind: Any, - }, - Pattern { - pattern: Glob { - glob: "**/.DS_Store", - re: "(?-u)^(?:/?|.*/)//.DS_Store$", - opts: GlobOptions { - case_insensitive: false, - literal_separator: false, - backslash_escape: true, - empty_alternates: false, - }, - tokens: Tokens( - [ - RecursivePrefix, - Literal( - '.', - ), - Literal( - 'D', - ), - Literal( - 'S', - ), - Literal( - '_', - ), - Literal( - 'S', - ), - Literal( - 't', - ), - Literal( - 'o', - ), - Literal( - 'r', - ), - Literal( - 'e', - ), - ], - ), - }, - original: ".DS_Store", - source: File { - path: ".xvcignore", - line: 6, - }, - effect: Ignore, - relativity: Anywhere, - path_kind: Any, - }, - ], - poisoned: false, - .. - }, - whitelist_patterns: RwLock { - data: [], - poisoned: false, - .. - }, - whitelist_set: RwLock { - data: GlobSet { - len: 0, - strats: [], - }, - poisoned: false, - .. - }, - ignore_set: RwLock { - data: GlobSet { - len: 3, - strats: [ - Extension( - ExtensionStrategy( - {}, - ), - ), - BasenameLiteral( - BasenameLiteralStrategy( - { - [ - 46, - 120, - 118, - 99, - ]: [ - 0, - ], - [ - 46, - 68, - 83, - 95, - 83, - 116, - 111, - 114, - 101, - ]: [ - 2, - ], - [ - 46, - 103, - 105, - 116, - ]: [ - 1, - ], - }, - ), - ), - Literal( - LiteralStrategy( - {}, - ), - ), - Suffix( - SuffixStrategy { - matcher: AhoCorasick( - dfa::DFA( - D 000000: /x00 => 0 - F 000001: - >000002: /x00 => 2 - 000003: /x00 => 0 - match kind: Standard - prefilter: false - state length: 4 - pattern length: 0 - shortest pattern length: 18446744073709551615 - longest pattern length: 0 - alphabet length: 1 - stride: 1 - byte classes: ByteClasses(0 => [0-255]) - memory usage: 16 - ) - , - ), - map: [], - longest: 0, - }, - ), - Prefix( - PrefixStrategy { - matcher: AhoCorasick( - dfa::DFA( - D 000000: /x00 => 0 - F 000001: - >000002: /x00 => 2 - 000003: /x00 => 0 - match kind: Standard - prefilter: false - state length: 4 - pattern length: 0 - shortest pattern length: 18446744073709551615 - longest pattern length: 0 - alphabet length: 1 - stride: 1 - byte classes: ByteClasses(0 => [0-255]) - memory usage: 16 - ) - , - ), - map: [], - longest: 0, - }, - ), - RequiredExtension( - RequiredExtensionStrategy( - {}, - ), - ), - Regex( - RegexSetStrategy { - matcher: Regex { - imp: RegexI { - strat: Core { - info: RegexInfo( - RegexInfoI { - config: Config { - match_kind: Some( - All, - ), - utf8_empty: Some( - false, - ), - autopre: None, - pre: None, - which_captures: None, - nfa_size_limit: Some( - Some( - 10485760, - ), - ), - onepass_size_limit: None, - hybrid_cache_capacity: Some( - 10485760, - ), - hybrid: None, - dfa: None, - dfa_size_limit: None, - dfa_state_limit: None, - onepass: None, - backtrack: None, - byte_classes: None, - line_terminator: None, - }, - props: [], - props_union: Properties( - PropertiesI { - minimum_len: None, - maximum_len: None, - look_set: ∅, - look_set_prefix: ∅, - look_set_suffix: ∅, - look_set_prefix_any: ∅, - look_set_suffix_any: ∅, - utf8: true, - explicit_captures_len: 0, - static_explicit_captures_len: None, - literal: false, - alternation_literal: true, - }, - ), - }, - ), - pre: None, - nfa: thompson::NFA( - ^000000: FAIL - - transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) - ) - , - nfarev: Some( - thompson::NFA( - ^000000: FAIL - - transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) - ) - , - ), - pikevm: PikeVM( - PikeVMEngine( - PikeVM { - config: Config { - match_kind: Some( - All, - ), - pre: Some( - None, - ), - }, - nfa: thompson::NFA( - ^000000: FAIL - - transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) - ) - , - }, - ), - ), - backtrack: BoundedBacktracker( - None, - ), - onepass: OnePass( - None, - ), - hybrid: Hybrid( - Some( - HybridEngine( - Regex { - forward: DFA { - config: Config { - match_kind: Some( - All, - ), - pre: Some( - None, - ), - starts_for_each_pattern: Some( - true, - ), - byte_classes: Some( - true, - ), - unicode_word_boundary: Some( - true, - ), - quitset: None, - specialize_start_states: Some( - false, - ), - cache_capacity: Some( - 10485760, - ), - skip_cache_capacity_check: Some( - false, - ), - minimum_cache_clear_count: Some( - Some( - 3, - ), - ), - minimum_bytes_per_state: Some( - Some( - 10, - ), - ), - }, - nfa: thompson::NFA( - ^000000: FAIL - - transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) - ) - , - stride2: 1, - start_map: StartByteMap{/x00 => NonWordByte, /x01 => NonWordByte, /x02 => NonWordByte, /x03 => NonWordByte, /x04 => NonWordByte, /x05 => NonWordByte, /x06 => NonWordByte, /x07 => NonWordByte, /x08 => NonWordByte, /t => NonWordByte, - => LineLF, /x0B => NonWordByte, /x0C => NonWordByte, /r => LineCR, /x0E => NonWordByte, /x0F => NonWordByte, /x10 => NonWordByte, /x11 => NonWordByte, /x12 => NonWordByte, /x13 => NonWordByte, /x14 => NonWordByte, /x15 => NonWordByte, /x16 => NonWordByte, /x17 => NonWordByte, /x18 => NonWordByte, /x19 => NonWordByte, /x1A => NonWordByte, /x1B => NonWordByte, /x1C => NonWordByte, /x1D => NonWordByte, /x1E => NonWordByte, /x1F => NonWordByte, ' ' => NonWordByte, ! => NonWordByte, /" => NonWordByte, # => NonWordByte, $ => NonWordByte, % => NonWordByte, & => NonWordByte, /' => NonWordByte, ( => NonWordByte, ) => NonWordByte, * => NonWordByte, + => NonWordByte, , => NonWordByte, - => NonWordByte, . => NonWordByte, / => NonWordByte, 0 => WordByte, 1 => WordByte, 2 => WordByte, 3 => WordByte, 4 => WordByte, 5 => WordByte, 6 => WordByte, 7 => WordByte, 8 => WordByte, 9 => WordByte, : => NonWordByte, ; => NonWordByte, < => NonWordByte, = => NonWordByte, > => NonWordByte, ? => NonWordByte, @ => NonWordByte, A => WordByte, B => WordByte, C => WordByte, D => WordByte, E => WordByte, F => WordByte, G => WordByte, H => WordByte, I => WordByte, J => WordByte, K => WordByte, L => WordByte, M => WordByte, N => WordByte, O => WordByte, P => WordByte, Q => WordByte, R => WordByte, S => WordByte, T => WordByte, U => WordByte, V => WordByte, W => WordByte, X => WordByte, Y => WordByte, Z => WordByte, [ => NonWordByte, // => NonWordByte, ] => NonWordByte, ^ => NonWordByte, _ => WordByte, ` => NonWordByte, a => WordByte, b => WordByte, c => WordByte, d => WordByte, e => WordByte, f => WordByte, g => WordByte, h => WordByte, i => WordByte, j => WordByte, k => WordByte, l => WordByte, m => WordByte, n => WordByte, o => WordByte, p => WordByte, q => WordByte, r => WordByte, s => WordByte, t => WordByte, u => WordByte, v => WordByte, w => WordByte, x => WordByte, y => WordByte, z => WordByte, { => NonWordByte, | => NonWordByte, } => NonWordByte, ~ => NonWordByte, /x7F => NonWordByte, /x80 => NonWordByte, /x81 => NonWordByte, /x82 => NonWordByte, /x83 => NonWordByte, /x84 => NonWordByte, /x85 => NonWordByte, /x86 => NonWordByte, /x87 => NonWordByte, /x88 => NonWordByte, /x89 => NonWordByte, /x8A => NonWordByte, /x8B => NonWordByte, /x8C => NonWordByte, /x8D => NonWordByte, /x8E => NonWordByte, /x8F => NonWordByte, /x90 => NonWordByte, /x91 => NonWordByte, /x92 => NonWordByte, /x93 => NonWordByte, /x94 => NonWordByte, /x95 => NonWordByte, /x96 => NonWordByte, /x97 => NonWordByte, /x98 => NonWordByte, /x99 => NonWordByte, /x9A => NonWordByte, /x9B => NonWordByte, /x9C => NonWordByte, /x9D => NonWordByte, /x9E => NonWordByte, /x9F => NonWordByte, /xA0 => NonWordByte, /xA1 => NonWordByte, /xA2 => NonWordByte, /xA3 => NonWordByte, /xA4 => NonWordByte, /xA5 => NonWordByte, /xA6 => NonWordByte, /xA7 => NonWordByte, /xA8 => NonWordByte, /xA9 => NonWordByte, /xAA => NonWordByte, /xAB => NonWordByte, /xAC => NonWordByte, /xAD => NonWordByte, /xAE => NonWordByte, /xAF => NonWordByte, /xB0 => NonWordByte, /xB1 => NonWordByte, /xB2 => NonWordByte, /xB3 => NonWordByte, /xB4 => NonWordByte, /xB5 => NonWordByte, /xB6 => NonWordByte, /xB7 => NonWordByte, /xB8 => NonWordByte, /xB9 => NonWordByte, /xBA => NonWordByte, /xBB => NonWordByte, /xBC => NonWordByte, /xBD => NonWordByte, /xBE => NonWordByte, /xBF => NonWordByte, /xC0 => NonWordByte, /xC1 => NonWordByte, /xC2 => NonWordByte, /xC3 => NonWordByte, /xC4 => NonWordByte, /xC5 => NonWordByte, /xC6 => NonWordByte, /xC7 => NonWordByte, /xC8 => NonWordByte, /xC9 => NonWordByte, /xCA => NonWordByte, /xCB => NonWordByte, /xCC => NonWordByte, /xCD => NonWordByte, /xCE => NonWordByte, /xCF => NonWordByte, /xD0 => NonWordByte, /xD1 => NonWordByte, /xD2 => NonWordByte, /xD3 => NonWordByte, /xD4 => NonWordByte, /xD5 => NonWordByte, /xD6 => NonWordByte, /xD7 => NonWordByte, /xD8 => NonWordByte, /xD9 => NonWordByte, /xDA => NonWordByte, /xDB => NonWordByte, /xDC => NonWordByte, /xDD => NonWordByte, /xDE => NonWordByte, /xDF => NonWordByte, /xE0 => NonWordByte, /xE1 => NonWordByte, /xE2 => NonWordByte, /xE3 => NonWordByte, /xE4 => NonWordByte, /xE5 => NonWordByte, /xE6 => NonWordByte, /xE7 => NonWordByte, /xE8 => NonWordByte, /xE9 => NonWordByte, /xEA => NonWordByte, /xEB => NonWordByte, /xEC => NonWordByte, /xED => NonWordByte, /xEE => NonWordByte, /xEF => NonWordByte, /xF0 => NonWordByte, /xF1 => NonWordByte, /xF2 => NonWordByte, /xF3 => NonWordByte, /xF4 => NonWordByte, /xF5 => NonWordByte, /xF6 => NonWordByte, /xF7 => NonWordByte, /xF8 => NonWordByte, /xF9 => NonWordByte, /xFA => NonWordByte, /xFB => NonWordByte, /xFC => NonWordByte, /xFD => NonWordByte, /xFE => NonWordByte, /xFF => NonWordByte}, - classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]), - quitset: ByteSet { - bits: {}, - }, - cache_capacity: 10485760, - }, - reverse: DFA { - config: Config { - match_kind: Some( - All, - ), - pre: Some( - None, - ), - starts_for_each_pattern: Some( - true, - ), - byte_classes: Some( - true, - ), - unicode_word_boundary: Some( - true, - ), - quitset: None, - specialize_start_states: Some( - false, - ), - cache_capacity: Some( - 10485760, - ), - skip_cache_capacity_check: Some( - false, - ), - minimum_cache_clear_count: Some( - Some( - 3, - ), - ), - minimum_bytes_per_state: Some( - Some( - 10, - ), - ), - }, - nfa: thompson::NFA( - ^000000: FAIL - - transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) - ) - , - stride2: 1, - start_map: StartByteMap{/x00 => NonWordByte, /x01 => NonWordByte, /x02 => NonWordByte, /x03 => NonWordByte, /x04 => NonWordByte, /x05 => NonWordByte, /x06 => NonWordByte, /x07 => NonWordByte, /x08 => NonWordByte, /t => NonWordByte, - => LineLF, /x0B => NonWordByte, /x0C => NonWordByte, /r => LineCR, /x0E => NonWordByte, /x0F => NonWordByte, /x10 => NonWordByte, /x11 => NonWordByte, /x12 => NonWordByte, /x13 => NonWordByte, /x14 => NonWordByte, /x15 => NonWordByte, /x16 => NonWordByte, /x17 => NonWordByte, /x18 => NonWordByte, /x19 => NonWordByte, /x1A => NonWordByte, /x1B => NonWordByte, /x1C => NonWordByte, /x1D => NonWordByte, /x1E => NonWordByte, /x1F => NonWordByte, ' ' => NonWordByte, ! => NonWordByte, /" => NonWordByte, # => NonWordByte, $ => NonWordByte, % => NonWordByte, & => NonWordByte, /' => NonWordByte, ( => NonWordByte, ) => NonWordByte, * => NonWordByte, + => NonWordByte, , => NonWordByte, - => NonWordByte, . => NonWordByte, / => NonWordByte, 0 => WordByte, 1 => WordByte, 2 => WordByte, 3 => WordByte, 4 => WordByte, 5 => WordByte, 6 => WordByte, 7 => WordByte, 8 => WordByte, 9 => WordByte, : => NonWordByte, ; => NonWordByte, < => NonWordByte, = => NonWordByte, > => NonWordByte, ? => NonWordByte, @ => NonWordByte, A => WordByte, B => WordByte, C => WordByte, D => WordByte, E => WordByte, F => WordByte, G => WordByte, H => WordByte, I => WordByte, J => WordByte, K => WordByte, L => WordByte, M => WordByte, N => WordByte, O => WordByte, P => WordByte, Q => WordByte, R => WordByte, S => WordByte, T => WordByte, U => WordByte, V => WordByte, W => WordByte, X => WordByte, Y => WordByte, Z => WordByte, [ => NonWordByte, // => NonWordByte, ] => NonWordByte, ^ => NonWordByte, _ => WordByte, ` => NonWordByte, a => WordByte, b => WordByte, c => WordByte, d => WordByte, e => WordByte, f => WordByte, g => WordByte, h => WordByte, i => WordByte, j => WordByte, k => WordByte, l => WordByte, m => WordByte, n => WordByte, o => WordByte, p => WordByte, q => WordByte, r => WordByte, s => WordByte, t => WordByte, u => WordByte, v => WordByte, w => WordByte, x => WordByte, y => WordByte, z => WordByte, { => NonWordByte, | => NonWordByte, } => NonWordByte, ~ => NonWordByte, /x7F => NonWordByte, /x80 => NonWordByte, /x81 => NonWordByte, /x82 => NonWordByte, /x83 => NonWordByte, /x84 => NonWordByte, /x85 => NonWordByte, /x86 => NonWordByte, /x87 => NonWordByte, /x88 => NonWordByte, /x89 => NonWordByte, /x8A => NonWordByte, /x8B => NonWordByte, /x8C => NonWordByte, /x8D => NonWordByte, /x8E => NonWordByte, /x8F => NonWordByte, /x90 => NonWordByte, /x91 => NonWordByte, /x92 => NonWordByte, /x93 => NonWordByte, /x94 => NonWordByte, /x95 => NonWordByte, /x96 => NonWordByte, /x97 => NonWordByte, /x98 => NonWordByte, /x99 => NonWordByte, /x9A => NonWordByte, /x9B => NonWordByte, /x9C => NonWordByte, /x9D => NonWordByte, /x9E => NonWordByte, /x9F => NonWordByte, /xA0 => NonWordByte, /xA1 => NonWordByte, /xA2 => NonWordByte, /xA3 => NonWordByte, /xA4 => NonWordByte, /xA5 => NonWordByte, /xA6 => NonWordByte, /xA7 => NonWordByte, /xA8 => NonWordByte, /xA9 => NonWordByte, /xAA => NonWordByte, /xAB => NonWordByte, /xAC => NonWordByte, /xAD => NonWordByte, /xAE => NonWordByte, /xAF => NonWordByte, /xB0 => NonWordByte, /xB1 => NonWordByte, /xB2 => NonWordByte, /xB3 => NonWordByte, /xB4 => NonWordByte, /xB5 => NonWordByte, /xB6 => NonWordByte, /xB7 => NonWordByte, /xB8 => NonWordByte, /xB9 => NonWordByte, /xBA => NonWordByte, /xBB => NonWordByte, /xBC => NonWordByte, /xBD => NonWordByte, /xBE => NonWordByte, /xBF => NonWordByte, /xC0 => NonWordByte, /xC1 => NonWordByte, /xC2 => NonWordByte, /xC3 => NonWordByte, /xC4 => NonWordByte, /xC5 => NonWordByte, /xC6 => NonWordByte, /xC7 => NonWordByte, /xC8 => NonWordByte, /xC9 => NonWordByte, /xCA => NonWordByte, /xCB => NonWordByte, /xCC => NonWordByte, /xCD => NonWordByte, /xCE => NonWordByte, /xCF => NonWordByte, /xD0 => NonWordByte, /xD1 => NonWordByte, /xD2 => NonWordByte, /xD3 => NonWordByte, /xD4 => NonWordByte, /xD5 => NonWordByte, /xD6 => NonWordByte, /xD7 => NonWordByte, /xD8 => NonWordByte, /xD9 => NonWordByte, /xDA => NonWordByte, /xDB => NonWordByte, /xDC => NonWordByte, /xDD => NonWordByte, /xDE => NonWordByte, /xDF => NonWordByte, /xE0 => NonWordByte, /xE1 => NonWordByte, /xE2 => NonWordByte, /xE3 => NonWordByte, /xE4 => NonWordByte, /xE5 => NonWordByte, /xE6 => NonWordByte, /xE7 => NonWordByte, /xE8 => NonWordByte, /xE9 => NonWordByte, /xEA => NonWordByte, /xEB => NonWordByte, /xEC => NonWordByte, /xED => NonWordByte, /xEE => NonWordByte, /xEF => NonWordByte, /xF0 => NonWordByte, /xF1 => NonWordByte, /xF2 => NonWordByte, /xF3 => NonWordByte, /xF4 => NonWordByte, /xF5 => NonWordByte, /xF6 => NonWordByte, /xF7 => NonWordByte, /xF8 => NonWordByte, /xF9 => NonWordByte, /xFA => NonWordByte, /xFB => NonWordByte, /xFC => NonWordByte, /xFD => NonWordByte, /xFE => NonWordByte, /xFF => NonWordByte}, - classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]), - quitset: ByteSet { - bits: {}, - }, - cache_capacity: 10485760, - }, - }, - ), - ), - ), - dfa: DFA( - None, - ), - }, - info: RegexInfo( - RegexInfoI { - config: Config { - match_kind: Some( - All, - ), - utf8_empty: Some( - false, - ), - autopre: None, - pre: None, - which_captures: None, - nfa_size_limit: Some( - Some( - 10485760, - ), - ), - onepass_size_limit: None, - hybrid_cache_capacity: Some( - 10485760, - ), - hybrid: None, - dfa: None, - dfa_size_limit: None, - dfa_state_limit: None, - onepass: None, - backtrack: None, - byte_classes: None, - line_terminator: None, - }, - props: [], - props_union: Properties( - PropertiesI { - minimum_len: None, - maximum_len: None, - look_set: ∅, - look_set_prefix: ∅, - look_set_suffix: ∅, - look_set_prefix_any: ∅, - look_set_suffix_any: ∅, - utf8: true, - explicit_captures_len: 0, - static_explicit_captures_len: None, - literal: false, - alternation_literal: true, - }, - ), - }, - ), - }, - pool: Pool( - Pool { - stacks: [ - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - ], - owner: 0, - owner_val: UnsafeCell { .. }, - }, - ), - }, - map: [], - patset: Pool( - Pool { - stacks: [ - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - ], - owner: 0, - owner_val: UnsafeCell { .. }, - }, - ), - }, - ), - ], - }, - poisoned: false, - .. - }, -} -[TRACE][walker/src/notify.rs::170] watcher: FsEventWatcher { - paths: 0x00006000030a8120, - since_when: 18446744073709551615, - latency: 0.0, - flags: 18, - event_handler: 0x00006000015a4010, - runloop: Some( - ( - 0x0000600000bb40c0, - JoinHandle { .. }, - ), - ), - recursive_info: { - "[CWD]": true, - }, -} -[TRACE][core/src/util/pmp.rs::131] background_thread: Mutex { - data: JoinHandle { .. }, - poisoned: false, - .. -} -[TRACE][core/src/util/pmp.rs::59] watcher: FsEventWatcher { - paths: 0x00006000030a8120, - since_when: 18446744073709551615, - latency: 0.0, - flags: 18, - event_handler: 0x00006000015a4010, - runloop: Some( - ( - 0x0000600000bb40c0, - JoinHandle { .. }, - ), - ), - recursive_info: { - "[CWD]": true, - }, -} -[TRACE][core/src/util/pmp.rs::91] fs_event_index: 0 -[TRACE][core/src/util/pmp.rs::93] kill_signal_index: 1 -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][pipeline/src/pipeline/mod.rs::302] &pmp: XvcPathMetadataProvider { - xvc_root: XvcRootInner { - absolute_path: AbsolutePath( - "[CWD]", - ), - xvc_dir: AbsolutePath( - "[CWD]/.xvc", - ), - store_dir: AbsolutePath( - "[CWD]/.xvc/store", - ), - config: XvcConfig { - current_dir: XvcConfigOption { - source: Runtime, - option: AbsolutePath( - "[CWD]", - ), - }, - config_maps: [ - XvcConfigMap { - source: Default, - map: { - "file.list.format": String( - "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", - ), - "file.track.force": Boolean( - false, - ), - "pipeline.default": String( - "default", - ), - "git.auto_commit": Boolean( - true, - ), - "file.recheck.method": String( - "copy", - ), - "pipeline.process_pool_size": Integer( - 4, - ), - "git.auto_stage": Boolean( - false, - ), - "git.use_git": Boolean( - true, - ), - "cache.algorithm": String( - "blake3", - ), - "file.track.no_parallel": Boolean( - false, - ), - "file.list.recursive": Boolean( - false, - ), - "file.list.no_summary": Boolean( - false, - ), - "file.track.no_commit": Boolean( - false, - ), - "pipeline.current_pipeline": String( - "default", - ), - "file.carry-in.no_parallel": Boolean( - false, - ), - "core.guid": String( - "603c2d6a0d2e6ea9", - ), - "file.list.show_dot_files": Boolean( - false, - ), - "file.list.sort": String( - "name-desc", - ), - "file.track.text_or_binary": String( - "auto", - ), - "file.carry-in.force": Boolean( - false, - ), - "pipeline.default_params_file": String( - "params.yaml", - ), - "git.command": String( - "git", - ), - "core.verbosity": String( - "error", - ), - }, - }, - XvcConfigMap { - source: Project, - map: { - "file.list.recursive": Boolean( - false, - ), - "pipeline.default": String( - "default", - ), - "pipeline.default_params_file": String( - "params.yaml", - ), - "file.track.no_commit": Boolean( - false, - ), - "git.use_git": Boolean( - true, - ), - "git.auto_stage": Boolean( - false, - ), - "file.track.no_parallel": Boolean( - false, - ), - "file.list.no_summary": Boolean( - false, - ), - "file.list.sort": String( - "name-desc", - ), - "core.guid": String( - "860d1874a44e1d89", - ), - "file.list.show_dot_files": Boolean( - false, - ), - "git.command": String( - "git", - ), - "file.track.text_or_binary": String( - "auto", - ), - "pipeline.process_pool_size": Integer( - 4, - ), - "file.recheck.method": String( - "copy", - ), - "file.list.format": String( - "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", - ), - "file.carry-in.no_parallel": Boolean( - false, - ), - "file.carry-in.force": Boolean( - false, - ), - "pipeline.current_pipeline": String( - "default", - ), - "file.track.force": Boolean( - false, - ), - "core.verbosity": String( - "error", - ), - "git.auto_commit": Boolean( - true, - ), - "cache.algorithm": String( - "blake3", - ), - }, - }, - XvcConfigMap { - source: Local, - map: {}, - }, - XvcConfigMap { - source: Environment, - map: { - "TRYCMD_TESTS": String( - "storage,file,pipeline,core", - ), - }, - }, - XvcConfigMap { - source: CommandLine, - map: { - "core.verbosity": String( - "debug", - ), - "core.quiet": Boolean( - false, - ), - }, - }, - ], - the_config: { - "pipeline.current_pipeline": XvcConfigValue { - source: Project, - value: String( - "default", - ), - }, - "cache.algorithm": XvcConfigValue { - source: Project, - value: String( - "blake3", - ), - }, - "pipeline.default_params_file": XvcConfigValue { - source: Project, - value: String( - "params.yaml", - ), - }, - "git.command": XvcConfigValue { - source: Project, - value: String( - "git", - ), - }, - "file.track.no_commit": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "core.verbosity": XvcConfigValue { - source: CommandLine, - value: String( - "debug", - ), - }, - "file.track.text_or_binary": XvcConfigValue { - source: Project, - value: String( - "auto", - ), - }, - "core.guid": XvcConfigValue { - source: Project, - value: String( - "860d1874a44e1d89", - ), - }, - "file.list.show_dot_files": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "file.recheck.method": XvcConfigValue { - source: Project, - value: String( - "copy", - ), - }, - "file.track.no_parallel": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "file.list.no_summary": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "TRYCMD_TESTS": XvcConfigValue { - source: Environment, - value: String( - "storage,file,pipeline,core", - ), - }, - "file.list.sort": XvcConfigValue { - source: Project, - value: String( - "name-desc", - ), - }, - "core.quiet": XvcConfigValue { - source: CommandLine, - value: Boolean( - false, - ), - }, - "git.auto_stage": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "git.use_git": XvcConfigValue { - source: Project, - value: Boolean( - true, - ), - }, - "file.carry-in.no_parallel": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "pipeline.default": XvcConfigValue { - source: Project, - value: String( - "default", - ), - }, - "file.list.recursive": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "file.track.force": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "file.list.format": XvcConfigValue { - source: Project, - value: String( - "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", - ), - }, - "pipeline.process_pool_size": XvcConfigValue { - source: Project, - value: Integer( - 4, - ), - }, - "file.carry-in.force": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "git.auto_commit": XvcConfigValue { - source: Project, - value: Boolean( - true, - ), - }, - }, - init_params: XvcConfigParams { - default_configuration: " -[core] -# The repository id. Please do not delete or change it. -# This is used to identify the repository and generate paths in storages. -# In the future it may be used to in other ways. -guid = /"603c2d6a0d2e6ea9/" -# Default verbosity level. -# One of /"error/", /"warn/", /"info/" -verbosity = /"error/" - -[git] -# Automate git operations. -# Turning this off leads Xvc to behave as if it's not in a Git repository. -# Not recommended unless you're really not using Git -use_git = true -# Command to run Git process. -# You can set this to an absolute path to specify an executable -# If set to a non-absolute path, the executable will be searched in $PATH. -command = /"git/" - -# Commit changes in .xvc/ directory after commands. -# You can set this to false if you want to commit manually. -auto_commit = true - -# Stage changes in .xvc/ directory without committing. -# auto_commit implies auto_stage. -# If you want to commit manually but don't want to stage after individual Xvc commands, you can set this to true. -auto_stage = false - -[cache] -# The hash algorithm used for the cache. -# It may take blake3, blake2, sha2 or sha3 as values. -# All algorithms are selected to produce 256-bit hashes, so sha2 means SHA2-256, blake2 means BLAKE2s, etc. -# The cache path is produced by prepending algorithm name to the cache. -# Blake3 files are in .xvc/b3/, while sha2 files are in .xvc/s2/ etc. -algorithm = /"blake3/" - -[file] - -[file.track] - -# Don't move file content to cache after xvc file track -no_commit = false -# Force to track files even if they are already tracked. -force = false - -# Xvc calculates file content digest differently for text and binary files. -# This option controls whether to treat files as text or binary. -# It may take auto, text or binary as values. -# Auto check each file individually and treat it as text if it's text. -text_or_binary = /"auto/" - -# Don't use parallelism in track operations. -# Note that some of the operations are implemented in parallel by default, and this option affects some heavier operations. -no_parallel = false - -[file.list] - -# Format for `xvc file list` rows. You can reorder or remove columns. -# The following are the keys for each row: -# - {acd64}: actual content digest. All 64 digits from the workspace file's content. -# - {acd8}: actual content digest. First 8 digits the file content digest. -# - {aft}: actual file type. Whether the entry is a file (F), directory (D), -# symlink (S), hardlink (H) or reflink (R). -# - {asz}: actual size. The size of the workspace file in bytes. It uses MB, -# GB and TB to represent sizes larger than 1MB. -# - {ats}: actual timestamp. The timestamp of the workspace file. -# - {cst}: cache status. One of /"=/", /">/", /" 0 - F 000001: - >000002: /x00 => 2 - 000003: /x00 => 0 - match kind: Standard - prefilter: false - state length: 4 - pattern length: 0 - shortest pattern length: 18446744073709551615 - longest pattern length: 0 - alphabet length: 1 - stride: 1 - byte classes: ByteClasses(0 => [0-255]) - memory usage: 16 - ) - , - ), - map: [], - longest: 0, - }, - ), - Prefix( - PrefixStrategy { - matcher: AhoCorasick( - dfa::DFA( - D 000000: /x00 => 0 - F 000001: - >000002: /x00 => 2 - 000003: /x00 => 0 - match kind: Standard - prefilter: false - state length: 4 - pattern length: 0 - shortest pattern length: 18446744073709551615 - longest pattern length: 0 - alphabet length: 1 - stride: 1 - byte classes: ByteClasses(0 => [0-255]) - memory usage: 16 - ) - , - ), - map: [], - longest: 0, - }, - ), - RequiredExtension( - RequiredExtensionStrategy( - {}, - ), - ), - Regex( - RegexSetStrategy { - matcher: Regex { - imp: RegexI { - strat: Core { - info: RegexInfo( - RegexInfoI { - config: Config { - match_kind: Some( - All, - ), - utf8_empty: Some( - false, - ), - autopre: None, - pre: None, - which_captures: None, - nfa_size_limit: Some( - Some( - 10485760, - ), - ), - onepass_size_limit: None, - hybrid_cache_capacity: Some( - 10485760, - ), - hybrid: None, - dfa: None, - dfa_size_limit: None, - dfa_state_limit: None, - onepass: None, - backtrack: None, - byte_classes: None, - line_terminator: None, - }, - props: [], - props_union: Properties( - PropertiesI { - minimum_len: None, - maximum_len: None, - look_set: ∅, - look_set_prefix: ∅, - look_set_suffix: ∅, - look_set_prefix_any: ∅, - look_set_suffix_any: ∅, - utf8: true, - explicit_captures_len: 0, - static_explicit_captures_len: None, - literal: false, - alternation_literal: true, - }, - ), - }, - ), - pre: None, - nfa: thompson::NFA( - ^000000: FAIL - - transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) - ) - , - nfarev: Some( - thompson::NFA( - ^000000: FAIL - - transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) - ) - , - ), - pikevm: PikeVM( - PikeVMEngine( - PikeVM { - config: Config { - match_kind: Some( - All, - ), - pre: Some( - None, - ), - }, - nfa: thompson::NFA( - ^000000: FAIL - - transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) - ) - , - }, - ), - ), - backtrack: BoundedBacktracker( - None, - ), - onepass: OnePass( - None, - ), - hybrid: Hybrid( - Some( - HybridEngine( - Regex { - forward: DFA { - config: Config { - match_kind: Some( - All, - ), - pre: Some( - None, - ), - starts_for_each_pattern: Some( - true, - ), - byte_classes: Some( - true, - ), - unicode_word_boundary: Some( - true, - ), - quitset: None, - specialize_start_states: Some( - false, - ), - cache_capacity: Some( - 10485760, - ), - skip_cache_capacity_check: Some( - false, - ), - minimum_cache_clear_count: Some( - Some( - 3, - ), - ), - minimum_bytes_per_state: Some( - Some( - 10, - ), - ), - }, - nfa: thompson::NFA( - ^000000: FAIL - - transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) - ) - , - stride2: 1, - start_map: StartByteMap{/x00 => NonWordByte, /x01 => NonWordByte, /x02 => NonWordByte, /x03 => NonWordByte, /x04 => NonWordByte, /x05 => NonWordByte, /x06 => NonWordByte, /x07 => NonWordByte, /x08 => NonWordByte, /t => NonWordByte, - => LineLF, /x0B => NonWordByte, /x0C => NonWordByte, /r => LineCR, /x0E => NonWordByte, /x0F => NonWordByte, /x10 => NonWordByte, /x11 => NonWordByte, /x12 => NonWordByte, /x13 => NonWordByte, /x14 => NonWordByte, /x15 => NonWordByte, /x16 => NonWordByte, /x17 => NonWordByte, /x18 => NonWordByte, /x19 => NonWordByte, /x1A => NonWordByte, /x1B => NonWordByte, /x1C => NonWordByte, /x1D => NonWordByte, /x1E => NonWordByte, /x1F => NonWordByte, ' ' => NonWordByte, ! => NonWordByte, /" => NonWordByte, # => NonWordByte, $ => NonWordByte, % => NonWordByte, & => NonWordByte, /' => NonWordByte, ( => NonWordByte, ) => NonWordByte, * => NonWordByte, + => NonWordByte, , => NonWordByte, - => NonWordByte, . => NonWordByte, / => NonWordByte, 0 => WordByte, 1 => WordByte, 2 => WordByte, 3 => WordByte, 4 => WordByte, 5 => WordByte, 6 => WordByte, 7 => WordByte, 8 => WordByte, 9 => WordByte, : => NonWordByte, ; => NonWordByte, < => NonWordByte, = => NonWordByte, > => NonWordByte, ? => NonWordByte, @ => NonWordByte, A => WordByte, B => WordByte, C => WordByte, D => WordByte, E => WordByte, F => WordByte, G => WordByte, H => WordByte, I => WordByte, J => WordByte, K => WordByte, L => WordByte, M => WordByte, N => WordByte, O => WordByte, P => WordByte, Q => WordByte, R => WordByte, S => WordByte, T => WordByte, U => WordByte, V => WordByte, W => WordByte, X => WordByte, Y => WordByte, Z => WordByte, [ => NonWordByte, // => NonWordByte, ] => NonWordByte, ^ => NonWordByte, _ => WordByte, ` => NonWordByte, a => WordByte, b => WordByte, c => WordByte, d => WordByte, e => WordByte, f => WordByte, g => WordByte, h => WordByte, i => WordByte, j => WordByte, k => WordByte, l => WordByte, m => WordByte, n => WordByte, o => WordByte, p => WordByte, q => WordByte, r => WordByte, s => WordByte, t => WordByte, u => WordByte, v => WordByte, w => WordByte, x => WordByte, y => WordByte, z => WordByte, { => NonWordByte, | => NonWordByte, } => NonWordByte, ~ => NonWordByte, /x7F => NonWordByte, /x80 => NonWordByte, /x81 => NonWordByte, /x82 => NonWordByte, /x83 => NonWordByte, /x84 => NonWordByte, /x85 => NonWordByte, /x86 => NonWordByte, /x87 => NonWordByte, /x88 => NonWordByte, /x89 => NonWordByte, /x8A => NonWordByte, /x8B => NonWordByte, /x8C => NonWordByte, /x8D => NonWordByte, /x8E => NonWordByte, /x8F => NonWordByte, /x90 => NonWordByte, /x91 => NonWordByte, /x92 => NonWordByte, /x93 => NonWordByte, /x94 => NonWordByte, /x95 => NonWordByte, /x96 => NonWordByte, /x97 => NonWordByte, /x98 => NonWordByte, /x99 => NonWordByte, /x9A => NonWordByte, /x9B => NonWordByte, /x9C => NonWordByte, /x9D => NonWordByte, /x9E => NonWordByte, /x9F => NonWordByte, /xA0 => NonWordByte, /xA1 => NonWordByte, /xA2 => NonWordByte, /xA3 => NonWordByte, /xA4 => NonWordByte, /xA5 => NonWordByte, /xA6 => NonWordByte, /xA7 => NonWordByte, /xA8 => NonWordByte, /xA9 => NonWordByte, /xAA => NonWordByte, /xAB => NonWordByte, /xAC => NonWordByte, /xAD => NonWordByte, /xAE => NonWordByte, /xAF => NonWordByte, /xB0 => NonWordByte, /xB1 => NonWordByte, /xB2 => NonWordByte, /xB3 => NonWordByte, /xB4 => NonWordByte, /xB5 => NonWordByte, /xB6 => NonWordByte, /xB7 => NonWordByte, /xB8 => NonWordByte, /xB9 => NonWordByte, /xBA => NonWordByte, /xBB => NonWordByte, /xBC => NonWordByte, /xBD => NonWordByte, /xBE => NonWordByte, /xBF => NonWordByte, /xC0 => NonWordByte, /xC1 => NonWordByte, /xC2 => NonWordByte, /xC3 => NonWordByte, /xC4 => NonWordByte, /xC5 => NonWordByte, /xC6 => NonWordByte, /xC7 => NonWordByte, /xC8 => NonWordByte, /xC9 => NonWordByte, /xCA => NonWordByte, /xCB => NonWordByte, /xCC => NonWordByte, /xCD => NonWordByte, /xCE => NonWordByte, /xCF => NonWordByte, /xD0 => NonWordByte, /xD1 => NonWordByte, /xD2 => NonWordByte, /xD3 => NonWordByte, /xD4 => NonWordByte, /xD5 => NonWordByte, /xD6 => NonWordByte, /xD7 => NonWordByte, /xD8 => NonWordByte, /xD9 => NonWordByte, /xDA => NonWordByte, /xDB => NonWordByte, /xDC => NonWordByte, /xDD => NonWordByte, /xDE => NonWordByte, /xDF => NonWordByte, /xE0 => NonWordByte, /xE1 => NonWordByte, /xE2 => NonWordByte, /xE3 => NonWordByte, /xE4 => NonWordByte, /xE5 => NonWordByte, /xE6 => NonWordByte, /xE7 => NonWordByte, /xE8 => NonWordByte, /xE9 => NonWordByte, /xEA => NonWordByte, /xEB => NonWordByte, /xEC => NonWordByte, /xED => NonWordByte, /xEE => NonWordByte, /xEF => NonWordByte, /xF0 => NonWordByte, /xF1 => NonWordByte, /xF2 => NonWordByte, /xF3 => NonWordByte, /xF4 => NonWordByte, /xF5 => NonWordByte, /xF6 => NonWordByte, /xF7 => NonWordByte, /xF8 => NonWordByte, /xF9 => NonWordByte, /xFA => NonWordByte, /xFB => NonWordByte, /xFC => NonWordByte, /xFD => NonWordByte, /xFE => NonWordByte, /xFF => NonWordByte}, - classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]), - quitset: ByteSet { - bits: {}, - }, - cache_capacity: 10485760, - }, - reverse: DFA { - config: Config { - match_kind: Some( - All, - ), - pre: Some( - None, - ), - starts_for_each_pattern: Some( - true, - ), - byte_classes: Some( - true, - ), - unicode_word_boundary: Some( - true, - ), - quitset: None, - specialize_start_states: Some( - false, - ), - cache_capacity: Some( - 10485760, - ), - skip_cache_capacity_check: Some( - false, - ), - minimum_cache_clear_count: Some( - Some( - 3, - ), - ), - minimum_bytes_per_state: Some( - Some( - 10, - ), - ), - }, - nfa: thompson::NFA( - ^000000: FAIL - - transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) - ) - , - stride2: 1, - start_map: StartByteMap{/x00 => NonWordByte, /x01 => NonWordByte, /x02 => NonWordByte, /x03 => NonWordByte, /x04 => NonWordByte, /x05 => NonWordByte, /x06 => NonWordByte, /x07 => NonWordByte, /x08 => NonWordByte, /t => NonWordByte, - => LineLF, /x0B => NonWordByte, /x0C => NonWordByte, /r => LineCR, /x0E => NonWordByte, /x0F => NonWordByte, /x10 => NonWordByte, /x11 => NonWordByte, /x12 => NonWordByte, /x13 => NonWordByte, /x14 => NonWordByte, /x15 => NonWordByte, /x16 => NonWordByte, /x17 => NonWordByte, /x18 => NonWordByte, /x19 => NonWordByte, /x1A => NonWordByte, /x1B => NonWordByte, /x1C => NonWordByte, /x1D => NonWordByte, /x1E => NonWordByte, /x1F => NonWordByte, ' ' => NonWordByte, ! => NonWordByte, /" => NonWordByte, # => NonWordByte, $ => NonWordByte, % => NonWordByte, & => NonWordByte, /' => NonWordByte, ( => NonWordByte, ) => NonWordByte, * => NonWordByte, + => NonWordByte, , => NonWordByte, - => NonWordByte, . => NonWordByte, / => NonWordByte, 0 => WordByte, 1 => WordByte, 2 => WordByte, 3 => WordByte, 4 => WordByte, 5 => WordByte, 6 => WordByte, 7 => WordByte, 8 => WordByte, 9 => WordByte, : => NonWordByte, ; => NonWordByte, < => NonWordByte, = => NonWordByte, > => NonWordByte, ? => NonWordByte, @ => NonWordByte, A => WordByte, B => WordByte, C => WordByte, D => WordByte, E => WordByte, F => WordByte, G => WordByte, H => WordByte, I => WordByte, J => WordByte, K => WordByte, L => WordByte, M => WordByte, N => WordByte, O => WordByte, P => WordByte, Q => WordByte, R => WordByte, S => WordByte, T => WordByte, U => WordByte, V => WordByte, W => WordByte, X => WordByte, Y => WordByte, Z => WordByte, [ => NonWordByte, // => NonWordByte, ] => NonWordByte, ^ => NonWordByte, _ => WordByte, ` => NonWordByte, a => WordByte, b => WordByte, c => WordByte, d => WordByte, e => WordByte, f => WordByte, g => WordByte, h => WordByte, i => WordByte, j => WordByte, k => WordByte, l => WordByte, m => WordByte, n => WordByte, o => WordByte, p => WordByte, q => WordByte, r => WordByte, s => WordByte, t => WordByte, u => WordByte, v => WordByte, w => WordByte, x => WordByte, y => WordByte, z => WordByte, { => NonWordByte, | => NonWordByte, } => NonWordByte, ~ => NonWordByte, /x7F => NonWordByte, /x80 => NonWordByte, /x81 => NonWordByte, /x82 => NonWordByte, /x83 => NonWordByte, /x84 => NonWordByte, /x85 => NonWordByte, /x86 => NonWordByte, /x87 => NonWordByte, /x88 => NonWordByte, /x89 => NonWordByte, /x8A => NonWordByte, /x8B => NonWordByte, /x8C => NonWordByte, /x8D => NonWordByte, /x8E => NonWordByte, /x8F => NonWordByte, /x90 => NonWordByte, /x91 => NonWordByte, /x92 => NonWordByte, /x93 => NonWordByte, /x94 => NonWordByte, /x95 => NonWordByte, /x96 => NonWordByte, /x97 => NonWordByte, /x98 => NonWordByte, /x99 => NonWordByte, /x9A => NonWordByte, /x9B => NonWordByte, /x9C => NonWordByte, /x9D => NonWordByte, /x9E => NonWordByte, /x9F => NonWordByte, /xA0 => NonWordByte, /xA1 => NonWordByte, /xA2 => NonWordByte, /xA3 => NonWordByte, /xA4 => NonWordByte, /xA5 => NonWordByte, /xA6 => NonWordByte, /xA7 => NonWordByte, /xA8 => NonWordByte, /xA9 => NonWordByte, /xAA => NonWordByte, /xAB => NonWordByte, /xAC => NonWordByte, /xAD => NonWordByte, /xAE => NonWordByte, /xAF => NonWordByte, /xB0 => NonWordByte, /xB1 => NonWordByte, /xB2 => NonWordByte, /xB3 => NonWordByte, /xB4 => NonWordByte, /xB5 => NonWordByte, /xB6 => NonWordByte, /xB7 => NonWordByte, /xB8 => NonWordByte, /xB9 => NonWordByte, /xBA => NonWordByte, /xBB => NonWordByte, /xBC => NonWordByte, /xBD => NonWordByte, /xBE => NonWordByte, /xBF => NonWordByte, /xC0 => NonWordByte, /xC1 => NonWordByte, /xC2 => NonWordByte, /xC3 => NonWordByte, /xC4 => NonWordByte, /xC5 => NonWordByte, /xC6 => NonWordByte, /xC7 => NonWordByte, /xC8 => NonWordByte, /xC9 => NonWordByte, /xCA => NonWordByte, /xCB => NonWordByte, /xCC => NonWordByte, /xCD => NonWordByte, /xCE => NonWordByte, /xCF => NonWordByte, /xD0 => NonWordByte, /xD1 => NonWordByte, /xD2 => NonWordByte, /xD3 => NonWordByte, /xD4 => NonWordByte, /xD5 => NonWordByte, /xD6 => NonWordByte, /xD7 => NonWordByte, /xD8 => NonWordByte, /xD9 => NonWordByte, /xDA => NonWordByte, /xDB => NonWordByte, /xDC => NonWordByte, /xDD => NonWordByte, /xDE => NonWordByte, /xDF => NonWordByte, /xE0 => NonWordByte, /xE1 => NonWordByte, /xE2 => NonWordByte, /xE3 => NonWordByte, /xE4 => NonWordByte, /xE5 => NonWordByte, /xE6 => NonWordByte, /xE7 => NonWordByte, /xE8 => NonWordByte, /xE9 => NonWordByte, /xEA => NonWordByte, /xEB => NonWordByte, /xEC => NonWordByte, /xED => NonWordByte, /xEE => NonWordByte, /xEF => NonWordByte, /xF0 => NonWordByte, /xF1 => NonWordByte, /xF2 => NonWordByte, /xF3 => NonWordByte, /xF4 => NonWordByte, /xF5 => NonWordByte, /xF6 => NonWordByte, /xF7 => NonWordByte, /xF8 => NonWordByte, /xF9 => NonWordByte, /xFA => NonWordByte, /xFB => NonWordByte, /xFC => NonWordByte, /xFD => NonWordByte, /xFE => NonWordByte, /xFF => NonWordByte}, - classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]), - quitset: ByteSet { - bits: {}, - }, - cache_capacity: 10485760, - }, - }, - ), - ), - ), - dfa: DFA( - None, - ), - }, - info: RegexInfo( - RegexInfoI { - config: Config { - match_kind: Some( - All, - ), - utf8_empty: Some( - false, - ), - autopre: None, - pre: None, - which_captures: None, - nfa_size_limit: Some( - Some( - 10485760, - ), - ), - onepass_size_limit: None, - hybrid_cache_capacity: Some( - 10485760, - ), - hybrid: None, - dfa: None, - dfa_size_limit: None, - dfa_state_limit: None, - onepass: None, - backtrack: None, - byte_classes: None, - line_terminator: None, - }, - props: [], - props_union: Properties( - PropertiesI { - minimum_len: None, - maximum_len: None, - look_set: ∅, - look_set_prefix: ∅, - look_set_suffix: ∅, - look_set_prefix_any: ∅, - look_set_suffix_any: ∅, - utf8: true, - explicit_captures_len: 0, - static_explicit_captures_len: None, - literal: false, - alternation_literal: true, - }, - ), - }, - ), - }, - pool: Pool( - Pool { - stacks: [ - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - ], - owner: 0, - owner_val: UnsafeCell { .. }, - }, - ), - }, - map: [], - patset: Pool( - Pool { - stacks: [ - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - ], - owner: 0, - owner_val: UnsafeCell { .. }, - }, - ), - }, - ), - ], - }, - poisoned: false, - .. - }, - }, -} -[TRACE][pipeline/src/pipeline/mod.rs::305] pipeline_len: 1 -[TRACE][pipeline/src/pipeline/mod.rs::331] &dependency_graph: { - XvcEntity( - 2, - 16393284406203488121, - ): [], -} -[TRACE][pipeline/src/pipeline/mod.rs::343] &dependency_graph: { - XvcEntity( - 2, - 16393284406203488121, - ): [], -} [INFO][pipeline/src/pipeline/mod.rs::347] Pipeline Graph: digraph { - 0 [ label = "(2, 16393284406203488121)" ] + 0 [ label = "(2, 9818981151839479067)" ] } -[TRACE][pipeline/src/pipeline/mod.rs::412] step_states: RwLock { - data: HStore { - map: { - XvcEntity( - 2, - 16393284406203488121, - ): Begin( - FromInit, - ), - }, - }, - poisoned: false, - .. -} -[TRACE][/Users/iex/.cargo/registry/src/index.crates.io-6f17d22bba15001f/notify-6.1.1/src/fsevent.rs::536] FSEvent: path = `[CWD]/.xvc/store/xvc-dependency-store`, flag = StreamFlags(ITEM_CREATED | IS_DIR) -[TRACE][walker/src/notify.rs::56] event: Ok( - Event { - kind: Create( - Folder, - ), - paths: [ - "[CWD]/.xvc/store/xvc-dependency-store", - ], - attr:tracker: None, - attr:flag: None, - attr:info: None, - attr:source: None, - }, -) -[TRACE][walker/src/lib.rs::748] is_abs: true -[TRACE][walker/src/lib.rs::752] path_str: "[CWD]/.xvc/store/xvc-dependency-store" -[TRACE][walker/src/lib.rs::754] final_slash: false -[TRACE][walker/src/lib.rs::776] path: "/.xvc/store/xvc-dependency-store" -[TRACE][/Users/iex/.cargo/registry/src/index.crates.io-6f17d22bba15001f/notify-6.1.1/src/fsevent.rs::536] FSEvent: path = `[CWD]/.xvc/store/xvc-dependency-xvc-step-r1n-store`, flag = StreamFlags(ITEM_CREATED | IS_DIR) -[TRACE][walker/src/notify.rs::56] event: Ok( - Event { - kind: Create( - Folder, - ), - paths: [ - "[CWD]/.xvc/store/xvc-dependency-xvc-step-r1n-store", - ], - attr:tracker: None, - attr:flag: None, - attr:info: None, - attr:source: None, - }, -) -[TRACE][walker/src/lib.rs::748] is_abs: true -[TRACE][walker/src/lib.rs::752] path_str: "[CWD]/.xvc/store/xvc-dependency-xvc-step-r1n-store" -[TRACE][walker/src/lib.rs::754] final_slash: false -[TRACE][walker/src/lib.rs::776] path: "/.xvc/store/xvc-dependency-xvc-step-r1n-store" -[TRACE][/Users/iex/.cargo/registry/src/index.crates.io-6f17d22bba15001f/notify-6.1.1/src/fsevent.rs::536] FSEvent: path = `[CWD]/.xvc/store/xvc-output-store`, flag = StreamFlags(ITEM_CREATED | IS_DIR) -[TRACE][walker/src/notify.rs::56] event: Ok( - Event { - kind: Create( - Folder, - ), - paths: [ - "[CWD]/.xvc/store/xvc-output-store", - ], - attr:tracker: None, - attr:flag: None, - attr:info: None, - attr:source: None, - }, -) -[TRACE][walker/src/lib.rs::748] is_abs: true -[TRACE][walker/src/lib.rs::752] path_str: "[CWD]/.xvc/store/xvc-output-store" -[TRACE][walker/src/lib.rs::754] final_slash: false -[TRACE][walker/src/lib.rs::776] path: "/.xvc/store/xvc-output-store" -[TRACE][/Users/iex/.cargo/registry/src/index.crates.io-6f17d22bba15001f/notify-6.1.1/src/fsevent.rs::536] FSEvent: path = `[CWD]/.xvc/store/xvc-output-xvc-step-r1n-store`, flag = StreamFlags(ITEM_CREATED | IS_DIR) -[TRACE][walker/src/notify.rs::56] event: Ok( - Event { - kind: Create( - Folder, - ), - paths: [ - "[CWD]/.xvc/store/xvc-output-xvc-step-r1n-store", - ], - attr:tracker: None, - attr:flag: None, - attr:info: None, - attr:source: None, - }, -) -[TRACE][walker/src/lib.rs::748] is_abs: true -[TRACE][walker/src/lib.rs::752] path_str: "[CWD]/.xvc/store/xvc-output-xvc-step-r1n-store" -[TRACE][walker/src/lib.rs::754] final_slash: false -[TRACE][walker/src/lib.rs::776] path: "/.xvc/store/xvc-output-xvc-step-r1n-store" -[TRACE][/Users/iex/.cargo/registry/src/index.crates.io-6f17d22bba15001f/notify-6.1.1/src/fsevent.rs::536] FSEvent: path = `[CWD]/.xvc/store/xvc-pipeline-run-dir-store`, flag = StreamFlags(ITEM_CREATED | IS_DIR) -[TRACE][walker/src/notify.rs::56] event: Ok( - Event { - kind: Create( - Folder, - ), - paths: [ - "[CWD]/.xvc/store/xvc-pipeline-run-dir-store", - ], - attr:tracker: None, - attr:flag: None, - attr:info: None, - attr:source: None, - }, -) -[TRACE][walker/src/lib.rs::748] is_abs: true -[TRACE][walker/src/lib.rs::752] path_str: "[CWD]/.xvc/store/xvc-pipeline-run-dir-store" -[TRACE][walker/src/lib.rs::754] final_slash: false -[TRACE][walker/src/lib.rs::776] path: "/.xvc/store/xvc-pipeline-run-dir-store" -[TRACE][core/src/util/pmp.rs::99] index: 0 -[TRACE][core/src/util/pmp.rs::102] fs_event: Ok( - Some( - Create { - path: "[CWD]/.xvc/store/xvc-dependency-store", - metadata: Metadata { - file_type: FileType( - FileType { - mode: 16877, - }, - ), - is_dir: true, - is_file: false, - permissions: Permissions( - FilePermissions { - mode: 16877, - }, - ), - modified: Ok( - SystemTime { - tv_sec: 1722024005, - tv_nsec: 893699709, - }, - ), - accessed: Ok( - SystemTime { - tv_sec: 1722024005, - tv_nsec: 893699709, - }, - ), - created: Ok( - SystemTime { - tv_sec: 1722024005, - tv_nsec: 893699709, - }, - ), - .. - }, - }, - ), -) -[TRACE][core/src/types/xvcpath.rs::88] abs_path: "[CWD]/.xvc/store/xvc-dependency-store" -[TRACE][core/src/types/xvcpath.rs::89] current_dir: AbsolutePath( - "[CWD]", -) -[TRACE][core/src/types/xvcpath.rs::90] xvc_root.absolute_path(): AbsolutePath( - "[CWD]", -) -[TRACE][core/src/util/pmp.rs::65] "Creating {} with {}": "Creating {} with {}" -[TRACE][core/src/util/pmp.rs::65] xvc_path: XvcPath( - ".xvc/store/xvc-dependency-store", -) -[TRACE][core/src/util/pmp.rs::65] xvc_md: XvcMetadata { - file_type: Directory, - size: Some( - 64, - ), - modified: Some( - SystemTime { - tv_sec: 1722024005, - tv_nsec: 893699709, - }, - ), -} -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::99] index: 0 -[TRACE][core/src/util/pmp.rs::102] fs_event: Ok( - Some( - Create { - path: "[CWD]/.xvc/store/xvc-dependency-xvc-step-r1n-store", - metadata: Metadata { - file_type: FileType( - FileType { - mode: 16877, - }, - ), - is_dir: true, - is_file: false, - permissions: Permissions( - FilePermissions { - mode: 16877, - }, - ), - modified: Ok( - SystemTime { - tv_sec: 1722024005, - tv_nsec: 893766208, - }, - ), - accessed: Ok( - SystemTime { - tv_sec: 1722024005, - tv_nsec: 893766208, - }, - ), - created: Ok( - SystemTime { - tv_sec: 1722024005, - tv_nsec: 893766208, - }, - ), - .. - }, - }, - ), -) -[TRACE][core/src/types/xvcpath.rs::88] abs_path: "[CWD]/.xvc/store/xvc-dependency-xvc-step-r1n-store" -[TRACE][core/src/types/xvcpath.rs::89] current_dir: AbsolutePath( - "[CWD]", -) -[TRACE][core/src/types/xvcpath.rs::90] xvc_root.absolute_path(): AbsolutePath( - "[CWD]", -) -[TRACE][core/src/util/pmp.rs::65] "Creating {} with {}": "Creating {} with {}" -[TRACE][core/src/util/pmp.rs::65] xvc_path: XvcPath( - ".xvc/store/xvc-dependency-xvc-step-r1n-store", -) -[TRACE][core/src/util/pmp.rs::65] xvc_md: XvcMetadata { - file_type: Directory, - size: Some( - 64, - ), - modified: Some( - SystemTime { - tv_sec: 1722024005, - tv_nsec: 893766208, - }, - ), -} -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::99] index: 0 -[TRACE][core/src/util/pmp.rs::102] fs_event: Ok( - Some( - Create { - path: "[CWD]/.xvc/store/xvc-output-store", - metadata: Metadata { - file_type: FileType( - FileType { - mode: 16877, - }, - ), - is_dir: true, - is_file: false, - permissions: Permissions( - FilePermissions { - mode: 16877, - }, - ), - modified: Ok( - SystemTime { - tv_sec: 1722024005, - tv_nsec: 893895957, - }, - ), - accessed: Ok( - SystemTime { - tv_sec: 1722024005, - tv_nsec: 893895957, - }, - ), - created: Ok( - SystemTime { - tv_sec: 1722024005, - tv_nsec: 893895957, - }, - ), - .. - }, - }, - ), -) -[TRACE][core/src/types/xvcpath.rs::88] abs_path: "[CWD]/.xvc/store/xvc-output-store" -[TRACE][core/src/types/xvcpath.rs::89] current_dir: AbsolutePath( - "[CWD]", -) -[TRACE][core/src/types/xvcpath.rs::90] xvc_root.absolute_path(): AbsolutePath( - "[CWD]", -) -[TRACE][core/src/util/pmp.rs::65] "Creating {} with {}": "Creating {} with {}" -[TRACE][core/src/util/pmp.rs::65] xvc_path: XvcPath( - ".xvc/store/xvc-output-store", -) -[TRACE][core/src/util/pmp.rs::65] xvc_md: XvcMetadata { - file_type: Directory, - size: Some( - 64, - ), - modified: Some( - SystemTime { - tv_sec: 1722024005, - tv_nsec: 893895957, - }, - ), -} -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::99] index: 0 -[TRACE][core/src/util/pmp.rs::102] fs_event: Ok( - Some( - Create { - path: "[CWD]/.xvc/store/xvc-output-xvc-step-r1n-store", - metadata: Metadata { - file_type: FileType( - FileType { - mode: 16877, - }, - ), - is_dir: true, - is_file: false, - permissions: Permissions( - FilePermissions { - mode: 16877, - }, - ), - modified: Ok( - SystemTime { - tv_sec: 1722024005, - tv_nsec: 893950706, - }, - ), - accessed: Ok( - SystemTime { - tv_sec: 1722024005, - tv_nsec: 893950706, - }, - ), - created: Ok( - SystemTime { - tv_sec: 1722024005, - tv_nsec: 893950706, - }, - ), - .. - }, - }, - ), -) -[TRACE][core/src/types/xvcpath.rs::88] abs_path: "[CWD]/.xvc/store/xvc-output-xvc-step-r1n-store" -[TRACE][core/src/types/xvcpath.rs::89] current_dir: AbsolutePath( - "[CWD]", -) -[TRACE][core/src/types/xvcpath.rs::90] xvc_root.absolute_path(): AbsolutePath( - "[CWD]", -) -[TRACE][core/src/util/pmp.rs::65] "Creating {} with {}": "Creating {} with {}" -[TRACE][core/src/util/pmp.rs::65] xvc_path: XvcPath( - ".xvc/store/xvc-output-xvc-step-r1n-store", -) -[TRACE][core/src/util/pmp.rs::65] xvc_md: XvcMetadata { - file_type: Directory, - size: Some( - 64, - ), - modified: Some( - SystemTime { - tv_sec: 1722024005, - tv_nsec: 893950706, - }, - ), -} -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::99] index: 0 -[TRACE][core/src/util/pmp.rs::102] fs_event: Ok( - Some( - Create { - path: "[CWD]/.xvc/store/xvc-pipeline-run-dir-store", - metadata: Metadata { - file_type: FileType( - FileType { - mode: 16877, - }, - ), - is_dir: true, - is_file: false, - permissions: Permissions( - FilePermissions { - mode: 16877, - }, - ), - modified: Ok( - SystemTime { - tv_sec: 1722024005, - tv_nsec: 902113188, - }, - ), - accessed: Ok( - SystemTime { - tv_sec: 1722024005, - tv_nsec: 902113188, - }, - ), - created: Ok( - SystemTime { - tv_sec: 1722024005, - tv_nsec: 902113188, - }, - ), - .. - }, - }, - ), -) -[TRACE][core/src/types/xvcpath.rs::88] abs_path: "[CWD]/.xvc/store/xvc-pipeline-run-dir-store" -[TRACE][core/src/types/xvcpath.rs::89] current_dir: AbsolutePath( - "[CWD]", -) -[TRACE][core/src/types/xvcpath.rs::90] xvc_root.absolute_path(): AbsolutePath( - "[CWD]", -) -[TRACE][core/src/util/pmp.rs::65] "Creating {} with {}": "Creating {} with {}" -[TRACE][core/src/util/pmp.rs::65] xvc_path: XvcPath( - ".xvc/store/xvc-pipeline-run-dir-store", -) -[TRACE][core/src/util/pmp.rs::65] xvc_md: XvcMetadata { - file_type: Directory, - size: Some( - 64, - ), - modified: Some( - SystemTime { - tv_sec: 1722024005, - tv_nsec: 902113188, - }, - ), -} -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][pipeline/src/pipeline/mod.rs::583] select: Select { .. } -[TRACE][pipeline/src/pipeline/mod.rs::508] &step_thread_store: HStore { - map: { - XvcEntity( - 2, - 16393284406203488121, - ): ScopedJoinHandle { .. }, - }, -} -[TRACE][pipeline/src/pipeline/mod.rs::512] (step_e, &jh): ( - XvcEntity( - 2, - 16393284406203488121, - ), - ScopedJoinHandle { .. }, -) -[TRACE][pipeline/src/pipeline/mod.rs::623] params.recorded_dependencies: R1NStore { - parents: XvcStore { - map: { - XvcEntity( - 2, - 16393284406203488121, - ): XvcStep { - name: "average-age", - }, - }, - entity_index: { - XvcStep { - name: "average-age", - }: [ - XvcEntity( - 2, - 16393284406203488121, - ), - ], - }, - previous: EventLog( - [ - Add { - entity: XvcEntity( - 2, - 16393284406203488121, - ), - value: XvcStep { - name: "average-age", - }, - }, - Add { - entity: XvcEntity( - 2, - 16393284406203488121, - ), - value: XvcStep { - name: "average-age", - }, - }, - ], - ), - current: EventLog( - [], - ), - }, - children: XvcStore { - map: {}, - entity_index: {}, - previous: EventLog( - [], - ), - current: EventLog( - [], - ), - }, - child_parents: XvcStore { - map: {}, - entity_index: {}, - previous: EventLog( - [], - ), - current: EventLog( - [], - ), - }, -} -[TRACE][pipeline/src/pipeline/mod.rs::624] step_e: XvcEntity( - 2, - 16393284406203488121, -) -[TRACE][pipeline/src/pipeline/mod.rs::564] dep_neighbors: Neighbors { - iter: Iter( - [], - ), - ty: PhantomData, -} -[TRACE][pipeline/src/pipeline/mod.rs::625] dependency_steps(step_e, params.dependency_graph)?: {} -[TRACE][pipeline/src/pipeline/mod.rs::564] dep_neighbors: Neighbors { - iter: Iter( - [], - ), - ty: PhantomData, -} -[TRACE][pipeline/src/pipeline/mod.rs::662] &step_state: Begin( - FromInit, -) -[TRACE][pipeline/src/pipeline/mod.rs::772] step.name: "average-age" -[TRACE][pipeline/src/pipeline/mod.rs::773] &r_next_state: WaitingDependencySteps( - FromRunConditional, -) -[TRACE][pipeline/src/pipeline/mod.rs::775] &step_state: WaitingDependencySteps( - FromRunConditional, -) -[TRACE][pipeline/src/pipeline/mod.rs::662] &step_state: WaitingDependencySteps( - FromRunConditional, -) -[TRACE][pipeline/src/pipeline/mod.rs::772] step.name: "average-age" -[TRACE][pipeline/src/pipeline/mod.rs::773] &r_next_state: CheckingOutputs( - FromDependencyStepsFinishedSuccessfully, -) -[TRACE][pipeline/src/pipeline/mod.rs::775] &step_state: CheckingOutputs( - FromDependencyStepsFinishedSuccessfully, -) -[TRACE][pipeline/src/pipeline/mod.rs::662] &step_state: CheckingOutputs( - FromDependencyStepsFinishedSuccessfully, -) -[TRACE][pipeline/src/pipeline/mod.rs::772] step.name: "average-age" -[TRACE][pipeline/src/pipeline/mod.rs::773] &r_next_state: CheckingSuperficialDiffs( - FromCheckedOutputs, -) -[TRACE][pipeline/src/pipeline/mod.rs::775] &step_state: CheckingSuperficialDiffs( - FromCheckedOutputs, -) -[TRACE][pipeline/src/pipeline/mod.rs::662] &step_state: CheckingSuperficialDiffs( - FromCheckedOutputs, -) -[TRACE][pipeline/src/pipeline/mod.rs::1010] parent_entity: XvcEntity( - 2, - 16393284406203488121, -) -[TRACE][pipeline/src/pipeline/mod.rs::1013] deps: HStore { - map: {}, -} -[TRACE][pipeline/src/pipeline/mod.rs::1016] params.step.name: "average-age" -[TRACE][pipeline/src/pipeline/mod.rs::772] step.name: "average-age" -[TRACE][pipeline/src/pipeline/mod.rs::773] &r_next_state: CheckingThoroughDiffs( - FromSuperficialDiffsChanged, -) -[TRACE][pipeline/src/pipeline/mod.rs::775] &step_state: CheckingThoroughDiffs( - FromSuperficialDiffsChanged, -) -[TRACE][pipeline/src/pipeline/mod.rs::662] &step_state: CheckingThoroughDiffs( - FromSuperficialDiffsChanged, -) -[TRACE][pipeline/src/pipeline/mod.rs::1067] deps: HStore { - map: {}, -} -[TRACE][pipeline/src/pipeline/mod.rs::772] step.name: "average-age" -[TRACE][pipeline/src/pipeline/mod.rs::773] &r_next_state: ComparingDiffsAndOutputs( - FromThoroughDiffsChanged, -) -[TRACE][pipeline/src/pipeline/mod.rs::775] &step_state: ComparingDiffsAndOutputs( - FromThoroughDiffsChanged, -) -[TRACE][pipeline/src/pipeline/mod.rs::662] &step_state: ComparingDiffsAndOutputs( - FromThoroughDiffsChanged, -) -[TRACE][pipeline/src/pipeline/mod.rs::772] step.name: "average-age" -[TRACE][pipeline/src/pipeline/mod.rs::773] &r_next_state: WaitingToRun( - FromDiffsHasChanged, -) -[TRACE][pipeline/src/pipeline/mod.rs::775] &step_state: WaitingToRun( - FromDiffsHasChanged, -) -[TRACE][pipeline/src/pipeline/mod.rs::662] &step_state: WaitingToRun( - FromDiffsHasChanged, -) -[TRACE][pipeline/src/pipeline/mod.rs::583] select: Select { .. } -[TRACE][pipeline/src/pipeline/mod.rs::583] select: Select { .. } -[TRACE][pipeline/src/pipeline/mod.rs::583] select: Select { .. } -[TRACE][pipeline/src/pipeline/mod.rs::583] select: Select { .. } -[TRACE][pipeline/src/pipeline/mod.rs::583] select: Select { .. } -[TRACE][pipeline/src/pipeline/mod.rs::583] select: Select { .. } -[TRACE][pipeline/src/pipeline/mod.rs::583] select: Select { .. } [INFO] No dependency steps for step average-age [INFO] [average-age] Dependencies has changed -[TRACE][pipeline/src/pipeline/mod.rs::1541] params: StepStateParams { - xvc_root: XvcRootInner { - absolute_path: AbsolutePath( - "[CWD]", - ), - xvc_dir: AbsolutePath( - "[CWD]/.xvc", - ), - store_dir: AbsolutePath( - "[CWD]/.xvc/store", - ), - config: XvcConfig { - current_dir: XvcConfigOption { - source: Runtime, - option: AbsolutePath( - "[CWD]", - ), - }, - config_maps: [ - XvcConfigMap { - source: Default, - map: { - "file.list.format": String( - "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", - ), - "file.track.force": Boolean( - false, - ), - "pipeline.default": String( - "default", - ), - "git.auto_commit": Boolean( - true, - ), - "file.recheck.method": String( - "copy", - ), - "pipeline.process_pool_size": Integer( - 4, - ), - "git.auto_stage": Boolean( - false, - ), - "git.use_git": Boolean( - true, - ), - "cache.algorithm": String( - "blake3", - ), - "file.track.no_parallel": Boolean( - false, - ), - "file.list.recursive": Boolean( - false, - ), - "file.list.no_summary": Boolean( - false, - ), - "file.track.no_commit": Boolean( - false, - ), - "pipeline.current_pipeline": String( - "default", - ), - "file.carry-in.no_parallel": Boolean( - false, - ), - "core.guid": String( - "603c2d6a0d2e6ea9", - ), - "file.list.show_dot_files": Boolean( - false, - ), - "file.list.sort": String( - "name-desc", - ), - "file.track.text_or_binary": String( - "auto", - ), - "file.carry-in.force": Boolean( - false, - ), - "pipeline.default_params_file": String( - "params.yaml", - ), - "git.command": String( - "git", - ), - "core.verbosity": String( - "error", - ), - }, - }, - XvcConfigMap { - source: Project, - map: { - "file.list.recursive": Boolean( - false, - ), - "pipeline.default": String( - "default", - ), - "pipeline.default_params_file": String( - "params.yaml", - ), - "file.track.no_commit": Boolean( - false, - ), - "git.use_git": Boolean( - true, - ), - "git.auto_stage": Boolean( - false, - ), - "file.track.no_parallel": Boolean( - false, - ), - "file.list.no_summary": Boolean( - false, - ), - "file.list.sort": String( - "name-desc", - ), - "core.guid": String( - "860d1874a44e1d89", - ), - "file.list.show_dot_files": Boolean( - false, - ), - "git.command": String( - "git", - ), - "file.track.text_or_binary": String( - "auto", - ), - "pipeline.process_pool_size": Integer( - 4, - ), - "file.recheck.method": String( - "copy", - ), - "file.list.format": String( - "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", - ), - "file.carry-in.no_parallel": Boolean( - false, - ), - "file.carry-in.force": Boolean( - false, - ), - "pipeline.current_pipeline": String( - "default", - ), - "file.track.force": Boolean( - false, - ), - "core.verbosity": String( - "error", - ), - "git.auto_commit": Boolean( - true, - ), - "cache.algorithm": String( - "blake3", - ), - }, - }, - XvcConfigMap { - source: Local, - map: {}, - }, - XvcConfigMap { - source: Environment, - map: { - "TRYCMD_TESTS": String( - "storage,file,pipeline,core", - ), - }, - }, - XvcConfigMap { - source: CommandLine, - map: { - "core.verbosity": String( - "debug", - ), - "core.quiet": Boolean( - false, - ), - }, - }, - ], - the_config: { - "pipeline.current_pipeline": XvcConfigValue { - source: Project, - value: String( - "default", - ), - }, - "cache.algorithm": XvcConfigValue { - source: Project, - value: String( - "blake3", - ), - }, - "pipeline.default_params_file": XvcConfigValue { - source: Project, - value: String( - "params.yaml", - ), - }, - "git.command": XvcConfigValue { - source: Project, - value: String( - "git", - ), - }, - "file.track.no_commit": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "core.verbosity": XvcConfigValue { - source: CommandLine, - value: String( - "debug", - ), - }, - "file.track.text_or_binary": XvcConfigValue { - source: Project, - value: String( - "auto", - ), - }, - "core.guid": XvcConfigValue { - source: Project, - value: String( - "860d1874a44e1d89", - ), - }, - "file.list.show_dot_files": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "file.recheck.method": XvcConfigValue { - source: Project, - value: String( - "copy", - ), - }, - "file.track.no_parallel": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "file.list.no_summary": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "TRYCMD_TESTS": XvcConfigValue { - source: Environment, - value: String( - "storage,file,pipeline,core", - ), - }, - "file.list.sort": XvcConfigValue { - source: Project, - value: String( - "name-desc", - ), - }, - "core.quiet": XvcConfigValue { - source: CommandLine, - value: Boolean( - false, - ), - }, - "git.auto_stage": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "git.use_git": XvcConfigValue { - source: Project, - value: Boolean( - true, - ), - }, - "file.carry-in.no_parallel": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "pipeline.default": XvcConfigValue { - source: Project, - value: String( - "default", - ), - }, - "file.list.recursive": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "file.track.force": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "file.list.format": XvcConfigValue { - source: Project, - value: String( - "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", - ), - }, - "pipeline.process_pool_size": XvcConfigValue { - source: Project, - value: Integer( - 4, - ), - }, - "file.carry-in.force": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "git.auto_commit": XvcConfigValue { - source: Project, - value: Boolean( - true, - ), - }, - }, - init_params: XvcConfigParams { - default_configuration: " -[core] -# The repository id. Please do not delete or change it. -# This is used to identify the repository and generate paths in storages. -# In the future it may be used to in other ways. -guid = /"603c2d6a0d2e6ea9/" -# Default verbosity level. -# One of /"error/", /"warn/", /"info/" -verbosity = /"error/" - -[git] -# Automate git operations. -# Turning this off leads Xvc to behave as if it's not in a Git repository. -# Not recommended unless you're really not using Git -use_git = true -# Command to run Git process. -# You can set this to an absolute path to specify an executable -# If set to a non-absolute path, the executable will be searched in $PATH. -command = /"git/" - -# Commit changes in .xvc/ directory after commands. -# You can set this to false if you want to commit manually. -auto_commit = true - -# Stage changes in .xvc/ directory without committing. -# auto_commit implies auto_stage. -# If you want to commit manually but don't want to stage after individual Xvc commands, you can set this to true. -auto_stage = false - -[cache] -# The hash algorithm used for the cache. -# It may take blake3, blake2, sha2 or sha3 as values. -# All algorithms are selected to produce 256-bit hashes, so sha2 means SHA2-256, blake2 means BLAKE2s, etc. -# The cache path is produced by prepending algorithm name to the cache. -# Blake3 files are in .xvc/b3/, while sha2 files are in .xvc/s2/ etc. -algorithm = /"blake3/" - -[file] - -[file.track] - -# Don't move file content to cache after xvc file track -no_commit = false -# Force to track files even if they are already tracked. -force = false - -# Xvc calculates file content digest differently for text and binary files. -# This option controls whether to treat files as text or binary. -# It may take auto, text or binary as values. -# Auto check each file individually and treat it as text if it's text. -text_or_binary = /"auto/" - -# Don't use parallelism in track operations. -# Note that some of the operations are implemented in parallel by default, and this option affects some heavier operations. -no_parallel = false - -[file.list] - -# Format for `xvc file list` rows. You can reorder or remove columns. -# The following are the keys for each row: -# - {acd64}: actual content digest. All 64 digits from the workspace file's content. -# - {acd8}: actual content digest. First 8 digits the file content digest. -# - {aft}: actual file type. Whether the entry is a file (F), directory (D), -# symlink (S), hardlink (H) or reflink (R). -# - {asz}: actual size. The size of the workspace file in bytes. It uses MB, -# GB and TB to represent sizes larger than 1MB. -# - {ats}: actual timestamp. The timestamp of the workspace file. -# - {cst}: cache status. One of /"=/", /">/", /"/", /" 0 - F 000001: - >000002: /x00 => 2 - 000003: /x00 => 0 - match kind: Standard - prefilter: false - state length: 4 - pattern length: 0 - shortest pattern length: 18446744073709551615 - longest pattern length: 0 - alphabet length: 1 - stride: 1 - byte classes: ByteClasses(0 => [0-255]) - memory usage: 16 - ) - , - ), - map: [], - longest: 0, - }, - ), - Prefix( - PrefixStrategy { - matcher: AhoCorasick( - dfa::DFA( - D 000000: /x00 => 0 - F 000001: - >000002: /x00 => 2 - 000003: /x00 => 0 - match kind: Standard - prefilter: false - state length: 4 - pattern length: 0 - shortest pattern length: 18446744073709551615 - longest pattern length: 0 - alphabet length: 1 - stride: 1 - byte classes: ByteClasses(0 => [0-255]) - memory usage: 16 - ) - , - ), - map: [], - longest: 0, - }, - ), - RequiredExtension( - RequiredExtensionStrategy( - {}, - ), - ), - Regex( - RegexSetStrategy { - matcher: Regex { - imp: RegexI { - strat: Core { - info: RegexInfo( - RegexInfoI { - config: Config { - match_kind: Some( - All, - ), - utf8_empty: Some( - false, - ), - autopre: None, - pre: None, - which_captures: None, - nfa_size_limit: Some( - Some( - 10485760, - ), - ), - onepass_size_limit: None, - hybrid_cache_capacity: Some( - 10485760, - ), - hybrid: None, - dfa: None, - dfa_size_limit: None, - dfa_state_limit: None, - onepass: None, - backtrack: None, - byte_classes: None, - line_terminator: None, - }, - props: [], - props_union: Properties( - PropertiesI { - minimum_len: None, - maximum_len: None, - look_set: ∅, - look_set_prefix: ∅, - look_set_suffix: ∅, - look_set_prefix_any: ∅, - look_set_suffix_any: ∅, - utf8: true, - explicit_captures_len: 0, - static_explicit_captures_len: None, - literal: false, - alternation_literal: true, - }, - ), - }, - ), - pre: None, - nfa: thompson::NFA( - ^000000: FAIL - - transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) - ) - , - nfarev: Some( - thompson::NFA( - ^000000: FAIL - - transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) - ) - , - ), - pikevm: PikeVM( - PikeVMEngine( - PikeVM { - config: Config { - match_kind: Some( - All, - ), - pre: Some( - None, - ), - }, - nfa: thompson::NFA( - ^000000: FAIL - - transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) - ) - , - }, - ), - ), - backtrack: BoundedBacktracker( - None, - ), - onepass: OnePass( - None, - ), - hybrid: Hybrid( - Some( - HybridEngine( - Regex { - forward: DFA { - config: Config { - match_kind: Some( - All, - ), - pre: Some( - None, - ), - starts_for_each_pattern: Some( - true, - ), - byte_classes: Some( - true, - ), - unicode_word_boundary: Some( - true, - ), - quitset: None, - specialize_start_states: Some( - false, - ), - cache_capacity: Some( - 10485760, - ), - skip_cache_capacity_check: Some( - false, - ), - minimum_cache_clear_count: Some( - Some( - 3, - ), - ), - minimum_bytes_per_state: Some( - Some( - 10, - ), - ), - }, - nfa: thompson::NFA( - ^000000: FAIL - - transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) - ) - , - stride2: 1, - start_map: StartByteMap{/x00 => NonWordByte, /x01 => NonWordByte, /x02 => NonWordByte, /x03 => NonWordByte, /x04 => NonWordByte, /x05 => NonWordByte, /x06 => NonWordByte, /x07 => NonWordByte, /x08 => NonWordByte, /t => NonWordByte, - => LineLF, /x0B => NonWordByte, /x0C => NonWordByte, /r => LineCR, /x0E => NonWordByte, /x0F => NonWordByte, /x10 => NonWordByte, /x11 => NonWordByte, /x12 => NonWordByte, /x13 => NonWordByte, /x14 => NonWordByte, /x15 => NonWordByte, /x16 => NonWordByte, /x17 => NonWordByte, /x18 => NonWordByte, /x19 => NonWordByte, /x1A => NonWordByte, /x1B => NonWordByte, /x1C => NonWordByte, /x1D => NonWordByte, /x1E => NonWordByte, /x1F => NonWordByte, ' ' => NonWordByte, ! => NonWordByte, /" => NonWordByte, # => NonWordByte, $ => NonWordByte, % => NonWordByte, & => NonWordByte, /' => NonWordByte, ( => NonWordByte, ) => NonWordByte, * => NonWordByte, + => NonWordByte, , => NonWordByte, - => NonWordByte, . => NonWordByte, / => NonWordByte, 0 => WordByte, 1 => WordByte, 2 => WordByte, 3 => WordByte, 4 => WordByte, 5 => WordByte, 6 => WordByte, 7 => WordByte, 8 => WordByte, 9 => WordByte, : => NonWordByte, ; => NonWordByte, < => NonWordByte, = => NonWordByte, > => NonWordByte, ? => NonWordByte, @ => NonWordByte, A => WordByte, B => WordByte, C => WordByte, D => WordByte, E => WordByte, F => WordByte, G => WordByte, H => WordByte, I => WordByte, J => WordByte, K => WordByte, L => WordByte, M => WordByte, N => WordByte, O => WordByte, P => WordByte, Q => WordByte, R => WordByte, S => WordByte, T => WordByte, U => WordByte, V => WordByte, W => WordByte, X => WordByte, Y => WordByte, Z => WordByte, [ => NonWordByte, // => NonWordByte, ] => NonWordByte, ^ => NonWordByte, _ => WordByte, ` => NonWordByte, a => WordByte, b => WordByte, c => WordByte, d => WordByte, e => WordByte, f => WordByte, g => WordByte, h => WordByte, i => WordByte, j => WordByte, k => WordByte, l => WordByte, m => WordByte, n => WordByte, o => WordByte, p => WordByte, q => WordByte, r => WordByte, s => WordByte, t => WordByte, u => WordByte, v => WordByte, w => WordByte, x => WordByte, y => WordByte, z => WordByte, { => NonWordByte, | => NonWordByte, } => NonWordByte, ~ => NonWordByte, /x7F => NonWordByte, /x80 => NonWordByte, /x81 => NonWordByte, /x82 => NonWordByte, /x83 => NonWordByte, /x84 => NonWordByte, /x85 => NonWordByte, /x86 => NonWordByte, /x87 => NonWordByte, /x88 => NonWordByte, /x89 => NonWordByte, /x8A => NonWordByte, /x8B => NonWordByte, /x8C => NonWordByte, /x8D => NonWordByte, /x8E => NonWordByte, /x8F => NonWordByte, /x90 => NonWordByte, /x91 => NonWordByte, /x92 => NonWordByte, /x93 => NonWordByte, /x94 => NonWordByte, /x95 => NonWordByte, /x96 => NonWordByte, /x97 => NonWordByte, /x98 => NonWordByte, /x99 => NonWordByte, /x9A => NonWordByte, /x9B => NonWordByte, /x9C => NonWordByte, /x9D => NonWordByte, /x9E => NonWordByte, /x9F => NonWordByte, /xA0 => NonWordByte, /xA1 => NonWordByte, /xA2 => NonWordByte, /xA3 => NonWordByte, /xA4 => NonWordByte, /xA5 => NonWordByte, /xA6 => NonWordByte, /xA7 => NonWordByte, /xA8 => NonWordByte, /xA9 => NonWordByte, /xAA => NonWordByte, /xAB => NonWordByte, /xAC => NonWordByte, /xAD => NonWordByte, /xAE => NonWordByte, /xAF => NonWordByte, /xB0 => NonWordByte, /xB1 => NonWordByte, /xB2 => NonWordByte, /xB3 => NonWordByte, /xB4 => NonWordByte, /xB5 => NonWordByte, /xB6 => NonWordByte, /xB7 => NonWordByte, /xB8 => NonWordByte, /xB9 => NonWordByte, /xBA => NonWordByte, /xBB => NonWordByte, /xBC => NonWordByte, /xBD => NonWordByte, /xBE => NonWordByte, /xBF => NonWordByte, /xC0 => NonWordByte, /xC1 => NonWordByte, /xC2 => NonWordByte, /xC3 => NonWordByte, /xC4 => NonWordByte, /xC5 => NonWordByte, /xC6 => NonWordByte, /xC7 => NonWordByte, /xC8 => NonWordByte, /xC9 => NonWordByte, /xCA => NonWordByte, /xCB => NonWordByte, /xCC => NonWordByte, /xCD => NonWordByte, /xCE => NonWordByte, /xCF => NonWordByte, /xD0 => NonWordByte, /xD1 => NonWordByte, /xD2 => NonWordByte, /xD3 => NonWordByte, /xD4 => NonWordByte, /xD5 => NonWordByte, /xD6 => NonWordByte, /xD7 => NonWordByte, /xD8 => NonWordByte, /xD9 => NonWordByte, /xDA => NonWordByte, /xDB => NonWordByte, /xDC => NonWordByte, /xDD => NonWordByte, /xDE => NonWordByte, /xDF => NonWordByte, /xE0 => NonWordByte, /xE1 => NonWordByte, /xE2 => NonWordByte, /xE3 => NonWordByte, /xE4 => NonWordByte, /xE5 => NonWordByte, /xE6 => NonWordByte, /xE7 => NonWordByte, /xE8 => NonWordByte, /xE9 => NonWordByte, /xEA => NonWordByte, /xEB => NonWordByte, /xEC => NonWordByte, /xED => NonWordByte, /xEE => NonWordByte, /xEF => NonWordByte, /xF0 => NonWordByte, /xF1 => NonWordByte, /xF2 => NonWordByte, /xF3 => NonWordByte, /xF4 => NonWordByte, /xF5 => NonWordByte, /xF6 => NonWordByte, /xF7 => NonWordByte, /xF8 => NonWordByte, /xF9 => NonWordByte, /xFA => NonWordByte, /xFB => NonWordByte, /xFC => NonWordByte, /xFD => NonWordByte, /xFE => NonWordByte, /xFF => NonWordByte}, - classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]), - quitset: ByteSet { - bits: {}, - }, - cache_capacity: 10485760, - }, - reverse: DFA { - config: Config { - match_kind: Some( - All, - ), - pre: Some( - None, - ), - starts_for_each_pattern: Some( - true, - ), - byte_classes: Some( - true, - ), - unicode_word_boundary: Some( - true, - ), - quitset: None, - specialize_start_states: Some( - false, - ), - cache_capacity: Some( - 10485760, - ), - skip_cache_capacity_check: Some( - false, - ), - minimum_cache_clear_count: Some( - Some( - 3, - ), - ), - minimum_bytes_per_state: Some( - Some( - 10, - ), - ), - }, - nfa: thompson::NFA( - ^000000: FAIL - - transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) - ) - , - stride2: 1, - start_map: StartByteMap{/x00 => NonWordByte, /x01 => NonWordByte, /x02 => NonWordByte, /x03 => NonWordByte, /x04 => NonWordByte, /x05 => NonWordByte, /x06 => NonWordByte, /x07 => NonWordByte, /x08 => NonWordByte, /t => NonWordByte, - => LineLF, /x0B => NonWordByte, /x0C => NonWordByte, /r => LineCR, /x0E => NonWordByte, /x0F => NonWordByte, /x10 => NonWordByte, /x11 => NonWordByte, /x12 => NonWordByte, /x13 => NonWordByte, /x14 => NonWordByte, /x15 => NonWordByte, /x16 => NonWordByte, /x17 => NonWordByte, /x18 => NonWordByte, /x19 => NonWordByte, /x1A => NonWordByte, /x1B => NonWordByte, /x1C => NonWordByte, /x1D => NonWordByte, /x1E => NonWordByte, /x1F => NonWordByte, ' ' => NonWordByte, ! => NonWordByte, /" => NonWordByte, # => NonWordByte, $ => NonWordByte, % => NonWordByte, & => NonWordByte, /' => NonWordByte, ( => NonWordByte, ) => NonWordByte, * => NonWordByte, + => NonWordByte, , => NonWordByte, - => NonWordByte, . => NonWordByte, / => NonWordByte, 0 => WordByte, 1 => WordByte, 2 => WordByte, 3 => WordByte, 4 => WordByte, 5 => WordByte, 6 => WordByte, 7 => WordByte, 8 => WordByte, 9 => WordByte, : => NonWordByte, ; => NonWordByte, < => NonWordByte, = => NonWordByte, > => NonWordByte, ? => NonWordByte, @ => NonWordByte, A => WordByte, B => WordByte, C => WordByte, D => WordByte, E => WordByte, F => WordByte, G => WordByte, H => WordByte, I => WordByte, J => WordByte, K => WordByte, L => WordByte, M => WordByte, N => WordByte, O => WordByte, P => WordByte, Q => WordByte, R => WordByte, S => WordByte, T => WordByte, U => WordByte, V => WordByte, W => WordByte, X => WordByte, Y => WordByte, Z => WordByte, [ => NonWordByte, // => NonWordByte, ] => NonWordByte, ^ => NonWordByte, _ => WordByte, ` => NonWordByte, a => WordByte, b => WordByte, c => WordByte, d => WordByte, e => WordByte, f => WordByte, g => WordByte, h => WordByte, i => WordByte, j => WordByte, k => WordByte, l => WordByte, m => WordByte, n => WordByte, o => WordByte, p => WordByte, q => WordByte, r => WordByte, s => WordByte, t => WordByte, u => WordByte, v => WordByte, w => WordByte, x => WordByte, y => WordByte, z => WordByte, { => NonWordByte, | => NonWordByte, } => NonWordByte, ~ => NonWordByte, /x7F => NonWordByte, /x80 => NonWordByte, /x81 => NonWordByte, /x82 => NonWordByte, /x83 => NonWordByte, /x84 => NonWordByte, /x85 => NonWordByte, /x86 => NonWordByte, /x87 => NonWordByte, /x88 => NonWordByte, /x89 => NonWordByte, /x8A => NonWordByte, /x8B => NonWordByte, /x8C => NonWordByte, /x8D => NonWordByte, /x8E => NonWordByte, /x8F => NonWordByte, /x90 => NonWordByte, /x91 => NonWordByte, /x92 => NonWordByte, /x93 => NonWordByte, /x94 => NonWordByte, /x95 => NonWordByte, /x96 => NonWordByte, /x97 => NonWordByte, /x98 => NonWordByte, /x99 => NonWordByte, /x9A => NonWordByte, /x9B => NonWordByte, /x9C => NonWordByte, /x9D => NonWordByte, /x9E => NonWordByte, /x9F => NonWordByte, /xA0 => NonWordByte, /xA1 => NonWordByte, /xA2 => NonWordByte, /xA3 => NonWordByte, /xA4 => NonWordByte, /xA5 => NonWordByte, /xA6 => NonWordByte, /xA7 => NonWordByte, /xA8 => NonWordByte, /xA9 => NonWordByte, /xAA => NonWordByte, /xAB => NonWordByte, /xAC => NonWordByte, /xAD => NonWordByte, /xAE => NonWordByte, /xAF => NonWordByte, /xB0 => NonWordByte, /xB1 => NonWordByte, /xB2 => NonWordByte, /xB3 => NonWordByte, /xB4 => NonWordByte, /xB5 => NonWordByte, /xB6 => NonWordByte, /xB7 => NonWordByte, /xB8 => NonWordByte, /xB9 => NonWordByte, /xBA => NonWordByte, /xBB => NonWordByte, /xBC => NonWordByte, /xBD => NonWordByte, /xBE => NonWordByte, /xBF => NonWordByte, /xC0 => NonWordByte, /xC1 => NonWordByte, /xC2 => NonWordByte, /xC3 => NonWordByte, /xC4 => NonWordByte, /xC5 => NonWordByte, /xC6 => NonWordByte, /xC7 => NonWordByte, /xC8 => NonWordByte, /xC9 => NonWordByte, /xCA => NonWordByte, /xCB => NonWordByte, /xCC => NonWordByte, /xCD => NonWordByte, /xCE => NonWordByte, /xCF => NonWordByte, /xD0 => NonWordByte, /xD1 => NonWordByte, /xD2 => NonWordByte, /xD3 => NonWordByte, /xD4 => NonWordByte, /xD5 => NonWordByte, /xD6 => NonWordByte, /xD7 => NonWordByte, /xD8 => NonWordByte, /xD9 => NonWordByte, /xDA => NonWordByte, /xDB => NonWordByte, /xDC => NonWordByte, /xDD => NonWordByte, /xDE => NonWordByte, /xDF => NonWordByte, /xE0 => NonWordByte, /xE1 => NonWordByte, /xE2 => NonWordByte, /xE3 => NonWordByte, /xE4 => NonWordByte, /xE5 => NonWordByte, /xE6 => NonWordByte, /xE7 => NonWordByte, /xE8 => NonWordByte, /xE9 => NonWordByte, /xEA => NonWordByte, /xEB => NonWordByte, /xEC => NonWordByte, /xED => NonWordByte, /xEE => NonWordByte, /xEF => NonWordByte, /xF0 => NonWordByte, /xF1 => NonWordByte, /xF2 => NonWordByte, /xF3 => NonWordByte, /xF4 => NonWordByte, /xF5 => NonWordByte, /xF6 => NonWordByte, /xF7 => NonWordByte, /xF8 => NonWordByte, /xF9 => NonWordByte, /xFA => NonWordByte, /xFB => NonWordByte, /xFC => NonWordByte, /xFD => NonWordByte, /xFE => NonWordByte, /xFF => NonWordByte}, - classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]), - quitset: ByteSet { - bits: {}, - }, - cache_capacity: 10485760, - }, - }, - ), - ), - ), - dfa: DFA( - None, - ), - }, - info: RegexInfo( - RegexInfoI { - config: Config { - match_kind: Some( - All, - ), - utf8_empty: Some( - false, - ), - autopre: None, - pre: None, - which_captures: None, - nfa_size_limit: Some( - Some( - 10485760, - ), - ), - onepass_size_limit: None, - hybrid_cache_capacity: Some( - 10485760, - ), - hybrid: None, - dfa: None, - dfa_size_limit: None, - dfa_state_limit: None, - onepass: None, - backtrack: None, - byte_classes: None, - line_terminator: None, - }, - props: [], - props_union: Properties( - PropertiesI { - minimum_len: None, - maximum_len: None, - look_set: ∅, - look_set_prefix: ∅, - look_set_suffix: ∅, - look_set_prefix_any: ∅, - look_set_suffix_any: ∅, - utf8: true, - explicit_captures_len: 0, - static_explicit_captures_len: None, - literal: false, - alternation_literal: true, - }, - ), - }, - ), - }, - pool: Pool( - Pool { - stacks: [ - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - ], - owner: 4, - owner_val: UnsafeCell { .. }, - }, - ), - }, - map: [], - patset: Pool( - Pool { - stacks: [ - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - ], - owner: 0, - owner_val: UnsafeCell { .. }, - }, - ), - }, - ), - ], - }, - poisoned: false, - .. - }, - }, - }, - run_conditions: RunConditions { - never: false, - always: true, - ignore_broken_dep_steps: true, - ignore_missing_outputs: true, - }, - pipeline_rundir: XvcPath( - "", - ), - terminate_timeout_processes: true, - algorithm: Blake3, - command_process: RwLock { - data: CommandProcess { - environment: {}, - step: XvcStep { - name: "average-age", - }, - step_command: XvcStepCommand { - command: "sqlite3 people.db 'SELECT AVG(Age) FROM People;'", - }, - birth: None, - process: None, - stdout_sender: Sender { .. }, - stderr_sender: Sender { .. }, - stdout_receiver: Receiver { .. }, - stderr_receiver: Receiver { .. }, - }, - poisoned: false, - .. - }, - available_process_slots: RwLock { - data: 4, - poisoned: false, - .. - }, - process_poll_milliseconds: 10, - dependency_diffs: RwLock { - data: HStore { - map: {}, - }, - poisoned: false, - .. - }, - output_diffs: RwLock { - data: HStore { - map: {}, - }, - poisoned: false, - .. - }, - step_e: XvcEntity( - 2, - 16393284406203488121, - ), - step: XvcStep { - name: "average-age", - }, - step_command: XvcStepCommand { - command: "sqlite3 people.db 'SELECT AVG(Age) FROM People;'", - }, - current_states: RwLock { - data: HStore { - map: { - XvcEntity( - 2, - 16393284406203488121, - ): WaitingToRun( - FromDiffsHasChanged, - ), - }, - }, - poisoned: false, - .. - }, - step_timeout: 10000s, - all_steps: HStore { - map: { - XvcEntity( - 2, - 16393284406203488121, - ): XvcStep { - name: "average-age", - }, - }, - }, - recorded_dependencies: R1NStore { - parents: XvcStore { - map: { - XvcEntity( - 2, - 16393284406203488121, - ): XvcStep { - name: "average-age", - }, - }, - entity_index: { - XvcStep { - name: "average-age", - }: [ - XvcEntity( - 2, - 16393284406203488121, - ), - ], - }, - previous: EventLog( - [ - Add { - entity: XvcEntity( - 2, - 16393284406203488121, - ), - value: XvcStep { - name: "average-age", - }, - }, - Add { - entity: XvcEntity( - 2, - 16393284406203488121, - ), - value: XvcStep { - name: "average-age", - }, - }, - ], - ), - current: EventLog( - [], - ), - }, - children: XvcStore { - map: {}, - entity_index: {}, - previous: EventLog( - [], - ), - current: EventLog( - [], - ), - }, - child_parents: XvcStore { - map: {}, - entity_index: {}, - previous: EventLog( - [], - ), - current: EventLog( - [], - ), - }, - }, - step_dependencies: {}, - step_outputs: HStore { - map: {}, - }, -} -[TRACE][pipeline/src/pipeline/mod.rs::772] step.name: "average-age" -[TRACE][pipeline/src/pipeline/mod.rs::773] &r_next_state: Running( - FromStartProcess, -) -[TRACE][pipeline/src/pipeline/mod.rs::775] &step_state: Running( - FromStartProcess, -) -[TRACE][pipeline/src/pipeline/mod.rs::662] &step_state: Running( - FromStartProcess, -) -[TRACE][pipeline/src/pipeline/command.rs::96] self.environment: {} -[TRACE][pipeline/src/pipeline/mod.rs::583] select: Select { .. } -[TRACE][pipeline/src/pipeline/mod.rs::772] step.name: "average-age" -[TRACE][pipeline/src/pipeline/mod.rs::773] &r_next_state: Running( - FromWaitProcess, -) -[TRACE][pipeline/src/pipeline/mod.rs::775] &step_state: Running( - FromWaitProcess, -) -[TRACE][pipeline/src/pipeline/mod.rs::662] &step_state: Running( - FromWaitProcess, -) -[TRACE][pipeline/src/pipeline/mod.rs::583] select: Select { .. } -[TRACE][pipeline/src/pipeline/mod.rs::1420] params: StepStateParams { - xvc_root: XvcRootInner { - absolute_path: AbsolutePath( - "[CWD]", - ), - xvc_dir: AbsolutePath( - "[CWD]/.xvc", - ), - store_dir: AbsolutePath( - "[CWD]/.xvc/store", - ), - config: XvcConfig { - current_dir: XvcConfigOption { - source: Runtime, - option: AbsolutePath( - "[CWD]", - ), - }, - config_maps: [ - XvcConfigMap { - source: Default, - map: { - "file.list.format": String( - "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", - ), - "file.track.force": Boolean( - false, - ), - "pipeline.default": String( - "default", - ), - "git.auto_commit": Boolean( - true, - ), - "file.recheck.method": String( - "copy", - ), - "pipeline.process_pool_size": Integer( - 4, - ), - "git.auto_stage": Boolean( - false, - ), - "git.use_git": Boolean( - true, - ), - "cache.algorithm": String( - "blake3", - ), - "file.track.no_parallel": Boolean( - false, - ), - "file.list.recursive": Boolean( - false, - ), - "file.list.no_summary": Boolean( - false, - ), - "file.track.no_commit": Boolean( - false, - ), - "pipeline.current_pipeline": String( - "default", - ), - "file.carry-in.no_parallel": Boolean( - false, - ), - "core.guid": String( - "603c2d6a0d2e6ea9", - ), - "file.list.show_dot_files": Boolean( - false, - ), - "file.list.sort": String( - "name-desc", - ), - "file.track.text_or_binary": String( - "auto", - ), - "file.carry-in.force": Boolean( - false, - ), - "pipeline.default_params_file": String( - "params.yaml", - ), - "git.command": String( - "git", - ), - "core.verbosity": String( - "error", - ), - }, - }, - XvcConfigMap { - source: Project, - map: { - "file.list.recursive": Boolean( - false, - ), - "pipeline.default": String( - "default", - ), - "pipeline.default_params_file": String( - "params.yaml", - ), - "file.track.no_commit": Boolean( - false, - ), - "git.use_git": Boolean( - true, - ), - "git.auto_stage": Boolean( - false, - ), - "file.track.no_parallel": Boolean( - false, - ), - "file.list.no_summary": Boolean( - false, - ), - "file.list.sort": String( - "name-desc", - ), - "core.guid": String( - "860d1874a44e1d89", - ), - "file.list.show_dot_files": Boolean( - false, - ), - "git.command": String( - "git", - ), - "file.track.text_or_binary": String( - "auto", - ), - "pipeline.process_pool_size": Integer( - 4, - ), - "file.recheck.method": String( - "copy", - ), - "file.list.format": String( - "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", - ), - "file.carry-in.no_parallel": Boolean( - false, - ), - "file.carry-in.force": Boolean( - false, - ), - "pipeline.current_pipeline": String( - "default", - ), - "file.track.force": Boolean( - false, - ), - "core.verbosity": String( - "error", - ), - "git.auto_commit": Boolean( - true, - ), - "cache.algorithm": String( - "blake3", - ), - }, - }, - XvcConfigMap { - source: Local, - map: {}, - }, - XvcConfigMap { - source: Environment, - map: { - "TRYCMD_TESTS": String( - "storage,file,pipeline,core", - ), - }, - }, - XvcConfigMap { - source: CommandLine, - map: { - "core.verbosity": String( - "debug", - ), - "core.quiet": Boolean( - false, - ), - }, - }, - ], - the_config: { - "pipeline.current_pipeline": XvcConfigValue { - source: Project, - value: String( - "default", - ), - }, - "cache.algorithm": XvcConfigValue { - source: Project, - value: String( - "blake3", - ), - }, - "pipeline.default_params_file": XvcConfigValue { - source: Project, - value: String( - "params.yaml", - ), - }, - "git.command": XvcConfigValue { - source: Project, - value: String( - "git", - ), - }, - "file.track.no_commit": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "core.verbosity": XvcConfigValue { - source: CommandLine, - value: String( - "debug", - ), - }, - "file.track.text_or_binary": XvcConfigValue { - source: Project, - value: String( - "auto", - ), - }, - "core.guid": XvcConfigValue { - source: Project, - value: String( - "860d1874a44e1d89", - ), - }, - "file.list.show_dot_files": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "file.recheck.method": XvcConfigValue { - source: Project, - value: String( - "copy", - ), - }, - "file.track.no_parallel": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "file.list.no_summary": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "TRYCMD_TESTS": XvcConfigValue { - source: Environment, - value: String( - "storage,file,pipeline,core", - ), - }, - "file.list.sort": XvcConfigValue { - source: Project, - value: String( - "name-desc", - ), - }, - "core.quiet": XvcConfigValue { - source: CommandLine, - value: Boolean( - false, - ), - }, - "git.auto_stage": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "git.use_git": XvcConfigValue { - source: Project, - value: Boolean( - true, - ), - }, - "file.carry-in.no_parallel": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "pipeline.default": XvcConfigValue { - source: Project, - value: String( - "default", - ), - }, - "file.list.recursive": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "file.track.force": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "file.list.format": XvcConfigValue { - source: Project, - value: String( - "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", - ), - }, - "pipeline.process_pool_size": XvcConfigValue { - source: Project, - value: Integer( - 4, - ), - }, - "file.carry-in.force": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "git.auto_commit": XvcConfigValue { - source: Project, - value: Boolean( - true, - ), - }, - }, - init_params: XvcConfigParams { - default_configuration: " -[core] -# The repository id. Please do not delete or change it. -# This is used to identify the repository and generate paths in storages. -# In the future it may be used to in other ways. -guid = /"603c2d6a0d2e6ea9/" -# Default verbosity level. -# One of /"error/", /"warn/", /"info/" -verbosity = /"error/" - -[git] -# Automate git operations. -# Turning this off leads Xvc to behave as if it's not in a Git repository. -# Not recommended unless you're really not using Git -use_git = true -# Command to run Git process. -# You can set this to an absolute path to specify an executable -# If set to a non-absolute path, the executable will be searched in $PATH. -command = /"git/" - -# Commit changes in .xvc/ directory after commands. -# You can set this to false if you want to commit manually. -auto_commit = true - -# Stage changes in .xvc/ directory without committing. -# auto_commit implies auto_stage. -# If you want to commit manually but don't want to stage after individual Xvc commands, you can set this to true. -auto_stage = false - -[cache] -# The hash algorithm used for the cache. -# It may take blake3, blake2, sha2 or sha3 as values. -# All algorithms are selected to produce 256-bit hashes, so sha2 means SHA2-256, blake2 means BLAKE2s, etc. -# The cache path is produced by prepending algorithm name to the cache. -# Blake3 files are in .xvc/b3/, while sha2 files are in .xvc/s2/ etc. -algorithm = /"blake3/" - -[file] - -[file.track] - -# Don't move file content to cache after xvc file track -no_commit = false -# Force to track files even if they are already tracked. -force = false - -# Xvc calculates file content digest differently for text and binary files. -# This option controls whether to treat files as text or binary. -# It may take auto, text or binary as values. -# Auto check each file individually and treat it as text if it's text. -text_or_binary = /"auto/" - -# Don't use parallelism in track operations. -# Note that some of the operations are implemented in parallel by default, and this option affects some heavier operations. -no_parallel = false - -[file.list] - -# Format for `xvc file list` rows. You can reorder or remove columns. -# The following are the keys for each row: -# - {acd64}: actual content digest. All 64 digits from the workspace file's content. -# - {acd8}: actual content digest. First 8 digits the file content digest. -# - {aft}: actual file type. Whether the entry is a file (F), directory (D), -# symlink (S), hardlink (H) or reflink (R). -# - {asz}: actual size. The size of the workspace file in bytes. It uses MB, -# GB and TB to represent sizes larger than 1MB. -# - {ats}: actual timestamp. The timestamp of the workspace file. -# - {cst}: cache status. One of /"=/", /">/", /"/", /" 0 - F 000001: - >000002: /x00 => 2 - 000003: /x00 => 0 - match kind: Standard - prefilter: false - state length: 4 - pattern length: 0 - shortest pattern length: 18446744073709551615 - longest pattern length: 0 - alphabet length: 1 - stride: 1 - byte classes: ByteClasses(0 => [0-255]) - memory usage: 16 - ) - , - ), - map: [], - longest: 0, - }, - ), - Prefix( - PrefixStrategy { - matcher: AhoCorasick( - dfa::DFA( - D 000000: /x00 => 0 - F 000001: - >000002: /x00 => 2 - 000003: /x00 => 0 - match kind: Standard - prefilter: false - state length: 4 - pattern length: 0 - shortest pattern length: 18446744073709551615 - longest pattern length: 0 - alphabet length: 1 - stride: 1 - byte classes: ByteClasses(0 => [0-255]) - memory usage: 16 - ) - , - ), - map: [], - longest: 0, - }, - ), - RequiredExtension( - RequiredExtensionStrategy( - {}, - ), - ), - Regex( - RegexSetStrategy { - matcher: Regex { - imp: RegexI { - strat: Core { - info: RegexInfo( - RegexInfoI { - config: Config { - match_kind: Some( - All, - ), - utf8_empty: Some( - false, - ), - autopre: None, - pre: None, - which_captures: None, - nfa_size_limit: Some( - Some( - 10485760, - ), - ), - onepass_size_limit: None, - hybrid_cache_capacity: Some( - 10485760, - ), - hybrid: None, - dfa: None, - dfa_size_limit: None, - dfa_state_limit: None, - onepass: None, - backtrack: None, - byte_classes: None, - line_terminator: None, - }, - props: [], - props_union: Properties( - PropertiesI { - minimum_len: None, - maximum_len: None, - look_set: ∅, - look_set_prefix: ∅, - look_set_suffix: ∅, - look_set_prefix_any: ∅, - look_set_suffix_any: ∅, - utf8: true, - explicit_captures_len: 0, - static_explicit_captures_len: None, - literal: false, - alternation_literal: true, - }, - ), - }, - ), - pre: None, - nfa: thompson::NFA( - ^000000: FAIL - - transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) - ) - , - nfarev: Some( - thompson::NFA( - ^000000: FAIL - - transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) - ) - , - ), - pikevm: PikeVM( - PikeVMEngine( - PikeVM { - config: Config { - match_kind: Some( - All, - ), - pre: Some( - None, - ), - }, - nfa: thompson::NFA( - ^000000: FAIL - - transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) - ) - , - }, - ), - ), - backtrack: BoundedBacktracker( - None, - ), - onepass: OnePass( - None, - ), - hybrid: Hybrid( - Some( - HybridEngine( - Regex { - forward: DFA { - config: Config { - match_kind: Some( - All, - ), - pre: Some( - None, - ), - starts_for_each_pattern: Some( - true, - ), - byte_classes: Some( - true, - ), - unicode_word_boundary: Some( - true, - ), - quitset: None, - specialize_start_states: Some( - false, - ), - cache_capacity: Some( - 10485760, - ), - skip_cache_capacity_check: Some( - false, - ), - minimum_cache_clear_count: Some( - Some( - 3, - ), - ), - minimum_bytes_per_state: Some( - Some( - 10, - ), - ), - }, - nfa: thompson::NFA( - ^000000: FAIL - - transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) - ) - , - stride2: 1, - start_map: StartByteMap{/x00 => NonWordByte, /x01 => NonWordByte, /x02 => NonWordByte, /x03 => NonWordByte, /x04 => NonWordByte, /x05 => NonWordByte, /x06 => NonWordByte, /x07 => NonWordByte, /x08 => NonWordByte, /t => NonWordByte, - => LineLF, /x0B => NonWordByte, /x0C => NonWordByte, /r => LineCR, /x0E => NonWordByte, /x0F => NonWordByte, /x10 => NonWordByte, /x11 => NonWordByte, /x12 => NonWordByte, /x13 => NonWordByte, /x14 => NonWordByte, /x15 => NonWordByte, /x16 => NonWordByte, /x17 => NonWordByte, /x18 => NonWordByte, /x19 => NonWordByte, /x1A => NonWordByte, /x1B => NonWordByte, /x1C => NonWordByte, /x1D => NonWordByte, /x1E => NonWordByte, /x1F => NonWordByte, ' ' => NonWordByte, ! => NonWordByte, /" => NonWordByte, # => NonWordByte, $ => NonWordByte, % => NonWordByte, & => NonWordByte, /' => NonWordByte, ( => NonWordByte, ) => NonWordByte, * => NonWordByte, + => NonWordByte, , => NonWordByte, - => NonWordByte, . => NonWordByte, / => NonWordByte, 0 => WordByte, 1 => WordByte, 2 => WordByte, 3 => WordByte, 4 => WordByte, 5 => WordByte, 6 => WordByte, 7 => WordByte, 8 => WordByte, 9 => WordByte, : => NonWordByte, ; => NonWordByte, < => NonWordByte, = => NonWordByte, > => NonWordByte, ? => NonWordByte, @ => NonWordByte, A => WordByte, B => WordByte, C => WordByte, D => WordByte, E => WordByte, F => WordByte, G => WordByte, H => WordByte, I => WordByte, J => WordByte, K => WordByte, L => WordByte, M => WordByte, N => WordByte, O => WordByte, P => WordByte, Q => WordByte, R => WordByte, S => WordByte, T => WordByte, U => WordByte, V => WordByte, W => WordByte, X => WordByte, Y => WordByte, Z => WordByte, [ => NonWordByte, // => NonWordByte, ] => NonWordByte, ^ => NonWordByte, _ => WordByte, ` => NonWordByte, a => WordByte, b => WordByte, c => WordByte, d => WordByte, e => WordByte, f => WordByte, g => WordByte, h => WordByte, i => WordByte, j => WordByte, k => WordByte, l => WordByte, m => WordByte, n => WordByte, o => WordByte, p => WordByte, q => WordByte, r => WordByte, s => WordByte, t => WordByte, u => WordByte, v => WordByte, w => WordByte, x => WordByte, y => WordByte, z => WordByte, { => NonWordByte, | => NonWordByte, } => NonWordByte, ~ => NonWordByte, /x7F => NonWordByte, /x80 => NonWordByte, /x81 => NonWordByte, /x82 => NonWordByte, /x83 => NonWordByte, /x84 => NonWordByte, /x85 => NonWordByte, /x86 => NonWordByte, /x87 => NonWordByte, /x88 => NonWordByte, /x89 => NonWordByte, /x8A => NonWordByte, /x8B => NonWordByte, /x8C => NonWordByte, /x8D => NonWordByte, /x8E => NonWordByte, /x8F => NonWordByte, /x90 => NonWordByte, /x91 => NonWordByte, /x92 => NonWordByte, /x93 => NonWordByte, /x94 => NonWordByte, /x95 => NonWordByte, /x96 => NonWordByte, /x97 => NonWordByte, /x98 => NonWordByte, /x99 => NonWordByte, /x9A => NonWordByte, /x9B => NonWordByte, /x9C => NonWordByte, /x9D => NonWordByte, /x9E => NonWordByte, /x9F => NonWordByte, /xA0 => NonWordByte, /xA1 => NonWordByte, /xA2 => NonWordByte, /xA3 => NonWordByte, /xA4 => NonWordByte, /xA5 => NonWordByte, /xA6 => NonWordByte, /xA7 => NonWordByte, /xA8 => NonWordByte, /xA9 => NonWordByte, /xAA => NonWordByte, /xAB => NonWordByte, /xAC => NonWordByte, /xAD => NonWordByte, /xAE => NonWordByte, /xAF => NonWordByte, /xB0 => NonWordByte, /xB1 => NonWordByte, /xB2 => NonWordByte, /xB3 => NonWordByte, /xB4 => NonWordByte, /xB5 => NonWordByte, /xB6 => NonWordByte, /xB7 => NonWordByte, /xB8 => NonWordByte, /xB9 => NonWordByte, /xBA => NonWordByte, /xBB => NonWordByte, /xBC => NonWordByte, /xBD => NonWordByte, /xBE => NonWordByte, /xBF => NonWordByte, /xC0 => NonWordByte, /xC1 => NonWordByte, /xC2 => NonWordByte, /xC3 => NonWordByte, /xC4 => NonWordByte, /xC5 => NonWordByte, /xC6 => NonWordByte, /xC7 => NonWordByte, /xC8 => NonWordByte, /xC9 => NonWordByte, /xCA => NonWordByte, /xCB => NonWordByte, /xCC => NonWordByte, /xCD => NonWordByte, /xCE => NonWordByte, /xCF => NonWordByte, /xD0 => NonWordByte, /xD1 => NonWordByte, /xD2 => NonWordByte, /xD3 => NonWordByte, /xD4 => NonWordByte, /xD5 => NonWordByte, /xD6 => NonWordByte, /xD7 => NonWordByte, /xD8 => NonWordByte, /xD9 => NonWordByte, /xDA => NonWordByte, /xDB => NonWordByte, /xDC => NonWordByte, /xDD => NonWordByte, /xDE => NonWordByte, /xDF => NonWordByte, /xE0 => NonWordByte, /xE1 => NonWordByte, /xE2 => NonWordByte, /xE3 => NonWordByte, /xE4 => NonWordByte, /xE5 => NonWordByte, /xE6 => NonWordByte, /xE7 => NonWordByte, /xE8 => NonWordByte, /xE9 => NonWordByte, /xEA => NonWordByte, /xEB => NonWordByte, /xEC => NonWordByte, /xED => NonWordByte, /xEE => NonWordByte, /xEF => NonWordByte, /xF0 => NonWordByte, /xF1 => NonWordByte, /xF2 => NonWordByte, /xF3 => NonWordByte, /xF4 => NonWordByte, /xF5 => NonWordByte, /xF6 => NonWordByte, /xF7 => NonWordByte, /xF8 => NonWordByte, /xF9 => NonWordByte, /xFA => NonWordByte, /xFB => NonWordByte, /xFC => NonWordByte, /xFD => NonWordByte, /xFE => NonWordByte, /xFF => NonWordByte}, - classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]), - quitset: ByteSet { - bits: {}, - }, - cache_capacity: 10485760, - }, - reverse: DFA { - config: Config { - match_kind: Some( - All, - ), - pre: Some( - None, - ), - starts_for_each_pattern: Some( - true, - ), - byte_classes: Some( - true, - ), - unicode_word_boundary: Some( - true, - ), - quitset: None, - specialize_start_states: Some( - false, - ), - cache_capacity: Some( - 10485760, - ), - skip_cache_capacity_check: Some( - false, - ), - minimum_cache_clear_count: Some( - Some( - 3, - ), - ), - minimum_bytes_per_state: Some( - Some( - 10, - ), - ), - }, - nfa: thompson::NFA( - ^000000: FAIL - - transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) - ) - , - stride2: 1, - start_map: StartByteMap{/x00 => NonWordByte, /x01 => NonWordByte, /x02 => NonWordByte, /x03 => NonWordByte, /x04 => NonWordByte, /x05 => NonWordByte, /x06 => NonWordByte, /x07 => NonWordByte, /x08 => NonWordByte, /t => NonWordByte, - => LineLF, /x0B => NonWordByte, /x0C => NonWordByte, /r => LineCR, /x0E => NonWordByte, /x0F => NonWordByte, /x10 => NonWordByte, /x11 => NonWordByte, /x12 => NonWordByte, /x13 => NonWordByte, /x14 => NonWordByte, /x15 => NonWordByte, /x16 => NonWordByte, /x17 => NonWordByte, /x18 => NonWordByte, /x19 => NonWordByte, /x1A => NonWordByte, /x1B => NonWordByte, /x1C => NonWordByte, /x1D => NonWordByte, /x1E => NonWordByte, /x1F => NonWordByte, ' ' => NonWordByte, ! => NonWordByte, /" => NonWordByte, # => NonWordByte, $ => NonWordByte, % => NonWordByte, & => NonWordByte, /' => NonWordByte, ( => NonWordByte, ) => NonWordByte, * => NonWordByte, + => NonWordByte, , => NonWordByte, - => NonWordByte, . => NonWordByte, / => NonWordByte, 0 => WordByte, 1 => WordByte, 2 => WordByte, 3 => WordByte, 4 => WordByte, 5 => WordByte, 6 => WordByte, 7 => WordByte, 8 => WordByte, 9 => WordByte, : => NonWordByte, ; => NonWordByte, < => NonWordByte, = => NonWordByte, > => NonWordByte, ? => NonWordByte, @ => NonWordByte, A => WordByte, B => WordByte, C => WordByte, D => WordByte, E => WordByte, F => WordByte, G => WordByte, H => WordByte, I => WordByte, J => WordByte, K => WordByte, L => WordByte, M => WordByte, N => WordByte, O => WordByte, P => WordByte, Q => WordByte, R => WordByte, S => WordByte, T => WordByte, U => WordByte, V => WordByte, W => WordByte, X => WordByte, Y => WordByte, Z => WordByte, [ => NonWordByte, // => NonWordByte, ] => NonWordByte, ^ => NonWordByte, _ => WordByte, ` => NonWordByte, a => WordByte, b => WordByte, c => WordByte, d => WordByte, e => WordByte, f => WordByte, g => WordByte, h => WordByte, i => WordByte, j => WordByte, k => WordByte, l => WordByte, m => WordByte, n => WordByte, o => WordByte, p => WordByte, q => WordByte, r => WordByte, s => WordByte, t => WordByte, u => WordByte, v => WordByte, w => WordByte, x => WordByte, y => WordByte, z => WordByte, { => NonWordByte, | => NonWordByte, } => NonWordByte, ~ => NonWordByte, /x7F => NonWordByte, /x80 => NonWordByte, /x81 => NonWordByte, /x82 => NonWordByte, /x83 => NonWordByte, /x84 => NonWordByte, /x85 => NonWordByte, /x86 => NonWordByte, /x87 => NonWordByte, /x88 => NonWordByte, /x89 => NonWordByte, /x8A => NonWordByte, /x8B => NonWordByte, /x8C => NonWordByte, /x8D => NonWordByte, /x8E => NonWordByte, /x8F => NonWordByte, /x90 => NonWordByte, /x91 => NonWordByte, /x92 => NonWordByte, /x93 => NonWordByte, /x94 => NonWordByte, /x95 => NonWordByte, /x96 => NonWordByte, /x97 => NonWordByte, /x98 => NonWordByte, /x99 => NonWordByte, /x9A => NonWordByte, /x9B => NonWordByte, /x9C => NonWordByte, /x9D => NonWordByte, /x9E => NonWordByte, /x9F => NonWordByte, /xA0 => NonWordByte, /xA1 => NonWordByte, /xA2 => NonWordByte, /xA3 => NonWordByte, /xA4 => NonWordByte, /xA5 => NonWordByte, /xA6 => NonWordByte, /xA7 => NonWordByte, /xA8 => NonWordByte, /xA9 => NonWordByte, /xAA => NonWordByte, /xAB => NonWordByte, /xAC => NonWordByte, /xAD => NonWordByte, /xAE => NonWordByte, /xAF => NonWordByte, /xB0 => NonWordByte, /xB1 => NonWordByte, /xB2 => NonWordByte, /xB3 => NonWordByte, /xB4 => NonWordByte, /xB5 => NonWordByte, /xB6 => NonWordByte, /xB7 => NonWordByte, /xB8 => NonWordByte, /xB9 => NonWordByte, /xBA => NonWordByte, /xBB => NonWordByte, /xBC => NonWordByte, /xBD => NonWordByte, /xBE => NonWordByte, /xBF => NonWordByte, /xC0 => NonWordByte, /xC1 => NonWordByte, /xC2 => NonWordByte, /xC3 => NonWordByte, /xC4 => NonWordByte, /xC5 => NonWordByte, /xC6 => NonWordByte, /xC7 => NonWordByte, /xC8 => NonWordByte, /xC9 => NonWordByte, /xCA => NonWordByte, /xCB => NonWordByte, /xCC => NonWordByte, /xCD => NonWordByte, /xCE => NonWordByte, /xCF => NonWordByte, /xD0 => NonWordByte, /xD1 => NonWordByte, /xD2 => NonWordByte, /xD3 => NonWordByte, /xD4 => NonWordByte, /xD5 => NonWordByte, /xD6 => NonWordByte, /xD7 => NonWordByte, /xD8 => NonWordByte, /xD9 => NonWordByte, /xDA => NonWordByte, /xDB => NonWordByte, /xDC => NonWordByte, /xDD => NonWordByte, /xDE => NonWordByte, /xDF => NonWordByte, /xE0 => NonWordByte, /xE1 => NonWordByte, /xE2 => NonWordByte, /xE3 => NonWordByte, /xE4 => NonWordByte, /xE5 => NonWordByte, /xE6 => NonWordByte, /xE7 => NonWordByte, /xE8 => NonWordByte, /xE9 => NonWordByte, /xEA => NonWordByte, /xEB => NonWordByte, /xEC => NonWordByte, /xED => NonWordByte, /xEE => NonWordByte, /xEF => NonWordByte, /xF0 => NonWordByte, /xF1 => NonWordByte, /xF2 => NonWordByte, /xF3 => NonWordByte, /xF4 => NonWordByte, /xF5 => NonWordByte, /xF6 => NonWordByte, /xF7 => NonWordByte, /xF8 => NonWordByte, /xF9 => NonWordByte, /xFA => NonWordByte, /xFB => NonWordByte, /xFC => NonWordByte, /xFD => NonWordByte, /xFE => NonWordByte, /xFF => NonWordByte}, - classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]), - quitset: ByteSet { - bits: {}, - }, - cache_capacity: 10485760, - }, - }, - ), - ), - ), - dfa: DFA( - None, - ), - }, - info: RegexInfo( - RegexInfoI { - config: Config { - match_kind: Some( - All, - ), - utf8_empty: Some( - false, - ), - autopre: None, - pre: None, - which_captures: None, - nfa_size_limit: Some( - Some( - 10485760, - ), - ), - onepass_size_limit: None, - hybrid_cache_capacity: Some( - 10485760, - ), - hybrid: None, - dfa: None, - dfa_size_limit: None, - dfa_state_limit: None, - onepass: None, - backtrack: None, - byte_classes: None, - line_terminator: None, - }, - props: [], - props_union: Properties( - PropertiesI { - minimum_len: None, - maximum_len: None, - look_set: ∅, - look_set_prefix: ∅, - look_set_suffix: ∅, - look_set_prefix_any: ∅, - look_set_suffix_any: ∅, - utf8: true, - explicit_captures_len: 0, - static_explicit_captures_len: None, - literal: false, - alternation_literal: true, - }, - ), - }, - ), - }, - pool: Pool( - Pool { - stacks: [ - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - ], - owner: 4, - owner_val: UnsafeCell { .. }, - }, - ), - }, - map: [], - patset: Pool( - Pool { - stacks: [ - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - ], - owner: 0, - owner_val: UnsafeCell { .. }, - }, - ), - }, - ), - ], - }, - poisoned: false, - .. - }, - }, - }, - run_conditions: RunConditions { - never: false, - always: true, - ignore_broken_dep_steps: true, - ignore_missing_outputs: true, - }, - pipeline_rundir: XvcPath( - "", - ), - terminate_timeout_processes: true, - algorithm: Blake3, - command_process: RwLock { - data: CommandProcess { - environment: {}, - step: XvcStep { - name: "average-age", - }, - step_command: XvcStepCommand { - command: "sqlite3 people.db 'SELECT AVG(Age) FROM People;'", - }, - birth: Some( - Instant { - tv_sec: 807955, - tv_nsec: 611297291, - }, - ), - process: Some( - Popen { - stdin: None, - stdout: Some( - File { - fd: 5, - read: true, - write: false, - }, - ), - stderr: Some( - File { - fd: 7, - read: true, - write: false, - }, - ), - child_state: Running { - pid: 29330, - ext: (), - }, - detached: true, - }, - ), - stdout_sender: Sender { .. }, - stderr_sender: Sender { .. }, - stdout_receiver: Receiver { .. }, - stderr_receiver: Receiver { .. }, - }, - poisoned: false, - .. - }, - available_process_slots: RwLock { - data: 3, - poisoned: false, - .. - }, - process_poll_milliseconds: 10, - dependency_diffs: RwLock { - data: HStore { - map: {}, - }, - poisoned: false, - .. - }, - output_diffs: RwLock { - data: HStore { - map: {}, - }, - poisoned: false, - .. - }, - step_e: XvcEntity( - 2, - 16393284406203488121, - ), - step: XvcStep { - name: "average-age", - }, - step_command: XvcStepCommand { - command: "sqlite3 people.db 'SELECT AVG(Age) FROM People;'", - }, - current_states: RwLock { - data: HStore { - map: { - XvcEntity( - 2, - 16393284406203488121, - ): Running( - FromWaitProcess, - ), - }, - }, - poisoned: false, - .. - }, - step_timeout: 10000s, - all_steps: HStore { - map: { - XvcEntity( - 2, - 16393284406203488121, - ): XvcStep { - name: "average-age", - }, - }, - }, - recorded_dependencies: R1NStore { - parents: XvcStore { - map: { - XvcEntity( - 2, - 16393284406203488121, - ): XvcStep { - name: "average-age", - }, - }, - entity_index: { - XvcStep { - name: "average-age", - }: [ - XvcEntity( - 2, - 16393284406203488121, - ), - ], - }, - previous: EventLog( - [ - Add { - entity: XvcEntity( - 2, - 16393284406203488121, - ), - value: XvcStep { - name: "average-age", - }, - }, - Add { - entity: XvcEntity( - 2, - 16393284406203488121, - ), - value: XvcStep { - name: "average-age", - }, - }, - ], - ), - current: EventLog( - [], - ), - }, - children: XvcStore { - map: {}, - entity_index: {}, - previous: EventLog( - [], - ), - current: EventLog( - [], - ), - }, - child_parents: XvcStore { - map: {}, - entity_index: {}, - previous: EventLog( - [], - ), - current: EventLog( - [], - ), - }, - }, - step_dependencies: {}, - step_outputs: HStore { - map: {}, - }, -} -[TRACE][pipeline/src/pipeline/mod.rs::1463] &process: Popen { - stdin: None, - stdout: Some( - File { - fd: 5, - read: true, - write: false, - }, - ), - stderr: Some( - File { - fd: 7, - read: true, - write: false, - }, - ), - child_state: Running { - pid: 29330, - ext: (), - }, - detached: true, -} -[TRACE][pipeline/src/pipeline/mod.rs::1468] process: Popen { - stdin: None, - stdout: Some( - File { - fd: 5, - read: true, - write: false, - }, - ), - stderr: Some( - File { - fd: 7, - read: true, - write: false, - }, - ), - child_state: Running { - pid: 29330, - ext: (), - }, - detached: true, -} -[DEBUG] Step average-age with command sqlite3 people.db 'SELECT AVG(Age) FROM People;' is still running -[TRACE][pipeline/src/pipeline/mod.rs::593] s: Running( - FromWaitProcess, -) -[TRACE][pipeline/src/pipeline/mod.rs::583] select: Select { .. } [WARN] [ERR] [average-age] Error: in prepare, no such table: People -[TRACE][/Users/iex/.cargo/registry/src/index.crates.io-6f17d22bba15001f/notify-6.1.1/src/fsevent.rs::536] FSEvent: path = `[CWD]/people.db`, flag = StreamFlags(ITEM_CREATED | IS_FILE) -[TRACE][walker/src/notify.rs::56] event: Ok( - Event { - kind: Create( - File, - ), - paths: [ - "[CWD]/people.db", - ], - attr:tracker: None, - attr:flag: None, - attr:info: None, - attr:source: None, - }, -) -[TRACE][walker/src/lib.rs::748] is_abs: true -[TRACE][walker/src/lib.rs::752] path_str: "[CWD]/people.db" -[TRACE][walker/src/lib.rs::754] final_slash: false -[TRACE][walker/src/lib.rs::776] path: "/people.db" -[TRACE][core/src/util/pmp.rs::99] index: 0 -[TRACE][core/src/util/pmp.rs::102] fs_event: Ok( - Some( - Create { - path: "[CWD]/people.db", - metadata: Metadata { - file_type: FileType( - FileType { - mode: 33188, - }, - ), - is_dir: false, - is_file: true, - permissions: Permissions( - FilePermissions { - mode: 33188, - }, - ), - modified: Ok( - SystemTime { - tv_sec: 1722024005, - tv_nsec: 932064150, - }, - ), - accessed: Ok( - SystemTime { - tv_sec: 1722024005, - tv_nsec: 932064150, - }, - ), - created: Ok( - SystemTime { - tv_sec: 1722024005, - tv_nsec: 932064150, - }, - ), - .. - }, - }, - ), -) -[TRACE][core/src/types/xvcpath.rs::88] abs_path: "[CWD]/people.db" -[TRACE][core/src/types/xvcpath.rs::89] current_dir: AbsolutePath( - "[CWD]", -) -[TRACE][core/src/types/xvcpath.rs::90] xvc_root.absolute_path(): AbsolutePath( - "[CWD]", -) -[TRACE][core/src/util/pmp.rs::65] "Creating {} with {}": "Creating {} with {}" -[TRACE][core/src/util/pmp.rs::65] xvc_path: XvcPath( - "people.db", -) -[TRACE][core/src/util/pmp.rs::65] xvc_md: XvcMetadata { - file_type: File, - size: Some( - 0, - ), - modified: Some( - SystemTime { - tv_sec: 1722024005, - tv_nsec: 932064150, - }, - ), -} -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][pipeline/src/pipeline/mod.rs::593] s: Running( - FromWaitProcess, -) -[TRACE][pipeline/src/pipeline/mod.rs::583] select: Select { .. } -[TRACE][pipeline/src/pipeline/mod.rs::1463] &process: Popen { - stdin: None, - stdout: Some( - File { - fd: 5, - read: true, - write: false, - }, - ), - stderr: Some( - File { - fd: 7, - read: true, - write: false, - }, - ), - child_state: Running { - pid: 29330, - ext: (), - }, - detached: true, -} -[TRACE][pipeline/src/pipeline/mod.rs::1515] return_state: Some( - Broken( - FromProcessReturnedNonZero, - ), -) [ERROR] Step average-age finished UNSUCCESSFULLY with command sqlite3 people.db 'SELECT AVG(Age) FROM People;' -[TRACE][pipeline/src/pipeline/mod.rs::1521] params: StepStateParams { - xvc_root: XvcRootInner { - absolute_path: AbsolutePath( - "[CWD]", - ), - xvc_dir: AbsolutePath( - "[CWD]/.xvc", - ), - store_dir: AbsolutePath( - "[CWD]/.xvc/store", - ), - config: XvcConfig { - current_dir: XvcConfigOption { - source: Runtime, - option: AbsolutePath( - "[CWD]", - ), - }, - config_maps: [ - XvcConfigMap { - source: Default, - map: { - "file.list.format": String( - "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", - ), - "file.track.force": Boolean( - false, - ), - "pipeline.default": String( - "default", - ), - "git.auto_commit": Boolean( - true, - ), - "file.recheck.method": String( - "copy", - ), - "pipeline.process_pool_size": Integer( - 4, - ), - "git.auto_stage": Boolean( - false, - ), - "git.use_git": Boolean( - true, - ), - "cache.algorithm": String( - "blake3", - ), - "file.track.no_parallel": Boolean( - false, - ), - "file.list.recursive": Boolean( - false, - ), - "file.list.no_summary": Boolean( - false, - ), - "file.track.no_commit": Boolean( - false, - ), - "pipeline.current_pipeline": String( - "default", - ), - "file.carry-in.no_parallel": Boolean( - false, - ), - "core.guid": String( - "603c2d6a0d2e6ea9", - ), - "file.list.show_dot_files": Boolean( - false, - ), - "file.list.sort": String( - "name-desc", - ), - "file.track.text_or_binary": String( - "auto", - ), - "file.carry-in.force": Boolean( - false, - ), - "pipeline.default_params_file": String( - "params.yaml", - ), - "git.command": String( - "git", - ), - "core.verbosity": String( - "error", - ), - }, - }, - XvcConfigMap { - source: Project, - map: { - "file.list.recursive": Boolean( - false, - ), - "pipeline.default": String( - "default", - ), - "pipeline.default_params_file": String( - "params.yaml", - ), - "file.track.no_commit": Boolean( - false, - ), - "git.use_git": Boolean( - true, - ), - "git.auto_stage": Boolean( - false, - ), - "file.track.no_parallel": Boolean( - false, - ), - "file.list.no_summary": Boolean( - false, - ), - "file.list.sort": String( - "name-desc", - ), - "core.guid": String( - "860d1874a44e1d89", - ), - "file.list.show_dot_files": Boolean( - false, - ), - "git.command": String( - "git", - ), - "file.track.text_or_binary": String( - "auto", - ), - "pipeline.process_pool_size": Integer( - 4, - ), - "file.recheck.method": String( - "copy", - ), - "file.list.format": String( - "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", - ), - "file.carry-in.no_parallel": Boolean( - false, - ), - "file.carry-in.force": Boolean( - false, - ), - "pipeline.current_pipeline": String( - "default", - ), - "file.track.force": Boolean( - false, - ), - "core.verbosity": String( - "error", - ), - "git.auto_commit": Boolean( - true, - ), - "cache.algorithm": String( - "blake3", - ), - }, - }, - XvcConfigMap { - source: Local, - map: {}, - }, - XvcConfigMap { - source: Environment, - map: { - "TRYCMD_TESTS": String( - "storage,file,pipeline,core", - ), - }, - }, - XvcConfigMap { - source: CommandLine, - map: { - "core.verbosity": String( - "debug", - ), - "core.quiet": Boolean( - false, - ), - }, - }, - ], - the_config: { - "pipeline.current_pipeline": XvcConfigValue { - source: Project, - value: String( - "default", - ), - }, - "cache.algorithm": XvcConfigValue { - source: Project, - value: String( - "blake3", - ), - }, - "pipeline.default_params_file": XvcConfigValue { - source: Project, - value: String( - "params.yaml", - ), - }, - "git.command": XvcConfigValue { - source: Project, - value: String( - "git", - ), - }, - "file.track.no_commit": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "core.verbosity": XvcConfigValue { - source: CommandLine, - value: String( - "debug", - ), - }, - "file.track.text_or_binary": XvcConfigValue { - source: Project, - value: String( - "auto", - ), - }, - "core.guid": XvcConfigValue { - source: Project, - value: String( - "860d1874a44e1d89", - ), - }, - "file.list.show_dot_files": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "file.recheck.method": XvcConfigValue { - source: Project, - value: String( - "copy", - ), - }, - "file.track.no_parallel": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "file.list.no_summary": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "TRYCMD_TESTS": XvcConfigValue { - source: Environment, - value: String( - "storage,file,pipeline,core", - ), - }, - "file.list.sort": XvcConfigValue { - source: Project, - value: String( - "name-desc", - ), - }, - "core.quiet": XvcConfigValue { - source: CommandLine, - value: Boolean( - false, - ), - }, - "git.auto_stage": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "git.use_git": XvcConfigValue { - source: Project, - value: Boolean( - true, - ), - }, - "file.carry-in.no_parallel": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "pipeline.default": XvcConfigValue { - source: Project, - value: String( - "default", - ), - }, - "file.list.recursive": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "file.track.force": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "file.list.format": XvcConfigValue { - source: Project, - value: String( - "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", - ), - }, - "pipeline.process_pool_size": XvcConfigValue { - source: Project, - value: Integer( - 4, - ), - }, - "file.carry-in.force": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "git.auto_commit": XvcConfigValue { - source: Project, - value: Boolean( - true, - ), - }, - }, - init_params: XvcConfigParams { - default_configuration: " -[core] -# The repository id. Please do not delete or change it. -# This is used to identify the repository and generate paths in storages. -# In the future it may be used to in other ways. -guid = /"603c2d6a0d2e6ea9/" -# Default verbosity level. -# One of /"error/", /"warn/", /"info/" -verbosity = /"error/" - -[git] -# Automate git operations. -# Turning this off leads Xvc to behave as if it's not in a Git repository. -# Not recommended unless you're really not using Git -use_git = true -# Command to run Git process. -# You can set this to an absolute path to specify an executable -# If set to a non-absolute path, the executable will be searched in $PATH. -command = /"git/" - -# Commit changes in .xvc/ directory after commands. -# You can set this to false if you want to commit manually. -auto_commit = true - -# Stage changes in .xvc/ directory without committing. -# auto_commit implies auto_stage. -# If you want to commit manually but don't want to stage after individual Xvc commands, you can set this to true. -auto_stage = false - -[cache] -# The hash algorithm used for the cache. -# It may take blake3, blake2, sha2 or sha3 as values. -# All algorithms are selected to produce 256-bit hashes, so sha2 means SHA2-256, blake2 means BLAKE2s, etc. -# The cache path is produced by prepending algorithm name to the cache. -# Blake3 files are in .xvc/b3/, while sha2 files are in .xvc/s2/ etc. -algorithm = /"blake3/" - -[file] - -[file.track] - -# Don't move file content to cache after xvc file track -no_commit = false -# Force to track files even if they are already tracked. -force = false - -# Xvc calculates file content digest differently for text and binary files. -# This option controls whether to treat files as text or binary. -# It may take auto, text or binary as values. -# Auto check each file individually and treat it as text if it's text. -text_or_binary = /"auto/" - -# Don't use parallelism in track operations. -# Note that some of the operations are implemented in parallel by default, and this option affects some heavier operations. -no_parallel = false - -[file.list] - -# Format for `xvc file list` rows. You can reorder or remove columns. -# The following are the keys for each row: -# - {acd64}: actual content digest. All 64 digits from the workspace file's content. -# - {acd8}: actual content digest. First 8 digits the file content digest. -# - {aft}: actual file type. Whether the entry is a file (F), directory (D), -# symlink (S), hardlink (H) or reflink (R). -# - {asz}: actual size. The size of the workspace file in bytes. It uses MB, -# GB and TB to represent sizes larger than 1MB. -# - {ats}: actual timestamp. The timestamp of the workspace file. -# - {cst}: cache status. One of /"=/", /">/", /"/", /" 0 - F 000001: - >000002: /x00 => 2 - 000003: /x00 => 0 - match kind: Standard - prefilter: false - state length: 4 - pattern length: 0 - shortest pattern length: 18446744073709551615 - longest pattern length: 0 - alphabet length: 1 - stride: 1 - byte classes: ByteClasses(0 => [0-255]) - memory usage: 16 - ) - , - ), - map: [], - longest: 0, - }, - ), - Prefix( - PrefixStrategy { - matcher: AhoCorasick( - dfa::DFA( - D 000000: /x00 => 0 - F 000001: - >000002: /x00 => 2 - 000003: /x00 => 0 - match kind: Standard - prefilter: false - state length: 4 - pattern length: 0 - shortest pattern length: 18446744073709551615 - longest pattern length: 0 - alphabet length: 1 - stride: 1 - byte classes: ByteClasses(0 => [0-255]) - memory usage: 16 - ) - , - ), - map: [], - longest: 0, - }, - ), - RequiredExtension( - RequiredExtensionStrategy( - {}, - ), - ), - Regex( - RegexSetStrategy { - matcher: Regex { - imp: RegexI { - strat: Core { - info: RegexInfo( - RegexInfoI { - config: Config { - match_kind: Some( - All, - ), - utf8_empty: Some( - false, - ), - autopre: None, - pre: None, - which_captures: None, - nfa_size_limit: Some( - Some( - 10485760, - ), - ), - onepass_size_limit: None, - hybrid_cache_capacity: Some( - 10485760, - ), - hybrid: None, - dfa: None, - dfa_size_limit: None, - dfa_state_limit: None, - onepass: None, - backtrack: None, - byte_classes: None, - line_terminator: None, - }, - props: [], - props_union: Properties( - PropertiesI { - minimum_len: None, - maximum_len: None, - look_set: ∅, - look_set_prefix: ∅, - look_set_suffix: ∅, - look_set_prefix_any: ∅, - look_set_suffix_any: ∅, - utf8: true, - explicit_captures_len: 0, - static_explicit_captures_len: None, - literal: false, - alternation_literal: true, - }, - ), - }, - ), - pre: None, - nfa: thompson::NFA( - ^000000: FAIL - - transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) - ) - , - nfarev: Some( - thompson::NFA( - ^000000: FAIL - - transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) - ) - , - ), - pikevm: PikeVM( - PikeVMEngine( - PikeVM { - config: Config { - match_kind: Some( - All, - ), - pre: Some( - None, - ), - }, - nfa: thompson::NFA( - ^000000: FAIL - - transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) - ) - , - }, - ), - ), - backtrack: BoundedBacktracker( - None, - ), - onepass: OnePass( - None, - ), - hybrid: Hybrid( - Some( - HybridEngine( - Regex { - forward: DFA { - config: Config { - match_kind: Some( - All, - ), - pre: Some( - None, - ), - starts_for_each_pattern: Some( - true, - ), - byte_classes: Some( - true, - ), - unicode_word_boundary: Some( - true, - ), - quitset: None, - specialize_start_states: Some( - false, - ), - cache_capacity: Some( - 10485760, - ), - skip_cache_capacity_check: Some( - false, - ), - minimum_cache_clear_count: Some( - Some( - 3, - ), - ), - minimum_bytes_per_state: Some( - Some( - 10, - ), - ), - }, - nfa: thompson::NFA( - ^000000: FAIL - - transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) - ) - , - stride2: 1, - start_map: StartByteMap{/x00 => NonWordByte, /x01 => NonWordByte, /x02 => NonWordByte, /x03 => NonWordByte, /x04 => NonWordByte, /x05 => NonWordByte, /x06 => NonWordByte, /x07 => NonWordByte, /x08 => NonWordByte, /t => NonWordByte, - => LineLF, /x0B => NonWordByte, /x0C => NonWordByte, /r => LineCR, /x0E => NonWordByte, /x0F => NonWordByte, /x10 => NonWordByte, /x11 => NonWordByte, /x12 => NonWordByte, /x13 => NonWordByte, /x14 => NonWordByte, /x15 => NonWordByte, /x16 => NonWordByte, /x17 => NonWordByte, /x18 => NonWordByte, /x19 => NonWordByte, /x1A => NonWordByte, /x1B => NonWordByte, /x1C => NonWordByte, /x1D => NonWordByte, /x1E => NonWordByte, /x1F => NonWordByte, ' ' => NonWordByte, ! => NonWordByte, /" => NonWordByte, # => NonWordByte, $ => NonWordByte, % => NonWordByte, & => NonWordByte, /' => NonWordByte, ( => NonWordByte, ) => NonWordByte, * => NonWordByte, + => NonWordByte, , => NonWordByte, - => NonWordByte, . => NonWordByte, / => NonWordByte, 0 => WordByte, 1 => WordByte, 2 => WordByte, 3 => WordByte, 4 => WordByte, 5 => WordByte, 6 => WordByte, 7 => WordByte, 8 => WordByte, 9 => WordByte, : => NonWordByte, ; => NonWordByte, < => NonWordByte, = => NonWordByte, > => NonWordByte, ? => NonWordByte, @ => NonWordByte, A => WordByte, B => WordByte, C => WordByte, D => WordByte, E => WordByte, F => WordByte, G => WordByte, H => WordByte, I => WordByte, J => WordByte, K => WordByte, L => WordByte, M => WordByte, N => WordByte, O => WordByte, P => WordByte, Q => WordByte, R => WordByte, S => WordByte, T => WordByte, U => WordByte, V => WordByte, W => WordByte, X => WordByte, Y => WordByte, Z => WordByte, [ => NonWordByte, // => NonWordByte, ] => NonWordByte, ^ => NonWordByte, _ => WordByte, ` => NonWordByte, a => WordByte, b => WordByte, c => WordByte, d => WordByte, e => WordByte, f => WordByte, g => WordByte, h => WordByte, i => WordByte, j => WordByte, k => WordByte, l => WordByte, m => WordByte, n => WordByte, o => WordByte, p => WordByte, q => WordByte, r => WordByte, s => WordByte, t => WordByte, u => WordByte, v => WordByte, w => WordByte, x => WordByte, y => WordByte, z => WordByte, { => NonWordByte, | => NonWordByte, } => NonWordByte, ~ => NonWordByte, /x7F => NonWordByte, /x80 => NonWordByte, /x81 => NonWordByte, /x82 => NonWordByte, /x83 => NonWordByte, /x84 => NonWordByte, /x85 => NonWordByte, /x86 => NonWordByte, /x87 => NonWordByte, /x88 => NonWordByte, /x89 => NonWordByte, /x8A => NonWordByte, /x8B => NonWordByte, /x8C => NonWordByte, /x8D => NonWordByte, /x8E => NonWordByte, /x8F => NonWordByte, /x90 => NonWordByte, /x91 => NonWordByte, /x92 => NonWordByte, /x93 => NonWordByte, /x94 => NonWordByte, /x95 => NonWordByte, /x96 => NonWordByte, /x97 => NonWordByte, /x98 => NonWordByte, /x99 => NonWordByte, /x9A => NonWordByte, /x9B => NonWordByte, /x9C => NonWordByte, /x9D => NonWordByte, /x9E => NonWordByte, /x9F => NonWordByte, /xA0 => NonWordByte, /xA1 => NonWordByte, /xA2 => NonWordByte, /xA3 => NonWordByte, /xA4 => NonWordByte, /xA5 => NonWordByte, /xA6 => NonWordByte, /xA7 => NonWordByte, /xA8 => NonWordByte, /xA9 => NonWordByte, /xAA => NonWordByte, /xAB => NonWordByte, /xAC => NonWordByte, /xAD => NonWordByte, /xAE => NonWordByte, /xAF => NonWordByte, /xB0 => NonWordByte, /xB1 => NonWordByte, /xB2 => NonWordByte, /xB3 => NonWordByte, /xB4 => NonWordByte, /xB5 => NonWordByte, /xB6 => NonWordByte, /xB7 => NonWordByte, /xB8 => NonWordByte, /xB9 => NonWordByte, /xBA => NonWordByte, /xBB => NonWordByte, /xBC => NonWordByte, /xBD => NonWordByte, /xBE => NonWordByte, /xBF => NonWordByte, /xC0 => NonWordByte, /xC1 => NonWordByte, /xC2 => NonWordByte, /xC3 => NonWordByte, /xC4 => NonWordByte, /xC5 => NonWordByte, /xC6 => NonWordByte, /xC7 => NonWordByte, /xC8 => NonWordByte, /xC9 => NonWordByte, /xCA => NonWordByte, /xCB => NonWordByte, /xCC => NonWordByte, /xCD => NonWordByte, /xCE => NonWordByte, /xCF => NonWordByte, /xD0 => NonWordByte, /xD1 => NonWordByte, /xD2 => NonWordByte, /xD3 => NonWordByte, /xD4 => NonWordByte, /xD5 => NonWordByte, /xD6 => NonWordByte, /xD7 => NonWordByte, /xD8 => NonWordByte, /xD9 => NonWordByte, /xDA => NonWordByte, /xDB => NonWordByte, /xDC => NonWordByte, /xDD => NonWordByte, /xDE => NonWordByte, /xDF => NonWordByte, /xE0 => NonWordByte, /xE1 => NonWordByte, /xE2 => NonWordByte, /xE3 => NonWordByte, /xE4 => NonWordByte, /xE5 => NonWordByte, /xE6 => NonWordByte, /xE7 => NonWordByte, /xE8 => NonWordByte, /xE9 => NonWordByte, /xEA => NonWordByte, /xEB => NonWordByte, /xEC => NonWordByte, /xED => NonWordByte, /xEE => NonWordByte, /xEF => NonWordByte, /xF0 => NonWordByte, /xF1 => NonWordByte, /xF2 => NonWordByte, /xF3 => NonWordByte, /xF4 => NonWordByte, /xF5 => NonWordByte, /xF6 => NonWordByte, /xF7 => NonWordByte, /xF8 => NonWordByte, /xF9 => NonWordByte, /xFA => NonWordByte, /xFB => NonWordByte, /xFC => NonWordByte, /xFD => NonWordByte, /xFE => NonWordByte, /xFF => NonWordByte}, - classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]), - quitset: ByteSet { - bits: {}, - }, - cache_capacity: 10485760, - }, - reverse: DFA { - config: Config { - match_kind: Some( - All, - ), - pre: Some( - None, - ), - starts_for_each_pattern: Some( - true, - ), - byte_classes: Some( - true, - ), - unicode_word_boundary: Some( - true, - ), - quitset: None, - specialize_start_states: Some( - false, - ), - cache_capacity: Some( - 10485760, - ), - skip_cache_capacity_check: Some( - false, - ), - minimum_cache_clear_count: Some( - Some( - 3, - ), - ), - minimum_bytes_per_state: Some( - Some( - 10, - ), - ), - }, - nfa: thompson::NFA( - ^000000: FAIL - - transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) - ) - , - stride2: 1, - start_map: StartByteMap{/x00 => NonWordByte, /x01 => NonWordByte, /x02 => NonWordByte, /x03 => NonWordByte, /x04 => NonWordByte, /x05 => NonWordByte, /x06 => NonWordByte, /x07 => NonWordByte, /x08 => NonWordByte, /t => NonWordByte, - => LineLF, /x0B => NonWordByte, /x0C => NonWordByte, /r => LineCR, /x0E => NonWordByte, /x0F => NonWordByte, /x10 => NonWordByte, /x11 => NonWordByte, /x12 => NonWordByte, /x13 => NonWordByte, /x14 => NonWordByte, /x15 => NonWordByte, /x16 => NonWordByte, /x17 => NonWordByte, /x18 => NonWordByte, /x19 => NonWordByte, /x1A => NonWordByte, /x1B => NonWordByte, /x1C => NonWordByte, /x1D => NonWordByte, /x1E => NonWordByte, /x1F => NonWordByte, ' ' => NonWordByte, ! => NonWordByte, /" => NonWordByte, # => NonWordByte, $ => NonWordByte, % => NonWordByte, & => NonWordByte, /' => NonWordByte, ( => NonWordByte, ) => NonWordByte, * => NonWordByte, + => NonWordByte, , => NonWordByte, - => NonWordByte, . => NonWordByte, / => NonWordByte, 0 => WordByte, 1 => WordByte, 2 => WordByte, 3 => WordByte, 4 => WordByte, 5 => WordByte, 6 => WordByte, 7 => WordByte, 8 => WordByte, 9 => WordByte, : => NonWordByte, ; => NonWordByte, < => NonWordByte, = => NonWordByte, > => NonWordByte, ? => NonWordByte, @ => NonWordByte, A => WordByte, B => WordByte, C => WordByte, D => WordByte, E => WordByte, F => WordByte, G => WordByte, H => WordByte, I => WordByte, J => WordByte, K => WordByte, L => WordByte, M => WordByte, N => WordByte, O => WordByte, P => WordByte, Q => WordByte, R => WordByte, S => WordByte, T => WordByte, U => WordByte, V => WordByte, W => WordByte, X => WordByte, Y => WordByte, Z => WordByte, [ => NonWordByte, // => NonWordByte, ] => NonWordByte, ^ => NonWordByte, _ => WordByte, ` => NonWordByte, a => WordByte, b => WordByte, c => WordByte, d => WordByte, e => WordByte, f => WordByte, g => WordByte, h => WordByte, i => WordByte, j => WordByte, k => WordByte, l => WordByte, m => WordByte, n => WordByte, o => WordByte, p => WordByte, q => WordByte, r => WordByte, s => WordByte, t => WordByte, u => WordByte, v => WordByte, w => WordByte, x => WordByte, y => WordByte, z => WordByte, { => NonWordByte, | => NonWordByte, } => NonWordByte, ~ => NonWordByte, /x7F => NonWordByte, /x80 => NonWordByte, /x81 => NonWordByte, /x82 => NonWordByte, /x83 => NonWordByte, /x84 => NonWordByte, /x85 => NonWordByte, /x86 => NonWordByte, /x87 => NonWordByte, /x88 => NonWordByte, /x89 => NonWordByte, /x8A => NonWordByte, /x8B => NonWordByte, /x8C => NonWordByte, /x8D => NonWordByte, /x8E => NonWordByte, /x8F => NonWordByte, /x90 => NonWordByte, /x91 => NonWordByte, /x92 => NonWordByte, /x93 => NonWordByte, /x94 => NonWordByte, /x95 => NonWordByte, /x96 => NonWordByte, /x97 => NonWordByte, /x98 => NonWordByte, /x99 => NonWordByte, /x9A => NonWordByte, /x9B => NonWordByte, /x9C => NonWordByte, /x9D => NonWordByte, /x9E => NonWordByte, /x9F => NonWordByte, /xA0 => NonWordByte, /xA1 => NonWordByte, /xA2 => NonWordByte, /xA3 => NonWordByte, /xA4 => NonWordByte, /xA5 => NonWordByte, /xA6 => NonWordByte, /xA7 => NonWordByte, /xA8 => NonWordByte, /xA9 => NonWordByte, /xAA => NonWordByte, /xAB => NonWordByte, /xAC => NonWordByte, /xAD => NonWordByte, /xAE => NonWordByte, /xAF => NonWordByte, /xB0 => NonWordByte, /xB1 => NonWordByte, /xB2 => NonWordByte, /xB3 => NonWordByte, /xB4 => NonWordByte, /xB5 => NonWordByte, /xB6 => NonWordByte, /xB7 => NonWordByte, /xB8 => NonWordByte, /xB9 => NonWordByte, /xBA => NonWordByte, /xBB => NonWordByte, /xBC => NonWordByte, /xBD => NonWordByte, /xBE => NonWordByte, /xBF => NonWordByte, /xC0 => NonWordByte, /xC1 => NonWordByte, /xC2 => NonWordByte, /xC3 => NonWordByte, /xC4 => NonWordByte, /xC5 => NonWordByte, /xC6 => NonWordByte, /xC7 => NonWordByte, /xC8 => NonWordByte, /xC9 => NonWordByte, /xCA => NonWordByte, /xCB => NonWordByte, /xCC => NonWordByte, /xCD => NonWordByte, /xCE => NonWordByte, /xCF => NonWordByte, /xD0 => NonWordByte, /xD1 => NonWordByte, /xD2 => NonWordByte, /xD3 => NonWordByte, /xD4 => NonWordByte, /xD5 => NonWordByte, /xD6 => NonWordByte, /xD7 => NonWordByte, /xD8 => NonWordByte, /xD9 => NonWordByte, /xDA => NonWordByte, /xDB => NonWordByte, /xDC => NonWordByte, /xDD => NonWordByte, /xDE => NonWordByte, /xDF => NonWordByte, /xE0 => NonWordByte, /xE1 => NonWordByte, /xE2 => NonWordByte, /xE3 => NonWordByte, /xE4 => NonWordByte, /xE5 => NonWordByte, /xE6 => NonWordByte, /xE7 => NonWordByte, /xE8 => NonWordByte, /xE9 => NonWordByte, /xEA => NonWordByte, /xEB => NonWordByte, /xEC => NonWordByte, /xED => NonWordByte, /xEE => NonWordByte, /xEF => NonWordByte, /xF0 => NonWordByte, /xF1 => NonWordByte, /xF2 => NonWordByte, /xF3 => NonWordByte, /xF4 => NonWordByte, /xF5 => NonWordByte, /xF6 => NonWordByte, /xF7 => NonWordByte, /xF8 => NonWordByte, /xF9 => NonWordByte, /xFA => NonWordByte, /xFB => NonWordByte, /xFC => NonWordByte, /xFD => NonWordByte, /xFE => NonWordByte, /xFF => NonWordByte}, - classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]), - quitset: ByteSet { - bits: {}, - }, - cache_capacity: 10485760, - }, - }, - ), - ), - ), - dfa: DFA( - None, - ), - }, - info: RegexInfo( - RegexInfoI { - config: Config { - match_kind: Some( - All, - ), - utf8_empty: Some( - false, - ), - autopre: None, - pre: None, - which_captures: None, - nfa_size_limit: Some( - Some( - 10485760, - ), - ), - onepass_size_limit: None, - hybrid_cache_capacity: Some( - 10485760, - ), - hybrid: None, - dfa: None, - dfa_size_limit: None, - dfa_state_limit: None, - onepass: None, - backtrack: None, - byte_classes: None, - line_terminator: None, - }, - props: [], - props_union: Properties( - PropertiesI { - minimum_len: None, - maximum_len: None, - look_set: ∅, - look_set_prefix: ∅, - look_set_suffix: ∅, - look_set_prefix_any: ∅, - look_set_suffix_any: ∅, - utf8: true, - explicit_captures_len: 0, - static_explicit_captures_len: None, - literal: false, - alternation_literal: true, - }, - ), - }, - ), - }, - pool: Pool( - Pool { - stacks: [ - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - ], - owner: 4, - owner_val: UnsafeCell { .. }, - }, - ), - }, - map: [], - patset: Pool( - Pool { - stacks: [ - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - ], - owner: 0, - owner_val: UnsafeCell { .. }, - }, - ), - }, - ), - ], - }, - poisoned: false, - .. - }, - }, - }, - run_conditions: RunConditions { - never: false, - always: true, - ignore_broken_dep_steps: true, - ignore_missing_outputs: true, - }, - pipeline_rundir: XvcPath( - "", - ), - terminate_timeout_processes: true, - algorithm: Blake3, - command_process: RwLock { - data: CommandProcess { - environment: {}, - step: XvcStep { - name: "average-age", - }, - step_command: XvcStepCommand { - command: "sqlite3 people.db 'SELECT AVG(Age) FROM People;'", - }, - birth: Some( - Instant { - tv_sec: 807955, - tv_nsec: 611297291, - }, - ), - process: Some( - Popen { - stdin: None, - stdout: Some( - File { - fd: 5, - read: true, - write: false, - }, - ), - stderr: Some( - File { - fd: 7, - read: true, - write: false, - }, - ), - child_state: Finished( - Exited( - 1, - ), - ), - detached: true, - }, - ), - stdout_sender: Sender { .. }, - stderr_sender: Sender { .. }, - stdout_receiver: Receiver { .. }, - stderr_receiver: Receiver { .. }, - }, - poisoned: false, - .. - }, - available_process_slots: RwLock { - data: , - poisoned: false, - .. - }, - process_poll_milliseconds: 10, - dependency_diffs: RwLock { - data: HStore { - map: {}, - }, - poisoned: false, - .. - }, - output_diffs: RwLock { - data: HStore { - map: {}, - }, - poisoned: false, - .. - }, - step_e: XvcEntity( - 2, - 16393284406203488121, - ), - step: XvcStep { - name: "average-age", - }, - step_command: XvcStepCommand { - command: "sqlite3 people.db 'SELECT AVG(Age) FROM People;'", - }, - current_states: RwLock { - data: HStore { - map: { - XvcEntity( - 2, - 16393284406203488121, - ): Running( - FromWaitProcess, - ), - }, - }, - poisoned: false, - .. - }, - step_timeout: 10000s, - all_steps: HStore { - map: { - XvcEntity( - 2, - 16393284406203488121, - ): XvcStep { - name: "average-age", - }, - }, - }, - recorded_dependencies: R1NStore { - parents: XvcStore { - map: { - XvcEntity( - 2, - 16393284406203488121, - ): XvcStep { - name: "average-age", - }, - }, - entity_index: { - XvcStep { - name: "average-age", - }: [ - XvcEntity( - 2, - 16393284406203488121, - ), - ], - }, - previous: EventLog( - [ - Add { - entity: XvcEntity( - 2, - 16393284406203488121, - ), - value: XvcStep { - name: "average-age", - }, - }, - Add { - entity: XvcEntity( - 2, - 16393284406203488121, - ), - value: XvcStep { - name: "average-age", - }, - }, - ], - ), - current: EventLog( - [], - ), - }, - children: XvcStore { - map: {}, - entity_index: {}, - previous: EventLog( - [], - ), - current: EventLog( - [], - ), - }, - child_parents: XvcStore { - map: {}, - entity_index: {}, - previous: EventLog( - [], - ), - current: EventLog( - [], - ), - }, - }, - step_dependencies: {}, - step_outputs: HStore { - map: {}, - }, -} -[TRACE][pipeline/src/pipeline/mod.rs::772] step.name: "average-age" -[TRACE][pipeline/src/pipeline/mod.rs::773] &r_next_state: Broken( - FromProcessReturnedNonZero, -) -[TRACE][pipeline/src/pipeline/mod.rs::775] &step_state: Broken( - FromProcessReturnedNonZero, -) -[TRACE][pipeline/src/pipeline/mod.rs::662] &step_state: Broken( - FromProcessReturnedNonZero, -) -[TRACE][pipeline/src/pipeline/mod.rs::583] select: Select { .. } -[TRACE][pipeline/src/pipeline/mod.rs::519] "Before state updater": "Before state updater" -[TRACE][pipeline/src/pipeline/mod.rs::593] s: Broken( - FromProcessReturnedNonZero, -) -[TRACE][pipeline/src/pipeline/mod.rs::532] step_states: RwLock { - data: HStore { - map: { - XvcEntity( - 2, - 16393284406203488121, - ): Broken( - FromProcessReturnedNonZero, - ), - }, - }, - poisoned: false, - .. -} -[TRACE][pipeline/src/pipeline/mod.rs::536] done_successfully: Ok( - false, -) -[TRACE][core/src/util/pmp.rs::185] self.background_thread: Mutex { - data: JoinHandle { .. }, - poisoned: false, - .. -} -[TRACE][core/src/util/pmp.rs::190] self.background_thread: Mutex { - data: JoinHandle { .. }, - poisoned: false, - .. -} -[TRACE][lib/src/cli/mod.rs::376] "Before handle_git_automation": "Before handle_git_automation" -[TRACE][core/src/util/pmp.rs::99] index: 1 -[TRACE][lib/src/git.rs::30] args: [ - "-C", - "[CWD]", - "diff", - "--name-only", - "--cached", -] -[DEBUG] Using Git: /opt/homebrew/bin/git -[TRACE][lib/src/git.rs::61] git_diff_staged_out: "" -[TRACE][lib/src/git.rs::30] args: [ - "-C", - "[CWD]", - "add", - "--verbose", - "[CWD]/.xvc", - "*.gitignore", - "*.xvcignore", -] -[TRACE][lib/src/git.rs::179] git_add_output: "" -[DEBUG] No files to commit -[DEBUG] Command completed successfully. ``` From 0939cfa28570905a23aae6bf750a35ae497de034 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Fri, 26 Jul 2024 23:22:50 +0300 Subject: [PATCH 071/257] renamed the file --- ...y-sqlite.md => xvc-pipeline-step-dependency-sqlite-query.md} | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) rename book/src/ref/{xvc-pipeline-step-dependency-sqlite.md => xvc-pipeline-step-dependency-sqlite-query.md} (98%) diff --git a/book/src/ref/xvc-pipeline-step-dependency-sqlite.md b/book/src/ref/xvc-pipeline-step-dependency-sqlite-query.md similarity index 98% rename from book/src/ref/xvc-pipeline-step-dependency-sqlite.md rename to book/src/ref/xvc-pipeline-step-dependency-sqlite-query.md index ae454ca56..f24f37676 100644 --- a/book/src/ref/xvc-pipeline-step-dependency-sqlite.md +++ b/book/src/ref/xvc-pipeline-step-dependency-sqlite-query.md @@ -61,7 +61,7 @@ Let's run the step without a dependency first. $ xvc -vv pipeline run [INFO][pipeline/src/pipeline/mod.rs::347] Pipeline Graph: digraph { - 0 [ label = "(2, 9818981151839479067)" ] + 0 [ label = "(2, 12770542937637433432)" ] } From 55a80bfdc8149bfbf36f0d4cfda965b441a4f306 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Fri, 26 Jul 2024 23:25:30 +0300 Subject: [PATCH 072/257] remove verbosity --- .../src/ref/xvc-pipeline-step-dependency-sqlite-query.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/book/src/ref/xvc-pipeline-step-dependency-sqlite-query.md b/book/src/ref/xvc-pipeline-step-dependency-sqlite-query.md index f24f37676..6ced236ef 100644 --- a/book/src/ref/xvc-pipeline-step-dependency-sqlite-query.md +++ b/book/src/ref/xvc-pipeline-step-dependency-sqlite-query.md @@ -58,18 +58,19 @@ $ xvc pipeline step new --step-name average-age --command "sqlite3 people.db 'SE Let's run the step without a dependency first. ```console -$ xvc -vv pipeline run +$ xvc pipeline run [INFO][pipeline/src/pipeline/mod.rs::347] Pipeline Graph: digraph { - 0 [ label = "(2, 12770542937637433432)" ] + 0 [ label = "(2, 579168168868227462)" ] } [INFO] No dependency steps for step average-age [INFO] [average-age] Dependencies has changed -[WARN] [ERR] [average-age] Error: in prepare, no such table: People +[OUT] [average-age] 34.6666666666667 + +[DONE] average-age (sqlite3 people.db 'SELECT AVG(Age) FROM People;') -[ERROR] Step average-age finished UNSUCCESSFULLY with command sqlite3 people.db 'SELECT AVG(Age) FROM People;' ``` From 369fe32a1149e2dff0a47b00e8c5be76487a6e65 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Fri, 26 Jul 2024 23:29:40 +0300 Subject: [PATCH 073/257] added compare --- book/src/ref/xvc-pipeline-step-dependency-sqlite-query.md | 8 -------- pipeline/src/pipeline/deps/compare.rs | 2 +- 2 files changed, 1 insertion(+), 9 deletions(-) diff --git a/book/src/ref/xvc-pipeline-step-dependency-sqlite-query.md b/book/src/ref/xvc-pipeline-step-dependency-sqlite-query.md index 6ced236ef..6e2f63aad 100644 --- a/book/src/ref/xvc-pipeline-step-dependency-sqlite-query.md +++ b/book/src/ref/xvc-pipeline-step-dependency-sqlite-query.md @@ -59,14 +59,6 @@ Let's run the step without a dependency first. ```console $ xvc pipeline run -[INFO][pipeline/src/pipeline/mod.rs::347] Pipeline Graph: -digraph { - 0 [ label = "(2, 579168168868227462)" ] -} - - -[INFO] No dependency steps for step average-age -[INFO] [average-age] Dependencies has changed [OUT] [average-age] 34.6666666666667 [DONE] average-age (sqlite3 people.db 'SELECT AVG(Age) FROM People;') diff --git a/pipeline/src/pipeline/deps/compare.rs b/pipeline/src/pipeline/deps/compare.rs index 6d0ddb725..428e18351 100644 --- a/pipeline/src/pipeline/deps/compare.rs +++ b/pipeline/src/pipeline/deps/compare.rs @@ -471,7 +471,7 @@ pub fn superficial_compare_dependency( XvcDependency::Glob(dep) => diff_of_dep(superficial_compare_glob(cmp_params, dep)?), XvcDependency::Regex(dep) => diff_of_dep(superficial_compare_regex(cmp_params, dep)?), XvcDependency::Lines(dep) => diff_of_dep(superficial_compare_lines(cmp_params, dep)?), - XvcDependency::SqliteQueryDigest(dep) => todo!(), + XvcDependency::SqliteQueryDigest(dep) => diff_of_dep(superficial_compare_query_digest(cmp_params, dep)?), }; Ok(diff) From 5dae1d7b9e86d9e1635a489d041d152aeb68ed8d Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Fri, 26 Jul 2024 23:33:53 +0300 Subject: [PATCH 074/257] fix superficial --- pipeline/src/pipeline/deps/compare.rs | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/pipeline/src/pipeline/deps/compare.rs b/pipeline/src/pipeline/deps/compare.rs index 428e18351..d8c5bc2e8 100644 --- a/pipeline/src/pipeline/deps/compare.rs +++ b/pipeline/src/pipeline/deps/compare.rs @@ -581,7 +581,9 @@ fn superficial_compare_query_digest( cmp_params: &StepStateParams, record: &SqliteQueryDep, ) -> Result> { - let actual = SqliteQueryDep::new(record.path.clone(), record.query.clone()); + let actual = SqliteQueryDep::new(record.path.clone(), record.query.clone()) + .update_metadata(cmp_params.pmp.get(&record.path)); + Ok(SqliteQueryDep::diff_superficial(record, &actual)) } From 926b8bea6623ee636685fed23632a41bf5fe46e8 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Fri, 26 Jul 2024 23:36:23 +0300 Subject: [PATCH 075/257] bump version --- config/Cargo.toml | 6 +++--- core/Cargo.toml | 12 ++++++------ ecs/Cargo.toml | 4 ++-- file/Cargo.toml | 16 ++++++++-------- lib/Cargo.toml | 20 ++++++++++---------- logging/Cargo.toml | 2 +- pipeline/Cargo.toml | 16 ++++++++-------- storage/Cargo.toml | 14 +++++++------- test_helper/Cargo.toml | 4 ++-- walker/Cargo.toml | 6 +++--- workflow_tests/Cargo.toml | 22 +++++++++++----------- 11 files changed, 61 insertions(+), 61 deletions(-) diff --git a/config/Cargo.toml b/config/Cargo.toml index e09394202..b3342fbc3 100644 --- a/config/Cargo.toml +++ b/config/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-config" -version = "0.6.9-alpha.0" +version = "0.6.9-alpha.1" edition = "2021" description = "Xvc configuration management" authors = ["Emre Şahin "] @@ -20,8 +20,8 @@ debug = true [dependencies] -xvc-logging = { version = "0.6.9-alpha.0", path = "../logging" } -xvc-walker = { version = "0.6.9-alpha.0", path = "../walker" } +xvc-logging = { version = "0.6.9-alpha.1", path = "../logging" } +xvc-walker = { version = "0.6.9-alpha.1", path = "../walker" } ## Cli and config diff --git a/core/Cargo.toml b/core/Cargo.toml index f7da1bd24..18057939e 100644 --- a/core/Cargo.toml +++ b/core/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-core" -version = "0.6.9-alpha.0" +version = "0.6.9-alpha.1" edition = "2021" description = "Xvc core for common elements for all commands" authors = ["Emre Şahin "] @@ -19,10 +19,10 @@ crate-type = ["rlib"] debug = true [dependencies] -xvc-config = { version = "0.6.9-alpha.0", path = "../config" } -xvc-logging = { version = "0.6.9-alpha.0", path = "../logging" } -xvc-ecs = { version = "0.6.9-alpha.0", path = "../ecs" } -xvc-walker = { version = "0.6.9-alpha.0", path = "../walker" } +xvc-config = { version = "0.6.9-alpha.1", path = "../config" } +xvc-logging = { version = "0.6.9-alpha.1", path = "../logging" } +xvc-ecs = { version = "0.6.9-alpha.1", path = "../ecs" } +xvc-walker = { version = "0.6.9-alpha.1", path = "../walker" } ## Cli and config clap = { version = "^4.4", features = ["derive"] } @@ -82,6 +82,6 @@ derive_more = "^0.99" itertools = "^0.13" [dev-dependencies] -xvc-test-helper = { version = "0.6.9-alpha.0", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.9-alpha.1", path = "../test_helper/" } proptest = "^1.4" test-case = "^3.3" diff --git a/ecs/Cargo.toml b/ecs/Cargo.toml index b1b97a565..e3f8d2d66 100644 --- a/ecs/Cargo.toml +++ b/ecs/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-ecs" -version = "0.6.9-alpha.0" +version = "0.6.9-alpha.1" edition = "2021" description = "Entity-Component System for Xvc" authors = ["Emre Şahin "] @@ -19,7 +19,7 @@ crate-type = ["rlib"] debug = true [dependencies] -xvc-logging = { version = "0.6.9-alpha.0", path = "../logging" } +xvc-logging = { version = "0.6.9-alpha.1", path = "../logging" } ## Serialization serde = { version = "^1.0", features = ["derive"] } diff --git a/file/Cargo.toml b/file/Cargo.toml index c0a9c50f4..46fcd66a1 100644 --- a/file/Cargo.toml +++ b/file/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-file" -version = "0.6.9-alpha.0" +version = "0.6.9-alpha.1" edition = "2021" description = "File tracking, versioning, upload and download functions for Xvc" authors = ["Emre Şahin "] @@ -24,12 +24,12 @@ bench = true debug = true [dependencies] -xvc-logging = { version = "0.6.9-alpha.0", path = "../logging" } -xvc-config = { version = "0.6.9-alpha.0", path = "../config" } -xvc-core = { version = "0.6.9-alpha.0", path = "../core" } -xvc-ecs = { version = "0.6.9-alpha.0", path = "../ecs" } -xvc-walker = { version = "0.6.9-alpha.0", path = "../walker" } -xvc-storage = { version = "0.6.9-alpha.0", path = "../storage" } +xvc-logging = { version = "0.6.9-alpha.1", path = "../logging" } +xvc-config = { version = "0.6.9-alpha.1", path = "../config" } +xvc-core = { version = "0.6.9-alpha.1", path = "../core" } +xvc-ecs = { version = "0.6.9-alpha.1", path = "../ecs" } +xvc-walker = { version = "0.6.9-alpha.1", path = "../walker" } +xvc-storage = { version = "0.6.9-alpha.1", path = "../storage" } ## Cli and config @@ -94,5 +94,5 @@ parse-size = "^1.0" [dev-dependencies] -xvc-test-helper = { version = "0.6.9-alpha.0", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.9-alpha.1", path = "../test_helper/" } shellfn = "^0.1" diff --git a/lib/Cargo.toml b/lib/Cargo.toml index 329260f57..99fa9dbbe 100644 --- a/lib/Cargo.toml +++ b/lib/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc" -version = "0.6.9-alpha.0" +version = "0.6.9-alpha.1" edition = "2021" description = "An MLOps tool to manage data files and pipelines on top of Git" authors = ["Emre Şahin "] @@ -23,14 +23,14 @@ path = "src/main.rs" debug = true [dependencies] -xvc-config = { version = "0.6.9-alpha.0", path = "../config" } -xvc-core = { version = "0.6.9-alpha.0", path = "../core" } -xvc-logging = { version = "0.6.9-alpha.0", path = "../logging" } -xvc-ecs = { version = "0.6.9-alpha.0", path = "../ecs" } -xvc-file = { version = "0.6.9-alpha.0", path = "../file" } -xvc-pipeline = { version = "0.6.9-alpha.0", path = "../pipeline" } -xvc-walker = { version = "0.6.9-alpha.0", path = "../walker" } -xvc-storage = { version = "0.6.9-alpha.0", path = "../storage" } +xvc-config = { version = "0.6.9-alpha.1", path = "../config" } +xvc-core = { version = "0.6.9-alpha.1", path = "../core" } +xvc-logging = { version = "0.6.9-alpha.1", path = "../logging" } +xvc-ecs = { version = "0.6.9-alpha.1", path = "../ecs" } +xvc-file = { version = "0.6.9-alpha.1", path = "../file" } +xvc-pipeline = { version = "0.6.9-alpha.1", path = "../pipeline" } +xvc-walker = { version = "0.6.9-alpha.1", path = "../walker" } +xvc-storage = { version = "0.6.9-alpha.1", path = "../storage" } ## Cli and config @@ -113,4 +113,4 @@ jwalk = "^0.8" proptest = "^1.4" shellfn = "^0.1" test-case = "^3.3" -xvc-test-helper = { version = "0.6.9-alpha.0", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.9-alpha.1", path = "../test_helper/" } diff --git a/logging/Cargo.toml b/logging/Cargo.toml index 4c61aead5..7ff7d4deb 100644 --- a/logging/Cargo.toml +++ b/logging/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-logging" -version = "0.6.9-alpha.0" +version = "0.6.9-alpha.1" edition = "2021" description = "Logging crate for Xvc" authors = ["Emre Şahin "] diff --git a/pipeline/Cargo.toml b/pipeline/Cargo.toml index 6e7d37b96..9cbae3c3a 100644 --- a/pipeline/Cargo.toml +++ b/pipeline/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-pipeline" -version = "0.6.9-alpha.0" +version = "0.6.9-alpha.1" edition = "2021" description = "Xvc data pipeline management" authors = ["Emre Şahin "] @@ -19,12 +19,12 @@ debug = true [dependencies] -xvc-config = { version = "0.6.9-alpha.0", path = "../config" } -xvc-core = { version = "0.6.9-alpha.0", path = "../core" } -xvc-ecs = { version = "0.6.9-alpha.0", path = "../ecs" } -xvc-logging = { version = "0.6.9-alpha.0", path = "../logging" } -xvc-walker = { version = "0.6.9-alpha.0", path = "../walker" } -xvc-file = { version = "0.6.9-alpha.0", path = "../file" } +xvc-config = { version = "0.6.9-alpha.1", path = "../config" } +xvc-core = { version = "0.6.9-alpha.1", path = "../core" } +xvc-ecs = { version = "0.6.9-alpha.1", path = "../ecs" } +xvc-logging = { version = "0.6.9-alpha.1", path = "../logging" } +xvc-walker = { version = "0.6.9-alpha.1", path = "../walker" } +xvc-file = { version = "0.6.9-alpha.1", path = "../file" } ## Cli and config clap = { version = "^4.4", features = ["derive"] } @@ -98,5 +98,5 @@ itertools = "^0.13" derive_more = "^0.99" [dev-dependencies] -xvc-test-helper = { version = "0.6.9-alpha.0", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.9-alpha.1", path = "../test_helper/" } test-case = "^3.3" diff --git a/storage/Cargo.toml b/storage/Cargo.toml index 408e47114..52ab7802d 100644 --- a/storage/Cargo.toml +++ b/storage/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-storage" -version = "0.6.9-alpha.0" +version = "0.6.9-alpha.1" edition = "2021" description = "Xvc remote and local storage management" authors = ["Emre Şahin "] @@ -20,11 +20,11 @@ debug = true [dependencies] -xvc-logging = { version = "0.6.9-alpha.0", path = "../logging" } -xvc-config = { version = "0.6.9-alpha.0", path = "../config" } -xvc-core = { version = "0.6.9-alpha.0", path = "../core" } -xvc-ecs = { version = "0.6.9-alpha.0", path = "../ecs" } -xvc-walker = { version = "0.6.9-alpha.0", path = "../walker" } +xvc-logging = { version = "0.6.9-alpha.1", path = "../logging" } +xvc-config = { version = "0.6.9-alpha.1", path = "../config" } +xvc-core = { version = "0.6.9-alpha.1", path = "../core" } +xvc-ecs = { version = "0.6.9-alpha.1", path = "../ecs" } +xvc-walker = { version = "0.6.9-alpha.1", path = "../walker" } ## Cli and config clap = { version = "^4.4", features = ["derive"] } @@ -109,7 +109,7 @@ digital-ocean = ["async"] [dev-dependencies] -xvc-test-helper = { version = "0.6.9-alpha.0", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.9-alpha.1", path = "../test_helper/" } shellfn = "^0.1" [package.metadata.cargo-udeps.ignore] diff --git a/test_helper/Cargo.toml b/test_helper/Cargo.toml index d388d4270..c72722af4 100644 --- a/test_helper/Cargo.toml +++ b/test_helper/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-test-helper" -version = "0.6.9-alpha.0" +version = "0.6.9-alpha.1" edition = "2021" description = "Unit test helper functions for Xvc" authors = ["Emre Şahin "] @@ -23,7 +23,7 @@ debug = true # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html [dependencies] -xvc-logging = { version = "0.6.9-alpha.0", path = "../logging/" } +xvc-logging = { version = "0.6.9-alpha.1", path = "../logging/" } rand = "^0.8" log = "^0.4" diff --git a/walker/Cargo.toml b/walker/Cargo.toml index 82dedf82a..3905d687f 100644 --- a/walker/Cargo.toml +++ b/walker/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-walker" -version = "0.6.9-alpha.0" +version = "0.6.9-alpha.1" edition = "2021" description = "Xvc parallel file system walker with ignore features" authors = ["Emre Şahin "] @@ -19,7 +19,7 @@ crate-type = ["rlib"] debug = true [dependencies] -xvc-logging = { version = "0.6.9-alpha.0", path = "../logging" } +xvc-logging = { version = "0.6.9-alpha.1", path = "../logging" } globset = "^0.4" ## Parallelization @@ -41,7 +41,7 @@ itertools = "^0.13" regex = "^1.10" [dev-dependencies] -xvc-test-helper = { path = "../test_helper/", version = "0.6.9-alpha.0" } +xvc-test-helper = { path = "../test_helper/", version = "0.6.9-alpha.1" } test-case = "^3.3" [package.metadata.cargo-udeps.ignore] diff --git a/workflow_tests/Cargo.toml b/workflow_tests/Cargo.toml index 6933f8b9d..729eba794 100644 --- a/workflow_tests/Cargo.toml +++ b/workflow_tests/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-workflow-tests" -version = "0.6.9-alpha.0" +version = "0.6.9-alpha.1" edition = "2021" description = "Integration test suite for Xvc" authors = ["Emre Şahin "] @@ -23,15 +23,15 @@ debug = true [dependencies] -xvc = { version = "0.6.9-alpha.0", path = "../lib" } -xvc-config = { version = "0.6.9-alpha.0", path = "../config" } -xvc-core = { version = "0.6.9-alpha.0", path = "../core" } -xvc-logging = { version = "0.6.9-alpha.0", path = "../logging" } -xvc-ecs = { version = "0.6.9-alpha.0", path = "../ecs" } -xvc-file = { version = "0.6.9-alpha.0", path = "../file" } -xvc-pipeline = { version = "0.6.9-alpha.0", path = "../pipeline" } -xvc-walker = { version = "0.6.9-alpha.0", path = "../walker" } -xvc-storage = { version = "0.6.9-alpha.0", path = "../storage" } +xvc = { version = "0.6.9-alpha.1", path = "../lib" } +xvc-config = { version = "0.6.9-alpha.1", path = "../config" } +xvc-core = { version = "0.6.9-alpha.1", path = "../core" } +xvc-logging = { version = "0.6.9-alpha.1", path = "../logging" } +xvc-ecs = { version = "0.6.9-alpha.1", path = "../ecs" } +xvc-file = { version = "0.6.9-alpha.1", path = "../file" } +xvc-pipeline = { version = "0.6.9-alpha.1", path = "../pipeline" } +xvc-walker = { version = "0.6.9-alpha.1", path = "../walker" } +xvc-storage = { version = "0.6.9-alpha.1", path = "../storage" } ## packages for testing clap = { version = "^4.4", features = ["derive"] } @@ -70,7 +70,7 @@ proptest = "^1.4" test-case = "^3.3" globset = "^0.4" escargot = "^0.5" -xvc-test-helper = { version = "0.6.9-alpha.0", path = "../test_helper" } +xvc-test-helper = { version = "0.6.9-alpha.1", path = "../test_helper" } shellfn = "^0.1" jwalk = "^0.8" anyhow = "^1.0" From 066b856fb5da9ab6f2cda028f870457ab23d7fda Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Fri, 26 Jul 2024 23:43:12 +0300 Subject: [PATCH 076/257] debug --- ...c-pipeline-step-dependency-sqlite-query.md | 34 +++++++++++++------ 1 file changed, 23 insertions(+), 11 deletions(-) diff --git a/book/src/ref/xvc-pipeline-step-dependency-sqlite-query.md b/book/src/ref/xvc-pipeline-step-dependency-sqlite-query.md index 6e2f63aad..9702f5507 100644 --- a/book/src/ref/xvc-pipeline-step-dependency-sqlite-query.md +++ b/book/src/ref/xvc-pipeline-step-dependency-sqlite-query.md @@ -73,18 +73,22 @@ $ xvc pipeline step dependency --step-name average-age --sqlite-query people.db ``` -```note +```admonition note The dependency query is run everytime the pipeline runs. It's expected to be lightweight to avoid performance issues. ``` So, when the number of people in the table changes, the step will run. Initially it doesn't keep track of the query results, so it will run again. ```console -$ xvc pipeline run -thread '' panicked at pipeline/src/pipeline/deps/compare.rs:474:50: -not yet implemented +$ xvc -vvvv pipeline run +? interrupted +thread '' panicked at pipeline/src/pipeline/mod.rs:1081:32: +SqliteError { source: FromSqlConversionFailure(18446744073709551615, Null, InvalidType) } note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace -[ERROR] Error in step thread: Any { .. } +thread '' panicked at lib/src/cli/mod.rs:251:52: +[PANIC] SqliteError { source: FromSqlConversionFailure(18446744073709551615, Null, InvalidType) }, [pipeline/src/pipeline/mod.rs::1081] +thread '' panicked at pipeline/src/pipeline/mod.rs:514:17: +called `Result::unwrap()` on an `Err` value: "SendError(..)" ``` @@ -92,10 +96,14 @@ But it won't run the step a second time, as the table didn't change. ```console $ xvc pipeline run -thread '' panicked at pipeline/src/pipeline/deps/compare.rs:474:50: -not yet implemented +? interrupted +thread '' panicked at pipeline/src/pipeline/mod.rs:1081:32: +SqliteError { source: FromSqlConversionFailure(18446744073709551615, Null, InvalidType) } note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace -[ERROR] Error in step thread: Any { .. } +thread '' panicked at lib/src/cli/mod.rs:251:52: +[PANIC] SqliteError { source: FromSqlConversionFailure(18446744073709551615, Null, InvalidType) }, [pipeline/src/pipeline/mod.rs::1081] +thread '' panicked at pipeline/src/pipeline/mod.rs:514:17: +called `Result::unwrap()` on an `Err` value: "SendError(..)" ``` @@ -109,10 +117,14 @@ This time, the step will run again as the result from dependency query (`SELECT ```console $ xvc pipeline run -thread '' panicked at pipeline/src/pipeline/deps/compare.rs:474:50: -not yet implemented +? interrupted +thread '' panicked at pipeline/src/pipeline/mod.rs:1081:32: +thread '' panicked at lib/src/cli/mod.rs:251SqliteError { source: FromSqlConversionFailure(18446744073709551615, Null, InvalidType) } note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace -[ERROR] Error in step thread: Any { .. } +:52: +[PANIC] SqliteError { source: FromSqlConversionFailure(18446744073709551615, Null, InvalidType) }, [pipeline/src/pipeline/mod.rs::1081] +thread '' panicked at pipeline/src/pipeline/mod.rs:514:17: +called `Result::unwrap()` on an `Err` value: "SendError(..)" ``` From 4311bb08b2688fe2106fcc1eb8677cc8cd47a2ea Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sun, 28 Jul 2024 15:17:35 +0300 Subject: [PATCH 077/257] debug --- ...c-pipeline-step-dependency-sqlite-query.md | 2432 ++++++++++++++++- pipeline/src/pipeline/deps/sqlite_query.rs | 2 + 2 files changed, 2429 insertions(+), 5 deletions(-) diff --git a/book/src/ref/xvc-pipeline-step-dependency-sqlite-query.md b/book/src/ref/xvc-pipeline-step-dependency-sqlite-query.md index 9702f5507..114186515 100644 --- a/book/src/ref/xvc-pipeline-step-dependency-sqlite-query.md +++ b/book/src/ref/xvc-pipeline-step-dependency-sqlite-query.md @@ -81,14 +81,2436 @@ So, when the number of people in the table changes, the step will run. Initially ```console $ xvc -vvvv pipeline run -? interrupted +? 101 +[DEBUG][logging/src/lib.rs::237] Terminal logger enabled with level: Trace +[TRACE][core/src/types/xvcroot.rs::263] xvc_candidate: "[CWD]/.xvc" +[TRACE][core/src/types/xvcroot.rs::265] parent: "[CWD]" +[DEBUG][config/src/error.rs::72] Config source for level "system" not found at "/Users/iex/Library/Application Support/com.emresult.xvc" +[DEBUG][config/src/error.rs::72] Config source for level "global" not found at "/Users/iex/Library/Application Support/xvc" +[TRACE][ecs/src/ecs/mod.rs::229] dir: "[CWD]/.xvc/ec" +[TRACE][ecs/src/ecs/mod.rs::239] files: [ + "[CWD]/.xvc/ec/1722026597763090", + "[CWD]/.xvc/ec/1722026597766408", + "[CWD]/.xvc/ec/1722026597840236", + "[CWD]/.xvc/ec/1722026598025010", +] +[TRACE][core/src/types/xvcroot.rs::263] xvc_candidate: "[CWD]/.xvc" +[TRACE][core/src/types/xvcroot.rs::265] parent: "[CWD]" +[TRACE][pipeline/src/pipeline/mod.rs::285] pipeline_e: XvcEntity( + 1, + 4733698130976263725, +) +[TRACE][pipeline/src/pipeline/mod.rs::290] pipeline_steps: HStore { + map: { + XvcEntity( + 2, + 17595501963111217905, + ): XvcStep { + name: "average-age", + }, + }, +} +[TRACE][pipeline/src/pipeline/mod.rs::293] consider_changed: XvcStore { + map: { + XvcEntity( + 2, + 17595501963111217905, + ): ByDependencies, + }, + entity_index: { + ByDependencies: [ + XvcEntity( + 2, + 17595501963111217905, + ), + ], + }, + previous: EventLog( + [ + Add { + entity: XvcEntity( + 2, + 17595501963111217905, + ), + value: ByDependencies, + }, + ], + ), + current: EventLog( + [], + ), +} +[TRACE][pipeline/src/pipeline/mod.rs::296] all_deps.parents.len(): 1 +[TRACE][pipeline/src/pipeline/mod.rs::297] all_deps.children.len(): 1 +[TRACE][pipeline/src/pipeline/mod.rs::299] all_outs.parents.len(): 1 +[TRACE][pipeline/src/pipeline/mod.rs::300] all_outs.children.len(): 0 +[DEBUG][/Users/iex/.cargo/registry/src/index.crates.io-6f17d22bba15001f/globset-0.4.14/src/lib.rs::453] built glob set; 0 literals, 2 basenames, 0 extensions, 0 prefixes, 0 suffixes, 0 required extensions, 0 regexes +[TRACE][walker/src/lib.rs::488] ignore_fn: ".xvcignore" +[TRACE][walker/src/lib.rs::490] ignore_root: "[CWD]" +[TRACE][walker/src/lib.rs::498] entry.path(): "[CWD]/.xvc" +[TRACE][walker/src/lib.rs::498] entry.path(): "[CWD]/.git" +[TRACE][walker/src/lib.rs::503] ignore_path: "[CWD]/.xvcignore" +[TRACE][walker/src/lib.rs::597] ignore_root: "[CWD]" +[TRACE][walker/src/lib.rs::598] ignore_path: "[CWD]/.xvcignore" +[TRACE][walker/src/lib.rs::606] &content: " +# Add patterns of files xvc should ignore, which could improve +# the performance. +# It's in the same format as .gitignore files. + +.DS_Store +" +[DEBUG][/Users/iex/.cargo/registry/src/index.crates.io-6f17d22bba15001f/globset-0.4.14/src/lib.rs::453] built glob set; 0 literals, 3 basenames, 0 extensions, 0 prefixes, 0 suffixes, 0 required extensions, 0 regexes +[TRACE][walker/src/lib.rs::748] is_abs: true +[TRACE][walker/src/lib.rs::752] path_str: "[CWD]/.xvc" +[TRACE][walker/src/lib.rs::754] final_slash: false +[TRACE][walker/src/lib.rs::776] path: "/.xvc" +[TRACE][walker/src/lib.rs::748] is_abs: true +[TRACE][walker/src/lib.rs::752] path_str: "[CWD]/.git" +[TRACE][walker/src/lib.rs::754] final_slash: false +[TRACE][walker/src/lib.rs::776] path: "/.git" +[TRACE][core/src/util/pmp.rs::41] ignore_rules: IgnoreRules { + root: "[CWD]", + ignore_patterns: RwLock { + data: [ + Pattern { + pattern: Glob { + glob: "**/.xvc", + re: "(?-u)^(?:/?|.*/)//.xvc$", + opts: GlobOptions { + case_insensitive: false, + literal_separator: false, + backslash_escape: true, + empty_alternates: false, + }, + tokens: Tokens( + [ + RecursivePrefix, + Literal( + '.', + ), + Literal( + 'x', + ), + Literal( + 'v', + ), + Literal( + 'c', + ), + ], + ), + }, + original: ".xvc", + source: Global, + effect: Ignore, + relativity: Anywhere, + path_kind: Any, + }, + Pattern { + pattern: Glob { + glob: "**/.git", + re: "(?-u)^(?:/?|.*/)//.git$", + opts: GlobOptions { + case_insensitive: false, + literal_separator: false, + backslash_escape: true, + empty_alternates: false, + }, + tokens: Tokens( + [ + RecursivePrefix, + Literal( + '.', + ), + Literal( + 'g', + ), + Literal( + 'i', + ), + Literal( + 't', + ), + ], + ), + }, + original: ".git", + source: Global, + effect: Ignore, + relativity: Anywhere, + path_kind: Any, + }, + Pattern { + pattern: Glob { + glob: "**/.DS_Store", + re: "(?-u)^(?:/?|.*/)//.DS_Store$", + opts: GlobOptions { + case_insensitive: false, + literal_separator: false, + backslash_escape: true, + empty_alternates: false, + }, + tokens: Tokens( + [ + RecursivePrefix, + Literal( + '.', + ), + Literal( + 'D', + ), + Literal( + 'S', + ), + Literal( + '_', + ), + Literal( + 'S', + ), + Literal( + 't', + ), + Literal( + 'o', + ), + Literal( + 'r', + ), + Literal( + 'e', + ), + ], + ), + }, + original: ".DS_Store", + source: File { + path: ".xvcignore", + line: 6, + }, + effect: Ignore, + relativity: Anywhere, + path_kind: Any, + }, + ], + poisoned: false, + .. + }, + whitelist_patterns: RwLock { + data: [], + poisoned: false, + .. + }, + whitelist_set: RwLock { + data: GlobSet { + len: 0, + strats: [], + }, + poisoned: false, + .. + }, + ignore_set: RwLock { + data: GlobSet { + len: 3, + strats: [ + Extension( + ExtensionStrategy( + {}, + ), + ), + BasenameLiteral( + BasenameLiteralStrategy( + { + [ + 46, + 120, + 118, + 99, + ]: [ + 0, + ], + [ + 46, + 68, + 83, + 95, + 83, + 116, + 111, + 114, + 101, + ]: [ + 2, + ], + [ + 46, + 103, + 105, + 116, + ]: [ + 1, + ], + }, + ), + ), + Literal( + LiteralStrategy( + {}, + ), + ), + Suffix( + SuffixStrategy { + matcher: AhoCorasick( + dfa::DFA( + D 000000: /x00 => 0 + F 000001: + >000002: /x00 => 2 + 000003: /x00 => 0 + match kind: Standard + prefilter: false + state length: 4 + pattern length: 0 + shortest pattern length: 18446744073709551615 + longest pattern length: 0 + alphabet length: 1 + stride: 1 + byte classes: ByteClasses(0 => [0-255]) + memory usage: 16 + ) + , + ), + map: [], + longest: 0, + }, + ), + Prefix( + PrefixStrategy { + matcher: AhoCorasick( + dfa::DFA( + D 000000: /x00 => 0 + F 000001: + >000002: /x00 => 2 + 000003: /x00 => 0 + match kind: Standard + prefilter: false + state length: 4 + pattern length: 0 + shortest pattern length: 18446744073709551615 + longest pattern length: 0 + alphabet length: 1 + stride: 1 + byte classes: ByteClasses(0 => [0-255]) + memory usage: 16 + ) + , + ), + map: [], + longest: 0, + }, + ), + RequiredExtension( + RequiredExtensionStrategy( + {}, + ), + ), + Regex( + RegexSetStrategy { + matcher: Regex { + imp: RegexI { + strat: Core { + info: RegexInfo( + RegexInfoI { + config: Config { + match_kind: Some( + All, + ), + utf8_empty: Some( + false, + ), + autopre: None, + pre: None, + which_captures: None, + nfa_size_limit: Some( + Some( + 10485760, + ), + ), + onepass_size_limit: None, + hybrid_cache_capacity: Some( + 10485760, + ), + hybrid: None, + dfa: None, + dfa_size_limit: None, + dfa_state_limit: None, + onepass: None, + backtrack: None, + byte_classes: None, + line_terminator: None, + }, + props: [], + props_union: Properties( + PropertiesI { + minimum_len: None, + maximum_len: None, + look_set: ∅, + look_set_prefix: ∅, + look_set_suffix: ∅, + look_set_prefix_any: ∅, + look_set_suffix_any: ∅, + utf8: true, + explicit_captures_len: 0, + static_explicit_captures_len: None, + literal: false, + alternation_literal: true, + }, + ), + }, + ), + pre: None, + nfa: thompson::NFA( + ^000000: FAIL + + transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) + ) + , + nfarev: Some( + thompson::NFA( + ^000000: FAIL + + transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) + ) + , + ), + pikevm: PikeVM( + PikeVMEngine( + PikeVM { + config: Config { + match_kind: Some( + All, + ), + pre: Some( + None, + ), + }, + nfa: thompson::NFA( + ^000000: FAIL + + transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) + ) + , + }, + ), + ), + backtrack: BoundedBacktracker( + None, + ), + onepass: OnePass( + None, + ), + hybrid: Hybrid( + Some( + HybridEngine( + Regex { + forward: DFA { + config: Config { + match_kind: Some( + All, + ), + pre: Some( + None, + ), + starts_for_each_pattern: Some( + true, + ), + byte_classes: Some( + true, + ), + unicode_word_boundary: Some( + true, + ), + quitset: None, + specialize_start_states: Some( + false, + ), + cache_capacity: Some( + 10485760, + ), + skip_cache_capacity_check: Some( + false, + ), + minimum_cache_clear_count: Some( + Some( + 3, + ), + ), + minimum_bytes_per_state: Some( + Some( + 10, + ), + ), + }, + nfa: thompson::NFA( + ^000000: FAIL + + transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) + ) + , + stride2: 1, + start_map: StartByteMap{/x00 => NonWordByte, /x01 => NonWordByte, /x02 => NonWordByte, /x03 => NonWordByte, /x04 => NonWordByte, /x05 => NonWordByte, /x06 => NonWordByte, /x07 => NonWordByte, /x08 => NonWordByte, /t => NonWordByte, + => LineLF, /x0B => NonWordByte, /x0C => NonWordByte, /r => LineCR, /x0E => NonWordByte, /x0F => NonWordByte, /x10 => NonWordByte, /x11 => NonWordByte, /x12 => NonWordByte, /x13 => NonWordByte, /x14 => NonWordByte, /x15 => NonWordByte, /x16 => NonWordByte, /x17 => NonWordByte, /x18 => NonWordByte, /x19 => NonWordByte, /x1A => NonWordByte, /x1B => NonWordByte, /x1C => NonWordByte, /x1D => NonWordByte, /x1E => NonWordByte, /x1F => NonWordByte, ' ' => NonWordByte, ! => NonWordByte, /" => NonWordByte, # => NonWordByte, $ => NonWordByte, % => NonWordByte, & => NonWordByte, /' => NonWordByte, ( => NonWordByte, ) => NonWordByte, * => NonWordByte, + => NonWordByte, , => NonWordByte, - => NonWordByte, . => NonWordByte, / => NonWordByte, 0 => WordByte, 1 => WordByte, 2 => WordByte, 3 => WordByte, 4 => WordByte, 5 => WordByte, 6 => WordByte, 7 => WordByte, 8 => WordByte, 9 => WordByte, : => NonWordByte, ; => NonWordByte, < => NonWordByte, = => NonWordByte, > => NonWordByte, ? => NonWordByte, @ => NonWordByte, A => WordByte, B => WordByte, C => WordByte, D => WordByte, E => WordByte, F => WordByte, G => WordByte, H => WordByte, I => WordByte, J => WordByte, K => WordByte, L => WordByte, M => WordByte, N => WordByte, O => WordByte, P => WordByte, Q => WordByte, R => WordByte, S => WordByte, T => WordByte, U => WordByte, V => WordByte, W => WordByte, X => WordByte, Y => WordByte, Z => WordByte, [ => NonWordByte, // => NonWordByte, ] => NonWordByte, ^ => NonWordByte, _ => WordByte, ` => NonWordByte, a => WordByte, b => WordByte, c => WordByte, d => WordByte, e => WordByte, f => WordByte, g => WordByte, h => WordByte, i => WordByte, j => WordByte, k => WordByte, l => WordByte, m => WordByte, n => WordByte, o => WordByte, p => WordByte, q => WordByte, r => WordByte, s => WordByte, t => WordByte, u => WordByte, v => WordByte, w => WordByte, x => WordByte, y => WordByte, z => WordByte, { => NonWordByte, | => NonWordByte, } => NonWordByte, ~ => NonWordByte, /x7F => NonWordByte, /x80 => NonWordByte, /x81 => NonWordByte, /x82 => NonWordByte, /x83 => NonWordByte, /x84 => NonWordByte, /x85 => NonWordByte, /x86 => NonWordByte, /x87 => NonWordByte, /x88 => NonWordByte, /x89 => NonWordByte, /x8A => NonWordByte, /x8B => NonWordByte, /x8C => NonWordByte, /x8D => NonWordByte, /x8E => NonWordByte, /x8F => NonWordByte, /x90 => NonWordByte, /x91 => NonWordByte, /x92 => NonWordByte, /x93 => NonWordByte, /x94 => NonWordByte, /x95 => NonWordByte, /x96 => NonWordByte, /x97 => NonWordByte, /x98 => NonWordByte, /x99 => NonWordByte, /x9A => NonWordByte, /x9B => NonWordByte, /x9C => NonWordByte, /x9D => NonWordByte, /x9E => NonWordByte, /x9F => NonWordByte, /xA0 => NonWordByte, /xA1 => NonWordByte, /xA2 => NonWordByte, /xA3 => NonWordByte, /xA4 => NonWordByte, /xA5 => NonWordByte, /xA6 => NonWordByte, /xA7 => NonWordByte, /xA8 => NonWordByte, /xA9 => NonWordByte, /xAA => NonWordByte, /xAB => NonWordByte, /xAC => NonWordByte, /xAD => NonWordByte, /xAE => NonWordByte, /xAF => NonWordByte, /xB0 => NonWordByte, /xB1 => NonWordByte, /xB2 => NonWordByte, /xB3 => NonWordByte, /xB4 => NonWordByte, /xB5 => NonWordByte, /xB6 => NonWordByte, /xB7 => NonWordByte, /xB8 => NonWordByte, /xB9 => NonWordByte, /xBA => NonWordByte, /xBB => NonWordByte, /xBC => NonWordByte, /xBD => NonWordByte, /xBE => NonWordByte, /xBF => NonWordByte, /xC0 => NonWordByte, /xC1 => NonWordByte, /xC2 => NonWordByte, /xC3 => NonWordByte, /xC4 => NonWordByte, /xC5 => NonWordByte, /xC6 => NonWordByte, /xC7 => NonWordByte, /xC8 => NonWordByte, /xC9 => NonWordByte, /xCA => NonWordByte, /xCB => NonWordByte, /xCC => NonWordByte, /xCD => NonWordByte, /xCE => NonWordByte, /xCF => NonWordByte, /xD0 => NonWordByte, /xD1 => NonWordByte, /xD2 => NonWordByte, /xD3 => NonWordByte, /xD4 => NonWordByte, /xD5 => NonWordByte, /xD6 => NonWordByte, /xD7 => NonWordByte, /xD8 => NonWordByte, /xD9 => NonWordByte, /xDA => NonWordByte, /xDB => NonWordByte, /xDC => NonWordByte, /xDD => NonWordByte, /xDE => NonWordByte, /xDF => NonWordByte, /xE0 => NonWordByte, /xE1 => NonWordByte, /xE2 => NonWordByte, /xE3 => NonWordByte, /xE4 => NonWordByte, /xE5 => NonWordByte, /xE6 => NonWordByte, /xE7 => NonWordByte, /xE8 => NonWordByte, /xE9 => NonWordByte, /xEA => NonWordByte, /xEB => NonWordByte, /xEC => NonWordByte, /xED => NonWordByte, /xEE => NonWordByte, /xEF => NonWordByte, /xF0 => NonWordByte, /xF1 => NonWordByte, /xF2 => NonWordByte, /xF3 => NonWordByte, /xF4 => NonWordByte, /xF5 => NonWordByte, /xF6 => NonWordByte, /xF7 => NonWordByte, /xF8 => NonWordByte, /xF9 => NonWordByte, /xFA => NonWordByte, /xFB => NonWordByte, /xFC => NonWordByte, /xFD => NonWordByte, /xFE => NonWordByte, /xFF => NonWordByte}, + classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]), + quitset: ByteSet { + bits: {}, + }, + cache_capacity: 10485760, + }, + reverse: DFA { + config: Config { + match_kind: Some( + All, + ), + pre: Some( + None, + ), + starts_for_each_pattern: Some( + true, + ), + byte_classes: Some( + true, + ), + unicode_word_boundary: Some( + true, + ), + quitset: None, + specialize_start_states: Some( + false, + ), + cache_capacity: Some( + 10485760, + ), + skip_cache_capacity_check: Some( + false, + ), + minimum_cache_clear_count: Some( + Some( + 3, + ), + ), + minimum_bytes_per_state: Some( + Some( + 10, + ), + ), + }, + nfa: thompson::NFA( + ^000000: FAIL + + transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) + ) + , + stride2: 1, + start_map: StartByteMap{/x00 => NonWordByte, /x01 => NonWordByte, /x02 => NonWordByte, /x03 => NonWordByte, /x04 => NonWordByte, /x05 => NonWordByte, /x06 => NonWordByte, /x07 => NonWordByte, /x08 => NonWordByte, /t => NonWordByte, + => LineLF, /x0B => NonWordByte, /x0C => NonWordByte, /r => LineCR, /x0E => NonWordByte, /x0F => NonWordByte, /x10 => NonWordByte, /x11 => NonWordByte, /x12 => NonWordByte, /x13 => NonWordByte, /x14 => NonWordByte, /x15 => NonWordByte, /x16 => NonWordByte, /x17 => NonWordByte, /x18 => NonWordByte, /x19 => NonWordByte, /x1A => NonWordByte, /x1B => NonWordByte, /x1C => NonWordByte, /x1D => NonWordByte, /x1E => NonWordByte, /x1F => NonWordByte, ' ' => NonWordByte, ! => NonWordByte, /" => NonWordByte, # => NonWordByte, $ => NonWordByte, % => NonWordByte, & => NonWordByte, /' => NonWordByte, ( => NonWordByte, ) => NonWordByte, * => NonWordByte, + => NonWordByte, , => NonWordByte, - => NonWordByte, . => NonWordByte, / => NonWordByte, 0 => WordByte, 1 => WordByte, 2 => WordByte, 3 => WordByte, 4 => WordByte, 5 => WordByte, 6 => WordByte, 7 => WordByte, 8 => WordByte, 9 => WordByte, : => NonWordByte, ; => NonWordByte, < => NonWordByte, = => NonWordByte, > => NonWordByte, ? => NonWordByte, @ => NonWordByte, A => WordByte, B => WordByte, C => WordByte, D => WordByte, E => WordByte, F => WordByte, G => WordByte, H => WordByte, I => WordByte, J => WordByte, K => WordByte, L => WordByte, M => WordByte, N => WordByte, O => WordByte, P => WordByte, Q => WordByte, R => WordByte, S => WordByte, T => WordByte, U => WordByte, V => WordByte, W => WordByte, X => WordByte, Y => WordByte, Z => WordByte, [ => NonWordByte, // => NonWordByte, ] => NonWordByte, ^ => NonWordByte, _ => WordByte, ` => NonWordByte, a => WordByte, b => WordByte, c => WordByte, d => WordByte, e => WordByte, f => WordByte, g => WordByte, h => WordByte, i => WordByte, j => WordByte, k => WordByte, l => WordByte, m => WordByte, n => WordByte, o => WordByte, p => WordByte, q => WordByte, r => WordByte, s => WordByte, t => WordByte, u => WordByte, v => WordByte, w => WordByte, x => WordByte, y => WordByte, z => WordByte, { => NonWordByte, | => NonWordByte, } => NonWordByte, ~ => NonWordByte, /x7F => NonWordByte, /x80 => NonWordByte, /x81 => NonWordByte, /x82 => NonWordByte, /x83 => NonWordByte, /x84 => NonWordByte, /x85 => NonWordByte, /x86 => NonWordByte, /x87 => NonWordByte, /x88 => NonWordByte, /x89 => NonWordByte, /x8A => NonWordByte, /x8B => NonWordByte, /x8C => NonWordByte, /x8D => NonWordByte, /x8E => NonWordByte, /x8F => NonWordByte, /x90 => NonWordByte, /x91 => NonWordByte, /x92 => NonWordByte, /x93 => NonWordByte, /x94 => NonWordByte, /x95 => NonWordByte, /x96 => NonWordByte, /x97 => NonWordByte, /x98 => NonWordByte, /x99 => NonWordByte, /x9A => NonWordByte, /x9B => NonWordByte, /x9C => NonWordByte, /x9D => NonWordByte, /x9E => NonWordByte, /x9F => NonWordByte, /xA0 => NonWordByte, /xA1 => NonWordByte, /xA2 => NonWordByte, /xA3 => NonWordByte, /xA4 => NonWordByte, /xA5 => NonWordByte, /xA6 => NonWordByte, /xA7 => NonWordByte, /xA8 => NonWordByte, /xA9 => NonWordByte, /xAA => NonWordByte, /xAB => NonWordByte, /xAC => NonWordByte, /xAD => NonWordByte, /xAE => NonWordByte, /xAF => NonWordByte, /xB0 => NonWordByte, /xB1 => NonWordByte, /xB2 => NonWordByte, /xB3 => NonWordByte, /xB4 => NonWordByte, /xB5 => NonWordByte, /xB6 => NonWordByte, /xB7 => NonWordByte, /xB8 => NonWordByte, /xB9 => NonWordByte, /xBA => NonWordByte, /xBB => NonWordByte, /xBC => NonWordByte, /xBD => NonWordByte, /xBE => NonWordByte, /xBF => NonWordByte, /xC0 => NonWordByte, /xC1 => NonWordByte, /xC2 => NonWordByte, /xC3 => NonWordByte, /xC4 => NonWordByte, /xC5 => NonWordByte, /xC6 => NonWordByte, /xC7 => NonWordByte, /xC8 => NonWordByte, /xC9 => NonWordByte, /xCA => NonWordByte, /xCB => NonWordByte, /xCC => NonWordByte, /xCD => NonWordByte, /xCE => NonWordByte, /xCF => NonWordByte, /xD0 => NonWordByte, /xD1 => NonWordByte, /xD2 => NonWordByte, /xD3 => NonWordByte, /xD4 => NonWordByte, /xD5 => NonWordByte, /xD6 => NonWordByte, /xD7 => NonWordByte, /xD8 => NonWordByte, /xD9 => NonWordByte, /xDA => NonWordByte, /xDB => NonWordByte, /xDC => NonWordByte, /xDD => NonWordByte, /xDE => NonWordByte, /xDF => NonWordByte, /xE0 => NonWordByte, /xE1 => NonWordByte, /xE2 => NonWordByte, /xE3 => NonWordByte, /xE4 => NonWordByte, /xE5 => NonWordByte, /xE6 => NonWordByte, /xE7 => NonWordByte, /xE8 => NonWordByte, /xE9 => NonWordByte, /xEA => NonWordByte, /xEB => NonWordByte, /xEC => NonWordByte, /xED => NonWordByte, /xEE => NonWordByte, /xEF => NonWordByte, /xF0 => NonWordByte, /xF1 => NonWordByte, /xF2 => NonWordByte, /xF3 => NonWordByte, /xF4 => NonWordByte, /xF5 => NonWordByte, /xF6 => NonWordByte, /xF7 => NonWordByte, /xF8 => NonWordByte, /xF9 => NonWordByte, /xFA => NonWordByte, /xFB => NonWordByte, /xFC => NonWordByte, /xFD => NonWordByte, /xFE => NonWordByte, /xFF => NonWordByte}, + classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]), + quitset: ByteSet { + bits: {}, + }, + cache_capacity: 10485760, + }, + }, + ), + ), + ), + dfa: DFA( + None, + ), + }, + info: RegexInfo( + RegexInfoI { + config: Config { + match_kind: Some( + All, + ), + utf8_empty: Some( + false, + ), + autopre: None, + pre: None, + which_captures: None, + nfa_size_limit: Some( + Some( + 10485760, + ), + ), + onepass_size_limit: None, + hybrid_cache_capacity: Some( + 10485760, + ), + hybrid: None, + dfa: None, + dfa_size_limit: None, + dfa_state_limit: None, + onepass: None, + backtrack: None, + byte_classes: None, + line_terminator: None, + }, + props: [], + props_union: Properties( + PropertiesI { + minimum_len: None, + maximum_len: None, + look_set: ∅, + look_set_prefix: ∅, + look_set_suffix: ∅, + look_set_prefix_any: ∅, + look_set_suffix_any: ∅, + utf8: true, + explicit_captures_len: 0, + static_explicit_captures_len: None, + literal: false, + alternation_literal: true, + }, + ), + }, + ), + }, + pool: Pool( + Pool { + stacks: [ + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + ], + owner: 0, + owner_val: UnsafeCell { .. }, + }, + ), + }, + map: [], + patset: Pool( + Pool { + stacks: [ + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + ], + owner: 0, + owner_val: UnsafeCell { .. }, + }, + ), + }, + ), + ], + }, + poisoned: false, + .. + }, +} +[TRACE][walker/src/notify.rs::170] watcher: FsEventWatcher { + paths: 0x0000600000d70120, + since_when: 18446744073709551615, + latency: 0.0, + flags: 18, + event_handler: 0x0000600002870010, + runloop: Some( + ( + 0x00006000036740c0, + JoinHandle { .. }, + ), + ), + recursive_info: { + "[CWD]": true, + }, +} +[TRACE][core/src/util/pmp.rs::131] background_thread: Mutex { + data: JoinHandle { .. }, + poisoned: false, + .. +} +[TRACE][core/src/util/pmp.rs::59] watcher: FsEventWatcher { + paths: 0x0000600000d70120, + since_when: 18446744073709551615, + latency: 0.0, + flags: 18, + event_handler: 0x0000600002870010, + runloop: Some( + ( + 0x00006000036740c0, + JoinHandle { .. }, + ), + ), + recursive_info: { + "[CWD]": true, + }, +} +[TRACE][core/src/util/pmp.rs::91] fs_event_index: 0 +[TRACE][core/src/util/pmp.rs::93] kill_signal_index: 1 +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][pipeline/src/pipeline/mod.rs::302] &pmp: XvcPathMetadataProvider { + xvc_root: XvcRootInner { + absolute_path: AbsolutePath( + "[CWD]", + ), + xvc_dir: AbsolutePath( + "[CWD]/.xvc", + ), + store_dir: AbsolutePath( + "[CWD]/.xvc/store", + ), + config: XvcConfig { + current_dir: XvcConfigOption { + source: Runtime, + option: AbsolutePath( + "[CWD]", + ), + }, + config_maps: [ + XvcConfigMap { + source: Default, + map: { + "pipeline.default": String( + "default", + ), + "git.command": String( + "git", + ), + "file.track.text_or_binary": String( + "auto", + ), + "core.verbosity": String( + "error", + ), + "core.guid": String( + "2563bb26e79eff77", + ), + "file.list.show_dot_files": Boolean( + false, + ), + "file.track.no_parallel": Boolean( + false, + ), + "file.list.no_summary": Boolean( + false, + ), + "git.auto_commit": Boolean( + true, + ), + "pipeline.current_pipeline": String( + "default", + ), + "file.recheck.method": String( + "copy", + ), + "pipeline.process_pool_size": Integer( + 4, + ), + "file.carry-in.no_parallel": Boolean( + false, + ), + "file.carry-in.force": Boolean( + false, + ), + "file.track.force": Boolean( + false, + ), + "pipeline.default_params_file": String( + "params.yaml", + ), + "git.auto_stage": Boolean( + false, + ), + "file.list.recursive": Boolean( + false, + ), + "file.list.format": String( + "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", + ), + "file.track.no_commit": Boolean( + false, + ), + "cache.algorithm": String( + "blake3", + ), + "git.use_git": Boolean( + true, + ), + "file.list.sort": String( + "name-desc", + ), + }, + }, + XvcConfigMap { + source: Project, + map: { + "core.verbosity": String( + "error", + ), + "git.auto_commit": Boolean( + true, + ), + "git.use_git": Boolean( + true, + ), + "file.track.no_parallel": Boolean( + false, + ), + "file.recheck.method": String( + "copy", + ), + "pipeline.default_params_file": String( + "params.yaml", + ), + "git.auto_stage": Boolean( + false, + ), + "file.track.no_commit": Boolean( + false, + ), + "file.list.recursive": Boolean( + false, + ), + "pipeline.default": String( + "default", + ), + "file.list.format": String( + "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", + ), + "file.list.show_dot_files": Boolean( + false, + ), + "file.track.text_or_binary": String( + "auto", + ), + "cache.algorithm": String( + "blake3", + ), + "file.list.sort": String( + "name-desc", + ), + "core.guid": String( + "fff3d263419bb95f", + ), + "pipeline.current_pipeline": String( + "default", + ), + "file.carry-in.no_parallel": Boolean( + false, + ), + "file.carry-in.force": Boolean( + false, + ), + "git.command": String( + "git", + ), + "file.list.no_summary": Boolean( + false, + ), + "pipeline.process_pool_size": Integer( + 4, + ), + "file.track.force": Boolean( + false, + ), + }, + }, + XvcConfigMap { + source: Local, + map: {}, + }, + XvcConfigMap { + source: Environment, + map: { + "TRYCMD_TESTS": String( + "storage,file,pipeline,core", + ), + }, + }, + XvcConfigMap { + source: CommandLine, + map: { + "core.verbosity": String( + "debug", + ), + "core.quiet": Boolean( + false, + ), + }, + }, + ], + the_config: { + "pipeline.current_pipeline": XvcConfigValue { + source: Project, + value: String( + "default", + ), + }, + "file.track.force": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "cache.algorithm": XvcConfigValue { + source: Project, + value: String( + "blake3", + ), + }, + "git.use_git": XvcConfigValue { + source: Project, + value: Boolean( + true, + ), + }, + "file.recheck.method": XvcConfigValue { + source: Project, + value: String( + "copy", + ), + }, + "file.track.text_or_binary": XvcConfigValue { + source: Project, + value: String( + "auto", + ), + }, + "core.guid": XvcConfigValue { + source: Project, + value: String( + "fff3d263419bb95f", + ), + }, + "pipeline.default_params_file": XvcConfigValue { + source: Project, + value: String( + "params.yaml", + ), + }, + "file.carry-in.force": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "file.list.format": XvcConfigValue { + source: Project, + value: String( + "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", + ), + }, + "git.auto_stage": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "git.command": XvcConfigValue { + source: Project, + value: String( + "git", + ), + }, + "file.list.recursive": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "core.quiet": XvcConfigValue { + source: CommandLine, + value: Boolean( + false, + ), + }, + "file.carry-in.no_parallel": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "TRYCMD_TESTS": XvcConfigValue { + source: Environment, + value: String( + "storage,file,pipeline,core", + ), + }, + "git.auto_commit": XvcConfigValue { + source: Project, + value: Boolean( + true, + ), + }, + "file.list.sort": XvcConfigValue { + source: Project, + value: String( + "name-desc", + ), + }, + "file.track.no_parallel": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "file.list.show_dot_files": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "pipeline.default": XvcConfigValue { + source: Project, + value: String( + "default", + ), + }, + "core.verbosity": XvcConfigValue { + source: CommandLine, + value: String( + "debug", + ), + }, + "file.list.no_summary": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "pipeline.process_pool_size": XvcConfigValue { + source: Project, + value: Integer( + 4, + ), + }, + "file.track.no_commit": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + }, + init_params: XvcConfigParams { + default_configuration: " +[core] +# The repository id. Please do not delete or change it. +# This is used to identify the repository and generate paths in storages. +# In the future it may be used to in other ways. +guid = /"2563bb26e79eff77/" +# Default verbosity level. +# One of /"error/", /"warn/", /"info/" +verbosity = /"error/" + +[git] +# Automate git operations. +# Turning this off leads Xvc to behave as if it's not in a Git repository. +# Not recommended unless you're really not using Git +use_git = true +# Command to run Git process. +# You can set this to an absolute path to specify an executable +# If set to a non-absolute path, the executable will be searched in $PATH. +command = /"git/" + +# Commit changes in .xvc/ directory after commands. +# You can set this to false if you want to commit manually. +auto_commit = true + +# Stage changes in .xvc/ directory without committing. +# auto_commit implies auto_stage. +# If you want to commit manually but don't want to stage after individual Xvc commands, you can set this to true. +auto_stage = false + +[cache] +# The hash algorithm used for the cache. +# It may take blake3, blake2, sha2 or sha3 as values. +# All algorithms are selected to produce 256-bit hashes, so sha2 means SHA2-256, blake2 means BLAKE2s, etc. +# The cache path is produced by prepending algorithm name to the cache. +# Blake3 files are in .xvc/b3/, while sha2 files are in .xvc/s2/ etc. +algorithm = /"blake3/" + +[file] + +[file.track] + +# Don't move file content to cache after xvc file track +no_commit = false +# Force to track files even if they are already tracked. +force = false + +# Xvc calculates file content digest differently for text and binary files. +# This option controls whether to treat files as text or binary. +# It may take auto, text or binary as values. +# Auto check each file individually and treat it as text if it's text. +text_or_binary = /"auto/" + +# Don't use parallelism in track operations. +# Note that some of the operations are implemented in parallel by default, and this option affects some heavier operations. +no_parallel = false + +[file.list] + +# Format for `xvc file list` rows. You can reorder or remove columns. +# The following are the keys for each row: +# - {acd64}: actual content digest. All 64 digits from the workspace file's content. +# - {acd8}: actual content digest. First 8 digits the file content digest. +# - {aft}: actual file type. Whether the entry is a file (F), directory (D), +# symlink (S), hardlink (H) or reflink (R). +# - {asz}: actual size. The size of the workspace file in bytes. It uses MB, +# GB and TB to represent sizes larger than 1MB. +# - {ats}: actual timestamp. The timestamp of the workspace file. +# - {cst}: cache status. One of /"=/", /">/", /" 0 + F 000001: + >000002: /x00 => 2 + 000003: /x00 => 0 + match kind: Standard + prefilter: false + state length: 4 + pattern length: 0 + shortest pattern length: 18446744073709551615 + longest pattern length: 0 + alphabet length: 1 + stride: 1 + byte classes: ByteClasses(0 => [0-255]) + memory usage: 16 + ) + , + ), + map: [], + longest: 0, + }, + ), + Prefix( + PrefixStrategy { + matcher: AhoCorasick( + dfa::DFA( + D 000000: /x00 => 0 + F 000001: + >000002: /x00 => 2 + 000003: /x00 => 0 + match kind: Standard + prefilter: false + state length: 4 + pattern length: 0 + shortest pattern length: 18446744073709551615 + longest pattern length: 0 + alphabet length: 1 + stride: 1 + byte classes: ByteClasses(0 => [0-255]) + memory usage: 16 + ) + , + ), + map: [], + longest: 0, + }, + ), + RequiredExtension( + RequiredExtensionStrategy( + {}, + ), + ), + Regex( + RegexSetStrategy { + matcher: Regex { + imp: RegexI { + strat: Core { + info: RegexInfo( + RegexInfoI { + config: Config { + match_kind: Some( + All, + ), + utf8_empty: Some( + false, + ), + autopre: None, + pre: None, + which_captures: None, + nfa_size_limit: Some( + Some( + 10485760, + ), + ), + onepass_size_limit: None, + hybrid_cache_capacity: Some( + 10485760, + ), + hybrid: None, + dfa: None, + dfa_size_limit: None, + dfa_state_limit: None, + onepass: None, + backtrack: None, + byte_classes: None, + line_terminator: None, + }, + props: [], + props_union: Properties( + PropertiesI { + minimum_len: None, + maximum_len: None, + look_set: ∅, + look_set_prefix: ∅, + look_set_suffix: ∅, + look_set_prefix_any: ∅, + look_set_suffix_any: ∅, + utf8: true, + explicit_captures_len: 0, + static_explicit_captures_len: None, + literal: false, + alternation_literal: true, + }, + ), + }, + ), + pre: None, + nfa: thompson::NFA( + ^000000: FAIL + + transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) + ) + , + nfarev: Some( + thompson::NFA( + ^000000: FAIL + + transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) + ) + , + ), + pikevm: PikeVM( + PikeVMEngine( + PikeVM { + config: Config { + match_kind: Some( + All, + ), + pre: Some( + None, + ), + }, + nfa: thompson::NFA( + ^000000: FAIL + + transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) + ) + , + }, + ), + ), + backtrack: BoundedBacktracker( + None, + ), + onepass: OnePass( + None, + ), + hybrid: Hybrid( + Some( + HybridEngine( + Regex { + forward: DFA { + config: Config { + match_kind: Some( + All, + ), + pre: Some( + None, + ), + starts_for_each_pattern: Some( + true, + ), + byte_classes: Some( + true, + ), + unicode_word_boundary: Some( + true, + ), + quitset: None, + specialize_start_states: Some( + false, + ), + cache_capacity: Some( + 10485760, + ), + skip_cache_capacity_check: Some( + false, + ), + minimum_cache_clear_count: Some( + Some( + 3, + ), + ), + minimum_bytes_per_state: Some( + Some( + 10, + ), + ), + }, + nfa: thompson::NFA( + ^000000: FAIL + + transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) + ) + , + stride2: 1, + start_map: StartByteMap{/x00 => NonWordByte, /x01 => NonWordByte, /x02 => NonWordByte, /x03 => NonWordByte, /x04 => NonWordByte, /x05 => NonWordByte, /x06 => NonWordByte, /x07 => NonWordByte, /x08 => NonWordByte, /t => NonWordByte, + => LineLF, /x0B => NonWordByte, /x0C => NonWordByte, /r => LineCR, /x0E => NonWordByte, /x0F => NonWordByte, /x10 => NonWordByte, /x11 => NonWordByte, /x12 => NonWordByte, /x13 => NonWordByte, /x14 => NonWordByte, /x15 => NonWordByte, /x16 => NonWordByte, /x17 => NonWordByte, /x18 => NonWordByte, /x19 => NonWordByte, /x1A => NonWordByte, /x1B => NonWordByte, /x1C => NonWordByte, /x1D => NonWordByte, /x1E => NonWordByte, /x1F => NonWordByte, ' ' => NonWordByte, ! => NonWordByte, /" => NonWordByte, # => NonWordByte, $ => NonWordByte, % => NonWordByte, & => NonWordByte, /' => NonWordByte, ( => NonWordByte, ) => NonWordByte, * => NonWordByte, + => NonWordByte, , => NonWordByte, - => NonWordByte, . => NonWordByte, / => NonWordByte, 0 => WordByte, 1 => WordByte, 2 => WordByte, 3 => WordByte, 4 => WordByte, 5 => WordByte, 6 => WordByte, 7 => WordByte, 8 => WordByte, 9 => WordByte, : => NonWordByte, ; => NonWordByte, < => NonWordByte, = => NonWordByte, > => NonWordByte, ? => NonWordByte, @ => NonWordByte, A => WordByte, B => WordByte, C => WordByte, D => WordByte, E => WordByte, F => WordByte, G => WordByte, H => WordByte, I => WordByte, J => WordByte, K => WordByte, L => WordByte, M => WordByte, N => WordByte, O => WordByte, P => WordByte, Q => WordByte, R => WordByte, S => WordByte, T => WordByte, U => WordByte, V => WordByte, W => WordByte, X => WordByte, Y => WordByte, Z => WordByte, [ => NonWordByte, // => NonWordByte, ] => NonWordByte, ^ => NonWordByte, _ => WordByte, ` => NonWordByte, a => WordByte, b => WordByte, c => WordByte, d => WordByte, e => WordByte, f => WordByte, g => WordByte, h => WordByte, i => WordByte, j => WordByte, k => WordByte, l => WordByte, m => WordByte, n => WordByte, o => WordByte, p => WordByte, q => WordByte, r => WordByte, s => WordByte, t => WordByte, u => WordByte, v => WordByte, w => WordByte, x => WordByte, y => WordByte, z => WordByte, { => NonWordByte, | => NonWordByte, } => NonWordByte, ~ => NonWordByte, /x7F => NonWordByte, /x80 => NonWordByte, /x81 => NonWordByte, /x82 => NonWordByte, /x83 => NonWordByte, /x84 => NonWordByte, /x85 => NonWordByte, /x86 => NonWordByte, /x87 => NonWordByte, /x88 => NonWordByte, /x89 => NonWordByte, /x8A => NonWordByte, /x8B => NonWordByte, /x8C => NonWordByte, /x8D => NonWordByte, /x8E => NonWordByte, /x8F => NonWordByte, /x90 => NonWordByte, /x91 => NonWordByte, /x92 => NonWordByte, /x93 => NonWordByte, /x94 => NonWordByte, /x95 => NonWordByte, /x96 => NonWordByte, /x97 => NonWordByte, /x98 => NonWordByte, /x99 => NonWordByte, /x9A => NonWordByte, /x9B => NonWordByte, /x9C => NonWordByte, /x9D => NonWordByte, /x9E => NonWordByte, /x9F => NonWordByte, /xA0 => NonWordByte, /xA1 => NonWordByte, /xA2 => NonWordByte, /xA3 => NonWordByte, /xA4 => NonWordByte, /xA5 => NonWordByte, /xA6 => NonWordByte, /xA7 => NonWordByte, /xA8 => NonWordByte, /xA9 => NonWordByte, /xAA => NonWordByte, /xAB => NonWordByte, /xAC => NonWordByte, /xAD => NonWordByte, /xAE => NonWordByte, /xAF => NonWordByte, /xB0 => NonWordByte, /xB1 => NonWordByte, /xB2 => NonWordByte, /xB3 => NonWordByte, /xB4 => NonWordByte, /xB5 => NonWordByte, /xB6 => NonWordByte, /xB7 => NonWordByte, /xB8 => NonWordByte, /xB9 => NonWordByte, /xBA => NonWordByte, /xBB => NonWordByte, /xBC => NonWordByte, /xBD => NonWordByte, /xBE => NonWordByte, /xBF => NonWordByte, /xC0 => NonWordByte, /xC1 => NonWordByte, /xC2 => NonWordByte, /xC3 => NonWordByte, /xC4 => NonWordByte, /xC5 => NonWordByte, /xC6 => NonWordByte, /xC7 => NonWordByte, /xC8 => NonWordByte, /xC9 => NonWordByte, /xCA => NonWordByte, /xCB => NonWordByte, /xCC => NonWordByte, /xCD => NonWordByte, /xCE => NonWordByte, /xCF => NonWordByte, /xD0 => NonWordByte, /xD1 => NonWordByte, /xD2 => NonWordByte, /xD3 => NonWordByte, /xD4 => NonWordByte, /xD5 => NonWordByte, /xD6 => NonWordByte, /xD7 => NonWordByte, /xD8 => NonWordByte, /xD9 => NonWordByte, /xDA => NonWordByte, /xDB => NonWordByte, /xDC => NonWordByte, /xDD => NonWordByte, /xDE => NonWordByte, /xDF => NonWordByte, /xE0 => NonWordByte, /xE1 => NonWordByte, /xE2 => NonWordByte, /xE3 => NonWordByte, /xE4 => NonWordByte, /xE5 => NonWordByte, /xE6 => NonWordByte, /xE7 => NonWordByte, /xE8 => NonWordByte, /xE9 => NonWordByte, /xEA => NonWordByte, /xEB => NonWordByte, /xEC => NonWordByte, /xED => NonWordByte, /xEE => NonWordByte, /xEF => NonWordByte, /xF0 => NonWordByte, /xF1 => NonWordByte, /xF2 => NonWordByte, /xF3 => NonWordByte, /xF4 => NonWordByte, /xF5 => NonWordByte, /xF6 => NonWordByte, /xF7 => NonWordByte, /xF8 => NonWordByte, /xF9 => NonWordByte, /xFA => NonWordByte, /xFB => NonWordByte, /xFC => NonWordByte, /xFD => NonWordByte, /xFE => NonWordByte, /xFF => NonWordByte}, + classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]), + quitset: ByteSet { + bits: {}, + }, + cache_capacity: 10485760, + }, + reverse: DFA { + config: Config { + match_kind: Some( + All, + ), + pre: Some( + None, + ), + starts_for_each_pattern: Some( + true, + ), + byte_classes: Some( + true, + ), + unicode_word_boundary: Some( + true, + ), + quitset: None, + specialize_start_states: Some( + false, + ), + cache_capacity: Some( + 10485760, + ), + skip_cache_capacity_check: Some( + false, + ), + minimum_cache_clear_count: Some( + Some( + 3, + ), + ), + minimum_bytes_per_state: Some( + Some( + 10, + ), + ), + }, + nfa: thompson::NFA( + ^000000: FAIL + + transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) + ) + , + stride2: 1, + start_map: StartByteMap{/x00 => NonWordByte, /x01 => NonWordByte, /x02 => NonWordByte, /x03 => NonWordByte, /x04 => NonWordByte, /x05 => NonWordByte, /x06 => NonWordByte, /x07 => NonWordByte, /x08 => NonWordByte, /t => NonWordByte, + => LineLF, /x0B => NonWordByte, /x0C => NonWordByte, /r => LineCR, /x0E => NonWordByte, /x0F => NonWordByte, /x10 => NonWordByte, /x11 => NonWordByte, /x12 => NonWordByte, /x13 => NonWordByte, /x14 => NonWordByte, /x15 => NonWordByte, /x16 => NonWordByte, /x17 => NonWordByte, /x18 => NonWordByte, /x19 => NonWordByte, /x1A => NonWordByte, /x1B => NonWordByte, /x1C => NonWordByte, /x1D => NonWordByte, /x1E => NonWordByte, /x1F => NonWordByte, ' ' => NonWordByte, ! => NonWordByte, /" => NonWordByte, # => NonWordByte, $ => NonWordByte, % => NonWordByte, & => NonWordByte, /' => NonWordByte, ( => NonWordByte, ) => NonWordByte, * => NonWordByte, + => NonWordByte, , => NonWordByte, - => NonWordByte, . => NonWordByte, / => NonWordByte, 0 => WordByte, 1 => WordByte, 2 => WordByte, 3 => WordByte, 4 => WordByte, 5 => WordByte, 6 => WordByte, 7 => WordByte, 8 => WordByte, 9 => WordByte, : => NonWordByte, ; => NonWordByte, < => NonWordByte, = => NonWordByte, > => NonWordByte, ? => NonWordByte, @ => NonWordByte, A => WordByte, B => WordByte, C => WordByte, D => WordByte, E => WordByte, F => WordByte, G => WordByte, H => WordByte, I => WordByte, J => WordByte, K => WordByte, L => WordByte, M => WordByte, N => WordByte, O => WordByte, P => WordByte, Q => WordByte, R => WordByte, S => WordByte, T => WordByte, U => WordByte, V => WordByte, W => WordByte, X => WordByte, Y => WordByte, Z => WordByte, [ => NonWordByte, // => NonWordByte, ] => NonWordByte, ^ => NonWordByte, _ => WordByte, ` => NonWordByte, a => WordByte, b => WordByte, c => WordByte, d => WordByte, e => WordByte, f => WordByte, g => WordByte, h => WordByte, i => WordByte, j => WordByte, k => WordByte, l => WordByte, m => WordByte, n => WordByte, o => WordByte, p => WordByte, q => WordByte, r => WordByte, s => WordByte, t => WordByte, u => WordByte, v => WordByte, w => WordByte, x => WordByte, y => WordByte, z => WordByte, { => NonWordByte, | => NonWordByte, } => NonWordByte, ~ => NonWordByte, /x7F => NonWordByte, /x80 => NonWordByte, /x81 => NonWordByte, /x82 => NonWordByte, /x83 => NonWordByte, /x84 => NonWordByte, /x85 => NonWordByte, /x86 => NonWordByte, /x87 => NonWordByte, /x88 => NonWordByte, /x89 => NonWordByte, /x8A => NonWordByte, /x8B => NonWordByte, /x8C => NonWordByte, /x8D => NonWordByte, /x8E => NonWordByte, /x8F => NonWordByte, /x90 => NonWordByte, /x91 => NonWordByte, /x92 => NonWordByte, /x93 => NonWordByte, /x94 => NonWordByte, /x95 => NonWordByte, /x96 => NonWordByte, /x97 => NonWordByte, /x98 => NonWordByte, /x99 => NonWordByte, /x9A => NonWordByte, /x9B => NonWordByte, /x9C => NonWordByte, /x9D => NonWordByte, /x9E => NonWordByte, /x9F => NonWordByte, /xA0 => NonWordByte, /xA1 => NonWordByte, /xA2 => NonWordByte, /xA3 => NonWordByte, /xA4 => NonWordByte, /xA5 => NonWordByte, /xA6 => NonWordByte, /xA7 => NonWordByte, /xA8 => NonWordByte, /xA9 => NonWordByte, /xAA => NonWordByte, /xAB => NonWordByte, /xAC => NonWordByte, /xAD => NonWordByte, /xAE => NonWordByte, /xAF => NonWordByte, /xB0 => NonWordByte, /xB1 => NonWordByte, /xB2 => NonWordByte, /xB3 => NonWordByte, /xB4 => NonWordByte, /xB5 => NonWordByte, /xB6 => NonWordByte, /xB7 => NonWordByte, /xB8 => NonWordByte, /xB9 => NonWordByte, /xBA => NonWordByte, /xBB => NonWordByte, /xBC => NonWordByte, /xBD => NonWordByte, /xBE => NonWordByte, /xBF => NonWordByte, /xC0 => NonWordByte, /xC1 => NonWordByte, /xC2 => NonWordByte, /xC3 => NonWordByte, /xC4 => NonWordByte, /xC5 => NonWordByte, /xC6 => NonWordByte, /xC7 => NonWordByte, /xC8 => NonWordByte, /xC9 => NonWordByte, /xCA => NonWordByte, /xCB => NonWordByte, /xCC => NonWordByte, /xCD => NonWordByte, /xCE => NonWordByte, /xCF => NonWordByte, /xD0 => NonWordByte, /xD1 => NonWordByte, /xD2 => NonWordByte, /xD3 => NonWordByte, /xD4 => NonWordByte, /xD5 => NonWordByte, /xD6 => NonWordByte, /xD7 => NonWordByte, /xD8 => NonWordByte, /xD9 => NonWordByte, /xDA => NonWordByte, /xDB => NonWordByte, /xDC => NonWordByte, /xDD => NonWordByte, /xDE => NonWordByte, /xDF => NonWordByte, /xE0 => NonWordByte, /xE1 => NonWordByte, /xE2 => NonWordByte, /xE3 => NonWordByte, /xE4 => NonWordByte, /xE5 => NonWordByte, /xE6 => NonWordByte, /xE7 => NonWordByte, /xE8 => NonWordByte, /xE9 => NonWordByte, /xEA => NonWordByte, /xEB => NonWordByte, /xEC => NonWordByte, /xED => NonWordByte, /xEE => NonWordByte, /xEF => NonWordByte, /xF0 => NonWordByte, /xF1 => NonWordByte, /xF2 => NonWordByte, /xF3 => NonWordByte, /xF4 => NonWordByte, /xF5 => NonWordByte, /xF6 => NonWordByte, /xF7 => NonWordByte, /xF8 => NonWordByte, /xF9 => NonWordByte, /xFA => NonWordByte, /xFB => NonWordByte, /xFC => NonWordByte, /xFD => NonWordByte, /xFE => NonWordByte, /xFF => NonWordByte}, + classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]), + quitset: ByteSet { + bits: {}, + }, + cache_capacity: 10485760, + }, + }, + ), + ), + ), + dfa: DFA( + None, + ), + }, + info: RegexInfo( + RegexInfoI { + config: Config { + match_kind: Some( + All, + ), + utf8_empty: Some( + false, + ), + autopre: None, + pre: None, + which_captures: None, + nfa_size_limit: Some( + Some( + 10485760, + ), + ), + onepass_size_limit: None, + hybrid_cache_capacity: Some( + 10485760, + ), + hybrid: None, + dfa: None, + dfa_size_limit: None, + dfa_state_limit: None, + onepass: None, + backtrack: None, + byte_classes: None, + line_terminator: None, + }, + props: [], + props_union: Properties( + PropertiesI { + minimum_len: None, + maximum_len: None, + look_set: ∅, + look_set_prefix: ∅, + look_set_suffix: ∅, + look_set_prefix_any: ∅, + look_set_suffix_any: ∅, + utf8: true, + explicit_captures_len: 0, + static_explicit_captures_len: None, + literal: false, + alternation_literal: true, + }, + ), + }, + ), + }, + pool: Pool( + Pool { + stacks: [ + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + ], + owner: 0, + owner_val: UnsafeCell { .. }, + }, + ), + }, + map: [], + patset: Pool( + Pool { + stacks: [ + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + ], + owner: 0, + owner_val: UnsafeCell { .. }, + }, + ), + }, + ), + ], + }, + poisoned: false, + .. + }, + }, +} +[TRACE][pipeline/src/pipeline/mod.rs::305] pipeline_len: 1 +[TRACE][pipeline/src/pipeline/mod.rs::331] &dependency_graph: { + XvcEntity( + 2, + 17595501963111217905, + ): [], +} +[TRACE][pipeline/src/pipeline/mod.rs::343] &dependency_graph: { + XvcEntity( + 2, + 17595501963111217905, + ): [], +} +[INFO][pipeline/src/pipeline/mod.rs::347] Pipeline Graph: +digraph { + 0 [ label = "(2, 17595501963111217905)" ] +} + + +[TRACE][pipeline/src/pipeline/mod.rs::412] step_states: RwLock { + data: HStore { + map: { + XvcEntity( + 2, + 17595501963111217905, + ): Begin( + FromInit, + ), + }, + }, + poisoned: false, + .. +} +[TRACE][pipeline/src/pipeline/mod.rs::508] &step_thread_store: HStore { + map: { + XvcEntity( + 2, + 17595501963111217905, + ): ScopedJoinHandle { .. }, + }, +} +[TRACE][pipeline/src/pipeline/mod.rs::512] (step_e, &jh): ( + XvcEntity( + 2, + 17595501963111217905, + ), + ScopedJoinHandle { .. }, +) +[TRACE][pipeline/src/pipeline/mod.rs::583] select: Select { .. } +[TRACE][pipeline/src/pipeline/mod.rs::623] params.recorded_dependencies: R1NStore { + parents: XvcStore { + map: { + XvcEntity( + 2, + 17595501963111217905, + ): XvcStep { + name: "average-age", + }, + }, + entity_index: { + XvcStep { + name: "average-age", + }: [ + XvcEntity( + 2, + 17595501963111217905, + ), + ], + }, + previous: EventLog( + [ + Add { + entity: XvcEntity( + 2, + 17595501963111217905, + ), + value: XvcStep { + name: "average-age", + }, + }, + Add { + entity: XvcEntity( + 2, + 17595501963111217905, + ), + value: XvcStep { + name: "average-age", + }, + }, + ], + ), + current: EventLog( + [], + ), + }, + children: XvcStore { + map: { + XvcEntity( + 3, + 15601036540397786949, + ): SqliteQueryDigest( + SqliteQueryDep { + path: XvcPath( + "people.db", + ), + query: "SELECT count(*) FROM People;", + query_digest: None, + xvc_metadata: None, + }, + ), + }, + entity_index: { + SqliteQueryDigest( + SqliteQueryDep { + path: XvcPath( + "people.db", + ), + query: "SELECT count(*) FROM People;", + query_digest: None, + xvc_metadata: None, + }, + ): [ + XvcEntity( + 3, + 15601036540397786949, + ), + ], + }, + previous: EventLog( + [ + Add { + entity: XvcEntity( + 3, + 15601036540397786949, + ), + value: SqliteQueryDigest( + SqliteQueryDep { + path: XvcPath( + "people.db", + ), + query: "SELECT count(*) FROM People;", + query_digest: None, + xvc_metadata: None, + }, + ), + }, + ], + ), + current: EventLog( + [], + ), + }, + child_parents: XvcStore { + map: { + XvcEntity( + 3, + 15601036540397786949, + ): ChildEntity( + XvcEntity( + 2, + 17595501963111217905, + ), + PhantomData, + PhantomData, + ), + }, + entity_index: { + ChildEntity( + XvcEntity( + 2, + 17595501963111217905, + ), + PhantomData, + PhantomData, + ): [ + XvcEntity( + 3, + 15601036540397786949, + ), + ], + }, + previous: EventLog( + [ + Add { + entity: XvcEntity( + 3, + 15601036540397786949, + ), + value: ChildEntity( + XvcEntity( + 2, + 17595501963111217905, + ), + PhantomData, + PhantomData, + ), + }, + ], + ), + current: EventLog( + [], + ), + }, +} +[TRACE][pipeline/src/pipeline/mod.rs::624] step_e: XvcEntity( + 2, + 17595501963111217905, +) +[TRACE][pipeline/src/pipeline/mod.rs::564] dep_neighbors: Neighbors { + iter: Iter( + [], + ), + ty: PhantomData, +} +[TRACE][pipeline/src/pipeline/mod.rs::625] dependency_steps(step_e, params.dependency_graph)?: {} +[TRACE][pipeline/src/pipeline/mod.rs::564] dep_neighbors: Neighbors { + iter: Iter( + [], + ), + ty: PhantomData, +} +[TRACE][pipeline/src/pipeline/mod.rs::662] &step_state: Begin( + FromInit, +) +[TRACE][pipeline/src/pipeline/mod.rs::772] step.name: "average-age" +[TRACE][pipeline/src/pipeline/mod.rs::773] &r_next_state: WaitingDependencySteps( + FromRunConditional, +) +[TRACE][pipeline/src/pipeline/mod.rs::775] &step_state: WaitingDependencySteps( + FromRunConditional, +) +[TRACE][pipeline/src/pipeline/mod.rs::662] &step_state: WaitingDependencySteps( + FromRunConditional, +) +[TRACE][pipeline/src/pipeline/mod.rs::772] step.name: "average-age" +[TRACE][pipeline/src/pipeline/mod.rs::773] &r_next_state: CheckingOutputs( + FromDependencyStepsFinishedSuccessfully, +) +[TRACE][pipeline/src/pipeline/mod.rs::775] &step_state: CheckingOutputs( + FromDependencyStepsFinishedSuccessfully, +) +[TRACE][pipeline/src/pipeline/mod.rs::662] &step_state: CheckingOutputs( + FromDependencyStepsFinishedSuccessfully, +) +[TRACE][pipeline/src/pipeline/mod.rs::772] step.name: "average-age" +[TRACE][pipeline/src/pipeline/mod.rs::773] &r_next_state: CheckingSuperficialDiffs( + FromCheckedOutputs, +) +[TRACE][pipeline/src/pipeline/mod.rs::775] &step_state: CheckingSuperficialDiffs( + FromCheckedOutputs, +) +[TRACE][pipeline/src/pipeline/mod.rs::662] &step_state: CheckingSuperficialDiffs( + FromCheckedOutputs, +) +[TRACE][pipeline/src/pipeline/mod.rs::1010] parent_entity: XvcEntity( + 2, + 17595501963111217905, +) +[TRACE][pipeline/src/pipeline/mod.rs::583] select: Select { .. } +[TRACE][pipeline/src/pipeline/mod.rs::1013] deps: HStore { + map: { + XvcEntity( + 3, + 15601036540397786949, + ): SqliteQueryDigest( + SqliteQueryDep { + path: XvcPath( + "people.db", + ), + query: "SELECT count(*) FROM People;", + query_digest: None, + xvc_metadata: None, + }, + ), + }, +} +[TRACE][pipeline/src/pipeline/deps/compare.rs::449] &stored: SqliteQueryDigest( + SqliteQueryDep { + path: XvcPath( + "people.db", + ), + query: "SELECT count(*) FROM People;", + query_digest: None, + xvc_metadata: None, + }, +) +[TRACE][core/src/util/pmp.rs::145] path: XvcPath( + "people.db", +) +[TRACE][core/src/util/pmp.rs::171] xvc_path: XvcPath( + "people.db", +) +[TRACE][core/src/util/pmp.rs::173] path: AbsolutePath( + "[CWD]/people.db", +) +[TRACE][pipeline/src/pipeline/mod.rs::583] select: Select { .. } +[TRACE][core/src/util/pmp.rs::175] &md: Ok( + Metadata { + file_type: FileType( + FileType { + mode: 33188, + }, + ), + is_dir: false, + is_file: true, + permissions: Permissions( + FilePermissions { + mode: 33188, + }, + ), + modified: Ok( + SystemTime { + tv_sec: 1722024927, + tv_nsec: 262703931, + }, + ), + accessed: Ok( + SystemTime { + tv_sec: 1722025051, + tv_nsec: 673004106, + }, + ), + created: Ok( + SystemTime { + tv_sec: 1722024927, + tv_nsec: 261929105, + }, + ), + .. + }, +) +[TRACE][pipeline/src/pipeline/mod.rs::583] select: Select { .. } +[TRACE][pipeline/src/pipeline/mod.rs::583] select: Select { .. } +[TRACE][core/src/util/pmp.rs::152] &md: Some( + XvcMetadata { + file_type: File, + size: Some( + 8192, + ), + modified: Some( + SystemTime { + tv_sec: 1722024927, + tv_nsec: 262703931, + }, + ), + }, +) +[TRACE][pipeline/src/pipeline/mod.rs::1030] step_dependency_diffs: HStore { + map: { + XvcEntity( + 3, + 15601036540397786949, + ): RecordMissing { + actual: SqliteQueryDigest( + SqliteQueryDep { + path: XvcPath( + "people.db", + ), + query: "SELECT count(*) FROM People;", + query_digest: None, + xvc_metadata: Some( + XvcMetadata { + file_type: File, + size: Some( + 8192, + ), + modified: Some( + SystemTime { + tv_sec: 1722024927, + tv_nsec: 262703931, + }, + ), + }, + ), + }, + ), + }, + }, +} +[TRACE][pipeline/src/pipeline/mod.rs::1036] diff: RecordMissing { + actual: SqliteQueryDigest( + SqliteQueryDep { + path: XvcPath( + "people.db", + ), + query: "SELECT count(*) FROM People;", + query_digest: None, + xvc_metadata: Some( + XvcMetadata { + file_type: File, + size: Some( + 8192, + ), + modified: Some( + SystemTime { + tv_sec: 1722024927, + tv_nsec: 262703931, + }, + ), + }, + ), + }, + ), +} +[INFO] No dependency steps for step average-age +[TRACE][pipeline/src/pipeline/mod.rs::1037] diff.changed(): true +[TRACE][pipeline/src/pipeline/mod.rs::1042] changed: true +[TRACE][pipeline/src/pipeline/mod.rs::772] step.name: "average-age" +[TRACE][pipeline/src/pipeline/mod.rs::773] &r_next_state: CheckingThoroughDiffs( + FromSuperficialDiffsChanged, +) +[TRACE][pipeline/src/pipeline/mod.rs::775] &step_state: CheckingThoroughDiffs( + FromSuperficialDiffsChanged, +) +[TRACE][pipeline/src/pipeline/mod.rs::662] &step_state: CheckingThoroughDiffs( + FromSuperficialDiffsChanged, +) +[TRACE][pipeline/src/pipeline/mod.rs::1067] deps: HStore { + map: { + XvcEntity( + 3, + 15601036540397786949, + ): SqliteQueryDigest( + SqliteQueryDep { + path: XvcPath( + "people.db", + ), + query: "SELECT count(*) FROM People;", + query_digest: None, + xvc_metadata: None, + }, + ), + }, +} +[TRACE][core/src/util/pmp.rs::145] path: XvcPath( + "people.db", +) +[TRACE][core/src/util/pmp.rs::152] &md: Some( + XvcMetadata { + file_type: File, + size: Some( + 8192, + ), + modified: Some( + SystemTime { + tv_sec: 1722024927, + tv_nsec: 262703931, + }, + ), + }, +) +[TRACE][pipeline/src/pipeline/mod.rs::583] select: Select { .. } thread '' panicked at pipeline/src/pipeline/mod.rs:1081:32: SqliteError { source: FromSqlConversionFailure(18446744073709551615, Null, InvalidType) } note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace -thread '' panicked at lib/src/cli/mod.rs:251:52: +thread '' panicked at lib/src/cli/mod.rs:288:52: [PANIC] SqliteError { source: FromSqlConversionFailure(18446744073709551615, Null, InvalidType) }, [pipeline/src/pipeline/mod.rs::1081] -thread '' panicked at pipeline/src/pipeline/mod.rs:514:17: +[TRACE][pipeline/src/pipeline/mod.rs::519] "Before state updater": "Before state updater" +[TRACE][pipeline/src/pipeline/mod.rs::593] s: CheckingThoroughDiffs( + FromSuperficialDiffsChanged, +) +[TRACE][pipeline/src/pipeline/mod.rs::532] step_states: RwLock { + data: HStore { + map: { + XvcEntity( + 2, + 17595501963111217905, + ): CheckingThoroughDiffs( + FromSuperficialDiffsChanged, + ), + }, + }, + poisoned: false, + .. +} +[TRACE][pipeline/src/pipeline/mod.rs::536] done_successfully: Ok( + false, +) +[TRACE][core/src/util/pmp.rs::185] self.background_thread: Mutex { + data: JoinHandle { .. }, + poisoned: false, + .. +} +[TRACE][core/src/util/pmp.rs::190] self.background_thread: Mutex { + data: JoinHandle { .. }, + poisoned: false, + .. +} +[TRACE][lib/src/cli/mod.rs::376] "Before handle_git_automation": "Before handle_git_automation" +[TRACE][core/src/util/pmp.rs::99] index: 1 +thread '' panicked at lib/src/git.rs:155:5: called `Result::unwrap()` on an `Err` value: "SendError(..)" +thread 'main' panicked at lib/src/cli/mod.rs:402:52: +called `Result::unwrap()` on an `Err` value: Any { .. } ``` @@ -119,9 +2541,9 @@ This time, the step will run again as the result from dependency query (`SELECT $ xvc pipeline run ? interrupted thread '' panicked at pipeline/src/pipeline/mod.rs:1081:32: -thread '' panicked at lib/src/cli/mod.rs:251SqliteError { source: FromSqlConversionFailure(18446744073709551615, Null, InvalidType) } +SqliteError { source: FromSqlConversionFailure(18446744073709551615, Null, InvalidType) } note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace -:52: +thread '' panicked at lib/src/cli/mod.rs:251:52: [PANIC] SqliteError { source: FromSqlConversionFailure(18446744073709551615, Null, InvalidType) }, [pipeline/src/pipeline/mod.rs::1081] thread '' panicked at pipeline/src/pipeline/mod.rs:514:17: called `Result::unwrap()` on an `Err` value: "SendError(..)" diff --git a/pipeline/src/pipeline/deps/sqlite_query.rs b/pipeline/src/pipeline/deps/sqlite_query.rs index 56402d31f..376238626 100644 --- a/pipeline/src/pipeline/deps/sqlite_query.rs +++ b/pipeline/src/pipeline/deps/sqlite_query.rs @@ -2,6 +2,7 @@ //! and checks whether the result of that query has changed. It doesn't run the query if the //! metadata of the database file hasn't changed. use serde::{Deserialize, Serialize}; +use xvc_logging::watch; use crate::XvcDependency; use rusqlite::{Connection, OpenFlags}; use fallible_iterator::FallibleIterator; @@ -67,6 +68,7 @@ impl SqliteQueryDep { // TODO: Add salting with the repo id here? let mut els = String::new(); while let Ok(col) = row.get_ref(i) { + watch!(col); els.push_str(col.as_str()?); i += 1; } From df6e2e80abd3adaad15357ce43a5acf6a4ab1a0d Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sun, 28 Jul 2024 15:45:16 +0300 Subject: [PATCH 078/257] remove watch in select --- pipeline/src/pipeline/mod.rs | 2 -- 1 file changed, 2 deletions(-) diff --git a/pipeline/src/pipeline/mod.rs b/pipeline/src/pipeline/mod.rs index acbd4254a..1952cf90f 100644 --- a/pipeline/src/pipeline/mod.rs +++ b/pipeline/src/pipeline/mod.rs @@ -580,7 +580,6 @@ fn step_state_bulletin( select.recv(r); } loop { - watch!(select); if let Ok(index) = select.ready_timeout(Duration::from_millis(10)) { let res = state_senders[index].1.recv()?; if let Some(state) = res { @@ -590,7 +589,6 @@ fn step_state_bulletin( } } else { if current_states.read()?.iter().all(|(_, s)| { - watch!(s); matches!( s, XvcStepState::DoneByRunning(_) From 817eb363555bc10bd224dc02b6de242701bb6545 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sun, 28 Jul 2024 15:52:28 +0300 Subject: [PATCH 079/257] handle sqlite type --- ...c-pipeline-step-dependency-sqlite-query.md | 730 ++++++++++++------ pipeline/src/pipeline/deps/sqlite_query.rs | 15 +- 2 files changed, 504 insertions(+), 241 deletions(-) diff --git a/book/src/ref/xvc-pipeline-step-dependency-sqlite-query.md b/book/src/ref/xvc-pipeline-step-dependency-sqlite-query.md index 114186515..1fdae75dc 100644 --- a/book/src/ref/xvc-pipeline-step-dependency-sqlite-query.md +++ b/book/src/ref/xvc-pipeline-step-dependency-sqlite-query.md @@ -81,7 +81,7 @@ So, when the number of people in the table changes, the step will run. Initially ```console $ xvc -vvvv pipeline run -? 101 +? interrupted [DEBUG][logging/src/lib.rs::237] Terminal logger enabled with level: Trace [TRACE][core/src/types/xvcroot.rs::263] xvc_candidate: "[CWD]/.xvc" [TRACE][core/src/types/xvcroot.rs::265] parent: "[CWD]" @@ -89,22 +89,22 @@ $ xvc -vvvv pipeline run [DEBUG][config/src/error.rs::72] Config source for level "global" not found at "/Users/iex/Library/Application Support/xvc" [TRACE][ecs/src/ecs/mod.rs::229] dir: "[CWD]/.xvc/ec" [TRACE][ecs/src/ecs/mod.rs::239] files: [ - "[CWD]/.xvc/ec/1722026597763090", - "[CWD]/.xvc/ec/1722026597766408", - "[CWD]/.xvc/ec/1722026597840236", - "[CWD]/.xvc/ec/1722026598025010", + "[CWD]/.xvc/ec/1722170724738869", + "[CWD]/.xvc/ec/1722170724741681", + "[CWD]/.xvc/ec/1722170724810551", + "[CWD]/.xvc/ec/1722170724986226", ] [TRACE][core/src/types/xvcroot.rs::263] xvc_candidate: "[CWD]/.xvc" [TRACE][core/src/types/xvcroot.rs::265] parent: "[CWD]" [TRACE][pipeline/src/pipeline/mod.rs::285] pipeline_e: XvcEntity( 1, - 4733698130976263725, + 13031353410878765396, ) [TRACE][pipeline/src/pipeline/mod.rs::290] pipeline_steps: HStore { map: { XvcEntity( 2, - 17595501963111217905, + 3920065547014771711, ): XvcStep { name: "average-age", }, @@ -114,14 +114,14 @@ $ xvc -vvvv pipeline run map: { XvcEntity( 2, - 17595501963111217905, + 3920065547014771711, ): ByDependencies, }, entity_index: { ByDependencies: [ XvcEntity( 2, - 17595501963111217905, + 3920065547014771711, ), ], }, @@ -130,7 +130,7 @@ $ xvc -vvvv pipeline run Add { entity: XvcEntity( 2, - 17595501963111217905, + 3920065547014771711, ), value: ByDependencies, }, @@ -815,14 +815,14 @@ $ xvc -vvvv pipeline run }, } [TRACE][walker/src/notify.rs::170] watcher: FsEventWatcher { - paths: 0x0000600000d70120, + paths: 0x000060000137c000, since_when: 18446744073709551615, latency: 0.0, flags: 18, - event_handler: 0x0000600002870010, + event_handler: 0x0000600003660010, runloop: Some( ( - 0x00006000036740c0, + 0x000060000286c0c0, JoinHandle { .. }, ), ), @@ -836,14 +836,14 @@ $ xvc -vvvv pipeline run .. } [TRACE][core/src/util/pmp.rs::59] watcher: FsEventWatcher { - paths: 0x0000600000d70120, + paths: 0x000060000137c000, since_when: 18446744073709551615, latency: 0.0, flags: 18, - event_handler: 0x0000600002870010, + event_handler: 0x0000600003660010, runloop: Some( ( - 0x00006000036740c0, + 0x000060000286c0c0, JoinHandle { .. }, ), ), @@ -876,148 +876,148 @@ $ xvc -vvvv pipeline run XvcConfigMap { source: Default, map: { - "pipeline.default": String( - "default", + "pipeline.process_pool_size": Integer( + 4, ), "git.command": String( "git", ), - "file.track.text_or_binary": String( - "auto", + "file.recheck.method": String( + "copy", + ), + "file.list.sort": String( + "name-desc", + ), + "file.track.no_commit": Boolean( + false, ), "core.verbosity": String( "error", ), - "core.guid": String( - "2563bb26e79eff77", - ), - "file.list.show_dot_files": Boolean( + "file.carry-in.no_parallel": Boolean( false, ), - "file.track.no_parallel": Boolean( + "file.track.force": Boolean( false, ), - "file.list.no_summary": Boolean( + "pipeline.default": String( + "default", + ), + "file.track.no_parallel": Boolean( false, ), - "git.auto_commit": Boolean( + "git.use_git": Boolean( true, ), "pipeline.current_pipeline": String( "default", ), - "file.recheck.method": String( - "copy", - ), - "pipeline.process_pool_size": Integer( - 4, - ), - "file.carry-in.no_parallel": Boolean( - false, - ), - "file.carry-in.force": Boolean( + "file.list.recursive": Boolean( false, ), - "file.track.force": Boolean( + "git.auto_stage": Boolean( false, ), "pipeline.default_params_file": String( "params.yaml", ), - "git.auto_stage": Boolean( + "git.auto_commit": Boolean( + true, + ), + "file.list.show_dot_files": Boolean( false, ), - "file.list.recursive": Boolean( + "core.guid": String( + "6dd0d3b5a9776de4", + ), + "file.carry-in.force": Boolean( + false, + ), + "file.list.no_summary": Boolean( false, ), "file.list.format": String( "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", ), - "file.track.no_commit": Boolean( - false, - ), "cache.algorithm": String( "blake3", ), - "git.use_git": Boolean( - true, - ), - "file.list.sort": String( - "name-desc", + "file.track.text_or_binary": String( + "auto", ), }, }, XvcConfigMap { source: Project, map: { - "core.verbosity": String( - "error", - ), - "git.auto_commit": Boolean( - true, - ), - "git.use_git": Boolean( - true, - ), - "file.track.no_parallel": Boolean( + "file.list.no_summary": Boolean( false, ), - "file.recheck.method": String( - "copy", + "pipeline.process_pool_size": Integer( + 4, ), "pipeline.default_params_file": String( "params.yaml", ), - "git.auto_stage": Boolean( - false, + "file.track.text_or_binary": String( + "auto", ), - "file.track.no_commit": Boolean( + "file.list.show_dot_files": Boolean( false, ), + "git.auto_commit": Boolean( + true, + ), "file.list.recursive": Boolean( false, ), - "pipeline.default": String( + "file.recheck.method": String( + "copy", + ), + "file.track.no_commit": Boolean( + false, + ), + "pipeline.current_pipeline": String( "default", ), "file.list.format": String( "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", ), - "file.list.show_dot_files": Boolean( + "git.command": String( + "git", + ), + "file.carry-in.no_parallel": Boolean( false, ), - "file.track.text_or_binary": String( - "auto", + "file.track.no_parallel": Boolean( + false, ), - "cache.algorithm": String( - "blake3", + "file.carry-in.force": Boolean( + false, + ), + "core.verbosity": String( + "error", ), "file.list.sort": String( "name-desc", ), "core.guid": String( - "fff3d263419bb95f", - ), - "pipeline.current_pipeline": String( - "default", + "6142ae8952816bc7", ), - "file.carry-in.no_parallel": Boolean( - false, + "cache.algorithm": String( + "blake3", ), - "file.carry-in.force": Boolean( + "file.track.force": Boolean( false, ), - "git.command": String( - "git", + "git.use_git": Boolean( + true, ), - "file.list.no_summary": Boolean( + "git.auto_stage": Boolean( false, ), - "pipeline.process_pool_size": Integer( - 4, - ), - "file.track.force": Boolean( - false, + "pipeline.default": String( + "default", ), }, }, @@ -1036,32 +1036,32 @@ $ xvc -vvvv pipeline run XvcConfigMap { source: CommandLine, map: { - "core.verbosity": String( - "debug", - ), "core.quiet": Boolean( false, ), + "core.verbosity": String( + "debug", + ), }, }, ], the_config: { - "pipeline.current_pipeline": XvcConfigValue { - source: Project, + "TRYCMD_TESTS": XvcConfigValue { + source: Environment, value: String( - "default", + "storage,file,pipeline,core", ), }, - "file.track.force": XvcConfigValue { + "file.list.recursive": XvcConfigValue { source: Project, value: Boolean( false, ), }, - "cache.algorithm": XvcConfigValue { + "file.list.no_summary": XvcConfigValue { source: Project, - value: String( - "blake3", + value: Boolean( + false, ), }, "git.use_git": XvcConfigValue { @@ -1070,31 +1070,37 @@ $ xvc -vvvv pipeline run true, ), }, - "file.recheck.method": XvcConfigValue { + "core.guid": XvcConfigValue { source: Project, value: String( - "copy", + "6142ae8952816bc7", ), }, - "file.track.text_or_binary": XvcConfigValue { + "file.recheck.method": XvcConfigValue { source: Project, value: String( - "auto", + "copy", ), }, - "core.guid": XvcConfigValue { + "pipeline.default": XvcConfigValue { source: Project, value: String( - "fff3d263419bb95f", + "default", ), }, - "pipeline.default_params_file": XvcConfigValue { + "pipeline.process_pool_size": XvcConfigValue { source: Project, + value: Integer( + 4, + ), + }, + "core.verbosity": XvcConfigValue { + source: CommandLine, value: String( - "params.yaml", + "debug", ), }, - "file.carry-in.force": XvcConfigValue { + "file.list.show_dot_files": XvcConfigValue { source: Project, value: Boolean( false, @@ -1106,10 +1112,10 @@ $ xvc -vvvv pipeline run "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", ), }, - "git.auto_stage": XvcConfigValue { + "file.track.text_or_binary": XvcConfigValue { source: Project, - value: Boolean( - false, + value: String( + "auto", ), }, "git.command": XvcConfigValue { @@ -1118,28 +1124,22 @@ $ xvc -vvvv pipeline run "git", ), }, - "file.list.recursive": XvcConfigValue { + "file.track.no_parallel": XvcConfigValue { source: Project, value: Boolean( false, ), }, - "core.quiet": XvcConfigValue { - source: CommandLine, - value: Boolean( - false, - ), - }, "file.carry-in.no_parallel": XvcConfigValue { source: Project, value: Boolean( false, ), }, - "TRYCMD_TESTS": XvcConfigValue { - source: Environment, + "pipeline.current_pipeline": XvcConfigValue { + source: Project, value: String( - "storage,file,pipeline,core", + "default", ), }, "git.auto_commit": XvcConfigValue { @@ -1148,46 +1148,40 @@ $ xvc -vvvv pipeline run true, ), }, - "file.list.sort": XvcConfigValue { - source: Project, - value: String( - "name-desc", - ), - }, - "file.track.no_parallel": XvcConfigValue { + "file.carry-in.force": XvcConfigValue { source: Project, value: Boolean( false, ), }, - "file.list.show_dot_files": XvcConfigValue { + "cache.algorithm": XvcConfigValue { source: Project, + value: String( + "blake3", + ), + }, + "core.quiet": XvcConfigValue { + source: CommandLine, value: Boolean( false, ), }, - "pipeline.default": XvcConfigValue { + "file.list.sort": XvcConfigValue { source: Project, value: String( - "default", - ), - }, - "core.verbosity": XvcConfigValue { - source: CommandLine, - value: String( - "debug", + "name-desc", ), }, - "file.list.no_summary": XvcConfigValue { + "git.auto_stage": XvcConfigValue { source: Project, value: Boolean( false, ), }, - "pipeline.process_pool_size": XvcConfigValue { + "pipeline.default_params_file": XvcConfigValue { source: Project, - value: Integer( - 4, + value: String( + "params.yaml", ), }, "file.track.no_commit": XvcConfigValue { @@ -1196,6 +1190,12 @@ $ xvc -vvvv pipeline run false, ), }, + "file.track.force": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, }, init_params: XvcConfigParams { default_configuration: " @@ -1203,7 +1203,7 @@ $ xvc -vvvv pipeline run # The repository id. Please do not delete or change it. # This is used to identify the repository and generate paths in storages. # In the future it may be used to in other ways. -guid = /"2563bb26e79eff77/" +guid = /"6dd0d3b5a9776de4/" # Default verbosity level. # One of /"error/", /"warn/", /"info/" verbosity = /"error/" @@ -1354,7 +1354,7 @@ process_pool_size = 4 ), entity_generator: XvcEntityGenerator { counter: 4, - random: 2745833803509197134, + random: 16503095231746129767, dirty: false, }, }, @@ -2021,18 +2021,18 @@ process_pool_size = 4 [TRACE][pipeline/src/pipeline/mod.rs::331] &dependency_graph: { XvcEntity( 2, - 17595501963111217905, + 3920065547014771711, ): [], } [TRACE][pipeline/src/pipeline/mod.rs::343] &dependency_graph: { XvcEntity( 2, - 17595501963111217905, + 3920065547014771711, ): [], } [INFO][pipeline/src/pipeline/mod.rs::347] Pipeline Graph: digraph { - 0 [ label = "(2, 17595501963111217905)" ] + 0 [ label = "(2, 3920065547014771711)" ] } @@ -2041,7 +2041,7 @@ digraph { map: { XvcEntity( 2, - 17595501963111217905, + 3920065547014771711, ): Begin( FromInit, ), @@ -2054,24 +2054,23 @@ digraph { map: { XvcEntity( 2, - 17595501963111217905, + 3920065547014771711, ): ScopedJoinHandle { .. }, }, } [TRACE][pipeline/src/pipeline/mod.rs::512] (step_e, &jh): ( XvcEntity( 2, - 17595501963111217905, + 3920065547014771711, ), ScopedJoinHandle { .. }, ) -[TRACE][pipeline/src/pipeline/mod.rs::583] select: Select { .. } -[TRACE][pipeline/src/pipeline/mod.rs::623] params.recorded_dependencies: R1NStore { +[TRACE][pipeline/src/pipeline/mod.rs::621] params.recorded_dependencies: R1NStore { parents: XvcStore { map: { XvcEntity( 2, - 17595501963111217905, + 3920065547014771711, ): XvcStep { name: "average-age", }, @@ -2082,7 +2081,7 @@ digraph { }: [ XvcEntity( 2, - 17595501963111217905, + 3920065547014771711, ), ], }, @@ -2091,7 +2090,7 @@ digraph { Add { entity: XvcEntity( 2, - 17595501963111217905, + 3920065547014771711, ), value: XvcStep { name: "average-age", @@ -2100,7 +2099,7 @@ digraph { Add { entity: XvcEntity( 2, - 17595501963111217905, + 3920065547014771711, ), value: XvcStep { name: "average-age", @@ -2116,7 +2115,7 @@ digraph { map: { XvcEntity( 3, - 15601036540397786949, + 324224710907113891, ): SqliteQueryDigest( SqliteQueryDep { path: XvcPath( @@ -2141,7 +2140,7 @@ digraph { ): [ XvcEntity( 3, - 15601036540397786949, + 324224710907113891, ), ], }, @@ -2150,7 +2149,7 @@ digraph { Add { entity: XvcEntity( 3, - 15601036540397786949, + 324224710907113891, ), value: SqliteQueryDigest( SqliteQueryDep { @@ -2173,11 +2172,11 @@ digraph { map: { XvcEntity( 3, - 15601036540397786949, + 324224710907113891, ): ChildEntity( XvcEntity( 2, - 17595501963111217905, + 3920065547014771711, ), PhantomData, PhantomData, @@ -2187,14 +2186,14 @@ digraph { ChildEntity( XvcEntity( 2, - 17595501963111217905, + 3920065547014771711, ), PhantomData, PhantomData, ): [ XvcEntity( 3, - 15601036540397786949, + 324224710907113891, ), ], }, @@ -2203,12 +2202,12 @@ digraph { Add { entity: XvcEntity( 3, - 15601036540397786949, + 324224710907113891, ), value: ChildEntity( XvcEntity( 2, - 17595501963111217905, + 3920065547014771711, ), PhantomData, PhantomData, @@ -2221,9 +2220,9 @@ digraph { ), }, } -[TRACE][pipeline/src/pipeline/mod.rs::624] step_e: XvcEntity( +[TRACE][pipeline/src/pipeline/mod.rs::622] step_e: XvcEntity( 2, - 17595501963111217905, + 3920065547014771711, ) [TRACE][pipeline/src/pipeline/mod.rs::564] dep_neighbors: Neighbors { iter: Iter( @@ -2231,56 +2230,56 @@ digraph { ), ty: PhantomData, } -[TRACE][pipeline/src/pipeline/mod.rs::625] dependency_steps(step_e, params.dependency_graph)?: {} +[TRACE][pipeline/src/pipeline/mod.rs::623] dependency_steps(step_e, params.dependency_graph)?: {} [TRACE][pipeline/src/pipeline/mod.rs::564] dep_neighbors: Neighbors { iter: Iter( [], ), ty: PhantomData, } -[TRACE][pipeline/src/pipeline/mod.rs::662] &step_state: Begin( +[TRACE][pipeline/src/pipeline/mod.rs::660] &step_state: Begin( FromInit, ) -[TRACE][pipeline/src/pipeline/mod.rs::772] step.name: "average-age" -[TRACE][pipeline/src/pipeline/mod.rs::773] &r_next_state: WaitingDependencySteps( +[TRACE][pipeline/src/pipeline/mod.rs::770] step.name: "average-age" +[TRACE][pipeline/src/pipeline/mod.rs::771] &r_next_state: WaitingDependencySteps( FromRunConditional, ) -[TRACE][pipeline/src/pipeline/mod.rs::775] &step_state: WaitingDependencySteps( +[TRACE][pipeline/src/pipeline/mod.rs::773] &step_state: WaitingDependencySteps( FromRunConditional, ) -[TRACE][pipeline/src/pipeline/mod.rs::662] &step_state: WaitingDependencySteps( +[TRACE][pipeline/src/pipeline/mod.rs::660] &step_state: WaitingDependencySteps( FromRunConditional, ) -[TRACE][pipeline/src/pipeline/mod.rs::772] step.name: "average-age" -[TRACE][pipeline/src/pipeline/mod.rs::773] &r_next_state: CheckingOutputs( +[TRACE][pipeline/src/pipeline/mod.rs::770] step.name: "average-age" +[TRACE][pipeline/src/pipeline/mod.rs::771] &r_next_state: CheckingOutputs( FromDependencyStepsFinishedSuccessfully, ) -[TRACE][pipeline/src/pipeline/mod.rs::775] &step_state: CheckingOutputs( +[TRACE][pipeline/src/pipeline/mod.rs::773] &step_state: CheckingOutputs( FromDependencyStepsFinishedSuccessfully, ) -[TRACE][pipeline/src/pipeline/mod.rs::662] &step_state: CheckingOutputs( +[TRACE][pipeline/src/pipeline/mod.rs::660] &step_state: CheckingOutputs( FromDependencyStepsFinishedSuccessfully, ) -[TRACE][pipeline/src/pipeline/mod.rs::772] step.name: "average-age" -[TRACE][pipeline/src/pipeline/mod.rs::773] &r_next_state: CheckingSuperficialDiffs( +[TRACE][pipeline/src/pipeline/mod.rs::770] step.name: "average-age" +[TRACE][pipeline/src/pipeline/mod.rs::771] &r_next_state: CheckingSuperficialDiffs( FromCheckedOutputs, ) -[TRACE][pipeline/src/pipeline/mod.rs::775] &step_state: CheckingSuperficialDiffs( +[TRACE][pipeline/src/pipeline/mod.rs::773] &step_state: CheckingSuperficialDiffs( FromCheckedOutputs, ) -[TRACE][pipeline/src/pipeline/mod.rs::662] &step_state: CheckingSuperficialDiffs( +[TRACE][pipeline/src/pipeline/mod.rs::660] &step_state: CheckingSuperficialDiffs( FromCheckedOutputs, ) -[TRACE][pipeline/src/pipeline/mod.rs::1010] parent_entity: XvcEntity( +[TRACE][pipeline/src/pipeline/mod.rs::1008] parent_entity: XvcEntity( 2, - 17595501963111217905, + 3920065547014771711, ) -[TRACE][pipeline/src/pipeline/mod.rs::583] select: Select { .. } -[TRACE][pipeline/src/pipeline/mod.rs::1013] deps: HStore { +[INFO] No dependency steps for step average-age +[TRACE][pipeline/src/pipeline/mod.rs::1011] deps: HStore { map: { XvcEntity( 3, - 15601036540397786949, + 324224710907113891, ): SqliteQueryDigest( SqliteQueryDep { path: XvcPath( @@ -2312,7 +2311,6 @@ digraph { [TRACE][core/src/util/pmp.rs::173] path: AbsolutePath( "[CWD]/people.db", ) -[TRACE][pipeline/src/pipeline/mod.rs::583] select: Select { .. } [TRACE][core/src/util/pmp.rs::175] &md: Ok( Metadata { file_type: FileType( @@ -2348,8 +2346,6 @@ digraph { .. }, ) -[TRACE][pipeline/src/pipeline/mod.rs::583] select: Select { .. } -[TRACE][pipeline/src/pipeline/mod.rs::583] select: Select { .. } [TRACE][core/src/util/pmp.rs::152] &md: Some( XvcMetadata { file_type: File, @@ -2364,11 +2360,11 @@ digraph { ), }, ) -[TRACE][pipeline/src/pipeline/mod.rs::1030] step_dependency_diffs: HStore { +[TRACE][pipeline/src/pipeline/mod.rs::1028] step_dependency_diffs: HStore { map: { XvcEntity( 3, - 15601036540397786949, + 324224710907113891, ): RecordMissing { actual: SqliteQueryDigest( SqliteQueryDep { @@ -2396,7 +2392,7 @@ digraph { }, }, } -[TRACE][pipeline/src/pipeline/mod.rs::1036] diff: RecordMissing { +[TRACE][pipeline/src/pipeline/mod.rs::1034] diff: RecordMissing { actual: SqliteQueryDigest( SqliteQueryDep { path: XvcPath( @@ -2421,24 +2417,23 @@ digraph { }, ), } -[INFO] No dependency steps for step average-age -[TRACE][pipeline/src/pipeline/mod.rs::1037] diff.changed(): true -[TRACE][pipeline/src/pipeline/mod.rs::1042] changed: true -[TRACE][pipeline/src/pipeline/mod.rs::772] step.name: "average-age" -[TRACE][pipeline/src/pipeline/mod.rs::773] &r_next_state: CheckingThoroughDiffs( +[TRACE][pipeline/src/pipeline/mod.rs::1035] diff.changed(): true +[TRACE][pipeline/src/pipeline/mod.rs::1040] changed: true +[TRACE][pipeline/src/pipeline/mod.rs::770] step.name: "average-age" +[TRACE][pipeline/src/pipeline/mod.rs::771] &r_next_state: CheckingThoroughDiffs( FromSuperficialDiffsChanged, ) -[TRACE][pipeline/src/pipeline/mod.rs::775] &step_state: CheckingThoroughDiffs( +[TRACE][pipeline/src/pipeline/mod.rs::773] &step_state: CheckingThoroughDiffs( FromSuperficialDiffsChanged, ) -[TRACE][pipeline/src/pipeline/mod.rs::662] &step_state: CheckingThoroughDiffs( +[TRACE][pipeline/src/pipeline/mod.rs::660] &step_state: CheckingThoroughDiffs( FromSuperficialDiffsChanged, ) -[TRACE][pipeline/src/pipeline/mod.rs::1067] deps: HStore { +[TRACE][pipeline/src/pipeline/mod.rs::1065] deps: HStore { map: { XvcEntity( 3, - 15601036540397786949, + 324224710907113891, ): SqliteQueryDigest( SqliteQueryDep { path: XvcPath( @@ -2468,49 +2463,304 @@ digraph { ), }, ) -[TRACE][pipeline/src/pipeline/mod.rs::583] select: Select { .. } -thread '' panicked at pipeline/src/pipeline/mod.rs:1081:32: +[TRACE][pipeline/src/pipeline/deps/sqlite_query.rs::71] col: Integer( + 18, +) +thread '' panicked at pipeline/src/pipeline/mod.rs:1079:32: SqliteError { source: FromSqlConversionFailure(18446744073709551615, Null, InvalidType) } note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace thread '' panicked at lib/src/cli/mod.rs:288:52: -[PANIC] SqliteError { source: FromSqlConversionFailure(18446744073709551615, Null, InvalidType) }, [pipeline/src/pipeline/mod.rs::1081] -[TRACE][pipeline/src/pipeline/mod.rs::519] "Before state updater": "Before state updater" -[TRACE][pipeline/src/pipeline/mod.rs::593] s: CheckingThoroughDiffs( - FromSuperficialDiffsChanged, -) -[TRACE][pipeline/src/pipeline/mod.rs::532] step_states: RwLock { - data: HStore { - map: { - XvcEntity( - 2, - 17595501963111217905, - ): CheckingThoroughDiffs( - FromSuperficialDiffsChanged, - ), - }, - }, - poisoned: false, - .. -} -[TRACE][pipeline/src/pipeline/mod.rs::536] done_successfully: Ok( - false, -) -[TRACE][core/src/util/pmp.rs::185] self.background_thread: Mutex { - data: JoinHandle { .. }, - poisoned: false, - .. -} -[TRACE][core/src/util/pmp.rs::190] self.background_thread: Mutex { - data: JoinHandle { .. }, - poisoned: false, - .. -} -[TRACE][lib/src/cli/mod.rs::376] "Before handle_git_automation": "Before handle_git_automation" -[TRACE][core/src/util/pmp.rs::99] index: 1 -thread '' panicked at lib/src/git.rs:155:5: +[PANIC] SqliteError { source: FromSqlConversionFailure(18446744073709551615, Null, InvalidType) }, [pipeline/src/pipeline/mod.rs::1079] +thread '' panicked at pipeline/src/pipeline/mod.rs:514:17: called `Result::unwrap()` on an `Err` value: "SendError(..)" -thread 'main' panicked at lib/src/cli/mod.rs:402:52: -called `Result::unwrap()` on an `Err` value: Any { .. } +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" ``` @@ -2519,11 +2769,11 @@ But it won't run the step a second time, as the table didn't change. ```console $ xvc pipeline run ? interrupted -thread '' panicked at pipeline/src/pipeline/mod.rs:1081:32: +thread '' panicked at pipeline/src/pipeline/mod.rs:1079:32: SqliteError { source: FromSqlConversionFailure(18446744073709551615, Null, InvalidType) } -note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace -thread '' panicked at lib/src/cli/mod.rs:251:52: -[PANIC] SqliteError { source: FromSqlConversionFailure(18446744073709551615, Null, InvalidType) }, [pipeline/src/pipeline/mod.rs::1081] +thread 'note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace +' panicked at lib/src/cli/mod.rs:251:52: +[PANIC] SqliteError { source: FromSqlConversionFailure(18446744073709551615, Null, InvalidType) }, [pipeline/src/pipeline/mod.rs::1079] thread '' panicked at pipeline/src/pipeline/mod.rs:514:17: called `Result::unwrap()` on an `Err` value: "SendError(..)" @@ -2540,11 +2790,11 @@ This time, the step will run again as the result from dependency query (`SELECT ```console $ xvc pipeline run ? interrupted -thread '' panicked at pipeline/src/pipeline/mod.rs:1081:32: +thread '' panicked at pipeline/src/pipeline/mod.rs:1079:32: SqliteError { source: FromSqlConversionFailure(18446744073709551615, Null, InvalidType) } note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace thread '' panicked at lib/src/cli/mod.rs:251:52: -[PANIC] SqliteError { source: FromSqlConversionFailure(18446744073709551615, Null, InvalidType) }, [pipeline/src/pipeline/mod.rs::1081] +[PANIC] SqliteError { source: FromSqlConversionFailure(18446744073709551615, Null, InvalidType) }, [pipeline/src/pipeline/mod.rs::1079] thread '' panicked at pipeline/src/pipeline/mod.rs:514:17: called `Result::unwrap()` on an `Err` value: "SendError(..)" diff --git a/pipeline/src/pipeline/deps/sqlite_query.rs b/pipeline/src/pipeline/deps/sqlite_query.rs index 376238626..14a5a3b67 100644 --- a/pipeline/src/pipeline/deps/sqlite_query.rs +++ b/pipeline/src/pipeline/deps/sqlite_query.rs @@ -69,7 +69,20 @@ impl SqliteQueryDep { let mut els = String::new(); while let Ok(col) = row.get_ref(i) { watch!(col); - els.push_str(col.as_str()?); + match col.data_type() { + rusqlite::types::Type::Text => { + els.push_str(col.as_str()?); + } + rusqlite::types::Type::Integer => { + els.push_str(col.as_i64()?.to_string().as_str()); + } + rusqlite::types::Type::Real => { + els.push_str(col.as_f64()?.to_string().as_str()); + } + _ => { + els.push_str(col.as_str()?); + } + } i += 1; } Ok(els) From d063ad1b5f4c065af44cc47efb675d25236010c5 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sun, 28 Jul 2024 15:54:02 +0300 Subject: [PATCH 080/257] Debug off --- ...c-pipeline-step-dependency-sqlite-query.md | 6920 +++++++++++++++-- 1 file changed, 6423 insertions(+), 497 deletions(-) diff --git a/book/src/ref/xvc-pipeline-step-dependency-sqlite-query.md b/book/src/ref/xvc-pipeline-step-dependency-sqlite-query.md index 1fdae75dc..7f4fe94a4 100644 --- a/book/src/ref/xvc-pipeline-step-dependency-sqlite-query.md +++ b/book/src/ref/xvc-pipeline-step-dependency-sqlite-query.md @@ -80,8 +80,7 @@ The dependency query is run everytime the pipeline runs. It's expected to be lig So, when the number of people in the table changes, the step will run. Initially it doesn't keep track of the query results, so it will run again. ```console -$ xvc -vvvv pipeline run -? interrupted +$ xvc pipeline run [DEBUG][logging/src/lib.rs::237] Terminal logger enabled with level: Trace [TRACE][core/src/types/xvcroot.rs::263] xvc_candidate: "[CWD]/.xvc" [TRACE][core/src/types/xvcroot.rs::265] parent: "[CWD]" @@ -89,22 +88,22 @@ $ xvc -vvvv pipeline run [DEBUG][config/src/error.rs::72] Config source for level "global" not found at "/Users/iex/Library/Application Support/xvc" [TRACE][ecs/src/ecs/mod.rs::229] dir: "[CWD]/.xvc/ec" [TRACE][ecs/src/ecs/mod.rs::239] files: [ - "[CWD]/.xvc/ec/1722170724738869", - "[CWD]/.xvc/ec/1722170724741681", - "[CWD]/.xvc/ec/1722170724810551", - "[CWD]/.xvc/ec/1722170724986226", + "[CWD]/.xvc/ec/1722171155879620", + "[CWD]/.xvc/ec/1722171155882390", + "[CWD]/.xvc/ec/1722171155967050", + "[CWD]/.xvc/ec/1722171156186854", ] [TRACE][core/src/types/xvcroot.rs::263] xvc_candidate: "[CWD]/.xvc" [TRACE][core/src/types/xvcroot.rs::265] parent: "[CWD]" [TRACE][pipeline/src/pipeline/mod.rs::285] pipeline_e: XvcEntity( 1, - 13031353410878765396, + 1872241824668543094, ) [TRACE][pipeline/src/pipeline/mod.rs::290] pipeline_steps: HStore { map: { XvcEntity( 2, - 3920065547014771711, + 9270302598983016314, ): XvcStep { name: "average-age", }, @@ -114,14 +113,14 @@ $ xvc -vvvv pipeline run map: { XvcEntity( 2, - 3920065547014771711, + 9270302598983016314, ): ByDependencies, }, entity_index: { ByDependencies: [ XvcEntity( 2, - 3920065547014771711, + 9270302598983016314, ), ], }, @@ -130,7 +129,7 @@ $ xvc -vvvv pipeline run Add { entity: XvcEntity( 2, - 3920065547014771711, + 9270302598983016314, ), value: ByDependencies, }, @@ -470,14 +469,14 @@ $ xvc -vvvv pipeline run pre: None, nfa: thompson::NFA( ^000000: FAIL - + transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) ) , nfarev: Some( thompson::NFA( ^000000: FAIL - + transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) ) , @@ -495,7 +494,7 @@ $ xvc -vvvv pipeline run }, nfa: thompson::NFA( ^000000: FAIL - + transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) ) , @@ -552,12 +551,12 @@ $ xvc -vvvv pipeline run }, nfa: thompson::NFA( ^000000: FAIL - + transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) ) , stride2: 1, - start_map: StartByteMap{/x00 => NonWordByte, /x01 => NonWordByte, /x02 => NonWordByte, /x03 => NonWordByte, /x04 => NonWordByte, /x05 => NonWordByte, /x06 => NonWordByte, /x07 => NonWordByte, /x08 => NonWordByte, /t => NonWordByte, + start_map: StartByteMap{/x00 => NonWordByte, /x01 => NonWordByte, /x02 => NonWordByte, /x03 => NonWordByte, /x04 => NonWordByte, /x05 => NonWordByte, /x06 => NonWordByte, /x07 => NonWordByte, /x08 => NonWordByte, /t => NonWordByte, => LineLF, /x0B => NonWordByte, /x0C => NonWordByte, /r => LineCR, /x0E => NonWordByte, /x0F => NonWordByte, /x10 => NonWordByte, /x11 => NonWordByte, /x12 => NonWordByte, /x13 => NonWordByte, /x14 => NonWordByte, /x15 => NonWordByte, /x16 => NonWordByte, /x17 => NonWordByte, /x18 => NonWordByte, /x19 => NonWordByte, /x1A => NonWordByte, /x1B => NonWordByte, /x1C => NonWordByte, /x1D => NonWordByte, /x1E => NonWordByte, /x1F => NonWordByte, ' ' => NonWordByte, ! => NonWordByte, /" => NonWordByte, # => NonWordByte, $ => NonWordByte, % => NonWordByte, & => NonWordByte, /' => NonWordByte, ( => NonWordByte, ) => NonWordByte, * => NonWordByte, + => NonWordByte, , => NonWordByte, - => NonWordByte, . => NonWordByte, / => NonWordByte, 0 => WordByte, 1 => WordByte, 2 => WordByte, 3 => WordByte, 4 => WordByte, 5 => WordByte, 6 => WordByte, 7 => WordByte, 8 => WordByte, 9 => WordByte, : => NonWordByte, ; => NonWordByte, < => NonWordByte, = => NonWordByte, > => NonWordByte, ? => NonWordByte, @ => NonWordByte, A => WordByte, B => WordByte, C => WordByte, D => WordByte, E => WordByte, F => WordByte, G => WordByte, H => WordByte, I => WordByte, J => WordByte, K => WordByte, L => WordByte, M => WordByte, N => WordByte, O => WordByte, P => WordByte, Q => WordByte, R => WordByte, S => WordByte, T => WordByte, U => WordByte, V => WordByte, W => WordByte, X => WordByte, Y => WordByte, Z => WordByte, [ => NonWordByte, // => NonWordByte, ] => NonWordByte, ^ => NonWordByte, _ => WordByte, ` => NonWordByte, a => WordByte, b => WordByte, c => WordByte, d => WordByte, e => WordByte, f => WordByte, g => WordByte, h => WordByte, i => WordByte, j => WordByte, k => WordByte, l => WordByte, m => WordByte, n => WordByte, o => WordByte, p => WordByte, q => WordByte, r => WordByte, s => WordByte, t => WordByte, u => WordByte, v => WordByte, w => WordByte, x => WordByte, y => WordByte, z => WordByte, { => NonWordByte, | => NonWordByte, } => NonWordByte, ~ => NonWordByte, /x7F => NonWordByte, /x80 => NonWordByte, /x81 => NonWordByte, /x82 => NonWordByte, /x83 => NonWordByte, /x84 => NonWordByte, /x85 => NonWordByte, /x86 => NonWordByte, /x87 => NonWordByte, /x88 => NonWordByte, /x89 => NonWordByte, /x8A => NonWordByte, /x8B => NonWordByte, /x8C => NonWordByte, /x8D => NonWordByte, /x8E => NonWordByte, /x8F => NonWordByte, /x90 => NonWordByte, /x91 => NonWordByte, /x92 => NonWordByte, /x93 => NonWordByte, /x94 => NonWordByte, /x95 => NonWordByte, /x96 => NonWordByte, /x97 => NonWordByte, /x98 => NonWordByte, /x99 => NonWordByte, /x9A => NonWordByte, /x9B => NonWordByte, /x9C => NonWordByte, /x9D => NonWordByte, /x9E => NonWordByte, /x9F => NonWordByte, /xA0 => NonWordByte, /xA1 => NonWordByte, /xA2 => NonWordByte, /xA3 => NonWordByte, /xA4 => NonWordByte, /xA5 => NonWordByte, /xA6 => NonWordByte, /xA7 => NonWordByte, /xA8 => NonWordByte, /xA9 => NonWordByte, /xAA => NonWordByte, /xAB => NonWordByte, /xAC => NonWordByte, /xAD => NonWordByte, /xAE => NonWordByte, /xAF => NonWordByte, /xB0 => NonWordByte, /xB1 => NonWordByte, /xB2 => NonWordByte, /xB3 => NonWordByte, /xB4 => NonWordByte, /xB5 => NonWordByte, /xB6 => NonWordByte, /xB7 => NonWordByte, /xB8 => NonWordByte, /xB9 => NonWordByte, /xBA => NonWordByte, /xBB => NonWordByte, /xBC => NonWordByte, /xBD => NonWordByte, /xBE => NonWordByte, /xBF => NonWordByte, /xC0 => NonWordByte, /xC1 => NonWordByte, /xC2 => NonWordByte, /xC3 => NonWordByte, /xC4 => NonWordByte, /xC5 => NonWordByte, /xC6 => NonWordByte, /xC7 => NonWordByte, /xC8 => NonWordByte, /xC9 => NonWordByte, /xCA => NonWordByte, /xCB => NonWordByte, /xCC => NonWordByte, /xCD => NonWordByte, /xCE => NonWordByte, /xCF => NonWordByte, /xD0 => NonWordByte, /xD1 => NonWordByte, /xD2 => NonWordByte, /xD3 => NonWordByte, /xD4 => NonWordByte, /xD5 => NonWordByte, /xD6 => NonWordByte, /xD7 => NonWordByte, /xD8 => NonWordByte, /xD9 => NonWordByte, /xDA => NonWordByte, /xDB => NonWordByte, /xDC => NonWordByte, /xDD => NonWordByte, /xDE => NonWordByte, /xDF => NonWordByte, /xE0 => NonWordByte, /xE1 => NonWordByte, /xE2 => NonWordByte, /xE3 => NonWordByte, /xE4 => NonWordByte, /xE5 => NonWordByte, /xE6 => NonWordByte, /xE7 => NonWordByte, /xE8 => NonWordByte, /xE9 => NonWordByte, /xEA => NonWordByte, /xEB => NonWordByte, /xEC => NonWordByte, /xED => NonWordByte, /xEE => NonWordByte, /xEF => NonWordByte, /xF0 => NonWordByte, /xF1 => NonWordByte, /xF2 => NonWordByte, /xF3 => NonWordByte, /xF4 => NonWordByte, /xF5 => NonWordByte, /xF6 => NonWordByte, /xF7 => NonWordByte, /xF8 => NonWordByte, /xF9 => NonWordByte, /xFA => NonWordByte, /xFB => NonWordByte, /xFC => NonWordByte, /xFD => NonWordByte, /xFE => NonWordByte, /xFF => NonWordByte}, classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]), quitset: ByteSet { @@ -605,12 +604,12 @@ $ xvc -vvvv pipeline run }, nfa: thompson::NFA( ^000000: FAIL - + transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) ) , stride2: 1, - start_map: StartByteMap{/x00 => NonWordByte, /x01 => NonWordByte, /x02 => NonWordByte, /x03 => NonWordByte, /x04 => NonWordByte, /x05 => NonWordByte, /x06 => NonWordByte, /x07 => NonWordByte, /x08 => NonWordByte, /t => NonWordByte, + start_map: StartByteMap{/x00 => NonWordByte, /x01 => NonWordByte, /x02 => NonWordByte, /x03 => NonWordByte, /x04 => NonWordByte, /x05 => NonWordByte, /x06 => NonWordByte, /x07 => NonWordByte, /x08 => NonWordByte, /t => NonWordByte, => LineLF, /x0B => NonWordByte, /x0C => NonWordByte, /r => LineCR, /x0E => NonWordByte, /x0F => NonWordByte, /x10 => NonWordByte, /x11 => NonWordByte, /x12 => NonWordByte, /x13 => NonWordByte, /x14 => NonWordByte, /x15 => NonWordByte, /x16 => NonWordByte, /x17 => NonWordByte, /x18 => NonWordByte, /x19 => NonWordByte, /x1A => NonWordByte, /x1B => NonWordByte, /x1C => NonWordByte, /x1D => NonWordByte, /x1E => NonWordByte, /x1F => NonWordByte, ' ' => NonWordByte, ! => NonWordByte, /" => NonWordByte, # => NonWordByte, $ => NonWordByte, % => NonWordByte, & => NonWordByte, /' => NonWordByte, ( => NonWordByte, ) => NonWordByte, * => NonWordByte, + => NonWordByte, , => NonWordByte, - => NonWordByte, . => NonWordByte, / => NonWordByte, 0 => WordByte, 1 => WordByte, 2 => WordByte, 3 => WordByte, 4 => WordByte, 5 => WordByte, 6 => WordByte, 7 => WordByte, 8 => WordByte, 9 => WordByte, : => NonWordByte, ; => NonWordByte, < => NonWordByte, = => NonWordByte, > => NonWordByte, ? => NonWordByte, @ => NonWordByte, A => WordByte, B => WordByte, C => WordByte, D => WordByte, E => WordByte, F => WordByte, G => WordByte, H => WordByte, I => WordByte, J => WordByte, K => WordByte, L => WordByte, M => WordByte, N => WordByte, O => WordByte, P => WordByte, Q => WordByte, R => WordByte, S => WordByte, T => WordByte, U => WordByte, V => WordByte, W => WordByte, X => WordByte, Y => WordByte, Z => WordByte, [ => NonWordByte, // => NonWordByte, ] => NonWordByte, ^ => NonWordByte, _ => WordByte, ` => NonWordByte, a => WordByte, b => WordByte, c => WordByte, d => WordByte, e => WordByte, f => WordByte, g => WordByte, h => WordByte, i => WordByte, j => WordByte, k => WordByte, l => WordByte, m => WordByte, n => WordByte, o => WordByte, p => WordByte, q => WordByte, r => WordByte, s => WordByte, t => WordByte, u => WordByte, v => WordByte, w => WordByte, x => WordByte, y => WordByte, z => WordByte, { => NonWordByte, | => NonWordByte, } => NonWordByte, ~ => NonWordByte, /x7F => NonWordByte, /x80 => NonWordByte, /x81 => NonWordByte, /x82 => NonWordByte, /x83 => NonWordByte, /x84 => NonWordByte, /x85 => NonWordByte, /x86 => NonWordByte, /x87 => NonWordByte, /x88 => NonWordByte, /x89 => NonWordByte, /x8A => NonWordByte, /x8B => NonWordByte, /x8C => NonWordByte, /x8D => NonWordByte, /x8E => NonWordByte, /x8F => NonWordByte, /x90 => NonWordByte, /x91 => NonWordByte, /x92 => NonWordByte, /x93 => NonWordByte, /x94 => NonWordByte, /x95 => NonWordByte, /x96 => NonWordByte, /x97 => NonWordByte, /x98 => NonWordByte, /x99 => NonWordByte, /x9A => NonWordByte, /x9B => NonWordByte, /x9C => NonWordByte, /x9D => NonWordByte, /x9E => NonWordByte, /x9F => NonWordByte, /xA0 => NonWordByte, /xA1 => NonWordByte, /xA2 => NonWordByte, /xA3 => NonWordByte, /xA4 => NonWordByte, /xA5 => NonWordByte, /xA6 => NonWordByte, /xA7 => NonWordByte, /xA8 => NonWordByte, /xA9 => NonWordByte, /xAA => NonWordByte, /xAB => NonWordByte, /xAC => NonWordByte, /xAD => NonWordByte, /xAE => NonWordByte, /xAF => NonWordByte, /xB0 => NonWordByte, /xB1 => NonWordByte, /xB2 => NonWordByte, /xB3 => NonWordByte, /xB4 => NonWordByte, /xB5 => NonWordByte, /xB6 => NonWordByte, /xB7 => NonWordByte, /xB8 => NonWordByte, /xB9 => NonWordByte, /xBA => NonWordByte, /xBB => NonWordByte, /xBC => NonWordByte, /xBD => NonWordByte, /xBE => NonWordByte, /xBF => NonWordByte, /xC0 => NonWordByte, /xC1 => NonWordByte, /xC2 => NonWordByte, /xC3 => NonWordByte, /xC4 => NonWordByte, /xC5 => NonWordByte, /xC6 => NonWordByte, /xC7 => NonWordByte, /xC8 => NonWordByte, /xC9 => NonWordByte, /xCA => NonWordByte, /xCB => NonWordByte, /xCC => NonWordByte, /xCD => NonWordByte, /xCE => NonWordByte, /xCF => NonWordByte, /xD0 => NonWordByte, /xD1 => NonWordByte, /xD2 => NonWordByte, /xD3 => NonWordByte, /xD4 => NonWordByte, /xD5 => NonWordByte, /xD6 => NonWordByte, /xD7 => NonWordByte, /xD8 => NonWordByte, /xD9 => NonWordByte, /xDA => NonWordByte, /xDB => NonWordByte, /xDC => NonWordByte, /xDD => NonWordByte, /xDE => NonWordByte, /xDF => NonWordByte, /xE0 => NonWordByte, /xE1 => NonWordByte, /xE2 => NonWordByte, /xE3 => NonWordByte, /xE4 => NonWordByte, /xE5 => NonWordByte, /xE6 => NonWordByte, /xE7 => NonWordByte, /xE8 => NonWordByte, /xE9 => NonWordByte, /xEA => NonWordByte, /xEB => NonWordByte, /xEC => NonWordByte, /xED => NonWordByte, /xEE => NonWordByte, /xEF => NonWordByte, /xF0 => NonWordByte, /xF1 => NonWordByte, /xF2 => NonWordByte, /xF3 => NonWordByte, /xF4 => NonWordByte, /xF5 => NonWordByte, /xF6 => NonWordByte, /xF7 => NonWordByte, /xF8 => NonWordByte, /xF9 => NonWordByte, /xFA => NonWordByte, /xFB => NonWordByte, /xFC => NonWordByte, /xFD => NonWordByte, /xFE => NonWordByte, /xFF => NonWordByte}, classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]), quitset: ByteSet { @@ -815,14 +814,14 @@ $ xvc -vvvv pipeline run }, } [TRACE][walker/src/notify.rs::170] watcher: FsEventWatcher { - paths: 0x000060000137c000, + paths: 0x0000600001ae8000, since_when: 18446744073709551615, latency: 0.0, flags: 18, - event_handler: 0x0000600003660010, + event_handler: 0x0000600003ff0010, runloop: Some( ( - 0x000060000286c0c0, + 0x00006000021fc0c0, JoinHandle { .. }, ), ), @@ -836,14 +835,14 @@ $ xvc -vvvv pipeline run .. } [TRACE][core/src/util/pmp.rs::59] watcher: FsEventWatcher { - paths: 0x000060000137c000, + paths: 0x0000600001ae8000, since_when: 18446744073709551615, latency: 0.0, flags: 18, - event_handler: 0x0000600003660010, + event_handler: 0x0000600003ff0010, runloop: Some( ( - 0x000060000286c0c0, + 0x00006000021fc0c0, JoinHandle { .. }, ), ), @@ -876,35 +875,23 @@ $ xvc -vvvv pipeline run XvcConfigMap { source: Default, map: { - "pipeline.process_pool_size": Integer( - 4, - ), - "git.command": String( - "git", - ), - "file.recheck.method": String( - "copy", + "pipeline.default_params_file": String( + "params.yaml", ), - "file.list.sort": String( - "name-desc", + "file.carry-in.no_parallel": Boolean( + false, ), - "file.track.no_commit": Boolean( + "file.carry-in.force": Boolean( false, ), "core.verbosity": String( "error", ), - "file.carry-in.no_parallel": Boolean( - false, - ), - "file.track.force": Boolean( - false, - ), - "pipeline.default": String( - "default", + "file.recheck.method": String( + "copy", ), - "file.track.no_parallel": Boolean( - false, + "file.list.sort": String( + "name-desc", ), "git.use_git": Boolean( true, @@ -912,28 +899,31 @@ $ xvc -vvvv pipeline run "pipeline.current_pipeline": String( "default", ), - "file.list.recursive": Boolean( - false, - ), - "git.auto_stage": Boolean( - false, - ), - "pipeline.default_params_file": String( - "params.yaml", + "core.guid": String( + "919ce8649f3e32f7", ), "git.auto_commit": Boolean( true, ), - "file.list.show_dot_files": Boolean( + "pipeline.default": String( + "default", + ), + "pipeline.process_pool_size": Integer( + 4, + ), + "git.command": String( + "git", + ), + "file.track.no_commit": Boolean( false, ), - "core.guid": String( - "6dd0d3b5a9776de4", + "file.track.force": Boolean( + false, ), - "file.carry-in.force": Boolean( + "file.list.show_dot_files": Boolean( false, ), - "file.list.no_summary": Boolean( + "file.list.recursive": Boolean( false, ), "file.list.format": String( @@ -942,82 +932,91 @@ $ xvc -vvvv pipeline run "cache.algorithm": String( "blake3", ), + "git.auto_stage": Boolean( + false, + ), + "file.list.no_summary": Boolean( + false, + ), "file.track.text_or_binary": String( "auto", ), + "file.track.no_parallel": Boolean( + false, + ), }, }, XvcConfigMap { source: Project, map: { - "file.list.no_summary": Boolean( + "core.guid": String( + "59a2d95849757118", + ), + "cache.algorithm": String( + "blake3", + ), + "file.list.recursive": Boolean( false, ), - "pipeline.process_pool_size": Integer( - 4, + "git.command": String( + "git", ), "pipeline.default_params_file": String( "params.yaml", ), - "file.track.text_or_binary": String( - "auto", + "file.track.force": Boolean( + false, ), - "file.list.show_dot_files": Boolean( + "file.track.no_parallel": Boolean( false, ), - "git.auto_commit": Boolean( + "git.use_git": Boolean( true, ), - "file.list.recursive": Boolean( + "file.track.no_commit": Boolean( false, ), "file.recheck.method": String( "copy", ), - "file.track.no_commit": Boolean( + "file.list.no_summary": Boolean( false, ), "pipeline.current_pipeline": String( "default", ), - "file.list.format": String( - "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", - ), - "git.command": String( - "git", - ), - "file.carry-in.no_parallel": Boolean( + "git.auto_stage": Boolean( false, ), - "file.track.no_parallel": Boolean( - false, + "git.auto_commit": Boolean( + true, ), - "file.carry-in.force": Boolean( + "file.list.show_dot_files": Boolean( false, ), - "core.verbosity": String( - "error", + "file.list.format": String( + "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", ), "file.list.sort": String( "name-desc", ), - "core.guid": String( - "6142ae8952816bc7", - ), - "cache.algorithm": String( - "blake3", + "pipeline.process_pool_size": Integer( + 4, ), - "file.track.force": Boolean( + "file.carry-in.no_parallel": Boolean( false, ), - "git.use_git": Boolean( - true, + "pipeline.default": String( + "default", ), - "git.auto_stage": Boolean( + "file.carry-in.force": Boolean( false, ), - "pipeline.default": String( - "default", + "core.verbosity": String( + "error", + ), + "file.track.text_or_binary": String( + "auto", ), }, }, @@ -1046,86 +1045,86 @@ $ xvc -vvvv pipeline run }, ], the_config: { - "TRYCMD_TESTS": XvcConfigValue { - source: Environment, + "pipeline.default_params_file": XvcConfigValue { + source: Project, value: String( - "storage,file,pipeline,core", + "params.yaml", ), }, - "file.list.recursive": XvcConfigValue { + "file.list.sort": XvcConfigValue { source: Project, - value: Boolean( - false, + value: String( + "name-desc", ), }, - "file.list.no_summary": XvcConfigValue { + "git.use_git": XvcConfigValue { source: Project, value: Boolean( - false, + true, ), }, - "git.use_git": XvcConfigValue { + "git.auto_commit": XvcConfigValue { source: Project, value: Boolean( true, ), }, - "core.guid": XvcConfigValue { - source: Project, + "core.verbosity": XvcConfigValue { + source: CommandLine, value: String( - "6142ae8952816bc7", + "debug", ), }, - "file.recheck.method": XvcConfigValue { + "git.auto_stage": XvcConfigValue { source: Project, - value: String( - "copy", + value: Boolean( + false, ), }, - "pipeline.default": XvcConfigValue { + "file.carry-in.force": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "pipeline.current_pipeline": XvcConfigValue { source: Project, value: String( "default", ), }, + "cache.algorithm": XvcConfigValue { + source: Project, + value: String( + "blake3", + ), + }, "pipeline.process_pool_size": XvcConfigValue { source: Project, value: Integer( 4, ), }, - "core.verbosity": XvcConfigValue { - source: CommandLine, - value: String( - "debug", - ), - }, - "file.list.show_dot_files": XvcConfigValue { + "file.list.no_summary": XvcConfigValue { source: Project, value: Boolean( false, ), }, - "file.list.format": XvcConfigValue { - source: Project, - value: String( - "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", - ), - }, - "file.track.text_or_binary": XvcConfigValue { + "file.track.no_parallel": XvcConfigValue { source: Project, - value: String( - "auto", + value: Boolean( + false, ), }, - "git.command": XvcConfigValue { + "file.recheck.method": XvcConfigValue { source: Project, value: String( - "git", + "copy", ), }, - "file.track.no_parallel": XvcConfigValue { - source: Project, + "core.quiet": XvcConfigValue { + source: CommandLine, value: Boolean( false, ), @@ -1136,61 +1135,61 @@ $ xvc -vvvv pipeline run false, ), }, - "pipeline.current_pipeline": XvcConfigValue { + "core.guid": XvcConfigValue { source: Project, value: String( - "default", + "59a2d95849757118", ), }, - "git.auto_commit": XvcConfigValue { - source: Project, - value: Boolean( - true, - ), - }, - "file.carry-in.force": XvcConfigValue { + "pipeline.default": XvcConfigValue { source: Project, - value: Boolean( - false, + value: String( + "default", ), }, - "cache.algorithm": XvcConfigValue { + "git.command": XvcConfigValue { source: Project, value: String( - "blake3", + "git", ), }, - "core.quiet": XvcConfigValue { - source: CommandLine, + "file.track.force": XvcConfigValue { + source: Project, value: Boolean( false, ), }, - "file.list.sort": XvcConfigValue { + "file.track.no_commit": XvcConfigValue { source: Project, - value: String( - "name-desc", + value: Boolean( + false, ), }, - "git.auto_stage": XvcConfigValue { + "file.list.recursive": XvcConfigValue { source: Project, value: Boolean( false, ), }, - "pipeline.default_params_file": XvcConfigValue { + "file.list.format": XvcConfigValue { source: Project, value: String( - "params.yaml", + "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", ), }, - "file.track.no_commit": XvcConfigValue { + "file.track.text_or_binary": XvcConfigValue { source: Project, - value: Boolean( - false, + value: String( + "auto", ), }, - "file.track.force": XvcConfigValue { + "TRYCMD_TESTS": XvcConfigValue { + source: Environment, + value: String( + "storage,file,pipeline,core", + ), + }, + "file.list.show_dot_files": XvcConfigValue { source: Project, value: Boolean( false, @@ -1203,7 +1202,7 @@ $ xvc -vvvv pipeline run # The repository id. Please do not delete or change it. # This is used to identify the repository and generate paths in storages. # In the future it may be used to in other ways. -guid = /"6dd0d3b5a9776de4/" +guid = /"919ce8649f3e32f7/" # Default verbosity level. # One of /"error/", /"warn/", /"info/" verbosity = /"error/" @@ -1319,7 +1318,7 @@ default = /"default/" default_params_file = /"params.yaml/" # Number of command processes to run concurrently process_pool_size = 4 -# +# ", current_dir: AbsolutePath( @@ -1354,7 +1353,7 @@ process_pool_size = 4 ), entity_generator: XvcEntityGenerator { counter: 4, - random: 16503095231746129767, + random: 13513277701724141613, dirty: false, }, }, @@ -1672,14 +1671,14 @@ process_pool_size = 4 pre: None, nfa: thompson::NFA( ^000000: FAIL - + transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) ) , nfarev: Some( thompson::NFA( ^000000: FAIL - + transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) ) , @@ -1697,7 +1696,7 @@ process_pool_size = 4 }, nfa: thompson::NFA( ^000000: FAIL - + transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) ) , @@ -1754,12 +1753,12 @@ process_pool_size = 4 }, nfa: thompson::NFA( ^000000: FAIL - + transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) ) , stride2: 1, - start_map: StartByteMap{/x00 => NonWordByte, /x01 => NonWordByte, /x02 => NonWordByte, /x03 => NonWordByte, /x04 => NonWordByte, /x05 => NonWordByte, /x06 => NonWordByte, /x07 => NonWordByte, /x08 => NonWordByte, /t => NonWordByte, + start_map: StartByteMap{/x00 => NonWordByte, /x01 => NonWordByte, /x02 => NonWordByte, /x03 => NonWordByte, /x04 => NonWordByte, /x05 => NonWordByte, /x06 => NonWordByte, /x07 => NonWordByte, /x08 => NonWordByte, /t => NonWordByte, => LineLF, /x0B => NonWordByte, /x0C => NonWordByte, /r => LineCR, /x0E => NonWordByte, /x0F => NonWordByte, /x10 => NonWordByte, /x11 => NonWordByte, /x12 => NonWordByte, /x13 => NonWordByte, /x14 => NonWordByte, /x15 => NonWordByte, /x16 => NonWordByte, /x17 => NonWordByte, /x18 => NonWordByte, /x19 => NonWordByte, /x1A => NonWordByte, /x1B => NonWordByte, /x1C => NonWordByte, /x1D => NonWordByte, /x1E => NonWordByte, /x1F => NonWordByte, ' ' => NonWordByte, ! => NonWordByte, /" => NonWordByte, # => NonWordByte, $ => NonWordByte, % => NonWordByte, & => NonWordByte, /' => NonWordByte, ( => NonWordByte, ) => NonWordByte, * => NonWordByte, + => NonWordByte, , => NonWordByte, - => NonWordByte, . => NonWordByte, / => NonWordByte, 0 => WordByte, 1 => WordByte, 2 => WordByte, 3 => WordByte, 4 => WordByte, 5 => WordByte, 6 => WordByte, 7 => WordByte, 8 => WordByte, 9 => WordByte, : => NonWordByte, ; => NonWordByte, < => NonWordByte, = => NonWordByte, > => NonWordByte, ? => NonWordByte, @ => NonWordByte, A => WordByte, B => WordByte, C => WordByte, D => WordByte, E => WordByte, F => WordByte, G => WordByte, H => WordByte, I => WordByte, J => WordByte, K => WordByte, L => WordByte, M => WordByte, N => WordByte, O => WordByte, P => WordByte, Q => WordByte, R => WordByte, S => WordByte, T => WordByte, U => WordByte, V => WordByte, W => WordByte, X => WordByte, Y => WordByte, Z => WordByte, [ => NonWordByte, // => NonWordByte, ] => NonWordByte, ^ => NonWordByte, _ => WordByte, ` => NonWordByte, a => WordByte, b => WordByte, c => WordByte, d => WordByte, e => WordByte, f => WordByte, g => WordByte, h => WordByte, i => WordByte, j => WordByte, k => WordByte, l => WordByte, m => WordByte, n => WordByte, o => WordByte, p => WordByte, q => WordByte, r => WordByte, s => WordByte, t => WordByte, u => WordByte, v => WordByte, w => WordByte, x => WordByte, y => WordByte, z => WordByte, { => NonWordByte, | => NonWordByte, } => NonWordByte, ~ => NonWordByte, /x7F => NonWordByte, /x80 => NonWordByte, /x81 => NonWordByte, /x82 => NonWordByte, /x83 => NonWordByte, /x84 => NonWordByte, /x85 => NonWordByte, /x86 => NonWordByte, /x87 => NonWordByte, /x88 => NonWordByte, /x89 => NonWordByte, /x8A => NonWordByte, /x8B => NonWordByte, /x8C => NonWordByte, /x8D => NonWordByte, /x8E => NonWordByte, /x8F => NonWordByte, /x90 => NonWordByte, /x91 => NonWordByte, /x92 => NonWordByte, /x93 => NonWordByte, /x94 => NonWordByte, /x95 => NonWordByte, /x96 => NonWordByte, /x97 => NonWordByte, /x98 => NonWordByte, /x99 => NonWordByte, /x9A => NonWordByte, /x9B => NonWordByte, /x9C => NonWordByte, /x9D => NonWordByte, /x9E => NonWordByte, /x9F => NonWordByte, /xA0 => NonWordByte, /xA1 => NonWordByte, /xA2 => NonWordByte, /xA3 => NonWordByte, /xA4 => NonWordByte, /xA5 => NonWordByte, /xA6 => NonWordByte, /xA7 => NonWordByte, /xA8 => NonWordByte, /xA9 => NonWordByte, /xAA => NonWordByte, /xAB => NonWordByte, /xAC => NonWordByte, /xAD => NonWordByte, /xAE => NonWordByte, /xAF => NonWordByte, /xB0 => NonWordByte, /xB1 => NonWordByte, /xB2 => NonWordByte, /xB3 => NonWordByte, /xB4 => NonWordByte, /xB5 => NonWordByte, /xB6 => NonWordByte, /xB7 => NonWordByte, /xB8 => NonWordByte, /xB9 => NonWordByte, /xBA => NonWordByte, /xBB => NonWordByte, /xBC => NonWordByte, /xBD => NonWordByte, /xBE => NonWordByte, /xBF => NonWordByte, /xC0 => NonWordByte, /xC1 => NonWordByte, /xC2 => NonWordByte, /xC3 => NonWordByte, /xC4 => NonWordByte, /xC5 => NonWordByte, /xC6 => NonWordByte, /xC7 => NonWordByte, /xC8 => NonWordByte, /xC9 => NonWordByte, /xCA => NonWordByte, /xCB => NonWordByte, /xCC => NonWordByte, /xCD => NonWordByte, /xCE => NonWordByte, /xCF => NonWordByte, /xD0 => NonWordByte, /xD1 => NonWordByte, /xD2 => NonWordByte, /xD3 => NonWordByte, /xD4 => NonWordByte, /xD5 => NonWordByte, /xD6 => NonWordByte, /xD7 => NonWordByte, /xD8 => NonWordByte, /xD9 => NonWordByte, /xDA => NonWordByte, /xDB => NonWordByte, /xDC => NonWordByte, /xDD => NonWordByte, /xDE => NonWordByte, /xDF => NonWordByte, /xE0 => NonWordByte, /xE1 => NonWordByte, /xE2 => NonWordByte, /xE3 => NonWordByte, /xE4 => NonWordByte, /xE5 => NonWordByte, /xE6 => NonWordByte, /xE7 => NonWordByte, /xE8 => NonWordByte, /xE9 => NonWordByte, /xEA => NonWordByte, /xEB => NonWordByte, /xEC => NonWordByte, /xED => NonWordByte, /xEE => NonWordByte, /xEF => NonWordByte, /xF0 => NonWordByte, /xF1 => NonWordByte, /xF2 => NonWordByte, /xF3 => NonWordByte, /xF4 => NonWordByte, /xF5 => NonWordByte, /xF6 => NonWordByte, /xF7 => NonWordByte, /xF8 => NonWordByte, /xF9 => NonWordByte, /xFA => NonWordByte, /xFB => NonWordByte, /xFC => NonWordByte, /xFD => NonWordByte, /xFE => NonWordByte, /xFF => NonWordByte}, classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]), quitset: ByteSet { @@ -1807,12 +1806,12 @@ process_pool_size = 4 }, nfa: thompson::NFA( ^000000: FAIL - + transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) ) , stride2: 1, - start_map: StartByteMap{/x00 => NonWordByte, /x01 => NonWordByte, /x02 => NonWordByte, /x03 => NonWordByte, /x04 => NonWordByte, /x05 => NonWordByte, /x06 => NonWordByte, /x07 => NonWordByte, /x08 => NonWordByte, /t => NonWordByte, + start_map: StartByteMap{/x00 => NonWordByte, /x01 => NonWordByte, /x02 => NonWordByte, /x03 => NonWordByte, /x04 => NonWordByte, /x05 => NonWordByte, /x06 => NonWordByte, /x07 => NonWordByte, /x08 => NonWordByte, /t => NonWordByte, => LineLF, /x0B => NonWordByte, /x0C => NonWordByte, /r => LineCR, /x0E => NonWordByte, /x0F => NonWordByte, /x10 => NonWordByte, /x11 => NonWordByte, /x12 => NonWordByte, /x13 => NonWordByte, /x14 => NonWordByte, /x15 => NonWordByte, /x16 => NonWordByte, /x17 => NonWordByte, /x18 => NonWordByte, /x19 => NonWordByte, /x1A => NonWordByte, /x1B => NonWordByte, /x1C => NonWordByte, /x1D => NonWordByte, /x1E => NonWordByte, /x1F => NonWordByte, ' ' => NonWordByte, ! => NonWordByte, /" => NonWordByte, # => NonWordByte, $ => NonWordByte, % => NonWordByte, & => NonWordByte, /' => NonWordByte, ( => NonWordByte, ) => NonWordByte, * => NonWordByte, + => NonWordByte, , => NonWordByte, - => NonWordByte, . => NonWordByte, / => NonWordByte, 0 => WordByte, 1 => WordByte, 2 => WordByte, 3 => WordByte, 4 => WordByte, 5 => WordByte, 6 => WordByte, 7 => WordByte, 8 => WordByte, 9 => WordByte, : => NonWordByte, ; => NonWordByte, < => NonWordByte, = => NonWordByte, > => NonWordByte, ? => NonWordByte, @ => NonWordByte, A => WordByte, B => WordByte, C => WordByte, D => WordByte, E => WordByte, F => WordByte, G => WordByte, H => WordByte, I => WordByte, J => WordByte, K => WordByte, L => WordByte, M => WordByte, N => WordByte, O => WordByte, P => WordByte, Q => WordByte, R => WordByte, S => WordByte, T => WordByte, U => WordByte, V => WordByte, W => WordByte, X => WordByte, Y => WordByte, Z => WordByte, [ => NonWordByte, // => NonWordByte, ] => NonWordByte, ^ => NonWordByte, _ => WordByte, ` => NonWordByte, a => WordByte, b => WordByte, c => WordByte, d => WordByte, e => WordByte, f => WordByte, g => WordByte, h => WordByte, i => WordByte, j => WordByte, k => WordByte, l => WordByte, m => WordByte, n => WordByte, o => WordByte, p => WordByte, q => WordByte, r => WordByte, s => WordByte, t => WordByte, u => WordByte, v => WordByte, w => WordByte, x => WordByte, y => WordByte, z => WordByte, { => NonWordByte, | => NonWordByte, } => NonWordByte, ~ => NonWordByte, /x7F => NonWordByte, /x80 => NonWordByte, /x81 => NonWordByte, /x82 => NonWordByte, /x83 => NonWordByte, /x84 => NonWordByte, /x85 => NonWordByte, /x86 => NonWordByte, /x87 => NonWordByte, /x88 => NonWordByte, /x89 => NonWordByte, /x8A => NonWordByte, /x8B => NonWordByte, /x8C => NonWordByte, /x8D => NonWordByte, /x8E => NonWordByte, /x8F => NonWordByte, /x90 => NonWordByte, /x91 => NonWordByte, /x92 => NonWordByte, /x93 => NonWordByte, /x94 => NonWordByte, /x95 => NonWordByte, /x96 => NonWordByte, /x97 => NonWordByte, /x98 => NonWordByte, /x99 => NonWordByte, /x9A => NonWordByte, /x9B => NonWordByte, /x9C => NonWordByte, /x9D => NonWordByte, /x9E => NonWordByte, /x9F => NonWordByte, /xA0 => NonWordByte, /xA1 => NonWordByte, /xA2 => NonWordByte, /xA3 => NonWordByte, /xA4 => NonWordByte, /xA5 => NonWordByte, /xA6 => NonWordByte, /xA7 => NonWordByte, /xA8 => NonWordByte, /xA9 => NonWordByte, /xAA => NonWordByte, /xAB => NonWordByte, /xAC => NonWordByte, /xAD => NonWordByte, /xAE => NonWordByte, /xAF => NonWordByte, /xB0 => NonWordByte, /xB1 => NonWordByte, /xB2 => NonWordByte, /xB3 => NonWordByte, /xB4 => NonWordByte, /xB5 => NonWordByte, /xB6 => NonWordByte, /xB7 => NonWordByte, /xB8 => NonWordByte, /xB9 => NonWordByte, /xBA => NonWordByte, /xBB => NonWordByte, /xBC => NonWordByte, /xBD => NonWordByte, /xBE => NonWordByte, /xBF => NonWordByte, /xC0 => NonWordByte, /xC1 => NonWordByte, /xC2 => NonWordByte, /xC3 => NonWordByte, /xC4 => NonWordByte, /xC5 => NonWordByte, /xC6 => NonWordByte, /xC7 => NonWordByte, /xC8 => NonWordByte, /xC9 => NonWordByte, /xCA => NonWordByte, /xCB => NonWordByte, /xCC => NonWordByte, /xCD => NonWordByte, /xCE => NonWordByte, /xCF => NonWordByte, /xD0 => NonWordByte, /xD1 => NonWordByte, /xD2 => NonWordByte, /xD3 => NonWordByte, /xD4 => NonWordByte, /xD5 => NonWordByte, /xD6 => NonWordByte, /xD7 => NonWordByte, /xD8 => NonWordByte, /xD9 => NonWordByte, /xDA => NonWordByte, /xDB => NonWordByte, /xDC => NonWordByte, /xDD => NonWordByte, /xDE => NonWordByte, /xDF => NonWordByte, /xE0 => NonWordByte, /xE1 => NonWordByte, /xE2 => NonWordByte, /xE3 => NonWordByte, /xE4 => NonWordByte, /xE5 => NonWordByte, /xE6 => NonWordByte, /xE7 => NonWordByte, /xE8 => NonWordByte, /xE9 => NonWordByte, /xEA => NonWordByte, /xEB => NonWordByte, /xEC => NonWordByte, /xED => NonWordByte, /xEE => NonWordByte, /xEF => NonWordByte, /xF0 => NonWordByte, /xF1 => NonWordByte, /xF2 => NonWordByte, /xF3 => NonWordByte, /xF4 => NonWordByte, /xF5 => NonWordByte, /xF6 => NonWordByte, /xF7 => NonWordByte, /xF8 => NonWordByte, /xF9 => NonWordByte, /xFA => NonWordByte, /xFB => NonWordByte, /xFC => NonWordByte, /xFD => NonWordByte, /xFE => NonWordByte, /xFF => NonWordByte}, classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]), quitset: ByteSet { @@ -2021,18 +2020,18 @@ process_pool_size = 4 [TRACE][pipeline/src/pipeline/mod.rs::331] &dependency_graph: { XvcEntity( 2, - 3920065547014771711, + 9270302598983016314, ): [], } [TRACE][pipeline/src/pipeline/mod.rs::343] &dependency_graph: { XvcEntity( 2, - 3920065547014771711, + 9270302598983016314, ): [], } [INFO][pipeline/src/pipeline/mod.rs::347] Pipeline Graph: digraph { - 0 [ label = "(2, 3920065547014771711)" ] + 0 [ label = "(2, 9270302598983016314)" ] } @@ -2041,7 +2040,7 @@ digraph { map: { XvcEntity( 2, - 3920065547014771711, + 9270302598983016314, ): Begin( FromInit, ), @@ -2054,14 +2053,14 @@ digraph { map: { XvcEntity( 2, - 3920065547014771711, + 9270302598983016314, ): ScopedJoinHandle { .. }, }, } [TRACE][pipeline/src/pipeline/mod.rs::512] (step_e, &jh): ( XvcEntity( 2, - 3920065547014771711, + 9270302598983016314, ), ScopedJoinHandle { .. }, ) @@ -2070,7 +2069,7 @@ digraph { map: { XvcEntity( 2, - 3920065547014771711, + 9270302598983016314, ): XvcStep { name: "average-age", }, @@ -2081,7 +2080,7 @@ digraph { }: [ XvcEntity( 2, - 3920065547014771711, + 9270302598983016314, ), ], }, @@ -2090,7 +2089,7 @@ digraph { Add { entity: XvcEntity( 2, - 3920065547014771711, + 9270302598983016314, ), value: XvcStep { name: "average-age", @@ -2099,7 +2098,7 @@ digraph { Add { entity: XvcEntity( 2, - 3920065547014771711, + 9270302598983016314, ), value: XvcStep { name: "average-age", @@ -2115,7 +2114,7 @@ digraph { map: { XvcEntity( 3, - 324224710907113891, + 9022053985516504033, ): SqliteQueryDigest( SqliteQueryDep { path: XvcPath( @@ -2140,7 +2139,7 @@ digraph { ): [ XvcEntity( 3, - 324224710907113891, + 9022053985516504033, ), ], }, @@ -2149,7 +2148,7 @@ digraph { Add { entity: XvcEntity( 3, - 324224710907113891, + 9022053985516504033, ), value: SqliteQueryDigest( SqliteQueryDep { @@ -2172,11 +2171,11 @@ digraph { map: { XvcEntity( 3, - 324224710907113891, + 9022053985516504033, ): ChildEntity( XvcEntity( 2, - 3920065547014771711, + 9270302598983016314, ), PhantomData, PhantomData, @@ -2186,14 +2185,14 @@ digraph { ChildEntity( XvcEntity( 2, - 3920065547014771711, + 9270302598983016314, ), PhantomData, PhantomData, ): [ XvcEntity( 3, - 324224710907113891, + 9022053985516504033, ), ], }, @@ -2202,12 +2201,12 @@ digraph { Add { entity: XvcEntity( 3, - 324224710907113891, + 9022053985516504033, ), value: ChildEntity( XvcEntity( 2, - 3920065547014771711, + 9270302598983016314, ), PhantomData, PhantomData, @@ -2222,7 +2221,7 @@ digraph { } [TRACE][pipeline/src/pipeline/mod.rs::622] step_e: XvcEntity( 2, - 3920065547014771711, + 9270302598983016314, ) [TRACE][pipeline/src/pipeline/mod.rs::564] dep_neighbors: Neighbors { iter: Iter( @@ -2251,6 +2250,7 @@ digraph { FromRunConditional, ) [TRACE][pipeline/src/pipeline/mod.rs::770] step.name: "average-age" +[INFO] No dependency steps for step average-age [TRACE][pipeline/src/pipeline/mod.rs::771] &r_next_state: CheckingOutputs( FromDependencyStepsFinishedSuccessfully, ) @@ -2272,14 +2272,13 @@ digraph { ) [TRACE][pipeline/src/pipeline/mod.rs::1008] parent_entity: XvcEntity( 2, - 3920065547014771711, + 9270302598983016314, ) -[INFO] No dependency steps for step average-age [TRACE][pipeline/src/pipeline/mod.rs::1011] deps: HStore { map: { XvcEntity( 3, - 324224710907113891, + 9022053985516504033, ): SqliteQueryDigest( SqliteQueryDep { path: XvcPath( @@ -2364,7 +2363,7 @@ digraph { map: { XvcEntity( 3, - 324224710907113891, + 9022053985516504033, ): RecordMissing { actual: SqliteQueryDigest( SqliteQueryDep { @@ -2433,7 +2432,7 @@ digraph { map: { XvcEntity( 3, - 324224710907113891, + 9022053985516504033, ): SqliteQueryDigest( SqliteQueryDep { path: XvcPath( @@ -2466,301 +2465,6244 @@ digraph { [TRACE][pipeline/src/pipeline/deps/sqlite_query.rs::71] col: Integer( 18, ) -thread '' panicked at pipeline/src/pipeline/mod.rs:1079:32: -SqliteError { source: FromSqlConversionFailure(18446744073709551615, Null, InvalidType) } -note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace -thread '' panicked at lib/src/cli/mod.rs:288:52: -[PANIC] SqliteError { source: FromSqlConversionFailure(18446744073709551615, Null, InvalidType) }, [pipeline/src/pipeline/mod.rs::1079] -thread '' panicked at pipeline/src/pipeline/mod.rs:514:17: -called `Result::unwrap()` on an `Err` value: "SendError(..)" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" +[TRACE][pipeline/src/pipeline/mod.rs::770] step.name: "average-age" +[TRACE][pipeline/src/pipeline/mod.rs::771] &r_next_state: ComparingDiffsAndOutputs( + FromThoroughDiffsChanged, +) +[TRACE][pipeline/src/pipeline/mod.rs::773] &step_state: ComparingDiffsAndOutputs( + FromThoroughDiffsChanged, +) +[TRACE][pipeline/src/pipeline/mod.rs::660] &step_state: ComparingDiffsAndOutputs( + FromThoroughDiffsChanged, +) +[TRACE][pipeline/src/pipeline/mod.rs::770] step.name: "average-age" +[TRACE][pipeline/src/pipeline/mod.rs::771] &r_next_state: WaitingToRun( + FromDiffsHasChanged, +) +[TRACE][pipeline/src/pipeline/mod.rs::773] &step_state: WaitingToRun( + FromDiffsHasChanged, +) +[TRACE][pipeline/src/pipeline/mod.rs::660] &step_state: WaitingToRun( + FromDiffsHasChanged, +) +[INFO] [average-age] Dependencies has changed +[TRACE][pipeline/src/pipeline/mod.rs::1539] params: StepStateParams { + xvc_root: XvcRootInner { + absolute_path: AbsolutePath( + "[CWD]", + ), + xvc_dir: AbsolutePath( + "[CWD]/.xvc", + ), + store_dir: AbsolutePath( + "[CWD]/.xvc/store", + ), + config: XvcConfig { + current_dir: XvcConfigOption { + source: Runtime, + option: AbsolutePath( + "[CWD]", + ), + }, + config_maps: [ + XvcConfigMap { + source: Default, + map: { + "pipeline.default_params_file": String( + "params.yaml", + ), + "file.carry-in.no_parallel": Boolean( + false, + ), + "file.carry-in.force": Boolean( + false, + ), + "core.verbosity": String( + "error", + ), + "file.recheck.method": String( + "copy", + ), + "file.list.sort": String( + "name-desc", + ), + "git.use_git": Boolean( + true, + ), + "pipeline.current_pipeline": String( + "default", + ), + "core.guid": String( + "919ce8649f3e32f7", + ), + "git.auto_commit": Boolean( + true, + ), + "pipeline.default": String( + "default", + ), + "pipeline.process_pool_size": Integer( + 4, + ), + "git.command": String( + "git", + ), + "file.track.no_commit": Boolean( + false, + ), + "file.track.force": Boolean( + false, + ), + "file.list.show_dot_files": Boolean( + false, + ), + "file.list.recursive": Boolean( + false, + ), + "file.list.format": String( + "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", + ), + "cache.algorithm": String( + "blake3", + ), + "git.auto_stage": Boolean( + false, + ), + "file.list.no_summary": Boolean( + false, + ), + "file.track.text_or_binary": String( + "auto", + ), + "file.track.no_parallel": Boolean( + false, + ), + }, + }, + XvcConfigMap { + source: Project, + map: { + "core.guid": String( + "59a2d95849757118", + ), + "cache.algorithm": String( + "blake3", + ), + "file.list.recursive": Boolean( + false, + ), + "git.command": String( + "git", + ), + "pipeline.default_params_file": String( + "params.yaml", + ), + "file.track.force": Boolean( + false, + ), + "file.track.no_parallel": Boolean( + false, + ), + "git.use_git": Boolean( + true, + ), + "file.track.no_commit": Boolean( + false, + ), + "file.recheck.method": String( + "copy", + ), + "file.list.no_summary": Boolean( + false, + ), + "pipeline.current_pipeline": String( + "default", + ), + "git.auto_stage": Boolean( + false, + ), + "git.auto_commit": Boolean( + true, + ), + "file.list.show_dot_files": Boolean( + false, + ), + "file.list.format": String( + "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", + ), + "file.list.sort": String( + "name-desc", + ), + "pipeline.process_pool_size": Integer( + 4, + ), + "file.carry-in.no_parallel": Boolean( + false, + ), + "pipeline.default": String( + "default", + ), + "file.carry-in.force": Boolean( + false, + ), + "core.verbosity": String( + "error", + ), + "file.track.text_or_binary": String( + "auto", + ), + }, + }, + XvcConfigMap { + source: Local, + map: {}, + }, + XvcConfigMap { + source: Environment, + map: { + "TRYCMD_TESTS": String( + "storage,file,pipeline,core", + ), + }, + }, + XvcConfigMap { + source: CommandLine, + map: { + "core.quiet": Boolean( + false, + ), + "core.verbosity": String( + "debug", + ), + }, + }, + ], + the_config: { + "pipeline.default_params_file": XvcConfigValue { + source: Project, + value: String( + "params.yaml", + ), + }, + "file.list.sort": XvcConfigValue { + source: Project, + value: String( + "name-desc", + ), + }, + "git.use_git": XvcConfigValue { + source: Project, + value: Boolean( + true, + ), + }, + "git.auto_commit": XvcConfigValue { + source: Project, + value: Boolean( + true, + ), + }, + "core.verbosity": XvcConfigValue { + source: CommandLine, + value: String( + "debug", + ), + }, + "git.auto_stage": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "file.carry-in.force": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "pipeline.current_pipeline": XvcConfigValue { + source: Project, + value: String( + "default", + ), + }, + "cache.algorithm": XvcConfigValue { + source: Project, + value: String( + "blake3", + ), + }, + "pipeline.process_pool_size": XvcConfigValue { + source: Project, + value: Integer( + 4, + ), + }, + "file.list.no_summary": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "file.track.no_parallel": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "file.recheck.method": XvcConfigValue { + source: Project, + value: String( + "copy", + ), + }, + "core.quiet": XvcConfigValue { + source: CommandLine, + value: Boolean( + false, + ), + }, + "file.carry-in.no_parallel": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "core.guid": XvcConfigValue { + source: Project, + value: String( + "59a2d95849757118", + ), + }, + "pipeline.default": XvcConfigValue { + source: Project, + value: String( + "default", + ), + }, + "git.command": XvcConfigValue { + source: Project, + value: String( + "git", + ), + }, + "file.track.force": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "file.track.no_commit": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "file.list.recursive": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "file.list.format": XvcConfigValue { + source: Project, + value: String( + "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", + ), + }, + "file.track.text_or_binary": XvcConfigValue { + source: Project, + value: String( + "auto", + ), + }, + "TRYCMD_TESTS": XvcConfigValue { + source: Environment, + value: String( + "storage,file,pipeline,core", + ), + }, + "file.list.show_dot_files": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + }, + init_params: XvcConfigParams { + default_configuration: " +[core] +# The repository id. Please do not delete or change it. +# This is used to identify the repository and generate paths in storages. +# In the future it may be used to in other ways. +guid = /"919ce8649f3e32f7/" +# Default verbosity level. +# One of /"error/", /"warn/", /"info/" +verbosity = /"error/" + +[git] +# Automate git operations. +# Turning this off leads Xvc to behave as if it's not in a Git repository. +# Not recommended unless you're really not using Git +use_git = true +# Command to run Git process. +# You can set this to an absolute path to specify an executable +# If set to a non-absolute path, the executable will be searched in $PATH. +command = /"git/" + +# Commit changes in .xvc/ directory after commands. +# You can set this to false if you want to commit manually. +auto_commit = true + +# Stage changes in .xvc/ directory without committing. +# auto_commit implies auto_stage. +# If you want to commit manually but don't want to stage after individual Xvc commands, you can set this to true. +auto_stage = false + +[cache] +# The hash algorithm used for the cache. +# It may take blake3, blake2, sha2 or sha3 as values. +# All algorithms are selected to produce 256-bit hashes, so sha2 means SHA2-256, blake2 means BLAKE2s, etc. +# The cache path is produced by prepending algorithm name to the cache. +# Blake3 files are in .xvc/b3/, while sha2 files are in .xvc/s2/ etc. +algorithm = /"blake3/" + +[file] + +[file.track] + +# Don't move file content to cache after xvc file track +no_commit = false +# Force to track files even if they are already tracked. +force = false + +# Xvc calculates file content digest differently for text and binary files. +# This option controls whether to treat files as text or binary. +# It may take auto, text or binary as values. +# Auto check each file individually and treat it as text if it's text. +text_or_binary = /"auto/" + +# Don't use parallelism in track operations. +# Note that some of the operations are implemented in parallel by default, and this option affects some heavier operations. +no_parallel = false + +[file.list] + +# Format for `xvc file list` rows. You can reorder or remove columns. +# The following are the keys for each row: +# - {acd64}: actual content digest. All 64 digits from the workspace file's content. +# - {acd8}: actual content digest. First 8 digits the file content digest. +# - {aft}: actual file type. Whether the entry is a file (F), directory (D), +# symlink (S), hardlink (H) or reflink (R). +# - {asz}: actual size. The size of the workspace file in bytes. It uses MB, +# GB and TB to represent sizes larger than 1MB. +# - {ats}: actual timestamp. The timestamp of the workspace file. +# - {cst}: cache status. One of /"=/", /">/", /"/", /" 0 + F 000001: + >000002: /x00 => 2 + 000003: /x00 => 0 + match kind: Standard + prefilter: false + state length: 4 + pattern length: 0 + shortest pattern length: 18446744073709551615 + longest pattern length: 0 + alphabet length: 1 + stride: 1 + byte classes: ByteClasses(0 => [0-255]) + memory usage: 16 + ) + , + ), + map: [], + longest: 0, + }, + ), + Prefix( + PrefixStrategy { + matcher: AhoCorasick( + dfa::DFA( + D 000000: /x00 => 0 + F 000001: + >000002: /x00 => 2 + 000003: /x00 => 0 + match kind: Standard + prefilter: false + state length: 4 + pattern length: 0 + shortest pattern length: 18446744073709551615 + longest pattern length: 0 + alphabet length: 1 + stride: 1 + byte classes: ByteClasses(0 => [0-255]) + memory usage: 16 + ) + , + ), + map: [], + longest: 0, + }, + ), + RequiredExtension( + RequiredExtensionStrategy( + {}, + ), + ), + Regex( + RegexSetStrategy { + matcher: Regex { + imp: RegexI { + strat: Core { + info: RegexInfo( + RegexInfoI { + config: Config { + match_kind: Some( + All, + ), + utf8_empty: Some( + false, + ), + autopre: None, + pre: None, + which_captures: None, + nfa_size_limit: Some( + Some( + 10485760, + ), + ), + onepass_size_limit: None, + hybrid_cache_capacity: Some( + 10485760, + ), + hybrid: None, + dfa: None, + dfa_size_limit: None, + dfa_state_limit: None, + onepass: None, + backtrack: None, + byte_classes: None, + line_terminator: None, + }, + props: [], + props_union: Properties( + PropertiesI { + minimum_len: None, + maximum_len: None, + look_set: ∅, + look_set_prefix: ∅, + look_set_suffix: ∅, + look_set_prefix_any: ∅, + look_set_suffix_any: ∅, + utf8: true, + explicit_captures_len: 0, + static_explicit_captures_len: None, + literal: false, + alternation_literal: true, + }, + ), + }, + ), + pre: None, + nfa: thompson::NFA( + ^000000: FAIL + + transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) + ) + , + nfarev: Some( + thompson::NFA( + ^000000: FAIL + + transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) + ) + , + ), + pikevm: PikeVM( + PikeVMEngine( + PikeVM { + config: Config { + match_kind: Some( + All, + ), + pre: Some( + None, + ), + }, + nfa: thompson::NFA( + ^000000: FAIL + + transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) + ) + , + }, + ), + ), + backtrack: BoundedBacktracker( + None, + ), + onepass: OnePass( + None, + ), + hybrid: Hybrid( + Some( + HybridEngine( + Regex { + forward: DFA { + config: Config { + match_kind: Some( + All, + ), + pre: Some( + None, + ), + starts_for_each_pattern: Some( + true, + ), + byte_classes: Some( + true, + ), + unicode_word_boundary: Some( + true, + ), + quitset: None, + specialize_start_states: Some( + false, + ), + cache_capacity: Some( + 10485760, + ), + skip_cache_capacity_check: Some( + false, + ), + minimum_cache_clear_count: Some( + Some( + 3, + ), + ), + minimum_bytes_per_state: Some( + Some( + 10, + ), + ), + }, + nfa: thompson::NFA( + ^000000: FAIL + + transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) + ) + , + stride2: 1, + start_map: StartByteMap{/x00 => NonWordByte, /x01 => NonWordByte, /x02 => NonWordByte, /x03 => NonWordByte, /x04 => NonWordByte, /x05 => NonWordByte, /x06 => NonWordByte, /x07 => NonWordByte, /x08 => NonWordByte, /t => NonWordByte, + => LineLF, /x0B => NonWordByte, /x0C => NonWordByte, /r => LineCR, /x0E => NonWordByte, /x0F => NonWordByte, /x10 => NonWordByte, /x11 => NonWordByte, /x12 => NonWordByte, /x13 => NonWordByte, /x14 => NonWordByte, /x15 => NonWordByte, /x16 => NonWordByte, /x17 => NonWordByte, /x18 => NonWordByte, /x19 => NonWordByte, /x1A => NonWordByte, /x1B => NonWordByte, /x1C => NonWordByte, /x1D => NonWordByte, /x1E => NonWordByte, /x1F => NonWordByte, ' ' => NonWordByte, ! => NonWordByte, /" => NonWordByte, # => NonWordByte, $ => NonWordByte, % => NonWordByte, & => NonWordByte, /' => NonWordByte, ( => NonWordByte, ) => NonWordByte, * => NonWordByte, + => NonWordByte, , => NonWordByte, - => NonWordByte, . => NonWordByte, / => NonWordByte, 0 => WordByte, 1 => WordByte, 2 => WordByte, 3 => WordByte, 4 => WordByte, 5 => WordByte, 6 => WordByte, 7 => WordByte, 8 => WordByte, 9 => WordByte, : => NonWordByte, ; => NonWordByte, < => NonWordByte, = => NonWordByte, > => NonWordByte, ? => NonWordByte, @ => NonWordByte, A => WordByte, B => WordByte, C => WordByte, D => WordByte, E => WordByte, F => WordByte, G => WordByte, H => WordByte, I => WordByte, J => WordByte, K => WordByte, L => WordByte, M => WordByte, N => WordByte, O => WordByte, P => WordByte, Q => WordByte, R => WordByte, S => WordByte, T => WordByte, U => WordByte, V => WordByte, W => WordByte, X => WordByte, Y => WordByte, Z => WordByte, [ => NonWordByte, // => NonWordByte, ] => NonWordByte, ^ => NonWordByte, _ => WordByte, ` => NonWordByte, a => WordByte, b => WordByte, c => WordByte, d => WordByte, e => WordByte, f => WordByte, g => WordByte, h => WordByte, i => WordByte, j => WordByte, k => WordByte, l => WordByte, m => WordByte, n => WordByte, o => WordByte, p => WordByte, q => WordByte, r => WordByte, s => WordByte, t => WordByte, u => WordByte, v => WordByte, w => WordByte, x => WordByte, y => WordByte, z => WordByte, { => NonWordByte, | => NonWordByte, } => NonWordByte, ~ => NonWordByte, /x7F => NonWordByte, /x80 => NonWordByte, /x81 => NonWordByte, /x82 => NonWordByte, /x83 => NonWordByte, /x84 => NonWordByte, /x85 => NonWordByte, /x86 => NonWordByte, /x87 => NonWordByte, /x88 => NonWordByte, /x89 => NonWordByte, /x8A => NonWordByte, /x8B => NonWordByte, /x8C => NonWordByte, /x8D => NonWordByte, /x8E => NonWordByte, /x8F => NonWordByte, /x90 => NonWordByte, /x91 => NonWordByte, /x92 => NonWordByte, /x93 => NonWordByte, /x94 => NonWordByte, /x95 => NonWordByte, /x96 => NonWordByte, /x97 => NonWordByte, /x98 => NonWordByte, /x99 => NonWordByte, /x9A => NonWordByte, /x9B => NonWordByte, /x9C => NonWordByte, /x9D => NonWordByte, /x9E => NonWordByte, /x9F => NonWordByte, /xA0 => NonWordByte, /xA1 => NonWordByte, /xA2 => NonWordByte, /xA3 => NonWordByte, /xA4 => NonWordByte, /xA5 => NonWordByte, /xA6 => NonWordByte, /xA7 => NonWordByte, /xA8 => NonWordByte, /xA9 => NonWordByte, /xAA => NonWordByte, /xAB => NonWordByte, /xAC => NonWordByte, /xAD => NonWordByte, /xAE => NonWordByte, /xAF => NonWordByte, /xB0 => NonWordByte, /xB1 => NonWordByte, /xB2 => NonWordByte, /xB3 => NonWordByte, /xB4 => NonWordByte, /xB5 => NonWordByte, /xB6 => NonWordByte, /xB7 => NonWordByte, /xB8 => NonWordByte, /xB9 => NonWordByte, /xBA => NonWordByte, /xBB => NonWordByte, /xBC => NonWordByte, /xBD => NonWordByte, /xBE => NonWordByte, /xBF => NonWordByte, /xC0 => NonWordByte, /xC1 => NonWordByte, /xC2 => NonWordByte, /xC3 => NonWordByte, /xC4 => NonWordByte, /xC5 => NonWordByte, /xC6 => NonWordByte, /xC7 => NonWordByte, /xC8 => NonWordByte, /xC9 => NonWordByte, /xCA => NonWordByte, /xCB => NonWordByte, /xCC => NonWordByte, /xCD => NonWordByte, /xCE => NonWordByte, /xCF => NonWordByte, /xD0 => NonWordByte, /xD1 => NonWordByte, /xD2 => NonWordByte, /xD3 => NonWordByte, /xD4 => NonWordByte, /xD5 => NonWordByte, /xD6 => NonWordByte, /xD7 => NonWordByte, /xD8 => NonWordByte, /xD9 => NonWordByte, /xDA => NonWordByte, /xDB => NonWordByte, /xDC => NonWordByte, /xDD => NonWordByte, /xDE => NonWordByte, /xDF => NonWordByte, /xE0 => NonWordByte, /xE1 => NonWordByte, /xE2 => NonWordByte, /xE3 => NonWordByte, /xE4 => NonWordByte, /xE5 => NonWordByte, /xE6 => NonWordByte, /xE7 => NonWordByte, /xE8 => NonWordByte, /xE9 => NonWordByte, /xEA => NonWordByte, /xEB => NonWordByte, /xEC => NonWordByte, /xED => NonWordByte, /xEE => NonWordByte, /xEF => NonWordByte, /xF0 => NonWordByte, /xF1 => NonWordByte, /xF2 => NonWordByte, /xF3 => NonWordByte, /xF4 => NonWordByte, /xF5 => NonWordByte, /xF6 => NonWordByte, /xF7 => NonWordByte, /xF8 => NonWordByte, /xF9 => NonWordByte, /xFA => NonWordByte, /xFB => NonWordByte, /xFC => NonWordByte, /xFD => NonWordByte, /xFE => NonWordByte, /xFF => NonWordByte}, + classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]), + quitset: ByteSet { + bits: {}, + }, + cache_capacity: 10485760, + }, + reverse: DFA { + config: Config { + match_kind: Some( + All, + ), + pre: Some( + None, + ), + starts_for_each_pattern: Some( + true, + ), + byte_classes: Some( + true, + ), + unicode_word_boundary: Some( + true, + ), + quitset: None, + specialize_start_states: Some( + false, + ), + cache_capacity: Some( + 10485760, + ), + skip_cache_capacity_check: Some( + false, + ), + minimum_cache_clear_count: Some( + Some( + 3, + ), + ), + minimum_bytes_per_state: Some( + Some( + 10, + ), + ), + }, + nfa: thompson::NFA( + ^000000: FAIL + + transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) + ) + , + stride2: 1, + start_map: StartByteMap{/x00 => NonWordByte, /x01 => NonWordByte, /x02 => NonWordByte, /x03 => NonWordByte, /x04 => NonWordByte, /x05 => NonWordByte, /x06 => NonWordByte, /x07 => NonWordByte, /x08 => NonWordByte, /t => NonWordByte, + => LineLF, /x0B => NonWordByte, /x0C => NonWordByte, /r => LineCR, /x0E => NonWordByte, /x0F => NonWordByte, /x10 => NonWordByte, /x11 => NonWordByte, /x12 => NonWordByte, /x13 => NonWordByte, /x14 => NonWordByte, /x15 => NonWordByte, /x16 => NonWordByte, /x17 => NonWordByte, /x18 => NonWordByte, /x19 => NonWordByte, /x1A => NonWordByte, /x1B => NonWordByte, /x1C => NonWordByte, /x1D => NonWordByte, /x1E => NonWordByte, /x1F => NonWordByte, ' ' => NonWordByte, ! => NonWordByte, /" => NonWordByte, # => NonWordByte, $ => NonWordByte, % => NonWordByte, & => NonWordByte, /' => NonWordByte, ( => NonWordByte, ) => NonWordByte, * => NonWordByte, + => NonWordByte, , => NonWordByte, - => NonWordByte, . => NonWordByte, / => NonWordByte, 0 => WordByte, 1 => WordByte, 2 => WordByte, 3 => WordByte, 4 => WordByte, 5 => WordByte, 6 => WordByte, 7 => WordByte, 8 => WordByte, 9 => WordByte, : => NonWordByte, ; => NonWordByte, < => NonWordByte, = => NonWordByte, > => NonWordByte, ? => NonWordByte, @ => NonWordByte, A => WordByte, B => WordByte, C => WordByte, D => WordByte, E => WordByte, F => WordByte, G => WordByte, H => WordByte, I => WordByte, J => WordByte, K => WordByte, L => WordByte, M => WordByte, N => WordByte, O => WordByte, P => WordByte, Q => WordByte, R => WordByte, S => WordByte, T => WordByte, U => WordByte, V => WordByte, W => WordByte, X => WordByte, Y => WordByte, Z => WordByte, [ => NonWordByte, // => NonWordByte, ] => NonWordByte, ^ => NonWordByte, _ => WordByte, ` => NonWordByte, a => WordByte, b => WordByte, c => WordByte, d => WordByte, e => WordByte, f => WordByte, g => WordByte, h => WordByte, i => WordByte, j => WordByte, k => WordByte, l => WordByte, m => WordByte, n => WordByte, o => WordByte, p => WordByte, q => WordByte, r => WordByte, s => WordByte, t => WordByte, u => WordByte, v => WordByte, w => WordByte, x => WordByte, y => WordByte, z => WordByte, { => NonWordByte, | => NonWordByte, } => NonWordByte, ~ => NonWordByte, /x7F => NonWordByte, /x80 => NonWordByte, /x81 => NonWordByte, /x82 => NonWordByte, /x83 => NonWordByte, /x84 => NonWordByte, /x85 => NonWordByte, /x86 => NonWordByte, /x87 => NonWordByte, /x88 => NonWordByte, /x89 => NonWordByte, /x8A => NonWordByte, /x8B => NonWordByte, /x8C => NonWordByte, /x8D => NonWordByte, /x8E => NonWordByte, /x8F => NonWordByte, /x90 => NonWordByte, /x91 => NonWordByte, /x92 => NonWordByte, /x93 => NonWordByte, /x94 => NonWordByte, /x95 => NonWordByte, /x96 => NonWordByte, /x97 => NonWordByte, /x98 => NonWordByte, /x99 => NonWordByte, /x9A => NonWordByte, /x9B => NonWordByte, /x9C => NonWordByte, /x9D => NonWordByte, /x9E => NonWordByte, /x9F => NonWordByte, /xA0 => NonWordByte, /xA1 => NonWordByte, /xA2 => NonWordByte, /xA3 => NonWordByte, /xA4 => NonWordByte, /xA5 => NonWordByte, /xA6 => NonWordByte, /xA7 => NonWordByte, /xA8 => NonWordByte, /xA9 => NonWordByte, /xAA => NonWordByte, /xAB => NonWordByte, /xAC => NonWordByte, /xAD => NonWordByte, /xAE => NonWordByte, /xAF => NonWordByte, /xB0 => NonWordByte, /xB1 => NonWordByte, /xB2 => NonWordByte, /xB3 => NonWordByte, /xB4 => NonWordByte, /xB5 => NonWordByte, /xB6 => NonWordByte, /xB7 => NonWordByte, /xB8 => NonWordByte, /xB9 => NonWordByte, /xBA => NonWordByte, /xBB => NonWordByte, /xBC => NonWordByte, /xBD => NonWordByte, /xBE => NonWordByte, /xBF => NonWordByte, /xC0 => NonWordByte, /xC1 => NonWordByte, /xC2 => NonWordByte, /xC3 => NonWordByte, /xC4 => NonWordByte, /xC5 => NonWordByte, /xC6 => NonWordByte, /xC7 => NonWordByte, /xC8 => NonWordByte, /xC9 => NonWordByte, /xCA => NonWordByte, /xCB => NonWordByte, /xCC => NonWordByte, /xCD => NonWordByte, /xCE => NonWordByte, /xCF => NonWordByte, /xD0 => NonWordByte, /xD1 => NonWordByte, /xD2 => NonWordByte, /xD3 => NonWordByte, /xD4 => NonWordByte, /xD5 => NonWordByte, /xD6 => NonWordByte, /xD7 => NonWordByte, /xD8 => NonWordByte, /xD9 => NonWordByte, /xDA => NonWordByte, /xDB => NonWordByte, /xDC => NonWordByte, /xDD => NonWordByte, /xDE => NonWordByte, /xDF => NonWordByte, /xE0 => NonWordByte, /xE1 => NonWordByte, /xE2 => NonWordByte, /xE3 => NonWordByte, /xE4 => NonWordByte, /xE5 => NonWordByte, /xE6 => NonWordByte, /xE7 => NonWordByte, /xE8 => NonWordByte, /xE9 => NonWordByte, /xEA => NonWordByte, /xEB => NonWordByte, /xEC => NonWordByte, /xED => NonWordByte, /xEE => NonWordByte, /xEF => NonWordByte, /xF0 => NonWordByte, /xF1 => NonWordByte, /xF2 => NonWordByte, /xF3 => NonWordByte, /xF4 => NonWordByte, /xF5 => NonWordByte, /xF6 => NonWordByte, /xF7 => NonWordByte, /xF8 => NonWordByte, /xF9 => NonWordByte, /xFA => NonWordByte, /xFB => NonWordByte, /xFC => NonWordByte, /xFD => NonWordByte, /xFE => NonWordByte, /xFF => NonWordByte}, + classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]), + quitset: ByteSet { + bits: {}, + }, + cache_capacity: 10485760, + }, + }, + ), + ), + ), + dfa: DFA( + None, + ), + }, + info: RegexInfo( + RegexInfoI { + config: Config { + match_kind: Some( + All, + ), + utf8_empty: Some( + false, + ), + autopre: None, + pre: None, + which_captures: None, + nfa_size_limit: Some( + Some( + 10485760, + ), + ), + onepass_size_limit: None, + hybrid_cache_capacity: Some( + 10485760, + ), + hybrid: None, + dfa: None, + dfa_size_limit: None, + dfa_state_limit: None, + onepass: None, + backtrack: None, + byte_classes: None, + line_terminator: None, + }, + props: [], + props_union: Properties( + PropertiesI { + minimum_len: None, + maximum_len: None, + look_set: ∅, + look_set_prefix: ∅, + look_set_suffix: ∅, + look_set_prefix_any: ∅, + look_set_suffix_any: ∅, + utf8: true, + explicit_captures_len: 0, + static_explicit_captures_len: None, + literal: false, + alternation_literal: true, + }, + ), + }, + ), + }, + pool: Pool( + Pool { + stacks: [ + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + ], + owner: 0, + owner_val: UnsafeCell { .. }, + }, + ), + }, + map: [], + patset: Pool( + Pool { + stacks: [ + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + ], + owner: 0, + owner_val: UnsafeCell { .. }, + }, + ), + }, + ), + ], + }, + poisoned: false, + .. + }, + }, + }, + run_conditions: RunConditions { + never: false, + always: false, + ignore_broken_dep_steps: false, + ignore_missing_outputs: true, + }, + pipeline_rundir: XvcPath( + "", + ), + terminate_timeout_processes: true, + algorithm: Blake3, + command_process: RwLock { + data: CommandProcess { + environment: {}, + step: XvcStep { + name: "average-age", + }, + step_command: XvcStepCommand { + command: "sqlite3 people.db 'SELECT AVG(Age) FROM People;'", + }, + birth: None, + process: None, + stdout_sender: Sender { .. }, + stderr_sender: Sender { .. }, + stdout_receiver: Receiver { .. }, + stderr_receiver: Receiver { .. }, + }, + poisoned: false, + .. + }, + available_process_slots: RwLock { + data: 4, + poisoned: false, + .. + }, + process_poll_milliseconds: 10, + dependency_diffs: RwLock { + data: HStore { + map: { + XvcEntity( + 3, + 9022053985516504033, + ): RecordMissing { + actual: SqliteQueryDigest( + SqliteQueryDep { + path: XvcPath( + "people.db", + ), + query: "SELECT count(*) FROM People;", + query_digest: Some( + ContentDigest( + XvcDigest { + algorithm: Blake3, + digest: [ + 11, + 225, + 214, + 145, + 174, + 151, + 99, + 217, + 216, + 197, + 211, + 26, + 216, + 218, + 115, + 209, + 161, + 95, + 15, + 52, + 174, + 24, + 193, + 209, + 218, + 91, + 154, + 207, + 247, + 217, + 245, + 9, + ], + }, + ), + ), + xvc_metadata: Some( + XvcMetadata { + file_type: File, + size: Some( + 8192, + ), + modified: Some( + SystemTime { + tv_sec: 1722024927, + tv_nsec: 262703931, + }, + ), + }, + ), + }, + ), + }, + }, + }, + poisoned: false, + .. + }, + output_diffs: RwLock { + data: HStore { + map: {}, + }, + poisoned: false, + .. + }, + step_e: XvcEntity( + 2, + 9270302598983016314, + ), + step: XvcStep { + name: "average-age", + }, + step_command: XvcStepCommand { + command: "sqlite3 people.db 'SELECT AVG(Age) FROM People;'", + }, + current_states: RwLock { + data: HStore { + map: { + XvcEntity( + 2, + 9270302598983016314, + ): WaitingToRun( + FromDiffsHasChanged, + ), + }, + }, + poisoned: false, + .. + }, + step_timeout: 10000s, + all_steps: HStore { + map: { + XvcEntity( + 2, + 9270302598983016314, + ): XvcStep { + name: "average-age", + }, + }, + }, + recorded_dependencies: R1NStore { + parents: XvcStore { + map: { + XvcEntity( + 2, + 9270302598983016314, + ): XvcStep { + name: "average-age", + }, + }, + entity_index: { + XvcStep { + name: "average-age", + }: [ + XvcEntity( + 2, + 9270302598983016314, + ), + ], + }, + previous: EventLog( + [ + Add { + entity: XvcEntity( + 2, + 9270302598983016314, + ), + value: XvcStep { + name: "average-age", + }, + }, + Add { + entity: XvcEntity( + 2, + 9270302598983016314, + ), + value: XvcStep { + name: "average-age", + }, + }, + ], + ), + current: EventLog( + [], + ), + }, + children: XvcStore { + map: { + XvcEntity( + 3, + 9022053985516504033, + ): SqliteQueryDigest( + SqliteQueryDep { + path: XvcPath( + "people.db", + ), + query: "SELECT count(*) FROM People;", + query_digest: None, + xvc_metadata: None, + }, + ), + }, + entity_index: { + SqliteQueryDigest( + SqliteQueryDep { + path: XvcPath( + "people.db", + ), + query: "SELECT count(*) FROM People;", + query_digest: None, + xvc_metadata: None, + }, + ): [ + XvcEntity( + 3, + 9022053985516504033, + ), + ], + }, + previous: EventLog( + [ + Add { + entity: XvcEntity( + 3, + 9022053985516504033, + ), + value: SqliteQueryDigest( + SqliteQueryDep { + path: XvcPath( + "people.db", + ), + query: "SELECT count(*) FROM People;", + query_digest: None, + xvc_metadata: None, + }, + ), + }, + ], + ), + current: EventLog( + [], + ), + }, + child_parents: XvcStore { + map: { + XvcEntity( + 3, + 9022053985516504033, + ): ChildEntity( + XvcEntity( + 2, + 9270302598983016314, + ), + PhantomData, + PhantomData, + ), + }, + entity_index: { + ChildEntity( + XvcEntity( + 2, + 9270302598983016314, + ), + PhantomData, + PhantomData, + ): [ + XvcEntity( + 3, + 9022053985516504033, + ), + ], + }, + previous: EventLog( + [ + Add { + entity: XvcEntity( + 3, + 9022053985516504033, + ), + value: ChildEntity( + XvcEntity( + 2, + 9270302598983016314, + ), + PhantomData, + PhantomData, + ), + }, + ], + ), + current: EventLog( + [], + ), + }, + }, + step_dependencies: {}, + step_outputs: HStore { + map: {}, + }, +} +[TRACE][pipeline/src/pipeline/mod.rs::770] step.name: "average-age" +[TRACE][pipeline/src/pipeline/mod.rs::771] &r_next_state: Running( + FromStartProcess, +) +[TRACE][pipeline/src/pipeline/mod.rs::773] &step_state: Running( + FromStartProcess, +) +[TRACE][pipeline/src/pipeline/mod.rs::660] &step_state: Running( + FromStartProcess, +) +[TRACE][pipeline/src/pipeline/command.rs::96] self.environment: {} +[TRACE][pipeline/src/pipeline/mod.rs::770] step.name: "average-age" +[TRACE][pipeline/src/pipeline/mod.rs::771] &r_next_state: Running( + FromWaitProcess, +) +[TRACE][pipeline/src/pipeline/mod.rs::773] &step_state: Running( + FromWaitProcess, +) +[TRACE][pipeline/src/pipeline/mod.rs::660] &step_state: Running( + FromWaitProcess, +) +[TRACE][pipeline/src/pipeline/mod.rs::1418] params: StepStateParams { + xvc_root: XvcRootInner { + absolute_path: AbsolutePath( + "[CWD]", + ), + xvc_dir: AbsolutePath( + "[CWD]/.xvc", + ), + store_dir: AbsolutePath( + "[CWD]/.xvc/store", + ), + config: XvcConfig { + current_dir: XvcConfigOption { + source: Runtime, + option: AbsolutePath( + "[CWD]", + ), + }, + config_maps: [ + XvcConfigMap { + source: Default, + map: { + "pipeline.default_params_file": String( + "params.yaml", + ), + "file.carry-in.no_parallel": Boolean( + false, + ), + "file.carry-in.force": Boolean( + false, + ), + "core.verbosity": String( + "error", + ), + "file.recheck.method": String( + "copy", + ), + "file.list.sort": String( + "name-desc", + ), + "git.use_git": Boolean( + true, + ), + "pipeline.current_pipeline": String( + "default", + ), + "core.guid": String( + "919ce8649f3e32f7", + ), + "git.auto_commit": Boolean( + true, + ), + "pipeline.default": String( + "default", + ), + "pipeline.process_pool_size": Integer( + 4, + ), + "git.command": String( + "git", + ), + "file.track.no_commit": Boolean( + false, + ), + "file.track.force": Boolean( + false, + ), + "file.list.show_dot_files": Boolean( + false, + ), + "file.list.recursive": Boolean( + false, + ), + "file.list.format": String( + "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", + ), + "cache.algorithm": String( + "blake3", + ), + "git.auto_stage": Boolean( + false, + ), + "file.list.no_summary": Boolean( + false, + ), + "file.track.text_or_binary": String( + "auto", + ), + "file.track.no_parallel": Boolean( + false, + ), + }, + }, + XvcConfigMap { + source: Project, + map: { + "core.guid": String( + "59a2d95849757118", + ), + "cache.algorithm": String( + "blake3", + ), + "file.list.recursive": Boolean( + false, + ), + "git.command": String( + "git", + ), + "pipeline.default_params_file": String( + "params.yaml", + ), + "file.track.force": Boolean( + false, + ), + "file.track.no_parallel": Boolean( + false, + ), + "git.use_git": Boolean( + true, + ), + "file.track.no_commit": Boolean( + false, + ), + "file.recheck.method": String( + "copy", + ), + "file.list.no_summary": Boolean( + false, + ), + "pipeline.current_pipeline": String( + "default", + ), + "git.auto_stage": Boolean( + false, + ), + "git.auto_commit": Boolean( + true, + ), + "file.list.show_dot_files": Boolean( + false, + ), + "file.list.format": String( + "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", + ), + "file.list.sort": String( + "name-desc", + ), + "pipeline.process_pool_size": Integer( + 4, + ), + "file.carry-in.no_parallel": Boolean( + false, + ), + "pipeline.default": String( + "default", + ), + "file.carry-in.force": Boolean( + false, + ), + "core.verbosity": String( + "error", + ), + "file.track.text_or_binary": String( + "auto", + ), + }, + }, + XvcConfigMap { + source: Local, + map: {}, + }, + XvcConfigMap { + source: Environment, + map: { + "TRYCMD_TESTS": String( + "storage,file,pipeline,core", + ), + }, + }, + XvcConfigMap { + source: CommandLine, + map: { + "core.quiet": Boolean( + false, + ), + "core.verbosity": String( + "debug", + ), + }, + }, + ], + the_config: { + "pipeline.default_params_file": XvcConfigValue { + source: Project, + value: String( + "params.yaml", + ), + }, + "file.list.sort": XvcConfigValue { + source: Project, + value: String( + "name-desc", + ), + }, + "git.use_git": XvcConfigValue { + source: Project, + value: Boolean( + true, + ), + }, + "git.auto_commit": XvcConfigValue { + source: Project, + value: Boolean( + true, + ), + }, + "core.verbosity": XvcConfigValue { + source: CommandLine, + value: String( + "debug", + ), + }, + "git.auto_stage": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "file.carry-in.force": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "pipeline.current_pipeline": XvcConfigValue { + source: Project, + value: String( + "default", + ), + }, + "cache.algorithm": XvcConfigValue { + source: Project, + value: String( + "blake3", + ), + }, + "pipeline.process_pool_size": XvcConfigValue { + source: Project, + value: Integer( + 4, + ), + }, + "file.list.no_summary": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "file.track.no_parallel": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "file.recheck.method": XvcConfigValue { + source: Project, + value: String( + "copy", + ), + }, + "core.quiet": XvcConfigValue { + source: CommandLine, + value: Boolean( + false, + ), + }, + "file.carry-in.no_parallel": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "core.guid": XvcConfigValue { + source: Project, + value: String( + "59a2d95849757118", + ), + }, + "pipeline.default": XvcConfigValue { + source: Project, + value: String( + "default", + ), + }, + "git.command": XvcConfigValue { + source: Project, + value: String( + "git", + ), + }, + "file.track.force": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "file.track.no_commit": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "file.list.recursive": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "file.list.format": XvcConfigValue { + source: Project, + value: String( + "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", + ), + }, + "file.track.text_or_binary": XvcConfigValue { + source: Project, + value: String( + "auto", + ), + }, + "TRYCMD_TESTS": XvcConfigValue { + source: Environment, + value: String( + "storage,file,pipeline,core", + ), + }, + "file.list.show_dot_files": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + }, + init_params: XvcConfigParams { + default_configuration: " +[core] +# The repository id. Please do not delete or change it. +# This is used to identify the repository and generate paths in storages. +# In the future it may be used to in other ways. +guid = /"919ce8649f3e32f7/" +# Default verbosity level. +# One of /"error/", /"warn/", /"info/" +verbosity = /"error/" + +[git] +# Automate git operations. +# Turning this off leads Xvc to behave as if it's not in a Git repository. +# Not recommended unless you're really not using Git +use_git = true +# Command to run Git process. +# You can set this to an absolute path to specify an executable +# If set to a non-absolute path, the executable will be searched in $PATH. +command = /"git/" + +# Commit changes in .xvc/ directory after commands. +# You can set this to false if you want to commit manually. +auto_commit = true + +# Stage changes in .xvc/ directory without committing. +# auto_commit implies auto_stage. +# If you want to commit manually but don't want to stage after individual Xvc commands, you can set this to true. +auto_stage = false + +[cache] +# The hash algorithm used for the cache. +# It may take blake3, blake2, sha2 or sha3 as values. +# All algorithms are selected to produce 256-bit hashes, so sha2 means SHA2-256, blake2 means BLAKE2s, etc. +# The cache path is produced by prepending algorithm name to the cache. +# Blake3 files are in .xvc/b3/, while sha2 files are in .xvc/s2/ etc. +algorithm = /"blake3/" + +[file] + +[file.track] + +# Don't move file content to cache after xvc file track +no_commit = false +# Force to track files even if they are already tracked. +force = false + +# Xvc calculates file content digest differently for text and binary files. +# This option controls whether to treat files as text or binary. +# It may take auto, text or binary as values. +# Auto check each file individually and treat it as text if it's text. +text_or_binary = /"auto/" + +# Don't use parallelism in track operations. +# Note that some of the operations are implemented in parallel by default, and this option affects some heavier operations. +no_parallel = false + +[file.list] + +# Format for `xvc file list` rows. You can reorder or remove columns. +# The following are the keys for each row: +# - {acd64}: actual content digest. All 64 digits from the workspace file's content. +# - {acd8}: actual content digest. First 8 digits the file content digest. +# - {aft}: actual file type. Whether the entry is a file (F), directory (D), +# symlink (S), hardlink (H) or reflink (R). +# - {asz}: actual size. The size of the workspace file in bytes. It uses MB, +# GB and TB to represent sizes larger than 1MB. +# - {ats}: actual timestamp. The timestamp of the workspace file. +# - {cst}: cache status. One of /"=/", /">/", /"/", /" 0 + F 000001: + >000002: /x00 => 2 + 000003: /x00 => 0 + match kind: Standard + prefilter: false + state length: 4 + pattern length: 0 + shortest pattern length: 18446744073709551615 + longest pattern length: 0 + alphabet length: 1 + stride: 1 + byte classes: ByteClasses(0 => [0-255]) + memory usage: 16 + ) + , + ), + map: [], + longest: 0, + }, + ), + Prefix( + PrefixStrategy { + matcher: AhoCorasick( + dfa::DFA( + D 000000: /x00 => 0 + F 000001: + >000002: /x00 => 2 + 000003: /x00 => 0 + match kind: Standard + prefilter: false + state length: 4 + pattern length: 0 + shortest pattern length: 18446744073709551615 + longest pattern length: 0 + alphabet length: 1 + stride: 1 + byte classes: ByteClasses(0 => [0-255]) + memory usage: 16 + ) + , + ), + map: [], + longest: 0, + }, + ), + RequiredExtension( + RequiredExtensionStrategy( + {}, + ), + ), + Regex( + RegexSetStrategy { + matcher: Regex { + imp: RegexI { + strat: Core { + info: RegexInfo( + RegexInfoI { + config: Config { + match_kind: Some( + All, + ), + utf8_empty: Some( + false, + ), + autopre: None, + pre: None, + which_captures: None, + nfa_size_limit: Some( + Some( + 10485760, + ), + ), + onepass_size_limit: None, + hybrid_cache_capacity: Some( + 10485760, + ), + hybrid: None, + dfa: None, + dfa_size_limit: None, + dfa_state_limit: None, + onepass: None, + backtrack: None, + byte_classes: None, + line_terminator: None, + }, + props: [], + props_union: Properties( + PropertiesI { + minimum_len: None, + maximum_len: None, + look_set: ∅, + look_set_prefix: ∅, + look_set_suffix: ∅, + look_set_prefix_any: ∅, + look_set_suffix_any: ∅, + utf8: true, + explicit_captures_len: 0, + static_explicit_captures_len: None, + literal: false, + alternation_literal: true, + }, + ), + }, + ), + pre: None, + nfa: thompson::NFA( + ^000000: FAIL + + transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) + ) + , + nfarev: Some( + thompson::NFA( + ^000000: FAIL + + transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) + ) + , + ), + pikevm: PikeVM( + PikeVMEngine( + PikeVM { + config: Config { + match_kind: Some( + All, + ), + pre: Some( + None, + ), + }, + nfa: thompson::NFA( + ^000000: FAIL + + transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) + ) + , + }, + ), + ), + backtrack: BoundedBacktracker( + None, + ), + onepass: OnePass( + None, + ), + hybrid: Hybrid( + Some( + HybridEngine( + Regex { + forward: DFA { + config: Config { + match_kind: Some( + All, + ), + pre: Some( + None, + ), + starts_for_each_pattern: Some( + true, + ), + byte_classes: Some( + true, + ), + unicode_word_boundary: Some( + true, + ), + quitset: None, + specialize_start_states: Some( + false, + ), + cache_capacity: Some( + 10485760, + ), + skip_cache_capacity_check: Some( + false, + ), + minimum_cache_clear_count: Some( + Some( + 3, + ), + ), + minimum_bytes_per_state: Some( + Some( + 10, + ), + ), + }, + nfa: thompson::NFA( + ^000000: FAIL + + transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) + ) + , + stride2: 1, + start_map: StartByteMap{/x00 => NonWordByte, /x01 => NonWordByte, /x02 => NonWordByte, /x03 => NonWordByte, /x04 => NonWordByte, /x05 => NonWordByte, /x06 => NonWordByte, /x07 => NonWordByte, /x08 => NonWordByte, /t => NonWordByte, + => LineLF, /x0B => NonWordByte, /x0C => NonWordByte, /r => LineCR, /x0E => NonWordByte, /x0F => NonWordByte, /x10 => NonWordByte, /x11 => NonWordByte, /x12 => NonWordByte, /x13 => NonWordByte, /x14 => NonWordByte, /x15 => NonWordByte, /x16 => NonWordByte, /x17 => NonWordByte, /x18 => NonWordByte, /x19 => NonWordByte, /x1A => NonWordByte, /x1B => NonWordByte, /x1C => NonWordByte, /x1D => NonWordByte, /x1E => NonWordByte, /x1F => NonWordByte, ' ' => NonWordByte, ! => NonWordByte, /" => NonWordByte, # => NonWordByte, $ => NonWordByte, % => NonWordByte, & => NonWordByte, /' => NonWordByte, ( => NonWordByte, ) => NonWordByte, * => NonWordByte, + => NonWordByte, , => NonWordByte, - => NonWordByte, . => NonWordByte, / => NonWordByte, 0 => WordByte, 1 => WordByte, 2 => WordByte, 3 => WordByte, 4 => WordByte, 5 => WordByte, 6 => WordByte, 7 => WordByte, 8 => WordByte, 9 => WordByte, : => NonWordByte, ; => NonWordByte, < => NonWordByte, = => NonWordByte, > => NonWordByte, ? => NonWordByte, @ => NonWordByte, A => WordByte, B => WordByte, C => WordByte, D => WordByte, E => WordByte, F => WordByte, G => WordByte, H => WordByte, I => WordByte, J => WordByte, K => WordByte, L => WordByte, M => WordByte, N => WordByte, O => WordByte, P => WordByte, Q => WordByte, R => WordByte, S => WordByte, T => WordByte, U => WordByte, V => WordByte, W => WordByte, X => WordByte, Y => WordByte, Z => WordByte, [ => NonWordByte, // => NonWordByte, ] => NonWordByte, ^ => NonWordByte, _ => WordByte, ` => NonWordByte, a => WordByte, b => WordByte, c => WordByte, d => WordByte, e => WordByte, f => WordByte, g => WordByte, h => WordByte, i => WordByte, j => WordByte, k => WordByte, l => WordByte, m => WordByte, n => WordByte, o => WordByte, p => WordByte, q => WordByte, r => WordByte, s => WordByte, t => WordByte, u => WordByte, v => WordByte, w => WordByte, x => WordByte, y => WordByte, z => WordByte, { => NonWordByte, | => NonWordByte, } => NonWordByte, ~ => NonWordByte, /x7F => NonWordByte, /x80 => NonWordByte, /x81 => NonWordByte, /x82 => NonWordByte, /x83 => NonWordByte, /x84 => NonWordByte, /x85 => NonWordByte, /x86 => NonWordByte, /x87 => NonWordByte, /x88 => NonWordByte, /x89 => NonWordByte, /x8A => NonWordByte, /x8B => NonWordByte, /x8C => NonWordByte, /x8D => NonWordByte, /x8E => NonWordByte, /x8F => NonWordByte, /x90 => NonWordByte, /x91 => NonWordByte, /x92 => NonWordByte, /x93 => NonWordByte, /x94 => NonWordByte, /x95 => NonWordByte, /x96 => NonWordByte, /x97 => NonWordByte, /x98 => NonWordByte, /x99 => NonWordByte, /x9A => NonWordByte, /x9B => NonWordByte, /x9C => NonWordByte, /x9D => NonWordByte, /x9E => NonWordByte, /x9F => NonWordByte, /xA0 => NonWordByte, /xA1 => NonWordByte, /xA2 => NonWordByte, /xA3 => NonWordByte, /xA4 => NonWordByte, /xA5 => NonWordByte, /xA6 => NonWordByte, /xA7 => NonWordByte, /xA8 => NonWordByte, /xA9 => NonWordByte, /xAA => NonWordByte, /xAB => NonWordByte, /xAC => NonWordByte, /xAD => NonWordByte, /xAE => NonWordByte, /xAF => NonWordByte, /xB0 => NonWordByte, /xB1 => NonWordByte, /xB2 => NonWordByte, /xB3 => NonWordByte, /xB4 => NonWordByte, /xB5 => NonWordByte, /xB6 => NonWordByte, /xB7 => NonWordByte, /xB8 => NonWordByte, /xB9 => NonWordByte, /xBA => NonWordByte, /xBB => NonWordByte, /xBC => NonWordByte, /xBD => NonWordByte, /xBE => NonWordByte, /xBF => NonWordByte, /xC0 => NonWordByte, /xC1 => NonWordByte, /xC2 => NonWordByte, /xC3 => NonWordByte, /xC4 => NonWordByte, /xC5 => NonWordByte, /xC6 => NonWordByte, /xC7 => NonWordByte, /xC8 => NonWordByte, /xC9 => NonWordByte, /xCA => NonWordByte, /xCB => NonWordByte, /xCC => NonWordByte, /xCD => NonWordByte, /xCE => NonWordByte, /xCF => NonWordByte, /xD0 => NonWordByte, /xD1 => NonWordByte, /xD2 => NonWordByte, /xD3 => NonWordByte, /xD4 => NonWordByte, /xD5 => NonWordByte, /xD6 => NonWordByte, /xD7 => NonWordByte, /xD8 => NonWordByte, /xD9 => NonWordByte, /xDA => NonWordByte, /xDB => NonWordByte, /xDC => NonWordByte, /xDD => NonWordByte, /xDE => NonWordByte, /xDF => NonWordByte, /xE0 => NonWordByte, /xE1 => NonWordByte, /xE2 => NonWordByte, /xE3 => NonWordByte, /xE4 => NonWordByte, /xE5 => NonWordByte, /xE6 => NonWordByte, /xE7 => NonWordByte, /xE8 => NonWordByte, /xE9 => NonWordByte, /xEA => NonWordByte, /xEB => NonWordByte, /xEC => NonWordByte, /xED => NonWordByte, /xEE => NonWordByte, /xEF => NonWordByte, /xF0 => NonWordByte, /xF1 => NonWordByte, /xF2 => NonWordByte, /xF3 => NonWordByte, /xF4 => NonWordByte, /xF5 => NonWordByte, /xF6 => NonWordByte, /xF7 => NonWordByte, /xF8 => NonWordByte, /xF9 => NonWordByte, /xFA => NonWordByte, /xFB => NonWordByte, /xFC => NonWordByte, /xFD => NonWordByte, /xFE => NonWordByte, /xFF => NonWordByte}, + classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]), + quitset: ByteSet { + bits: {}, + }, + cache_capacity: 10485760, + }, + reverse: DFA { + config: Config { + match_kind: Some( + All, + ), + pre: Some( + None, + ), + starts_for_each_pattern: Some( + true, + ), + byte_classes: Some( + true, + ), + unicode_word_boundary: Some( + true, + ), + quitset: None, + specialize_start_states: Some( + false, + ), + cache_capacity: Some( + 10485760, + ), + skip_cache_capacity_check: Some( + false, + ), + minimum_cache_clear_count: Some( + Some( + 3, + ), + ), + minimum_bytes_per_state: Some( + Some( + 10, + ), + ), + }, + nfa: thompson::NFA( + ^000000: FAIL + + transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) + ) + , + stride2: 1, + start_map: StartByteMap{/x00 => NonWordByte, /x01 => NonWordByte, /x02 => NonWordByte, /x03 => NonWordByte, /x04 => NonWordByte, /x05 => NonWordByte, /x06 => NonWordByte, /x07 => NonWordByte, /x08 => NonWordByte, /t => NonWordByte, + => LineLF, /x0B => NonWordByte, /x0C => NonWordByte, /r => LineCR, /x0E => NonWordByte, /x0F => NonWordByte, /x10 => NonWordByte, /x11 => NonWordByte, /x12 => NonWordByte, /x13 => NonWordByte, /x14 => NonWordByte, /x15 => NonWordByte, /x16 => NonWordByte, /x17 => NonWordByte, /x18 => NonWordByte, /x19 => NonWordByte, /x1A => NonWordByte, /x1B => NonWordByte, /x1C => NonWordByte, /x1D => NonWordByte, /x1E => NonWordByte, /x1F => NonWordByte, ' ' => NonWordByte, ! => NonWordByte, /" => NonWordByte, # => NonWordByte, $ => NonWordByte, % => NonWordByte, & => NonWordByte, /' => NonWordByte, ( => NonWordByte, ) => NonWordByte, * => NonWordByte, + => NonWordByte, , => NonWordByte, - => NonWordByte, . => NonWordByte, / => NonWordByte, 0 => WordByte, 1 => WordByte, 2 => WordByte, 3 => WordByte, 4 => WordByte, 5 => WordByte, 6 => WordByte, 7 => WordByte, 8 => WordByte, 9 => WordByte, : => NonWordByte, ; => NonWordByte, < => NonWordByte, = => NonWordByte, > => NonWordByte, ? => NonWordByte, @ => NonWordByte, A => WordByte, B => WordByte, C => WordByte, D => WordByte, E => WordByte, F => WordByte, G => WordByte, H => WordByte, I => WordByte, J => WordByte, K => WordByte, L => WordByte, M => WordByte, N => WordByte, O => WordByte, P => WordByte, Q => WordByte, R => WordByte, S => WordByte, T => WordByte, U => WordByte, V => WordByte, W => WordByte, X => WordByte, Y => WordByte, Z => WordByte, [ => NonWordByte, // => NonWordByte, ] => NonWordByte, ^ => NonWordByte, _ => WordByte, ` => NonWordByte, a => WordByte, b => WordByte, c => WordByte, d => WordByte, e => WordByte, f => WordByte, g => WordByte, h => WordByte, i => WordByte, j => WordByte, k => WordByte, l => WordByte, m => WordByte, n => WordByte, o => WordByte, p => WordByte, q => WordByte, r => WordByte, s => WordByte, t => WordByte, u => WordByte, v => WordByte, w => WordByte, x => WordByte, y => WordByte, z => WordByte, { => NonWordByte, | => NonWordByte, } => NonWordByte, ~ => NonWordByte, /x7F => NonWordByte, /x80 => NonWordByte, /x81 => NonWordByte, /x82 => NonWordByte, /x83 => NonWordByte, /x84 => NonWordByte, /x85 => NonWordByte, /x86 => NonWordByte, /x87 => NonWordByte, /x88 => NonWordByte, /x89 => NonWordByte, /x8A => NonWordByte, /x8B => NonWordByte, /x8C => NonWordByte, /x8D => NonWordByte, /x8E => NonWordByte, /x8F => NonWordByte, /x90 => NonWordByte, /x91 => NonWordByte, /x92 => NonWordByte, /x93 => NonWordByte, /x94 => NonWordByte, /x95 => NonWordByte, /x96 => NonWordByte, /x97 => NonWordByte, /x98 => NonWordByte, /x99 => NonWordByte, /x9A => NonWordByte, /x9B => NonWordByte, /x9C => NonWordByte, /x9D => NonWordByte, /x9E => NonWordByte, /x9F => NonWordByte, /xA0 => NonWordByte, /xA1 => NonWordByte, /xA2 => NonWordByte, /xA3 => NonWordByte, /xA4 => NonWordByte, /xA5 => NonWordByte, /xA6 => NonWordByte, /xA7 => NonWordByte, /xA8 => NonWordByte, /xA9 => NonWordByte, /xAA => NonWordByte, /xAB => NonWordByte, /xAC => NonWordByte, /xAD => NonWordByte, /xAE => NonWordByte, /xAF => NonWordByte, /xB0 => NonWordByte, /xB1 => NonWordByte, /xB2 => NonWordByte, /xB3 => NonWordByte, /xB4 => NonWordByte, /xB5 => NonWordByte, /xB6 => NonWordByte, /xB7 => NonWordByte, /xB8 => NonWordByte, /xB9 => NonWordByte, /xBA => NonWordByte, /xBB => NonWordByte, /xBC => NonWordByte, /xBD => NonWordByte, /xBE => NonWordByte, /xBF => NonWordByte, /xC0 => NonWordByte, /xC1 => NonWordByte, /xC2 => NonWordByte, /xC3 => NonWordByte, /xC4 => NonWordByte, /xC5 => NonWordByte, /xC6 => NonWordByte, /xC7 => NonWordByte, /xC8 => NonWordByte, /xC9 => NonWordByte, /xCA => NonWordByte, /xCB => NonWordByte, /xCC => NonWordByte, /xCD => NonWordByte, /xCE => NonWordByte, /xCF => NonWordByte, /xD0 => NonWordByte, /xD1 => NonWordByte, /xD2 => NonWordByte, /xD3 => NonWordByte, /xD4 => NonWordByte, /xD5 => NonWordByte, /xD6 => NonWordByte, /xD7 => NonWordByte, /xD8 => NonWordByte, /xD9 => NonWordByte, /xDA => NonWordByte, /xDB => NonWordByte, /xDC => NonWordByte, /xDD => NonWordByte, /xDE => NonWordByte, /xDF => NonWordByte, /xE0 => NonWordByte, /xE1 => NonWordByte, /xE2 => NonWordByte, /xE3 => NonWordByte, /xE4 => NonWordByte, /xE5 => NonWordByte, /xE6 => NonWordByte, /xE7 => NonWordByte, /xE8 => NonWordByte, /xE9 => NonWordByte, /xEA => NonWordByte, /xEB => NonWordByte, /xEC => NonWordByte, /xED => NonWordByte, /xEE => NonWordByte, /xEF => NonWordByte, /xF0 => NonWordByte, /xF1 => NonWordByte, /xF2 => NonWordByte, /xF3 => NonWordByte, /xF4 => NonWordByte, /xF5 => NonWordByte, /xF6 => NonWordByte, /xF7 => NonWordByte, /xF8 => NonWordByte, /xF9 => NonWordByte, /xFA => NonWordByte, /xFB => NonWordByte, /xFC => NonWordByte, /xFD => NonWordByte, /xFE => NonWordByte, /xFF => NonWordByte}, + classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]), + quitset: ByteSet { + bits: {}, + }, + cache_capacity: 10485760, + }, + }, + ), + ), + ), + dfa: DFA( + None, + ), + }, + info: RegexInfo( + RegexInfoI { + config: Config { + match_kind: Some( + All, + ), + utf8_empty: Some( + false, + ), + autopre: None, + pre: None, + which_captures: None, + nfa_size_limit: Some( + Some( + 10485760, + ), + ), + onepass_size_limit: None, + hybrid_cache_capacity: Some( + 10485760, + ), + hybrid: None, + dfa: None, + dfa_size_limit: None, + dfa_state_limit: None, + onepass: None, + backtrack: None, + byte_classes: None, + line_terminator: None, + }, + props: [], + props_union: Properties( + PropertiesI { + minimum_len: None, + maximum_len: None, + look_set: ∅, + look_set_prefix: ∅, + look_set_suffix: ∅, + look_set_prefix_any: ∅, + look_set_suffix_any: ∅, + utf8: true, + explicit_captures_len: 0, + static_explicit_captures_len: None, + literal: false, + alternation_literal: true, + }, + ), + }, + ), + }, + pool: Pool( + Pool { + stacks: [ + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + ], + owner: 0, + owner_val: UnsafeCell { .. }, + }, + ), + }, + map: [], + patset: Pool( + Pool { + stacks: [ + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + ], + owner: 0, + owner_val: UnsafeCell { .. }, + }, + ), + }, + ), + ], + }, + poisoned: false, + .. + }, + }, + }, + run_conditions: RunConditions { + never: false, + always: false, + ignore_broken_dep_steps: false, + ignore_missing_outputs: true, + }, + pipeline_rundir: XvcPath( + "", + ), + terminate_timeout_processes: true, + algorithm: Blake3, + command_process: RwLock { + data: CommandProcess { + environment: {}, + step: XvcStep { + name: "average-age", + }, + step_command: XvcStepCommand { + command: "sqlite3 people.db 'SELECT AVG(Age) FROM People;'", + }, + birth: Some( + Instant { + tv_sec: 834347, + tv_nsec: 663124791, + }, + ), + process: Some( + Popen { + stdin: None, + stdout: Some( + File { + fd: 5, + read: true, + write: false, + }, + ), + stderr: Some( + File { + fd: 7, + read: true, + write: false, + }, + ), + child_state: Running { + pid: 21337, + ext: (), + }, + detached: true, + }, + ), + stdout_sender: Sender { .. }, + stderr_sender: Sender { .. }, + stdout_receiver: Receiver { .. }, + stderr_receiver: Receiver { .. }, + }, + poisoned: false, + .. + }, + available_process_slots: RwLock { + data: 3, + poisoned: false, + .. + }, + process_poll_milliseconds: 10, + dependency_diffs: RwLock { + data: HStore { + map: { + XvcEntity( + 3, + 9022053985516504033, + ): RecordMissing { + actual: SqliteQueryDigest( + SqliteQueryDep { + path: XvcPath( + "people.db", + ), + query: "SELECT count(*) FROM People;", + query_digest: Some( + ContentDigest( + XvcDigest { + algorithm: Blake3, + digest: [ + 11, + 225, + 214, + 145, + 174, + 151, + 99, + 217, + 216, + 197, + 211, + 26, + 216, + 218, + 115, + 209, + 161, + 95, + 15, + 52, + 174, + 24, + 193, + 209, + 218, + 91, + 154, + 207, + 247, + 217, + 245, + 9, + ], + }, + ), + ), + xvc_metadata: Some( + XvcMetadata { + file_type: File, + size: Some( + 8192, + ), + modified: Some( + SystemTime { + tv_sec: 1722024927, + tv_nsec: 262703931, + }, + ), + }, + ), + }, + ), + }, + }, + }, + poisoned: false, + .. + }, + output_diffs: RwLock { + data: HStore { + map: {}, + }, + poisoned: false, + .. + }, + step_e: XvcEntity( + 2, + 9270302598983016314, + ), + step: XvcStep { + name: "average-age", + }, + step_command: XvcStepCommand { + command: "sqlite3 people.db 'SELECT AVG(Age) FROM People;'", + }, + current_states: RwLock { + data: HStore { + map: { + XvcEntity( + 2, + 9270302598983016314, + ): Running( + FromWaitProcess, + ), + }, + }, + poisoned: false, + .. + }, + step_timeout: 10000s, + all_steps: HStore { + map: { + XvcEntity( + 2, + 9270302598983016314, + ): XvcStep { + name: "average-age", + }, + }, + }, + recorded_dependencies: R1NStore { + parents: XvcStore { + map: { + XvcEntity( + 2, + 9270302598983016314, + ): XvcStep { + name: "average-age", + }, + }, + entity_index: { + XvcStep { + name: "average-age", + }: [ + XvcEntity( + 2, + 9270302598983016314, + ), + ], + }, + previous: EventLog( + [ + Add { + entity: XvcEntity( + 2, + 9270302598983016314, + ), + value: XvcStep { + name: "average-age", + }, + }, + Add { + entity: XvcEntity( + 2, + 9270302598983016314, + ), + value: XvcStep { + name: "average-age", + }, + }, + ], + ), + current: EventLog( + [], + ), + }, + children: XvcStore { + map: { + XvcEntity( + 3, + 9022053985516504033, + ): SqliteQueryDigest( + SqliteQueryDep { + path: XvcPath( + "people.db", + ), + query: "SELECT count(*) FROM People;", + query_digest: None, + xvc_metadata: None, + }, + ), + }, + entity_index: { + SqliteQueryDigest( + SqliteQueryDep { + path: XvcPath( + "people.db", + ), + query: "SELECT count(*) FROM People;", + query_digest: None, + xvc_metadata: None, + }, + ): [ + XvcEntity( + 3, + 9022053985516504033, + ), + ], + }, + previous: EventLog( + [ + Add { + entity: XvcEntity( + 3, + 9022053985516504033, + ), + value: SqliteQueryDigest( + SqliteQueryDep { + path: XvcPath( + "people.db", + ), + query: "SELECT count(*) FROM People;", + query_digest: None, + xvc_metadata: None, + }, + ), + }, + ], + ), + current: EventLog( + [], + ), + }, + child_parents: XvcStore { + map: { + XvcEntity( + 3, + 9022053985516504033, + ): ChildEntity( + XvcEntity( + 2, + 9270302598983016314, + ), + PhantomData, + PhantomData, + ), + }, + entity_index: { + ChildEntity( + XvcEntity( + 2, + 9270302598983016314, + ), + PhantomData, + PhantomData, + ): [ + XvcEntity( + 3, + 9022053985516504033, + ), + ], + }, + previous: EventLog( + [ + Add { + entity: XvcEntity( + 3, + 9022053985516504033, + ), + value: ChildEntity( + XvcEntity( + 2, + 9270302598983016314, + ), + PhantomData, + PhantomData, + ), + }, + ], + ), + current: EventLog( + [], + ), + }, + }, + step_dependencies: {}, + step_outputs: HStore { + map: {}, + }, +} +[TRACE][pipeline/src/pipeline/mod.rs::1461] &process: Popen { + stdin: None, + stdout: Some( + File { + fd: 5, + read: true, + write: false, + }, + ), + stderr: Some( + File { + fd: 7, + read: true, + write: false, + }, + ), + child_state: Running { + pid: 21337, + ext: (), + }, + detached: true, +} +[TRACE][pipeline/src/pipeline/mod.rs::1466] process: Popen { + stdin: None, + stdout: Some( + File { + fd: 5, + read: true, + write: false, + }, + ), + stderr: Some( + File { + fd: 7, + read: true, + write: false, + }, + ), + child_state: Running { + pid: 21337, + ext: (), + }, + detached: true, +} +[DEBUG] Step average-age with command sqlite3 people.db 'SELECT AVG(Age) FROM People;' is still running +[OUT] [average-age] 34.6666666666667 + +[TRACE][pipeline/src/pipeline/mod.rs::1461] &process: Popen { + stdin: None, + stdout: Some( + File { + fd: 5, + read: true, + write: false, + }, + ), + stderr: Some( + File { + fd: 7, + read: true, + write: false, + }, + ), + child_state: Running { + pid: 21337, + ext: (), + }, + detached: true, +} +[DONE] average-age (sqlite3 people.db 'SELECT AVG(Age) FROM People;') +[TRACE][pipeline/src/pipeline/mod.rs::1513] return_state: Some( + DoneByRunning( + FromProcessCompletedSuccessfully, + ), +) + +[TRACE][pipeline/src/pipeline/mod.rs::1519] params: StepStateParams { + xvc_root: XvcRootInner { + absolute_path: AbsolutePath( + "[CWD]", + ), + xvc_dir: AbsolutePath( + "[CWD]/.xvc", + ), + store_dir: AbsolutePath( + "[CWD]/.xvc/store", + ), + config: XvcConfig { + current_dir: XvcConfigOption { + source: Runtime, + option: AbsolutePath( + "[CWD]", + ), + }, + config_maps: [ + XvcConfigMap { + source: Default, + map: { + "pipeline.default_params_file": String( + "params.yaml", + ), + "file.carry-in.no_parallel": Boolean( + false, + ), + "file.carry-in.force": Boolean( + false, + ), + "core.verbosity": String( + "error", + ), + "file.recheck.method": String( + "copy", + ), + "file.list.sort": String( + "name-desc", + ), + "git.use_git": Boolean( + true, + ), + "pipeline.current_pipeline": String( + "default", + ), + "core.guid": String( + "919ce8649f3e32f7", + ), + "git.auto_commit": Boolean( + true, + ), + "pipeline.default": String( + "default", + ), + "pipeline.process_pool_size": Integer( + 4, + ), + "git.command": String( + "git", + ), + "file.track.no_commit": Boolean( + false, + ), + "file.track.force": Boolean( + false, + ), + "file.list.show_dot_files": Boolean( + false, + ), + "file.list.recursive": Boolean( + false, + ), + "file.list.format": String( + "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", + ), + "cache.algorithm": String( + "blake3", + ), + "git.auto_stage": Boolean( + false, + ), + "file.list.no_summary": Boolean( + false, + ), + "file.track.text_or_binary": String( + "auto", + ), + "file.track.no_parallel": Boolean( + false, + ), + }, + }, + XvcConfigMap { + source: Project, + map: { + "core.guid": String( + "59a2d95849757118", + ), + "cache.algorithm": String( + "blake3", + ), + "file.list.recursive": Boolean( + false, + ), + "git.command": String( + "git", + ), + "pipeline.default_params_file": String( + "params.yaml", + ), + "file.track.force": Boolean( + false, + ), + "file.track.no_parallel": Boolean( + false, + ), + "git.use_git": Boolean( + true, + ), + "file.track.no_commit": Boolean( + false, + ), + "file.recheck.method": String( + "copy", + ), + "file.list.no_summary": Boolean( + false, + ), + "pipeline.current_pipeline": String( + "default", + ), + "git.auto_stage": Boolean( + false, + ), + "git.auto_commit": Boolean( + true, + ), + "file.list.show_dot_files": Boolean( + false, + ), + "file.list.format": String( + "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", + ), + "file.list.sort": String( + "name-desc", + ), + "pipeline.process_pool_size": Integer( + 4, + ), + "file.carry-in.no_parallel": Boolean( + false, + ), + "pipeline.default": String( + "default", + ), + "file.carry-in.force": Boolean( + false, + ), + "core.verbosity": String( + "error", + ), + "file.track.text_or_binary": String( + "auto", + ), + }, + }, + XvcConfigMap { + source: Local, + map: {}, + }, + XvcConfigMap { + source: Environment, + map: { + "TRYCMD_TESTS": String( + "storage,file,pipeline,core", + ), + }, + }, + XvcConfigMap { + source: CommandLine, + map: { + "core.quiet": Boolean( + false, + ), + "core.verbosity": String( + "debug", + ), + }, + }, + ], + the_config: { + "pipeline.default_params_file": XvcConfigValue { + source: Project, + value: String( + "params.yaml", + ), + }, + "file.list.sort": XvcConfigValue { + source: Project, + value: String( + "name-desc", + ), + }, + "git.use_git": XvcConfigValue { + source: Project, + value: Boolean( + true, + ), + }, + "git.auto_commit": XvcConfigValue { + source: Project, + value: Boolean( + true, + ), + }, + "core.verbosity": XvcConfigValue { + source: CommandLine, + value: String( + "debug", + ), + }, + "git.auto_stage": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "file.carry-in.force": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "pipeline.current_pipeline": XvcConfigValue { + source: Project, + value: String( + "default", + ), + }, + "cache.algorithm": XvcConfigValue { + source: Project, + value: String( + "blake3", + ), + }, + "pipeline.process_pool_size": XvcConfigValue { + source: Project, + value: Integer( + 4, + ), + }, + "file.list.no_summary": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "file.track.no_parallel": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "file.recheck.method": XvcConfigValue { + source: Project, + value: String( + "copy", + ), + }, + "core.quiet": XvcConfigValue { + source: CommandLine, + value: Boolean( + false, + ), + }, + "file.carry-in.no_parallel": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "core.guid": XvcConfigValue { + source: Project, + value: String( + "59a2d95849757118", + ), + }, + "pipeline.default": XvcConfigValue { + source: Project, + value: String( + "default", + ), + }, + "git.command": XvcConfigValue { + source: Project, + value: String( + "git", + ), + }, + "file.track.force": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "file.track.no_commit": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "file.list.recursive": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + "file.list.format": XvcConfigValue { + source: Project, + value: String( + "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", + ), + }, + "file.track.text_or_binary": XvcConfigValue { + source: Project, + value: String( + "auto", + ), + }, + "TRYCMD_TESTS": XvcConfigValue { + source: Environment, + value: String( + "storage,file,pipeline,core", + ), + }, + "file.list.show_dot_files": XvcConfigValue { + source: Project, + value: Boolean( + false, + ), + }, + }, + init_params: XvcConfigParams { + default_configuration: " +[core] +# The repository id. Please do not delete or change it. +# This is used to identify the repository and generate paths in storages. +# In the future it may be used to in other ways. +guid = /"919ce8649f3e32f7/" +# Default verbosity level. +# One of /"error/", /"warn/", /"info/" +verbosity = /"error/" + +[git] +# Automate git operations. +# Turning this off leads Xvc to behave as if it's not in a Git repository. +# Not recommended unless you're really not using Git +use_git = true +# Command to run Git process. +# You can set this to an absolute path to specify an executable +# If set to a non-absolute path, the executable will be searched in $PATH. +command = /"git/" + +# Commit changes in .xvc/ directory after commands. +# You can set this to false if you want to commit manually. +auto_commit = true + +# Stage changes in .xvc/ directory without committing. +# auto_commit implies auto_stage. +# If you want to commit manually but don't want to stage after individual Xvc commands, you can set this to true. +auto_stage = false + +[cache] +# The hash algorithm used for the cache. +# It may take blake3, blake2, sha2 or sha3 as values. +# All algorithms are selected to produce 256-bit hashes, so sha2 means SHA2-256, blake2 means BLAKE2s, etc. +# The cache path is produced by prepending algorithm name to the cache. +# Blake3 files are in .xvc/b3/, while sha2 files are in .xvc/s2/ etc. +algorithm = /"blake3/" + +[file] + +[file.track] + +# Don't move file content to cache after xvc file track +no_commit = false +# Force to track files even if they are already tracked. +force = false + +# Xvc calculates file content digest differently for text and binary files. +# This option controls whether to treat files as text or binary. +# It may take auto, text or binary as values. +# Auto check each file individually and treat it as text if it's text. +text_or_binary = /"auto/" + +# Don't use parallelism in track operations. +# Note that some of the operations are implemented in parallel by default, and this option affects some heavier operations. +no_parallel = false + +[file.list] + +# Format for `xvc file list` rows. You can reorder or remove columns. +# The following are the keys for each row: +# - {acd64}: actual content digest. All 64 digits from the workspace file's content. +# - {acd8}: actual content digest. First 8 digits the file content digest. +# - {aft}: actual file type. Whether the entry is a file (F), directory (D), +# symlink (S), hardlink (H) or reflink (R). +# - {asz}: actual size. The size of the workspace file in bytes. It uses MB, +# GB and TB to represent sizes larger than 1MB. +# - {ats}: actual timestamp. The timestamp of the workspace file. +# - {cst}: cache status. One of /"=/", /">/", /"/", /" 0 + F 000001: + >000002: /x00 => 2 + 000003: /x00 => 0 + match kind: Standard + prefilter: false + state length: 4 + pattern length: 0 + shortest pattern length: 18446744073709551615 + longest pattern length: 0 + alphabet length: 1 + stride: 1 + byte classes: ByteClasses(0 => [0-255]) + memory usage: 16 + ) + , + ), + map: [], + longest: 0, + }, + ), + Prefix( + PrefixStrategy { + matcher: AhoCorasick( + dfa::DFA( + D 000000: /x00 => 0 + F 000001: + >000002: /x00 => 2 + 000003: /x00 => 0 + match kind: Standard + prefilter: false + state length: 4 + pattern length: 0 + shortest pattern length: 18446744073709551615 + longest pattern length: 0 + alphabet length: 1 + stride: 1 + byte classes: ByteClasses(0 => [0-255]) + memory usage: 16 + ) + , + ), + map: [], + longest: 0, + }, + ), + RequiredExtension( + RequiredExtensionStrategy( + {}, + ), + ), + Regex( + RegexSetStrategy { + matcher: Regex { + imp: RegexI { + strat: Core { + info: RegexInfo( + RegexInfoI { + config: Config { + match_kind: Some( + All, + ), + utf8_empty: Some( + false, + ), + autopre: None, + pre: None, + which_captures: None, + nfa_size_limit: Some( + Some( + 10485760, + ), + ), + onepass_size_limit: None, + hybrid_cache_capacity: Some( + 10485760, + ), + hybrid: None, + dfa: None, + dfa_size_limit: None, + dfa_state_limit: None, + onepass: None, + backtrack: None, + byte_classes: None, + line_terminator: None, + }, + props: [], + props_union: Properties( + PropertiesI { + minimum_len: None, + maximum_len: None, + look_set: ∅, + look_set_prefix: ∅, + look_set_suffix: ∅, + look_set_prefix_any: ∅, + look_set_suffix_any: ∅, + utf8: true, + explicit_captures_len: 0, + static_explicit_captures_len: None, + literal: false, + alternation_literal: true, + }, + ), + }, + ), + pre: None, + nfa: thompson::NFA( + ^000000: FAIL + + transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) + ) + , + nfarev: Some( + thompson::NFA( + ^000000: FAIL + + transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) + ) + , + ), + pikevm: PikeVM( + PikeVMEngine( + PikeVM { + config: Config { + match_kind: Some( + All, + ), + pre: Some( + None, + ), + }, + nfa: thompson::NFA( + ^000000: FAIL + + transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) + ) + , + }, + ), + ), + backtrack: BoundedBacktracker( + None, + ), + onepass: OnePass( + None, + ), + hybrid: Hybrid( + Some( + HybridEngine( + Regex { + forward: DFA { + config: Config { + match_kind: Some( + All, + ), + pre: Some( + None, + ), + starts_for_each_pattern: Some( + true, + ), + byte_classes: Some( + true, + ), + unicode_word_boundary: Some( + true, + ), + quitset: None, + specialize_start_states: Some( + false, + ), + cache_capacity: Some( + 10485760, + ), + skip_cache_capacity_check: Some( + false, + ), + minimum_cache_clear_count: Some( + Some( + 3, + ), + ), + minimum_bytes_per_state: Some( + Some( + 10, + ), + ), + }, + nfa: thompson::NFA( + ^000000: FAIL + + transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) + ) + , + stride2: 1, + start_map: StartByteMap{/x00 => NonWordByte, /x01 => NonWordByte, /x02 => NonWordByte, /x03 => NonWordByte, /x04 => NonWordByte, /x05 => NonWordByte, /x06 => NonWordByte, /x07 => NonWordByte, /x08 => NonWordByte, /t => NonWordByte, + => LineLF, /x0B => NonWordByte, /x0C => NonWordByte, /r => LineCR, /x0E => NonWordByte, /x0F => NonWordByte, /x10 => NonWordByte, /x11 => NonWordByte, /x12 => NonWordByte, /x13 => NonWordByte, /x14 => NonWordByte, /x15 => NonWordByte, /x16 => NonWordByte, /x17 => NonWordByte, /x18 => NonWordByte, /x19 => NonWordByte, /x1A => NonWordByte, /x1B => NonWordByte, /x1C => NonWordByte, /x1D => NonWordByte, /x1E => NonWordByte, /x1F => NonWordByte, ' ' => NonWordByte, ! => NonWordByte, /" => NonWordByte, # => NonWordByte, $ => NonWordByte, % => NonWordByte, & => NonWordByte, /' => NonWordByte, ( => NonWordByte, ) => NonWordByte, * => NonWordByte, + => NonWordByte, , => NonWordByte, - => NonWordByte, . => NonWordByte, / => NonWordByte, 0 => WordByte, 1 => WordByte, 2 => WordByte, 3 => WordByte, 4 => WordByte, 5 => WordByte, 6 => WordByte, 7 => WordByte, 8 => WordByte, 9 => WordByte, : => NonWordByte, ; => NonWordByte, < => NonWordByte, = => NonWordByte, > => NonWordByte, ? => NonWordByte, @ => NonWordByte, A => WordByte, B => WordByte, C => WordByte, D => WordByte, E => WordByte, F => WordByte, G => WordByte, H => WordByte, I => WordByte, J => WordByte, K => WordByte, L => WordByte, M => WordByte, N => WordByte, O => WordByte, P => WordByte, Q => WordByte, R => WordByte, S => WordByte, T => WordByte, U => WordByte, V => WordByte, W => WordByte, X => WordByte, Y => WordByte, Z => WordByte, [ => NonWordByte, // => NonWordByte, ] => NonWordByte, ^ => NonWordByte, _ => WordByte, ` => NonWordByte, a => WordByte, b => WordByte, c => WordByte, d => WordByte, e => WordByte, f => WordByte, g => WordByte, h => WordByte, i => WordByte, j => WordByte, k => WordByte, l => WordByte, m => WordByte, n => WordByte, o => WordByte, p => WordByte, q => WordByte, r => WordByte, s => WordByte, t => WordByte, u => WordByte, v => WordByte, w => WordByte, x => WordByte, y => WordByte, z => WordByte, { => NonWordByte, | => NonWordByte, } => NonWordByte, ~ => NonWordByte, /x7F => NonWordByte, /x80 => NonWordByte, /x81 => NonWordByte, /x82 => NonWordByte, /x83 => NonWordByte, /x84 => NonWordByte, /x85 => NonWordByte, /x86 => NonWordByte, /x87 => NonWordByte, /x88 => NonWordByte, /x89 => NonWordByte, /x8A => NonWordByte, /x8B => NonWordByte, /x8C => NonWordByte, /x8D => NonWordByte, /x8E => NonWordByte, /x8F => NonWordByte, /x90 => NonWordByte, /x91 => NonWordByte, /x92 => NonWordByte, /x93 => NonWordByte, /x94 => NonWordByte, /x95 => NonWordByte, /x96 => NonWordByte, /x97 => NonWordByte, /x98 => NonWordByte, /x99 => NonWordByte, /x9A => NonWordByte, /x9B => NonWordByte, /x9C => NonWordByte, /x9D => NonWordByte, /x9E => NonWordByte, /x9F => NonWordByte, /xA0 => NonWordByte, /xA1 => NonWordByte, /xA2 => NonWordByte, /xA3 => NonWordByte, /xA4 => NonWordByte, /xA5 => NonWordByte, /xA6 => NonWordByte, /xA7 => NonWordByte, /xA8 => NonWordByte, /xA9 => NonWordByte, /xAA => NonWordByte, /xAB => NonWordByte, /xAC => NonWordByte, /xAD => NonWordByte, /xAE => NonWordByte, /xAF => NonWordByte, /xB0 => NonWordByte, /xB1 => NonWordByte, /xB2 => NonWordByte, /xB3 => NonWordByte, /xB4 => NonWordByte, /xB5 => NonWordByte, /xB6 => NonWordByte, /xB7 => NonWordByte, /xB8 => NonWordByte, /xB9 => NonWordByte, /xBA => NonWordByte, /xBB => NonWordByte, /xBC => NonWordByte, /xBD => NonWordByte, /xBE => NonWordByte, /xBF => NonWordByte, /xC0 => NonWordByte, /xC1 => NonWordByte, /xC2 => NonWordByte, /xC3 => NonWordByte, /xC4 => NonWordByte, /xC5 => NonWordByte, /xC6 => NonWordByte, /xC7 => NonWordByte, /xC8 => NonWordByte, /xC9 => NonWordByte, /xCA => NonWordByte, /xCB => NonWordByte, /xCC => NonWordByte, /xCD => NonWordByte, /xCE => NonWordByte, /xCF => NonWordByte, /xD0 => NonWordByte, /xD1 => NonWordByte, /xD2 => NonWordByte, /xD3 => NonWordByte, /xD4 => NonWordByte, /xD5 => NonWordByte, /xD6 => NonWordByte, /xD7 => NonWordByte, /xD8 => NonWordByte, /xD9 => NonWordByte, /xDA => NonWordByte, /xDB => NonWordByte, /xDC => NonWordByte, /xDD => NonWordByte, /xDE => NonWordByte, /xDF => NonWordByte, /xE0 => NonWordByte, /xE1 => NonWordByte, /xE2 => NonWordByte, /xE3 => NonWordByte, /xE4 => NonWordByte, /xE5 => NonWordByte, /xE6 => NonWordByte, /xE7 => NonWordByte, /xE8 => NonWordByte, /xE9 => NonWordByte, /xEA => NonWordByte, /xEB => NonWordByte, /xEC => NonWordByte, /xED => NonWordByte, /xEE => NonWordByte, /xEF => NonWordByte, /xF0 => NonWordByte, /xF1 => NonWordByte, /xF2 => NonWordByte, /xF3 => NonWordByte, /xF4 => NonWordByte, /xF5 => NonWordByte, /xF6 => NonWordByte, /xF7 => NonWordByte, /xF8 => NonWordByte, /xF9 => NonWordByte, /xFA => NonWordByte, /xFB => NonWordByte, /xFC => NonWordByte, /xFD => NonWordByte, /xFE => NonWordByte, /xFF => NonWordByte}, + classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]), + quitset: ByteSet { + bits: {}, + }, + cache_capacity: 10485760, + }, + reverse: DFA { + config: Config { + match_kind: Some( + All, + ), + pre: Some( + None, + ), + starts_for_each_pattern: Some( + true, + ), + byte_classes: Some( + true, + ), + unicode_word_boundary: Some( + true, + ), + quitset: None, + specialize_start_states: Some( + false, + ), + cache_capacity: Some( + 10485760, + ), + skip_cache_capacity_check: Some( + false, + ), + minimum_cache_clear_count: Some( + Some( + 3, + ), + ), + minimum_bytes_per_state: Some( + Some( + 10, + ), + ), + }, + nfa: thompson::NFA( + ^000000: FAIL + + transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) + ) + , + stride2: 1, + start_map: StartByteMap{/x00 => NonWordByte, /x01 => NonWordByte, /x02 => NonWordByte, /x03 => NonWordByte, /x04 => NonWordByte, /x05 => NonWordByte, /x06 => NonWordByte, /x07 => NonWordByte, /x08 => NonWordByte, /t => NonWordByte, + => LineLF, /x0B => NonWordByte, /x0C => NonWordByte, /r => LineCR, /x0E => NonWordByte, /x0F => NonWordByte, /x10 => NonWordByte, /x11 => NonWordByte, /x12 => NonWordByte, /x13 => NonWordByte, /x14 => NonWordByte, /x15 => NonWordByte, /x16 => NonWordByte, /x17 => NonWordByte, /x18 => NonWordByte, /x19 => NonWordByte, /x1A => NonWordByte, /x1B => NonWordByte, /x1C => NonWordByte, /x1D => NonWordByte, /x1E => NonWordByte, /x1F => NonWordByte, ' ' => NonWordByte, ! => NonWordByte, /" => NonWordByte, # => NonWordByte, $ => NonWordByte, % => NonWordByte, & => NonWordByte, /' => NonWordByte, ( => NonWordByte, ) => NonWordByte, * => NonWordByte, + => NonWordByte, , => NonWordByte, - => NonWordByte, . => NonWordByte, / => NonWordByte, 0 => WordByte, 1 => WordByte, 2 => WordByte, 3 => WordByte, 4 => WordByte, 5 => WordByte, 6 => WordByte, 7 => WordByte, 8 => WordByte, 9 => WordByte, : => NonWordByte, ; => NonWordByte, < => NonWordByte, = => NonWordByte, > => NonWordByte, ? => NonWordByte, @ => NonWordByte, A => WordByte, B => WordByte, C => WordByte, D => WordByte, E => WordByte, F => WordByte, G => WordByte, H => WordByte, I => WordByte, J => WordByte, K => WordByte, L => WordByte, M => WordByte, N => WordByte, O => WordByte, P => WordByte, Q => WordByte, R => WordByte, S => WordByte, T => WordByte, U => WordByte, V => WordByte, W => WordByte, X => WordByte, Y => WordByte, Z => WordByte, [ => NonWordByte, // => NonWordByte, ] => NonWordByte, ^ => NonWordByte, _ => WordByte, ` => NonWordByte, a => WordByte, b => WordByte, c => WordByte, d => WordByte, e => WordByte, f => WordByte, g => WordByte, h => WordByte, i => WordByte, j => WordByte, k => WordByte, l => WordByte, m => WordByte, n => WordByte, o => WordByte, p => WordByte, q => WordByte, r => WordByte, s => WordByte, t => WordByte, u => WordByte, v => WordByte, w => WordByte, x => WordByte, y => WordByte, z => WordByte, { => NonWordByte, | => NonWordByte, } => NonWordByte, ~ => NonWordByte, /x7F => NonWordByte, /x80 => NonWordByte, /x81 => NonWordByte, /x82 => NonWordByte, /x83 => NonWordByte, /x84 => NonWordByte, /x85 => NonWordByte, /x86 => NonWordByte, /x87 => NonWordByte, /x88 => NonWordByte, /x89 => NonWordByte, /x8A => NonWordByte, /x8B => NonWordByte, /x8C => NonWordByte, /x8D => NonWordByte, /x8E => NonWordByte, /x8F => NonWordByte, /x90 => NonWordByte, /x91 => NonWordByte, /x92 => NonWordByte, /x93 => NonWordByte, /x94 => NonWordByte, /x95 => NonWordByte, /x96 => NonWordByte, /x97 => NonWordByte, /x98 => NonWordByte, /x99 => NonWordByte, /x9A => NonWordByte, /x9B => NonWordByte, /x9C => NonWordByte, /x9D => NonWordByte, /x9E => NonWordByte, /x9F => NonWordByte, /xA0 => NonWordByte, /xA1 => NonWordByte, /xA2 => NonWordByte, /xA3 => NonWordByte, /xA4 => NonWordByte, /xA5 => NonWordByte, /xA6 => NonWordByte, /xA7 => NonWordByte, /xA8 => NonWordByte, /xA9 => NonWordByte, /xAA => NonWordByte, /xAB => NonWordByte, /xAC => NonWordByte, /xAD => NonWordByte, /xAE => NonWordByte, /xAF => NonWordByte, /xB0 => NonWordByte, /xB1 => NonWordByte, /xB2 => NonWordByte, /xB3 => NonWordByte, /xB4 => NonWordByte, /xB5 => NonWordByte, /xB6 => NonWordByte, /xB7 => NonWordByte, /xB8 => NonWordByte, /xB9 => NonWordByte, /xBA => NonWordByte, /xBB => NonWordByte, /xBC => NonWordByte, /xBD => NonWordByte, /xBE => NonWordByte, /xBF => NonWordByte, /xC0 => NonWordByte, /xC1 => NonWordByte, /xC2 => NonWordByte, /xC3 => NonWordByte, /xC4 => NonWordByte, /xC5 => NonWordByte, /xC6 => NonWordByte, /xC7 => NonWordByte, /xC8 => NonWordByte, /xC9 => NonWordByte, /xCA => NonWordByte, /xCB => NonWordByte, /xCC => NonWordByte, /xCD => NonWordByte, /xCE => NonWordByte, /xCF => NonWordByte, /xD0 => NonWordByte, /xD1 => NonWordByte, /xD2 => NonWordByte, /xD3 => NonWordByte, /xD4 => NonWordByte, /xD5 => NonWordByte, /xD6 => NonWordByte, /xD7 => NonWordByte, /xD8 => NonWordByte, /xD9 => NonWordByte, /xDA => NonWordByte, /xDB => NonWordByte, /xDC => NonWordByte, /xDD => NonWordByte, /xDE => NonWordByte, /xDF => NonWordByte, /xE0 => NonWordByte, /xE1 => NonWordByte, /xE2 => NonWordByte, /xE3 => NonWordByte, /xE4 => NonWordByte, /xE5 => NonWordByte, /xE6 => NonWordByte, /xE7 => NonWordByte, /xE8 => NonWordByte, /xE9 => NonWordByte, /xEA => NonWordByte, /xEB => NonWordByte, /xEC => NonWordByte, /xED => NonWordByte, /xEE => NonWordByte, /xEF => NonWordByte, /xF0 => NonWordByte, /xF1 => NonWordByte, /xF2 => NonWordByte, /xF3 => NonWordByte, /xF4 => NonWordByte, /xF5 => NonWordByte, /xF6 => NonWordByte, /xF7 => NonWordByte, /xF8 => NonWordByte, /xF9 => NonWordByte, /xFA => NonWordByte, /xFB => NonWordByte, /xFC => NonWordByte, /xFD => NonWordByte, /xFE => NonWordByte, /xFF => NonWordByte}, + classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]), + quitset: ByteSet { + bits: {}, + }, + cache_capacity: 10485760, + }, + }, + ), + ), + ), + dfa: DFA( + None, + ), + }, + info: RegexInfo( + RegexInfoI { + config: Config { + match_kind: Some( + All, + ), + utf8_empty: Some( + false, + ), + autopre: None, + pre: None, + which_captures: None, + nfa_size_limit: Some( + Some( + 10485760, + ), + ), + onepass_size_limit: None, + hybrid_cache_capacity: Some( + 10485760, + ), + hybrid: None, + dfa: None, + dfa_size_limit: None, + dfa_state_limit: None, + onepass: None, + backtrack: None, + byte_classes: None, + line_terminator: None, + }, + props: [], + props_union: Properties( + PropertiesI { + minimum_len: None, + maximum_len: None, + look_set: ∅, + look_set_prefix: ∅, + look_set_suffix: ∅, + look_set_prefix_any: ∅, + look_set_suffix_any: ∅, + utf8: true, + explicit_captures_len: 0, + static_explicit_captures_len: None, + literal: false, + alternation_literal: true, + }, + ), + }, + ), + }, + pool: Pool( + Pool { + stacks: [ + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + ], + owner: 0, + owner_val: UnsafeCell { .. }, + }, + ), + }, + map: [], + patset: Pool( + Pool { + stacks: [ + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + CacheLine( + Mutex { + data: [], + poisoned: false, + .. + }, + ), + ], + owner: 0, + owner_val: UnsafeCell { .. }, + }, + ), + }, + ), + ], + }, + poisoned: false, + .. + }, + }, + }, + run_conditions: RunConditions { + never: false, + always: false, + ignore_broken_dep_steps: false, + ignore_missing_outputs: true, + }, + pipeline_rundir: XvcPath( + "", + ), + terminate_timeout_processes: true, + algorithm: Blake3, + command_process: RwLock { + data: CommandProcess { + environment: {}, + step: XvcStep { + name: "average-age", + }, + step_command: XvcStepCommand { + command: "sqlite3 people.db 'SELECT AVG(Age) FROM People;'", + }, + birth: Some( + Instant { + tv_sec: 834347, + tv_nsec: 663124791, + }, + ), + process: Some( + Popen { + stdin: None, + stdout: Some( + File { + fd: 5, + read: true, + write: false, + }, + ), + stderr: Some( + File { + fd: 7, + read: true, + write: false, + }, + ), + child_state: Finished( + Exited( + 0, + ), + ), + detached: true, + }, + ), + stdout_sender: Sender { .. }, + stderr_sender: Sender { .. }, + stdout_receiver: Receiver { .. }, + stderr_receiver: Receiver { .. }, + }, + poisoned: false, + .. + }, + available_process_slots: RwLock { + data: , + poisoned: false, + .. + }, + process_poll_milliseconds: 10, + dependency_diffs: RwLock { + data: HStore { + map: { + XvcEntity( + 3, + 9022053985516504033, + ): RecordMissing { + actual: SqliteQueryDigest( + SqliteQueryDep { + path: XvcPath( + "people.db", + ), + query: "SELECT count(*) FROM People;", + query_digest: Some( + ContentDigest( + XvcDigest { + algorithm: Blake3, + digest: [ + 11, + 225, + 214, + 145, + 174, + 151, + 99, + 217, + 216, + 197, + 211, + 26, + 216, + 218, + 115, + 209, + 161, + 95, + 15, + 52, + 174, + 24, + 193, + 209, + 218, + 91, + 154, + 207, + 247, + 217, + 245, + 9, + ], + }, + ), + ), + xvc_metadata: Some( + XvcMetadata { + file_type: File, + size: Some( + 8192, + ), + modified: Some( + SystemTime { + tv_sec: 1722024927, + tv_nsec: 262703931, + }, + ), + }, + ), + }, + ), + }, + }, + }, + poisoned: false, + .. + }, + output_diffs: RwLock { + data: HStore { + map: {}, + }, + poisoned: false, + .. + }, + step_e: XvcEntity( + 2, + 9270302598983016314, + ), + step: XvcStep { + name: "average-age", + }, + step_command: XvcStepCommand { + command: "sqlite3 people.db 'SELECT AVG(Age) FROM People;'", + }, + current_states: RwLock { + data: HStore { + map: { + XvcEntity( + 2, + 9270302598983016314, + ): Running( + FromWaitProcess, + ), + }, + }, + poisoned: false, + .. + }, + step_timeout: 10000s, + all_steps: HStore { + map: { + XvcEntity( + 2, + 9270302598983016314, + ): XvcStep { + name: "average-age", + }, + }, + }, + recorded_dependencies: R1NStore { + parents: XvcStore { + map: { + XvcEntity( + 2, + 9270302598983016314, + ): XvcStep { + name: "average-age", + }, + }, + entity_index: { + XvcStep { + name: "average-age", + }: [ + XvcEntity( + 2, + 9270302598983016314, + ), + ], + }, + previous: EventLog( + [ + Add { + entity: XvcEntity( + 2, + 9270302598983016314, + ), + value: XvcStep { + name: "average-age", + }, + }, + Add { + entity: XvcEntity( + 2, + 9270302598983016314, + ), + value: XvcStep { + name: "average-age", + }, + }, + ], + ), + current: EventLog( + [], + ), + }, + children: XvcStore { + map: { + XvcEntity( + 3, + 9022053985516504033, + ): SqliteQueryDigest( + SqliteQueryDep { + path: XvcPath( + "people.db", + ), + query: "SELECT count(*) FROM People;", + query_digest: None, + xvc_metadata: None, + }, + ), + }, + entity_index: { + SqliteQueryDigest( + SqliteQueryDep { + path: XvcPath( + "people.db", + ), + query: "SELECT count(*) FROM People;", + query_digest: None, + xvc_metadata: None, + }, + ): [ + XvcEntity( + 3, + 9022053985516504033, + ), + ], + }, + previous: EventLog( + [ + Add { + entity: XvcEntity( + 3, + 9022053985516504033, + ), + value: SqliteQueryDigest( + SqliteQueryDep { + path: XvcPath( + "people.db", + ), + query: "SELECT count(*) FROM People;", + query_digest: None, + xvc_metadata: None, + }, + ), + }, + ], + ), + current: EventLog( + [], + ), + }, + child_parents: XvcStore { + map: { + XvcEntity( + 3, + 9022053985516504033, + ): ChildEntity( + XvcEntity( + 2, + 9270302598983016314, + ), + PhantomData, + PhantomData, + ), + }, + entity_index: { + ChildEntity( + XvcEntity( + 2, + 9270302598983016314, + ), + PhantomData, + PhantomData, + ): [ + XvcEntity( + 3, + 9022053985516504033, + ), + ], + }, + previous: EventLog( + [ + Add { + entity: XvcEntity( + 3, + 9022053985516504033, + ), + value: ChildEntity( + XvcEntity( + 2, + 9270302598983016314, + ), + PhantomData, + PhantomData, + ), + }, + ], + ), + current: EventLog( + [], + ), + }, + }, + step_dependencies: {}, + step_outputs: HStore { + map: {}, + }, +} +[TRACE][pipeline/src/pipeline/mod.rs::770] step.name: "average-age" +[TRACE][pipeline/src/pipeline/mod.rs::771] &r_next_state: DoneByRunning( + FromProcessCompletedSuccessfully, +) +[TRACE][pipeline/src/pipeline/mod.rs::773] &step_state: DoneByRunning( + FromProcessCompletedSuccessfully, +) +[TRACE][pipeline/src/pipeline/mod.rs::660] &step_state: DoneByRunning( + FromProcessCompletedSuccessfully, +) +[TRACE][pipeline/src/pipeline/mod.rs::519] "Before state updater": "Before state updater" +[TRACE][pipeline/src/pipeline/mod.rs::529] step_states: RwLock { + data: HStore { + map: { + XvcEntity( + 2, + 9270302598983016314, + ): DoneByRunning( + FromProcessCompletedSuccessfully, + ), + }, + }, + poisoned: false, + .. +} +[TRACE][pipeline/src/pipeline/mod.rs::536] done_successfully: Ok( + true, +) +[TRACE][core/src/util/pmp.rs::185] self.background_thread: Mutex { + data: JoinHandle { .. }, + poisoned: false, + .. +} +[TRACE][core/src/util/pmp.rs::190] self.background_thread: Mutex { + data: JoinHandle { .. }, + poisoned: false, + .. +} +[TRACE][core/src/util/pmp.rs::99] index: 1 +[TRACE][lib/src/cli/mod.rs::376] "Before handle_git_automation": "Before handle_git_automation" +[TRACE][lib/src/git.rs::30] args: [ + "-C", + "[CWD]", + "diff", + "--name-only", + "--cached", +] +[DEBUG] Using Git: /opt/homebrew/bin/git +[TRACE][lib/src/git.rs::61] git_diff_staged_out: "" +[TRACE][lib/src/git.rs::30] args: [ + "-C", + "[CWD]", + "add", + "--verbose", + "[CWD]/.xvc", + "*.gitignore", + "*.xvcignore", +] +[TRACE][lib/src/git.rs::179] git_add_output: "add '.xvc/store/xvc-dependency-store/1722171156330163.json' +" +[TRACE][lib/src/git.rs::30] args: [ + "-C", + "[CWD]", + "commit", + "-m", + "Xvc auto-commit after /'/Users/iex/github.com/iesahin/xvc/target/debug/xvc -vvvv pipeline run/'", +] +[DEBUG] Committing .xvc/ to git: [main f7aaa52] Xvc auto-commit after '/Users/iex/github.com/iesahin/xvc/target/debug/xvc -vvvv pipeline run' + 1 file changed, 1 insertion(+) + create mode 100644 .xvc/store/xvc-dependency-store/1722171156330163.json + +[DEBUG] Command completed successfully. ``` @@ -2768,14 +8710,6 @@ But it won't run the step a second time, as the table didn't change. ```console $ xvc pipeline run -? interrupted -thread '' panicked at pipeline/src/pipeline/mod.rs:1079:32: -SqliteError { source: FromSqlConversionFailure(18446744073709551615, Null, InvalidType) } -thread 'note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace -' panicked at lib/src/cli/mod.rs:251:52: -[PANIC] SqliteError { source: FromSqlConversionFailure(18446744073709551615, Null, InvalidType) }, [pipeline/src/pipeline/mod.rs::1079] -thread '' panicked at pipeline/src/pipeline/mod.rs:514:17: -called `Result::unwrap()` on an `Err` value: "SendError(..)" ``` @@ -2789,14 +8723,6 @@ This time, the step will run again as the result from dependency query (`SELECT ```console $ xvc pipeline run -? interrupted -thread '' panicked at pipeline/src/pipeline/mod.rs:1079:32: -SqliteError { source: FromSqlConversionFailure(18446744073709551615, Null, InvalidType) } -note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace -thread '' panicked at lib/src/cli/mod.rs:251:52: -[PANIC] SqliteError { source: FromSqlConversionFailure(18446744073709551615, Null, InvalidType) }, [pipeline/src/pipeline/mod.rs::1079] -thread '' panicked at pipeline/src/pipeline/mod.rs:514:17: -called `Result::unwrap()` on an `Err` value: "SendError(..)" ``` From 9fad80e170211c8928df8279d55b56378bed28a8 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sun, 28 Jul 2024 16:23:28 +0300 Subject: [PATCH 081/257] added bundled sqlite feature --- pipeline/Cargo.toml | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/pipeline/Cargo.toml b/pipeline/Cargo.toml index 9cbae3c3a..c32307a93 100644 --- a/pipeline/Cargo.toml +++ b/pipeline/Cargo.toml @@ -17,6 +17,9 @@ crate-type = ["rlib"] [profile.release] debug = true +[features] +default = [] +bundled_sqlite = ["dep:rusqlite/bundled"] [dependencies] xvc-config = { version = "0.6.9-alpha.1", path = "../config" } @@ -98,5 +101,6 @@ itertools = "^0.13" derive_more = "^0.99" [dev-dependencies] +default = [] xvc-test-helper = { version = "0.6.9-alpha.1", path = "../test_helper/" } test-case = "^3.3" From 0355ebfb8146aaba6e300ee5eb803fb4d9e0a033 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sun, 28 Jul 2024 16:24:38 +0300 Subject: [PATCH 082/257] renamed the feature and added to main --- lib/Cargo.toml | 1 + pipeline/Cargo.toml | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/Cargo.toml b/lib/Cargo.toml index 99fa9dbbe..7468a43ac 100644 --- a/lib/Cargo.toml +++ b/lib/Cargo.toml @@ -104,6 +104,7 @@ r2 = ["xvc-storage/r2"] gcs = ["xvc-storage/gcs"] wasabi = ["xvc-storage/wasabi"] digital-ocean = ["xvc-storage/digital-ocean"] +bundled_sqlite = ["xvc-pipeline/bundled-sqlite"] [dev-dependencies] diff --git a/pipeline/Cargo.toml b/pipeline/Cargo.toml index c32307a93..19f058d09 100644 --- a/pipeline/Cargo.toml +++ b/pipeline/Cargo.toml @@ -19,7 +19,7 @@ debug = true [features] default = [] -bundled_sqlite = ["dep:rusqlite/bundled"] +bundled-sqlite = ["dep:rusqlite/bundled"] [dependencies] xvc-config = { version = "0.6.9-alpha.1", path = "../config" } From d0f65a9274360bdc1e967743c2e53adc2092db29 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sun, 28 Jul 2024 16:25:17 +0300 Subject: [PATCH 083/257] rename feature --- lib/Cargo.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/Cargo.toml b/lib/Cargo.toml index 7468a43ac..cdd5b6f87 100644 --- a/lib/Cargo.toml +++ b/lib/Cargo.toml @@ -104,7 +104,7 @@ r2 = ["xvc-storage/r2"] gcs = ["xvc-storage/gcs"] wasabi = ["xvc-storage/wasabi"] digital-ocean = ["xvc-storage/digital-ocean"] -bundled_sqlite = ["xvc-pipeline/bundled-sqlite"] +bundled-sqlite = ["xvc-pipeline/bundled-sqlite"] [dev-dependencies] From 8a377a1c47f42ca11bf98e4cebecd7366fbc5a3f Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sun, 28 Jul 2024 16:26:12 +0300 Subject: [PATCH 084/257] fix --- pipeline/Cargo.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pipeline/Cargo.toml b/pipeline/Cargo.toml index 19f058d09..37a19fbc6 100644 --- a/pipeline/Cargo.toml +++ b/pipeline/Cargo.toml @@ -18,7 +18,7 @@ crate-type = ["rlib"] debug = true [features] -default = [] +default = [""] bundled-sqlite = ["dep:rusqlite/bundled"] [dependencies] From 2e5770db558dc18ab76b64c415f5e65534ec9dbd Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sun, 28 Jul 2024 16:42:03 +0300 Subject: [PATCH 085/257] remove debugs --- ...c-pipeline-step-dependency-sqlite-query.md | 8618 ----------------- 1 file changed, 8618 deletions(-) diff --git a/book/src/ref/xvc-pipeline-step-dependency-sqlite-query.md b/book/src/ref/xvc-pipeline-step-dependency-sqlite-query.md index 7f4fe94a4..906d8ceb6 100644 --- a/book/src/ref/xvc-pipeline-step-dependency-sqlite-query.md +++ b/book/src/ref/xvc-pipeline-step-dependency-sqlite-query.md @@ -81,8628 +81,10 @@ So, when the number of people in the table changes, the step will run. Initially ```console $ xvc pipeline run -[DEBUG][logging/src/lib.rs::237] Terminal logger enabled with level: Trace -[TRACE][core/src/types/xvcroot.rs::263] xvc_candidate: "[CWD]/.xvc" -[TRACE][core/src/types/xvcroot.rs::265] parent: "[CWD]" -[DEBUG][config/src/error.rs::72] Config source for level "system" not found at "/Users/iex/Library/Application Support/com.emresult.xvc" -[DEBUG][config/src/error.rs::72] Config source for level "global" not found at "/Users/iex/Library/Application Support/xvc" -[TRACE][ecs/src/ecs/mod.rs::229] dir: "[CWD]/.xvc/ec" -[TRACE][ecs/src/ecs/mod.rs::239] files: [ - "[CWD]/.xvc/ec/1722171155879620", - "[CWD]/.xvc/ec/1722171155882390", - "[CWD]/.xvc/ec/1722171155967050", - "[CWD]/.xvc/ec/1722171156186854", -] -[TRACE][core/src/types/xvcroot.rs::263] xvc_candidate: "[CWD]/.xvc" -[TRACE][core/src/types/xvcroot.rs::265] parent: "[CWD]" -[TRACE][pipeline/src/pipeline/mod.rs::285] pipeline_e: XvcEntity( - 1, - 1872241824668543094, -) -[TRACE][pipeline/src/pipeline/mod.rs::290] pipeline_steps: HStore { - map: { - XvcEntity( - 2, - 9270302598983016314, - ): XvcStep { - name: "average-age", - }, - }, -} -[TRACE][pipeline/src/pipeline/mod.rs::293] consider_changed: XvcStore { - map: { - XvcEntity( - 2, - 9270302598983016314, - ): ByDependencies, - }, - entity_index: { - ByDependencies: [ - XvcEntity( - 2, - 9270302598983016314, - ), - ], - }, - previous: EventLog( - [ - Add { - entity: XvcEntity( - 2, - 9270302598983016314, - ), - value: ByDependencies, - }, - ], - ), - current: EventLog( - [], - ), -} -[TRACE][pipeline/src/pipeline/mod.rs::296] all_deps.parents.len(): 1 -[TRACE][pipeline/src/pipeline/mod.rs::297] all_deps.children.len(): 1 -[TRACE][pipeline/src/pipeline/mod.rs::299] all_outs.parents.len(): 1 -[TRACE][pipeline/src/pipeline/mod.rs::300] all_outs.children.len(): 0 -[DEBUG][/Users/iex/.cargo/registry/src/index.crates.io-6f17d22bba15001f/globset-0.4.14/src/lib.rs::453] built glob set; 0 literals, 2 basenames, 0 extensions, 0 prefixes, 0 suffixes, 0 required extensions, 0 regexes -[TRACE][walker/src/lib.rs::488] ignore_fn: ".xvcignore" -[TRACE][walker/src/lib.rs::490] ignore_root: "[CWD]" -[TRACE][walker/src/lib.rs::498] entry.path(): "[CWD]/.xvc" -[TRACE][walker/src/lib.rs::498] entry.path(): "[CWD]/.git" -[TRACE][walker/src/lib.rs::503] ignore_path: "[CWD]/.xvcignore" -[TRACE][walker/src/lib.rs::597] ignore_root: "[CWD]" -[TRACE][walker/src/lib.rs::598] ignore_path: "[CWD]/.xvcignore" -[TRACE][walker/src/lib.rs::606] &content: " -# Add patterns of files xvc should ignore, which could improve -# the performance. -# It's in the same format as .gitignore files. - -.DS_Store -" -[DEBUG][/Users/iex/.cargo/registry/src/index.crates.io-6f17d22bba15001f/globset-0.4.14/src/lib.rs::453] built glob set; 0 literals, 3 basenames, 0 extensions, 0 prefixes, 0 suffixes, 0 required extensions, 0 regexes -[TRACE][walker/src/lib.rs::748] is_abs: true -[TRACE][walker/src/lib.rs::752] path_str: "[CWD]/.xvc" -[TRACE][walker/src/lib.rs::754] final_slash: false -[TRACE][walker/src/lib.rs::776] path: "/.xvc" -[TRACE][walker/src/lib.rs::748] is_abs: true -[TRACE][walker/src/lib.rs::752] path_str: "[CWD]/.git" -[TRACE][walker/src/lib.rs::754] final_slash: false -[TRACE][walker/src/lib.rs::776] path: "/.git" -[TRACE][core/src/util/pmp.rs::41] ignore_rules: IgnoreRules { - root: "[CWD]", - ignore_patterns: RwLock { - data: [ - Pattern { - pattern: Glob { - glob: "**/.xvc", - re: "(?-u)^(?:/?|.*/)//.xvc$", - opts: GlobOptions { - case_insensitive: false, - literal_separator: false, - backslash_escape: true, - empty_alternates: false, - }, - tokens: Tokens( - [ - RecursivePrefix, - Literal( - '.', - ), - Literal( - 'x', - ), - Literal( - 'v', - ), - Literal( - 'c', - ), - ], - ), - }, - original: ".xvc", - source: Global, - effect: Ignore, - relativity: Anywhere, - path_kind: Any, - }, - Pattern { - pattern: Glob { - glob: "**/.git", - re: "(?-u)^(?:/?|.*/)//.git$", - opts: GlobOptions { - case_insensitive: false, - literal_separator: false, - backslash_escape: true, - empty_alternates: false, - }, - tokens: Tokens( - [ - RecursivePrefix, - Literal( - '.', - ), - Literal( - 'g', - ), - Literal( - 'i', - ), - Literal( - 't', - ), - ], - ), - }, - original: ".git", - source: Global, - effect: Ignore, - relativity: Anywhere, - path_kind: Any, - }, - Pattern { - pattern: Glob { - glob: "**/.DS_Store", - re: "(?-u)^(?:/?|.*/)//.DS_Store$", - opts: GlobOptions { - case_insensitive: false, - literal_separator: false, - backslash_escape: true, - empty_alternates: false, - }, - tokens: Tokens( - [ - RecursivePrefix, - Literal( - '.', - ), - Literal( - 'D', - ), - Literal( - 'S', - ), - Literal( - '_', - ), - Literal( - 'S', - ), - Literal( - 't', - ), - Literal( - 'o', - ), - Literal( - 'r', - ), - Literal( - 'e', - ), - ], - ), - }, - original: ".DS_Store", - source: File { - path: ".xvcignore", - line: 6, - }, - effect: Ignore, - relativity: Anywhere, - path_kind: Any, - }, - ], - poisoned: false, - .. - }, - whitelist_patterns: RwLock { - data: [], - poisoned: false, - .. - }, - whitelist_set: RwLock { - data: GlobSet { - len: 0, - strats: [], - }, - poisoned: false, - .. - }, - ignore_set: RwLock { - data: GlobSet { - len: 3, - strats: [ - Extension( - ExtensionStrategy( - {}, - ), - ), - BasenameLiteral( - BasenameLiteralStrategy( - { - [ - 46, - 120, - 118, - 99, - ]: [ - 0, - ], - [ - 46, - 68, - 83, - 95, - 83, - 116, - 111, - 114, - 101, - ]: [ - 2, - ], - [ - 46, - 103, - 105, - 116, - ]: [ - 1, - ], - }, - ), - ), - Literal( - LiteralStrategy( - {}, - ), - ), - Suffix( - SuffixStrategy { - matcher: AhoCorasick( - dfa::DFA( - D 000000: /x00 => 0 - F 000001: - >000002: /x00 => 2 - 000003: /x00 => 0 - match kind: Standard - prefilter: false - state length: 4 - pattern length: 0 - shortest pattern length: 18446744073709551615 - longest pattern length: 0 - alphabet length: 1 - stride: 1 - byte classes: ByteClasses(0 => [0-255]) - memory usage: 16 - ) - , - ), - map: [], - longest: 0, - }, - ), - Prefix( - PrefixStrategy { - matcher: AhoCorasick( - dfa::DFA( - D 000000: /x00 => 0 - F 000001: - >000002: /x00 => 2 - 000003: /x00 => 0 - match kind: Standard - prefilter: false - state length: 4 - pattern length: 0 - shortest pattern length: 18446744073709551615 - longest pattern length: 0 - alphabet length: 1 - stride: 1 - byte classes: ByteClasses(0 => [0-255]) - memory usage: 16 - ) - , - ), - map: [], - longest: 0, - }, - ), - RequiredExtension( - RequiredExtensionStrategy( - {}, - ), - ), - Regex( - RegexSetStrategy { - matcher: Regex { - imp: RegexI { - strat: Core { - info: RegexInfo( - RegexInfoI { - config: Config { - match_kind: Some( - All, - ), - utf8_empty: Some( - false, - ), - autopre: None, - pre: None, - which_captures: None, - nfa_size_limit: Some( - Some( - 10485760, - ), - ), - onepass_size_limit: None, - hybrid_cache_capacity: Some( - 10485760, - ), - hybrid: None, - dfa: None, - dfa_size_limit: None, - dfa_state_limit: None, - onepass: None, - backtrack: None, - byte_classes: None, - line_terminator: None, - }, - props: [], - props_union: Properties( - PropertiesI { - minimum_len: None, - maximum_len: None, - look_set: ∅, - look_set_prefix: ∅, - look_set_suffix: ∅, - look_set_prefix_any: ∅, - look_set_suffix_any: ∅, - utf8: true, - explicit_captures_len: 0, - static_explicit_captures_len: None, - literal: false, - alternation_literal: true, - }, - ), - }, - ), - pre: None, - nfa: thompson::NFA( - ^000000: FAIL - - transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) - ) - , - nfarev: Some( - thompson::NFA( - ^000000: FAIL - - transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) - ) - , - ), - pikevm: PikeVM( - PikeVMEngine( - PikeVM { - config: Config { - match_kind: Some( - All, - ), - pre: Some( - None, - ), - }, - nfa: thompson::NFA( - ^000000: FAIL - - transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) - ) - , - }, - ), - ), - backtrack: BoundedBacktracker( - None, - ), - onepass: OnePass( - None, - ), - hybrid: Hybrid( - Some( - HybridEngine( - Regex { - forward: DFA { - config: Config { - match_kind: Some( - All, - ), - pre: Some( - None, - ), - starts_for_each_pattern: Some( - true, - ), - byte_classes: Some( - true, - ), - unicode_word_boundary: Some( - true, - ), - quitset: None, - specialize_start_states: Some( - false, - ), - cache_capacity: Some( - 10485760, - ), - skip_cache_capacity_check: Some( - false, - ), - minimum_cache_clear_count: Some( - Some( - 3, - ), - ), - minimum_bytes_per_state: Some( - Some( - 10, - ), - ), - }, - nfa: thompson::NFA( - ^000000: FAIL - - transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) - ) - , - stride2: 1, - start_map: StartByteMap{/x00 => NonWordByte, /x01 => NonWordByte, /x02 => NonWordByte, /x03 => NonWordByte, /x04 => NonWordByte, /x05 => NonWordByte, /x06 => NonWordByte, /x07 => NonWordByte, /x08 => NonWordByte, /t => NonWordByte, - => LineLF, /x0B => NonWordByte, /x0C => NonWordByte, /r => LineCR, /x0E => NonWordByte, /x0F => NonWordByte, /x10 => NonWordByte, /x11 => NonWordByte, /x12 => NonWordByte, /x13 => NonWordByte, /x14 => NonWordByte, /x15 => NonWordByte, /x16 => NonWordByte, /x17 => NonWordByte, /x18 => NonWordByte, /x19 => NonWordByte, /x1A => NonWordByte, /x1B => NonWordByte, /x1C => NonWordByte, /x1D => NonWordByte, /x1E => NonWordByte, /x1F => NonWordByte, ' ' => NonWordByte, ! => NonWordByte, /" => NonWordByte, # => NonWordByte, $ => NonWordByte, % => NonWordByte, & => NonWordByte, /' => NonWordByte, ( => NonWordByte, ) => NonWordByte, * => NonWordByte, + => NonWordByte, , => NonWordByte, - => NonWordByte, . => NonWordByte, / => NonWordByte, 0 => WordByte, 1 => WordByte, 2 => WordByte, 3 => WordByte, 4 => WordByte, 5 => WordByte, 6 => WordByte, 7 => WordByte, 8 => WordByte, 9 => WordByte, : => NonWordByte, ; => NonWordByte, < => NonWordByte, = => NonWordByte, > => NonWordByte, ? => NonWordByte, @ => NonWordByte, A => WordByte, B => WordByte, C => WordByte, D => WordByte, E => WordByte, F => WordByte, G => WordByte, H => WordByte, I => WordByte, J => WordByte, K => WordByte, L => WordByte, M => WordByte, N => WordByte, O => WordByte, P => WordByte, Q => WordByte, R => WordByte, S => WordByte, T => WordByte, U => WordByte, V => WordByte, W => WordByte, X => WordByte, Y => WordByte, Z => WordByte, [ => NonWordByte, // => NonWordByte, ] => NonWordByte, ^ => NonWordByte, _ => WordByte, ` => NonWordByte, a => WordByte, b => WordByte, c => WordByte, d => WordByte, e => WordByte, f => WordByte, g => WordByte, h => WordByte, i => WordByte, j => WordByte, k => WordByte, l => WordByte, m => WordByte, n => WordByte, o => WordByte, p => WordByte, q => WordByte, r => WordByte, s => WordByte, t => WordByte, u => WordByte, v => WordByte, w => WordByte, x => WordByte, y => WordByte, z => WordByte, { => NonWordByte, | => NonWordByte, } => NonWordByte, ~ => NonWordByte, /x7F => NonWordByte, /x80 => NonWordByte, /x81 => NonWordByte, /x82 => NonWordByte, /x83 => NonWordByte, /x84 => NonWordByte, /x85 => NonWordByte, /x86 => NonWordByte, /x87 => NonWordByte, /x88 => NonWordByte, /x89 => NonWordByte, /x8A => NonWordByte, /x8B => NonWordByte, /x8C => NonWordByte, /x8D => NonWordByte, /x8E => NonWordByte, /x8F => NonWordByte, /x90 => NonWordByte, /x91 => NonWordByte, /x92 => NonWordByte, /x93 => NonWordByte, /x94 => NonWordByte, /x95 => NonWordByte, /x96 => NonWordByte, /x97 => NonWordByte, /x98 => NonWordByte, /x99 => NonWordByte, /x9A => NonWordByte, /x9B => NonWordByte, /x9C => NonWordByte, /x9D => NonWordByte, /x9E => NonWordByte, /x9F => NonWordByte, /xA0 => NonWordByte, /xA1 => NonWordByte, /xA2 => NonWordByte, /xA3 => NonWordByte, /xA4 => NonWordByte, /xA5 => NonWordByte, /xA6 => NonWordByte, /xA7 => NonWordByte, /xA8 => NonWordByte, /xA9 => NonWordByte, /xAA => NonWordByte, /xAB => NonWordByte, /xAC => NonWordByte, /xAD => NonWordByte, /xAE => NonWordByte, /xAF => NonWordByte, /xB0 => NonWordByte, /xB1 => NonWordByte, /xB2 => NonWordByte, /xB3 => NonWordByte, /xB4 => NonWordByte, /xB5 => NonWordByte, /xB6 => NonWordByte, /xB7 => NonWordByte, /xB8 => NonWordByte, /xB9 => NonWordByte, /xBA => NonWordByte, /xBB => NonWordByte, /xBC => NonWordByte, /xBD => NonWordByte, /xBE => NonWordByte, /xBF => NonWordByte, /xC0 => NonWordByte, /xC1 => NonWordByte, /xC2 => NonWordByte, /xC3 => NonWordByte, /xC4 => NonWordByte, /xC5 => NonWordByte, /xC6 => NonWordByte, /xC7 => NonWordByte, /xC8 => NonWordByte, /xC9 => NonWordByte, /xCA => NonWordByte, /xCB => NonWordByte, /xCC => NonWordByte, /xCD => NonWordByte, /xCE => NonWordByte, /xCF => NonWordByte, /xD0 => NonWordByte, /xD1 => NonWordByte, /xD2 => NonWordByte, /xD3 => NonWordByte, /xD4 => NonWordByte, /xD5 => NonWordByte, /xD6 => NonWordByte, /xD7 => NonWordByte, /xD8 => NonWordByte, /xD9 => NonWordByte, /xDA => NonWordByte, /xDB => NonWordByte, /xDC => NonWordByte, /xDD => NonWordByte, /xDE => NonWordByte, /xDF => NonWordByte, /xE0 => NonWordByte, /xE1 => NonWordByte, /xE2 => NonWordByte, /xE3 => NonWordByte, /xE4 => NonWordByte, /xE5 => NonWordByte, /xE6 => NonWordByte, /xE7 => NonWordByte, /xE8 => NonWordByte, /xE9 => NonWordByte, /xEA => NonWordByte, /xEB => NonWordByte, /xEC => NonWordByte, /xED => NonWordByte, /xEE => NonWordByte, /xEF => NonWordByte, /xF0 => NonWordByte, /xF1 => NonWordByte, /xF2 => NonWordByte, /xF3 => NonWordByte, /xF4 => NonWordByte, /xF5 => NonWordByte, /xF6 => NonWordByte, /xF7 => NonWordByte, /xF8 => NonWordByte, /xF9 => NonWordByte, /xFA => NonWordByte, /xFB => NonWordByte, /xFC => NonWordByte, /xFD => NonWordByte, /xFE => NonWordByte, /xFF => NonWordByte}, - classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]), - quitset: ByteSet { - bits: {}, - }, - cache_capacity: 10485760, - }, - reverse: DFA { - config: Config { - match_kind: Some( - All, - ), - pre: Some( - None, - ), - starts_for_each_pattern: Some( - true, - ), - byte_classes: Some( - true, - ), - unicode_word_boundary: Some( - true, - ), - quitset: None, - specialize_start_states: Some( - false, - ), - cache_capacity: Some( - 10485760, - ), - skip_cache_capacity_check: Some( - false, - ), - minimum_cache_clear_count: Some( - Some( - 3, - ), - ), - minimum_bytes_per_state: Some( - Some( - 10, - ), - ), - }, - nfa: thompson::NFA( - ^000000: FAIL - - transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) - ) - , - stride2: 1, - start_map: StartByteMap{/x00 => NonWordByte, /x01 => NonWordByte, /x02 => NonWordByte, /x03 => NonWordByte, /x04 => NonWordByte, /x05 => NonWordByte, /x06 => NonWordByte, /x07 => NonWordByte, /x08 => NonWordByte, /t => NonWordByte, - => LineLF, /x0B => NonWordByte, /x0C => NonWordByte, /r => LineCR, /x0E => NonWordByte, /x0F => NonWordByte, /x10 => NonWordByte, /x11 => NonWordByte, /x12 => NonWordByte, /x13 => NonWordByte, /x14 => NonWordByte, /x15 => NonWordByte, /x16 => NonWordByte, /x17 => NonWordByte, /x18 => NonWordByte, /x19 => NonWordByte, /x1A => NonWordByte, /x1B => NonWordByte, /x1C => NonWordByte, /x1D => NonWordByte, /x1E => NonWordByte, /x1F => NonWordByte, ' ' => NonWordByte, ! => NonWordByte, /" => NonWordByte, # => NonWordByte, $ => NonWordByte, % => NonWordByte, & => NonWordByte, /' => NonWordByte, ( => NonWordByte, ) => NonWordByte, * => NonWordByte, + => NonWordByte, , => NonWordByte, - => NonWordByte, . => NonWordByte, / => NonWordByte, 0 => WordByte, 1 => WordByte, 2 => WordByte, 3 => WordByte, 4 => WordByte, 5 => WordByte, 6 => WordByte, 7 => WordByte, 8 => WordByte, 9 => WordByte, : => NonWordByte, ; => NonWordByte, < => NonWordByte, = => NonWordByte, > => NonWordByte, ? => NonWordByte, @ => NonWordByte, A => WordByte, B => WordByte, C => WordByte, D => WordByte, E => WordByte, F => WordByte, G => WordByte, H => WordByte, I => WordByte, J => WordByte, K => WordByte, L => WordByte, M => WordByte, N => WordByte, O => WordByte, P => WordByte, Q => WordByte, R => WordByte, S => WordByte, T => WordByte, U => WordByte, V => WordByte, W => WordByte, X => WordByte, Y => WordByte, Z => WordByte, [ => NonWordByte, // => NonWordByte, ] => NonWordByte, ^ => NonWordByte, _ => WordByte, ` => NonWordByte, a => WordByte, b => WordByte, c => WordByte, d => WordByte, e => WordByte, f => WordByte, g => WordByte, h => WordByte, i => WordByte, j => WordByte, k => WordByte, l => WordByte, m => WordByte, n => WordByte, o => WordByte, p => WordByte, q => WordByte, r => WordByte, s => WordByte, t => WordByte, u => WordByte, v => WordByte, w => WordByte, x => WordByte, y => WordByte, z => WordByte, { => NonWordByte, | => NonWordByte, } => NonWordByte, ~ => NonWordByte, /x7F => NonWordByte, /x80 => NonWordByte, /x81 => NonWordByte, /x82 => NonWordByte, /x83 => NonWordByte, /x84 => NonWordByte, /x85 => NonWordByte, /x86 => NonWordByte, /x87 => NonWordByte, /x88 => NonWordByte, /x89 => NonWordByte, /x8A => NonWordByte, /x8B => NonWordByte, /x8C => NonWordByte, /x8D => NonWordByte, /x8E => NonWordByte, /x8F => NonWordByte, /x90 => NonWordByte, /x91 => NonWordByte, /x92 => NonWordByte, /x93 => NonWordByte, /x94 => NonWordByte, /x95 => NonWordByte, /x96 => NonWordByte, /x97 => NonWordByte, /x98 => NonWordByte, /x99 => NonWordByte, /x9A => NonWordByte, /x9B => NonWordByte, /x9C => NonWordByte, /x9D => NonWordByte, /x9E => NonWordByte, /x9F => NonWordByte, /xA0 => NonWordByte, /xA1 => NonWordByte, /xA2 => NonWordByte, /xA3 => NonWordByte, /xA4 => NonWordByte, /xA5 => NonWordByte, /xA6 => NonWordByte, /xA7 => NonWordByte, /xA8 => NonWordByte, /xA9 => NonWordByte, /xAA => NonWordByte, /xAB => NonWordByte, /xAC => NonWordByte, /xAD => NonWordByte, /xAE => NonWordByte, /xAF => NonWordByte, /xB0 => NonWordByte, /xB1 => NonWordByte, /xB2 => NonWordByte, /xB3 => NonWordByte, /xB4 => NonWordByte, /xB5 => NonWordByte, /xB6 => NonWordByte, /xB7 => NonWordByte, /xB8 => NonWordByte, /xB9 => NonWordByte, /xBA => NonWordByte, /xBB => NonWordByte, /xBC => NonWordByte, /xBD => NonWordByte, /xBE => NonWordByte, /xBF => NonWordByte, /xC0 => NonWordByte, /xC1 => NonWordByte, /xC2 => NonWordByte, /xC3 => NonWordByte, /xC4 => NonWordByte, /xC5 => NonWordByte, /xC6 => NonWordByte, /xC7 => NonWordByte, /xC8 => NonWordByte, /xC9 => NonWordByte, /xCA => NonWordByte, /xCB => NonWordByte, /xCC => NonWordByte, /xCD => NonWordByte, /xCE => NonWordByte, /xCF => NonWordByte, /xD0 => NonWordByte, /xD1 => NonWordByte, /xD2 => NonWordByte, /xD3 => NonWordByte, /xD4 => NonWordByte, /xD5 => NonWordByte, /xD6 => NonWordByte, /xD7 => NonWordByte, /xD8 => NonWordByte, /xD9 => NonWordByte, /xDA => NonWordByte, /xDB => NonWordByte, /xDC => NonWordByte, /xDD => NonWordByte, /xDE => NonWordByte, /xDF => NonWordByte, /xE0 => NonWordByte, /xE1 => NonWordByte, /xE2 => NonWordByte, /xE3 => NonWordByte, /xE4 => NonWordByte, /xE5 => NonWordByte, /xE6 => NonWordByte, /xE7 => NonWordByte, /xE8 => NonWordByte, /xE9 => NonWordByte, /xEA => NonWordByte, /xEB => NonWordByte, /xEC => NonWordByte, /xED => NonWordByte, /xEE => NonWordByte, /xEF => NonWordByte, /xF0 => NonWordByte, /xF1 => NonWordByte, /xF2 => NonWordByte, /xF3 => NonWordByte, /xF4 => NonWordByte, /xF5 => NonWordByte, /xF6 => NonWordByte, /xF7 => NonWordByte, /xF8 => NonWordByte, /xF9 => NonWordByte, /xFA => NonWordByte, /xFB => NonWordByte, /xFC => NonWordByte, /xFD => NonWordByte, /xFE => NonWordByte, /xFF => NonWordByte}, - classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]), - quitset: ByteSet { - bits: {}, - }, - cache_capacity: 10485760, - }, - }, - ), - ), - ), - dfa: DFA( - None, - ), - }, - info: RegexInfo( - RegexInfoI { - config: Config { - match_kind: Some( - All, - ), - utf8_empty: Some( - false, - ), - autopre: None, - pre: None, - which_captures: None, - nfa_size_limit: Some( - Some( - 10485760, - ), - ), - onepass_size_limit: None, - hybrid_cache_capacity: Some( - 10485760, - ), - hybrid: None, - dfa: None, - dfa_size_limit: None, - dfa_state_limit: None, - onepass: None, - backtrack: None, - byte_classes: None, - line_terminator: None, - }, - props: [], - props_union: Properties( - PropertiesI { - minimum_len: None, - maximum_len: None, - look_set: ∅, - look_set_prefix: ∅, - look_set_suffix: ∅, - look_set_prefix_any: ∅, - look_set_suffix_any: ∅, - utf8: true, - explicit_captures_len: 0, - static_explicit_captures_len: None, - literal: false, - alternation_literal: true, - }, - ), - }, - ), - }, - pool: Pool( - Pool { - stacks: [ - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - ], - owner: 0, - owner_val: UnsafeCell { .. }, - }, - ), - }, - map: [], - patset: Pool( - Pool { - stacks: [ - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - ], - owner: 0, - owner_val: UnsafeCell { .. }, - }, - ), - }, - ), - ], - }, - poisoned: false, - .. - }, -} -[TRACE][walker/src/notify.rs::170] watcher: FsEventWatcher { - paths: 0x0000600001ae8000, - since_when: 18446744073709551615, - latency: 0.0, - flags: 18, - event_handler: 0x0000600003ff0010, - runloop: Some( - ( - 0x00006000021fc0c0, - JoinHandle { .. }, - ), - ), - recursive_info: { - "[CWD]": true, - }, -} -[TRACE][core/src/util/pmp.rs::131] background_thread: Mutex { - data: JoinHandle { .. }, - poisoned: false, - .. -} -[TRACE][core/src/util/pmp.rs::59] watcher: FsEventWatcher { - paths: 0x0000600001ae8000, - since_when: 18446744073709551615, - latency: 0.0, - flags: 18, - event_handler: 0x0000600003ff0010, - runloop: Some( - ( - 0x00006000021fc0c0, - JoinHandle { .. }, - ), - ), - recursive_info: { - "[CWD]": true, - }, -} -[TRACE][core/src/util/pmp.rs::91] fs_event_index: 0 -[TRACE][core/src/util/pmp.rs::93] kill_signal_index: 1 -[TRACE][core/src/util/pmp.rs::96] "pmp background updater ticks": "pmp background updater ticks" -[TRACE][pipeline/src/pipeline/mod.rs::302] &pmp: XvcPathMetadataProvider { - xvc_root: XvcRootInner { - absolute_path: AbsolutePath( - "[CWD]", - ), - xvc_dir: AbsolutePath( - "[CWD]/.xvc", - ), - store_dir: AbsolutePath( - "[CWD]/.xvc/store", - ), - config: XvcConfig { - current_dir: XvcConfigOption { - source: Runtime, - option: AbsolutePath( - "[CWD]", - ), - }, - config_maps: [ - XvcConfigMap { - source: Default, - map: { - "pipeline.default_params_file": String( - "params.yaml", - ), - "file.carry-in.no_parallel": Boolean( - false, - ), - "file.carry-in.force": Boolean( - false, - ), - "core.verbosity": String( - "error", - ), - "file.recheck.method": String( - "copy", - ), - "file.list.sort": String( - "name-desc", - ), - "git.use_git": Boolean( - true, - ), - "pipeline.current_pipeline": String( - "default", - ), - "core.guid": String( - "919ce8649f3e32f7", - ), - "git.auto_commit": Boolean( - true, - ), - "pipeline.default": String( - "default", - ), - "pipeline.process_pool_size": Integer( - 4, - ), - "git.command": String( - "git", - ), - "file.track.no_commit": Boolean( - false, - ), - "file.track.force": Boolean( - false, - ), - "file.list.show_dot_files": Boolean( - false, - ), - "file.list.recursive": Boolean( - false, - ), - "file.list.format": String( - "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", - ), - "cache.algorithm": String( - "blake3", - ), - "git.auto_stage": Boolean( - false, - ), - "file.list.no_summary": Boolean( - false, - ), - "file.track.text_or_binary": String( - "auto", - ), - "file.track.no_parallel": Boolean( - false, - ), - }, - }, - XvcConfigMap { - source: Project, - map: { - "core.guid": String( - "59a2d95849757118", - ), - "cache.algorithm": String( - "blake3", - ), - "file.list.recursive": Boolean( - false, - ), - "git.command": String( - "git", - ), - "pipeline.default_params_file": String( - "params.yaml", - ), - "file.track.force": Boolean( - false, - ), - "file.track.no_parallel": Boolean( - false, - ), - "git.use_git": Boolean( - true, - ), - "file.track.no_commit": Boolean( - false, - ), - "file.recheck.method": String( - "copy", - ), - "file.list.no_summary": Boolean( - false, - ), - "pipeline.current_pipeline": String( - "default", - ), - "git.auto_stage": Boolean( - false, - ), - "git.auto_commit": Boolean( - true, - ), - "file.list.show_dot_files": Boolean( - false, - ), - "file.list.format": String( - "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", - ), - "file.list.sort": String( - "name-desc", - ), - "pipeline.process_pool_size": Integer( - 4, - ), - "file.carry-in.no_parallel": Boolean( - false, - ), - "pipeline.default": String( - "default", - ), - "file.carry-in.force": Boolean( - false, - ), - "core.verbosity": String( - "error", - ), - "file.track.text_or_binary": String( - "auto", - ), - }, - }, - XvcConfigMap { - source: Local, - map: {}, - }, - XvcConfigMap { - source: Environment, - map: { - "TRYCMD_TESTS": String( - "storage,file,pipeline,core", - ), - }, - }, - XvcConfigMap { - source: CommandLine, - map: { - "core.quiet": Boolean( - false, - ), - "core.verbosity": String( - "debug", - ), - }, - }, - ], - the_config: { - "pipeline.default_params_file": XvcConfigValue { - source: Project, - value: String( - "params.yaml", - ), - }, - "file.list.sort": XvcConfigValue { - source: Project, - value: String( - "name-desc", - ), - }, - "git.use_git": XvcConfigValue { - source: Project, - value: Boolean( - true, - ), - }, - "git.auto_commit": XvcConfigValue { - source: Project, - value: Boolean( - true, - ), - }, - "core.verbosity": XvcConfigValue { - source: CommandLine, - value: String( - "debug", - ), - }, - "git.auto_stage": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "file.carry-in.force": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "pipeline.current_pipeline": XvcConfigValue { - source: Project, - value: String( - "default", - ), - }, - "cache.algorithm": XvcConfigValue { - source: Project, - value: String( - "blake3", - ), - }, - "pipeline.process_pool_size": XvcConfigValue { - source: Project, - value: Integer( - 4, - ), - }, - "file.list.no_summary": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "file.track.no_parallel": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "file.recheck.method": XvcConfigValue { - source: Project, - value: String( - "copy", - ), - }, - "core.quiet": XvcConfigValue { - source: CommandLine, - value: Boolean( - false, - ), - }, - "file.carry-in.no_parallel": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "core.guid": XvcConfigValue { - source: Project, - value: String( - "59a2d95849757118", - ), - }, - "pipeline.default": XvcConfigValue { - source: Project, - value: String( - "default", - ), - }, - "git.command": XvcConfigValue { - source: Project, - value: String( - "git", - ), - }, - "file.track.force": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "file.track.no_commit": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "file.list.recursive": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "file.list.format": XvcConfigValue { - source: Project, - value: String( - "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", - ), - }, - "file.track.text_or_binary": XvcConfigValue { - source: Project, - value: String( - "auto", - ), - }, - "TRYCMD_TESTS": XvcConfigValue { - source: Environment, - value: String( - "storage,file,pipeline,core", - ), - }, - "file.list.show_dot_files": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - }, - init_params: XvcConfigParams { - default_configuration: " -[core] -# The repository id. Please do not delete or change it. -# This is used to identify the repository and generate paths in storages. -# In the future it may be used to in other ways. -guid = /"919ce8649f3e32f7/" -# Default verbosity level. -# One of /"error/", /"warn/", /"info/" -verbosity = /"error/" - -[git] -# Automate git operations. -# Turning this off leads Xvc to behave as if it's not in a Git repository. -# Not recommended unless you're really not using Git -use_git = true -# Command to run Git process. -# You can set this to an absolute path to specify an executable -# If set to a non-absolute path, the executable will be searched in $PATH. -command = /"git/" - -# Commit changes in .xvc/ directory after commands. -# You can set this to false if you want to commit manually. -auto_commit = true - -# Stage changes in .xvc/ directory without committing. -# auto_commit implies auto_stage. -# If you want to commit manually but don't want to stage after individual Xvc commands, you can set this to true. -auto_stage = false - -[cache] -# The hash algorithm used for the cache. -# It may take blake3, blake2, sha2 or sha3 as values. -# All algorithms are selected to produce 256-bit hashes, so sha2 means SHA2-256, blake2 means BLAKE2s, etc. -# The cache path is produced by prepending algorithm name to the cache. -# Blake3 files are in .xvc/b3/, while sha2 files are in .xvc/s2/ etc. -algorithm = /"blake3/" - -[file] - -[file.track] - -# Don't move file content to cache after xvc file track -no_commit = false -# Force to track files even if they are already tracked. -force = false - -# Xvc calculates file content digest differently for text and binary files. -# This option controls whether to treat files as text or binary. -# It may take auto, text or binary as values. -# Auto check each file individually and treat it as text if it's text. -text_or_binary = /"auto/" - -# Don't use parallelism in track operations. -# Note that some of the operations are implemented in parallel by default, and this option affects some heavier operations. -no_parallel = false - -[file.list] - -# Format for `xvc file list` rows. You can reorder or remove columns. -# The following are the keys for each row: -# - {acd64}: actual content digest. All 64 digits from the workspace file's content. -# - {acd8}: actual content digest. First 8 digits the file content digest. -# - {aft}: actual file type. Whether the entry is a file (F), directory (D), -# symlink (S), hardlink (H) or reflink (R). -# - {asz}: actual size. The size of the workspace file in bytes. It uses MB, -# GB and TB to represent sizes larger than 1MB. -# - {ats}: actual timestamp. The timestamp of the workspace file. -# - {cst}: cache status. One of /"=/", /">/", /" 0 - F 000001: - >000002: /x00 => 2 - 000003: /x00 => 0 - match kind: Standard - prefilter: false - state length: 4 - pattern length: 0 - shortest pattern length: 18446744073709551615 - longest pattern length: 0 - alphabet length: 1 - stride: 1 - byte classes: ByteClasses(0 => [0-255]) - memory usage: 16 - ) - , - ), - map: [], - longest: 0, - }, - ), - Prefix( - PrefixStrategy { - matcher: AhoCorasick( - dfa::DFA( - D 000000: /x00 => 0 - F 000001: - >000002: /x00 => 2 - 000003: /x00 => 0 - match kind: Standard - prefilter: false - state length: 4 - pattern length: 0 - shortest pattern length: 18446744073709551615 - longest pattern length: 0 - alphabet length: 1 - stride: 1 - byte classes: ByteClasses(0 => [0-255]) - memory usage: 16 - ) - , - ), - map: [], - longest: 0, - }, - ), - RequiredExtension( - RequiredExtensionStrategy( - {}, - ), - ), - Regex( - RegexSetStrategy { - matcher: Regex { - imp: RegexI { - strat: Core { - info: RegexInfo( - RegexInfoI { - config: Config { - match_kind: Some( - All, - ), - utf8_empty: Some( - false, - ), - autopre: None, - pre: None, - which_captures: None, - nfa_size_limit: Some( - Some( - 10485760, - ), - ), - onepass_size_limit: None, - hybrid_cache_capacity: Some( - 10485760, - ), - hybrid: None, - dfa: None, - dfa_size_limit: None, - dfa_state_limit: None, - onepass: None, - backtrack: None, - byte_classes: None, - line_terminator: None, - }, - props: [], - props_union: Properties( - PropertiesI { - minimum_len: None, - maximum_len: None, - look_set: ∅, - look_set_prefix: ∅, - look_set_suffix: ∅, - look_set_prefix_any: ∅, - look_set_suffix_any: ∅, - utf8: true, - explicit_captures_len: 0, - static_explicit_captures_len: None, - literal: false, - alternation_literal: true, - }, - ), - }, - ), - pre: None, - nfa: thompson::NFA( - ^000000: FAIL - - transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) - ) - , - nfarev: Some( - thompson::NFA( - ^000000: FAIL - - transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) - ) - , - ), - pikevm: PikeVM( - PikeVMEngine( - PikeVM { - config: Config { - match_kind: Some( - All, - ), - pre: Some( - None, - ), - }, - nfa: thompson::NFA( - ^000000: FAIL - - transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) - ) - , - }, - ), - ), - backtrack: BoundedBacktracker( - None, - ), - onepass: OnePass( - None, - ), - hybrid: Hybrid( - Some( - HybridEngine( - Regex { - forward: DFA { - config: Config { - match_kind: Some( - All, - ), - pre: Some( - None, - ), - starts_for_each_pattern: Some( - true, - ), - byte_classes: Some( - true, - ), - unicode_word_boundary: Some( - true, - ), - quitset: None, - specialize_start_states: Some( - false, - ), - cache_capacity: Some( - 10485760, - ), - skip_cache_capacity_check: Some( - false, - ), - minimum_cache_clear_count: Some( - Some( - 3, - ), - ), - minimum_bytes_per_state: Some( - Some( - 10, - ), - ), - }, - nfa: thompson::NFA( - ^000000: FAIL - - transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) - ) - , - stride2: 1, - start_map: StartByteMap{/x00 => NonWordByte, /x01 => NonWordByte, /x02 => NonWordByte, /x03 => NonWordByte, /x04 => NonWordByte, /x05 => NonWordByte, /x06 => NonWordByte, /x07 => NonWordByte, /x08 => NonWordByte, /t => NonWordByte, - => LineLF, /x0B => NonWordByte, /x0C => NonWordByte, /r => LineCR, /x0E => NonWordByte, /x0F => NonWordByte, /x10 => NonWordByte, /x11 => NonWordByte, /x12 => NonWordByte, /x13 => NonWordByte, /x14 => NonWordByte, /x15 => NonWordByte, /x16 => NonWordByte, /x17 => NonWordByte, /x18 => NonWordByte, /x19 => NonWordByte, /x1A => NonWordByte, /x1B => NonWordByte, /x1C => NonWordByte, /x1D => NonWordByte, /x1E => NonWordByte, /x1F => NonWordByte, ' ' => NonWordByte, ! => NonWordByte, /" => NonWordByte, # => NonWordByte, $ => NonWordByte, % => NonWordByte, & => NonWordByte, /' => NonWordByte, ( => NonWordByte, ) => NonWordByte, * => NonWordByte, + => NonWordByte, , => NonWordByte, - => NonWordByte, . => NonWordByte, / => NonWordByte, 0 => WordByte, 1 => WordByte, 2 => WordByte, 3 => WordByte, 4 => WordByte, 5 => WordByte, 6 => WordByte, 7 => WordByte, 8 => WordByte, 9 => WordByte, : => NonWordByte, ; => NonWordByte, < => NonWordByte, = => NonWordByte, > => NonWordByte, ? => NonWordByte, @ => NonWordByte, A => WordByte, B => WordByte, C => WordByte, D => WordByte, E => WordByte, F => WordByte, G => WordByte, H => WordByte, I => WordByte, J => WordByte, K => WordByte, L => WordByte, M => WordByte, N => WordByte, O => WordByte, P => WordByte, Q => WordByte, R => WordByte, S => WordByte, T => WordByte, U => WordByte, V => WordByte, W => WordByte, X => WordByte, Y => WordByte, Z => WordByte, [ => NonWordByte, // => NonWordByte, ] => NonWordByte, ^ => NonWordByte, _ => WordByte, ` => NonWordByte, a => WordByte, b => WordByte, c => WordByte, d => WordByte, e => WordByte, f => WordByte, g => WordByte, h => WordByte, i => WordByte, j => WordByte, k => WordByte, l => WordByte, m => WordByte, n => WordByte, o => WordByte, p => WordByte, q => WordByte, r => WordByte, s => WordByte, t => WordByte, u => WordByte, v => WordByte, w => WordByte, x => WordByte, y => WordByte, z => WordByte, { => NonWordByte, | => NonWordByte, } => NonWordByte, ~ => NonWordByte, /x7F => NonWordByte, /x80 => NonWordByte, /x81 => NonWordByte, /x82 => NonWordByte, /x83 => NonWordByte, /x84 => NonWordByte, /x85 => NonWordByte, /x86 => NonWordByte, /x87 => NonWordByte, /x88 => NonWordByte, /x89 => NonWordByte, /x8A => NonWordByte, /x8B => NonWordByte, /x8C => NonWordByte, /x8D => NonWordByte, /x8E => NonWordByte, /x8F => NonWordByte, /x90 => NonWordByte, /x91 => NonWordByte, /x92 => NonWordByte, /x93 => NonWordByte, /x94 => NonWordByte, /x95 => NonWordByte, /x96 => NonWordByte, /x97 => NonWordByte, /x98 => NonWordByte, /x99 => NonWordByte, /x9A => NonWordByte, /x9B => NonWordByte, /x9C => NonWordByte, /x9D => NonWordByte, /x9E => NonWordByte, /x9F => NonWordByte, /xA0 => NonWordByte, /xA1 => NonWordByte, /xA2 => NonWordByte, /xA3 => NonWordByte, /xA4 => NonWordByte, /xA5 => NonWordByte, /xA6 => NonWordByte, /xA7 => NonWordByte, /xA8 => NonWordByte, /xA9 => NonWordByte, /xAA => NonWordByte, /xAB => NonWordByte, /xAC => NonWordByte, /xAD => NonWordByte, /xAE => NonWordByte, /xAF => NonWordByte, /xB0 => NonWordByte, /xB1 => NonWordByte, /xB2 => NonWordByte, /xB3 => NonWordByte, /xB4 => NonWordByte, /xB5 => NonWordByte, /xB6 => NonWordByte, /xB7 => NonWordByte, /xB8 => NonWordByte, /xB9 => NonWordByte, /xBA => NonWordByte, /xBB => NonWordByte, /xBC => NonWordByte, /xBD => NonWordByte, /xBE => NonWordByte, /xBF => NonWordByte, /xC0 => NonWordByte, /xC1 => NonWordByte, /xC2 => NonWordByte, /xC3 => NonWordByte, /xC4 => NonWordByte, /xC5 => NonWordByte, /xC6 => NonWordByte, /xC7 => NonWordByte, /xC8 => NonWordByte, /xC9 => NonWordByte, /xCA => NonWordByte, /xCB => NonWordByte, /xCC => NonWordByte, /xCD => NonWordByte, /xCE => NonWordByte, /xCF => NonWordByte, /xD0 => NonWordByte, /xD1 => NonWordByte, /xD2 => NonWordByte, /xD3 => NonWordByte, /xD4 => NonWordByte, /xD5 => NonWordByte, /xD6 => NonWordByte, /xD7 => NonWordByte, /xD8 => NonWordByte, /xD9 => NonWordByte, /xDA => NonWordByte, /xDB => NonWordByte, /xDC => NonWordByte, /xDD => NonWordByte, /xDE => NonWordByte, /xDF => NonWordByte, /xE0 => NonWordByte, /xE1 => NonWordByte, /xE2 => NonWordByte, /xE3 => NonWordByte, /xE4 => NonWordByte, /xE5 => NonWordByte, /xE6 => NonWordByte, /xE7 => NonWordByte, /xE8 => NonWordByte, /xE9 => NonWordByte, /xEA => NonWordByte, /xEB => NonWordByte, /xEC => NonWordByte, /xED => NonWordByte, /xEE => NonWordByte, /xEF => NonWordByte, /xF0 => NonWordByte, /xF1 => NonWordByte, /xF2 => NonWordByte, /xF3 => NonWordByte, /xF4 => NonWordByte, /xF5 => NonWordByte, /xF6 => NonWordByte, /xF7 => NonWordByte, /xF8 => NonWordByte, /xF9 => NonWordByte, /xFA => NonWordByte, /xFB => NonWordByte, /xFC => NonWordByte, /xFD => NonWordByte, /xFE => NonWordByte, /xFF => NonWordByte}, - classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]), - quitset: ByteSet { - bits: {}, - }, - cache_capacity: 10485760, - }, - reverse: DFA { - config: Config { - match_kind: Some( - All, - ), - pre: Some( - None, - ), - starts_for_each_pattern: Some( - true, - ), - byte_classes: Some( - true, - ), - unicode_word_boundary: Some( - true, - ), - quitset: None, - specialize_start_states: Some( - false, - ), - cache_capacity: Some( - 10485760, - ), - skip_cache_capacity_check: Some( - false, - ), - minimum_cache_clear_count: Some( - Some( - 3, - ), - ), - minimum_bytes_per_state: Some( - Some( - 10, - ), - ), - }, - nfa: thompson::NFA( - ^000000: FAIL - - transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) - ) - , - stride2: 1, - start_map: StartByteMap{/x00 => NonWordByte, /x01 => NonWordByte, /x02 => NonWordByte, /x03 => NonWordByte, /x04 => NonWordByte, /x05 => NonWordByte, /x06 => NonWordByte, /x07 => NonWordByte, /x08 => NonWordByte, /t => NonWordByte, - => LineLF, /x0B => NonWordByte, /x0C => NonWordByte, /r => LineCR, /x0E => NonWordByte, /x0F => NonWordByte, /x10 => NonWordByte, /x11 => NonWordByte, /x12 => NonWordByte, /x13 => NonWordByte, /x14 => NonWordByte, /x15 => NonWordByte, /x16 => NonWordByte, /x17 => NonWordByte, /x18 => NonWordByte, /x19 => NonWordByte, /x1A => NonWordByte, /x1B => NonWordByte, /x1C => NonWordByte, /x1D => NonWordByte, /x1E => NonWordByte, /x1F => NonWordByte, ' ' => NonWordByte, ! => NonWordByte, /" => NonWordByte, # => NonWordByte, $ => NonWordByte, % => NonWordByte, & => NonWordByte, /' => NonWordByte, ( => NonWordByte, ) => NonWordByte, * => NonWordByte, + => NonWordByte, , => NonWordByte, - => NonWordByte, . => NonWordByte, / => NonWordByte, 0 => WordByte, 1 => WordByte, 2 => WordByte, 3 => WordByte, 4 => WordByte, 5 => WordByte, 6 => WordByte, 7 => WordByte, 8 => WordByte, 9 => WordByte, : => NonWordByte, ; => NonWordByte, < => NonWordByte, = => NonWordByte, > => NonWordByte, ? => NonWordByte, @ => NonWordByte, A => WordByte, B => WordByte, C => WordByte, D => WordByte, E => WordByte, F => WordByte, G => WordByte, H => WordByte, I => WordByte, J => WordByte, K => WordByte, L => WordByte, M => WordByte, N => WordByte, O => WordByte, P => WordByte, Q => WordByte, R => WordByte, S => WordByte, T => WordByte, U => WordByte, V => WordByte, W => WordByte, X => WordByte, Y => WordByte, Z => WordByte, [ => NonWordByte, // => NonWordByte, ] => NonWordByte, ^ => NonWordByte, _ => WordByte, ` => NonWordByte, a => WordByte, b => WordByte, c => WordByte, d => WordByte, e => WordByte, f => WordByte, g => WordByte, h => WordByte, i => WordByte, j => WordByte, k => WordByte, l => WordByte, m => WordByte, n => WordByte, o => WordByte, p => WordByte, q => WordByte, r => WordByte, s => WordByte, t => WordByte, u => WordByte, v => WordByte, w => WordByte, x => WordByte, y => WordByte, z => WordByte, { => NonWordByte, | => NonWordByte, } => NonWordByte, ~ => NonWordByte, /x7F => NonWordByte, /x80 => NonWordByte, /x81 => NonWordByte, /x82 => NonWordByte, /x83 => NonWordByte, /x84 => NonWordByte, /x85 => NonWordByte, /x86 => NonWordByte, /x87 => NonWordByte, /x88 => NonWordByte, /x89 => NonWordByte, /x8A => NonWordByte, /x8B => NonWordByte, /x8C => NonWordByte, /x8D => NonWordByte, /x8E => NonWordByte, /x8F => NonWordByte, /x90 => NonWordByte, /x91 => NonWordByte, /x92 => NonWordByte, /x93 => NonWordByte, /x94 => NonWordByte, /x95 => NonWordByte, /x96 => NonWordByte, /x97 => NonWordByte, /x98 => NonWordByte, /x99 => NonWordByte, /x9A => NonWordByte, /x9B => NonWordByte, /x9C => NonWordByte, /x9D => NonWordByte, /x9E => NonWordByte, /x9F => NonWordByte, /xA0 => NonWordByte, /xA1 => NonWordByte, /xA2 => NonWordByte, /xA3 => NonWordByte, /xA4 => NonWordByte, /xA5 => NonWordByte, /xA6 => NonWordByte, /xA7 => NonWordByte, /xA8 => NonWordByte, /xA9 => NonWordByte, /xAA => NonWordByte, /xAB => NonWordByte, /xAC => NonWordByte, /xAD => NonWordByte, /xAE => NonWordByte, /xAF => NonWordByte, /xB0 => NonWordByte, /xB1 => NonWordByte, /xB2 => NonWordByte, /xB3 => NonWordByte, /xB4 => NonWordByte, /xB5 => NonWordByte, /xB6 => NonWordByte, /xB7 => NonWordByte, /xB8 => NonWordByte, /xB9 => NonWordByte, /xBA => NonWordByte, /xBB => NonWordByte, /xBC => NonWordByte, /xBD => NonWordByte, /xBE => NonWordByte, /xBF => NonWordByte, /xC0 => NonWordByte, /xC1 => NonWordByte, /xC2 => NonWordByte, /xC3 => NonWordByte, /xC4 => NonWordByte, /xC5 => NonWordByte, /xC6 => NonWordByte, /xC7 => NonWordByte, /xC8 => NonWordByte, /xC9 => NonWordByte, /xCA => NonWordByte, /xCB => NonWordByte, /xCC => NonWordByte, /xCD => NonWordByte, /xCE => NonWordByte, /xCF => NonWordByte, /xD0 => NonWordByte, /xD1 => NonWordByte, /xD2 => NonWordByte, /xD3 => NonWordByte, /xD4 => NonWordByte, /xD5 => NonWordByte, /xD6 => NonWordByte, /xD7 => NonWordByte, /xD8 => NonWordByte, /xD9 => NonWordByte, /xDA => NonWordByte, /xDB => NonWordByte, /xDC => NonWordByte, /xDD => NonWordByte, /xDE => NonWordByte, /xDF => NonWordByte, /xE0 => NonWordByte, /xE1 => NonWordByte, /xE2 => NonWordByte, /xE3 => NonWordByte, /xE4 => NonWordByte, /xE5 => NonWordByte, /xE6 => NonWordByte, /xE7 => NonWordByte, /xE8 => NonWordByte, /xE9 => NonWordByte, /xEA => NonWordByte, /xEB => NonWordByte, /xEC => NonWordByte, /xED => NonWordByte, /xEE => NonWordByte, /xEF => NonWordByte, /xF0 => NonWordByte, /xF1 => NonWordByte, /xF2 => NonWordByte, /xF3 => NonWordByte, /xF4 => NonWordByte, /xF5 => NonWordByte, /xF6 => NonWordByte, /xF7 => NonWordByte, /xF8 => NonWordByte, /xF9 => NonWordByte, /xFA => NonWordByte, /xFB => NonWordByte, /xFC => NonWordByte, /xFD => NonWordByte, /xFE => NonWordByte, /xFF => NonWordByte}, - classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]), - quitset: ByteSet { - bits: {}, - }, - cache_capacity: 10485760, - }, - }, - ), - ), - ), - dfa: DFA( - None, - ), - }, - info: RegexInfo( - RegexInfoI { - config: Config { - match_kind: Some( - All, - ), - utf8_empty: Some( - false, - ), - autopre: None, - pre: None, - which_captures: None, - nfa_size_limit: Some( - Some( - 10485760, - ), - ), - onepass_size_limit: None, - hybrid_cache_capacity: Some( - 10485760, - ), - hybrid: None, - dfa: None, - dfa_size_limit: None, - dfa_state_limit: None, - onepass: None, - backtrack: None, - byte_classes: None, - line_terminator: None, - }, - props: [], - props_union: Properties( - PropertiesI { - minimum_len: None, - maximum_len: None, - look_set: ∅, - look_set_prefix: ∅, - look_set_suffix: ∅, - look_set_prefix_any: ∅, - look_set_suffix_any: ∅, - utf8: true, - explicit_captures_len: 0, - static_explicit_captures_len: None, - literal: false, - alternation_literal: true, - }, - ), - }, - ), - }, - pool: Pool( - Pool { - stacks: [ - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - ], - owner: 0, - owner_val: UnsafeCell { .. }, - }, - ), - }, - map: [], - patset: Pool( - Pool { - stacks: [ - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - ], - owner: 0, - owner_val: UnsafeCell { .. }, - }, - ), - }, - ), - ], - }, - poisoned: false, - .. - }, - }, -} -[TRACE][pipeline/src/pipeline/mod.rs::305] pipeline_len: 1 -[TRACE][pipeline/src/pipeline/mod.rs::331] &dependency_graph: { - XvcEntity( - 2, - 9270302598983016314, - ): [], -} -[TRACE][pipeline/src/pipeline/mod.rs::343] &dependency_graph: { - XvcEntity( - 2, - 9270302598983016314, - ): [], -} -[INFO][pipeline/src/pipeline/mod.rs::347] Pipeline Graph: -digraph { - 0 [ label = "(2, 9270302598983016314)" ] -} - - -[TRACE][pipeline/src/pipeline/mod.rs::412] step_states: RwLock { - data: HStore { - map: { - XvcEntity( - 2, - 9270302598983016314, - ): Begin( - FromInit, - ), - }, - }, - poisoned: false, - .. -} -[TRACE][pipeline/src/pipeline/mod.rs::508] &step_thread_store: HStore { - map: { - XvcEntity( - 2, - 9270302598983016314, - ): ScopedJoinHandle { .. }, - }, -} -[TRACE][pipeline/src/pipeline/mod.rs::512] (step_e, &jh): ( - XvcEntity( - 2, - 9270302598983016314, - ), - ScopedJoinHandle { .. }, -) -[TRACE][pipeline/src/pipeline/mod.rs::621] params.recorded_dependencies: R1NStore { - parents: XvcStore { - map: { - XvcEntity( - 2, - 9270302598983016314, - ): XvcStep { - name: "average-age", - }, - }, - entity_index: { - XvcStep { - name: "average-age", - }: [ - XvcEntity( - 2, - 9270302598983016314, - ), - ], - }, - previous: EventLog( - [ - Add { - entity: XvcEntity( - 2, - 9270302598983016314, - ), - value: XvcStep { - name: "average-age", - }, - }, - Add { - entity: XvcEntity( - 2, - 9270302598983016314, - ), - value: XvcStep { - name: "average-age", - }, - }, - ], - ), - current: EventLog( - [], - ), - }, - children: XvcStore { - map: { - XvcEntity( - 3, - 9022053985516504033, - ): SqliteQueryDigest( - SqliteQueryDep { - path: XvcPath( - "people.db", - ), - query: "SELECT count(*) FROM People;", - query_digest: None, - xvc_metadata: None, - }, - ), - }, - entity_index: { - SqliteQueryDigest( - SqliteQueryDep { - path: XvcPath( - "people.db", - ), - query: "SELECT count(*) FROM People;", - query_digest: None, - xvc_metadata: None, - }, - ): [ - XvcEntity( - 3, - 9022053985516504033, - ), - ], - }, - previous: EventLog( - [ - Add { - entity: XvcEntity( - 3, - 9022053985516504033, - ), - value: SqliteQueryDigest( - SqliteQueryDep { - path: XvcPath( - "people.db", - ), - query: "SELECT count(*) FROM People;", - query_digest: None, - xvc_metadata: None, - }, - ), - }, - ], - ), - current: EventLog( - [], - ), - }, - child_parents: XvcStore { - map: { - XvcEntity( - 3, - 9022053985516504033, - ): ChildEntity( - XvcEntity( - 2, - 9270302598983016314, - ), - PhantomData, - PhantomData, - ), - }, - entity_index: { - ChildEntity( - XvcEntity( - 2, - 9270302598983016314, - ), - PhantomData, - PhantomData, - ): [ - XvcEntity( - 3, - 9022053985516504033, - ), - ], - }, - previous: EventLog( - [ - Add { - entity: XvcEntity( - 3, - 9022053985516504033, - ), - value: ChildEntity( - XvcEntity( - 2, - 9270302598983016314, - ), - PhantomData, - PhantomData, - ), - }, - ], - ), - current: EventLog( - [], - ), - }, -} -[TRACE][pipeline/src/pipeline/mod.rs::622] step_e: XvcEntity( - 2, - 9270302598983016314, -) -[TRACE][pipeline/src/pipeline/mod.rs::564] dep_neighbors: Neighbors { - iter: Iter( - [], - ), - ty: PhantomData, -} -[TRACE][pipeline/src/pipeline/mod.rs::623] dependency_steps(step_e, params.dependency_graph)?: {} -[TRACE][pipeline/src/pipeline/mod.rs::564] dep_neighbors: Neighbors { - iter: Iter( - [], - ), - ty: PhantomData, -} -[TRACE][pipeline/src/pipeline/mod.rs::660] &step_state: Begin( - FromInit, -) -[TRACE][pipeline/src/pipeline/mod.rs::770] step.name: "average-age" -[TRACE][pipeline/src/pipeline/mod.rs::771] &r_next_state: WaitingDependencySteps( - FromRunConditional, -) -[TRACE][pipeline/src/pipeline/mod.rs::773] &step_state: WaitingDependencySteps( - FromRunConditional, -) -[TRACE][pipeline/src/pipeline/mod.rs::660] &step_state: WaitingDependencySteps( - FromRunConditional, -) -[TRACE][pipeline/src/pipeline/mod.rs::770] step.name: "average-age" -[INFO] No dependency steps for step average-age -[TRACE][pipeline/src/pipeline/mod.rs::771] &r_next_state: CheckingOutputs( - FromDependencyStepsFinishedSuccessfully, -) -[TRACE][pipeline/src/pipeline/mod.rs::773] &step_state: CheckingOutputs( - FromDependencyStepsFinishedSuccessfully, -) -[TRACE][pipeline/src/pipeline/mod.rs::660] &step_state: CheckingOutputs( - FromDependencyStepsFinishedSuccessfully, -) -[TRACE][pipeline/src/pipeline/mod.rs::770] step.name: "average-age" -[TRACE][pipeline/src/pipeline/mod.rs::771] &r_next_state: CheckingSuperficialDiffs( - FromCheckedOutputs, -) -[TRACE][pipeline/src/pipeline/mod.rs::773] &step_state: CheckingSuperficialDiffs( - FromCheckedOutputs, -) -[TRACE][pipeline/src/pipeline/mod.rs::660] &step_state: CheckingSuperficialDiffs( - FromCheckedOutputs, -) -[TRACE][pipeline/src/pipeline/mod.rs::1008] parent_entity: XvcEntity( - 2, - 9270302598983016314, -) -[TRACE][pipeline/src/pipeline/mod.rs::1011] deps: HStore { - map: { - XvcEntity( - 3, - 9022053985516504033, - ): SqliteQueryDigest( - SqliteQueryDep { - path: XvcPath( - "people.db", - ), - query: "SELECT count(*) FROM People;", - query_digest: None, - xvc_metadata: None, - }, - ), - }, -} -[TRACE][pipeline/src/pipeline/deps/compare.rs::449] &stored: SqliteQueryDigest( - SqliteQueryDep { - path: XvcPath( - "people.db", - ), - query: "SELECT count(*) FROM People;", - query_digest: None, - xvc_metadata: None, - }, -) -[TRACE][core/src/util/pmp.rs::145] path: XvcPath( - "people.db", -) -[TRACE][core/src/util/pmp.rs::171] xvc_path: XvcPath( - "people.db", -) -[TRACE][core/src/util/pmp.rs::173] path: AbsolutePath( - "[CWD]/people.db", -) -[TRACE][core/src/util/pmp.rs::175] &md: Ok( - Metadata { - file_type: FileType( - FileType { - mode: 33188, - }, - ), - is_dir: false, - is_file: true, - permissions: Permissions( - FilePermissions { - mode: 33188, - }, - ), - modified: Ok( - SystemTime { - tv_sec: 1722024927, - tv_nsec: 262703931, - }, - ), - accessed: Ok( - SystemTime { - tv_sec: 1722025051, - tv_nsec: 673004106, - }, - ), - created: Ok( - SystemTime { - tv_sec: 1722024927, - tv_nsec: 261929105, - }, - ), - .. - }, -) -[TRACE][core/src/util/pmp.rs::152] &md: Some( - XvcMetadata { - file_type: File, - size: Some( - 8192, - ), - modified: Some( - SystemTime { - tv_sec: 1722024927, - tv_nsec: 262703931, - }, - ), - }, -) -[TRACE][pipeline/src/pipeline/mod.rs::1028] step_dependency_diffs: HStore { - map: { - XvcEntity( - 3, - 9022053985516504033, - ): RecordMissing { - actual: SqliteQueryDigest( - SqliteQueryDep { - path: XvcPath( - "people.db", - ), - query: "SELECT count(*) FROM People;", - query_digest: None, - xvc_metadata: Some( - XvcMetadata { - file_type: File, - size: Some( - 8192, - ), - modified: Some( - SystemTime { - tv_sec: 1722024927, - tv_nsec: 262703931, - }, - ), - }, - ), - }, - ), - }, - }, -} -[TRACE][pipeline/src/pipeline/mod.rs::1034] diff: RecordMissing { - actual: SqliteQueryDigest( - SqliteQueryDep { - path: XvcPath( - "people.db", - ), - query: "SELECT count(*) FROM People;", - query_digest: None, - xvc_metadata: Some( - XvcMetadata { - file_type: File, - size: Some( - 8192, - ), - modified: Some( - SystemTime { - tv_sec: 1722024927, - tv_nsec: 262703931, - }, - ), - }, - ), - }, - ), -} -[TRACE][pipeline/src/pipeline/mod.rs::1035] diff.changed(): true -[TRACE][pipeline/src/pipeline/mod.rs::1040] changed: true -[TRACE][pipeline/src/pipeline/mod.rs::770] step.name: "average-age" -[TRACE][pipeline/src/pipeline/mod.rs::771] &r_next_state: CheckingThoroughDiffs( - FromSuperficialDiffsChanged, -) -[TRACE][pipeline/src/pipeline/mod.rs::773] &step_state: CheckingThoroughDiffs( - FromSuperficialDiffsChanged, -) -[TRACE][pipeline/src/pipeline/mod.rs::660] &step_state: CheckingThoroughDiffs( - FromSuperficialDiffsChanged, -) -[TRACE][pipeline/src/pipeline/mod.rs::1065] deps: HStore { - map: { - XvcEntity( - 3, - 9022053985516504033, - ): SqliteQueryDigest( - SqliteQueryDep { - path: XvcPath( - "people.db", - ), - query: "SELECT count(*) FROM People;", - query_digest: None, - xvc_metadata: None, - }, - ), - }, -} -[TRACE][core/src/util/pmp.rs::145] path: XvcPath( - "people.db", -) -[TRACE][core/src/util/pmp.rs::152] &md: Some( - XvcMetadata { - file_type: File, - size: Some( - 8192, - ), - modified: Some( - SystemTime { - tv_sec: 1722024927, - tv_nsec: 262703931, - }, - ), - }, -) -[TRACE][pipeline/src/pipeline/deps/sqlite_query.rs::71] col: Integer( - 18, -) -[TRACE][pipeline/src/pipeline/mod.rs::770] step.name: "average-age" -[TRACE][pipeline/src/pipeline/mod.rs::771] &r_next_state: ComparingDiffsAndOutputs( - FromThoroughDiffsChanged, -) -[TRACE][pipeline/src/pipeline/mod.rs::773] &step_state: ComparingDiffsAndOutputs( - FromThoroughDiffsChanged, -) -[TRACE][pipeline/src/pipeline/mod.rs::660] &step_state: ComparingDiffsAndOutputs( - FromThoroughDiffsChanged, -) -[TRACE][pipeline/src/pipeline/mod.rs::770] step.name: "average-age" -[TRACE][pipeline/src/pipeline/mod.rs::771] &r_next_state: WaitingToRun( - FromDiffsHasChanged, -) -[TRACE][pipeline/src/pipeline/mod.rs::773] &step_state: WaitingToRun( - FromDiffsHasChanged, -) -[TRACE][pipeline/src/pipeline/mod.rs::660] &step_state: WaitingToRun( - FromDiffsHasChanged, -) -[INFO] [average-age] Dependencies has changed -[TRACE][pipeline/src/pipeline/mod.rs::1539] params: StepStateParams { - xvc_root: XvcRootInner { - absolute_path: AbsolutePath( - "[CWD]", - ), - xvc_dir: AbsolutePath( - "[CWD]/.xvc", - ), - store_dir: AbsolutePath( - "[CWD]/.xvc/store", - ), - config: XvcConfig { - current_dir: XvcConfigOption { - source: Runtime, - option: AbsolutePath( - "[CWD]", - ), - }, - config_maps: [ - XvcConfigMap { - source: Default, - map: { - "pipeline.default_params_file": String( - "params.yaml", - ), - "file.carry-in.no_parallel": Boolean( - false, - ), - "file.carry-in.force": Boolean( - false, - ), - "core.verbosity": String( - "error", - ), - "file.recheck.method": String( - "copy", - ), - "file.list.sort": String( - "name-desc", - ), - "git.use_git": Boolean( - true, - ), - "pipeline.current_pipeline": String( - "default", - ), - "core.guid": String( - "919ce8649f3e32f7", - ), - "git.auto_commit": Boolean( - true, - ), - "pipeline.default": String( - "default", - ), - "pipeline.process_pool_size": Integer( - 4, - ), - "git.command": String( - "git", - ), - "file.track.no_commit": Boolean( - false, - ), - "file.track.force": Boolean( - false, - ), - "file.list.show_dot_files": Boolean( - false, - ), - "file.list.recursive": Boolean( - false, - ), - "file.list.format": String( - "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", - ), - "cache.algorithm": String( - "blake3", - ), - "git.auto_stage": Boolean( - false, - ), - "file.list.no_summary": Boolean( - false, - ), - "file.track.text_or_binary": String( - "auto", - ), - "file.track.no_parallel": Boolean( - false, - ), - }, - }, - XvcConfigMap { - source: Project, - map: { - "core.guid": String( - "59a2d95849757118", - ), - "cache.algorithm": String( - "blake3", - ), - "file.list.recursive": Boolean( - false, - ), - "git.command": String( - "git", - ), - "pipeline.default_params_file": String( - "params.yaml", - ), - "file.track.force": Boolean( - false, - ), - "file.track.no_parallel": Boolean( - false, - ), - "git.use_git": Boolean( - true, - ), - "file.track.no_commit": Boolean( - false, - ), - "file.recheck.method": String( - "copy", - ), - "file.list.no_summary": Boolean( - false, - ), - "pipeline.current_pipeline": String( - "default", - ), - "git.auto_stage": Boolean( - false, - ), - "git.auto_commit": Boolean( - true, - ), - "file.list.show_dot_files": Boolean( - false, - ), - "file.list.format": String( - "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", - ), - "file.list.sort": String( - "name-desc", - ), - "pipeline.process_pool_size": Integer( - 4, - ), - "file.carry-in.no_parallel": Boolean( - false, - ), - "pipeline.default": String( - "default", - ), - "file.carry-in.force": Boolean( - false, - ), - "core.verbosity": String( - "error", - ), - "file.track.text_or_binary": String( - "auto", - ), - }, - }, - XvcConfigMap { - source: Local, - map: {}, - }, - XvcConfigMap { - source: Environment, - map: { - "TRYCMD_TESTS": String( - "storage,file,pipeline,core", - ), - }, - }, - XvcConfigMap { - source: CommandLine, - map: { - "core.quiet": Boolean( - false, - ), - "core.verbosity": String( - "debug", - ), - }, - }, - ], - the_config: { - "pipeline.default_params_file": XvcConfigValue { - source: Project, - value: String( - "params.yaml", - ), - }, - "file.list.sort": XvcConfigValue { - source: Project, - value: String( - "name-desc", - ), - }, - "git.use_git": XvcConfigValue { - source: Project, - value: Boolean( - true, - ), - }, - "git.auto_commit": XvcConfigValue { - source: Project, - value: Boolean( - true, - ), - }, - "core.verbosity": XvcConfigValue { - source: CommandLine, - value: String( - "debug", - ), - }, - "git.auto_stage": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "file.carry-in.force": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "pipeline.current_pipeline": XvcConfigValue { - source: Project, - value: String( - "default", - ), - }, - "cache.algorithm": XvcConfigValue { - source: Project, - value: String( - "blake3", - ), - }, - "pipeline.process_pool_size": XvcConfigValue { - source: Project, - value: Integer( - 4, - ), - }, - "file.list.no_summary": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "file.track.no_parallel": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "file.recheck.method": XvcConfigValue { - source: Project, - value: String( - "copy", - ), - }, - "core.quiet": XvcConfigValue { - source: CommandLine, - value: Boolean( - false, - ), - }, - "file.carry-in.no_parallel": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "core.guid": XvcConfigValue { - source: Project, - value: String( - "59a2d95849757118", - ), - }, - "pipeline.default": XvcConfigValue { - source: Project, - value: String( - "default", - ), - }, - "git.command": XvcConfigValue { - source: Project, - value: String( - "git", - ), - }, - "file.track.force": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "file.track.no_commit": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "file.list.recursive": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "file.list.format": XvcConfigValue { - source: Project, - value: String( - "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", - ), - }, - "file.track.text_or_binary": XvcConfigValue { - source: Project, - value: String( - "auto", - ), - }, - "TRYCMD_TESTS": XvcConfigValue { - source: Environment, - value: String( - "storage,file,pipeline,core", - ), - }, - "file.list.show_dot_files": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - }, - init_params: XvcConfigParams { - default_configuration: " -[core] -# The repository id. Please do not delete or change it. -# This is used to identify the repository and generate paths in storages. -# In the future it may be used to in other ways. -guid = /"919ce8649f3e32f7/" -# Default verbosity level. -# One of /"error/", /"warn/", /"info/" -verbosity = /"error/" - -[git] -# Automate git operations. -# Turning this off leads Xvc to behave as if it's not in a Git repository. -# Not recommended unless you're really not using Git -use_git = true -# Command to run Git process. -# You can set this to an absolute path to specify an executable -# If set to a non-absolute path, the executable will be searched in $PATH. -command = /"git/" - -# Commit changes in .xvc/ directory after commands. -# You can set this to false if you want to commit manually. -auto_commit = true - -# Stage changes in .xvc/ directory without committing. -# auto_commit implies auto_stage. -# If you want to commit manually but don't want to stage after individual Xvc commands, you can set this to true. -auto_stage = false - -[cache] -# The hash algorithm used for the cache. -# It may take blake3, blake2, sha2 or sha3 as values. -# All algorithms are selected to produce 256-bit hashes, so sha2 means SHA2-256, blake2 means BLAKE2s, etc. -# The cache path is produced by prepending algorithm name to the cache. -# Blake3 files are in .xvc/b3/, while sha2 files are in .xvc/s2/ etc. -algorithm = /"blake3/" - -[file] - -[file.track] - -# Don't move file content to cache after xvc file track -no_commit = false -# Force to track files even if they are already tracked. -force = false - -# Xvc calculates file content digest differently for text and binary files. -# This option controls whether to treat files as text or binary. -# It may take auto, text or binary as values. -# Auto check each file individually and treat it as text if it's text. -text_or_binary = /"auto/" - -# Don't use parallelism in track operations. -# Note that some of the operations are implemented in parallel by default, and this option affects some heavier operations. -no_parallel = false - -[file.list] - -# Format for `xvc file list` rows. You can reorder or remove columns. -# The following are the keys for each row: -# - {acd64}: actual content digest. All 64 digits from the workspace file's content. -# - {acd8}: actual content digest. First 8 digits the file content digest. -# - {aft}: actual file type. Whether the entry is a file (F), directory (D), -# symlink (S), hardlink (H) or reflink (R). -# - {asz}: actual size. The size of the workspace file in bytes. It uses MB, -# GB and TB to represent sizes larger than 1MB. -# - {ats}: actual timestamp. The timestamp of the workspace file. -# - {cst}: cache status. One of /"=/", /">/", /"/", /" 0 - F 000001: - >000002: /x00 => 2 - 000003: /x00 => 0 - match kind: Standard - prefilter: false - state length: 4 - pattern length: 0 - shortest pattern length: 18446744073709551615 - longest pattern length: 0 - alphabet length: 1 - stride: 1 - byte classes: ByteClasses(0 => [0-255]) - memory usage: 16 - ) - , - ), - map: [], - longest: 0, - }, - ), - Prefix( - PrefixStrategy { - matcher: AhoCorasick( - dfa::DFA( - D 000000: /x00 => 0 - F 000001: - >000002: /x00 => 2 - 000003: /x00 => 0 - match kind: Standard - prefilter: false - state length: 4 - pattern length: 0 - shortest pattern length: 18446744073709551615 - longest pattern length: 0 - alphabet length: 1 - stride: 1 - byte classes: ByteClasses(0 => [0-255]) - memory usage: 16 - ) - , - ), - map: [], - longest: 0, - }, - ), - RequiredExtension( - RequiredExtensionStrategy( - {}, - ), - ), - Regex( - RegexSetStrategy { - matcher: Regex { - imp: RegexI { - strat: Core { - info: RegexInfo( - RegexInfoI { - config: Config { - match_kind: Some( - All, - ), - utf8_empty: Some( - false, - ), - autopre: None, - pre: None, - which_captures: None, - nfa_size_limit: Some( - Some( - 10485760, - ), - ), - onepass_size_limit: None, - hybrid_cache_capacity: Some( - 10485760, - ), - hybrid: None, - dfa: None, - dfa_size_limit: None, - dfa_state_limit: None, - onepass: None, - backtrack: None, - byte_classes: None, - line_terminator: None, - }, - props: [], - props_union: Properties( - PropertiesI { - minimum_len: None, - maximum_len: None, - look_set: ∅, - look_set_prefix: ∅, - look_set_suffix: ∅, - look_set_prefix_any: ∅, - look_set_suffix_any: ∅, - utf8: true, - explicit_captures_len: 0, - static_explicit_captures_len: None, - literal: false, - alternation_literal: true, - }, - ), - }, - ), - pre: None, - nfa: thompson::NFA( - ^000000: FAIL - - transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) - ) - , - nfarev: Some( - thompson::NFA( - ^000000: FAIL - - transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) - ) - , - ), - pikevm: PikeVM( - PikeVMEngine( - PikeVM { - config: Config { - match_kind: Some( - All, - ), - pre: Some( - None, - ), - }, - nfa: thompson::NFA( - ^000000: FAIL - - transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) - ) - , - }, - ), - ), - backtrack: BoundedBacktracker( - None, - ), - onepass: OnePass( - None, - ), - hybrid: Hybrid( - Some( - HybridEngine( - Regex { - forward: DFA { - config: Config { - match_kind: Some( - All, - ), - pre: Some( - None, - ), - starts_for_each_pattern: Some( - true, - ), - byte_classes: Some( - true, - ), - unicode_word_boundary: Some( - true, - ), - quitset: None, - specialize_start_states: Some( - false, - ), - cache_capacity: Some( - 10485760, - ), - skip_cache_capacity_check: Some( - false, - ), - minimum_cache_clear_count: Some( - Some( - 3, - ), - ), - minimum_bytes_per_state: Some( - Some( - 10, - ), - ), - }, - nfa: thompson::NFA( - ^000000: FAIL - - transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) - ) - , - stride2: 1, - start_map: StartByteMap{/x00 => NonWordByte, /x01 => NonWordByte, /x02 => NonWordByte, /x03 => NonWordByte, /x04 => NonWordByte, /x05 => NonWordByte, /x06 => NonWordByte, /x07 => NonWordByte, /x08 => NonWordByte, /t => NonWordByte, - => LineLF, /x0B => NonWordByte, /x0C => NonWordByte, /r => LineCR, /x0E => NonWordByte, /x0F => NonWordByte, /x10 => NonWordByte, /x11 => NonWordByte, /x12 => NonWordByte, /x13 => NonWordByte, /x14 => NonWordByte, /x15 => NonWordByte, /x16 => NonWordByte, /x17 => NonWordByte, /x18 => NonWordByte, /x19 => NonWordByte, /x1A => NonWordByte, /x1B => NonWordByte, /x1C => NonWordByte, /x1D => NonWordByte, /x1E => NonWordByte, /x1F => NonWordByte, ' ' => NonWordByte, ! => NonWordByte, /" => NonWordByte, # => NonWordByte, $ => NonWordByte, % => NonWordByte, & => NonWordByte, /' => NonWordByte, ( => NonWordByte, ) => NonWordByte, * => NonWordByte, + => NonWordByte, , => NonWordByte, - => NonWordByte, . => NonWordByte, / => NonWordByte, 0 => WordByte, 1 => WordByte, 2 => WordByte, 3 => WordByte, 4 => WordByte, 5 => WordByte, 6 => WordByte, 7 => WordByte, 8 => WordByte, 9 => WordByte, : => NonWordByte, ; => NonWordByte, < => NonWordByte, = => NonWordByte, > => NonWordByte, ? => NonWordByte, @ => NonWordByte, A => WordByte, B => WordByte, C => WordByte, D => WordByte, E => WordByte, F => WordByte, G => WordByte, H => WordByte, I => WordByte, J => WordByte, K => WordByte, L => WordByte, M => WordByte, N => WordByte, O => WordByte, P => WordByte, Q => WordByte, R => WordByte, S => WordByte, T => WordByte, U => WordByte, V => WordByte, W => WordByte, X => WordByte, Y => WordByte, Z => WordByte, [ => NonWordByte, // => NonWordByte, ] => NonWordByte, ^ => NonWordByte, _ => WordByte, ` => NonWordByte, a => WordByte, b => WordByte, c => WordByte, d => WordByte, e => WordByte, f => WordByte, g => WordByte, h => WordByte, i => WordByte, j => WordByte, k => WordByte, l => WordByte, m => WordByte, n => WordByte, o => WordByte, p => WordByte, q => WordByte, r => WordByte, s => WordByte, t => WordByte, u => WordByte, v => WordByte, w => WordByte, x => WordByte, y => WordByte, z => WordByte, { => NonWordByte, | => NonWordByte, } => NonWordByte, ~ => NonWordByte, /x7F => NonWordByte, /x80 => NonWordByte, /x81 => NonWordByte, /x82 => NonWordByte, /x83 => NonWordByte, /x84 => NonWordByte, /x85 => NonWordByte, /x86 => NonWordByte, /x87 => NonWordByte, /x88 => NonWordByte, /x89 => NonWordByte, /x8A => NonWordByte, /x8B => NonWordByte, /x8C => NonWordByte, /x8D => NonWordByte, /x8E => NonWordByte, /x8F => NonWordByte, /x90 => NonWordByte, /x91 => NonWordByte, /x92 => NonWordByte, /x93 => NonWordByte, /x94 => NonWordByte, /x95 => NonWordByte, /x96 => NonWordByte, /x97 => NonWordByte, /x98 => NonWordByte, /x99 => NonWordByte, /x9A => NonWordByte, /x9B => NonWordByte, /x9C => NonWordByte, /x9D => NonWordByte, /x9E => NonWordByte, /x9F => NonWordByte, /xA0 => NonWordByte, /xA1 => NonWordByte, /xA2 => NonWordByte, /xA3 => NonWordByte, /xA4 => NonWordByte, /xA5 => NonWordByte, /xA6 => NonWordByte, /xA7 => NonWordByte, /xA8 => NonWordByte, /xA9 => NonWordByte, /xAA => NonWordByte, /xAB => NonWordByte, /xAC => NonWordByte, /xAD => NonWordByte, /xAE => NonWordByte, /xAF => NonWordByte, /xB0 => NonWordByte, /xB1 => NonWordByte, /xB2 => NonWordByte, /xB3 => NonWordByte, /xB4 => NonWordByte, /xB5 => NonWordByte, /xB6 => NonWordByte, /xB7 => NonWordByte, /xB8 => NonWordByte, /xB9 => NonWordByte, /xBA => NonWordByte, /xBB => NonWordByte, /xBC => NonWordByte, /xBD => NonWordByte, /xBE => NonWordByte, /xBF => NonWordByte, /xC0 => NonWordByte, /xC1 => NonWordByte, /xC2 => NonWordByte, /xC3 => NonWordByte, /xC4 => NonWordByte, /xC5 => NonWordByte, /xC6 => NonWordByte, /xC7 => NonWordByte, /xC8 => NonWordByte, /xC9 => NonWordByte, /xCA => NonWordByte, /xCB => NonWordByte, /xCC => NonWordByte, /xCD => NonWordByte, /xCE => NonWordByte, /xCF => NonWordByte, /xD0 => NonWordByte, /xD1 => NonWordByte, /xD2 => NonWordByte, /xD3 => NonWordByte, /xD4 => NonWordByte, /xD5 => NonWordByte, /xD6 => NonWordByte, /xD7 => NonWordByte, /xD8 => NonWordByte, /xD9 => NonWordByte, /xDA => NonWordByte, /xDB => NonWordByte, /xDC => NonWordByte, /xDD => NonWordByte, /xDE => NonWordByte, /xDF => NonWordByte, /xE0 => NonWordByte, /xE1 => NonWordByte, /xE2 => NonWordByte, /xE3 => NonWordByte, /xE4 => NonWordByte, /xE5 => NonWordByte, /xE6 => NonWordByte, /xE7 => NonWordByte, /xE8 => NonWordByte, /xE9 => NonWordByte, /xEA => NonWordByte, /xEB => NonWordByte, /xEC => NonWordByte, /xED => NonWordByte, /xEE => NonWordByte, /xEF => NonWordByte, /xF0 => NonWordByte, /xF1 => NonWordByte, /xF2 => NonWordByte, /xF3 => NonWordByte, /xF4 => NonWordByte, /xF5 => NonWordByte, /xF6 => NonWordByte, /xF7 => NonWordByte, /xF8 => NonWordByte, /xF9 => NonWordByte, /xFA => NonWordByte, /xFB => NonWordByte, /xFC => NonWordByte, /xFD => NonWordByte, /xFE => NonWordByte, /xFF => NonWordByte}, - classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]), - quitset: ByteSet { - bits: {}, - }, - cache_capacity: 10485760, - }, - reverse: DFA { - config: Config { - match_kind: Some( - All, - ), - pre: Some( - None, - ), - starts_for_each_pattern: Some( - true, - ), - byte_classes: Some( - true, - ), - unicode_word_boundary: Some( - true, - ), - quitset: None, - specialize_start_states: Some( - false, - ), - cache_capacity: Some( - 10485760, - ), - skip_cache_capacity_check: Some( - false, - ), - minimum_cache_clear_count: Some( - Some( - 3, - ), - ), - minimum_bytes_per_state: Some( - Some( - 10, - ), - ), - }, - nfa: thompson::NFA( - ^000000: FAIL - - transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) - ) - , - stride2: 1, - start_map: StartByteMap{/x00 => NonWordByte, /x01 => NonWordByte, /x02 => NonWordByte, /x03 => NonWordByte, /x04 => NonWordByte, /x05 => NonWordByte, /x06 => NonWordByte, /x07 => NonWordByte, /x08 => NonWordByte, /t => NonWordByte, - => LineLF, /x0B => NonWordByte, /x0C => NonWordByte, /r => LineCR, /x0E => NonWordByte, /x0F => NonWordByte, /x10 => NonWordByte, /x11 => NonWordByte, /x12 => NonWordByte, /x13 => NonWordByte, /x14 => NonWordByte, /x15 => NonWordByte, /x16 => NonWordByte, /x17 => NonWordByte, /x18 => NonWordByte, /x19 => NonWordByte, /x1A => NonWordByte, /x1B => NonWordByte, /x1C => NonWordByte, /x1D => NonWordByte, /x1E => NonWordByte, /x1F => NonWordByte, ' ' => NonWordByte, ! => NonWordByte, /" => NonWordByte, # => NonWordByte, $ => NonWordByte, % => NonWordByte, & => NonWordByte, /' => NonWordByte, ( => NonWordByte, ) => NonWordByte, * => NonWordByte, + => NonWordByte, , => NonWordByte, - => NonWordByte, . => NonWordByte, / => NonWordByte, 0 => WordByte, 1 => WordByte, 2 => WordByte, 3 => WordByte, 4 => WordByte, 5 => WordByte, 6 => WordByte, 7 => WordByte, 8 => WordByte, 9 => WordByte, : => NonWordByte, ; => NonWordByte, < => NonWordByte, = => NonWordByte, > => NonWordByte, ? => NonWordByte, @ => NonWordByte, A => WordByte, B => WordByte, C => WordByte, D => WordByte, E => WordByte, F => WordByte, G => WordByte, H => WordByte, I => WordByte, J => WordByte, K => WordByte, L => WordByte, M => WordByte, N => WordByte, O => WordByte, P => WordByte, Q => WordByte, R => WordByte, S => WordByte, T => WordByte, U => WordByte, V => WordByte, W => WordByte, X => WordByte, Y => WordByte, Z => WordByte, [ => NonWordByte, // => NonWordByte, ] => NonWordByte, ^ => NonWordByte, _ => WordByte, ` => NonWordByte, a => WordByte, b => WordByte, c => WordByte, d => WordByte, e => WordByte, f => WordByte, g => WordByte, h => WordByte, i => WordByte, j => WordByte, k => WordByte, l => WordByte, m => WordByte, n => WordByte, o => WordByte, p => WordByte, q => WordByte, r => WordByte, s => WordByte, t => WordByte, u => WordByte, v => WordByte, w => WordByte, x => WordByte, y => WordByte, z => WordByte, { => NonWordByte, | => NonWordByte, } => NonWordByte, ~ => NonWordByte, /x7F => NonWordByte, /x80 => NonWordByte, /x81 => NonWordByte, /x82 => NonWordByte, /x83 => NonWordByte, /x84 => NonWordByte, /x85 => NonWordByte, /x86 => NonWordByte, /x87 => NonWordByte, /x88 => NonWordByte, /x89 => NonWordByte, /x8A => NonWordByte, /x8B => NonWordByte, /x8C => NonWordByte, /x8D => NonWordByte, /x8E => NonWordByte, /x8F => NonWordByte, /x90 => NonWordByte, /x91 => NonWordByte, /x92 => NonWordByte, /x93 => NonWordByte, /x94 => NonWordByte, /x95 => NonWordByte, /x96 => NonWordByte, /x97 => NonWordByte, /x98 => NonWordByte, /x99 => NonWordByte, /x9A => NonWordByte, /x9B => NonWordByte, /x9C => NonWordByte, /x9D => NonWordByte, /x9E => NonWordByte, /x9F => NonWordByte, /xA0 => NonWordByte, /xA1 => NonWordByte, /xA2 => NonWordByte, /xA3 => NonWordByte, /xA4 => NonWordByte, /xA5 => NonWordByte, /xA6 => NonWordByte, /xA7 => NonWordByte, /xA8 => NonWordByte, /xA9 => NonWordByte, /xAA => NonWordByte, /xAB => NonWordByte, /xAC => NonWordByte, /xAD => NonWordByte, /xAE => NonWordByte, /xAF => NonWordByte, /xB0 => NonWordByte, /xB1 => NonWordByte, /xB2 => NonWordByte, /xB3 => NonWordByte, /xB4 => NonWordByte, /xB5 => NonWordByte, /xB6 => NonWordByte, /xB7 => NonWordByte, /xB8 => NonWordByte, /xB9 => NonWordByte, /xBA => NonWordByte, /xBB => NonWordByte, /xBC => NonWordByte, /xBD => NonWordByte, /xBE => NonWordByte, /xBF => NonWordByte, /xC0 => NonWordByte, /xC1 => NonWordByte, /xC2 => NonWordByte, /xC3 => NonWordByte, /xC4 => NonWordByte, /xC5 => NonWordByte, /xC6 => NonWordByte, /xC7 => NonWordByte, /xC8 => NonWordByte, /xC9 => NonWordByte, /xCA => NonWordByte, /xCB => NonWordByte, /xCC => NonWordByte, /xCD => NonWordByte, /xCE => NonWordByte, /xCF => NonWordByte, /xD0 => NonWordByte, /xD1 => NonWordByte, /xD2 => NonWordByte, /xD3 => NonWordByte, /xD4 => NonWordByte, /xD5 => NonWordByte, /xD6 => NonWordByte, /xD7 => NonWordByte, /xD8 => NonWordByte, /xD9 => NonWordByte, /xDA => NonWordByte, /xDB => NonWordByte, /xDC => NonWordByte, /xDD => NonWordByte, /xDE => NonWordByte, /xDF => NonWordByte, /xE0 => NonWordByte, /xE1 => NonWordByte, /xE2 => NonWordByte, /xE3 => NonWordByte, /xE4 => NonWordByte, /xE5 => NonWordByte, /xE6 => NonWordByte, /xE7 => NonWordByte, /xE8 => NonWordByte, /xE9 => NonWordByte, /xEA => NonWordByte, /xEB => NonWordByte, /xEC => NonWordByte, /xED => NonWordByte, /xEE => NonWordByte, /xEF => NonWordByte, /xF0 => NonWordByte, /xF1 => NonWordByte, /xF2 => NonWordByte, /xF3 => NonWordByte, /xF4 => NonWordByte, /xF5 => NonWordByte, /xF6 => NonWordByte, /xF7 => NonWordByte, /xF8 => NonWordByte, /xF9 => NonWordByte, /xFA => NonWordByte, /xFB => NonWordByte, /xFC => NonWordByte, /xFD => NonWordByte, /xFE => NonWordByte, /xFF => NonWordByte}, - classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]), - quitset: ByteSet { - bits: {}, - }, - cache_capacity: 10485760, - }, - }, - ), - ), - ), - dfa: DFA( - None, - ), - }, - info: RegexInfo( - RegexInfoI { - config: Config { - match_kind: Some( - All, - ), - utf8_empty: Some( - false, - ), - autopre: None, - pre: None, - which_captures: None, - nfa_size_limit: Some( - Some( - 10485760, - ), - ), - onepass_size_limit: None, - hybrid_cache_capacity: Some( - 10485760, - ), - hybrid: None, - dfa: None, - dfa_size_limit: None, - dfa_state_limit: None, - onepass: None, - backtrack: None, - byte_classes: None, - line_terminator: None, - }, - props: [], - props_union: Properties( - PropertiesI { - minimum_len: None, - maximum_len: None, - look_set: ∅, - look_set_prefix: ∅, - look_set_suffix: ∅, - look_set_prefix_any: ∅, - look_set_suffix_any: ∅, - utf8: true, - explicit_captures_len: 0, - static_explicit_captures_len: None, - literal: false, - alternation_literal: true, - }, - ), - }, - ), - }, - pool: Pool( - Pool { - stacks: [ - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - ], - owner: 0, - owner_val: UnsafeCell { .. }, - }, - ), - }, - map: [], - patset: Pool( - Pool { - stacks: [ - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - ], - owner: 0, - owner_val: UnsafeCell { .. }, - }, - ), - }, - ), - ], - }, - poisoned: false, - .. - }, - }, - }, - run_conditions: RunConditions { - never: false, - always: false, - ignore_broken_dep_steps: false, - ignore_missing_outputs: true, - }, - pipeline_rundir: XvcPath( - "", - ), - terminate_timeout_processes: true, - algorithm: Blake3, - command_process: RwLock { - data: CommandProcess { - environment: {}, - step: XvcStep { - name: "average-age", - }, - step_command: XvcStepCommand { - command: "sqlite3 people.db 'SELECT AVG(Age) FROM People;'", - }, - birth: None, - process: None, - stdout_sender: Sender { .. }, - stderr_sender: Sender { .. }, - stdout_receiver: Receiver { .. }, - stderr_receiver: Receiver { .. }, - }, - poisoned: false, - .. - }, - available_process_slots: RwLock { - data: 4, - poisoned: false, - .. - }, - process_poll_milliseconds: 10, - dependency_diffs: RwLock { - data: HStore { - map: { - XvcEntity( - 3, - 9022053985516504033, - ): RecordMissing { - actual: SqliteQueryDigest( - SqliteQueryDep { - path: XvcPath( - "people.db", - ), - query: "SELECT count(*) FROM People;", - query_digest: Some( - ContentDigest( - XvcDigest { - algorithm: Blake3, - digest: [ - 11, - 225, - 214, - 145, - 174, - 151, - 99, - 217, - 216, - 197, - 211, - 26, - 216, - 218, - 115, - 209, - 161, - 95, - 15, - 52, - 174, - 24, - 193, - 209, - 218, - 91, - 154, - 207, - 247, - 217, - 245, - 9, - ], - }, - ), - ), - xvc_metadata: Some( - XvcMetadata { - file_type: File, - size: Some( - 8192, - ), - modified: Some( - SystemTime { - tv_sec: 1722024927, - tv_nsec: 262703931, - }, - ), - }, - ), - }, - ), - }, - }, - }, - poisoned: false, - .. - }, - output_diffs: RwLock { - data: HStore { - map: {}, - }, - poisoned: false, - .. - }, - step_e: XvcEntity( - 2, - 9270302598983016314, - ), - step: XvcStep { - name: "average-age", - }, - step_command: XvcStepCommand { - command: "sqlite3 people.db 'SELECT AVG(Age) FROM People;'", - }, - current_states: RwLock { - data: HStore { - map: { - XvcEntity( - 2, - 9270302598983016314, - ): WaitingToRun( - FromDiffsHasChanged, - ), - }, - }, - poisoned: false, - .. - }, - step_timeout: 10000s, - all_steps: HStore { - map: { - XvcEntity( - 2, - 9270302598983016314, - ): XvcStep { - name: "average-age", - }, - }, - }, - recorded_dependencies: R1NStore { - parents: XvcStore { - map: { - XvcEntity( - 2, - 9270302598983016314, - ): XvcStep { - name: "average-age", - }, - }, - entity_index: { - XvcStep { - name: "average-age", - }: [ - XvcEntity( - 2, - 9270302598983016314, - ), - ], - }, - previous: EventLog( - [ - Add { - entity: XvcEntity( - 2, - 9270302598983016314, - ), - value: XvcStep { - name: "average-age", - }, - }, - Add { - entity: XvcEntity( - 2, - 9270302598983016314, - ), - value: XvcStep { - name: "average-age", - }, - }, - ], - ), - current: EventLog( - [], - ), - }, - children: XvcStore { - map: { - XvcEntity( - 3, - 9022053985516504033, - ): SqliteQueryDigest( - SqliteQueryDep { - path: XvcPath( - "people.db", - ), - query: "SELECT count(*) FROM People;", - query_digest: None, - xvc_metadata: None, - }, - ), - }, - entity_index: { - SqliteQueryDigest( - SqliteQueryDep { - path: XvcPath( - "people.db", - ), - query: "SELECT count(*) FROM People;", - query_digest: None, - xvc_metadata: None, - }, - ): [ - XvcEntity( - 3, - 9022053985516504033, - ), - ], - }, - previous: EventLog( - [ - Add { - entity: XvcEntity( - 3, - 9022053985516504033, - ), - value: SqliteQueryDigest( - SqliteQueryDep { - path: XvcPath( - "people.db", - ), - query: "SELECT count(*) FROM People;", - query_digest: None, - xvc_metadata: None, - }, - ), - }, - ], - ), - current: EventLog( - [], - ), - }, - child_parents: XvcStore { - map: { - XvcEntity( - 3, - 9022053985516504033, - ): ChildEntity( - XvcEntity( - 2, - 9270302598983016314, - ), - PhantomData, - PhantomData, - ), - }, - entity_index: { - ChildEntity( - XvcEntity( - 2, - 9270302598983016314, - ), - PhantomData, - PhantomData, - ): [ - XvcEntity( - 3, - 9022053985516504033, - ), - ], - }, - previous: EventLog( - [ - Add { - entity: XvcEntity( - 3, - 9022053985516504033, - ), - value: ChildEntity( - XvcEntity( - 2, - 9270302598983016314, - ), - PhantomData, - PhantomData, - ), - }, - ], - ), - current: EventLog( - [], - ), - }, - }, - step_dependencies: {}, - step_outputs: HStore { - map: {}, - }, -} -[TRACE][pipeline/src/pipeline/mod.rs::770] step.name: "average-age" -[TRACE][pipeline/src/pipeline/mod.rs::771] &r_next_state: Running( - FromStartProcess, -) -[TRACE][pipeline/src/pipeline/mod.rs::773] &step_state: Running( - FromStartProcess, -) -[TRACE][pipeline/src/pipeline/mod.rs::660] &step_state: Running( - FromStartProcess, -) -[TRACE][pipeline/src/pipeline/command.rs::96] self.environment: {} -[TRACE][pipeline/src/pipeline/mod.rs::770] step.name: "average-age" -[TRACE][pipeline/src/pipeline/mod.rs::771] &r_next_state: Running( - FromWaitProcess, -) -[TRACE][pipeline/src/pipeline/mod.rs::773] &step_state: Running( - FromWaitProcess, -) -[TRACE][pipeline/src/pipeline/mod.rs::660] &step_state: Running( - FromWaitProcess, -) -[TRACE][pipeline/src/pipeline/mod.rs::1418] params: StepStateParams { - xvc_root: XvcRootInner { - absolute_path: AbsolutePath( - "[CWD]", - ), - xvc_dir: AbsolutePath( - "[CWD]/.xvc", - ), - store_dir: AbsolutePath( - "[CWD]/.xvc/store", - ), - config: XvcConfig { - current_dir: XvcConfigOption { - source: Runtime, - option: AbsolutePath( - "[CWD]", - ), - }, - config_maps: [ - XvcConfigMap { - source: Default, - map: { - "pipeline.default_params_file": String( - "params.yaml", - ), - "file.carry-in.no_parallel": Boolean( - false, - ), - "file.carry-in.force": Boolean( - false, - ), - "core.verbosity": String( - "error", - ), - "file.recheck.method": String( - "copy", - ), - "file.list.sort": String( - "name-desc", - ), - "git.use_git": Boolean( - true, - ), - "pipeline.current_pipeline": String( - "default", - ), - "core.guid": String( - "919ce8649f3e32f7", - ), - "git.auto_commit": Boolean( - true, - ), - "pipeline.default": String( - "default", - ), - "pipeline.process_pool_size": Integer( - 4, - ), - "git.command": String( - "git", - ), - "file.track.no_commit": Boolean( - false, - ), - "file.track.force": Boolean( - false, - ), - "file.list.show_dot_files": Boolean( - false, - ), - "file.list.recursive": Boolean( - false, - ), - "file.list.format": String( - "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", - ), - "cache.algorithm": String( - "blake3", - ), - "git.auto_stage": Boolean( - false, - ), - "file.list.no_summary": Boolean( - false, - ), - "file.track.text_or_binary": String( - "auto", - ), - "file.track.no_parallel": Boolean( - false, - ), - }, - }, - XvcConfigMap { - source: Project, - map: { - "core.guid": String( - "59a2d95849757118", - ), - "cache.algorithm": String( - "blake3", - ), - "file.list.recursive": Boolean( - false, - ), - "git.command": String( - "git", - ), - "pipeline.default_params_file": String( - "params.yaml", - ), - "file.track.force": Boolean( - false, - ), - "file.track.no_parallel": Boolean( - false, - ), - "git.use_git": Boolean( - true, - ), - "file.track.no_commit": Boolean( - false, - ), - "file.recheck.method": String( - "copy", - ), - "file.list.no_summary": Boolean( - false, - ), - "pipeline.current_pipeline": String( - "default", - ), - "git.auto_stage": Boolean( - false, - ), - "git.auto_commit": Boolean( - true, - ), - "file.list.show_dot_files": Boolean( - false, - ), - "file.list.format": String( - "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", - ), - "file.list.sort": String( - "name-desc", - ), - "pipeline.process_pool_size": Integer( - 4, - ), - "file.carry-in.no_parallel": Boolean( - false, - ), - "pipeline.default": String( - "default", - ), - "file.carry-in.force": Boolean( - false, - ), - "core.verbosity": String( - "error", - ), - "file.track.text_or_binary": String( - "auto", - ), - }, - }, - XvcConfigMap { - source: Local, - map: {}, - }, - XvcConfigMap { - source: Environment, - map: { - "TRYCMD_TESTS": String( - "storage,file,pipeline,core", - ), - }, - }, - XvcConfigMap { - source: CommandLine, - map: { - "core.quiet": Boolean( - false, - ), - "core.verbosity": String( - "debug", - ), - }, - }, - ], - the_config: { - "pipeline.default_params_file": XvcConfigValue { - source: Project, - value: String( - "params.yaml", - ), - }, - "file.list.sort": XvcConfigValue { - source: Project, - value: String( - "name-desc", - ), - }, - "git.use_git": XvcConfigValue { - source: Project, - value: Boolean( - true, - ), - }, - "git.auto_commit": XvcConfigValue { - source: Project, - value: Boolean( - true, - ), - }, - "core.verbosity": XvcConfigValue { - source: CommandLine, - value: String( - "debug", - ), - }, - "git.auto_stage": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "file.carry-in.force": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "pipeline.current_pipeline": XvcConfigValue { - source: Project, - value: String( - "default", - ), - }, - "cache.algorithm": XvcConfigValue { - source: Project, - value: String( - "blake3", - ), - }, - "pipeline.process_pool_size": XvcConfigValue { - source: Project, - value: Integer( - 4, - ), - }, - "file.list.no_summary": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "file.track.no_parallel": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "file.recheck.method": XvcConfigValue { - source: Project, - value: String( - "copy", - ), - }, - "core.quiet": XvcConfigValue { - source: CommandLine, - value: Boolean( - false, - ), - }, - "file.carry-in.no_parallel": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "core.guid": XvcConfigValue { - source: Project, - value: String( - "59a2d95849757118", - ), - }, - "pipeline.default": XvcConfigValue { - source: Project, - value: String( - "default", - ), - }, - "git.command": XvcConfigValue { - source: Project, - value: String( - "git", - ), - }, - "file.track.force": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "file.track.no_commit": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "file.list.recursive": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "file.list.format": XvcConfigValue { - source: Project, - value: String( - "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", - ), - }, - "file.track.text_or_binary": XvcConfigValue { - source: Project, - value: String( - "auto", - ), - }, - "TRYCMD_TESTS": XvcConfigValue { - source: Environment, - value: String( - "storage,file,pipeline,core", - ), - }, - "file.list.show_dot_files": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - }, - init_params: XvcConfigParams { - default_configuration: " -[core] -# The repository id. Please do not delete or change it. -# This is used to identify the repository and generate paths in storages. -# In the future it may be used to in other ways. -guid = /"919ce8649f3e32f7/" -# Default verbosity level. -# One of /"error/", /"warn/", /"info/" -verbosity = /"error/" - -[git] -# Automate git operations. -# Turning this off leads Xvc to behave as if it's not in a Git repository. -# Not recommended unless you're really not using Git -use_git = true -# Command to run Git process. -# You can set this to an absolute path to specify an executable -# If set to a non-absolute path, the executable will be searched in $PATH. -command = /"git/" - -# Commit changes in .xvc/ directory after commands. -# You can set this to false if you want to commit manually. -auto_commit = true - -# Stage changes in .xvc/ directory without committing. -# auto_commit implies auto_stage. -# If you want to commit manually but don't want to stage after individual Xvc commands, you can set this to true. -auto_stage = false - -[cache] -# The hash algorithm used for the cache. -# It may take blake3, blake2, sha2 or sha3 as values. -# All algorithms are selected to produce 256-bit hashes, so sha2 means SHA2-256, blake2 means BLAKE2s, etc. -# The cache path is produced by prepending algorithm name to the cache. -# Blake3 files are in .xvc/b3/, while sha2 files are in .xvc/s2/ etc. -algorithm = /"blake3/" - -[file] - -[file.track] - -# Don't move file content to cache after xvc file track -no_commit = false -# Force to track files even if they are already tracked. -force = false - -# Xvc calculates file content digest differently for text and binary files. -# This option controls whether to treat files as text or binary. -# It may take auto, text or binary as values. -# Auto check each file individually and treat it as text if it's text. -text_or_binary = /"auto/" - -# Don't use parallelism in track operations. -# Note that some of the operations are implemented in parallel by default, and this option affects some heavier operations. -no_parallel = false - -[file.list] - -# Format for `xvc file list` rows. You can reorder or remove columns. -# The following are the keys for each row: -# - {acd64}: actual content digest. All 64 digits from the workspace file's content. -# - {acd8}: actual content digest. First 8 digits the file content digest. -# - {aft}: actual file type. Whether the entry is a file (F), directory (D), -# symlink (S), hardlink (H) or reflink (R). -# - {asz}: actual size. The size of the workspace file in bytes. It uses MB, -# GB and TB to represent sizes larger than 1MB. -# - {ats}: actual timestamp. The timestamp of the workspace file. -# - {cst}: cache status. One of /"=/", /">/", /"/", /" 0 - F 000001: - >000002: /x00 => 2 - 000003: /x00 => 0 - match kind: Standard - prefilter: false - state length: 4 - pattern length: 0 - shortest pattern length: 18446744073709551615 - longest pattern length: 0 - alphabet length: 1 - stride: 1 - byte classes: ByteClasses(0 => [0-255]) - memory usage: 16 - ) - , - ), - map: [], - longest: 0, - }, - ), - Prefix( - PrefixStrategy { - matcher: AhoCorasick( - dfa::DFA( - D 000000: /x00 => 0 - F 000001: - >000002: /x00 => 2 - 000003: /x00 => 0 - match kind: Standard - prefilter: false - state length: 4 - pattern length: 0 - shortest pattern length: 18446744073709551615 - longest pattern length: 0 - alphabet length: 1 - stride: 1 - byte classes: ByteClasses(0 => [0-255]) - memory usage: 16 - ) - , - ), - map: [], - longest: 0, - }, - ), - RequiredExtension( - RequiredExtensionStrategy( - {}, - ), - ), - Regex( - RegexSetStrategy { - matcher: Regex { - imp: RegexI { - strat: Core { - info: RegexInfo( - RegexInfoI { - config: Config { - match_kind: Some( - All, - ), - utf8_empty: Some( - false, - ), - autopre: None, - pre: None, - which_captures: None, - nfa_size_limit: Some( - Some( - 10485760, - ), - ), - onepass_size_limit: None, - hybrid_cache_capacity: Some( - 10485760, - ), - hybrid: None, - dfa: None, - dfa_size_limit: None, - dfa_state_limit: None, - onepass: None, - backtrack: None, - byte_classes: None, - line_terminator: None, - }, - props: [], - props_union: Properties( - PropertiesI { - minimum_len: None, - maximum_len: None, - look_set: ∅, - look_set_prefix: ∅, - look_set_suffix: ∅, - look_set_prefix_any: ∅, - look_set_suffix_any: ∅, - utf8: true, - explicit_captures_len: 0, - static_explicit_captures_len: None, - literal: false, - alternation_literal: true, - }, - ), - }, - ), - pre: None, - nfa: thompson::NFA( - ^000000: FAIL - - transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) - ) - , - nfarev: Some( - thompson::NFA( - ^000000: FAIL - - transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) - ) - , - ), - pikevm: PikeVM( - PikeVMEngine( - PikeVM { - config: Config { - match_kind: Some( - All, - ), - pre: Some( - None, - ), - }, - nfa: thompson::NFA( - ^000000: FAIL - - transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) - ) - , - }, - ), - ), - backtrack: BoundedBacktracker( - None, - ), - onepass: OnePass( - None, - ), - hybrid: Hybrid( - Some( - HybridEngine( - Regex { - forward: DFA { - config: Config { - match_kind: Some( - All, - ), - pre: Some( - None, - ), - starts_for_each_pattern: Some( - true, - ), - byte_classes: Some( - true, - ), - unicode_word_boundary: Some( - true, - ), - quitset: None, - specialize_start_states: Some( - false, - ), - cache_capacity: Some( - 10485760, - ), - skip_cache_capacity_check: Some( - false, - ), - minimum_cache_clear_count: Some( - Some( - 3, - ), - ), - minimum_bytes_per_state: Some( - Some( - 10, - ), - ), - }, - nfa: thompson::NFA( - ^000000: FAIL - - transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) - ) - , - stride2: 1, - start_map: StartByteMap{/x00 => NonWordByte, /x01 => NonWordByte, /x02 => NonWordByte, /x03 => NonWordByte, /x04 => NonWordByte, /x05 => NonWordByte, /x06 => NonWordByte, /x07 => NonWordByte, /x08 => NonWordByte, /t => NonWordByte, - => LineLF, /x0B => NonWordByte, /x0C => NonWordByte, /r => LineCR, /x0E => NonWordByte, /x0F => NonWordByte, /x10 => NonWordByte, /x11 => NonWordByte, /x12 => NonWordByte, /x13 => NonWordByte, /x14 => NonWordByte, /x15 => NonWordByte, /x16 => NonWordByte, /x17 => NonWordByte, /x18 => NonWordByte, /x19 => NonWordByte, /x1A => NonWordByte, /x1B => NonWordByte, /x1C => NonWordByte, /x1D => NonWordByte, /x1E => NonWordByte, /x1F => NonWordByte, ' ' => NonWordByte, ! => NonWordByte, /" => NonWordByte, # => NonWordByte, $ => NonWordByte, % => NonWordByte, & => NonWordByte, /' => NonWordByte, ( => NonWordByte, ) => NonWordByte, * => NonWordByte, + => NonWordByte, , => NonWordByte, - => NonWordByte, . => NonWordByte, / => NonWordByte, 0 => WordByte, 1 => WordByte, 2 => WordByte, 3 => WordByte, 4 => WordByte, 5 => WordByte, 6 => WordByte, 7 => WordByte, 8 => WordByte, 9 => WordByte, : => NonWordByte, ; => NonWordByte, < => NonWordByte, = => NonWordByte, > => NonWordByte, ? => NonWordByte, @ => NonWordByte, A => WordByte, B => WordByte, C => WordByte, D => WordByte, E => WordByte, F => WordByte, G => WordByte, H => WordByte, I => WordByte, J => WordByte, K => WordByte, L => WordByte, M => WordByte, N => WordByte, O => WordByte, P => WordByte, Q => WordByte, R => WordByte, S => WordByte, T => WordByte, U => WordByte, V => WordByte, W => WordByte, X => WordByte, Y => WordByte, Z => WordByte, [ => NonWordByte, // => NonWordByte, ] => NonWordByte, ^ => NonWordByte, _ => WordByte, ` => NonWordByte, a => WordByte, b => WordByte, c => WordByte, d => WordByte, e => WordByte, f => WordByte, g => WordByte, h => WordByte, i => WordByte, j => WordByte, k => WordByte, l => WordByte, m => WordByte, n => WordByte, o => WordByte, p => WordByte, q => WordByte, r => WordByte, s => WordByte, t => WordByte, u => WordByte, v => WordByte, w => WordByte, x => WordByte, y => WordByte, z => WordByte, { => NonWordByte, | => NonWordByte, } => NonWordByte, ~ => NonWordByte, /x7F => NonWordByte, /x80 => NonWordByte, /x81 => NonWordByte, /x82 => NonWordByte, /x83 => NonWordByte, /x84 => NonWordByte, /x85 => NonWordByte, /x86 => NonWordByte, /x87 => NonWordByte, /x88 => NonWordByte, /x89 => NonWordByte, /x8A => NonWordByte, /x8B => NonWordByte, /x8C => NonWordByte, /x8D => NonWordByte, /x8E => NonWordByte, /x8F => NonWordByte, /x90 => NonWordByte, /x91 => NonWordByte, /x92 => NonWordByte, /x93 => NonWordByte, /x94 => NonWordByte, /x95 => NonWordByte, /x96 => NonWordByte, /x97 => NonWordByte, /x98 => NonWordByte, /x99 => NonWordByte, /x9A => NonWordByte, /x9B => NonWordByte, /x9C => NonWordByte, /x9D => NonWordByte, /x9E => NonWordByte, /x9F => NonWordByte, /xA0 => NonWordByte, /xA1 => NonWordByte, /xA2 => NonWordByte, /xA3 => NonWordByte, /xA4 => NonWordByte, /xA5 => NonWordByte, /xA6 => NonWordByte, /xA7 => NonWordByte, /xA8 => NonWordByte, /xA9 => NonWordByte, /xAA => NonWordByte, /xAB => NonWordByte, /xAC => NonWordByte, /xAD => NonWordByte, /xAE => NonWordByte, /xAF => NonWordByte, /xB0 => NonWordByte, /xB1 => NonWordByte, /xB2 => NonWordByte, /xB3 => NonWordByte, /xB4 => NonWordByte, /xB5 => NonWordByte, /xB6 => NonWordByte, /xB7 => NonWordByte, /xB8 => NonWordByte, /xB9 => NonWordByte, /xBA => NonWordByte, /xBB => NonWordByte, /xBC => NonWordByte, /xBD => NonWordByte, /xBE => NonWordByte, /xBF => NonWordByte, /xC0 => NonWordByte, /xC1 => NonWordByte, /xC2 => NonWordByte, /xC3 => NonWordByte, /xC4 => NonWordByte, /xC5 => NonWordByte, /xC6 => NonWordByte, /xC7 => NonWordByte, /xC8 => NonWordByte, /xC9 => NonWordByte, /xCA => NonWordByte, /xCB => NonWordByte, /xCC => NonWordByte, /xCD => NonWordByte, /xCE => NonWordByte, /xCF => NonWordByte, /xD0 => NonWordByte, /xD1 => NonWordByte, /xD2 => NonWordByte, /xD3 => NonWordByte, /xD4 => NonWordByte, /xD5 => NonWordByte, /xD6 => NonWordByte, /xD7 => NonWordByte, /xD8 => NonWordByte, /xD9 => NonWordByte, /xDA => NonWordByte, /xDB => NonWordByte, /xDC => NonWordByte, /xDD => NonWordByte, /xDE => NonWordByte, /xDF => NonWordByte, /xE0 => NonWordByte, /xE1 => NonWordByte, /xE2 => NonWordByte, /xE3 => NonWordByte, /xE4 => NonWordByte, /xE5 => NonWordByte, /xE6 => NonWordByte, /xE7 => NonWordByte, /xE8 => NonWordByte, /xE9 => NonWordByte, /xEA => NonWordByte, /xEB => NonWordByte, /xEC => NonWordByte, /xED => NonWordByte, /xEE => NonWordByte, /xEF => NonWordByte, /xF0 => NonWordByte, /xF1 => NonWordByte, /xF2 => NonWordByte, /xF3 => NonWordByte, /xF4 => NonWordByte, /xF5 => NonWordByte, /xF6 => NonWordByte, /xF7 => NonWordByte, /xF8 => NonWordByte, /xF9 => NonWordByte, /xFA => NonWordByte, /xFB => NonWordByte, /xFC => NonWordByte, /xFD => NonWordByte, /xFE => NonWordByte, /xFF => NonWordByte}, - classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]), - quitset: ByteSet { - bits: {}, - }, - cache_capacity: 10485760, - }, - reverse: DFA { - config: Config { - match_kind: Some( - All, - ), - pre: Some( - None, - ), - starts_for_each_pattern: Some( - true, - ), - byte_classes: Some( - true, - ), - unicode_word_boundary: Some( - true, - ), - quitset: None, - specialize_start_states: Some( - false, - ), - cache_capacity: Some( - 10485760, - ), - skip_cache_capacity_check: Some( - false, - ), - minimum_cache_clear_count: Some( - Some( - 3, - ), - ), - minimum_bytes_per_state: Some( - Some( - 10, - ), - ), - }, - nfa: thompson::NFA( - ^000000: FAIL - - transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) - ) - , - stride2: 1, - start_map: StartByteMap{/x00 => NonWordByte, /x01 => NonWordByte, /x02 => NonWordByte, /x03 => NonWordByte, /x04 => NonWordByte, /x05 => NonWordByte, /x06 => NonWordByte, /x07 => NonWordByte, /x08 => NonWordByte, /t => NonWordByte, - => LineLF, /x0B => NonWordByte, /x0C => NonWordByte, /r => LineCR, /x0E => NonWordByte, /x0F => NonWordByte, /x10 => NonWordByte, /x11 => NonWordByte, /x12 => NonWordByte, /x13 => NonWordByte, /x14 => NonWordByte, /x15 => NonWordByte, /x16 => NonWordByte, /x17 => NonWordByte, /x18 => NonWordByte, /x19 => NonWordByte, /x1A => NonWordByte, /x1B => NonWordByte, /x1C => NonWordByte, /x1D => NonWordByte, /x1E => NonWordByte, /x1F => NonWordByte, ' ' => NonWordByte, ! => NonWordByte, /" => NonWordByte, # => NonWordByte, $ => NonWordByte, % => NonWordByte, & => NonWordByte, /' => NonWordByte, ( => NonWordByte, ) => NonWordByte, * => NonWordByte, + => NonWordByte, , => NonWordByte, - => NonWordByte, . => NonWordByte, / => NonWordByte, 0 => WordByte, 1 => WordByte, 2 => WordByte, 3 => WordByte, 4 => WordByte, 5 => WordByte, 6 => WordByte, 7 => WordByte, 8 => WordByte, 9 => WordByte, : => NonWordByte, ; => NonWordByte, < => NonWordByte, = => NonWordByte, > => NonWordByte, ? => NonWordByte, @ => NonWordByte, A => WordByte, B => WordByte, C => WordByte, D => WordByte, E => WordByte, F => WordByte, G => WordByte, H => WordByte, I => WordByte, J => WordByte, K => WordByte, L => WordByte, M => WordByte, N => WordByte, O => WordByte, P => WordByte, Q => WordByte, R => WordByte, S => WordByte, T => WordByte, U => WordByte, V => WordByte, W => WordByte, X => WordByte, Y => WordByte, Z => WordByte, [ => NonWordByte, // => NonWordByte, ] => NonWordByte, ^ => NonWordByte, _ => WordByte, ` => NonWordByte, a => WordByte, b => WordByte, c => WordByte, d => WordByte, e => WordByte, f => WordByte, g => WordByte, h => WordByte, i => WordByte, j => WordByte, k => WordByte, l => WordByte, m => WordByte, n => WordByte, o => WordByte, p => WordByte, q => WordByte, r => WordByte, s => WordByte, t => WordByte, u => WordByte, v => WordByte, w => WordByte, x => WordByte, y => WordByte, z => WordByte, { => NonWordByte, | => NonWordByte, } => NonWordByte, ~ => NonWordByte, /x7F => NonWordByte, /x80 => NonWordByte, /x81 => NonWordByte, /x82 => NonWordByte, /x83 => NonWordByte, /x84 => NonWordByte, /x85 => NonWordByte, /x86 => NonWordByte, /x87 => NonWordByte, /x88 => NonWordByte, /x89 => NonWordByte, /x8A => NonWordByte, /x8B => NonWordByte, /x8C => NonWordByte, /x8D => NonWordByte, /x8E => NonWordByte, /x8F => NonWordByte, /x90 => NonWordByte, /x91 => NonWordByte, /x92 => NonWordByte, /x93 => NonWordByte, /x94 => NonWordByte, /x95 => NonWordByte, /x96 => NonWordByte, /x97 => NonWordByte, /x98 => NonWordByte, /x99 => NonWordByte, /x9A => NonWordByte, /x9B => NonWordByte, /x9C => NonWordByte, /x9D => NonWordByte, /x9E => NonWordByte, /x9F => NonWordByte, /xA0 => NonWordByte, /xA1 => NonWordByte, /xA2 => NonWordByte, /xA3 => NonWordByte, /xA4 => NonWordByte, /xA5 => NonWordByte, /xA6 => NonWordByte, /xA7 => NonWordByte, /xA8 => NonWordByte, /xA9 => NonWordByte, /xAA => NonWordByte, /xAB => NonWordByte, /xAC => NonWordByte, /xAD => NonWordByte, /xAE => NonWordByte, /xAF => NonWordByte, /xB0 => NonWordByte, /xB1 => NonWordByte, /xB2 => NonWordByte, /xB3 => NonWordByte, /xB4 => NonWordByte, /xB5 => NonWordByte, /xB6 => NonWordByte, /xB7 => NonWordByte, /xB8 => NonWordByte, /xB9 => NonWordByte, /xBA => NonWordByte, /xBB => NonWordByte, /xBC => NonWordByte, /xBD => NonWordByte, /xBE => NonWordByte, /xBF => NonWordByte, /xC0 => NonWordByte, /xC1 => NonWordByte, /xC2 => NonWordByte, /xC3 => NonWordByte, /xC4 => NonWordByte, /xC5 => NonWordByte, /xC6 => NonWordByte, /xC7 => NonWordByte, /xC8 => NonWordByte, /xC9 => NonWordByte, /xCA => NonWordByte, /xCB => NonWordByte, /xCC => NonWordByte, /xCD => NonWordByte, /xCE => NonWordByte, /xCF => NonWordByte, /xD0 => NonWordByte, /xD1 => NonWordByte, /xD2 => NonWordByte, /xD3 => NonWordByte, /xD4 => NonWordByte, /xD5 => NonWordByte, /xD6 => NonWordByte, /xD7 => NonWordByte, /xD8 => NonWordByte, /xD9 => NonWordByte, /xDA => NonWordByte, /xDB => NonWordByte, /xDC => NonWordByte, /xDD => NonWordByte, /xDE => NonWordByte, /xDF => NonWordByte, /xE0 => NonWordByte, /xE1 => NonWordByte, /xE2 => NonWordByte, /xE3 => NonWordByte, /xE4 => NonWordByte, /xE5 => NonWordByte, /xE6 => NonWordByte, /xE7 => NonWordByte, /xE8 => NonWordByte, /xE9 => NonWordByte, /xEA => NonWordByte, /xEB => NonWordByte, /xEC => NonWordByte, /xED => NonWordByte, /xEE => NonWordByte, /xEF => NonWordByte, /xF0 => NonWordByte, /xF1 => NonWordByte, /xF2 => NonWordByte, /xF3 => NonWordByte, /xF4 => NonWordByte, /xF5 => NonWordByte, /xF6 => NonWordByte, /xF7 => NonWordByte, /xF8 => NonWordByte, /xF9 => NonWordByte, /xFA => NonWordByte, /xFB => NonWordByte, /xFC => NonWordByte, /xFD => NonWordByte, /xFE => NonWordByte, /xFF => NonWordByte}, - classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]), - quitset: ByteSet { - bits: {}, - }, - cache_capacity: 10485760, - }, - }, - ), - ), - ), - dfa: DFA( - None, - ), - }, - info: RegexInfo( - RegexInfoI { - config: Config { - match_kind: Some( - All, - ), - utf8_empty: Some( - false, - ), - autopre: None, - pre: None, - which_captures: None, - nfa_size_limit: Some( - Some( - 10485760, - ), - ), - onepass_size_limit: None, - hybrid_cache_capacity: Some( - 10485760, - ), - hybrid: None, - dfa: None, - dfa_size_limit: None, - dfa_state_limit: None, - onepass: None, - backtrack: None, - byte_classes: None, - line_terminator: None, - }, - props: [], - props_union: Properties( - PropertiesI { - minimum_len: None, - maximum_len: None, - look_set: ∅, - look_set_prefix: ∅, - look_set_suffix: ∅, - look_set_prefix_any: ∅, - look_set_suffix_any: ∅, - utf8: true, - explicit_captures_len: 0, - static_explicit_captures_len: None, - literal: false, - alternation_literal: true, - }, - ), - }, - ), - }, - pool: Pool( - Pool { - stacks: [ - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - ], - owner: 0, - owner_val: UnsafeCell { .. }, - }, - ), - }, - map: [], - patset: Pool( - Pool { - stacks: [ - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - ], - owner: 0, - owner_val: UnsafeCell { .. }, - }, - ), - }, - ), - ], - }, - poisoned: false, - .. - }, - }, - }, - run_conditions: RunConditions { - never: false, - always: false, - ignore_broken_dep_steps: false, - ignore_missing_outputs: true, - }, - pipeline_rundir: XvcPath( - "", - ), - terminate_timeout_processes: true, - algorithm: Blake3, - command_process: RwLock { - data: CommandProcess { - environment: {}, - step: XvcStep { - name: "average-age", - }, - step_command: XvcStepCommand { - command: "sqlite3 people.db 'SELECT AVG(Age) FROM People;'", - }, - birth: Some( - Instant { - tv_sec: 834347, - tv_nsec: 663124791, - }, - ), - process: Some( - Popen { - stdin: None, - stdout: Some( - File { - fd: 5, - read: true, - write: false, - }, - ), - stderr: Some( - File { - fd: 7, - read: true, - write: false, - }, - ), - child_state: Running { - pid: 21337, - ext: (), - }, - detached: true, - }, - ), - stdout_sender: Sender { .. }, - stderr_sender: Sender { .. }, - stdout_receiver: Receiver { .. }, - stderr_receiver: Receiver { .. }, - }, - poisoned: false, - .. - }, - available_process_slots: RwLock { - data: 3, - poisoned: false, - .. - }, - process_poll_milliseconds: 10, - dependency_diffs: RwLock { - data: HStore { - map: { - XvcEntity( - 3, - 9022053985516504033, - ): RecordMissing { - actual: SqliteQueryDigest( - SqliteQueryDep { - path: XvcPath( - "people.db", - ), - query: "SELECT count(*) FROM People;", - query_digest: Some( - ContentDigest( - XvcDigest { - algorithm: Blake3, - digest: [ - 11, - 225, - 214, - 145, - 174, - 151, - 99, - 217, - 216, - 197, - 211, - 26, - 216, - 218, - 115, - 209, - 161, - 95, - 15, - 52, - 174, - 24, - 193, - 209, - 218, - 91, - 154, - 207, - 247, - 217, - 245, - 9, - ], - }, - ), - ), - xvc_metadata: Some( - XvcMetadata { - file_type: File, - size: Some( - 8192, - ), - modified: Some( - SystemTime { - tv_sec: 1722024927, - tv_nsec: 262703931, - }, - ), - }, - ), - }, - ), - }, - }, - }, - poisoned: false, - .. - }, - output_diffs: RwLock { - data: HStore { - map: {}, - }, - poisoned: false, - .. - }, - step_e: XvcEntity( - 2, - 9270302598983016314, - ), - step: XvcStep { - name: "average-age", - }, - step_command: XvcStepCommand { - command: "sqlite3 people.db 'SELECT AVG(Age) FROM People;'", - }, - current_states: RwLock { - data: HStore { - map: { - XvcEntity( - 2, - 9270302598983016314, - ): Running( - FromWaitProcess, - ), - }, - }, - poisoned: false, - .. - }, - step_timeout: 10000s, - all_steps: HStore { - map: { - XvcEntity( - 2, - 9270302598983016314, - ): XvcStep { - name: "average-age", - }, - }, - }, - recorded_dependencies: R1NStore { - parents: XvcStore { - map: { - XvcEntity( - 2, - 9270302598983016314, - ): XvcStep { - name: "average-age", - }, - }, - entity_index: { - XvcStep { - name: "average-age", - }: [ - XvcEntity( - 2, - 9270302598983016314, - ), - ], - }, - previous: EventLog( - [ - Add { - entity: XvcEntity( - 2, - 9270302598983016314, - ), - value: XvcStep { - name: "average-age", - }, - }, - Add { - entity: XvcEntity( - 2, - 9270302598983016314, - ), - value: XvcStep { - name: "average-age", - }, - }, - ], - ), - current: EventLog( - [], - ), - }, - children: XvcStore { - map: { - XvcEntity( - 3, - 9022053985516504033, - ): SqliteQueryDigest( - SqliteQueryDep { - path: XvcPath( - "people.db", - ), - query: "SELECT count(*) FROM People;", - query_digest: None, - xvc_metadata: None, - }, - ), - }, - entity_index: { - SqliteQueryDigest( - SqliteQueryDep { - path: XvcPath( - "people.db", - ), - query: "SELECT count(*) FROM People;", - query_digest: None, - xvc_metadata: None, - }, - ): [ - XvcEntity( - 3, - 9022053985516504033, - ), - ], - }, - previous: EventLog( - [ - Add { - entity: XvcEntity( - 3, - 9022053985516504033, - ), - value: SqliteQueryDigest( - SqliteQueryDep { - path: XvcPath( - "people.db", - ), - query: "SELECT count(*) FROM People;", - query_digest: None, - xvc_metadata: None, - }, - ), - }, - ], - ), - current: EventLog( - [], - ), - }, - child_parents: XvcStore { - map: { - XvcEntity( - 3, - 9022053985516504033, - ): ChildEntity( - XvcEntity( - 2, - 9270302598983016314, - ), - PhantomData, - PhantomData, - ), - }, - entity_index: { - ChildEntity( - XvcEntity( - 2, - 9270302598983016314, - ), - PhantomData, - PhantomData, - ): [ - XvcEntity( - 3, - 9022053985516504033, - ), - ], - }, - previous: EventLog( - [ - Add { - entity: XvcEntity( - 3, - 9022053985516504033, - ), - value: ChildEntity( - XvcEntity( - 2, - 9270302598983016314, - ), - PhantomData, - PhantomData, - ), - }, - ], - ), - current: EventLog( - [], - ), - }, - }, - step_dependencies: {}, - step_outputs: HStore { - map: {}, - }, -} -[TRACE][pipeline/src/pipeline/mod.rs::1461] &process: Popen { - stdin: None, - stdout: Some( - File { - fd: 5, - read: true, - write: false, - }, - ), - stderr: Some( - File { - fd: 7, - read: true, - write: false, - }, - ), - child_state: Running { - pid: 21337, - ext: (), - }, - detached: true, -} -[TRACE][pipeline/src/pipeline/mod.rs::1466] process: Popen { - stdin: None, - stdout: Some( - File { - fd: 5, - read: true, - write: false, - }, - ), - stderr: Some( - File { - fd: 7, - read: true, - write: false, - }, - ), - child_state: Running { - pid: 21337, - ext: (), - }, - detached: true, -} -[DEBUG] Step average-age with command sqlite3 people.db 'SELECT AVG(Age) FROM People;' is still running [OUT] [average-age] 34.6666666666667 -[TRACE][pipeline/src/pipeline/mod.rs::1461] &process: Popen { - stdin: None, - stdout: Some( - File { - fd: 5, - read: true, - write: false, - }, - ), - stderr: Some( - File { - fd: 7, - read: true, - write: false, - }, - ), - child_state: Running { - pid: 21337, - ext: (), - }, - detached: true, -} [DONE] average-age (sqlite3 people.db 'SELECT AVG(Age) FROM People;') -[TRACE][pipeline/src/pipeline/mod.rs::1513] return_state: Some( - DoneByRunning( - FromProcessCompletedSuccessfully, - ), -) - -[TRACE][pipeline/src/pipeline/mod.rs::1519] params: StepStateParams { - xvc_root: XvcRootInner { - absolute_path: AbsolutePath( - "[CWD]", - ), - xvc_dir: AbsolutePath( - "[CWD]/.xvc", - ), - store_dir: AbsolutePath( - "[CWD]/.xvc/store", - ), - config: XvcConfig { - current_dir: XvcConfigOption { - source: Runtime, - option: AbsolutePath( - "[CWD]", - ), - }, - config_maps: [ - XvcConfigMap { - source: Default, - map: { - "pipeline.default_params_file": String( - "params.yaml", - ), - "file.carry-in.no_parallel": Boolean( - false, - ), - "file.carry-in.force": Boolean( - false, - ), - "core.verbosity": String( - "error", - ), - "file.recheck.method": String( - "copy", - ), - "file.list.sort": String( - "name-desc", - ), - "git.use_git": Boolean( - true, - ), - "pipeline.current_pipeline": String( - "default", - ), - "core.guid": String( - "919ce8649f3e32f7", - ), - "git.auto_commit": Boolean( - true, - ), - "pipeline.default": String( - "default", - ), - "pipeline.process_pool_size": Integer( - 4, - ), - "git.command": String( - "git", - ), - "file.track.no_commit": Boolean( - false, - ), - "file.track.force": Boolean( - false, - ), - "file.list.show_dot_files": Boolean( - false, - ), - "file.list.recursive": Boolean( - false, - ), - "file.list.format": String( - "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", - ), - "cache.algorithm": String( - "blake3", - ), - "git.auto_stage": Boolean( - false, - ), - "file.list.no_summary": Boolean( - false, - ), - "file.track.text_or_binary": String( - "auto", - ), - "file.track.no_parallel": Boolean( - false, - ), - }, - }, - XvcConfigMap { - source: Project, - map: { - "core.guid": String( - "59a2d95849757118", - ), - "cache.algorithm": String( - "blake3", - ), - "file.list.recursive": Boolean( - false, - ), - "git.command": String( - "git", - ), - "pipeline.default_params_file": String( - "params.yaml", - ), - "file.track.force": Boolean( - false, - ), - "file.track.no_parallel": Boolean( - false, - ), - "git.use_git": Boolean( - true, - ), - "file.track.no_commit": Boolean( - false, - ), - "file.recheck.method": String( - "copy", - ), - "file.list.no_summary": Boolean( - false, - ), - "pipeline.current_pipeline": String( - "default", - ), - "git.auto_stage": Boolean( - false, - ), - "git.auto_commit": Boolean( - true, - ), - "file.list.show_dot_files": Boolean( - false, - ), - "file.list.format": String( - "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", - ), - "file.list.sort": String( - "name-desc", - ), - "pipeline.process_pool_size": Integer( - 4, - ), - "file.carry-in.no_parallel": Boolean( - false, - ), - "pipeline.default": String( - "default", - ), - "file.carry-in.force": Boolean( - false, - ), - "core.verbosity": String( - "error", - ), - "file.track.text_or_binary": String( - "auto", - ), - }, - }, - XvcConfigMap { - source: Local, - map: {}, - }, - XvcConfigMap { - source: Environment, - map: { - "TRYCMD_TESTS": String( - "storage,file,pipeline,core", - ), - }, - }, - XvcConfigMap { - source: CommandLine, - map: { - "core.quiet": Boolean( - false, - ), - "core.verbosity": String( - "debug", - ), - }, - }, - ], - the_config: { - "pipeline.default_params_file": XvcConfigValue { - source: Project, - value: String( - "params.yaml", - ), - }, - "file.list.sort": XvcConfigValue { - source: Project, - value: String( - "name-desc", - ), - }, - "git.use_git": XvcConfigValue { - source: Project, - value: Boolean( - true, - ), - }, - "git.auto_commit": XvcConfigValue { - source: Project, - value: Boolean( - true, - ), - }, - "core.verbosity": XvcConfigValue { - source: CommandLine, - value: String( - "debug", - ), - }, - "git.auto_stage": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "file.carry-in.force": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "pipeline.current_pipeline": XvcConfigValue { - source: Project, - value: String( - "default", - ), - }, - "cache.algorithm": XvcConfigValue { - source: Project, - value: String( - "blake3", - ), - }, - "pipeline.process_pool_size": XvcConfigValue { - source: Project, - value: Integer( - 4, - ), - }, - "file.list.no_summary": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "file.track.no_parallel": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "file.recheck.method": XvcConfigValue { - source: Project, - value: String( - "copy", - ), - }, - "core.quiet": XvcConfigValue { - source: CommandLine, - value: Boolean( - false, - ), - }, - "file.carry-in.no_parallel": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "core.guid": XvcConfigValue { - source: Project, - value: String( - "59a2d95849757118", - ), - }, - "pipeline.default": XvcConfigValue { - source: Project, - value: String( - "default", - ), - }, - "git.command": XvcConfigValue { - source: Project, - value: String( - "git", - ), - }, - "file.track.force": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "file.track.no_commit": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "file.list.recursive": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - "file.list.format": XvcConfigValue { - source: Project, - value: String( - "{{aft}}{{rrm}} {{asz}} {{ats}} {{rcd8}} {{acd8}} {{name}}", - ), - }, - "file.track.text_or_binary": XvcConfigValue { - source: Project, - value: String( - "auto", - ), - }, - "TRYCMD_TESTS": XvcConfigValue { - source: Environment, - value: String( - "storage,file,pipeline,core", - ), - }, - "file.list.show_dot_files": XvcConfigValue { - source: Project, - value: Boolean( - false, - ), - }, - }, - init_params: XvcConfigParams { - default_configuration: " -[core] -# The repository id. Please do not delete or change it. -# This is used to identify the repository and generate paths in storages. -# In the future it may be used to in other ways. -guid = /"919ce8649f3e32f7/" -# Default verbosity level. -# One of /"error/", /"warn/", /"info/" -verbosity = /"error/" - -[git] -# Automate git operations. -# Turning this off leads Xvc to behave as if it's not in a Git repository. -# Not recommended unless you're really not using Git -use_git = true -# Command to run Git process. -# You can set this to an absolute path to specify an executable -# If set to a non-absolute path, the executable will be searched in $PATH. -command = /"git/" - -# Commit changes in .xvc/ directory after commands. -# You can set this to false if you want to commit manually. -auto_commit = true - -# Stage changes in .xvc/ directory without committing. -# auto_commit implies auto_stage. -# If you want to commit manually but don't want to stage after individual Xvc commands, you can set this to true. -auto_stage = false - -[cache] -# The hash algorithm used for the cache. -# It may take blake3, blake2, sha2 or sha3 as values. -# All algorithms are selected to produce 256-bit hashes, so sha2 means SHA2-256, blake2 means BLAKE2s, etc. -# The cache path is produced by prepending algorithm name to the cache. -# Blake3 files are in .xvc/b3/, while sha2 files are in .xvc/s2/ etc. -algorithm = /"blake3/" - -[file] - -[file.track] - -# Don't move file content to cache after xvc file track -no_commit = false -# Force to track files even if they are already tracked. -force = false - -# Xvc calculates file content digest differently for text and binary files. -# This option controls whether to treat files as text or binary. -# It may take auto, text or binary as values. -# Auto check each file individually and treat it as text if it's text. -text_or_binary = /"auto/" - -# Don't use parallelism in track operations. -# Note that some of the operations are implemented in parallel by default, and this option affects some heavier operations. -no_parallel = false - -[file.list] - -# Format for `xvc file list` rows. You can reorder or remove columns. -# The following are the keys for each row: -# - {acd64}: actual content digest. All 64 digits from the workspace file's content. -# - {acd8}: actual content digest. First 8 digits the file content digest. -# - {aft}: actual file type. Whether the entry is a file (F), directory (D), -# symlink (S), hardlink (H) or reflink (R). -# - {asz}: actual size. The size of the workspace file in bytes. It uses MB, -# GB and TB to represent sizes larger than 1MB. -# - {ats}: actual timestamp. The timestamp of the workspace file. -# - {cst}: cache status. One of /"=/", /">/", /"/", /" 0 - F 000001: - >000002: /x00 => 2 - 000003: /x00 => 0 - match kind: Standard - prefilter: false - state length: 4 - pattern length: 0 - shortest pattern length: 18446744073709551615 - longest pattern length: 0 - alphabet length: 1 - stride: 1 - byte classes: ByteClasses(0 => [0-255]) - memory usage: 16 - ) - , - ), - map: [], - longest: 0, - }, - ), - Prefix( - PrefixStrategy { - matcher: AhoCorasick( - dfa::DFA( - D 000000: /x00 => 0 - F 000001: - >000002: /x00 => 2 - 000003: /x00 => 0 - match kind: Standard - prefilter: false - state length: 4 - pattern length: 0 - shortest pattern length: 18446744073709551615 - longest pattern length: 0 - alphabet length: 1 - stride: 1 - byte classes: ByteClasses(0 => [0-255]) - memory usage: 16 - ) - , - ), - map: [], - longest: 0, - }, - ), - RequiredExtension( - RequiredExtensionStrategy( - {}, - ), - ), - Regex( - RegexSetStrategy { - matcher: Regex { - imp: RegexI { - strat: Core { - info: RegexInfo( - RegexInfoI { - config: Config { - match_kind: Some( - All, - ), - utf8_empty: Some( - false, - ), - autopre: None, - pre: None, - which_captures: None, - nfa_size_limit: Some( - Some( - 10485760, - ), - ), - onepass_size_limit: None, - hybrid_cache_capacity: Some( - 10485760, - ), - hybrid: None, - dfa: None, - dfa_size_limit: None, - dfa_state_limit: None, - onepass: None, - backtrack: None, - byte_classes: None, - line_terminator: None, - }, - props: [], - props_union: Properties( - PropertiesI { - minimum_len: None, - maximum_len: None, - look_set: ∅, - look_set_prefix: ∅, - look_set_suffix: ∅, - look_set_prefix_any: ∅, - look_set_suffix_any: ∅, - utf8: true, - explicit_captures_len: 0, - static_explicit_captures_len: None, - literal: false, - alternation_literal: true, - }, - ), - }, - ), - pre: None, - nfa: thompson::NFA( - ^000000: FAIL - - transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) - ) - , - nfarev: Some( - thompson::NFA( - ^000000: FAIL - - transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) - ) - , - ), - pikevm: PikeVM( - PikeVMEngine( - PikeVM { - config: Config { - match_kind: Some( - All, - ), - pre: Some( - None, - ), - }, - nfa: thompson::NFA( - ^000000: FAIL - - transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) - ) - , - }, - ), - ), - backtrack: BoundedBacktracker( - None, - ), - onepass: OnePass( - None, - ), - hybrid: Hybrid( - Some( - HybridEngine( - Regex { - forward: DFA { - config: Config { - match_kind: Some( - All, - ), - pre: Some( - None, - ), - starts_for_each_pattern: Some( - true, - ), - byte_classes: Some( - true, - ), - unicode_word_boundary: Some( - true, - ), - quitset: None, - specialize_start_states: Some( - false, - ), - cache_capacity: Some( - 10485760, - ), - skip_cache_capacity_check: Some( - false, - ), - minimum_cache_clear_count: Some( - Some( - 3, - ), - ), - minimum_bytes_per_state: Some( - Some( - 10, - ), - ), - }, - nfa: thompson::NFA( - ^000000: FAIL - - transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) - ) - , - stride2: 1, - start_map: StartByteMap{/x00 => NonWordByte, /x01 => NonWordByte, /x02 => NonWordByte, /x03 => NonWordByte, /x04 => NonWordByte, /x05 => NonWordByte, /x06 => NonWordByte, /x07 => NonWordByte, /x08 => NonWordByte, /t => NonWordByte, - => LineLF, /x0B => NonWordByte, /x0C => NonWordByte, /r => LineCR, /x0E => NonWordByte, /x0F => NonWordByte, /x10 => NonWordByte, /x11 => NonWordByte, /x12 => NonWordByte, /x13 => NonWordByte, /x14 => NonWordByte, /x15 => NonWordByte, /x16 => NonWordByte, /x17 => NonWordByte, /x18 => NonWordByte, /x19 => NonWordByte, /x1A => NonWordByte, /x1B => NonWordByte, /x1C => NonWordByte, /x1D => NonWordByte, /x1E => NonWordByte, /x1F => NonWordByte, ' ' => NonWordByte, ! => NonWordByte, /" => NonWordByte, # => NonWordByte, $ => NonWordByte, % => NonWordByte, & => NonWordByte, /' => NonWordByte, ( => NonWordByte, ) => NonWordByte, * => NonWordByte, + => NonWordByte, , => NonWordByte, - => NonWordByte, . => NonWordByte, / => NonWordByte, 0 => WordByte, 1 => WordByte, 2 => WordByte, 3 => WordByte, 4 => WordByte, 5 => WordByte, 6 => WordByte, 7 => WordByte, 8 => WordByte, 9 => WordByte, : => NonWordByte, ; => NonWordByte, < => NonWordByte, = => NonWordByte, > => NonWordByte, ? => NonWordByte, @ => NonWordByte, A => WordByte, B => WordByte, C => WordByte, D => WordByte, E => WordByte, F => WordByte, G => WordByte, H => WordByte, I => WordByte, J => WordByte, K => WordByte, L => WordByte, M => WordByte, N => WordByte, O => WordByte, P => WordByte, Q => WordByte, R => WordByte, S => WordByte, T => WordByte, U => WordByte, V => WordByte, W => WordByte, X => WordByte, Y => WordByte, Z => WordByte, [ => NonWordByte, // => NonWordByte, ] => NonWordByte, ^ => NonWordByte, _ => WordByte, ` => NonWordByte, a => WordByte, b => WordByte, c => WordByte, d => WordByte, e => WordByte, f => WordByte, g => WordByte, h => WordByte, i => WordByte, j => WordByte, k => WordByte, l => WordByte, m => WordByte, n => WordByte, o => WordByte, p => WordByte, q => WordByte, r => WordByte, s => WordByte, t => WordByte, u => WordByte, v => WordByte, w => WordByte, x => WordByte, y => WordByte, z => WordByte, { => NonWordByte, | => NonWordByte, } => NonWordByte, ~ => NonWordByte, /x7F => NonWordByte, /x80 => NonWordByte, /x81 => NonWordByte, /x82 => NonWordByte, /x83 => NonWordByte, /x84 => NonWordByte, /x85 => NonWordByte, /x86 => NonWordByte, /x87 => NonWordByte, /x88 => NonWordByte, /x89 => NonWordByte, /x8A => NonWordByte, /x8B => NonWordByte, /x8C => NonWordByte, /x8D => NonWordByte, /x8E => NonWordByte, /x8F => NonWordByte, /x90 => NonWordByte, /x91 => NonWordByte, /x92 => NonWordByte, /x93 => NonWordByte, /x94 => NonWordByte, /x95 => NonWordByte, /x96 => NonWordByte, /x97 => NonWordByte, /x98 => NonWordByte, /x99 => NonWordByte, /x9A => NonWordByte, /x9B => NonWordByte, /x9C => NonWordByte, /x9D => NonWordByte, /x9E => NonWordByte, /x9F => NonWordByte, /xA0 => NonWordByte, /xA1 => NonWordByte, /xA2 => NonWordByte, /xA3 => NonWordByte, /xA4 => NonWordByte, /xA5 => NonWordByte, /xA6 => NonWordByte, /xA7 => NonWordByte, /xA8 => NonWordByte, /xA9 => NonWordByte, /xAA => NonWordByte, /xAB => NonWordByte, /xAC => NonWordByte, /xAD => NonWordByte, /xAE => NonWordByte, /xAF => NonWordByte, /xB0 => NonWordByte, /xB1 => NonWordByte, /xB2 => NonWordByte, /xB3 => NonWordByte, /xB4 => NonWordByte, /xB5 => NonWordByte, /xB6 => NonWordByte, /xB7 => NonWordByte, /xB8 => NonWordByte, /xB9 => NonWordByte, /xBA => NonWordByte, /xBB => NonWordByte, /xBC => NonWordByte, /xBD => NonWordByte, /xBE => NonWordByte, /xBF => NonWordByte, /xC0 => NonWordByte, /xC1 => NonWordByte, /xC2 => NonWordByte, /xC3 => NonWordByte, /xC4 => NonWordByte, /xC5 => NonWordByte, /xC6 => NonWordByte, /xC7 => NonWordByte, /xC8 => NonWordByte, /xC9 => NonWordByte, /xCA => NonWordByte, /xCB => NonWordByte, /xCC => NonWordByte, /xCD => NonWordByte, /xCE => NonWordByte, /xCF => NonWordByte, /xD0 => NonWordByte, /xD1 => NonWordByte, /xD2 => NonWordByte, /xD3 => NonWordByte, /xD4 => NonWordByte, /xD5 => NonWordByte, /xD6 => NonWordByte, /xD7 => NonWordByte, /xD8 => NonWordByte, /xD9 => NonWordByte, /xDA => NonWordByte, /xDB => NonWordByte, /xDC => NonWordByte, /xDD => NonWordByte, /xDE => NonWordByte, /xDF => NonWordByte, /xE0 => NonWordByte, /xE1 => NonWordByte, /xE2 => NonWordByte, /xE3 => NonWordByte, /xE4 => NonWordByte, /xE5 => NonWordByte, /xE6 => NonWordByte, /xE7 => NonWordByte, /xE8 => NonWordByte, /xE9 => NonWordByte, /xEA => NonWordByte, /xEB => NonWordByte, /xEC => NonWordByte, /xED => NonWordByte, /xEE => NonWordByte, /xEF => NonWordByte, /xF0 => NonWordByte, /xF1 => NonWordByte, /xF2 => NonWordByte, /xF3 => NonWordByte, /xF4 => NonWordByte, /xF5 => NonWordByte, /xF6 => NonWordByte, /xF7 => NonWordByte, /xF8 => NonWordByte, /xF9 => NonWordByte, /xFA => NonWordByte, /xFB => NonWordByte, /xFC => NonWordByte, /xFD => NonWordByte, /xFE => NonWordByte, /xFF => NonWordByte}, - classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]), - quitset: ByteSet { - bits: {}, - }, - cache_capacity: 10485760, - }, - reverse: DFA { - config: Config { - match_kind: Some( - All, - ), - pre: Some( - None, - ), - starts_for_each_pattern: Some( - true, - ), - byte_classes: Some( - true, - ), - unicode_word_boundary: Some( - true, - ), - quitset: None, - specialize_start_states: Some( - false, - ), - cache_capacity: Some( - 10485760, - ), - skip_cache_capacity_check: Some( - false, - ), - minimum_cache_clear_count: Some( - Some( - 3, - ), - ), - minimum_bytes_per_state: Some( - Some( - 10, - ), - ), - }, - nfa: thompson::NFA( - ^000000: FAIL - - transition equivalence classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]) - ) - , - stride2: 1, - start_map: StartByteMap{/x00 => NonWordByte, /x01 => NonWordByte, /x02 => NonWordByte, /x03 => NonWordByte, /x04 => NonWordByte, /x05 => NonWordByte, /x06 => NonWordByte, /x07 => NonWordByte, /x08 => NonWordByte, /t => NonWordByte, - => LineLF, /x0B => NonWordByte, /x0C => NonWordByte, /r => LineCR, /x0E => NonWordByte, /x0F => NonWordByte, /x10 => NonWordByte, /x11 => NonWordByte, /x12 => NonWordByte, /x13 => NonWordByte, /x14 => NonWordByte, /x15 => NonWordByte, /x16 => NonWordByte, /x17 => NonWordByte, /x18 => NonWordByte, /x19 => NonWordByte, /x1A => NonWordByte, /x1B => NonWordByte, /x1C => NonWordByte, /x1D => NonWordByte, /x1E => NonWordByte, /x1F => NonWordByte, ' ' => NonWordByte, ! => NonWordByte, /" => NonWordByte, # => NonWordByte, $ => NonWordByte, % => NonWordByte, & => NonWordByte, /' => NonWordByte, ( => NonWordByte, ) => NonWordByte, * => NonWordByte, + => NonWordByte, , => NonWordByte, - => NonWordByte, . => NonWordByte, / => NonWordByte, 0 => WordByte, 1 => WordByte, 2 => WordByte, 3 => WordByte, 4 => WordByte, 5 => WordByte, 6 => WordByte, 7 => WordByte, 8 => WordByte, 9 => WordByte, : => NonWordByte, ; => NonWordByte, < => NonWordByte, = => NonWordByte, > => NonWordByte, ? => NonWordByte, @ => NonWordByte, A => WordByte, B => WordByte, C => WordByte, D => WordByte, E => WordByte, F => WordByte, G => WordByte, H => WordByte, I => WordByte, J => WordByte, K => WordByte, L => WordByte, M => WordByte, N => WordByte, O => WordByte, P => WordByte, Q => WordByte, R => WordByte, S => WordByte, T => WordByte, U => WordByte, V => WordByte, W => WordByte, X => WordByte, Y => WordByte, Z => WordByte, [ => NonWordByte, // => NonWordByte, ] => NonWordByte, ^ => NonWordByte, _ => WordByte, ` => NonWordByte, a => WordByte, b => WordByte, c => WordByte, d => WordByte, e => WordByte, f => WordByte, g => WordByte, h => WordByte, i => WordByte, j => WordByte, k => WordByte, l => WordByte, m => WordByte, n => WordByte, o => WordByte, p => WordByte, q => WordByte, r => WordByte, s => WordByte, t => WordByte, u => WordByte, v => WordByte, w => WordByte, x => WordByte, y => WordByte, z => WordByte, { => NonWordByte, | => NonWordByte, } => NonWordByte, ~ => NonWordByte, /x7F => NonWordByte, /x80 => NonWordByte, /x81 => NonWordByte, /x82 => NonWordByte, /x83 => NonWordByte, /x84 => NonWordByte, /x85 => NonWordByte, /x86 => NonWordByte, /x87 => NonWordByte, /x88 => NonWordByte, /x89 => NonWordByte, /x8A => NonWordByte, /x8B => NonWordByte, /x8C => NonWordByte, /x8D => NonWordByte, /x8E => NonWordByte, /x8F => NonWordByte, /x90 => NonWordByte, /x91 => NonWordByte, /x92 => NonWordByte, /x93 => NonWordByte, /x94 => NonWordByte, /x95 => NonWordByte, /x96 => NonWordByte, /x97 => NonWordByte, /x98 => NonWordByte, /x99 => NonWordByte, /x9A => NonWordByte, /x9B => NonWordByte, /x9C => NonWordByte, /x9D => NonWordByte, /x9E => NonWordByte, /x9F => NonWordByte, /xA0 => NonWordByte, /xA1 => NonWordByte, /xA2 => NonWordByte, /xA3 => NonWordByte, /xA4 => NonWordByte, /xA5 => NonWordByte, /xA6 => NonWordByte, /xA7 => NonWordByte, /xA8 => NonWordByte, /xA9 => NonWordByte, /xAA => NonWordByte, /xAB => NonWordByte, /xAC => NonWordByte, /xAD => NonWordByte, /xAE => NonWordByte, /xAF => NonWordByte, /xB0 => NonWordByte, /xB1 => NonWordByte, /xB2 => NonWordByte, /xB3 => NonWordByte, /xB4 => NonWordByte, /xB5 => NonWordByte, /xB6 => NonWordByte, /xB7 => NonWordByte, /xB8 => NonWordByte, /xB9 => NonWordByte, /xBA => NonWordByte, /xBB => NonWordByte, /xBC => NonWordByte, /xBD => NonWordByte, /xBE => NonWordByte, /xBF => NonWordByte, /xC0 => NonWordByte, /xC1 => NonWordByte, /xC2 => NonWordByte, /xC3 => NonWordByte, /xC4 => NonWordByte, /xC5 => NonWordByte, /xC6 => NonWordByte, /xC7 => NonWordByte, /xC8 => NonWordByte, /xC9 => NonWordByte, /xCA => NonWordByte, /xCB => NonWordByte, /xCC => NonWordByte, /xCD => NonWordByte, /xCE => NonWordByte, /xCF => NonWordByte, /xD0 => NonWordByte, /xD1 => NonWordByte, /xD2 => NonWordByte, /xD3 => NonWordByte, /xD4 => NonWordByte, /xD5 => NonWordByte, /xD6 => NonWordByte, /xD7 => NonWordByte, /xD8 => NonWordByte, /xD9 => NonWordByte, /xDA => NonWordByte, /xDB => NonWordByte, /xDC => NonWordByte, /xDD => NonWordByte, /xDE => NonWordByte, /xDF => NonWordByte, /xE0 => NonWordByte, /xE1 => NonWordByte, /xE2 => NonWordByte, /xE3 => NonWordByte, /xE4 => NonWordByte, /xE5 => NonWordByte, /xE6 => NonWordByte, /xE7 => NonWordByte, /xE8 => NonWordByte, /xE9 => NonWordByte, /xEA => NonWordByte, /xEB => NonWordByte, /xEC => NonWordByte, /xED => NonWordByte, /xEE => NonWordByte, /xEF => NonWordByte, /xF0 => NonWordByte, /xF1 => NonWordByte, /xF2 => NonWordByte, /xF3 => NonWordByte, /xF4 => NonWordByte, /xF5 => NonWordByte, /xF6 => NonWordByte, /xF7 => NonWordByte, /xF8 => NonWordByte, /xF9 => NonWordByte, /xFA => NonWordByte, /xFB => NonWordByte, /xFC => NonWordByte, /xFD => NonWordByte, /xFE => NonWordByte, /xFF => NonWordByte}, - classes: ByteClasses(0 => [/x00-/xFF], 1 => [EOI]), - quitset: ByteSet { - bits: {}, - }, - cache_capacity: 10485760, - }, - }, - ), - ), - ), - dfa: DFA( - None, - ), - }, - info: RegexInfo( - RegexInfoI { - config: Config { - match_kind: Some( - All, - ), - utf8_empty: Some( - false, - ), - autopre: None, - pre: None, - which_captures: None, - nfa_size_limit: Some( - Some( - 10485760, - ), - ), - onepass_size_limit: None, - hybrid_cache_capacity: Some( - 10485760, - ), - hybrid: None, - dfa: None, - dfa_size_limit: None, - dfa_state_limit: None, - onepass: None, - backtrack: None, - byte_classes: None, - line_terminator: None, - }, - props: [], - props_union: Properties( - PropertiesI { - minimum_len: None, - maximum_len: None, - look_set: ∅, - look_set_prefix: ∅, - look_set_suffix: ∅, - look_set_prefix_any: ∅, - look_set_suffix_any: ∅, - utf8: true, - explicit_captures_len: 0, - static_explicit_captures_len: None, - literal: false, - alternation_literal: true, - }, - ), - }, - ), - }, - pool: Pool( - Pool { - stacks: [ - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - ], - owner: 0, - owner_val: UnsafeCell { .. }, - }, - ), - }, - map: [], - patset: Pool( - Pool { - stacks: [ - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - CacheLine( - Mutex { - data: [], - poisoned: false, - .. - }, - ), - ], - owner: 0, - owner_val: UnsafeCell { .. }, - }, - ), - }, - ), - ], - }, - poisoned: false, - .. - }, - }, - }, - run_conditions: RunConditions { - never: false, - always: false, - ignore_broken_dep_steps: false, - ignore_missing_outputs: true, - }, - pipeline_rundir: XvcPath( - "", - ), - terminate_timeout_processes: true, - algorithm: Blake3, - command_process: RwLock { - data: CommandProcess { - environment: {}, - step: XvcStep { - name: "average-age", - }, - step_command: XvcStepCommand { - command: "sqlite3 people.db 'SELECT AVG(Age) FROM People;'", - }, - birth: Some( - Instant { - tv_sec: 834347, - tv_nsec: 663124791, - }, - ), - process: Some( - Popen { - stdin: None, - stdout: Some( - File { - fd: 5, - read: true, - write: false, - }, - ), - stderr: Some( - File { - fd: 7, - read: true, - write: false, - }, - ), - child_state: Finished( - Exited( - 0, - ), - ), - detached: true, - }, - ), - stdout_sender: Sender { .. }, - stderr_sender: Sender { .. }, - stdout_receiver: Receiver { .. }, - stderr_receiver: Receiver { .. }, - }, - poisoned: false, - .. - }, - available_process_slots: RwLock { - data: , - poisoned: false, - .. - }, - process_poll_milliseconds: 10, - dependency_diffs: RwLock { - data: HStore { - map: { - XvcEntity( - 3, - 9022053985516504033, - ): RecordMissing { - actual: SqliteQueryDigest( - SqliteQueryDep { - path: XvcPath( - "people.db", - ), - query: "SELECT count(*) FROM People;", - query_digest: Some( - ContentDigest( - XvcDigest { - algorithm: Blake3, - digest: [ - 11, - 225, - 214, - 145, - 174, - 151, - 99, - 217, - 216, - 197, - 211, - 26, - 216, - 218, - 115, - 209, - 161, - 95, - 15, - 52, - 174, - 24, - 193, - 209, - 218, - 91, - 154, - 207, - 247, - 217, - 245, - 9, - ], - }, - ), - ), - xvc_metadata: Some( - XvcMetadata { - file_type: File, - size: Some( - 8192, - ), - modified: Some( - SystemTime { - tv_sec: 1722024927, - tv_nsec: 262703931, - }, - ), - }, - ), - }, - ), - }, - }, - }, - poisoned: false, - .. - }, - output_diffs: RwLock { - data: HStore { - map: {}, - }, - poisoned: false, - .. - }, - step_e: XvcEntity( - 2, - 9270302598983016314, - ), - step: XvcStep { - name: "average-age", - }, - step_command: XvcStepCommand { - command: "sqlite3 people.db 'SELECT AVG(Age) FROM People;'", - }, - current_states: RwLock { - data: HStore { - map: { - XvcEntity( - 2, - 9270302598983016314, - ): Running( - FromWaitProcess, - ), - }, - }, - poisoned: false, - .. - }, - step_timeout: 10000s, - all_steps: HStore { - map: { - XvcEntity( - 2, - 9270302598983016314, - ): XvcStep { - name: "average-age", - }, - }, - }, - recorded_dependencies: R1NStore { - parents: XvcStore { - map: { - XvcEntity( - 2, - 9270302598983016314, - ): XvcStep { - name: "average-age", - }, - }, - entity_index: { - XvcStep { - name: "average-age", - }: [ - XvcEntity( - 2, - 9270302598983016314, - ), - ], - }, - previous: EventLog( - [ - Add { - entity: XvcEntity( - 2, - 9270302598983016314, - ), - value: XvcStep { - name: "average-age", - }, - }, - Add { - entity: XvcEntity( - 2, - 9270302598983016314, - ), - value: XvcStep { - name: "average-age", - }, - }, - ], - ), - current: EventLog( - [], - ), - }, - children: XvcStore { - map: { - XvcEntity( - 3, - 9022053985516504033, - ): SqliteQueryDigest( - SqliteQueryDep { - path: XvcPath( - "people.db", - ), - query: "SELECT count(*) FROM People;", - query_digest: None, - xvc_metadata: None, - }, - ), - }, - entity_index: { - SqliteQueryDigest( - SqliteQueryDep { - path: XvcPath( - "people.db", - ), - query: "SELECT count(*) FROM People;", - query_digest: None, - xvc_metadata: None, - }, - ): [ - XvcEntity( - 3, - 9022053985516504033, - ), - ], - }, - previous: EventLog( - [ - Add { - entity: XvcEntity( - 3, - 9022053985516504033, - ), - value: SqliteQueryDigest( - SqliteQueryDep { - path: XvcPath( - "people.db", - ), - query: "SELECT count(*) FROM People;", - query_digest: None, - xvc_metadata: None, - }, - ), - }, - ], - ), - current: EventLog( - [], - ), - }, - child_parents: XvcStore { - map: { - XvcEntity( - 3, - 9022053985516504033, - ): ChildEntity( - XvcEntity( - 2, - 9270302598983016314, - ), - PhantomData, - PhantomData, - ), - }, - entity_index: { - ChildEntity( - XvcEntity( - 2, - 9270302598983016314, - ), - PhantomData, - PhantomData, - ): [ - XvcEntity( - 3, - 9022053985516504033, - ), - ], - }, - previous: EventLog( - [ - Add { - entity: XvcEntity( - 3, - 9022053985516504033, - ), - value: ChildEntity( - XvcEntity( - 2, - 9270302598983016314, - ), - PhantomData, - PhantomData, - ), - }, - ], - ), - current: EventLog( - [], - ), - }, - }, - step_dependencies: {}, - step_outputs: HStore { - map: {}, - }, -} -[TRACE][pipeline/src/pipeline/mod.rs::770] step.name: "average-age" -[TRACE][pipeline/src/pipeline/mod.rs::771] &r_next_state: DoneByRunning( - FromProcessCompletedSuccessfully, -) -[TRACE][pipeline/src/pipeline/mod.rs::773] &step_state: DoneByRunning( - FromProcessCompletedSuccessfully, -) -[TRACE][pipeline/src/pipeline/mod.rs::660] &step_state: DoneByRunning( - FromProcessCompletedSuccessfully, -) -[TRACE][pipeline/src/pipeline/mod.rs::519] "Before state updater": "Before state updater" -[TRACE][pipeline/src/pipeline/mod.rs::529] step_states: RwLock { - data: HStore { - map: { - XvcEntity( - 2, - 9270302598983016314, - ): DoneByRunning( - FromProcessCompletedSuccessfully, - ), - }, - }, - poisoned: false, - .. -} -[TRACE][pipeline/src/pipeline/mod.rs::536] done_successfully: Ok( - true, -) -[TRACE][core/src/util/pmp.rs::185] self.background_thread: Mutex { - data: JoinHandle { .. }, - poisoned: false, - .. -} -[TRACE][core/src/util/pmp.rs::190] self.background_thread: Mutex { - data: JoinHandle { .. }, - poisoned: false, - .. -} -[TRACE][core/src/util/pmp.rs::99] index: 1 -[TRACE][lib/src/cli/mod.rs::376] "Before handle_git_automation": "Before handle_git_automation" -[TRACE][lib/src/git.rs::30] args: [ - "-C", - "[CWD]", - "diff", - "--name-only", - "--cached", -] -[DEBUG] Using Git: /opt/homebrew/bin/git -[TRACE][lib/src/git.rs::61] git_diff_staged_out: "" -[TRACE][lib/src/git.rs::30] args: [ - "-C", - "[CWD]", - "add", - "--verbose", - "[CWD]/.xvc", - "*.gitignore", - "*.xvcignore", -] -[TRACE][lib/src/git.rs::179] git_add_output: "add '.xvc/store/xvc-dependency-store/1722171156330163.json' -" -[TRACE][lib/src/git.rs::30] args: [ - "-C", - "[CWD]", - "commit", - "-m", - "Xvc auto-commit after /'/Users/iex/github.com/iesahin/xvc/target/debug/xvc -vvvv pipeline run/'", -] -[DEBUG] Committing .xvc/ to git: [main f7aaa52] Xvc auto-commit after '/Users/iex/github.com/iesahin/xvc/target/debug/xvc -vvvv pipeline run' - 1 file changed, 1 insertion(+) - create mode 100644 .xvc/store/xvc-dependency-store/1722171156330163.json -[DEBUG] Command completed successfully. ``` From 8fe3c8b8fa9f3718aa9ee6e221ed1faf44293f33 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sun, 28 Jul 2024 16:42:30 +0300 Subject: [PATCH 086/257] fix --- pipeline/Cargo.toml | 1 - 1 file changed, 1 deletion(-) diff --git a/pipeline/Cargo.toml b/pipeline/Cargo.toml index 37a19fbc6..b8a32007b 100644 --- a/pipeline/Cargo.toml +++ b/pipeline/Cargo.toml @@ -101,6 +101,5 @@ itertools = "^0.13" derive_more = "^0.99" [dev-dependencies] -default = [] xvc-test-helper = { version = "0.6.9-alpha.1", path = "../test_helper/" } test-case = "^3.3" From 597a2f94cf821e74ac6d3c962aed3e73dd09f5b2 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sun, 28 Jul 2024 16:42:58 +0300 Subject: [PATCH 087/257] fix --- pipeline/Cargo.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pipeline/Cargo.toml b/pipeline/Cargo.toml index b8a32007b..23a1bb7a9 100644 --- a/pipeline/Cargo.toml +++ b/pipeline/Cargo.toml @@ -19,7 +19,7 @@ debug = true [features] default = [""] -bundled-sqlite = ["dep:rusqlite/bundled"] +bundled-sqlite = ["rusqlite/bundled"] [dependencies] xvc-config = { version = "0.6.9-alpha.1", path = "../config" } From 7d39958d4723e376de75cb7a483270859bf94f81 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sun, 28 Jul 2024 16:43:21 +0300 Subject: [PATCH 088/257] default = [] --- pipeline/Cargo.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pipeline/Cargo.toml b/pipeline/Cargo.toml index 23a1bb7a9..17f309663 100644 --- a/pipeline/Cargo.toml +++ b/pipeline/Cargo.toml @@ -18,7 +18,7 @@ crate-type = ["rlib"] debug = true [features] -default = [""] +default = [] bundled-sqlite = ["rusqlite/bundled"] [dependencies] From dc91b5cd89d5ddf5eac92e35fe8316787c79e2ba Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sun, 28 Jul 2024 16:45:15 +0300 Subject: [PATCH 089/257] recheck fixes --- book/src/ref/xvc-file-recheck.md | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/book/src/ref/xvc-file-recheck.md b/book/src/ref/xvc-file-recheck.md index 7859955fc..e0d71a18e 100644 --- a/book/src/ref/xvc-file-recheck.md +++ b/book/src/ref/xvc-file-recheck.md @@ -15,7 +15,7 @@ Arguments: Options: --recheck-method How to track the file contents in cache: One of copy, symlink, hardlink, reflink. - + Note: Reflink uses copy if the underlying file system doesn't support it. --no-parallel @@ -98,9 +98,9 @@ $ rm -rf dir-0002/ $ xvc -v file recheck dir-0002/ $ ls -l dir-0002/ total 24 --rw-rw-rw- [..] file-0001.bin --rw-rw-rw- [..] file-0002.bin --rw-rw-rw- [..] file-0003.bin +-rw-rw-rw-[..] file-0001.bin +-rw-rw-rw-[..] file-0002.bin +-rw-rw-rw-[..] file-0003.bin ``` @@ -117,18 +117,18 @@ $ rm -rf dir-0002/ $ xvc -v file recheck dir-0002/ --as symlink $ ls -l dir-0002/ total 0 -lrwxr-xr-x [..] file-0001.bin -> [CWD]/.xvc/b3/3c9/255/424e13d9c38a37c5ddd376e1070cdd5de66996fbc82194c462f653856d/0.bin -lrwxr-xr-x [..] file-0002.bin -> [CWD]/.xvc/b3/6bc/65f/581e3a03edb127b63b71c5690be176e2fe265266f70abc65f72613f62e/0.bin -lrwxr-xr-x [..] file-0003.bin -> [CWD]/.xvc/b3/804/fb8/edbb122e735facd7f943c1bbe754e939a968f385c12f56b10411a4a015/0.bin +lrwxr-xr-x[..] file-0001.bin -> [CWD]/.xvc/b3/3c9/255/424e13d9c38a37c5ddd376e1070cdd5de66996fbc82194c462f653856d/0.bin +lrwxr-xr-x[..] file-0002.bin -> [CWD]/.xvc/b3/6bc/65f/581e3a03edb127b63b71c5690be176e2fe265266f70abc65f72613f62e/0.bin +lrwxr-xr-x[..] file-0003.bin -> [CWD]/.xvc/b3/804/fb8/edbb122e735facd7f943c1bbe754e939a968f385c12f56b10411a4a015/0.bin $ rm -rf dir-0002/ $ xvc -v file recheck dir-0002/ $ ls -l dir-0002/ total 0 -lrwxr-xr-x [..] file-0001.bin -> [CWD]/.xvc/b3/3c9/255/424e13d9c38a37c5ddd376e1070cdd5de66996fbc82194c462f653856d/0.bin -lrwxr-xr-x [..] file-0002.bin -> [CWD]/.xvc/b3/6bc/65f/581e3a03edb127b63b71c5690be176e2fe265266f70abc65f72613f62e/0.bin -lrwxr-xr-x [..] file-0003.bin -> [CWD]/.xvc/b3/804/fb8/edbb122e735facd7f943c1bbe754e939a968f385c12f56b10411a4a015/0.bin +lrwxr-xr-x[..] file-0001.bin -> [CWD]/.xvc/b3/3c9/255/424e13d9c38a37c5ddd376e1070cdd5de66996fbc82194c462f653856d/0.bin +lrwxr-xr-x[..] file-0002.bin -> [CWD]/.xvc/b3/6bc/65f/581e3a03edb127b63b71c5690be176e2fe265266f70abc65f72613f62e/0.bin +lrwxr-xr-x[..] file-0003.bin -> [CWD]/.xvc/b3/804/fb8/edbb122e735facd7f943c1bbe754e939a968f385c12f56b10411a4a015/0.bin ``` @@ -153,8 +153,8 @@ $ xvc -vv file recheck data.txt --as hardlink $ ls -l total[..] -drwxr-xr-x [..] dir-0001 -drwxr-xr-x [..] dir-0002 +drwxr-xr-x[..] dir-0001 +drwxr-xr-x[..] dir-0002 ``` From 909fa7972382361355113a0800665e12010f77da Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sun, 28 Jul 2024 17:49:55 +0300 Subject: [PATCH 090/257] add sqlite3 to binaries --- workflow_tests/tests/z_test_docs.rs | 1 + 1 file changed, 1 insertion(+) diff --git a/workflow_tests/tests/z_test_docs.rs b/workflow_tests/tests/z_test_docs.rs index 9d4e85aee..2104cd851 100644 --- a/workflow_tests/tests/z_test_docs.rs +++ b/workflow_tests/tests/z_test_docs.rs @@ -306,6 +306,7 @@ fn z_doc_tests() -> Result<()> { .register_bin("python3", which::which("python3")) .register_bin("dvc", which::which("dvc")) .register_bin("hyperfine", which::which("hyperfine")) + .register_bin("sqlite3", which::which("sqlite3")) .case("docs/*.md") .timeout(timeout) // We skip this for the time being. From b4b2bcf843c41909ac863f5bb02f4f91a26bbf2b Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sun, 28 Jul 2024 17:51:21 +0300 Subject: [PATCH 091/257] fix ref rest --- book/src/ref/xvc-file-recheck.md | 2 +- book/src/ref/xvc-pipeline-step-dependency-sqlite-query.md | 6 +++++- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/book/src/ref/xvc-file-recheck.md b/book/src/ref/xvc-file-recheck.md index e0d71a18e..cd97b76aa 100644 --- a/book/src/ref/xvc-file-recheck.md +++ b/book/src/ref/xvc-file-recheck.md @@ -15,7 +15,7 @@ Arguments: Options: --recheck-method How to track the file contents in cache: One of copy, symlink, hardlink, reflink. - + Note: Reflink uses copy if the underlying file system doesn't support it. --no-parallel diff --git a/book/src/ref/xvc-pipeline-step-dependency-sqlite-query.md b/book/src/ref/xvc-pipeline-step-dependency-sqlite-query.md index 906d8ceb6..9d04a6bab 100644 --- a/book/src/ref/xvc-pipeline-step-dependency-sqlite-query.md +++ b/book/src/ref/xvc-pipeline-step-dependency-sqlite-query.md @@ -105,9 +105,13 @@ This time, the step will run again as the result from dependency query (`SELECT ```console $ xvc pipeline run +[OUT] [average-age] 33.3684210526316 + +[DONE] average-age (sqlite3 people.db 'SELECT AVG(Age) FROM People;') + ``` -```note +```admonition note Xvc opens the database in read-only mode to avoid locking. ``` From 12d74b981a038e39d7f678ee13e2861e5cf666f8 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sun, 28 Jul 2024 17:51:38 +0300 Subject: [PATCH 092/257] add changes --- CHANGELOG.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 30283b35d..64c9595e0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,13 @@ tree, and subtasks are marked with indentation. ## Unreleased +- Added sqlite-query dependency to xvc pipelines. A step is invalidated to run when the output from an SQLite query changes. +- Moved bundled sqlite behind a feature flag, bundled_sqlite. + +### 0.6.8 (2024-07-15) + +- Modifications for Python bindings + ## 0.6.7 (2024-03-18) - Added `xvc file share` command to share files from S3 signed URLs @@ -41,7 +48,7 @@ tree, and subtasks are marked with indentation. - Updated README and added its commands to the tests - Updated openssl dependency to use vendored feature for Python bindings - Fixed clippy warnings -- Initial Python bindings in https://github.com/iesahin/xvc.py +- Initial Python bindings in ## 0.6.2 (2023-10-13) From 44f2f8397d29506b4db530aa0f29f12caa230b5b Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sun, 28 Jul 2024 17:56:04 +0300 Subject: [PATCH 093/257] Move reflink behind feature flag --- file/Cargo.toml | 6 +++++- lib/Cargo.toml | 3 ++- 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/file/Cargo.toml b/file/Cargo.toml index 46fcd66a1..4fd646e18 100644 --- a/file/Cargo.toml +++ b/file/Cargo.toml @@ -23,6 +23,10 @@ bench = true [profile.release] debug = true +[features] +default = ["reflink"] +reflink = ["dep:reflink"] + [dependencies] xvc-logging = { version = "0.6.9-alpha.1", path = "../logging" } xvc-config = { version = "0.6.9-alpha.1", path = "../config" } @@ -61,7 +65,7 @@ dashmap = { version = "^6.0", features = ["serde", "rayon"] } ## File system jwalk = "^0.8" walkdir = "^2.4" -reflink = "^0.1" +reflink = { version = "^0.1", optional = true } relative-path = { version = "^1.9", features = ["serde"] } path-absolutize = "^3.1" glob = "^0.3" diff --git a/lib/Cargo.toml b/lib/Cargo.toml index cdd5b6f87..72e12fa7f 100644 --- a/lib/Cargo.toml +++ b/lib/Cargo.toml @@ -97,7 +97,7 @@ predicates = "^3.0" [features] -default = ["s3", "minio", "r2", "gcs", "wasabi", "digital-ocean"] +default = ["s3", "minio", "r2", "gcs", "wasabi", "digital-ocean", "reflink"] s3 = ["xvc-storage/s3"] minio = ["xvc-storage/minio"] r2 = ["xvc-storage/r2"] @@ -105,6 +105,7 @@ gcs = ["xvc-storage/gcs"] wasabi = ["xvc-storage/wasabi"] digital-ocean = ["xvc-storage/digital-ocean"] bundled-sqlite = ["xvc-pipeline/bundled-sqlite"] +reflink = ["xvc-file/reflink"] [dev-dependencies] From 2f633a709212fa2caaa991e2c07aa5b0aaa05eb8 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sun, 28 Jul 2024 18:55:41 +0300 Subject: [PATCH 094/257] fixing feature flags --- book/src/ref/xvc-file-recheck.md | 2 +- core/src/types/recheckmethod.rs | 1 + file/Cargo.toml | 6 +++--- file/src/common/mod.rs | 1 + file/src/list/mod.rs | 1 + file/src/recheck/mod.rs | 3 ++- file/src/share/mod.rs | 1 - 7 files changed, 9 insertions(+), 6 deletions(-) diff --git a/book/src/ref/xvc-file-recheck.md b/book/src/ref/xvc-file-recheck.md index cd97b76aa..e0d71a18e 100644 --- a/book/src/ref/xvc-file-recheck.md +++ b/book/src/ref/xvc-file-recheck.md @@ -15,7 +15,7 @@ Arguments: Options: --recheck-method How to track the file contents in cache: One of copy, symlink, hardlink, reflink. - + Note: Reflink uses copy if the underlying file system doesn't support it. --no-parallel diff --git a/core/src/types/recheckmethod.rs b/core/src/types/recheckmethod.rs index 86400fa0a..8a1599f00 100644 --- a/core/src/types/recheckmethod.rs +++ b/core/src/types/recheckmethod.rs @@ -66,6 +66,7 @@ pub enum RecheckMethod { /// If the symbolic link in the workspace is deleted, the cached content is not /// affected. Symlink, + #[cfg(feature="reflink")] /// Make a reflink to the cached file in the working directory. /// Reflinks are similar to symlinks, but they are copied when the file is to be written. /// Currently reflinks are supported by Btrfs, CIFS, NFS 4.2, OCFS2, overlayfs, and XFS. diff --git a/file/Cargo.toml b/file/Cargo.toml index 4fd646e18..5033604a9 100644 --- a/file/Cargo.toml +++ b/file/Cargo.toml @@ -23,9 +23,6 @@ bench = true [profile.release] debug = true -[features] -default = ["reflink"] -reflink = ["dep:reflink"] [dependencies] xvc-logging = { version = "0.6.9-alpha.1", path = "../logging" } @@ -96,6 +93,9 @@ itertools = "^0.13" derive_more = "^0.99" parse-size = "^1.0" +[features] +default = ["reflink"] +reflink = ["dep:reflink"] [dev-dependencies] xvc-test-helper = { version = "0.6.9-alpha.1", path = "../test_helper/" } diff --git a/file/src/common/mod.rs b/file/src/common/mod.rs index ac3b25407..47e6ecb3a 100644 --- a/file/src/common/mod.rs +++ b/file/src/common/mod.rs @@ -337,6 +337,7 @@ pub fn recheck_from_cache( make_symlink(&cache_path, &path)?; info!(output_snd, "[SYMLINK] {} -> {}", cache_path, path); } + #[cfg(feature="reflink")] RecheckMethod::Reflink => { match reflink::reflink_or_copy(&cache_path, &path) { Ok(None) => { diff --git a/file/src/list/mod.rs b/file/src/list/mod.rs index e4f45aa2c..38aacbec0 100644 --- a/file/src/list/mod.rs +++ b/file/src/list/mod.rs @@ -250,6 +250,7 @@ fn format_recheck_method(recheck_method: RecheckMethod) -> String { RecheckMethod::Copy => "C".to_string(), RecheckMethod::Symlink => "S".to_string(), RecheckMethod::Hardlink => "H".to_string(), + #[cfg(feature="reflink")] RecheckMethod::Reflink => "R".to_string(), } } diff --git a/file/src/recheck/mod.rs b/file/src/recheck/mod.rs index 4c18faa2b..3c0e2e273 100644 --- a/file/src/recheck/mod.rs +++ b/file/src/recheck/mod.rs @@ -40,7 +40,8 @@ use xvc_logging::{error, info, uwr, warn, watch, XvcOutputSender}; pub struct RecheckCLI { /// How to track the file contents in cache: One of copy, symlink, hardlink, reflink. /// - /// Note: Reflink uses copy if the underlying file system doesn't support it. + /// Note: Reflink support requires "reflink" feature to be enabled and uses copy if the + /// underlying file system doesn't support it. #[arg(long, alias = "as")] pub recheck_method: Option, diff --git a/file/src/share/mod.rs b/file/src/share/mod.rs index 64347e629..281b6582a 100644 --- a/file/src/share/mod.rs +++ b/file/src/share/mod.rs @@ -1,5 +1,4 @@ //! Share files from S3 compatible storages for a limited time -use std::path::PathBuf; use crate::{common::load_targets_from_store, error, Result}; use clap::{command, Parser}; From 8232d021af553f460e213c88cebc4b81246e6320 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sun, 28 Jul 2024 19:02:58 +0300 Subject: [PATCH 095/257] add reflink feature to core --- core/Cargo.toml | 7 ++++++- lib/Cargo.toml | 2 +- 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/core/Cargo.toml b/core/Cargo.toml index 18057939e..891f013f3 100644 --- a/core/Cargo.toml +++ b/core/Cargo.toml @@ -15,8 +15,12 @@ keywords = ["file", "devops", "git", "versioning", "mlops"] name = "xvc_core" crate-type = ["rlib"] +[features] +default = ["reflink"] +reflink = [] + [profile.release] -debug = true +debug = false [dependencies] xvc-config = { version = "0.6.9-alpha.1", path = "../config" } @@ -81,6 +85,7 @@ cached = "^0.53" derive_more = "^0.99" itertools = "^0.13" + [dev-dependencies] xvc-test-helper = { version = "0.6.9-alpha.1", path = "../test_helper/" } proptest = "^1.4" diff --git a/lib/Cargo.toml b/lib/Cargo.toml index 72e12fa7f..b5cae196a 100644 --- a/lib/Cargo.toml +++ b/lib/Cargo.toml @@ -105,7 +105,7 @@ gcs = ["xvc-storage/gcs"] wasabi = ["xvc-storage/wasabi"] digital-ocean = ["xvc-storage/digital-ocean"] bundled-sqlite = ["xvc-pipeline/bundled-sqlite"] -reflink = ["xvc-file/reflink"] +reflink = ["xvc-file/reflink", "xvc-core/reflink"] [dev-dependencies] From 5f48ad1623a7a1799b0079ab975f2d6fa11b0df3 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sun, 28 Jul 2024 19:03:59 +0300 Subject: [PATCH 096/257] remove reflink feature from core --- core/Cargo.toml | 4 ---- core/src/types/recheckmethod.rs | 1 - 2 files changed, 5 deletions(-) diff --git a/core/Cargo.toml b/core/Cargo.toml index 891f013f3..1703c03d9 100644 --- a/core/Cargo.toml +++ b/core/Cargo.toml @@ -15,10 +15,6 @@ keywords = ["file", "devops", "git", "versioning", "mlops"] name = "xvc_core" crate-type = ["rlib"] -[features] -default = ["reflink"] -reflink = [] - [profile.release] debug = false diff --git a/core/src/types/recheckmethod.rs b/core/src/types/recheckmethod.rs index 8a1599f00..86400fa0a 100644 --- a/core/src/types/recheckmethod.rs +++ b/core/src/types/recheckmethod.rs @@ -66,7 +66,6 @@ pub enum RecheckMethod { /// If the symbolic link in the workspace is deleted, the cached content is not /// affected. Symlink, - #[cfg(feature="reflink")] /// Make a reflink to the cached file in the working directory. /// Reflinks are similar to symlinks, but they are copied when the file is to be written. /// Currently reflinks are supported by Btrfs, CIFS, NFS 4.2, OCFS2, overlayfs, and XFS. From 84bb318619fc6fba895d4b0866a53bac44dc92fc Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sun, 28 Jul 2024 19:04:07 +0300 Subject: [PATCH 097/257] remove reflink feature from core --- lib/Cargo.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/Cargo.toml b/lib/Cargo.toml index b5cae196a..72e12fa7f 100644 --- a/lib/Cargo.toml +++ b/lib/Cargo.toml @@ -105,7 +105,7 @@ gcs = ["xvc-storage/gcs"] wasabi = ["xvc-storage/wasabi"] digital-ocean = ["xvc-storage/digital-ocean"] bundled-sqlite = ["xvc-pipeline/bundled-sqlite"] -reflink = ["xvc-file/reflink", "xvc-core/reflink"] +reflink = ["xvc-file/reflink"] [dev-dependencies] From 21e171421952b1acac8e0b52ffe17537f6f57421 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sun, 28 Jul 2024 19:17:08 +0300 Subject: [PATCH 098/257] Turn off debug flags --- Cargo.toml | 2 +- book/src/ref/xvc-file-checkout.md | 2 +- book/src/ref/xvc-file-recheck.md | 4 ++-- config/Cargo.toml | 4 ---- core/Cargo.toml | 3 --- ecs/Cargo.toml | 3 --- file/Cargo.toml | 4 ---- lib/Cargo.toml | 3 --- logging/Cargo.toml | 4 ---- pipeline/Cargo.toml | 3 --- storage/Cargo.toml | 4 ---- test_helper/Cargo.toml | 3 --- walker/Cargo.toml | 3 --- workflow_tests/Cargo.toml | 4 ---- 14 files changed, 4 insertions(+), 42 deletions(-) diff --git a/Cargo.toml b/Cargo.toml index 60f274a38..481cb28d9 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -18,4 +18,4 @@ resolver = "2" no_individual_tags = true [profile.release] -debug = true +debug = false diff --git a/book/src/ref/xvc-file-checkout.md b/book/src/ref/xvc-file-checkout.md index 5892f5b20..bfd6a3d3c 100644 --- a/book/src/ref/xvc-file-checkout.md +++ b/book/src/ref/xvc-file-checkout.md @@ -19,7 +19,7 @@ Options: --recheck-method How to track the file contents in cache: One of copy, symlink, hardlink, reflink. - Note: Reflink uses copy if the underlying file system doesn't support it. + Note: Reflink support requires "reflink" feature to be enabled and uses copy if the underlying file system doesn't support it. --no-parallel Don't use parallelism diff --git a/book/src/ref/xvc-file-recheck.md b/book/src/ref/xvc-file-recheck.md index e0d71a18e..f0c2b1a39 100644 --- a/book/src/ref/xvc-file-recheck.md +++ b/book/src/ref/xvc-file-recheck.md @@ -15,8 +15,8 @@ Arguments: Options: --recheck-method How to track the file contents in cache: One of copy, symlink, hardlink, reflink. - - Note: Reflink uses copy if the underlying file system doesn't support it. + + Note: Reflink support requires "reflink" feature to be enabled and uses copy if the underlying file system doesn't support it. --no-parallel Don't use parallelism diff --git a/config/Cargo.toml b/config/Cargo.toml index b3342fbc3..863d91506 100644 --- a/config/Cargo.toml +++ b/config/Cargo.toml @@ -15,10 +15,6 @@ keywords = ["file", "devops", "git", "versioning", "mlops"] name = "xvc_config" crate-type = ["rlib"] -[profile.release] -debug = true - - [dependencies] xvc-logging = { version = "0.6.9-alpha.1", path = "../logging" } xvc-walker = { version = "0.6.9-alpha.1", path = "../walker" } diff --git a/core/Cargo.toml b/core/Cargo.toml index 1703c03d9..f7be40f6a 100644 --- a/core/Cargo.toml +++ b/core/Cargo.toml @@ -15,9 +15,6 @@ keywords = ["file", "devops", "git", "versioning", "mlops"] name = "xvc_core" crate-type = ["rlib"] -[profile.release] -debug = false - [dependencies] xvc-config = { version = "0.6.9-alpha.1", path = "../config" } xvc-logging = { version = "0.6.9-alpha.1", path = "../logging" } diff --git a/ecs/Cargo.toml b/ecs/Cargo.toml index e3f8d2d66..08e4d737c 100644 --- a/ecs/Cargo.toml +++ b/ecs/Cargo.toml @@ -15,9 +15,6 @@ keywords = ["file", "devops", "git", "versioning", "mlops"] name = "xvc_ecs" crate-type = ["rlib"] -[profile.release] -debug = true - [dependencies] xvc-logging = { version = "0.6.9-alpha.1", path = "../logging" } diff --git a/file/Cargo.toml b/file/Cargo.toml index 5033604a9..eced27c0a 100644 --- a/file/Cargo.toml +++ b/file/Cargo.toml @@ -20,10 +20,6 @@ name = "xvc-file" test = true bench = true -[profile.release] -debug = true - - [dependencies] xvc-logging = { version = "0.6.9-alpha.1", path = "../logging" } xvc-config = { version = "0.6.9-alpha.1", path = "../config" } diff --git a/lib/Cargo.toml b/lib/Cargo.toml index 72e12fa7f..01ac7c63d 100644 --- a/lib/Cargo.toml +++ b/lib/Cargo.toml @@ -19,9 +19,6 @@ crate-type = ["rlib"] name = "xvc" path = "src/main.rs" -[profile.release] -debug = true - [dependencies] xvc-config = { version = "0.6.9-alpha.1", path = "../config" } xvc-core = { version = "0.6.9-alpha.1", path = "../core" } diff --git a/logging/Cargo.toml b/logging/Cargo.toml index 7ff7d4deb..64421dfcf 100644 --- a/logging/Cargo.toml +++ b/logging/Cargo.toml @@ -15,10 +15,6 @@ keywords = ["file", "devops", "git", "versioning", "mlops"] name = "xvc_logging" crate-type = ["rlib"] - -[profile.release] -debug = true - [dependencies] ## Logging and errors diff --git a/pipeline/Cargo.toml b/pipeline/Cargo.toml index 17f309663..7a5c5db3f 100644 --- a/pipeline/Cargo.toml +++ b/pipeline/Cargo.toml @@ -14,9 +14,6 @@ keywords = ["file", "devops", "git", "versioning", "mlops"] name = "xvc_pipeline" crate-type = ["rlib"] -[profile.release] -debug = true - [features] default = [] bundled-sqlite = ["rusqlite/bundled"] diff --git a/storage/Cargo.toml b/storage/Cargo.toml index 52ab7802d..47f90ad99 100644 --- a/storage/Cargo.toml +++ b/storage/Cargo.toml @@ -15,10 +15,6 @@ keywords = ["file", "devops", "git", "versioning", "mlops"] name = "xvc_storage" crate-type = ["rlib"] -[profile.release] -debug = true - - [dependencies] xvc-logging = { version = "0.6.9-alpha.1", path = "../logging" } xvc-config = { version = "0.6.9-alpha.1", path = "../config" } diff --git a/test_helper/Cargo.toml b/test_helper/Cargo.toml index c72722af4..8c1474c93 100644 --- a/test_helper/Cargo.toml +++ b/test_helper/Cargo.toml @@ -17,9 +17,6 @@ name = "xvc_test_helper" name = "xvc-test-helper" path = "src/main.rs" -[profile.release] -debug = true - # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html [dependencies] diff --git a/walker/Cargo.toml b/walker/Cargo.toml index 3905d687f..82b134707 100644 --- a/walker/Cargo.toml +++ b/walker/Cargo.toml @@ -15,9 +15,6 @@ keywords = ["file", "devops", "git", "versioning", "mlops"] name = "xvc_walker" crate-type = ["rlib"] -[profile.release] -debug = true - [dependencies] xvc-logging = { version = "0.6.9-alpha.1", path = "../logging" } globset = "^0.4" diff --git a/workflow_tests/Cargo.toml b/workflow_tests/Cargo.toml index 729eba794..e8bf92895 100644 --- a/workflow_tests/Cargo.toml +++ b/workflow_tests/Cargo.toml @@ -18,10 +18,6 @@ path = "src/main.rs" name = "xvc_tests" crate-type = ["rlib"] -[profile.release] -debug = true - - [dependencies] xvc = { version = "0.6.9-alpha.1", path = "../lib" } xvc-config = { version = "0.6.9-alpha.1", path = "../config" } From 29fc0139ffe0589b1638e06547e571456339df29 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sun, 28 Jul 2024 19:56:03 +0300 Subject: [PATCH 099/257] add compile without default features --- book/src/SUMMARY.md | 146 ++-- .../intro/compile-without-default-features.md | 714 ++++++++++++++++++ 2 files changed, 790 insertions(+), 70 deletions(-) create mode 100644 book/src/intro/compile-without-default-features.md diff --git a/book/src/SUMMARY.md b/book/src/SUMMARY.md index d50b41715..fe7e88517 100644 --- a/book/src/SUMMARY.md +++ b/book/src/SUMMARY.md @@ -2,8 +2,10 @@ - [Introduction](./intro/index.md) - [Installation](./intro/install.md) + - [Compile without default features](./intro/compile-without-default-features.md) - [Configuration](./intro/configuration.md) - [Get Started](./start/index.md) + - [Xvc for Everyone](./start/everyone.md) - [Xvc for Data Versioning](./start/data.md) - [Xvc for Machine Learning](./start/ml.md) @@ -12,85 +14,89 @@ - [Benchmark between Xvc and DVC](./how-to/benchmark-versus-dvc.md) - [User's Guide](./how-to/index.md) + - [How to Compile Xvc](./how-to/compile.md) - [Xvc with Git Branches](./how-to/git-branches.md) - [Turn off Git Integration](./how-to/turn-off-git-automation.md) - [Create a Data Pipeline](./how-to/create-a-data-pipeline.md) - [Command Reference](./ref/xvc.md) - - [`xvc init`](./ref/xvc-init.md) - - [File Management (`xvc file`)](./ref/xvc-file.md) - - [`xvc file track`](./ref/xvc-file-track.md) - - [`xvc file untrack`](./ref/xvc-file-untrack.md) - - [`xvc file list`](./ref/xvc-file-list.md) - - [`xvc file hash`](./ref/xvc-file-hash.md) - - [`xvc file recheck`](./ref/xvc-file-recheck.md) - - [`xvc file carry-in`](./ref/xvc-file-carry-in.md) - - [`xvc file send`](./ref/xvc-file-send.md) - - [`xvc file bring`](./ref/xvc-file-bring.md) - - [`xvc file move`](./ref/xvc-file-move.md) - - [`xvc file copy`](./ref/xvc-file-copy.md) - - [`xvc file remove`](./ref/xvc-file-remove.md) - - [Data-Model Pipelines (`xvc pipeline`)](./ref/xvc-pipeline.md) - - [`xvc pipeline new`](./ref/xvc-pipeline-new.md) - - [`xvc pipeline list`](./ref/xvc-pipeline-list.md) - - [`xvc pipeline step`](./ref/xvc-pipeline-step.md) - - [`new`](./ref/xvc-pipeline-step-new.md) - - [`list`](./ref/xvc-pipeline-step-list.md) - - [`dependency`](./ref/xvc-pipeline-step-dependency.md) - - [`output`](./ref/xvc-pipeline-step-output.md) - - [`show`](./ref/xvc-pipeline-step-show.md) - - [`update`](./ref/xvc-pipeline-step-update.md) - - [`remove`](./ref/xvc-pipeline-step-remove.md) - - [`xvc pipeline run`](./ref/xvc-pipeline-run.md) - - [`xvc pipeline delete`](./ref/xvc-pipeline-delete.md) - - [`xvc pipeline export`](./ref/xvc-pipeline-export.md) - - [`xvc pipeline import`](./ref/xvc-pipeline-import.md) - - [`xvc pipeline update`](./ref/xvc-pipeline-update.md) - - [`xvc pipeline dag`](./ref/xvc-pipeline-dag.md) - - [(Remote) Storages (`xvc storage`)](./ref/xvc-storage.md) - - [`xvc storage list`](./ref/xvc-storage-list.md) - - [`xvc storage remove`](./ref/xvc-storage-remove.md) - - [`xvc storage new`](./ref/xvc-storage-new.md) - - [`local`](./ref/xvc-storage-new-local.md) - - [`generic`](./ref/xvc-storage-new-generic.md) - - [`rsync`](./ref/xvc-storage-new-rsync.md) - - [`s3`](./ref/xvc-storage-new-s3.md) - - [`gcs`](./ref/xvc-storage-new-gcs.md) - - [`minio`](./ref/xvc-storage-new-minio.md) - - [`r2`](./ref/xvc-storage-new-r2.md) - - [`wasabi`](./ref/xvc-storage-new-wasabi.md) - - [`digital-ocean`](./ref/xvc-storage-new-digital-ocean.md) - - [Server (`xvc server`)](./ref/xvc-server.md) - - [`xvc server start`](./ref/xvc-server-start.md) - - [Utilities](./ref/utilities.md) - - [`xvc root`](./ref/xvc-root.md) - - [`xvc check-ignore`](./ref/xvc-check-ignore.md) - - [`xvc aliases`](./ref/xvc-aliases.md) - - [`xvc deinit`](./ref/xvc-deinit.md) + + - [`xvc init`](./ref/xvc-init.md) + - [File Management (`xvc file`)](./ref/xvc-file.md) + - [`xvc file track`](./ref/xvc-file-track.md) + - [`xvc file untrack`](./ref/xvc-file-untrack.md) + - [`xvc file list`](./ref/xvc-file-list.md) + - [`xvc file hash`](./ref/xvc-file-hash.md) + - [`xvc file recheck`](./ref/xvc-file-recheck.md) + - [`xvc file carry-in`](./ref/xvc-file-carry-in.md) + - [`xvc file send`](./ref/xvc-file-send.md) + - [`xvc file bring`](./ref/xvc-file-bring.md) + - [`xvc file move`](./ref/xvc-file-move.md) + - [`xvc file copy`](./ref/xvc-file-copy.md) + - [`xvc file remove`](./ref/xvc-file-remove.md) + - [Data-Model Pipelines (`xvc pipeline`)](./ref/xvc-pipeline.md) + - [`xvc pipeline new`](./ref/xvc-pipeline-new.md) + - [`xvc pipeline list`](./ref/xvc-pipeline-list.md) + - [`xvc pipeline step`](./ref/xvc-pipeline-step.md) + - [`new`](./ref/xvc-pipeline-step-new.md) + - [`list`](./ref/xvc-pipeline-step-list.md) + - [`dependency`](./ref/xvc-pipeline-step-dependency.md) + - [`output`](./ref/xvc-pipeline-step-output.md) + - [`show`](./ref/xvc-pipeline-step-show.md) + - [`update`](./ref/xvc-pipeline-step-update.md) + - [`remove`](./ref/xvc-pipeline-step-remove.md) + - [`xvc pipeline run`](./ref/xvc-pipeline-run.md) + - [`xvc pipeline delete`](./ref/xvc-pipeline-delete.md) + - [`xvc pipeline export`](./ref/xvc-pipeline-export.md) + - [`xvc pipeline import`](./ref/xvc-pipeline-import.md) + - [`xvc pipeline update`](./ref/xvc-pipeline-update.md) + - [`xvc pipeline dag`](./ref/xvc-pipeline-dag.md) + - [(Remote) Storages (`xvc storage`)](./ref/xvc-storage.md) + - [`xvc storage list`](./ref/xvc-storage-list.md) + - [`xvc storage remove`](./ref/xvc-storage-remove.md) + - [`xvc storage new`](./ref/xvc-storage-new.md) + - [`local`](./ref/xvc-storage-new-local.md) + - [`generic`](./ref/xvc-storage-new-generic.md) + - [`rsync`](./ref/xvc-storage-new-rsync.md) + - [`s3`](./ref/xvc-storage-new-s3.md) + - [`gcs`](./ref/xvc-storage-new-gcs.md) + - [`minio`](./ref/xvc-storage-new-minio.md) + - [`r2`](./ref/xvc-storage-new-r2.md) + - [`wasabi`](./ref/xvc-storage-new-wasabi.md) + - [`digital-ocean`](./ref/xvc-storage-new-digital-ocean.md) + - [Server (`xvc server`)](./ref/xvc-server.md) + - [`xvc server start`](./ref/xvc-server-start.md) + - [Utilities](./ref/utilities.md) + - [`xvc root`](./ref/xvc-root.md) + - [`xvc check-ignore`](./ref/xvc-check-ignore.md) + - [`xvc aliases`](./ref/xvc-aliases.md) + - [`xvc deinit`](./ref/xvc-deinit.md) - [Rust API](./api/index.md) - - [`xvc`](./api/xvc.md) - - [`xvc-config`](./api/xvc-config.md) - - [`xvc-core`](./api/xvc-core.md) - - [`xvc-ecs`](./api/xvc-ecs.md) - - [`xvc-file`](./api/xvc-file.md) - - [`xvc-logging`](./api/xvc-logging.md) - - [`xvc-pipeline`](./api/xvc-pipeline.md) - - [`xvc-storage`](./api/xvc-storage.md) - - [`xvc-walker`](./api/xvc-walker.md) + + - [`xvc`](./api/xvc.md) + - [`xvc-config`](./api/xvc-config.md) + - [`xvc-core`](./api/xvc-core.md) + - [`xvc-ecs`](./api/xvc-ecs.md) + - [`xvc-file`](./api/xvc-file.md) + - [`xvc-logging`](./api/xvc-logging.md) + - [`xvc-pipeline`](./api/xvc-pipeline.md) + - [`xvc-storage`](./api/xvc-storage.md) + - [`xvc-walker`](./api/xvc-walker.md) - [Architecture](./arch/index.md) - - [Goals](./arch/goals.md) - - [Cache](./arch/cache.md) - - [Xvc ECS](./arch/ecs.md) - - [Pipeline State Machine](./arch/pipeline-state-machine.md) - - [Comparisons](./arch/delta.md) - - [Storages](./arch/storages.md) - - [Git and Xvc](./arch/git-and-xvc.md) - - [Concepts](./concepts/index.md) - - [Digest](./concepts/digest.md) - - [Associated Digest](./concepts/associated-digest.md) + + - [Goals](./arch/goals.md) + - [Cache](./arch/cache.md) + - [Xvc ECS](./arch/ecs.md) + - [Pipeline State Machine](./arch/pipeline-state-machine.md) + - [Comparisons](./arch/delta.md) + - [Storages](./arch/storages.md) + - [Git and Xvc](./arch/git-and-xvc.md) + - [Concepts](./concepts/index.md) + - [Digest](./concepts/digest.md) + - [Associated Digest](./concepts/associated-digest.md) - [Development](./development/index.md) - - [Conventions](./development/conventions.md) + - [Conventions](./development/conventions.md) diff --git a/book/src/intro/compile-without-default-features.md b/book/src/intro/compile-without-default-features.md new file mode 100644 index 000000000..0406fa742 --- /dev/null +++ b/book/src/intro/compile-without-default-features.md @@ -0,0 +1,714 @@ +# Compiling Xvc without default features + +You may want to customize the [feature set][features] when you want a smaller binary size. Not everyone needs all storage options and turning off them may result in smaller binary sizes. + +When you turn off all remote storage features, async runtime (`tokio`) is also excluded from binary. + +````bash +cargo build --no-default-features --release +warning: output filename collision. +The bin target `xvc` in package `xvc-workflow-tests v0.6.9-alpha.1 (/Users/iex/github.com/iesahin/xvc/workflow_tests)` has the same output filename as the bin target `xvc` in package `xvc v0.6.9-alpha.1 (/Users/iex/github.com/iesahin/xvc/lib)`. +Colliding filename is: /Users/iex/github.com/iesahin/xvc/target/release/xvc +The targets should have unique names. +Consider changing their names to be unique or compiling them separately. +This may become a hard error in the future; see . +warning: output filename collision. +The bin target `xvc` in package `xvc-workflow-tests v0.6.9-alpha.1 (/Users/iex/github.com/iesahin/xvc/workflow_tests)` has the same output filename as the bin target `xvc` in package `xvc v0.6.9-alpha.1 (/Users/iex/github.com/iesahin/xvc/lib)`. +Colliding filename is: /Users/iex/github.com/iesahin/xvc/target/release/xvc.dSYM +The targets should have unique names. +Consider changing their names to be unique or compiling them separately. +This may become a hard error in the future; see . + Compiling proc-macro2 v1.0.86 + Compiling unicode-ident v1.0.12 + Compiling libc v0.2.155 + Compiling cfg-if v1.0.0 + Compiling autocfg v1.3.0 + Compiling version_check v0.9.4 + Compiling serde v1.0.204 + Compiling memchr v2.7.4 + Compiling once_cell v1.19.0 + Compiling crossbeam-utils v0.8.20 + Compiling bitflags v2.6.0 + Compiling zerocopy v0.7.35 + Compiling allocator-api2 v0.2.18 + Compiling ahash v0.8.11 + Compiling bytes v1.6.1 + Compiling log v0.4.22 + Compiling itoa v1.0.11 + Compiling pin-project-lite v0.2.14 + Compiling core-foundation-sys v0.8.6 + Compiling aho-corasick v1.1.3 + Compiling rustix v0.38.34 + Compiling regex-syntax v0.8.4 + Compiling futures-core v0.3.30 + Compiling typenum v1.17.0 + Compiling equivalent v1.0.1 + Compiling ident_case v1.0.1 + Compiling fnv v1.0.7 + Compiling num-traits v0.2.19 + Compiling slab v0.4.9 + Compiling generic-array v0.14.7 + Compiling futures-sink v0.3.30 + Compiling errno v0.3.9 + Compiling regex-automata v0.4.7 + Compiling mio v0.8.11 + Compiling quote v1.0.36 + Compiling num_cpus v1.16.0 + Compiling syn v2.0.72 + Compiling socket2 v0.5.7 + Compiling crossbeam-epoch v0.9.18 + Compiling crossbeam-channel v0.5.13 + Compiling tokio v1.38.1 + Compiling hashbrown v0.14.5 + Compiling crossbeam-deque v0.8.5 + Compiling core-foundation v0.9.4 + Compiling indexmap v2.2.6 + Compiling crunchy v0.2.2 + Compiling thiserror v1.0.63 + Compiling fastrand v2.1.0 + Compiling ryu v1.0.18 + Compiling tempfile v3.10.1 + Compiling http v0.2.12 + Compiling security-framework-sys v2.11.1 + Compiling is-terminal v0.4.12 + Compiling futures-channel v0.3.30 + Compiling lock_api v0.4.12 + Compiling native-tls v0.2.12 + Compiling either v1.13.0 + Compiling serde_derive v1.0.204 + Compiling thiserror-impl v1.0.63 + Compiling futures-macro v0.3.30 + Compiling lazy_static v1.5.0 + Compiling tiny-keccak v2.0.2 + Compiling tinyvec_macros v0.1.1 + Compiling anstyle v1.0.7 + Compiling smallvec v1.13.2 + Compiling futures-io v0.3.30 + Compiling rayon-core v1.12.1 + Compiling futures-task v0.3.30 + Compiling pin-utils v0.1.0 + Compiling parking_lot_core v0.9.10 + Compiling futures-util v0.3.30 + Compiling tinyvec v1.8.0 + Compiling colored v1.9.4 + Compiling security-framework v2.11.1 + Compiling bstr v1.9.1 + Compiling getrandom v0.2.15 + Compiling percent-encoding v2.3.1 + Compiling anyhow v1.0.86 + Compiling scopeguard v1.2.0 + Compiling serde_json v1.0.120 + Compiling heck v0.5.0 + Compiling same-file v1.0.6 + Compiling walkdir v2.5.0 + Compiling form_urlencoded v1.2.1 + Compiling rand_core v0.6.4 + Compiling fern v0.6.2 + Compiling unicode-normalization v0.1.23 + Compiling block-buffer v0.10.4 + Compiling crypto-common v0.1.6 + Compiling regex v1.10.5 + Compiling tracing-core v0.1.32 + Compiling subtle v2.6.1 + Compiling httparse v1.9.4 + Compiling utf8parse v0.2.2 + Compiling ppv-lite86 v0.2.17 + Compiling unicode-bidi v0.3.15 + Compiling const-random-macro v0.1.16 + Compiling rand_chacha v0.3.1 + Compiling anstyle-parse v0.2.4 + Compiling tracing v0.1.40 + Compiling digest v0.10.7 + Compiling xvc-logging v0.6.9-alpha.1 (/Users/iex/github.com/iesahin/xvc/logging) + Compiling idna v0.5.0 + Compiling rayon v1.10.0 + Compiling globset v0.4.14 + Compiling tokio-util v0.7.11 + Compiling is_terminal_polyfill v1.70.0 + Compiling anstyle-query v1.1.0 + Compiling colorchoice v1.0.1 + Compiling rustversion v1.0.17 + Compiling strsim v0.11.1 + Compiling paste v1.0.15 + Compiling try-lock v0.2.5 + Compiling syn v1.0.109 + Compiling want v0.3.1 + Compiling url v2.5.2 + Compiling h2 v0.3.26 + Compiling darling_core v0.20.10 + Compiling anstream v0.6.14 + Compiling const-random v0.1.18 + Compiling rand v0.8.5 + Compiling http-body v0.4.6 + Compiling crossbeam-queue v0.3.11 + Compiling cpufeatures v0.2.12 + Compiling smartstring v1.0.1 + Compiling tower-service v0.3.2 + Compiling httpdate v1.0.3 + Compiling clap_lex v0.7.1 + Compiling system-configuration-sys v0.5.0 + Compiling adler v1.0.2 + Compiling miniz_oxide v0.7.4 + Compiling darling_macro v0.20.10 + Compiling hyper v0.14.30 + Compiling clap_builder v4.5.9 + Compiling crossbeam v0.8.4 + Compiling serde_spanned v0.6.6 + Compiling toml_datetime v0.6.6 + Compiling dlv-list v0.5.2 + Compiling tokio-native-tls v0.3.1 + Compiling clap_derive v4.5.8 + Compiling fsevent-sys v4.1.0 + Compiling filetime v0.2.23 + Compiling crc32fast v1.4.2 + Compiling time-core v0.1.2 + Compiling powerfmt v0.2.0 + Compiling static_assertions v1.1.0 + Compiling byteorder v1.5.0 + Compiling cc v1.1.6 + Compiling hashbrown v0.13.2 + Compiling winnow v0.6.14 + Compiling base64 v0.21.7 + Compiling num-conv v0.1.0 + Compiling time-macros v0.2.18 + Compiling ordered-multimap v0.6.0 + Compiling hyper-tls v0.5.0 + Compiling rmp v0.8.14 + Compiling blake3 v1.5.3 + Compiling deranged v0.3.11 + Compiling clap v4.5.9 + Compiling flate2 v1.0.30 + Compiling toml_edit v0.22.16 + Compiling notify v6.1.1 + Compiling strum_macros v0.26.4 + Compiling darling v0.20.10 + Compiling dashmap v6.0.1 + Compiling itertools v0.13.0 + Compiling dirs-sys-next v0.1.2 + Compiling unsafe-libyaml v0.2.11 + Compiling convert_case v0.4.0 + Compiling rxml_validation v0.9.1 + Compiling strsim v0.9.3 + Compiling home v0.5.9 + Compiling strum v0.26.3 + Compiling bitflags v1.3.2 + Compiling toml v0.8.15 + Compiling time v0.3.36 + Compiling system-configuration v0.5.1 + Compiling darling_core v0.10.2 + Compiling xvc-walker v0.6.9-alpha.1 (/Users/iex/github.com/iesahin/xvc/walker) + Compiling rxml v0.9.1 + Compiling serde_yaml v0.9.34+deprecated + Compiling cached_proc_macro v0.22.0 + Compiling directories-next v2.0.0 + Compiling rmp-serde v1.3.0 + Compiling async-compression v0.4.12 + Compiling rust-ini v0.19.0 + Compiling rustls-pemfile v1.0.4 + Compiling keccak v0.1.5 + Compiling sha2 v0.10.8 + Compiling attohttpc v0.26.1 + Compiling quick-xml v0.30.0 + Compiling serde_urlencoded v0.7.1 + Compiling hex v0.4.3 + Compiling futures-executor v0.3.30 + Compiling path-dedot v3.1.1 + Compiling encoding_rs v0.8.34 + Compiling web-time v1.1.0 + Compiling arrayvec v0.7.4 + Compiling constant_time_eq v0.3.0 + Compiling ipnet v2.9.0 + Compiling sync_wrapper v0.1.2 + Compiling mime v0.3.17 + Compiling arrayref v0.3.8 + Compiling cached_proc_macro_types v0.1.1 + Compiling cached v0.53.0 + Compiling reqwest v0.11.27 + Compiling aws-creds v0.36.0 + Compiling path-absolutize v3.1.1 + Compiling futures v0.3.30 + Compiling sha3 v0.10.8 + Compiling xvc-ecs v0.6.9-alpha.1 (/Users/iex/github.com/iesahin/xvc/ecs) + Compiling xvc-config v0.6.9-alpha.1 (/Users/iex/github.com/iesahin/xvc/config) + Compiling darling_macro v0.10.2 + Compiling minidom v0.15.2 + Compiling derive_more v0.99.18 + Compiling jwalk v0.8.1 + Compiling uuid v1.10.0 + Compiling relative-path v1.9.3 + Compiling blake2 v0.10.6 + Compiling hmac v0.12.1 + Compiling aws-region v0.25.4 + Compiling maybe-async v0.2.10 + Compiling async-trait v0.1.81 + Compiling tokio-stream v0.1.15 + Compiling pkg-config v0.3.30 + Compiling md5 v0.7.0 + Compiling vcpkg v0.2.15 + Compiling glob v0.3.1 + Compiling seahash v4.1.0 + Compiling darling v0.10.2 + Compiling which v6.0.1 + Compiling rust-s3 v0.34.0 + Compiling subprocess v0.2.9 + Compiling libsqlite3-sys v0.30.0 + Compiling xvc-core v0.6.9-alpha.1 (/Users/iex/github.com/iesahin/xvc/core) + Compiling iana-time-zone v0.1.60 + Compiling predicates-core v1.0.6 + Compiling doc-comment v0.3.3 + Compiling derive_builder v0.9.0 + Compiling humantime v2.1.0 + Compiling derive_builder_core v0.9.0 + Compiling chrono v0.4.38 + Compiling parking_lot v0.12.3 + Compiling float-cmp v0.9.0 + Compiling reflink v0.1.3 + Compiling parse-size v1.0.0 + Compiling normalize-line-endings v0.3.0 + Compiling difflib v0.4.0 + Compiling termtree v0.4.1 + Compiling predicates-tree v1.0.9 +warning: use of deprecated macro `EnumVariantNames`: please use `#[derive(VariantNames)]` instead + --> core/src/types/hashalgorithm.rs:27:5 + | +27 | EnumVariantNames, + | ^^^^^^^^^^^^^^^^ + | + = note: `#[warn(deprecated)]` on by default + + Compiling predicates v3.1.0 + Compiling crossterm v0.27.0 + Compiling ignore v0.4.22 + Compiling hashlink v0.9.1 + Compiling fallible-iterator v0.3.0 + Compiling fixedbitset v0.4.2 + Compiling unicode-width v0.1.13 + Compiling assert_cmd v2.0.14 + Compiling fallible-streaming-iterator v0.1.9 + Compiling globwalk v0.9.1 + Compiling comfy-table v7.1.1 + Compiling petgraph v0.6.5 + Compiling tabbycat v0.1.3 + Compiling sad_machine v1.0.0 + Compiling rusqlite v0.32.0 + Compiling git-version-macro v0.3.9 + Compiling wait-timeout v0.2.0 + Compiling git-version v0.3.9 + Compiling assert_fs v1.1.1 + Compiling xvc-test-helper v0.6.9-alpha.1 (/Users/iex/github.com/iesahin/xvc/test_helper) + Compiling xvc-storage v0.6.9-alpha.1 (/Users/iex/github.com/iesahin/xvc/storage) +warning: unused import: `panic` + --> storage/src/storage/mod.rs:35:26 + | +35 | use xvc_logging::{error, panic, watch, XvcOutputSender}; + | ^^^^^ + | + = note: `#[warn(unused_imports)]` on by default + +warning: unused variable: `xvc_root` + --> storage/src/storage/async_common.rs:273:50 + | +273 | fn init(&mut self, output: &XvcOutputSender, xvc_root: &XvcRoot) -> Result + | ^^^^^^^^ help: if this is intentional, prefix it with an underscore: `_xvc_root` + | + = note: `#[warn(unused_variables)]` on by default + +warning: unused variable: `xvc_root` + --> storage/src/storage/async_common.rs:313:9 + | +313 | xvc_root: &xvc_core::XvcRoot, + | ^^^^^^^^ help: if this is intentional, prefix it with an underscore: `_xvc_root` + +warning: unused variable: `xvc_root` + --> storage/src/storage/async_common.rs:327:9 + | +327 | xvc_root: &xvc_core::XvcRoot, + | ^^^^^^^^ help: if this is intentional, prefix it with an underscore: `_xvc_root` + +warning: unused variable: `xvc_root` + --> storage/src/storage/async_common.rs:340:9 + | +340 | xvc_root: &xvc_core::XvcRoot, + | ^^^^^^^^ help: if this is intentional, prefix it with an underscore: `_xvc_root` + +warning: unused variable: `output` + --> storage/src/storage/generic.rs:448:9 + | +448 | output: &XvcOutputSender, + | ^^^^^^ help: if this is intentional, prefix it with an underscore: `_output` + +warning: unused variable: `xvc_root` + --> storage/src/storage/generic.rs:449:9 + | +449 | xvc_root: &XvcRoot, + | ^^^^^^^^ help: if this is intentional, prefix it with an underscore: `_xvc_root` + +warning: unused variable: `path` + --> storage/src/storage/generic.rs:450:9 + | +450 | path: &XvcCachePath, + | ^^^^ help: if this is intentional, prefix it with an underscore: `_path` + +warning: unused variable: `period` + --> storage/src/storage/generic.rs:451:9 + | +451 | period: std::time::Duration, + | ^^^^^^ help: if this is intentional, prefix it with an underscore: `_period` + +warning: unused variable: `output` + --> storage/src/storage/rsync.rs:475:9 + | +475 | output: &XvcOutputSender, + | ^^^^^^ help: if this is intentional, prefix it with an underscore: `_output` + +warning: unused variable: `xvc_root` + --> storage/src/storage/rsync.rs:476:9 + | +476 | xvc_root: &XvcRoot, + | ^^^^^^^^ help: if this is intentional, prefix it with an underscore: `_xvc_root` + +warning: unused variable: `path` + --> storage/src/storage/rsync.rs:477:9 + | +477 | path: &XvcCachePath, + | ^^^^ help: if this is intentional, prefix it with an underscore: `_path` + +warning: unused variable: `period` + --> storage/src/storage/rsync.rs:478:9 + | +478 | period: std::time::Duration, + | ^^^^^^ help: if this is intentional, prefix it with an underscore: `_period` + +warning: missing documentation for a module + --> storage/src/storage/mod.rs:3:1 + | +3 | pub mod async_common; + | ^^^^^^^^^^^^^^^^^^^^ + | +note: the lint level is defined here + --> storage/src/lib.rs:6:9 + | +6 | #![warn(missing_docs)] + | ^^^^^^^^^^^^ + +warning: missing documentation for a module + --> storage/src/storage/mod.rs:4:1 + | +4 | pub mod common; + | ^^^^^^^^^^^^^^ + +warning: missing documentation for a trait + --> storage/src/storage/async_common.rs:33:1 + | +33 | pub trait XvcS3StorageOperations { + | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +warning: missing documentation for a method + --> storage/src/storage/async_common.rs:34:5 + | +34 | fn storage_prefix(&self) -> String; + | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +warning: missing documentation for a method + --> storage/src/storage/async_common.rs:35:5 + | +35 | fn guid(&self) -> &XvcStorageGuid; + | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +warning: missing documentation for a method + --> storage/src/storage/async_common.rs:36:5 + | +36 | fn get_bucket(&self) -> Result; + | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +warning: missing documentation for a method + --> storage/src/storage/async_common.rs:37:5 + | +37 | fn credentials(&self) -> Result; + | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +warning: missing documentation for a method + --> storage/src/storage/async_common.rs:38:5 + | +38 | fn bucket_name(&self) -> String; + | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +warning: missing documentation for a method + --> storage/src/storage/async_common.rs:39:5 + | +39 | fn build_storage_path(&self, cache_path: &XvcCachePath) -> XvcStoragePath { + | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +warning: missing documentation for a method + --> storage/src/storage/async_common.rs:48:5 + | +48 | fn region(&self) -> String; + | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +warning: missing documentation for a method + --> storage/src/storage/async_common.rs:49:5 + | +49 | async fn write_storage_guid(&self) -> Result<()> { + | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +warning: use of `async fn` in public traits is discouraged as auto trait bounds cannot be specified + --> storage/src/storage/async_common.rs:49:5 + | +49 | async fn write_storage_guid(&self) -> Result<()> { + | ^^^^^ + | + = note: you can suppress this lint if you plan to use the trait only in your own code, or do not care about auto traits like `Send` on the `Future` + = note: `#[warn(async_fn_in_trait)]` on by default +help: you can alternatively desugar to a normal `fn` that returns `impl Future` and add any desired bounds such as `Send`, but these cannot be relaxed without a breaking API change + | +49 ~ fn write_storage_guid(&self) -> impl std::future::Future> + Send {async { +50 | let guid_str = self.guid().to_string(); + ... +63 | } +64 ~ } } + | + +warning: missing documentation for a method + --> storage/src/storage/async_common.rs:66:5 + | +66 | async fn a_init(&mut self, output_snd: &XvcOutputSender) -> Result { + | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +warning: use of `async fn` in public traits is discouraged as auto trait bounds cannot be specified + --> storage/src/storage/async_common.rs:66:5 + | +66 | async fn a_init(&mut self, output_snd: &XvcOutputSender) -> Result { + | ^^^^^ + | + = note: you can suppress this lint if you plan to use the trait only in your own code, or do not care about auto traits like `Send` on the `Future` +help: you can alternatively desugar to a normal `fn` that returns `impl Future` and add any desired bounds such as `Send`, but these cannot be relaxed without a breaking API change + | +66 ~ fn a_init(&mut self, output_snd: &XvcOutputSender) -> impl std::future::Future> + Send {async { +67 | let res_response = self.write_storage_guid().await; + ... +77 | } +78 ~ } } + | + +warning: missing documentation for a method + --> storage/src/storage/async_common.rs:80:5 + | +80 | / async fn a_list( +81 | | &self, +82 | | output: &XvcOutputSender, +83 | | xvc_root: &xvc_core::XvcRoot, +84 | | ) -> Result { + | |____________________________________^ + +warning: use of `async fn` in public traits is discouraged as auto trait bounds cannot be specified + --> storage/src/storage/async_common.rs:80:5 + | +80 | async fn a_list( + | ^^^^^ + | + = note: you can suppress this lint if you plan to use the trait only in your own code, or do not care about auto traits like `Send` on the `Future` +help: you can alternatively desugar to a normal `fn` that returns `impl Future` and add any desired bounds such as `Send`, but these cannot be relaxed without a breaking API change + | +80 ~ fn a_list( +81 | &self, +82 | output: &XvcOutputSender, +83 | xvc_root: &xvc_core::XvcRoot, +84 ~ ) -> impl std::future::Future> + Send {async { +85 | let credentials = self.credentials()?; + ... +130| } +131~ } } + | + +warning: missing documentation for a method + --> storage/src/storage/async_common.rs:133:5 + | +133 | / async fn a_send( +134 | | &self, +135 | | output_snd: &XvcOutputSender, +136 | | xvc_root: &xvc_core::XvcRoot, +137 | | paths: &[xvc_core::XvcCachePath], +138 | | _force: bool, +139 | | ) -> crate::Result { + | |__________________________________________________^ + +warning: use of `async fn` in public traits is discouraged as auto trait bounds cannot be specified + --> storage/src/storage/async_common.rs:133:5 + | +133 | async fn a_send( + | ^^^^^ + | + = note: you can suppress this lint if you plan to use the trait only in your own code, or do not care about auto traits like `Send` on the `Future` +help: you can alternatively desugar to a normal `fn` that returns `impl Future` and add any desired bounds such as `Send`, but these cannot be relaxed without a breaking API change + | +133 ~ fn a_send( +134 | &self, + ... +138 | _force: bool, +139 ~ ) -> impl std::future::Future> + Send {async { +140 | let mut copied_paths = Vec::::new(); + ... +172 | }) +173 ~ } } + | + +warning: missing documentation for a method + --> storage/src/storage/async_common.rs:175:5 + | +175 | / async fn a_receive( +176 | | &self, +177 | | output_snd: &XvcOutputSender, +178 | | paths: &[xvc_core::XvcCachePath], +179 | | _force: bool, +180 | | ) -> Result<(XvcStorageTempDir, XvcStorageReceiveEvent)> { + | |____________________________________________________________^ + +warning: use of `async fn` in public traits is discouraged as auto trait bounds cannot be specified + --> storage/src/storage/async_common.rs:175:5 + | +175 | async fn a_receive( + | ^^^^^ + | + = note: you can suppress this lint if you plan to use the trait only in your own code, or do not care about auto traits like `Send` on the `Future` +help: you can alternatively desugar to a normal `fn` that returns `impl Future` and add any desired bounds such as `Send`, but these cannot be relaxed without a breaking API change + | +175 ~ fn a_receive( +176 | &self, + ... +179 | _force: bool, +180 ~ ) -> impl std::future::Future> + Send {async { +181 | let mut copied_paths = Vec::::new(); + ... +217 | )) +218 ~ } } + | + +warning: missing documentation for a method + --> storage/src/storage/async_common.rs:220:5 + | +220 | / async fn a_delete( +221 | | &self, +222 | | output: &XvcOutputSender, +223 | | paths: &[XvcCachePath], +224 | | ) -> Result { + | |______________________________________^ + +warning: use of `async fn` in public traits is discouraged as auto trait bounds cannot be specified + --> storage/src/storage/async_common.rs:220:5 + | +220 | async fn a_delete( + | ^^^^^ + | + = note: you can suppress this lint if you plan to use the trait only in your own code, or do not care about auto traits like `Send` on the `Future` +help: you can alternatively desugar to a normal `fn` that returns `impl Future` and add any desired bounds such as `Send`, but these cannot be relaxed without a breaking API change + | +220 ~ fn a_delete( +221 | &self, +222 | output: &XvcOutputSender, +223 | paths: &[XvcCachePath], +224 ~ ) -> impl std::future::Future> + Send {async { +225 | let mut deleted_paths = Vec::::new(); + ... +240 | }) +241 ~ } } + | + +warning: missing documentation for a method + --> storage/src/storage/async_common.rs:243:5 + | +243 | / async fn a_share( +244 | | &self, +245 | | output: &XvcOutputSender, +246 | | path: &XvcCachePath, +247 | | duration: std::time::Duration, +248 | | ) -> Result { + | |_____________________________________________^ + +warning: use of `async fn` in public traits is discouraged as auto trait bounds cannot be specified + --> storage/src/storage/async_common.rs:243:5 + | +243 | async fn a_share( + | ^^^^^ + | + = note: you can suppress this lint if you plan to use the trait only in your own code, or do not care about auto traits like `Send` on the `Future` +help: you can alternatively desugar to a normal `fn` that returns `impl Future` and add any desired bounds such as `Send`, but these cannot be relaxed without a breaking API change + | +243 ~ fn a_share( +244 | &self, + ... +247 | duration: std::time::Duration, +248 ~ ) -> impl std::future::Future> + Send {async { +249 | let bucket = self.get_bucket()?; + ... +268 | }) +269 ~ } } + | + + Compiling xvc-file v0.6.9-alpha.1 (/Users/iex/github.com/iesahin/xvc/file) +warning: unused import: `HStore` + --> file/src/share/mod.rs:7:15 + | +7 | use xvc_ecs::{HStore, XvcStore}; + | ^^^^^^ + | + = note: `#[warn(unused_imports)]` on by default + +warning: missing documentation for a function + --> file/src/share/mod.rs:29:1 + | +29 | pub fn cmd_share(output_snd: &XvcOutputSender, xvc_root: &XvcRoot, opts: ShareCLI) -> Result<()> { + | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + | +note: the lint level is defined here + --> file/src/lib.rs:7:9 + | +7 | #![warn(missing_docs)] + | ^^^^^^^^^^^^ + +warning: `xvc-core` (lib) generated 1 warning + Compiling xvc-pipeline v0.6.9-alpha.1 (/Users/iex/github.com/iesahin/xvc/pipeline) + Compiling xvc v0.6.9-alpha.1 (/Users/iex/github.com/iesahin/xvc/lib) +warning: missing documentation for a function + --> lib/src/cli/mod.rs:212:1 + | +212 | pub fn dispatch_with_root(cli_opts: cli::XvcCLI, xvc_root_opt: XvcRootOpt) -> Result { + | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + | +note: the lint level is defined here + --> lib/src/lib.rs:1:9 + | +1 | #![warn(missing_docs)] + | ^^^^^^^^^^^^ + + Compiling xvc-workflow-tests v0.6.9-alpha.1 (/Users/iex/github.com/iesahin/xvc/workflow_tests) +warning: `xvc-storage` (lib) generated 37 warnings (run `cargo fix --lib -p xvc-storage` to apply 1 suggestion) +warning: `xvc-file` (lib) generated 2 warnings (run `cargo fix --lib -p xvc-file` to apply 1 suggestion) +warning: `xvc` (lib) generated 1 warning + Finished `release` profile [optimized] target(s) in 41.49s +``` + +## Compiling Xvc without Reflink support + +[reflink] crate may cause compilation errors on platforms where it's not supported. + +Xvc adds a `reflink` feature flag that's turned on by default. When reflink +causes errors, you can turn off default features and select only those you'll +use. + +```bash +cargo build --no-default-features --features "reflink" --release +[..] + Finished `release` profile [optimized + debuginfo] target(s) in 56.40s +```` + +Note that when you supply `--no-default-features`, all other default features +like `s3` etc are also turned off. You'll have to specify which [features] you +want in the features list. Otherwise Xvc cannot connect to your storages. + +``` +cargo build --no-default-features --features "reflink,s3" --release +[..] + Finished `release` profile [optimized + debuginfo] target(s) in 56.40s +``` + +[features]: https://docs.rs/crate/xvc/latest/features From 3d8b1abc6562d49bbaba0bfbe3504477f38d80ed Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sun, 28 Jul 2024 19:57:50 +0300 Subject: [PATCH 100/257] update --- .gitignore | 1 + lib/Cargo.toml | 7 +++++++ .../chinese_mnist.zip | Bin .../chinese_mnist.zip | Bin .../image_to_numpy_array.py | 0 .../requirements.txt | 0 .../how-to-create-a-data-pipeline.in/train.py | 0 .../templates/how-to-git-branches.in/data.txt | 0 .../templates/intro-README.in/generate_data.py | 0 .../templates/intro-README.in/requirements.txt | 0 .../templates/intro-README.in/visualize.py | 0 .../templates/ref-xvc-file-carry-in.in/data.txt | 0 .../templates/ref-xvc-file-copy.in/data.txt | 0 .../templates/ref-xvc-file-move.in/data.txt | 0 .../templates/ref-xvc-file-recheck.in/.keep | 0 .../templates/ref-xvc-file-remove.in/data.txt | 0 .../templates/ref-xvc-file-untrack.in/data.txt | 0 .../ref-xvc-pipeline-import.in/pipeline.yaml | 0 .../data.txt | 0 .../data2.txt | 0 .../people.csv | 0 .../people.csv | 0 .../myparams.json | 0 .../myparams.toml | 0 .../myparams.yaml | 0 .../people.csv | 0 .../people.csv | 0 .../people.db | Bin {workflow_tests => lib}/tests/common/mod.rs | 0 {workflow_tests => lib}/tests/test_core_aliases.rs | 0 {workflow_tests => lib}/tests/test_core_root.rs | 0 {workflow_tests => lib}/tests/test_core_types.rs | 0 .../tests/test_core_util_file_pmp.rs | 0 .../tests/test_core_util_file_walker.rs | 0 .../tests/test_core_util_notify.rs | 0 .../tests/test_core_util_text_file.rs | 0 {workflow_tests => lib}/tests/test_core_xvc_path.rs | 0 {workflow_tests => lib}/tests/test_file_hash.rs | 0 {workflow_tests => lib}/tests/test_file_list.rs | 0 .../tests/test_file_recheck_parallel.rs | 0 .../tests/test_file_recheck_serial.rs | 0 .../tests/test_file_track_issue_104.rs | 0 .../tests/test_file_track_parallel.rs | 0 .../tests/test_file_track_serial.rs | 0 {workflow_tests => lib}/tests/test_init_1.rs | 0 {workflow_tests => lib}/tests/test_init_2.rs | 0 {workflow_tests => lib}/tests/test_init_3.rs | 0 {workflow_tests => lib}/tests/test_pipeline.rs | 0 .../tests/test_pipeline_command_process.rs | 0 {workflow_tests => lib}/tests/test_pipeline_dag.rs | 0 {workflow_tests => lib}/tests/test_pipeline_run.rs | 0 {workflow_tests => lib}/tests/test_storage_list.rs | 0 .../tests/test_storage_new_digital_ocean.rs | 0 .../tests/test_storage_new_gcs.rs | 0 .../tests/test_storage_new_generic_fs.rs | 0 .../tests/test_storage_new_generic_rsync.rs | 0 .../tests/test_storage_new_local.rs | 0 .../tests/test_storage_new_minio.rs | 0 .../tests/test_storage_new_r2.rs | 0 .../tests/test_storage_new_rsync.rs | 0 .../tests/test_storage_new_s3.rs | 0 .../tests/test_storage_new_wasabi.rs | 0 .../tests/test_walker_parallel.rs | 0 {workflow_tests => lib}/tests/test_walker_serial.rs | 0 {workflow_tests => lib}/tests/z_test_docs.rs | 0 workflow_tests/Cargo.toml | 2 +- 66 files changed, 9 insertions(+), 1 deletion(-) rename {workflow_tests => lib}/templates/how-to-benchmark-versus-dvc.in/chinese_mnist.zip (100%) rename {workflow_tests => lib}/templates/how-to-create-a-data-pipeline.in/chinese_mnist.zip (100%) rename {workflow_tests => lib}/templates/how-to-create-a-data-pipeline.in/image_to_numpy_array.py (100%) rename {workflow_tests => lib}/templates/how-to-create-a-data-pipeline.in/requirements.txt (100%) rename {workflow_tests => lib}/templates/how-to-create-a-data-pipeline.in/train.py (100%) rename {workflow_tests => lib}/templates/how-to-git-branches.in/data.txt (100%) rename {workflow_tests => lib}/templates/intro-README.in/generate_data.py (100%) rename {workflow_tests => lib}/templates/intro-README.in/requirements.txt (100%) rename {workflow_tests => lib}/templates/intro-README.in/visualize.py (100%) rename {workflow_tests => lib}/templates/ref-xvc-file-carry-in.in/data.txt (100%) rename {workflow_tests => lib}/templates/ref-xvc-file-copy.in/data.txt (100%) rename {workflow_tests => lib}/templates/ref-xvc-file-move.in/data.txt (100%) rename {workflow_tests => lib}/templates/ref-xvc-file-recheck.in/.keep (100%) rename {workflow_tests => lib}/templates/ref-xvc-file-remove.in/data.txt (100%) rename {workflow_tests => lib}/templates/ref-xvc-file-untrack.in/data.txt (100%) rename {workflow_tests => lib}/templates/ref-xvc-pipeline-import.in/pipeline.yaml (100%) rename {workflow_tests => lib}/templates/ref-xvc-pipeline-step-dependency-file.in/data.txt (100%) rename {workflow_tests => lib}/templates/ref-xvc-pipeline-step-dependency-file.in/data2.txt (100%) rename {workflow_tests => lib}/templates/ref-xvc-pipeline-step-dependency-line-items.in/people.csv (100%) rename {workflow_tests => lib}/templates/ref-xvc-pipeline-step-dependency-lines.in/people.csv (100%) rename {workflow_tests => lib}/templates/ref-xvc-pipeline-step-dependency-param.in/myparams.json (100%) rename {workflow_tests => lib}/templates/ref-xvc-pipeline-step-dependency-param.in/myparams.toml (100%) rename {workflow_tests => lib}/templates/ref-xvc-pipeline-step-dependency-param.in/myparams.yaml (100%) rename {workflow_tests => lib}/templates/ref-xvc-pipeline-step-dependency-regex-items.in/people.csv (100%) rename {workflow_tests => lib}/templates/ref-xvc-pipeline-step-dependency-regex.in/people.csv (100%) rename {workflow_tests => lib}/templates/ref-xvc-pipeline-step-dependency-sqlite-query.in/people.db (100%) rename {workflow_tests => lib}/tests/common/mod.rs (100%) rename {workflow_tests => lib}/tests/test_core_aliases.rs (100%) rename {workflow_tests => lib}/tests/test_core_root.rs (100%) rename {workflow_tests => lib}/tests/test_core_types.rs (100%) rename {workflow_tests => lib}/tests/test_core_util_file_pmp.rs (100%) rename {workflow_tests => lib}/tests/test_core_util_file_walker.rs (100%) rename {workflow_tests => lib}/tests/test_core_util_notify.rs (100%) rename {workflow_tests => lib}/tests/test_core_util_text_file.rs (100%) rename {workflow_tests => lib}/tests/test_core_xvc_path.rs (100%) rename {workflow_tests => lib}/tests/test_file_hash.rs (100%) rename {workflow_tests => lib}/tests/test_file_list.rs (100%) rename {workflow_tests => lib}/tests/test_file_recheck_parallel.rs (100%) rename {workflow_tests => lib}/tests/test_file_recheck_serial.rs (100%) rename {workflow_tests => lib}/tests/test_file_track_issue_104.rs (100%) rename {workflow_tests => lib}/tests/test_file_track_parallel.rs (100%) rename {workflow_tests => lib}/tests/test_file_track_serial.rs (100%) rename {workflow_tests => lib}/tests/test_init_1.rs (100%) rename {workflow_tests => lib}/tests/test_init_2.rs (100%) rename {workflow_tests => lib}/tests/test_init_3.rs (100%) rename {workflow_tests => lib}/tests/test_pipeline.rs (100%) rename {workflow_tests => lib}/tests/test_pipeline_command_process.rs (100%) rename {workflow_tests => lib}/tests/test_pipeline_dag.rs (100%) rename {workflow_tests => lib}/tests/test_pipeline_run.rs (100%) rename {workflow_tests => lib}/tests/test_storage_list.rs (100%) rename {workflow_tests => lib}/tests/test_storage_new_digital_ocean.rs (100%) rename {workflow_tests => lib}/tests/test_storage_new_gcs.rs (100%) rename {workflow_tests => lib}/tests/test_storage_new_generic_fs.rs (100%) rename {workflow_tests => lib}/tests/test_storage_new_generic_rsync.rs (100%) rename {workflow_tests => lib}/tests/test_storage_new_local.rs (100%) rename {workflow_tests => lib}/tests/test_storage_new_minio.rs (100%) rename {workflow_tests => lib}/tests/test_storage_new_r2.rs (100%) rename {workflow_tests => lib}/tests/test_storage_new_rsync.rs (100%) rename {workflow_tests => lib}/tests/test_storage_new_s3.rs (100%) rename {workflow_tests => lib}/tests/test_storage_new_wasabi.rs (100%) rename {workflow_tests => lib}/tests/test_walker_parallel.rs (100%) rename {workflow_tests => lib}/tests/test_walker_serial.rs (100%) rename {workflow_tests => lib}/tests/z_test_docs.rs (100%) diff --git a/.gitignore b/.gitignore index 8f93bfeed..d576b81b5 100644 --- a/.gitignore +++ b/.gitignore @@ -16,3 +16,4 @@ Cargo.lock *.swp flamegraph.svg workflow_tests/docs +lib/docs diff --git a/lib/Cargo.toml b/lib/Cargo.toml index 01ac7c63d..bbc0e0988 100644 --- a/lib/Cargo.toml +++ b/lib/Cargo.toml @@ -106,10 +106,17 @@ reflink = ["xvc-file/reflink"] [dev-dependencies] +anyhow = "^1.0" +assert_cmd = "^2.0" +assert_fs = "^1.1" +escargot = "^0.5" fs_extra = "^1.3" globset = "^0.4" jwalk = "^0.8" +predicates = "^3.0" proptest = "^1.4" shellfn = "^0.1" test-case = "^3.3" +trycmd = "^0.15" +which = "^6.0" xvc-test-helper = { version = "0.6.9-alpha.1", path = "../test_helper/" } diff --git a/workflow_tests/templates/how-to-benchmark-versus-dvc.in/chinese_mnist.zip b/lib/templates/how-to-benchmark-versus-dvc.in/chinese_mnist.zip similarity index 100% rename from workflow_tests/templates/how-to-benchmark-versus-dvc.in/chinese_mnist.zip rename to lib/templates/how-to-benchmark-versus-dvc.in/chinese_mnist.zip diff --git a/workflow_tests/templates/how-to-create-a-data-pipeline.in/chinese_mnist.zip b/lib/templates/how-to-create-a-data-pipeline.in/chinese_mnist.zip similarity index 100% rename from workflow_tests/templates/how-to-create-a-data-pipeline.in/chinese_mnist.zip rename to lib/templates/how-to-create-a-data-pipeline.in/chinese_mnist.zip diff --git a/workflow_tests/templates/how-to-create-a-data-pipeline.in/image_to_numpy_array.py b/lib/templates/how-to-create-a-data-pipeline.in/image_to_numpy_array.py similarity index 100% rename from workflow_tests/templates/how-to-create-a-data-pipeline.in/image_to_numpy_array.py rename to lib/templates/how-to-create-a-data-pipeline.in/image_to_numpy_array.py diff --git a/workflow_tests/templates/how-to-create-a-data-pipeline.in/requirements.txt b/lib/templates/how-to-create-a-data-pipeline.in/requirements.txt similarity index 100% rename from workflow_tests/templates/how-to-create-a-data-pipeline.in/requirements.txt rename to lib/templates/how-to-create-a-data-pipeline.in/requirements.txt diff --git a/workflow_tests/templates/how-to-create-a-data-pipeline.in/train.py b/lib/templates/how-to-create-a-data-pipeline.in/train.py similarity index 100% rename from workflow_tests/templates/how-to-create-a-data-pipeline.in/train.py rename to lib/templates/how-to-create-a-data-pipeline.in/train.py diff --git a/workflow_tests/templates/how-to-git-branches.in/data.txt b/lib/templates/how-to-git-branches.in/data.txt similarity index 100% rename from workflow_tests/templates/how-to-git-branches.in/data.txt rename to lib/templates/how-to-git-branches.in/data.txt diff --git a/workflow_tests/templates/intro-README.in/generate_data.py b/lib/templates/intro-README.in/generate_data.py similarity index 100% rename from workflow_tests/templates/intro-README.in/generate_data.py rename to lib/templates/intro-README.in/generate_data.py diff --git a/workflow_tests/templates/intro-README.in/requirements.txt b/lib/templates/intro-README.in/requirements.txt similarity index 100% rename from workflow_tests/templates/intro-README.in/requirements.txt rename to lib/templates/intro-README.in/requirements.txt diff --git a/workflow_tests/templates/intro-README.in/visualize.py b/lib/templates/intro-README.in/visualize.py similarity index 100% rename from workflow_tests/templates/intro-README.in/visualize.py rename to lib/templates/intro-README.in/visualize.py diff --git a/workflow_tests/templates/ref-xvc-file-carry-in.in/data.txt b/lib/templates/ref-xvc-file-carry-in.in/data.txt similarity index 100% rename from workflow_tests/templates/ref-xvc-file-carry-in.in/data.txt rename to lib/templates/ref-xvc-file-carry-in.in/data.txt diff --git a/workflow_tests/templates/ref-xvc-file-copy.in/data.txt b/lib/templates/ref-xvc-file-copy.in/data.txt similarity index 100% rename from workflow_tests/templates/ref-xvc-file-copy.in/data.txt rename to lib/templates/ref-xvc-file-copy.in/data.txt diff --git a/workflow_tests/templates/ref-xvc-file-move.in/data.txt b/lib/templates/ref-xvc-file-move.in/data.txt similarity index 100% rename from workflow_tests/templates/ref-xvc-file-move.in/data.txt rename to lib/templates/ref-xvc-file-move.in/data.txt diff --git a/workflow_tests/templates/ref-xvc-file-recheck.in/.keep b/lib/templates/ref-xvc-file-recheck.in/.keep similarity index 100% rename from workflow_tests/templates/ref-xvc-file-recheck.in/.keep rename to lib/templates/ref-xvc-file-recheck.in/.keep diff --git a/workflow_tests/templates/ref-xvc-file-remove.in/data.txt b/lib/templates/ref-xvc-file-remove.in/data.txt similarity index 100% rename from workflow_tests/templates/ref-xvc-file-remove.in/data.txt rename to lib/templates/ref-xvc-file-remove.in/data.txt diff --git a/workflow_tests/templates/ref-xvc-file-untrack.in/data.txt b/lib/templates/ref-xvc-file-untrack.in/data.txt similarity index 100% rename from workflow_tests/templates/ref-xvc-file-untrack.in/data.txt rename to lib/templates/ref-xvc-file-untrack.in/data.txt diff --git a/workflow_tests/templates/ref-xvc-pipeline-import.in/pipeline.yaml b/lib/templates/ref-xvc-pipeline-import.in/pipeline.yaml similarity index 100% rename from workflow_tests/templates/ref-xvc-pipeline-import.in/pipeline.yaml rename to lib/templates/ref-xvc-pipeline-import.in/pipeline.yaml diff --git a/workflow_tests/templates/ref-xvc-pipeline-step-dependency-file.in/data.txt b/lib/templates/ref-xvc-pipeline-step-dependency-file.in/data.txt similarity index 100% rename from workflow_tests/templates/ref-xvc-pipeline-step-dependency-file.in/data.txt rename to lib/templates/ref-xvc-pipeline-step-dependency-file.in/data.txt diff --git a/workflow_tests/templates/ref-xvc-pipeline-step-dependency-file.in/data2.txt b/lib/templates/ref-xvc-pipeline-step-dependency-file.in/data2.txt similarity index 100% rename from workflow_tests/templates/ref-xvc-pipeline-step-dependency-file.in/data2.txt rename to lib/templates/ref-xvc-pipeline-step-dependency-file.in/data2.txt diff --git a/workflow_tests/templates/ref-xvc-pipeline-step-dependency-line-items.in/people.csv b/lib/templates/ref-xvc-pipeline-step-dependency-line-items.in/people.csv similarity index 100% rename from workflow_tests/templates/ref-xvc-pipeline-step-dependency-line-items.in/people.csv rename to lib/templates/ref-xvc-pipeline-step-dependency-line-items.in/people.csv diff --git a/workflow_tests/templates/ref-xvc-pipeline-step-dependency-lines.in/people.csv b/lib/templates/ref-xvc-pipeline-step-dependency-lines.in/people.csv similarity index 100% rename from workflow_tests/templates/ref-xvc-pipeline-step-dependency-lines.in/people.csv rename to lib/templates/ref-xvc-pipeline-step-dependency-lines.in/people.csv diff --git a/workflow_tests/templates/ref-xvc-pipeline-step-dependency-param.in/myparams.json b/lib/templates/ref-xvc-pipeline-step-dependency-param.in/myparams.json similarity index 100% rename from workflow_tests/templates/ref-xvc-pipeline-step-dependency-param.in/myparams.json rename to lib/templates/ref-xvc-pipeline-step-dependency-param.in/myparams.json diff --git a/workflow_tests/templates/ref-xvc-pipeline-step-dependency-param.in/myparams.toml b/lib/templates/ref-xvc-pipeline-step-dependency-param.in/myparams.toml similarity index 100% rename from workflow_tests/templates/ref-xvc-pipeline-step-dependency-param.in/myparams.toml rename to lib/templates/ref-xvc-pipeline-step-dependency-param.in/myparams.toml diff --git a/workflow_tests/templates/ref-xvc-pipeline-step-dependency-param.in/myparams.yaml b/lib/templates/ref-xvc-pipeline-step-dependency-param.in/myparams.yaml similarity index 100% rename from workflow_tests/templates/ref-xvc-pipeline-step-dependency-param.in/myparams.yaml rename to lib/templates/ref-xvc-pipeline-step-dependency-param.in/myparams.yaml diff --git a/workflow_tests/templates/ref-xvc-pipeline-step-dependency-regex-items.in/people.csv b/lib/templates/ref-xvc-pipeline-step-dependency-regex-items.in/people.csv similarity index 100% rename from workflow_tests/templates/ref-xvc-pipeline-step-dependency-regex-items.in/people.csv rename to lib/templates/ref-xvc-pipeline-step-dependency-regex-items.in/people.csv diff --git a/workflow_tests/templates/ref-xvc-pipeline-step-dependency-regex.in/people.csv b/lib/templates/ref-xvc-pipeline-step-dependency-regex.in/people.csv similarity index 100% rename from workflow_tests/templates/ref-xvc-pipeline-step-dependency-regex.in/people.csv rename to lib/templates/ref-xvc-pipeline-step-dependency-regex.in/people.csv diff --git a/workflow_tests/templates/ref-xvc-pipeline-step-dependency-sqlite-query.in/people.db b/lib/templates/ref-xvc-pipeline-step-dependency-sqlite-query.in/people.db similarity index 100% rename from workflow_tests/templates/ref-xvc-pipeline-step-dependency-sqlite-query.in/people.db rename to lib/templates/ref-xvc-pipeline-step-dependency-sqlite-query.in/people.db diff --git a/workflow_tests/tests/common/mod.rs b/lib/tests/common/mod.rs similarity index 100% rename from workflow_tests/tests/common/mod.rs rename to lib/tests/common/mod.rs diff --git a/workflow_tests/tests/test_core_aliases.rs b/lib/tests/test_core_aliases.rs similarity index 100% rename from workflow_tests/tests/test_core_aliases.rs rename to lib/tests/test_core_aliases.rs diff --git a/workflow_tests/tests/test_core_root.rs b/lib/tests/test_core_root.rs similarity index 100% rename from workflow_tests/tests/test_core_root.rs rename to lib/tests/test_core_root.rs diff --git a/workflow_tests/tests/test_core_types.rs b/lib/tests/test_core_types.rs similarity index 100% rename from workflow_tests/tests/test_core_types.rs rename to lib/tests/test_core_types.rs diff --git a/workflow_tests/tests/test_core_util_file_pmp.rs b/lib/tests/test_core_util_file_pmp.rs similarity index 100% rename from workflow_tests/tests/test_core_util_file_pmp.rs rename to lib/tests/test_core_util_file_pmp.rs diff --git a/workflow_tests/tests/test_core_util_file_walker.rs b/lib/tests/test_core_util_file_walker.rs similarity index 100% rename from workflow_tests/tests/test_core_util_file_walker.rs rename to lib/tests/test_core_util_file_walker.rs diff --git a/workflow_tests/tests/test_core_util_notify.rs b/lib/tests/test_core_util_notify.rs similarity index 100% rename from workflow_tests/tests/test_core_util_notify.rs rename to lib/tests/test_core_util_notify.rs diff --git a/workflow_tests/tests/test_core_util_text_file.rs b/lib/tests/test_core_util_text_file.rs similarity index 100% rename from workflow_tests/tests/test_core_util_text_file.rs rename to lib/tests/test_core_util_text_file.rs diff --git a/workflow_tests/tests/test_core_xvc_path.rs b/lib/tests/test_core_xvc_path.rs similarity index 100% rename from workflow_tests/tests/test_core_xvc_path.rs rename to lib/tests/test_core_xvc_path.rs diff --git a/workflow_tests/tests/test_file_hash.rs b/lib/tests/test_file_hash.rs similarity index 100% rename from workflow_tests/tests/test_file_hash.rs rename to lib/tests/test_file_hash.rs diff --git a/workflow_tests/tests/test_file_list.rs b/lib/tests/test_file_list.rs similarity index 100% rename from workflow_tests/tests/test_file_list.rs rename to lib/tests/test_file_list.rs diff --git a/workflow_tests/tests/test_file_recheck_parallel.rs b/lib/tests/test_file_recheck_parallel.rs similarity index 100% rename from workflow_tests/tests/test_file_recheck_parallel.rs rename to lib/tests/test_file_recheck_parallel.rs diff --git a/workflow_tests/tests/test_file_recheck_serial.rs b/lib/tests/test_file_recheck_serial.rs similarity index 100% rename from workflow_tests/tests/test_file_recheck_serial.rs rename to lib/tests/test_file_recheck_serial.rs diff --git a/workflow_tests/tests/test_file_track_issue_104.rs b/lib/tests/test_file_track_issue_104.rs similarity index 100% rename from workflow_tests/tests/test_file_track_issue_104.rs rename to lib/tests/test_file_track_issue_104.rs diff --git a/workflow_tests/tests/test_file_track_parallel.rs b/lib/tests/test_file_track_parallel.rs similarity index 100% rename from workflow_tests/tests/test_file_track_parallel.rs rename to lib/tests/test_file_track_parallel.rs diff --git a/workflow_tests/tests/test_file_track_serial.rs b/lib/tests/test_file_track_serial.rs similarity index 100% rename from workflow_tests/tests/test_file_track_serial.rs rename to lib/tests/test_file_track_serial.rs diff --git a/workflow_tests/tests/test_init_1.rs b/lib/tests/test_init_1.rs similarity index 100% rename from workflow_tests/tests/test_init_1.rs rename to lib/tests/test_init_1.rs diff --git a/workflow_tests/tests/test_init_2.rs b/lib/tests/test_init_2.rs similarity index 100% rename from workflow_tests/tests/test_init_2.rs rename to lib/tests/test_init_2.rs diff --git a/workflow_tests/tests/test_init_3.rs b/lib/tests/test_init_3.rs similarity index 100% rename from workflow_tests/tests/test_init_3.rs rename to lib/tests/test_init_3.rs diff --git a/workflow_tests/tests/test_pipeline.rs b/lib/tests/test_pipeline.rs similarity index 100% rename from workflow_tests/tests/test_pipeline.rs rename to lib/tests/test_pipeline.rs diff --git a/workflow_tests/tests/test_pipeline_command_process.rs b/lib/tests/test_pipeline_command_process.rs similarity index 100% rename from workflow_tests/tests/test_pipeline_command_process.rs rename to lib/tests/test_pipeline_command_process.rs diff --git a/workflow_tests/tests/test_pipeline_dag.rs b/lib/tests/test_pipeline_dag.rs similarity index 100% rename from workflow_tests/tests/test_pipeline_dag.rs rename to lib/tests/test_pipeline_dag.rs diff --git a/workflow_tests/tests/test_pipeline_run.rs b/lib/tests/test_pipeline_run.rs similarity index 100% rename from workflow_tests/tests/test_pipeline_run.rs rename to lib/tests/test_pipeline_run.rs diff --git a/workflow_tests/tests/test_storage_list.rs b/lib/tests/test_storage_list.rs similarity index 100% rename from workflow_tests/tests/test_storage_list.rs rename to lib/tests/test_storage_list.rs diff --git a/workflow_tests/tests/test_storage_new_digital_ocean.rs b/lib/tests/test_storage_new_digital_ocean.rs similarity index 100% rename from workflow_tests/tests/test_storage_new_digital_ocean.rs rename to lib/tests/test_storage_new_digital_ocean.rs diff --git a/workflow_tests/tests/test_storage_new_gcs.rs b/lib/tests/test_storage_new_gcs.rs similarity index 100% rename from workflow_tests/tests/test_storage_new_gcs.rs rename to lib/tests/test_storage_new_gcs.rs diff --git a/workflow_tests/tests/test_storage_new_generic_fs.rs b/lib/tests/test_storage_new_generic_fs.rs similarity index 100% rename from workflow_tests/tests/test_storage_new_generic_fs.rs rename to lib/tests/test_storage_new_generic_fs.rs diff --git a/workflow_tests/tests/test_storage_new_generic_rsync.rs b/lib/tests/test_storage_new_generic_rsync.rs similarity index 100% rename from workflow_tests/tests/test_storage_new_generic_rsync.rs rename to lib/tests/test_storage_new_generic_rsync.rs diff --git a/workflow_tests/tests/test_storage_new_local.rs b/lib/tests/test_storage_new_local.rs similarity index 100% rename from workflow_tests/tests/test_storage_new_local.rs rename to lib/tests/test_storage_new_local.rs diff --git a/workflow_tests/tests/test_storage_new_minio.rs b/lib/tests/test_storage_new_minio.rs similarity index 100% rename from workflow_tests/tests/test_storage_new_minio.rs rename to lib/tests/test_storage_new_minio.rs diff --git a/workflow_tests/tests/test_storage_new_r2.rs b/lib/tests/test_storage_new_r2.rs similarity index 100% rename from workflow_tests/tests/test_storage_new_r2.rs rename to lib/tests/test_storage_new_r2.rs diff --git a/workflow_tests/tests/test_storage_new_rsync.rs b/lib/tests/test_storage_new_rsync.rs similarity index 100% rename from workflow_tests/tests/test_storage_new_rsync.rs rename to lib/tests/test_storage_new_rsync.rs diff --git a/workflow_tests/tests/test_storage_new_s3.rs b/lib/tests/test_storage_new_s3.rs similarity index 100% rename from workflow_tests/tests/test_storage_new_s3.rs rename to lib/tests/test_storage_new_s3.rs diff --git a/workflow_tests/tests/test_storage_new_wasabi.rs b/lib/tests/test_storage_new_wasabi.rs similarity index 100% rename from workflow_tests/tests/test_storage_new_wasabi.rs rename to lib/tests/test_storage_new_wasabi.rs diff --git a/workflow_tests/tests/test_walker_parallel.rs b/lib/tests/test_walker_parallel.rs similarity index 100% rename from workflow_tests/tests/test_walker_parallel.rs rename to lib/tests/test_walker_parallel.rs diff --git a/workflow_tests/tests/test_walker_serial.rs b/lib/tests/test_walker_serial.rs similarity index 100% rename from workflow_tests/tests/test_walker_serial.rs rename to lib/tests/test_walker_serial.rs diff --git a/workflow_tests/tests/z_test_docs.rs b/lib/tests/z_test_docs.rs similarity index 100% rename from workflow_tests/tests/z_test_docs.rs rename to lib/tests/z_test_docs.rs diff --git a/workflow_tests/Cargo.toml b/workflow_tests/Cargo.toml index e8bf92895..95d92621d 100644 --- a/workflow_tests/Cargo.toml +++ b/workflow_tests/Cargo.toml @@ -11,7 +11,7 @@ categories = ["command-line-utilities", "development-tools"] keywords = ["file", "devops", "git", "versioning", "mlops"] [[bin]] -name = "xvc" +name = "xvc-tester" path = "src/main.rs" [lib] From 77fed5c101f32ba05af98b64f2bc14b1883a72d2 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sun, 28 Jul 2024 20:00:44 +0300 Subject: [PATCH 101/257] Added docs dir to lib --- lib/docs/.keep | 0 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 lib/docs/.keep diff --git a/lib/docs/.keep b/lib/docs/.keep new file mode 100644 index 000000000..e69de29bb From 6b60a8e198becaa9d13fe4317842b6b2daec3abd Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sun, 28 Jul 2024 20:03:15 +0300 Subject: [PATCH 102/257] Remove workflow_tests --- workflow_tests/Cargo.toml | 78 ---------------------------------- workflow_tests/docs/.gitignore | 10 ----- workflow_tests/src/lib.rs | 19 --------- workflow_tests/src/main.rs | 14 ------ 4 files changed, 121 deletions(-) delete mode 100644 workflow_tests/Cargo.toml delete mode 100644 workflow_tests/docs/.gitignore delete mode 100755 workflow_tests/src/lib.rs delete mode 100755 workflow_tests/src/main.rs diff --git a/workflow_tests/Cargo.toml b/workflow_tests/Cargo.toml deleted file mode 100644 index 95d92621d..000000000 --- a/workflow_tests/Cargo.toml +++ /dev/null @@ -1,78 +0,0 @@ -[package] -name = "xvc-workflow-tests" -version = "0.6.9-alpha.1" -edition = "2021" -description = "Integration test suite for Xvc" -authors = ["Emre Şahin "] -license = "GPL-3.0" -homepage = "https://xvc.dev" -repository = "https://github.com/iesahin/xvc" -categories = ["command-line-utilities", "development-tools"] -keywords = ["file", "devops", "git", "versioning", "mlops"] - -[[bin]] -name = "xvc-tester" -path = "src/main.rs" - -[lib] -name = "xvc_tests" -crate-type = ["rlib"] - -[dependencies] -xvc = { version = "0.6.9-alpha.1", path = "../lib" } -xvc-config = { version = "0.6.9-alpha.1", path = "../config" } -xvc-core = { version = "0.6.9-alpha.1", path = "../core" } -xvc-logging = { version = "0.6.9-alpha.1", path = "../logging" } -xvc-ecs = { version = "0.6.9-alpha.1", path = "../ecs" } -xvc-file = { version = "0.6.9-alpha.1", path = "../file" } -xvc-pipeline = { version = "0.6.9-alpha.1", path = "../pipeline" } -xvc-walker = { version = "0.6.9-alpha.1", path = "../walker" } -xvc-storage = { version = "0.6.9-alpha.1", path = "../storage" } - -## packages for testing -clap = { version = "^4.4", features = ["derive"] } - -itertools = "^0.13" -serde = { version = "^1.0", features = ["derive"] } -serde_yaml = "^0.9" -serde_json = "^1.0" -crossbeam-channel = "^0.5" -subprocess = "^0.2" -log = "^0.4" -regex = "^1.10" -rand = "^0.8" -which = "^6.0" - -[features] -default = ["s3", "minio", "r2", "gcs", "digital-ocean"] -s3 = ["xvc-storage/s3"] -minio = ["xvc-storage/minio"] -r2 = ["xvc-storage/r2"] -gcs = ["xvc-storage/gcs"] -wasabi = ["xvc-storage/wasabi"] -digital-ocean = ["xvc-storage/digital-ocean"] -test-s3 = ["s3"] -test-minio = ["minio"] -test-digital-ocean = ["digital-ocean"] -test-wasabi = ["wasabi"] -test-gcs = ["gcs"] -test-r2 = ["r2"] -test-rsync = [] -test-generic-rsync = [] - - -[dev-dependencies] -proptest = "^1.4" -test-case = "^3.3" -globset = "^0.4" -escargot = "^0.5" -xvc-test-helper = { version = "0.6.9-alpha.1", path = "../test_helper" } -shellfn = "^0.1" -jwalk = "^0.8" -anyhow = "^1.0" -assert_cmd = "^2.0" -assert_fs = "^1.1" -predicates = "^3.0" -trycmd = "^0.15" -which = "^6.0" -fs_extra = "^1.3" diff --git a/workflow_tests/docs/.gitignore b/workflow_tests/docs/.gitignore deleted file mode 100644 index 76a17297e..000000000 --- a/workflow_tests/docs/.gitignore +++ /dev/null @@ -1,10 +0,0 @@ -## These are produced in test time -intro/* -!intro/.keep -start/* -!start/.keep -ref/* -!ref/.keep -how-to/* -!how-to/.keep - diff --git a/workflow_tests/src/lib.rs b/workflow_tests/src/lib.rs deleted file mode 100755 index 94fa246fb..000000000 --- a/workflow_tests/src/lib.rs +++ /dev/null @@ -1,19 +0,0 @@ -#![forbid(unsafe_code)] - -pub use xvc_config as config; -pub use xvc_core as core; -pub use xvc_ecs as ecs; -pub use xvc_file as file; -pub use xvc_logging as logging; -pub use xvc_pipeline as pipeline; - -pub use xvc_logging::watch; - -use xvc::{cli::XvcCLI, error::Result, XvcRootOpt}; - -/// Run xvc in another process with `args` in `xvc_root_opt`. -/// -/// See [xvc::cli::test_dispatch]. -pub fn dispatch(cli_opts: XvcCLI) -> Result { - xvc::cli::dispatch(cli_opts) -} diff --git a/workflow_tests/src/main.rs b/workflow_tests/src/main.rs deleted file mode 100755 index ff4c0cd24..000000000 --- a/workflow_tests/src/main.rs +++ /dev/null @@ -1,14 +0,0 @@ -#![warn(missing_docs)] -//! The entry point for xvc cli - -use xvc::error::Result; - -/// The entry point of the `xvc` cli. -/// -/// It parses the command line arguments [xvc::cli::XvcCLI] and calls [xvc::cli::dispatch] -/// -fn main() -> Result<()> { - let cli_opts = xvc::cli::XvcCLI::from_args_os(std::env::args_os())?; - xvc::cli::dispatch(cli_opts)?; - Ok(()) -} From 9fbd0f6aca897e6a46f71a6c258e080dae13cf72 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sun, 28 Jul 2024 20:03:57 +0300 Subject: [PATCH 103/257] remove workflow crate refs --- CHANGELOG.md | 2 ++ Cargo.toml | 1 - 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 64c9595e0..6b7c76fb9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,8 @@ tree, and subtasks are marked with indentation. - Added sqlite-query dependency to xvc pipelines. A step is invalidated to run when the output from an SQLite query changes. - Moved bundled sqlite behind a feature flag, bundled_sqlite. +- Merged xvc-workflow-tests crate to the main xvc crate. +- Added compiling Xvc with non-default features document ### 0.6.8 (2024-07-15) diff --git a/Cargo.toml b/Cargo.toml index 481cb28d9..97083fdb7 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -10,7 +10,6 @@ members = [ "logging", "walker", "test_helper", - "workflow_tests", ] resolver = "2" From 9f946ac9a6a59e7277615b95c71e57e7f3f874fe Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sun, 28 Jul 2024 20:11:11 +0300 Subject: [PATCH 104/257] add test features --- .github/workflows/rust.yml | 4 ++-- lib/Cargo.toml | 19 ++++++++++++++++++- 2 files changed, 20 insertions(+), 3 deletions(-) diff --git a/.github/workflows/rust.yml b/.github/workflows/rust.yml index e762d3d4f..8c2a6694a 100644 --- a/.github/workflows/rust.yml +++ b/.github/workflows/rust.yml @@ -27,11 +27,11 @@ jobs: # rust: beta - build: nightly rust: nightly-2024-01-01 - test-args: --features test-s3,test-minio,test-r2,test-rsync,test-generic-rsync --no-fail-fast + test-args: --features test-ci --no-fail-fast ## for submitters other than me, I'll add another job here. # test-args: --no-fail-fast # test-args: --all-features - benches: true + # benches: true coverage: true env: AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} diff --git a/lib/Cargo.toml b/lib/Cargo.toml index bbc0e0988..25d1f63b2 100644 --- a/lib/Cargo.toml +++ b/lib/Cargo.toml @@ -103,7 +103,24 @@ wasabi = ["xvc-storage/wasabi"] digital-ocean = ["xvc-storage/digital-ocean"] bundled-sqlite = ["xvc-pipeline/bundled-sqlite"] reflink = ["xvc-file/reflink"] - +test-s3 = ["s3"] +test-minio = ["minio"] +test-r2 = ["r2"] +test-gcs = ["gcs"] +test-wasabi = ["wasabi"] +test-digital-ocean = ["digital-ocean"] +test-rsync = [] +test-generic-rsync = [] +test-ci = [ + "test-s3", + "test-minio", + "test-r2", + "test-gcs", + "test-wasabi", + "test-digital-ocean", + "test-rsync", + "test-generic-rsync", +] [dev-dependencies] anyhow = "^1.0" From 2b862f2e67f264e40378374ad435b27389fcae1a Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sun, 28 Jul 2024 20:12:52 +0300 Subject: [PATCH 105/257] ws --- .github/workflows/rust.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/rust.yml b/.github/workflows/rust.yml index 8c2a6694a..c7f011770 100644 --- a/.github/workflows/rust.yml +++ b/.github/workflows/rust.yml @@ -27,7 +27,7 @@ jobs: # rust: beta - build: nightly rust: nightly-2024-01-01 - test-args: --features test-ci --no-fail-fast + test-args: --features test-ci --no-fail-fast ## for submitters other than me, I'll add another job here. # test-args: --no-fail-fast # test-args: --all-features From 727ddcfe97ff1c50d7784693a0e85c3766a8f0f1 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sun, 28 Jul 2024 20:19:00 +0300 Subject: [PATCH 106/257] Bump version --- .../intro/compile-without-default-features.md | 681 +----------------- config/Cargo.toml | 6 +- core/Cargo.toml | 12 +- ecs/Cargo.toml | 4 +- file/Cargo.toml | 16 +- lib/Cargo.toml | 20 +- logging/Cargo.toml | 2 +- pipeline/Cargo.toml | 16 +- storage/Cargo.toml | 14 +- test_helper/Cargo.toml | 4 +- walker/Cargo.toml | 6 +- 11 files changed, 52 insertions(+), 729 deletions(-) diff --git a/book/src/intro/compile-without-default-features.md b/book/src/intro/compile-without-default-features.md index 0406fa742..cebb723a1 100644 --- a/book/src/intro/compile-without-default-features.md +++ b/book/src/intro/compile-without-default-features.md @@ -6,685 +6,8 @@ When you turn off all remote storage features, async runtime (`tokio`) is also e ````bash cargo build --no-default-features --release -warning: output filename collision. -The bin target `xvc` in package `xvc-workflow-tests v0.6.9-alpha.1 (/Users/iex/github.com/iesahin/xvc/workflow_tests)` has the same output filename as the bin target `xvc` in package `xvc v0.6.9-alpha.1 (/Users/iex/github.com/iesahin/xvc/lib)`. -Colliding filename is: /Users/iex/github.com/iesahin/xvc/target/release/xvc -The targets should have unique names. -Consider changing their names to be unique or compiling them separately. -This may become a hard error in the future; see . -warning: output filename collision. -The bin target `xvc` in package `xvc-workflow-tests v0.6.9-alpha.1 (/Users/iex/github.com/iesahin/xvc/workflow_tests)` has the same output filename as the bin target `xvc` in package `xvc v0.6.9-alpha.1 (/Users/iex/github.com/iesahin/xvc/lib)`. -Colliding filename is: /Users/iex/github.com/iesahin/xvc/target/release/xvc.dSYM -The targets should have unique names. -Consider changing their names to be unique or compiling them separately. -This may become a hard error in the future; see . - Compiling proc-macro2 v1.0.86 - Compiling unicode-ident v1.0.12 - Compiling libc v0.2.155 - Compiling cfg-if v1.0.0 - Compiling autocfg v1.3.0 - Compiling version_check v0.9.4 - Compiling serde v1.0.204 - Compiling memchr v2.7.4 - Compiling once_cell v1.19.0 - Compiling crossbeam-utils v0.8.20 - Compiling bitflags v2.6.0 - Compiling zerocopy v0.7.35 - Compiling allocator-api2 v0.2.18 - Compiling ahash v0.8.11 - Compiling bytes v1.6.1 - Compiling log v0.4.22 - Compiling itoa v1.0.11 - Compiling pin-project-lite v0.2.14 - Compiling core-foundation-sys v0.8.6 - Compiling aho-corasick v1.1.3 - Compiling rustix v0.38.34 - Compiling regex-syntax v0.8.4 - Compiling futures-core v0.3.30 - Compiling typenum v1.17.0 - Compiling equivalent v1.0.1 - Compiling ident_case v1.0.1 - Compiling fnv v1.0.7 - Compiling num-traits v0.2.19 - Compiling slab v0.4.9 - Compiling generic-array v0.14.7 - Compiling futures-sink v0.3.30 - Compiling errno v0.3.9 - Compiling regex-automata v0.4.7 - Compiling mio v0.8.11 - Compiling quote v1.0.36 - Compiling num_cpus v1.16.0 - Compiling syn v2.0.72 - Compiling socket2 v0.5.7 - Compiling crossbeam-epoch v0.9.18 - Compiling crossbeam-channel v0.5.13 - Compiling tokio v1.38.1 - Compiling hashbrown v0.14.5 - Compiling crossbeam-deque v0.8.5 - Compiling core-foundation v0.9.4 - Compiling indexmap v2.2.6 - Compiling crunchy v0.2.2 - Compiling thiserror v1.0.63 - Compiling fastrand v2.1.0 - Compiling ryu v1.0.18 - Compiling tempfile v3.10.1 - Compiling http v0.2.12 - Compiling security-framework-sys v2.11.1 - Compiling is-terminal v0.4.12 - Compiling futures-channel v0.3.30 - Compiling lock_api v0.4.12 - Compiling native-tls v0.2.12 - Compiling either v1.13.0 - Compiling serde_derive v1.0.204 - Compiling thiserror-impl v1.0.63 - Compiling futures-macro v0.3.30 - Compiling lazy_static v1.5.0 - Compiling tiny-keccak v2.0.2 - Compiling tinyvec_macros v0.1.1 - Compiling anstyle v1.0.7 - Compiling smallvec v1.13.2 - Compiling futures-io v0.3.30 - Compiling rayon-core v1.12.1 - Compiling futures-task v0.3.30 - Compiling pin-utils v0.1.0 - Compiling parking_lot_core v0.9.10 - Compiling futures-util v0.3.30 - Compiling tinyvec v1.8.0 - Compiling colored v1.9.4 - Compiling security-framework v2.11.1 - Compiling bstr v1.9.1 - Compiling getrandom v0.2.15 - Compiling percent-encoding v2.3.1 - Compiling anyhow v1.0.86 - Compiling scopeguard v1.2.0 - Compiling serde_json v1.0.120 - Compiling heck v0.5.0 - Compiling same-file v1.0.6 - Compiling walkdir v2.5.0 - Compiling form_urlencoded v1.2.1 - Compiling rand_core v0.6.4 - Compiling fern v0.6.2 - Compiling unicode-normalization v0.1.23 - Compiling block-buffer v0.10.4 - Compiling crypto-common v0.1.6 - Compiling regex v1.10.5 - Compiling tracing-core v0.1.32 - Compiling subtle v2.6.1 - Compiling httparse v1.9.4 - Compiling utf8parse v0.2.2 - Compiling ppv-lite86 v0.2.17 - Compiling unicode-bidi v0.3.15 - Compiling const-random-macro v0.1.16 - Compiling rand_chacha v0.3.1 - Compiling anstyle-parse v0.2.4 - Compiling tracing v0.1.40 - Compiling digest v0.10.7 - Compiling xvc-logging v0.6.9-alpha.1 (/Users/iex/github.com/iesahin/xvc/logging) - Compiling idna v0.5.0 - Compiling rayon v1.10.0 - Compiling globset v0.4.14 - Compiling tokio-util v0.7.11 - Compiling is_terminal_polyfill v1.70.0 - Compiling anstyle-query v1.1.0 - Compiling colorchoice v1.0.1 - Compiling rustversion v1.0.17 - Compiling strsim v0.11.1 - Compiling paste v1.0.15 - Compiling try-lock v0.2.5 - Compiling syn v1.0.109 - Compiling want v0.3.1 - Compiling url v2.5.2 - Compiling h2 v0.3.26 - Compiling darling_core v0.20.10 - Compiling anstream v0.6.14 - Compiling const-random v0.1.18 - Compiling rand v0.8.5 - Compiling http-body v0.4.6 - Compiling crossbeam-queue v0.3.11 - Compiling cpufeatures v0.2.12 - Compiling smartstring v1.0.1 - Compiling tower-service v0.3.2 - Compiling httpdate v1.0.3 - Compiling clap_lex v0.7.1 - Compiling system-configuration-sys v0.5.0 - Compiling adler v1.0.2 - Compiling miniz_oxide v0.7.4 - Compiling darling_macro v0.20.10 - Compiling hyper v0.14.30 - Compiling clap_builder v4.5.9 - Compiling crossbeam v0.8.4 - Compiling serde_spanned v0.6.6 - Compiling toml_datetime v0.6.6 - Compiling dlv-list v0.5.2 - Compiling tokio-native-tls v0.3.1 - Compiling clap_derive v4.5.8 - Compiling fsevent-sys v4.1.0 - Compiling filetime v0.2.23 - Compiling crc32fast v1.4.2 - Compiling time-core v0.1.2 - Compiling powerfmt v0.2.0 - Compiling static_assertions v1.1.0 - Compiling byteorder v1.5.0 - Compiling cc v1.1.6 - Compiling hashbrown v0.13.2 - Compiling winnow v0.6.14 - Compiling base64 v0.21.7 - Compiling num-conv v0.1.0 - Compiling time-macros v0.2.18 - Compiling ordered-multimap v0.6.0 - Compiling hyper-tls v0.5.0 - Compiling rmp v0.8.14 - Compiling blake3 v1.5.3 - Compiling deranged v0.3.11 - Compiling clap v4.5.9 - Compiling flate2 v1.0.30 - Compiling toml_edit v0.22.16 - Compiling notify v6.1.1 - Compiling strum_macros v0.26.4 - Compiling darling v0.20.10 - Compiling dashmap v6.0.1 - Compiling itertools v0.13.0 - Compiling dirs-sys-next v0.1.2 - Compiling unsafe-libyaml v0.2.11 - Compiling convert_case v0.4.0 - Compiling rxml_validation v0.9.1 - Compiling strsim v0.9.3 - Compiling home v0.5.9 - Compiling strum v0.26.3 - Compiling bitflags v1.3.2 - Compiling toml v0.8.15 - Compiling time v0.3.36 - Compiling system-configuration v0.5.1 - Compiling darling_core v0.10.2 - Compiling xvc-walker v0.6.9-alpha.1 (/Users/iex/github.com/iesahin/xvc/walker) - Compiling rxml v0.9.1 - Compiling serde_yaml v0.9.34+deprecated - Compiling cached_proc_macro v0.22.0 - Compiling directories-next v2.0.0 - Compiling rmp-serde v1.3.0 - Compiling async-compression v0.4.12 - Compiling rust-ini v0.19.0 - Compiling rustls-pemfile v1.0.4 - Compiling keccak v0.1.5 - Compiling sha2 v0.10.8 - Compiling attohttpc v0.26.1 - Compiling quick-xml v0.30.0 - Compiling serde_urlencoded v0.7.1 - Compiling hex v0.4.3 - Compiling futures-executor v0.3.30 - Compiling path-dedot v3.1.1 - Compiling encoding_rs v0.8.34 - Compiling web-time v1.1.0 - Compiling arrayvec v0.7.4 - Compiling constant_time_eq v0.3.0 - Compiling ipnet v2.9.0 - Compiling sync_wrapper v0.1.2 - Compiling mime v0.3.17 - Compiling arrayref v0.3.8 - Compiling cached_proc_macro_types v0.1.1 - Compiling cached v0.53.0 - Compiling reqwest v0.11.27 - Compiling aws-creds v0.36.0 - Compiling path-absolutize v3.1.1 - Compiling futures v0.3.30 - Compiling sha3 v0.10.8 - Compiling xvc-ecs v0.6.9-alpha.1 (/Users/iex/github.com/iesahin/xvc/ecs) - Compiling xvc-config v0.6.9-alpha.1 (/Users/iex/github.com/iesahin/xvc/config) - Compiling darling_macro v0.10.2 - Compiling minidom v0.15.2 - Compiling derive_more v0.99.18 - Compiling jwalk v0.8.1 - Compiling uuid v1.10.0 - Compiling relative-path v1.9.3 - Compiling blake2 v0.10.6 - Compiling hmac v0.12.1 - Compiling aws-region v0.25.4 - Compiling maybe-async v0.2.10 - Compiling async-trait v0.1.81 - Compiling tokio-stream v0.1.15 - Compiling pkg-config v0.3.30 - Compiling md5 v0.7.0 - Compiling vcpkg v0.2.15 - Compiling glob v0.3.1 - Compiling seahash v4.1.0 - Compiling darling v0.10.2 - Compiling which v6.0.1 - Compiling rust-s3 v0.34.0 - Compiling subprocess v0.2.9 - Compiling libsqlite3-sys v0.30.0 - Compiling xvc-core v0.6.9-alpha.1 (/Users/iex/github.com/iesahin/xvc/core) - Compiling iana-time-zone v0.1.60 - Compiling predicates-core v1.0.6 - Compiling doc-comment v0.3.3 - Compiling derive_builder v0.9.0 - Compiling humantime v2.1.0 - Compiling derive_builder_core v0.9.0 - Compiling chrono v0.4.38 - Compiling parking_lot v0.12.3 - Compiling float-cmp v0.9.0 - Compiling reflink v0.1.3 - Compiling parse-size v1.0.0 - Compiling normalize-line-endings v0.3.0 - Compiling difflib v0.4.0 - Compiling termtree v0.4.1 - Compiling predicates-tree v1.0.9 -warning: use of deprecated macro `EnumVariantNames`: please use `#[derive(VariantNames)]` instead - --> core/src/types/hashalgorithm.rs:27:5 - | -27 | EnumVariantNames, - | ^^^^^^^^^^^^^^^^ - | - = note: `#[warn(deprecated)]` on by default - - Compiling predicates v3.1.0 - Compiling crossterm v0.27.0 - Compiling ignore v0.4.22 - Compiling hashlink v0.9.1 - Compiling fallible-iterator v0.3.0 - Compiling fixedbitset v0.4.2 - Compiling unicode-width v0.1.13 - Compiling assert_cmd v2.0.14 - Compiling fallible-streaming-iterator v0.1.9 - Compiling globwalk v0.9.1 - Compiling comfy-table v7.1.1 - Compiling petgraph v0.6.5 - Compiling tabbycat v0.1.3 - Compiling sad_machine v1.0.0 - Compiling rusqlite v0.32.0 - Compiling git-version-macro v0.3.9 - Compiling wait-timeout v0.2.0 - Compiling git-version v0.3.9 - Compiling assert_fs v1.1.1 - Compiling xvc-test-helper v0.6.9-alpha.1 (/Users/iex/github.com/iesahin/xvc/test_helper) - Compiling xvc-storage v0.6.9-alpha.1 (/Users/iex/github.com/iesahin/xvc/storage) -warning: unused import: `panic` - --> storage/src/storage/mod.rs:35:26 - | -35 | use xvc_logging::{error, panic, watch, XvcOutputSender}; - | ^^^^^ - | - = note: `#[warn(unused_imports)]` on by default - -warning: unused variable: `xvc_root` - --> storage/src/storage/async_common.rs:273:50 - | -273 | fn init(&mut self, output: &XvcOutputSender, xvc_root: &XvcRoot) -> Result - | ^^^^^^^^ help: if this is intentional, prefix it with an underscore: `_xvc_root` - | - = note: `#[warn(unused_variables)]` on by default - -warning: unused variable: `xvc_root` - --> storage/src/storage/async_common.rs:313:9 - | -313 | xvc_root: &xvc_core::XvcRoot, - | ^^^^^^^^ help: if this is intentional, prefix it with an underscore: `_xvc_root` - -warning: unused variable: `xvc_root` - --> storage/src/storage/async_common.rs:327:9 - | -327 | xvc_root: &xvc_core::XvcRoot, - | ^^^^^^^^ help: if this is intentional, prefix it with an underscore: `_xvc_root` - -warning: unused variable: `xvc_root` - --> storage/src/storage/async_common.rs:340:9 - | -340 | xvc_root: &xvc_core::XvcRoot, - | ^^^^^^^^ help: if this is intentional, prefix it with an underscore: `_xvc_root` - -warning: unused variable: `output` - --> storage/src/storage/generic.rs:448:9 - | -448 | output: &XvcOutputSender, - | ^^^^^^ help: if this is intentional, prefix it with an underscore: `_output` - -warning: unused variable: `xvc_root` - --> storage/src/storage/generic.rs:449:9 - | -449 | xvc_root: &XvcRoot, - | ^^^^^^^^ help: if this is intentional, prefix it with an underscore: `_xvc_root` - -warning: unused variable: `path` - --> storage/src/storage/generic.rs:450:9 - | -450 | path: &XvcCachePath, - | ^^^^ help: if this is intentional, prefix it with an underscore: `_path` - -warning: unused variable: `period` - --> storage/src/storage/generic.rs:451:9 - | -451 | period: std::time::Duration, - | ^^^^^^ help: if this is intentional, prefix it with an underscore: `_period` - -warning: unused variable: `output` - --> storage/src/storage/rsync.rs:475:9 - | -475 | output: &XvcOutputSender, - | ^^^^^^ help: if this is intentional, prefix it with an underscore: `_output` - -warning: unused variable: `xvc_root` - --> storage/src/storage/rsync.rs:476:9 - | -476 | xvc_root: &XvcRoot, - | ^^^^^^^^ help: if this is intentional, prefix it with an underscore: `_xvc_root` - -warning: unused variable: `path` - --> storage/src/storage/rsync.rs:477:9 - | -477 | path: &XvcCachePath, - | ^^^^ help: if this is intentional, prefix it with an underscore: `_path` - -warning: unused variable: `period` - --> storage/src/storage/rsync.rs:478:9 - | -478 | period: std::time::Duration, - | ^^^^^^ help: if this is intentional, prefix it with an underscore: `_period` - -warning: missing documentation for a module - --> storage/src/storage/mod.rs:3:1 - | -3 | pub mod async_common; - | ^^^^^^^^^^^^^^^^^^^^ - | -note: the lint level is defined here - --> storage/src/lib.rs:6:9 - | -6 | #![warn(missing_docs)] - | ^^^^^^^^^^^^ - -warning: missing documentation for a module - --> storage/src/storage/mod.rs:4:1 - | -4 | pub mod common; - | ^^^^^^^^^^^^^^ - -warning: missing documentation for a trait - --> storage/src/storage/async_common.rs:33:1 - | -33 | pub trait XvcS3StorageOperations { - | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -warning: missing documentation for a method - --> storage/src/storage/async_common.rs:34:5 - | -34 | fn storage_prefix(&self) -> String; - | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -warning: missing documentation for a method - --> storage/src/storage/async_common.rs:35:5 - | -35 | fn guid(&self) -> &XvcStorageGuid; - | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -warning: missing documentation for a method - --> storage/src/storage/async_common.rs:36:5 - | -36 | fn get_bucket(&self) -> Result; - | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -warning: missing documentation for a method - --> storage/src/storage/async_common.rs:37:5 - | -37 | fn credentials(&self) -> Result; - | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -warning: missing documentation for a method - --> storage/src/storage/async_common.rs:38:5 - | -38 | fn bucket_name(&self) -> String; - | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -warning: missing documentation for a method - --> storage/src/storage/async_common.rs:39:5 - | -39 | fn build_storage_path(&self, cache_path: &XvcCachePath) -> XvcStoragePath { - | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -warning: missing documentation for a method - --> storage/src/storage/async_common.rs:48:5 - | -48 | fn region(&self) -> String; - | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -warning: missing documentation for a method - --> storage/src/storage/async_common.rs:49:5 - | -49 | async fn write_storage_guid(&self) -> Result<()> { - | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -warning: use of `async fn` in public traits is discouraged as auto trait bounds cannot be specified - --> storage/src/storage/async_common.rs:49:5 - | -49 | async fn write_storage_guid(&self) -> Result<()> { - | ^^^^^ - | - = note: you can suppress this lint if you plan to use the trait only in your own code, or do not care about auto traits like `Send` on the `Future` - = note: `#[warn(async_fn_in_trait)]` on by default -help: you can alternatively desugar to a normal `fn` that returns `impl Future` and add any desired bounds such as `Send`, but these cannot be relaxed without a breaking API change - | -49 ~ fn write_storage_guid(&self) -> impl std::future::Future> + Send {async { -50 | let guid_str = self.guid().to_string(); - ... -63 | } -64 ~ } } - | - -warning: missing documentation for a method - --> storage/src/storage/async_common.rs:66:5 - | -66 | async fn a_init(&mut self, output_snd: &XvcOutputSender) -> Result { - | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -warning: use of `async fn` in public traits is discouraged as auto trait bounds cannot be specified - --> storage/src/storage/async_common.rs:66:5 - | -66 | async fn a_init(&mut self, output_snd: &XvcOutputSender) -> Result { - | ^^^^^ - | - = note: you can suppress this lint if you plan to use the trait only in your own code, or do not care about auto traits like `Send` on the `Future` -help: you can alternatively desugar to a normal `fn` that returns `impl Future` and add any desired bounds such as `Send`, but these cannot be relaxed without a breaking API change - | -66 ~ fn a_init(&mut self, output_snd: &XvcOutputSender) -> impl std::future::Future> + Send {async { -67 | let res_response = self.write_storage_guid().await; - ... -77 | } -78 ~ } } - | - -warning: missing documentation for a method - --> storage/src/storage/async_common.rs:80:5 - | -80 | / async fn a_list( -81 | | &self, -82 | | output: &XvcOutputSender, -83 | | xvc_root: &xvc_core::XvcRoot, -84 | | ) -> Result { - | |____________________________________^ - -warning: use of `async fn` in public traits is discouraged as auto trait bounds cannot be specified - --> storage/src/storage/async_common.rs:80:5 - | -80 | async fn a_list( - | ^^^^^ - | - = note: you can suppress this lint if you plan to use the trait only in your own code, or do not care about auto traits like `Send` on the `Future` -help: you can alternatively desugar to a normal `fn` that returns `impl Future` and add any desired bounds such as `Send`, but these cannot be relaxed without a breaking API change - | -80 ~ fn a_list( -81 | &self, -82 | output: &XvcOutputSender, -83 | xvc_root: &xvc_core::XvcRoot, -84 ~ ) -> impl std::future::Future> + Send {async { -85 | let credentials = self.credentials()?; - ... -130| } -131~ } } - | - -warning: missing documentation for a method - --> storage/src/storage/async_common.rs:133:5 - | -133 | / async fn a_send( -134 | | &self, -135 | | output_snd: &XvcOutputSender, -136 | | xvc_root: &xvc_core::XvcRoot, -137 | | paths: &[xvc_core::XvcCachePath], -138 | | _force: bool, -139 | | ) -> crate::Result { - | |__________________________________________________^ - -warning: use of `async fn` in public traits is discouraged as auto trait bounds cannot be specified - --> storage/src/storage/async_common.rs:133:5 - | -133 | async fn a_send( - | ^^^^^ - | - = note: you can suppress this lint if you plan to use the trait only in your own code, or do not care about auto traits like `Send` on the `Future` -help: you can alternatively desugar to a normal `fn` that returns `impl Future` and add any desired bounds such as `Send`, but these cannot be relaxed without a breaking API change - | -133 ~ fn a_send( -134 | &self, - ... -138 | _force: bool, -139 ~ ) -> impl std::future::Future> + Send {async { -140 | let mut copied_paths = Vec::::new(); - ... -172 | }) -173 ~ } } - | - -warning: missing documentation for a method - --> storage/src/storage/async_common.rs:175:5 - | -175 | / async fn a_receive( -176 | | &self, -177 | | output_snd: &XvcOutputSender, -178 | | paths: &[xvc_core::XvcCachePath], -179 | | _force: bool, -180 | | ) -> Result<(XvcStorageTempDir, XvcStorageReceiveEvent)> { - | |____________________________________________________________^ - -warning: use of `async fn` in public traits is discouraged as auto trait bounds cannot be specified - --> storage/src/storage/async_common.rs:175:5 - | -175 | async fn a_receive( - | ^^^^^ - | - = note: you can suppress this lint if you plan to use the trait only in your own code, or do not care about auto traits like `Send` on the `Future` -help: you can alternatively desugar to a normal `fn` that returns `impl Future` and add any desired bounds such as `Send`, but these cannot be relaxed without a breaking API change - | -175 ~ fn a_receive( -176 | &self, - ... -179 | _force: bool, -180 ~ ) -> impl std::future::Future> + Send {async { -181 | let mut copied_paths = Vec::::new(); - ... -217 | )) -218 ~ } } - | - -warning: missing documentation for a method - --> storage/src/storage/async_common.rs:220:5 - | -220 | / async fn a_delete( -221 | | &self, -222 | | output: &XvcOutputSender, -223 | | paths: &[XvcCachePath], -224 | | ) -> Result { - | |______________________________________^ - -warning: use of `async fn` in public traits is discouraged as auto trait bounds cannot be specified - --> storage/src/storage/async_common.rs:220:5 - | -220 | async fn a_delete( - | ^^^^^ - | - = note: you can suppress this lint if you plan to use the trait only in your own code, or do not care about auto traits like `Send` on the `Future` -help: you can alternatively desugar to a normal `fn` that returns `impl Future` and add any desired bounds such as `Send`, but these cannot be relaxed without a breaking API change - | -220 ~ fn a_delete( -221 | &self, -222 | output: &XvcOutputSender, -223 | paths: &[XvcCachePath], -224 ~ ) -> impl std::future::Future> + Send {async { -225 | let mut deleted_paths = Vec::::new(); - ... -240 | }) -241 ~ } } - | - -warning: missing documentation for a method - --> storage/src/storage/async_common.rs:243:5 - | -243 | / async fn a_share( -244 | | &self, -245 | | output: &XvcOutputSender, -246 | | path: &XvcCachePath, -247 | | duration: std::time::Duration, -248 | | ) -> Result { - | |_____________________________________________^ - -warning: use of `async fn` in public traits is discouraged as auto trait bounds cannot be specified - --> storage/src/storage/async_common.rs:243:5 - | -243 | async fn a_share( - | ^^^^^ - | - = note: you can suppress this lint if you plan to use the trait only in your own code, or do not care about auto traits like `Send` on the `Future` -help: you can alternatively desugar to a normal `fn` that returns `impl Future` and add any desired bounds such as `Send`, but these cannot be relaxed without a breaking API change - | -243 ~ fn a_share( -244 | &self, - ... -247 | duration: std::time::Duration, -248 ~ ) -> impl std::future::Future> + Send {async { -249 | let bucket = self.get_bucket()?; - ... -268 | }) -269 ~ } } - | - - Compiling xvc-file v0.6.9-alpha.1 (/Users/iex/github.com/iesahin/xvc/file) -warning: unused import: `HStore` - --> file/src/share/mod.rs:7:15 - | -7 | use xvc_ecs::{HStore, XvcStore}; - | ^^^^^^ - | - = note: `#[warn(unused_imports)]` on by default - -warning: missing documentation for a function - --> file/src/share/mod.rs:29:1 - | -29 | pub fn cmd_share(output_snd: &XvcOutputSender, xvc_root: &XvcRoot, opts: ShareCLI) -> Result<()> { - | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - | -note: the lint level is defined here - --> file/src/lib.rs:7:9 - | -7 | #![warn(missing_docs)] - | ^^^^^^^^^^^^ - -warning: `xvc-core` (lib) generated 1 warning - Compiling xvc-pipeline v0.6.9-alpha.1 (/Users/iex/github.com/iesahin/xvc/pipeline) - Compiling xvc v0.6.9-alpha.1 (/Users/iex/github.com/iesahin/xvc/lib) -warning: missing documentation for a function - --> lib/src/cli/mod.rs:212:1 - | -212 | pub fn dispatch_with_root(cli_opts: cli::XvcCLI, xvc_root_opt: XvcRootOpt) -> Result { - | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - | -note: the lint level is defined here - --> lib/src/lib.rs:1:9 - | -1 | #![warn(missing_docs)] - | ^^^^^^^^^^^^ - - Compiling xvc-workflow-tests v0.6.9-alpha.1 (/Users/iex/github.com/iesahin/xvc/workflow_tests) -warning: `xvc-storage` (lib) generated 37 warnings (run `cargo fix --lib -p xvc-storage` to apply 1 suggestion) -warning: `xvc-file` (lib) generated 2 warnings (run `cargo fix --lib -p xvc-file` to apply 1 suggestion) -warning: `xvc` (lib) generated 1 warning - Finished `release` profile [optimized] target(s) in 41.49s +[..] + Finished `release` profile [optimized] target(s) in 4.65s ``` ## Compiling Xvc without Reflink support diff --git a/config/Cargo.toml b/config/Cargo.toml index 863d91506..e4afb897b 100644 --- a/config/Cargo.toml +++ b/config/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-config" -version = "0.6.9-alpha.1" +version = "0.6.9-alpha.2" edition = "2021" description = "Xvc configuration management" authors = ["Emre Şahin "] @@ -16,8 +16,8 @@ name = "xvc_config" crate-type = ["rlib"] [dependencies] -xvc-logging = { version = "0.6.9-alpha.1", path = "../logging" } -xvc-walker = { version = "0.6.9-alpha.1", path = "../walker" } +xvc-logging = { version = "0.6.9-alpha.2", path = "../logging" } +xvc-walker = { version = "0.6.9-alpha.2", path = "../walker" } ## Cli and config diff --git a/core/Cargo.toml b/core/Cargo.toml index f7be40f6a..b9c54ea46 100644 --- a/core/Cargo.toml +++ b/core/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-core" -version = "0.6.9-alpha.1" +version = "0.6.9-alpha.2" edition = "2021" description = "Xvc core for common elements for all commands" authors = ["Emre Şahin "] @@ -16,10 +16,10 @@ name = "xvc_core" crate-type = ["rlib"] [dependencies] -xvc-config = { version = "0.6.9-alpha.1", path = "../config" } -xvc-logging = { version = "0.6.9-alpha.1", path = "../logging" } -xvc-ecs = { version = "0.6.9-alpha.1", path = "../ecs" } -xvc-walker = { version = "0.6.9-alpha.1", path = "../walker" } +xvc-config = { version = "0.6.9-alpha.2", path = "../config" } +xvc-logging = { version = "0.6.9-alpha.2", path = "../logging" } +xvc-ecs = { version = "0.6.9-alpha.2", path = "../ecs" } +xvc-walker = { version = "0.6.9-alpha.2", path = "../walker" } ## Cli and config clap = { version = "^4.4", features = ["derive"] } @@ -80,6 +80,6 @@ itertools = "^0.13" [dev-dependencies] -xvc-test-helper = { version = "0.6.9-alpha.1", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.9-alpha.2", path = "../test_helper/" } proptest = "^1.4" test-case = "^3.3" diff --git a/ecs/Cargo.toml b/ecs/Cargo.toml index 08e4d737c..4ce125745 100644 --- a/ecs/Cargo.toml +++ b/ecs/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-ecs" -version = "0.6.9-alpha.1" +version = "0.6.9-alpha.2" edition = "2021" description = "Entity-Component System for Xvc" authors = ["Emre Şahin "] @@ -16,7 +16,7 @@ name = "xvc_ecs" crate-type = ["rlib"] [dependencies] -xvc-logging = { version = "0.6.9-alpha.1", path = "../logging" } +xvc-logging = { version = "0.6.9-alpha.2", path = "../logging" } ## Serialization serde = { version = "^1.0", features = ["derive"] } diff --git a/file/Cargo.toml b/file/Cargo.toml index eced27c0a..4a4c0c854 100644 --- a/file/Cargo.toml +++ b/file/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-file" -version = "0.6.9-alpha.1" +version = "0.6.9-alpha.2" edition = "2021" description = "File tracking, versioning, upload and download functions for Xvc" authors = ["Emre Şahin "] @@ -21,12 +21,12 @@ test = true bench = true [dependencies] -xvc-logging = { version = "0.6.9-alpha.1", path = "../logging" } -xvc-config = { version = "0.6.9-alpha.1", path = "../config" } -xvc-core = { version = "0.6.9-alpha.1", path = "../core" } -xvc-ecs = { version = "0.6.9-alpha.1", path = "../ecs" } -xvc-walker = { version = "0.6.9-alpha.1", path = "../walker" } -xvc-storage = { version = "0.6.9-alpha.1", path = "../storage" } +xvc-logging = { version = "0.6.9-alpha.2", path = "../logging" } +xvc-config = { version = "0.6.9-alpha.2", path = "../config" } +xvc-core = { version = "0.6.9-alpha.2", path = "../core" } +xvc-ecs = { version = "0.6.9-alpha.2", path = "../ecs" } +xvc-walker = { version = "0.6.9-alpha.2", path = "../walker" } +xvc-storage = { version = "0.6.9-alpha.2", path = "../storage" } ## Cli and config @@ -94,5 +94,5 @@ default = ["reflink"] reflink = ["dep:reflink"] [dev-dependencies] -xvc-test-helper = { version = "0.6.9-alpha.1", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.9-alpha.2", path = "../test_helper/" } shellfn = "^0.1" diff --git a/lib/Cargo.toml b/lib/Cargo.toml index 25d1f63b2..4accd7387 100644 --- a/lib/Cargo.toml +++ b/lib/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc" -version = "0.6.9-alpha.1" +version = "0.6.9-alpha.2" edition = "2021" description = "An MLOps tool to manage data files and pipelines on top of Git" authors = ["Emre Şahin "] @@ -20,14 +20,14 @@ name = "xvc" path = "src/main.rs" [dependencies] -xvc-config = { version = "0.6.9-alpha.1", path = "../config" } -xvc-core = { version = "0.6.9-alpha.1", path = "../core" } -xvc-logging = { version = "0.6.9-alpha.1", path = "../logging" } -xvc-ecs = { version = "0.6.9-alpha.1", path = "../ecs" } -xvc-file = { version = "0.6.9-alpha.1", path = "../file" } -xvc-pipeline = { version = "0.6.9-alpha.1", path = "../pipeline" } -xvc-walker = { version = "0.6.9-alpha.1", path = "../walker" } -xvc-storage = { version = "0.6.9-alpha.1", path = "../storage" } +xvc-config = { version = "0.6.9-alpha.2", path = "../config" } +xvc-core = { version = "0.6.9-alpha.2", path = "../core" } +xvc-logging = { version = "0.6.9-alpha.2", path = "../logging" } +xvc-ecs = { version = "0.6.9-alpha.2", path = "../ecs" } +xvc-file = { version = "0.6.9-alpha.2", path = "../file" } +xvc-pipeline = { version = "0.6.9-alpha.2", path = "../pipeline" } +xvc-walker = { version = "0.6.9-alpha.2", path = "../walker" } +xvc-storage = { version = "0.6.9-alpha.2", path = "../storage" } ## Cli and config @@ -136,4 +136,4 @@ shellfn = "^0.1" test-case = "^3.3" trycmd = "^0.15" which = "^6.0" -xvc-test-helper = { version = "0.6.9-alpha.1", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.9-alpha.2", path = "../test_helper/" } diff --git a/logging/Cargo.toml b/logging/Cargo.toml index 64421dfcf..b4a921875 100644 --- a/logging/Cargo.toml +++ b/logging/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-logging" -version = "0.6.9-alpha.1" +version = "0.6.9-alpha.2" edition = "2021" description = "Logging crate for Xvc" authors = ["Emre Şahin "] diff --git a/pipeline/Cargo.toml b/pipeline/Cargo.toml index 7a5c5db3f..5940cd09b 100644 --- a/pipeline/Cargo.toml +++ b/pipeline/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-pipeline" -version = "0.6.9-alpha.1" +version = "0.6.9-alpha.2" edition = "2021" description = "Xvc data pipeline management" authors = ["Emre Şahin "] @@ -19,12 +19,12 @@ default = [] bundled-sqlite = ["rusqlite/bundled"] [dependencies] -xvc-config = { version = "0.6.9-alpha.1", path = "../config" } -xvc-core = { version = "0.6.9-alpha.1", path = "../core" } -xvc-ecs = { version = "0.6.9-alpha.1", path = "../ecs" } -xvc-logging = { version = "0.6.9-alpha.1", path = "../logging" } -xvc-walker = { version = "0.6.9-alpha.1", path = "../walker" } -xvc-file = { version = "0.6.9-alpha.1", path = "../file" } +xvc-config = { version = "0.6.9-alpha.2", path = "../config" } +xvc-core = { version = "0.6.9-alpha.2", path = "../core" } +xvc-ecs = { version = "0.6.9-alpha.2", path = "../ecs" } +xvc-logging = { version = "0.6.9-alpha.2", path = "../logging" } +xvc-walker = { version = "0.6.9-alpha.2", path = "../walker" } +xvc-file = { version = "0.6.9-alpha.2", path = "../file" } ## Cli and config clap = { version = "^4.4", features = ["derive"] } @@ -98,5 +98,5 @@ itertools = "^0.13" derive_more = "^0.99" [dev-dependencies] -xvc-test-helper = { version = "0.6.9-alpha.1", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.9-alpha.2", path = "../test_helper/" } test-case = "^3.3" diff --git a/storage/Cargo.toml b/storage/Cargo.toml index 47f90ad99..b64f20762 100644 --- a/storage/Cargo.toml +++ b/storage/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-storage" -version = "0.6.9-alpha.1" +version = "0.6.9-alpha.2" edition = "2021" description = "Xvc remote and local storage management" authors = ["Emre Şahin "] @@ -16,11 +16,11 @@ name = "xvc_storage" crate-type = ["rlib"] [dependencies] -xvc-logging = { version = "0.6.9-alpha.1", path = "../logging" } -xvc-config = { version = "0.6.9-alpha.1", path = "../config" } -xvc-core = { version = "0.6.9-alpha.1", path = "../core" } -xvc-ecs = { version = "0.6.9-alpha.1", path = "../ecs" } -xvc-walker = { version = "0.6.9-alpha.1", path = "../walker" } +xvc-logging = { version = "0.6.9-alpha.2", path = "../logging" } +xvc-config = { version = "0.6.9-alpha.2", path = "../config" } +xvc-core = { version = "0.6.9-alpha.2", path = "../core" } +xvc-ecs = { version = "0.6.9-alpha.2", path = "../ecs" } +xvc-walker = { version = "0.6.9-alpha.2", path = "../walker" } ## Cli and config clap = { version = "^4.4", features = ["derive"] } @@ -105,7 +105,7 @@ digital-ocean = ["async"] [dev-dependencies] -xvc-test-helper = { version = "0.6.9-alpha.1", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.9-alpha.2", path = "../test_helper/" } shellfn = "^0.1" [package.metadata.cargo-udeps.ignore] diff --git a/test_helper/Cargo.toml b/test_helper/Cargo.toml index 8c1474c93..cbbb6d87e 100644 --- a/test_helper/Cargo.toml +++ b/test_helper/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-test-helper" -version = "0.6.9-alpha.1" +version = "0.6.9-alpha.2" edition = "2021" description = "Unit test helper functions for Xvc" authors = ["Emre Şahin "] @@ -20,7 +20,7 @@ path = "src/main.rs" # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html [dependencies] -xvc-logging = { version = "0.6.9-alpha.1", path = "../logging/" } +xvc-logging = { version = "0.6.9-alpha.2", path = "../logging/" } rand = "^0.8" log = "^0.4" diff --git a/walker/Cargo.toml b/walker/Cargo.toml index 82b134707..13ae57dee 100644 --- a/walker/Cargo.toml +++ b/walker/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-walker" -version = "0.6.9-alpha.1" +version = "0.6.9-alpha.2" edition = "2021" description = "Xvc parallel file system walker with ignore features" authors = ["Emre Şahin "] @@ -16,7 +16,7 @@ name = "xvc_walker" crate-type = ["rlib"] [dependencies] -xvc-logging = { version = "0.6.9-alpha.1", path = "../logging" } +xvc-logging = { version = "0.6.9-alpha.2", path = "../logging" } globset = "^0.4" ## Parallelization @@ -38,7 +38,7 @@ itertools = "^0.13" regex = "^1.10" [dev-dependencies] -xvc-test-helper = { path = "../test_helper/", version = "0.6.9-alpha.1" } +xvc-test-helper = { path = "../test_helper/", version = "0.6.9-alpha.2" } test-case = "^3.3" [package.metadata.cargo-udeps.ignore] From e719133d65f9f0220187e38423f2505bb9951933 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sun, 28 Jul 2024 20:37:27 +0300 Subject: [PATCH 107/257] fix import errors --- lib/tests/test_file_recheck_serial.rs | 3 +-- lib/tests/test_init_3.rs | 1 - lib/tests/test_pipeline.rs | 2 +- lib/tests/test_pipeline_command_process.rs | 2 +- 4 files changed, 3 insertions(+), 5 deletions(-) diff --git a/lib/tests/test_file_recheck_serial.rs b/lib/tests/test_file_recheck_serial.rs index fa7b758a9..29a057bc5 100644 --- a/lib/tests/test_file_recheck_serial.rs +++ b/lib/tests/test_file_recheck_serial.rs @@ -4,11 +4,10 @@ use std::{fs, path::PathBuf}; use crate::common::run_in_temp_xvc_dir; use log::LevelFilter; -use xvc::error::Result; +use xvc::{error::Result, watch}; use xvc_config::XvcVerbosity; use xvc_core::XvcRoot; use xvc_test_helper::{create_directory_tree, generate_filled_file}; -use xvc_tests::watch; fn create_directory_hierarchy() -> Result { let temp_dir: XvcRoot = run_in_temp_xvc_dir()?; diff --git a/lib/tests/test_init_3.rs b/lib/tests/test_init_3.rs index 14bca87ff..5300d538b 100644 --- a/lib/tests/test_init_3.rs +++ b/lib/tests/test_init_3.rs @@ -5,7 +5,6 @@ use std::fs; use xvc::init::InitCLI; use xvc::error::Result; -use xvc_tests::watch; use common::*; diff --git a/lib/tests/test_pipeline.rs b/lib/tests/test_pipeline.rs index 23416f0ac..a443df5d7 100644 --- a/lib/tests/test_pipeline.rs +++ b/lib/tests/test_pipeline.rs @@ -3,7 +3,7 @@ mod common; use common::*; use xvc::error::Result; use xvc_config::XvcVerbosity; -use xvc_tests::watch; +use xvc::watch; #[test] fn test_pipeline() -> Result<()> { diff --git a/lib/tests/test_pipeline_command_process.rs b/lib/tests/test_pipeline_command_process.rs index 11fed0ad4..7e5e86856 100644 --- a/lib/tests/test_pipeline_command_process.rs +++ b/lib/tests/test_pipeline_command_process.rs @@ -8,7 +8,7 @@ use xvc_pipeline::{CommandProcess, XvcStep, XvcStepCommand}; use xvc::error::Result; use xvc_test_helper::{create_directory_tree, test_logging}; -use xvc_tests::watch; +use xvc::watch; #[test] fn test_pipeline_command_process() -> Result<()> { From 281f7edd919313e2f1b1db7c62b24293c7996ac8 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sun, 28 Jul 2024 20:47:48 +0300 Subject: [PATCH 108/257] fix warning --- core/src/types/hashalgorithm.rs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/core/src/types/hashalgorithm.rs b/core/src/types/hashalgorithm.rs index ab4c174f9..18dfbd1fa 100644 --- a/core/src/types/hashalgorithm.rs +++ b/core/src/types/hashalgorithm.rs @@ -1,6 +1,6 @@ //! Available hash algorithms in Xvc use serde::{Deserialize, Serialize}; -use strum_macros::{Display as EnumDisplay, EnumString, EnumVariantNames}; +use strum_macros::{Display as EnumDisplay, EnumString, VariantNames}; use xvc_config::{conf, FromConfigKey}; /// The available content hash algorithms to get content addresses. @@ -24,7 +24,7 @@ use xvc_config::{conf, FromConfigKey}; Deserialize, EnumString, EnumDisplay, - EnumVariantNames, + VariantNames, )] pub enum HashAlgorithm { /// Do nothing. From 2bd4ea7382ad6324d2d3f2e2833bb5f64cfa3e8a Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sun, 28 Jul 2024 20:47:59 +0300 Subject: [PATCH 109/257] fix warnings --- file/src/share/mod.rs | 2 +- lib/src/cli/mod.rs | 2 ++ lib/tests/test_init_3.rs | 1 + lib/tests/test_pipeline_run.rs | 2 +- 4 files changed, 5 insertions(+), 2 deletions(-) diff --git a/file/src/share/mod.rs b/file/src/share/mod.rs index 281b6582a..4441219bb 100644 --- a/file/src/share/mod.rs +++ b/file/src/share/mod.rs @@ -4,7 +4,7 @@ use crate::{common::load_targets_from_store, error, Result}; use clap::{command, Parser}; use humantime; use xvc_core::{ContentDigest, XvcCachePath, XvcFileType, XvcMetadata, XvcRoot}; -use xvc_ecs::{HStore, XvcStore}; +use xvc_ecs::{XvcStore}; use xvc_logging::{uwo, watch, XvcOutputSender}; use xvc_storage::{storage::get_storage_record, StorageIdentifier, XvcStorageOperations}; diff --git a/lib/src/cli/mod.rs b/lib/src/cli/mod.rs index c7bcf9fca..2457964bc 100644 --- a/lib/src/cli/mod.rs +++ b/lib/src/cli/mod.rs @@ -209,6 +209,8 @@ pub fn run(args: &[&str]) -> Result { dispatch(cli_options) } +/// Run the supplied command within the optional [XvcRoot]. If xvc_root is None, it will be tried +/// to be loaded from `cli_opts.workdir`. pub fn dispatch_with_root(cli_opts: cli::XvcCLI, xvc_root_opt: XvcRootOpt) -> Result { // XvcRoot should be kept per repository and shouldn't change directory across runs diff --git a/lib/tests/test_init_3.rs b/lib/tests/test_init_3.rs index 5300d538b..78c6580d5 100644 --- a/lib/tests/test_init_3.rs +++ b/lib/tests/test_init_3.rs @@ -7,6 +7,7 @@ use xvc::init::InitCLI; use xvc::error::Result; use common::*; +use xvc::watch; // This tests the preexisting .gitignore rules bug // https://github.com/iesahin/xvc/issues/119 diff --git a/lib/tests/test_pipeline_run.rs b/lib/tests/test_pipeline_run.rs index 2dd34246f..e596718fb 100644 --- a/lib/tests/test_pipeline_run.rs +++ b/lib/tests/test_pipeline_run.rs @@ -7,7 +7,7 @@ use xvc_config::XvcVerbosity; use xvc::error::Result; -use xvc_tests::watch; +use xvc::watch; #[test] fn test_pipeline_run() -> Result<()> { From 5a70d9d6384b358e6e6dcb1c420f4d48377342b2 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sun, 28 Jul 2024 20:53:05 +0300 Subject: [PATCH 110/257] update --- file/src/recheck/mod.rs | 4 ++-- file/src/share/mod.rs | 4 +++- file/src/untrack/mod.rs | 2 +- lib/src/cli/mod.rs | 10 ++++------ 4 files changed, 10 insertions(+), 10 deletions(-) diff --git a/file/src/recheck/mod.rs b/file/src/recheck/mod.rs index 3c0e2e273..37d9c07b2 100644 --- a/file/src/recheck/mod.rs +++ b/file/src/recheck/mod.rs @@ -145,12 +145,12 @@ pub fn cmd_recheck( watch!(content_digest_diff.get(xe)); if content_digest_diff.contains_key(xe) && matches!( - content_digest_diff[&xe], + content_digest_diff[xe], Diff::::Different { .. } ) { let output_snd = output_snd.clone(); - let xp = &stored_xvc_path_store[&xe]; + let xp = &stored_xvc_path_store[xe]; error!( output_snd, "{} has changed on disk. Either carry in, force, or delete the target to recheck. ", diff --git a/file/src/share/mod.rs b/file/src/share/mod.rs index 4441219bb..8079b32e9 100644 --- a/file/src/share/mod.rs +++ b/file/src/share/mod.rs @@ -4,7 +4,7 @@ use crate::{common::load_targets_from_store, error, Result}; use clap::{command, Parser}; use humantime; use xvc_core::{ContentDigest, XvcCachePath, XvcFileType, XvcMetadata, XvcRoot}; -use xvc_ecs::{XvcStore}; +use xvc_ecs::XvcStore; use xvc_logging::{uwo, watch, XvcOutputSender}; use xvc_storage::{storage::get_storage_record, StorageIdentifier, XvcStorageOperations}; @@ -26,6 +26,8 @@ pub struct ShareCLI { target: String, } +/// Handler function for `xvc share` command. Runs the command with `opts` within `xvc_root` and +/// sends output to `output_snd`. pub fn cmd_share(output_snd: &XvcOutputSender, xvc_root: &XvcRoot, opts: ShareCLI) -> Result<()> { // TODO: TIDY UP these implementation to reuse code in other places let storage = get_storage_record(output_snd, xvc_root, &opts.storage)?; diff --git a/file/src/untrack/mod.rs b/file/src/untrack/mod.rs index 12505a190..0e7c45a66 100644 --- a/file/src/untrack/mod.rs +++ b/file/src/untrack/mod.rs @@ -67,7 +67,7 @@ pub fn cmd_untrack( for (xe, xvc_path) in untrack_targets.iter() { let abs_path = xvc_path.to_absolute_path(xvc_root); if !abs_path.symlink_metadata().unwrap().is_file() { - let content_digest = all_content_digests[&xe]; + let content_digest = all_content_digests[xe]; let recheck_method = RecheckMethod::Copy; uwr!( recheck_op_snd.send(Some(RecheckOperation::Recheck { diff --git a/lib/src/cli/mod.rs b/lib/src/cli/mod.rs index 2457964bc..3b82af6f1 100644 --- a/lib/src/cli/mod.rs +++ b/lib/src/cli/mod.rs @@ -471,19 +471,18 @@ fn get_xvc_config_params(cli_opts: &XvcCLI) -> XvcConfigParams { } fn get_term_log_level(verbosity: XvcVerbosity) -> LevelFilter { - let term_log_level = match verbosity { + match verbosity { XvcVerbosity::Quiet => LevelFilter::Off, XvcVerbosity::Default => LevelFilter::Error, XvcVerbosity::Warn => LevelFilter::Warn, XvcVerbosity::Info => LevelFilter::Info, XvcVerbosity::Debug => LevelFilter::Debug, XvcVerbosity::Trace => LevelFilter::Trace, - }; - term_log_level + } } fn get_verbosity(cli_opts: &XvcCLI) -> XvcVerbosity { - let verbosity = if cli_opts.quiet { + if cli_opts.quiet { XvcVerbosity::Quiet } else { match cli_opts.verbosity { @@ -493,6 +492,5 @@ fn get_verbosity(cli_opts: &XvcCLI) -> XvcVerbosity { 3 => XvcVerbosity::Debug, _ => XvcVerbosity::Trace, } - }; - verbosity + } } From 8494bdf6c8c297c51a6e70e520d6b0bde997427e Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sun, 28 Jul 2024 20:56:41 +0300 Subject: [PATCH 111/257] fix warnings --- lib/tests/test_storage_new_digital_ocean.rs | 2 +- lib/tests/test_storage_new_gcs.rs | 2 +- lib/tests/test_storage_new_r2.rs | 2 +- lib/tests/test_storage_new_rsync.rs | 4 ++-- 4 files changed, 5 insertions(+), 5 deletions(-) diff --git a/lib/tests/test_storage_new_digital_ocean.rs b/lib/tests/test_storage_new_digital_ocean.rs index 3d24ab985..167080481 100644 --- a/lib/tests/test_storage_new_digital_ocean.rs +++ b/lib/tests/test_storage_new_digital_ocean.rs @@ -165,7 +165,7 @@ fn test_storage_new_digital_ocean() -> Result<()> { "--storage-prefix", &storage_prefix, "--region", - ®ion, + region, ])?; watch!(out); diff --git a/lib/tests/test_storage_new_gcs.rs b/lib/tests/test_storage_new_gcs.rs index 39de6be73..44eadddca 100644 --- a/lib/tests/test_storage_new_gcs.rs +++ b/lib/tests/test_storage_new_gcs.rs @@ -165,7 +165,7 @@ fn test_storage_new_gcs() -> Result<()> { "--storage-prefix", &storage_prefix, "--region", - ®ion, + region, ])?; watch!(out); diff --git a/lib/tests/test_storage_new_r2.rs b/lib/tests/test_storage_new_r2.rs index 3a81a998c..bb852da65 100644 --- a/lib/tests/test_storage_new_r2.rs +++ b/lib/tests/test_storage_new_r2.rs @@ -170,7 +170,7 @@ fn test_storage_new_r2() -> Result<()> { "--storage-prefix", &storage_prefix, "--account-id", - &account_id, + account_id, ])?; watch!(out); diff --git a/lib/tests/test_storage_new_rsync.rs b/lib/tests/test_storage_new_rsync.rs index 6d57da0a9..0786cd93d 100644 --- a/lib/tests/test_storage_new_rsync.rs +++ b/lib/tests/test_storage_new_rsync.rs @@ -57,9 +57,9 @@ fn test_storage_new_rsync() -> Result<()> { "--name", "rsync-storage", "--host", - &test_host, + test_host, "--user", - &test_user, + test_user, "--storage-dir", &storage_dir_name, ])?; From f6a7510809c56a43f27ce29b4194a7d320bf82b9 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sun, 28 Jul 2024 22:12:26 +0300 Subject: [PATCH 112/257] fix warnings --- core/src/check_ignore/mod.rs | 11 ++++++++--- lib/src/git.rs | 2 +- lib/tests/test_storage_new_wasabi.rs | 2 +- pipeline/src/pipeline/api/update.rs | 2 +- storage/src/storage/async_common.rs | 29 ++++++++++++++++++++++++---- 5 files changed, 36 insertions(+), 10 deletions(-) diff --git a/core/src/check_ignore/mod.rs b/core/src/check_ignore/mod.rs index d4cd002b1..ceb313c62 100644 --- a/core/src/check_ignore/mod.rs +++ b/core/src/check_ignore/mod.rs @@ -109,15 +109,20 @@ fn check_ignore_stdin( let conf = xvc_root.config(); let current_dir = conf.current_dir()?; let mut buffer = String::new(); - for line in input.lines().flatten() { - let xvc_path = XvcPath::new(xvc_root, current_dir, &PathBuf::from(line))?; + let lines_iter = input.lines(); + lines_iter.map_while(|line| { + if let Ok(line) = line{ + XvcPath::new(xvc_root, current_dir, &PathBuf::from(line)).ok() + } else { + None + }}).for_each(|xvc_path| { let absolute_path = xvc_path.to_absolute_path(xvc_root); let res = check_ignore_line(ignore_rules, &absolute_path, opts.non_matching); if !res.trim().is_empty() { output!(output_snd, "{}", res); } buffer.clear(); - } + }); Ok(()) } diff --git a/lib/src/git.rs b/lib/src/git.rs index 38cb29b39..2e739eeb3 100644 --- a/lib/src/git.rs +++ b/lib/src/git.rs @@ -228,7 +228,7 @@ pub fn git_auto_stage( let res_git_add = exec_git( git_command, xvc_root_str, - &["add", &xvc_dir_str, "*.gitignore", "*.xvcignore"], + &["add", xvc_dir_str, "*.gitignore", "*.xvcignore"], )?; debug!(output_snd, "Staging .xvc/ to git: {res_git_add}"); Ok(()) diff --git a/lib/tests/test_storage_new_wasabi.rs b/lib/tests/test_storage_new_wasabi.rs index 957ba9419..154e0ea51 100644 --- a/lib/tests/test_storage_new_wasabi.rs +++ b/lib/tests/test_storage_new_wasabi.rs @@ -167,7 +167,7 @@ fn test_storage_new_wasabi() -> Result<()> { "--storage-prefix", &storage_prefix, "--endpoint", - &endpoint, + endpoint, ])?; watch!(out); diff --git a/pipeline/src/pipeline/api/update.rs b/pipeline/src/pipeline/api/update.rs index 9e46f0f31..febaacb68 100644 --- a/pipeline/src/pipeline/api/update.rs +++ b/pipeline/src/pipeline/api/update.rs @@ -63,7 +63,7 @@ pub fn cmd_update(xvc_root: &XvcRoot, opts: UpdateCLI) -> Result<()> { ); } if let Some(newname) = &rename { - pipeline.name = newname.to_owned(); + pipeline.name.clone_from(newname); } rs.left.update(*pipeline_e, pipeline); diff --git a/storage/src/storage/async_common.rs b/storage/src/storage/async_common.rs index f2ee1f5f5..61f659d70 100644 --- a/storage/src/storage/async_common.rs +++ b/storage/src/storage/async_common.rs @@ -30,12 +30,21 @@ use super::XvcStorageSendEvent; use super::XvcStorageTempDir; use super::XVC_STORAGE_GUID_FILENAME; -pub trait XvcS3StorageOperations { +/// Operations for S3 compatible storage services. Each service implements functions in this trait +/// for xvc file send and xvc file bring commands to work with the common functions. +pub(crate) trait XvcS3StorageOperations { + /// Prefix within the storage bucket if you want to separate Xvc files from the rest of the + /// bucket. fn storage_prefix(&self) -> String; + /// GUID for the storage. This is generated when the storage is first initialized. fn guid(&self) -> &XvcStorageGuid; + /// Get the bucket for the storage fn get_bucket(&self) -> Result; + /// Get the credentials for the fn credentials(&self) -> Result; + /// Name of the bucket fn bucket_name(&self) -> String; + /// Build the storage path for the S3 compatible storage fn build_storage_path(&self, cache_path: &XvcCachePath) -> XvcStoragePath { XvcStoragePath::from(format!( "{}/{}/{}", @@ -45,7 +54,10 @@ pub trait XvcS3StorageOperations { )) } + /// Region of the bucket fn region(&self) -> String; + + /// Write GUID to the storage when first initializing the storage async fn write_storage_guid(&self) -> Result<()> { let guid_str = self.guid().to_string(); let guid_bytes = guid_str.as_bytes(); @@ -63,6 +75,7 @@ pub trait XvcS3StorageOperations { } } + /// Initialze the bucket as Xvc storage by adding a GUID async fn a_init(&mut self, output_snd: &XvcOutputSender) -> Result { let res_response = self.write_storage_guid().await; @@ -77,6 +90,7 @@ pub trait XvcS3StorageOperations { } } + /// List files in the S3 compatible storage async fn a_list( &self, output: &XvcOutputSender, @@ -130,6 +144,7 @@ pub trait XvcS3StorageOperations { } } + /// Send files to S3 compatible storage async fn a_send( &self, output_snd: &XvcOutputSender, @@ -172,6 +187,7 @@ pub trait XvcS3StorageOperations { }) } + /// Receive files from S3 compatible storage async fn a_receive( &self, output_snd: &XvcOutputSender, @@ -217,6 +233,7 @@ pub trait XvcS3StorageOperations { )) } + /// Delete files from S3 compatible storage async fn a_delete( &self, output: &XvcOutputSender, @@ -240,6 +257,7 @@ pub trait XvcS3StorageOperations { }) } + /// Share files from S3 compatible storage for a duration with a signed url async fn a_share( &self, output: &XvcOutputSender, @@ -270,7 +288,8 @@ pub trait XvcS3StorageOperations { } impl XvcStorageOperations for T { - fn init(&mut self, output: &XvcOutputSender, xvc_root: &XvcRoot) -> Result + // FIXME: Do we need xvc_root here? + fn init(&mut self, output: &XvcOutputSender, _xvc_root: &XvcRoot) -> Result where Self: Sized, { @@ -310,7 +329,8 @@ impl XvcStorageOperations for T { fn receive( &self, output: &XvcOutputSender, - xvc_root: &xvc_core::XvcRoot, + // FIXME: Do we need xvc_root here? + _xvc_root: &xvc_core::XvcRoot, paths: &[xvc_core::XvcCachePath], force: bool, ) -> crate::Result<(XvcStorageTempDir, XvcStorageReceiveEvent)> { @@ -324,7 +344,8 @@ impl XvcStorageOperations for T { fn delete( &self, output: &XvcOutputSender, - xvc_root: &xvc_core::XvcRoot, + // FIXME: Do we need xvc_root? + _xvc_root: &xvc_core::XvcRoot, paths: &[xvc_core::XvcCachePath], ) -> crate::Result { let rt = tokio::runtime::Builder::new_multi_thread() From b523cd0b51740990b00815f6b60364169c48bcad Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sun, 28 Jul 2024 22:21:05 +0300 Subject: [PATCH 113/257] fix some more warnings --- pipeline/src/pipeline/step.rs | 2 ++ storage/src/storage/async_common.rs | 4 +++- storage/src/storage/generic.rs | 8 ++++---- storage/src/storage/mod.rs | 3 +-- storage/src/storage/rsync.rs | 8 ++++---- test_helper/src/lib.rs | 1 + 6 files changed, 15 insertions(+), 11 deletions(-) diff --git a/pipeline/src/pipeline/step.rs b/pipeline/src/pipeline/step.rs index 629525bcb..174d35ccb 100644 --- a/pipeline/src/pipeline/step.rs +++ b/pipeline/src/pipeline/step.rs @@ -29,6 +29,8 @@ pub struct StepCLI { /// Step management subcommands #[derive(Debug, Clone, Parser)] #[command()] +// This is just a command description and used once +#[allow(clippy::large_enum_variant)] pub enum StepSubCommand { /// List steps in a pipeline #[command()] diff --git a/storage/src/storage/async_common.rs b/storage/src/storage/async_common.rs index 61f659d70..8e5066907 100644 --- a/storage/src/storage/async_common.rs +++ b/storage/src/storage/async_common.rs @@ -1,3 +1,4 @@ +//! Home for async operations for S3 compatible storage services. use std::fs; use std::str::FromStr; @@ -358,7 +359,8 @@ impl XvcStorageOperations for T { fn share( &self, output: &XvcOutputSender, - xvc_root: &xvc_core::XvcRoot, + // FIXME: Do we need xvc_root here? + _xvc_root: &xvc_core::XvcRoot, path: &XvcCachePath, duration: std::time::Duration, ) -> Result { diff --git a/storage/src/storage/generic.rs b/storage/src/storage/generic.rs index 65815d67f..0d291587f 100644 --- a/storage/src/storage/generic.rs +++ b/storage/src/storage/generic.rs @@ -445,10 +445,10 @@ impl XvcStorageOperations for XvcGenericStorage { fn share( &self, - output: &XvcOutputSender, - xvc_root: &XvcRoot, - path: &XvcCachePath, - period: std::time::Duration, + _output: &XvcOutputSender, + _xvc_root: &XvcRoot, + _path: &XvcCachePath, + _period: std::time::Duration, ) -> Result { Err(Error::StorageDoesNotSupportSignedUrls) } diff --git a/storage/src/storage/mod.rs b/storage/src/storage/mod.rs index 235a8722c..c2b145d70 100644 --- a/storage/src/storage/mod.rs +++ b/storage/src/storage/mod.rs @@ -1,7 +1,6 @@ //! Cloud storage implementations for xvc. #[cfg(feature = "async")] pub mod async_common; -pub mod common; #[cfg(feature = "digital-ocean")] pub mod digital_ocean; pub mod event; @@ -32,7 +31,7 @@ pub use local::XvcLocalStorage; use serde::{Deserialize, Serialize}; use tempfile::TempDir; use uuid::Uuid; -use xvc_logging::{error, panic, watch, XvcOutputSender}; +use xvc_logging::{error, watch, XvcOutputSender}; use xvc_walker::AbsolutePath; use crate::{Error, Result, StorageIdentifier}; diff --git a/storage/src/storage/rsync.rs b/storage/src/storage/rsync.rs index aaa054440..7299a6658 100644 --- a/storage/src/storage/rsync.rs +++ b/storage/src/storage/rsync.rs @@ -472,10 +472,10 @@ impl XvcStorageOperations for XvcRsyncStorage { fn share( &self, - output: &XvcOutputSender, - xvc_root: &XvcRoot, - path: &XvcCachePath, - period: std::time::Duration, + _output: &XvcOutputSender, + _xvc_root: &XvcRoot, + _path: &XvcCachePath, + _period: std::time::Duration, ) -> Result { Err(Error::StorageDoesNotSupportSignedUrls) } diff --git a/test_helper/src/lib.rs b/test_helper/src/lib.rs index 8e5c3bb75..62beaffa4 100755 --- a/test_helper/src/lib.rs +++ b/test_helper/src/lib.rs @@ -129,6 +129,7 @@ pub fn temp_git_dir() -> PathBuf { pub fn generate_random_file(filename: &Path, size: usize, seed: Option) { let f = OpenOptions::new() .create(true) + .truncate(true) .write(true) .open(filename) .unwrap(); From 35e3c6a419cddb60101b16f378fe6438470c051a Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sun, 28 Jul 2024 22:21:40 +0300 Subject: [PATCH 114/257] turn off wasabi and gcs warnings --- lib/Cargo.toml | 2 -- 1 file changed, 2 deletions(-) diff --git a/lib/Cargo.toml b/lib/Cargo.toml index 4accd7387..ae3ca8042 100644 --- a/lib/Cargo.toml +++ b/lib/Cargo.toml @@ -115,8 +115,6 @@ test-ci = [ "test-s3", "test-minio", "test-r2", - "test-gcs", - "test-wasabi", "test-digital-ocean", "test-rsync", "test-generic-rsync", From 2232b52039171b52898472ceb24eb13a78795f38 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sun, 28 Jul 2024 22:22:26 +0300 Subject: [PATCH 115/257] bump version --- config/Cargo.toml | 6 +++--- core/Cargo.toml | 12 ++++++------ ecs/Cargo.toml | 4 ++-- file/Cargo.toml | 16 ++++++++-------- lib/Cargo.toml | 20 ++++++++++---------- logging/Cargo.toml | 2 +- pipeline/Cargo.toml | 16 ++++++++-------- storage/Cargo.toml | 14 +++++++------- test_helper/Cargo.toml | 4 ++-- walker/Cargo.toml | 6 +++--- 10 files changed, 50 insertions(+), 50 deletions(-) diff --git a/config/Cargo.toml b/config/Cargo.toml index e4afb897b..ba02a766c 100644 --- a/config/Cargo.toml +++ b/config/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-config" -version = "0.6.9-alpha.2" +version = "0.6.9-alpha.3" edition = "2021" description = "Xvc configuration management" authors = ["Emre Şahin "] @@ -16,8 +16,8 @@ name = "xvc_config" crate-type = ["rlib"] [dependencies] -xvc-logging = { version = "0.6.9-alpha.2", path = "../logging" } -xvc-walker = { version = "0.6.9-alpha.2", path = "../walker" } +xvc-logging = { version = "0.6.9-alpha.3", path = "../logging" } +xvc-walker = { version = "0.6.9-alpha.3", path = "../walker" } ## Cli and config diff --git a/core/Cargo.toml b/core/Cargo.toml index b9c54ea46..eb74ea6ab 100644 --- a/core/Cargo.toml +++ b/core/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-core" -version = "0.6.9-alpha.2" +version = "0.6.9-alpha.3" edition = "2021" description = "Xvc core for common elements for all commands" authors = ["Emre Şahin "] @@ -16,10 +16,10 @@ name = "xvc_core" crate-type = ["rlib"] [dependencies] -xvc-config = { version = "0.6.9-alpha.2", path = "../config" } -xvc-logging = { version = "0.6.9-alpha.2", path = "../logging" } -xvc-ecs = { version = "0.6.9-alpha.2", path = "../ecs" } -xvc-walker = { version = "0.6.9-alpha.2", path = "../walker" } +xvc-config = { version = "0.6.9-alpha.3", path = "../config" } +xvc-logging = { version = "0.6.9-alpha.3", path = "../logging" } +xvc-ecs = { version = "0.6.9-alpha.3", path = "../ecs" } +xvc-walker = { version = "0.6.9-alpha.3", path = "../walker" } ## Cli and config clap = { version = "^4.4", features = ["derive"] } @@ -80,6 +80,6 @@ itertools = "^0.13" [dev-dependencies] -xvc-test-helper = { version = "0.6.9-alpha.2", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.9-alpha.3", path = "../test_helper/" } proptest = "^1.4" test-case = "^3.3" diff --git a/ecs/Cargo.toml b/ecs/Cargo.toml index 4ce125745..116b7c7d4 100644 --- a/ecs/Cargo.toml +++ b/ecs/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-ecs" -version = "0.6.9-alpha.2" +version = "0.6.9-alpha.3" edition = "2021" description = "Entity-Component System for Xvc" authors = ["Emre Şahin "] @@ -16,7 +16,7 @@ name = "xvc_ecs" crate-type = ["rlib"] [dependencies] -xvc-logging = { version = "0.6.9-alpha.2", path = "../logging" } +xvc-logging = { version = "0.6.9-alpha.3", path = "../logging" } ## Serialization serde = { version = "^1.0", features = ["derive"] } diff --git a/file/Cargo.toml b/file/Cargo.toml index 4a4c0c854..8828dce0d 100644 --- a/file/Cargo.toml +++ b/file/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-file" -version = "0.6.9-alpha.2" +version = "0.6.9-alpha.3" edition = "2021" description = "File tracking, versioning, upload and download functions for Xvc" authors = ["Emre Şahin "] @@ -21,12 +21,12 @@ test = true bench = true [dependencies] -xvc-logging = { version = "0.6.9-alpha.2", path = "../logging" } -xvc-config = { version = "0.6.9-alpha.2", path = "../config" } -xvc-core = { version = "0.6.9-alpha.2", path = "../core" } -xvc-ecs = { version = "0.6.9-alpha.2", path = "../ecs" } -xvc-walker = { version = "0.6.9-alpha.2", path = "../walker" } -xvc-storage = { version = "0.6.9-alpha.2", path = "../storage" } +xvc-logging = { version = "0.6.9-alpha.3", path = "../logging" } +xvc-config = { version = "0.6.9-alpha.3", path = "../config" } +xvc-core = { version = "0.6.9-alpha.3", path = "../core" } +xvc-ecs = { version = "0.6.9-alpha.3", path = "../ecs" } +xvc-walker = { version = "0.6.9-alpha.3", path = "../walker" } +xvc-storage = { version = "0.6.9-alpha.3", path = "../storage" } ## Cli and config @@ -94,5 +94,5 @@ default = ["reflink"] reflink = ["dep:reflink"] [dev-dependencies] -xvc-test-helper = { version = "0.6.9-alpha.2", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.9-alpha.3", path = "../test_helper/" } shellfn = "^0.1" diff --git a/lib/Cargo.toml b/lib/Cargo.toml index ae3ca8042..0afca62f9 100644 --- a/lib/Cargo.toml +++ b/lib/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc" -version = "0.6.9-alpha.2" +version = "0.6.9-alpha.3" edition = "2021" description = "An MLOps tool to manage data files and pipelines on top of Git" authors = ["Emre Şahin "] @@ -20,14 +20,14 @@ name = "xvc" path = "src/main.rs" [dependencies] -xvc-config = { version = "0.6.9-alpha.2", path = "../config" } -xvc-core = { version = "0.6.9-alpha.2", path = "../core" } -xvc-logging = { version = "0.6.9-alpha.2", path = "../logging" } -xvc-ecs = { version = "0.6.9-alpha.2", path = "../ecs" } -xvc-file = { version = "0.6.9-alpha.2", path = "../file" } -xvc-pipeline = { version = "0.6.9-alpha.2", path = "../pipeline" } -xvc-walker = { version = "0.6.9-alpha.2", path = "../walker" } -xvc-storage = { version = "0.6.9-alpha.2", path = "../storage" } +xvc-config = { version = "0.6.9-alpha.3", path = "../config" } +xvc-core = { version = "0.6.9-alpha.3", path = "../core" } +xvc-logging = { version = "0.6.9-alpha.3", path = "../logging" } +xvc-ecs = { version = "0.6.9-alpha.3", path = "../ecs" } +xvc-file = { version = "0.6.9-alpha.3", path = "../file" } +xvc-pipeline = { version = "0.6.9-alpha.3", path = "../pipeline" } +xvc-walker = { version = "0.6.9-alpha.3", path = "../walker" } +xvc-storage = { version = "0.6.9-alpha.3", path = "../storage" } ## Cli and config @@ -134,4 +134,4 @@ shellfn = "^0.1" test-case = "^3.3" trycmd = "^0.15" which = "^6.0" -xvc-test-helper = { version = "0.6.9-alpha.2", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.9-alpha.3", path = "../test_helper/" } diff --git a/logging/Cargo.toml b/logging/Cargo.toml index b4a921875..a36449109 100644 --- a/logging/Cargo.toml +++ b/logging/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-logging" -version = "0.6.9-alpha.2" +version = "0.6.9-alpha.3" edition = "2021" description = "Logging crate for Xvc" authors = ["Emre Şahin "] diff --git a/pipeline/Cargo.toml b/pipeline/Cargo.toml index 5940cd09b..18a42e8fa 100644 --- a/pipeline/Cargo.toml +++ b/pipeline/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-pipeline" -version = "0.6.9-alpha.2" +version = "0.6.9-alpha.3" edition = "2021" description = "Xvc data pipeline management" authors = ["Emre Şahin "] @@ -19,12 +19,12 @@ default = [] bundled-sqlite = ["rusqlite/bundled"] [dependencies] -xvc-config = { version = "0.6.9-alpha.2", path = "../config" } -xvc-core = { version = "0.6.9-alpha.2", path = "../core" } -xvc-ecs = { version = "0.6.9-alpha.2", path = "../ecs" } -xvc-logging = { version = "0.6.9-alpha.2", path = "../logging" } -xvc-walker = { version = "0.6.9-alpha.2", path = "../walker" } -xvc-file = { version = "0.6.9-alpha.2", path = "../file" } +xvc-config = { version = "0.6.9-alpha.3", path = "../config" } +xvc-core = { version = "0.6.9-alpha.3", path = "../core" } +xvc-ecs = { version = "0.6.9-alpha.3", path = "../ecs" } +xvc-logging = { version = "0.6.9-alpha.3", path = "../logging" } +xvc-walker = { version = "0.6.9-alpha.3", path = "../walker" } +xvc-file = { version = "0.6.9-alpha.3", path = "../file" } ## Cli and config clap = { version = "^4.4", features = ["derive"] } @@ -98,5 +98,5 @@ itertools = "^0.13" derive_more = "^0.99" [dev-dependencies] -xvc-test-helper = { version = "0.6.9-alpha.2", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.9-alpha.3", path = "../test_helper/" } test-case = "^3.3" diff --git a/storage/Cargo.toml b/storage/Cargo.toml index b64f20762..288f5d9e7 100644 --- a/storage/Cargo.toml +++ b/storage/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-storage" -version = "0.6.9-alpha.2" +version = "0.6.9-alpha.3" edition = "2021" description = "Xvc remote and local storage management" authors = ["Emre Şahin "] @@ -16,11 +16,11 @@ name = "xvc_storage" crate-type = ["rlib"] [dependencies] -xvc-logging = { version = "0.6.9-alpha.2", path = "../logging" } -xvc-config = { version = "0.6.9-alpha.2", path = "../config" } -xvc-core = { version = "0.6.9-alpha.2", path = "../core" } -xvc-ecs = { version = "0.6.9-alpha.2", path = "../ecs" } -xvc-walker = { version = "0.6.9-alpha.2", path = "../walker" } +xvc-logging = { version = "0.6.9-alpha.3", path = "../logging" } +xvc-config = { version = "0.6.9-alpha.3", path = "../config" } +xvc-core = { version = "0.6.9-alpha.3", path = "../core" } +xvc-ecs = { version = "0.6.9-alpha.3", path = "../ecs" } +xvc-walker = { version = "0.6.9-alpha.3", path = "../walker" } ## Cli and config clap = { version = "^4.4", features = ["derive"] } @@ -105,7 +105,7 @@ digital-ocean = ["async"] [dev-dependencies] -xvc-test-helper = { version = "0.6.9-alpha.2", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.9-alpha.3", path = "../test_helper/" } shellfn = "^0.1" [package.metadata.cargo-udeps.ignore] diff --git a/test_helper/Cargo.toml b/test_helper/Cargo.toml index cbbb6d87e..04647f463 100644 --- a/test_helper/Cargo.toml +++ b/test_helper/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-test-helper" -version = "0.6.9-alpha.2" +version = "0.6.9-alpha.3" edition = "2021" description = "Unit test helper functions for Xvc" authors = ["Emre Şahin "] @@ -20,7 +20,7 @@ path = "src/main.rs" # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html [dependencies] -xvc-logging = { version = "0.6.9-alpha.2", path = "../logging/" } +xvc-logging = { version = "0.6.9-alpha.3", path = "../logging/" } rand = "^0.8" log = "^0.4" diff --git a/walker/Cargo.toml b/walker/Cargo.toml index 13ae57dee..3390501c6 100644 --- a/walker/Cargo.toml +++ b/walker/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-walker" -version = "0.6.9-alpha.2" +version = "0.6.9-alpha.3" edition = "2021" description = "Xvc parallel file system walker with ignore features" authors = ["Emre Şahin "] @@ -16,7 +16,7 @@ name = "xvc_walker" crate-type = ["rlib"] [dependencies] -xvc-logging = { version = "0.6.9-alpha.2", path = "../logging" } +xvc-logging = { version = "0.6.9-alpha.3", path = "../logging" } globset = "^0.4" ## Parallelization @@ -38,7 +38,7 @@ itertools = "^0.13" regex = "^1.10" [dev-dependencies] -xvc-test-helper = { path = "../test_helper/", version = "0.6.9-alpha.2" } +xvc-test-helper = { path = "../test_helper/", version = "0.6.9-alpha.3" } test-case = "^3.3" [package.metadata.cargo-udeps.ignore] From a30cc3dd4a0f5fceca658eafdd57e927d22a653d Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sun, 28 Jul 2024 23:18:38 +0300 Subject: [PATCH 116/257] fix docs errors --- .../intro/compile-without-default-features.md | 2 +- book/src/start/from-dvc.md | 254 ++++++++++-------- book/src/start/ml.md | 2 +- 3 files changed, 141 insertions(+), 117 deletions(-) diff --git a/book/src/intro/compile-without-default-features.md b/book/src/intro/compile-without-default-features.md index cebb723a1..b762b7d9d 100644 --- a/book/src/intro/compile-without-default-features.md +++ b/book/src/intro/compile-without-default-features.md @@ -28,7 +28,7 @@ Note that when you supply `--no-default-features`, all other default features like `s3` etc are also turned off. You'll have to specify which [features] you want in the features list. Otherwise Xvc cannot connect to your storages. -``` +```bash cargo build --no-default-features --features "reflink,s3" --release [..] Finished `release` profile [optimized + debuginfo] target(s) in 56.40s diff --git a/book/src/start/from-dvc.md b/book/src/start/from-dvc.md index 3d66ff2ad..1e5a11d07 100644 --- a/book/src/start/from-dvc.md +++ b/book/src/start/from-dvc.md @@ -18,7 +18,8 @@ Git after the introduction of its experiment tracking features. Before that, Git was optional (but recommended) for DVC. Xvc has the same optional and recommended reliance on Git but all features are -available without Git. +available without Git. Xvc uses Git with its CLI interface like a user, without +any reliance on a particular library. Both of these commands use hashing the content to detect changes in files. @@ -26,172 +27,195 @@ Both of these use DAGs to represent pipelines. ## Conceptual Differences -- What DVC calls "remote", Xvc calls "storage." This is to emphasize the - difference between Xvc storages and Git remotes. - -- What DVC calls "stage" in a data pipeline, Xvc calls "step." "Stage" has a - different meaning in the Git context, and I believe using the same word in a - different meaning increases the mental effort to describe and understand. - -- In DVC, there is a 1-1 correspondence between `dvc.yaml` files in a - repository and the pipelines. In Xvc, pipelines are more abstract. They are - defined with [`xvc pipeline`](/ref/xvc-pipeline.md) family of commands. No - single file contains a pipeline definition. You can [export - pipelines](/ref/xvc-pipeline-export.md) to YAML, JSON, and TOML, and [import - them](/ref/xvc-pipeline-import.md) after making changes. Xvc doesn't consider - any file format authoritative for pipelines, and their YAML/JSON/TOML - representation may change between versions. - -- DVC is more liberal in creating files among user files in the repository. - When you add a file to DVC with `dvc add`, DVC creates a `.dvc` file next to - it. Xvc only creates a `.xvc/` directory in the repository root and only - updates `.gitignore` files to hide tracked files from Git. You won't see any - files added next to your data files. - -- Cache type, (or rather [recheck](/concepts/recheck.md) method) that is - whether a file in the repository is linked to its cached version by copying, - reflink, symlink or hardlink is determined repository-wide in DVC. You can - either have all your cache links as symlinks, or hardlinks, etc. Xvc tracks - these per file, you can have one file symlinked to the cache, another file - copied from the cache, etc. +**stage vs. step**: What DVC calls "stage" in a data pipeline, Xvc calls +"step." "Stage" has a different meaning in the Git context, and I believe using +the same word in a different meaning increases the mental effort to describe +and understand. + +**remmote vs storage**: What DVC calls "remote", Xvc calls "storage." This is +to emphasize the difference between Xvc storages and Git remotes. + +**pipeline definitions**: In DVC, there is a 1-1 correspondence between +`dvc.yaml` files in a repository and the pipelines. When you want to create a +new pipeline, you create a new file in DVC. + +In Xvc, pipelines are abstract. They are defined with [`xvc +pipeline`](/ref/xvc-pipeline.md) family of commands. No single file contains a +pipeline definition. You can [export pipelines](/ref/xvc-pipeline-export.md) to +YAML, JSON, and TOML, and [import them](/ref/xvc-pipeline-import.md) after +making changes. Xvc doesn't consider any file format authoritative for +pipelines, and their YAML/JSON/TOML representation may change between versions. + +**Files in the user workspace;** DVC is more liberal in creating files among +user files in the repository. When you add a file to DVC with `dvc add`, DVC +creates a `.dvc` file next to it. Xvc only creates a `.xvc/` directory in the +repository root and only updates `.gitignore` files to hide tracked files from +Git. You won't see any files added next to your data files. + +**cache-type vs recheck-method**: Cache type, (or rather +[recheck](/concepts/recheck.md) method) that is whether a file in the +repository is linked to its cached version by copying, reflink, symlink or +hardlink is determined repository-wide in DVC. You can either have all your +cache links as symlinks, or hardlinks, etc. Xvc tracks these per file, you can +have one file symlinked to the cache, another file copied from the cache, etc. ## Command Differences -❗Some of the Xvc commands described here are still in development. +```admonition warning +Some of the Xvc commands described here are still in development. +``` -- While naming Xvc commands, I tried our best to avoid name clashes with Git. - Having both `git push` and `dvc push` commands may look beneficial for - description at first, as these two are analogous. However, giving the same name - also hides important details that are more difficult to emphasize later. (e.g. - DVC experiments are _Git objects_ that are pushed to _Git remotes_, while the - files changed during experiments are pushed to _DVC remotes._) +While naming Xvc commands, we tried our best to avoid name clashes with Git. +Having both `git push` and `dvc push` commands may look beneficial for +understanding at first, as these two are analogous. However, giving the same name +also hides important details that are more difficult to emphasize later. (e.g. +DVC experiments are _Git objects_ that are pushed to _Git remotes_, while the +files changed during experiments are pushed to _DVC remotes._) -- `dvc add` can be replaced by [`xvc file track`](/ref/xvc-file-track.md). `dvc +`dvc add` can be replaced by [`xvc file track`](/ref/xvc-file-track.md). `dvc add` creates a `.dvc` file (formatted in YAML) in the repository. Xvc doesn't - create separate files for tracked paths. +create separate files for tracked paths. -- Instead of deleting `.dvc` files to remove a file from DVC, you can use [`xvc +Instead of deleting `.dvc` files to remove a file from DVC, you can use [`xvc file untrack`](/ref/xvc-file-untrack.md). It can also restore all versions of - an untracked file to a directory. +an untracked file to a directory. -- `dvc check-ignore` can be replaced by `xvc check-ignore`. Xvc version can be - used against any other ignore filename. (`.gitignore`,`.ignore`, - `.fooignore`...) +`dvc check-ignore` can be replaced by `xvc check-ignore`. Xvc version can be +used against any other ignore filename. (`.gitignore`,`.ignore`, +`.fooignore`...) -- `dvc checkout` is replaced by [`xvc file recheck`](/ref/xvc-file-recheck.md). - There is a `--recheck-method` (shortened as `--as`) option in several Xvc - commands to tell whether to check out as symlink, hardlink, reflink or copy. +`dvc checkout` is replaced by [`xvc file recheck`](/ref/xvc-file-recheck.md). +There is a `--recheck-method` (shortened as `--as`) option in several Xvc +commands to tell whether to check out as symlink, hardlink, reflink or copy. -- `dvc commit` is replaced by [`xvc file carry-in`](/ref/xvc-file-carry-in). +`dvc commit` is replaced by [`xvc file carry-in`](/ref/xvc-file-carry-in). They +both cache the files if they are changed. -- There is no command similar to `dvc config`. You can either edit the - [configuration files](/intro/configuration.md), or modify configuration with - `-c` options in each run. You can also supply all configuration from the - environment. See [Configuration](/intro/configuration.md). +There is no command similar to `dvc config`. You can either edit the +[configuration files](/intro/configuration.md), or modify configuration with +`-c` options in each run. You can also supply all configuration from the +environment. See [Configuration](/intro/configuration.md). -- `dvc dag` is replaced by `xvc pipeline dag`. DVC version uses ASCII art to - present the pipeline. Xvc doesn't provide ASCII art, instead provides either a - Graphviz representation or mermaid diagram. +`dvc dag` is replaced by `xvc pipeline dag`. DVC version uses ASCII art to +present the pipeline. Xvc doesn't provide ASCII art, instead provides either a +Graphviz representation or mermaid diagram. -- `dvc data status` and `dvc status` can be replaced by `xvc file list`. Xvc - version doesn't provide information about the pipelines, or remote storages. +`dvc data status` and `dvc status` can be replaced by `xvc file list`. Xvc +version doesn't provide information about the pipelines, or remote storages. -- There is no command similar to `dvc destroy` in Xvc. There will be an [`xvc -deinit`](/ref/xvc-deinit.md) command at some point. +There is no command similar to `dvc destroy` in Xvc. There will be an [`xvc +deinit`](/ref/xvc-deinit.md) command at some point. Until then, you can just +delete `.xvc/` directory and all `.xvcignore` files in your repository to +_destroy._ -- There is no command similar to `dvc diff` in Xvc. +There is no command similar to `dvc diff` in Xvc. -- There is no command similar to `dvc doctor` or `dvc version`. Version - information should be visible in the help text. Unless [compiled from source - with feature flags](/intro/install.md), Xvc binaries don't have feature - differences. +There is no command similar to `dvc doctor` or `dvc version`. Version +information should be visible in the help text. Unless [compiled from source +with feature flags](/intro/install.md), Xvc binaries don't have feature +differences. -- Currently, there are no commands corresponding to `dvc exp` set of commands. - This is on the roadmap for Xvc. Scope, implementation, and actual commands may - differ. +Currently, there are no commands corresponding to `dvc exp` set of commands. +This is on the roadmap for Xvc. Scope, implementation, and actual commands may +differ. -- `dvc fetch` is replaced by [`xvc file bring +`dvc fetch` is replaced by [`xvc file bring --no-recheck`](/ref/xvc-file-bring.md). -- Instead of freezing "pipeline stages" as in `dvc freeze`, and unfreezing with - `dvc unfreeze`, `xvc pipeline step update --changed +Instead of freezing "pipeline stages" as in `dvc freeze`, and unfreezing with +`dvc unfreeze`, `xvc pipeline step update --changed [never|always|by_dependencies]` can be used to specify if/when to run a - pipeline step. +pipeline step. -- Instead of `dvc gc` to "garbage-collect" files, you can use [`xvc file +Instead of `dvc gc` to "garbage-collect" files, you can use [`xvc file remove`](/ref/xvc-file-remove.md) with various options. -- There is no corresponding command for `dvc get-url` in Xvc. You can use - `wget` or `curl` instead. +There is no corresponding command for `dvc get-url` in Xvc. You can use +`wget` or `curl` instead. -- Currently there is no command to replace `dvc get` and `dvc import`, and `dvc +Currently there is no command to replace `dvc get` and `dvc import`, and `dvc import-url`. URL dependencies are supported in the pipeline with `xvc pipeline step dependency --url`. -- Instead of `dvc install` like hooks, Xvc issues Git commands itself if - `git.auto_commit` , `git.auto_stage` configuration options are set. +Instead of `dvc install` like hooks, Xvc issues Git commands itself if +`git.auto_commit` , `git.auto_stage` configuration options are set. -- There is no corresponding command for `dvc list-url`. +There is no corresponding command for `dvc list-url`. -- `dvc list` is replaced by [`xvc file list`](/ref/xvc-file-list.md) for local - paths. Its remote capabilities are not implemented but is on the roadmap. +`dvc list` is replaced by [`xvc file list`](/ref/xvc-file-list.md) for local +paths. Its remote capabilities are not implemented but is on the roadmap. -- Xvc doesn't mix files from different repositories in the same storage. There - is an ID for each Xvc repo that's also used in remote storage paths. +Xvc doesn't mix files from different repositories in the same storage. There +is an ID for each Xvc repo that's also used in remote storage paths. -- Currently, there is no params/metrics tracking/diff similar to `dvc params`, - `dvc metrics` or `dvc plots` commands in Xvc. +Currently, there is no params/metrics tracking/diff similar to `dvc params`, +`dvc metrics` or `dvc plots` commands in Xvc. -- `dvc move` is replaced by [`xvc file move`](/ref/xvc-file-move.md). +`dvc move` is replaced by [`xvc file move`](/ref/xvc-file-move.md). -- `dvc push` is replaced by [`xvc file send`](/ref/xvc-file-send.md). +`dvc push` is replaced by [`xvc file send`](/ref/xvc-file-send.md). -- `dvc pull` is replaced by [`xvc file bring`](/xvc-file-bring.md). +`dvc pull` is replaced by [`xvc file bring`](/xvc-file-bring.md). -- There are no commands similar to `dvc queue` for experiments in Xvc. - Experiment tracking will probably be handled differently. +There are no commands similar to `dvc queue` for experiments in Xvc. +Experiment tracking will probably be handled differently. -- `dvc remote` set of commands are replaced by `xvc storage` set of commands. - You can use `xvc storage new` for adding new storages. Currently, there is no - "default remote" facility in Xvc. Instead of `dvc remote modify`, you can use - `xvc storage remove` and `xvc storage new`. +`dvc remote` set of commands are replaced by `xvc storage` set of commands. +You can use `xvc storage new` for adding new storages. Currently, there is no +"default remote" facility in Xvc. Instead of `dvc remote modify`, you can use +`xvc storage remove` and `xvc storage new`. -- There is no single command to replace `dvc remove`. For files, you can use - [`xvc file delete`](/ref/xvc-file-delete.md). For pipelines steps, you can use - ][`xvc pipeline step remove`](/ref/xvc-pipeline-step-remove.md) +There is no single command to replace `dvc remove`. For files, you can use +[`xvc file delete`](/ref/xvc-file-delete.md). For pipelines steps, you can use +][`xvc pipeline step remove`](/ref/xvc-pipeline-step-remove.md) -- Instead of `dvc repro`, Xvc has [`xvc pipeline +Instead of `dvc repro`, Xvc has [`xvc pipeline run`](/ref/xvc-pipeline-run.md). If you want to reproduce a pipeline, you can - use `xvc pipeline run` again. +use `xvc pipeline run` again. -- `xvc root` is for the same purpose as `dvc root`. +`xvc root` is for the same purpose as `dvc root`. -- `dvc run` (that defines a stage in DVC pipeline and immediately runs it) can - be replaced by [`xvc pipeline`](/ref/xvc-pipeline.md) set of commands. [`xvc +`dvc run` (that defines a stage in DVC pipeline and immediately runs it) can +be replaced by [`xvc pipeline`](/ref/xvc-pipeline.md) set of commands. [`xvc pipeline new`](/ref/xvc-pipeline-new.md) for a new pipeline, [`xvc pipeline step new`](/ref/xvc-pipeline-step-new.md) for a new step in the pipeline, [`xvc pipeline step dependency`](/ref/xvc-pipeline-step-dependency.md) to specify - dependencies of a step, [`xvc pipeline step +dependencies of a step, [`xvc pipeline step output`](/ref/xvc-pipeline-step-output.md) to specify outputs of a step and - [`xvc pipeline run`](/ref/xvc-pipeline-run.md) to run this pipeline. +[`xvc pipeline run`](/ref/xvc-pipeline-run.md) to run this pipeline. -- Instead of `dvc stage add`, we have [`xvc pipeline step +Instead of `dvc stage add`, we have [`xvc pipeline step new`](/ref/xvc-pipeline-step-new.md). For `dvc stage list`, we have [`xvc pipeline step list`](/ref/xvc-pipeline-step-list.md). -- There is no (need) for `dvc protect` or `dvc unprotect` commands in Xvc. - "Cache type" of DVC is not a repository-wide option, and called ["recheck - method"](/concepts/recheck.md). If you want to track a certain directory as - symlink, and another as hardlink, you can do so with `xvc file recheck --as`. - If you want identical files copied to one directory _and_ linked in another, - [`xvc file copy`](/ref/xvc-file-copy.md) can help. +There is no (need) for `dvc protect` or `dvc unprotect` commands in Xvc. +"Cache type" of Xvc is not a repository-wide option, and called ["recheck +method"](/concepts/recheck.md). If you want to track a certain directory as +symlink, and another as hardlink, you can do so with `xvc file recheck --as`. +If you want identical files copied to one directory _and_ linked in another, +[`xvc file copy`](/ref/xvc-file-copy.md) can help. -- DVC needs `dvc update` for external dependencies in pipelines. Xvc checks - their metadata like any other dependency before downloading and invalidates the - step if the URL/file has changed automatically. +DVC needs `dvc update` for external dependencies in pipelines. Xvc checks +their metadata like any other dependency before downloading and invalidates the +step if the URL/file has changed automatically. -- DVC leaves Git operations to the user, and automates them to a certain degree - with Git hooks. Xvc adds Git commits to the repository after operations by - default. +DVC leaves Git operations to the user, and automates them to a certain degree +with Git hooks. Xvc adds Git commits to the repository after operations by +default. + +## Extra Features of Xvc + +Xvc can use multiple of hashing functions, like BLAKE3, BLAKE2s, SHA2-256 and +SHA3-256. More can be added upon request. The only requirement for hashes is +having 32-hex digits (256 bits) of output. + +In its pipelines, Xvc has more flexibility in defining dependencies. DVC +supports files, directories and hyperparameters. Xvc supports additionally + +- globs +- text file lines defined by line numbers, +- text file lines defined by regular expressions, +- URLs +- Sqlite queries, ## Technical Differences diff --git a/book/src/start/ml.md b/book/src/start/ml.md index efea1d4c4..7bc66042c 100644 --- a/book/src/start/ml.md +++ b/book/src/start/ml.md @@ -168,7 +168,7 @@ $ xvc pipeline step dependency --step-name test --glob 'models/*' --directory da ? 2 error: unexpected argument '--directory' found -Usage: xvc pipeline step dependency <--step-name |--generic |--url |--file |--step |--glob_items |--glob |--param |--regex_items |--regex |--line_items |--lines > +Usage: xvc pipeline step dependency <--step-name |--generic |--url |--file |--step |--glob_items |--glob |--param |--regex_items |--regex |--line_items |--lines |--sqlite-query > For more information, try '--help'. From eb973fb2a558f88ae39d7ac025286dd883bf39d9 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Mon, 29 Jul 2024 22:20:10 +0300 Subject: [PATCH 117/257] debug yaml --- pipeline/src/pipeline/deps/param.rs | 2 ++ 1 file changed, 2 insertions(+) diff --git a/pipeline/src/pipeline/deps/param.rs b/pipeline/src/pipeline/deps/param.rs index b5438ace1..8d6556f5c 100644 --- a/pipeline/src/pipeline/deps/param.rs +++ b/pipeline/src/pipeline/deps/param.rs @@ -272,9 +272,11 @@ impl XvcParamValue { /// Loads the key (in the form of a.b.c) from a YAML document fn parse_yaml(all_content: &str, key: &str) -> Result { let yaml_map: YamlValue = serde_yaml::from_str(all_content)?; + watch!(yaml_map); let nested_keys: Vec<&str> = key.split('.').collect(); let mut current_scope: YamlValue = yaml_map; for k in &nested_keys { + watch!(k); if let Some(current_value) = current_scope.get(*k) { match current_value { YamlValue::Mapping(_) => { From 50fdbdff38d7eec8600340db9c29a57cf81b8b8f Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Wed, 31 Jul 2024 13:14:26 +0300 Subject: [PATCH 118/257] Add sqlite_bundled feature to Windows build --- .github/workflows/rust.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/rust.yml b/.github/workflows/rust.yml index 05fe94b4c..d380a996c 100644 --- a/.github/workflows/rust.yml +++ b/.github/workflows/rust.yml @@ -199,7 +199,7 @@ jobs: with: use-cross: false command: build - args: --release --target ${{ matrix.target }} + args: --release --target ${{ matrix.target }} --features bundled_sqlite - name: Package shell: bash From d637f2eaf2e97dcb0b0e46e1f601e9b249789974 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Fri, 2 Aug 2024 10:59:37 +0300 Subject: [PATCH 119/257] Update pipeline output --- pipeline/src/pipeline/mod.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pipeline/src/pipeline/mod.rs b/pipeline/src/pipeline/mod.rs index 1952cf90f..ea9599733 100644 --- a/pipeline/src/pipeline/mod.rs +++ b/pipeline/src/pipeline/mod.rs @@ -1486,7 +1486,7 @@ fn s_running_f_wait_process<'a>( Some(exit_code) => match exit_code { ExitStatus::Exited(0) => { - output!(params.output_snd, "[DONE] {} ({})\n", step.name, step_command); + output!(params.output_snd, "[DONE] [{}] ({})\n", step.name, step_command); return_state = Some(s.process_completed_successfully()); } , From 998dc35008f96170803f2d208e97319836d3cab1 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sat, 3 Aug 2024 18:36:34 +0300 Subject: [PATCH 120/257] Fix spelling --- core/src/types/diff.rs | 17 ++++++++--------- pipeline/src/pipeline/deps/glob_items.rs | 4 +++- pipeline/src/pipeline/mod.rs | 20 ++++---------------- 3 files changed, 15 insertions(+), 26 deletions(-) diff --git a/core/src/types/diff.rs b/core/src/types/diff.rs index 72bb7a832..ea0224e2b 100644 --- a/core/src/types/diff.rs +++ b/core/src/types/diff.rs @@ -16,7 +16,7 @@ use xvc_logging::{warn, watch}; /// Shows which information is identical, missing or different in diff calculations. /// -/// We use this to compare anything that's storable. +/// We use this to compare anything that's Storable. #[derive(Debug, PartialEq, Eq, PartialOrd, Ord, Serialize, Deserialize, Hash, Clone, Default)] #[serde(bound = "T: Serialize, for<'lt> T: Deserialize<'lt>")] pub enum Diff { @@ -39,9 +39,8 @@ pub enum Diff { /// The value found in workspace actual: T, }, - /// We skipped this comparison. - /// It's not an error, but it means we didn't compare this field. - /// It may be shortcutted, we don't care or irrelevant. + /// We skipped this comparison. It's not an error, but it means we didn't compare this field. + /// It may be shortcut, we don't care or irrelevant. #[default] Skipped, } @@ -195,7 +194,7 @@ pub fn apply_diff( impl Diff { /// Return true if the diff is not [Diff::Identical] or [Diff::Skipped]. - /// This is used to find out if has changed. + /// This is used to find out if T is changed. pub fn changed(&self) -> bool { match self { Diff::Identical => false, @@ -279,7 +278,7 @@ impl DiffStore4 } } -/// Used to find out if record and actual are different for type T. +/// Used to find out if record and actual are different for type T pub trait Diffable { /// The type of the entity to compare. type Item: Storable; @@ -289,7 +288,7 @@ pub trait Diffable { /// compare the content. ) /// /// This is to convert optional entities to diffs. - /// e.g. a file may be missing from the disk, but it may exist in the records. + /// For example, a file may be missing from the disk, but it may exist in the records. /// ((Some(record), None) -> Diff::ActualMissing) fn diff(record: Option<&Self::Item>, actual: Option<&Self::Item>) -> Diff { watch!(record); @@ -316,7 +315,7 @@ pub trait Diffable { } /// This is to compare two entities with a quick comparison. - /// e.g. metadata of a file, timestamp of a URL etc. + /// Example: metadata of a file, timestamp of a URL etc. /// You may need to update actual's metadata or timestamp before calling this. fn diff_superficial(record: &Self::Item, actual: &Self::Item) -> Diff { if record == actual { @@ -330,7 +329,7 @@ pub trait Diffable { } /// This is to calculate two entities with a thorough comparison. - /// e.g. content of a file, content of a URL etc. + /// Example: content of a file, content of a URL etc. /// You may need to update actual's content before calling this. fn diff_thorough(record: &Self::Item, actual: &Self::Item) -> Diff { Self::diff_superficial(record, actual) diff --git a/pipeline/src/pipeline/deps/glob_items.rs b/pipeline/src/pipeline/deps/glob_items.rs index 14831d5f3..ab7e9361e 100644 --- a/pipeline/src/pipeline/deps/glob_items.rs +++ b/pipeline/src/pipeline/deps/glob_items.rs @@ -47,6 +47,7 @@ impl GlobItemsDep { ) -> Result { let xvc_path_metadata_map = glob_paths(pmp, glob_root, &glob).map(|paths| paths.into_iter().collect())?; + watch!(xvc_path_metadata_map); // We don't calculate the content digest map immediately, we only do that in through comparison Ok(GlobItemsDep { glob, @@ -79,7 +80,7 @@ impl GlobItemsDep { algorithm: HashAlgorithm, ) -> Result { let mut xvc_path_content_digest_map = BTreeMap::new(); - + watch!(xvc_path_metadata_map); for (xvc_path, xvc_metadata) in self.xvc_path_metadata_map.iter() { let record_metadata = record.xvc_path_metadata_map.get(xvc_path); let content_digest = if XvcMetadata::diff(record_metadata, Some(xvc_metadata)).changed() @@ -92,6 +93,7 @@ impl GlobItemsDep { xvc_path_content_digest_map.insert(xvc_path.clone(), content_digest); } + watch!(xvc_path_content_digest_map); Ok(Self { xvc_path_content_digest_map, ..self diff --git a/pipeline/src/pipeline/mod.rs b/pipeline/src/pipeline/mod.rs index ea9599733..36c7c5263 100644 --- a/pipeline/src/pipeline/mod.rs +++ b/pipeline/src/pipeline/mod.rs @@ -418,10 +418,6 @@ pub fn the_grand_pipeline_loop( .try_into()?; let default_step_timeout: u64 = 10000; let terminate_on_timeout = true; - let _step_timeouts: HStore = pipeline_steps - .keys() - .map(|step_e| (*step_e, Duration::from_secs(default_step_timeout))) - .collect(); let step_commands = xvc_root.load_store::()?; @@ -607,14 +603,6 @@ fn step_state_bulletin( } fn step_state_handler(step_e: XvcEntity, params: StepThreadParams) -> Result<()> { - // We check all other steps states in Select. - // If we only block on this step's dependencies, two parallel steps will block each other forever. - let _other_steps: Vec = params - .steps - .iter() - .filter_map(|(e, _)| if *e != step_e { Some(*e) } else { None }) - .collect(); - let step_state_sender = params.state_sender; let current_states = params.current_states.clone(); let mut step_state = XvcStepState::begin(); @@ -787,9 +775,9 @@ fn s_no_need_to_run_f_run_never<'a>( s: &DoneWithoutRunningState, params: StepStateParams<'a>, ) -> StateTransition<'a> { - info!( + output!( params.output_snd, - "Step {} has run_never set to true. Skipping.", params.step.name + "[NEVER] [{}]", params.step.name ); Ok((s.keep_done(), params)) } @@ -798,9 +786,9 @@ fn s_no_need_to_run_f_diffs_not_changed<'a>( s: &DoneWithoutRunningState, params: StepStateParams<'a>, ) -> StateTransition<'a> { - info!( + output!( params.output_snd, - "Dependencies for step {} hasn't changed. Skipping.", params.step.name + "[SKIP] [{}]", params.step.name ); Ok((s.keep_done(), params)) } From b05e8e711466f28ab182f94742a128f34997ba99 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sat, 3 Aug 2024 18:37:12 +0300 Subject: [PATCH 121/257] remove caching from glob_paths, glob_includes and dir_includes --- core/src/util/file.rs | 21 --------------------- 1 file changed, 21 deletions(-) diff --git a/core/src/util/file.rs b/core/src/util/file.rs index 8a9d47709..36d833dfa 100644 --- a/core/src/util/file.rs +++ b/core/src/util/file.rs @@ -90,12 +90,6 @@ pub fn compiled_glob(pipeline_rundir: &Path, glob: &str) -> Result Result { if pmm.contains_key(path) { // Makes a prefix comparison to see whether dir includes the path From fc5b481051cef3ac41513472f6614a8cd407424c Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sat, 3 Aug 2024 18:37:25 +0300 Subject: [PATCH 122/257] remove some traces --- core/src/util/pmp.rs | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/core/src/util/pmp.rs b/core/src/util/pmp.rs index fd07b9fa7..80718d9e2 100644 --- a/core/src/util/pmp.rs +++ b/core/src/util/pmp.rs @@ -38,7 +38,6 @@ impl XvcPathMetadataProvider { pub fn new(output_sender: &XvcOutputSender, xvc_root: &XvcRoot) -> Result { let initial_rules = IgnoreRules::try_from_patterns(xvc_root, COMMON_IGNORE_PATTERNS)?; let ignore_rules = build_ignore_rules(initial_rules, xvc_root, XVCIGNORE_FILENAME)?; - watch!(ignore_rules); let path_map = Arc::new(RwLock::new(HashMap::new())); let (watcher, event_receiver) = make_watcher(ignore_rules.clone())?; @@ -58,18 +57,17 @@ impl XvcPathMetadataProvider { let watcher = watcher; watch!(watcher); - let handle_fs_event = |fs_event, pmm: Arc>| match fs_event { + let handle_fs_event = |fs_event, pmm: Arc>| { + match fs_event { PathEvent::Create { path, metadata } => { let xvc_path = XvcPath::new(&xvc_root, &xvc_root, &path).unwrap(); let xvc_md = XvcMetadata::from(metadata); - watch!("Creating {} with {}", xvc_path, xvc_md); let mut pmm = pmm.write().unwrap(); pmm.insert(xvc_path, xvc_md); } PathEvent::Update { path, metadata } => { let xvc_path = XvcPath::new(&xvc_root, &xvc_root, &path).unwrap(); let xvc_md = XvcMetadata::from(metadata); - watch!("Updating {} with {}", xvc_path, xvc_md); let mut pmm = pmm.write().unwrap(); pmm.insert(xvc_path, xvc_md); } @@ -80,11 +78,10 @@ impl XvcPathMetadataProvider { size: None, modified: None, }; - watch!("Deleting {}", xvc_path); let mut pmm = pmm.write().unwrap(); pmm.insert(xvc_path, xvc_md); } - }; + } }; let mut sel = Select::new(); let fs_event_index = sel.recv(&fs_receiver); @@ -100,6 +97,7 @@ impl XvcPathMetadataProvider { if index == fs_event_index { let fs_event = selection.recv(&fs_receiver); watch!(fs_event); + watch!(path_map.read()); match fs_event { Ok(Some(fs_event)) => { let pmm = path_map.clone(); @@ -228,6 +226,7 @@ impl XvcPathMetadataProvider { pub fn glob_paths(&self, glob: &str) -> Result { watch!(glob); self.update_with_glob(glob)?; + watch!(self.path_map.read()); let mut matches = XvcPathMetadataMap::new(); let pattern = glob::Pattern::new(glob)?; for (p, md) in self.path_map.read().unwrap().iter() { @@ -252,6 +251,7 @@ impl XvcPathMetadataProvider { impl Drop for XvcPathMetadataProvider { /// Stop the background thread when quit fn drop(&mut self) { + watch!("Dropping XvcPathMetadataProvider", self); // Ignore if the channel is closed let _ = self.stop(); } From 116b29eb0c76260a88017961dbcaefce7cf299e1 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sat, 3 Aug 2024 18:37:33 +0300 Subject: [PATCH 123/257] spelling --- pipeline/src/pipeline/api/step_dependency.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pipeline/src/pipeline/api/step_dependency.rs b/pipeline/src/pipeline/api/step_dependency.rs index 52346330b..115b09d41 100644 --- a/pipeline/src/pipeline/api/step_dependency.rs +++ b/pipeline/src/pipeline/api/step_dependency.rs @@ -396,7 +396,7 @@ impl<'a> XvcDependencyList<'a> { } Ok(self) } - /// Records dependencies the store, as childs of `self.step`. + /// Records dependencies the store, as children of `self.step`. pub fn record(&self) -> Result<()> { self.xvc_root .with_r1nstore_mut(|rs: &mut R1NStore| { From a76215c911004993eed73220f56191a732e522b5 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sat, 3 Aug 2024 18:37:51 +0300 Subject: [PATCH 124/257] tidy up comparisons and add a Diff implementation --- pipeline/src/pipeline/deps/glob_items.rs | 97 ++++++++++++++++++++---- 1 file changed, 82 insertions(+), 15 deletions(-) diff --git a/pipeline/src/pipeline/deps/glob_items.rs b/pipeline/src/pipeline/deps/glob_items.rs index ab7e9361e..0891c2906 100644 --- a/pipeline/src/pipeline/deps/glob_items.rs +++ b/pipeline/src/pipeline/deps/glob_items.rs @@ -5,10 +5,10 @@ use crate::{Result, XvcDependency}; use serde::{Deserialize, Serialize}; use xvc_core::types::diff::Diffable; use xvc_core::{ - glob_paths, ContentDigest, HashAlgorithm, XvcMetadata, XvcPath, XvcPathMetadataProvider, - XvcRoot, + glob_paths, ContentDigest, Diff, HashAlgorithm, XvcMetadata, XvcPath, XvcPathMetadataProvider, XvcRoot }; use xvc_ecs::persist; +use xvc_logging::watch; /// A path collection where each item is tracked separately. #[derive(Debug, PartialOrd, Ord, Clone, Eq, PartialEq, Serialize, Deserialize)] @@ -38,25 +38,35 @@ impl GlobItemsDep { } /// Create a new [GlobItemsDep] with the given glob pattern and fill the metadata map from the - /// given pmm. The content digest map is empty. + /// given pmp. The content digest map in the returned [Self] is empty. pub fn from_pmp( - _xvc_root: &XvcRoot, glob_root: &XvcPath, glob: String, pmp: &XvcPathMetadataProvider, ) -> Result { - let xvc_path_metadata_map = - glob_paths(pmp, glob_root, &glob).map(|paths| paths.into_iter().collect())?; - watch!(xvc_path_metadata_map); + let empty = Self::new(glob); // We don't calculate the content digest map immediately, we only do that in through comparison - Ok(GlobItemsDep { - glob, + empty.update_paths(glob_root, pmp) + } + + /// Update path list by rereading the file list from disk. This doesn't update content digests + /// of files. Use [Self::update_digests] for this. + pub fn update_paths(self, glob_root: &XvcPath, pmp: &XvcPathMetadataProvider) -> Result { + watch!(self.xvc_path_metadata_map); + let xvc_path_metadata_map = + glob_paths(pmp, glob_root, &self.glob).map(|paths| paths.into_iter().collect())?; + watch!(xvc_path_metadata_map); + + Ok(Self { xvc_path_metadata_map, - xvc_path_content_digest_map: BTreeMap::new(), + ..self }) } + - /// Update the content digest map for each path in the metadata map. + + /// Update the content digest map for each path in the metadata map. This doesn't update the + /// file list defined by glob. Use [Self::update_paths] for this. pub fn update_digests(self, xvc_root: &XvcRoot, algorithm: HashAlgorithm) -> Result { let mut xvc_path_content_digest_map = BTreeMap::new(); for (xvc_path, _xvc_metadata) in self.xvc_path_metadata_map.iter() { @@ -71,16 +81,22 @@ impl GlobItemsDep { }) } - /// Unlike update_digests, this only updates the changed paths' digest. - /// It checks the record's metadata for the identical path and only updates the digest if the metadata has changed. + /// Update the content digest map for each path in the metadata map. This doesn't update the + /// file list defined by glob. Use [Self::update_paths] for this. + /// + /// Calculates content digests when the path metadata is different from record's. This way only + /// the changed path's content digest is calculated. pub fn update_changed_paths_digests( - self, + mut self, record: &Self, xvc_root: &XvcRoot, + glob_root: &XvcPath, + pmp: &XvcPathMetadataProvider, algorithm: HashAlgorithm, ) -> Result { + // Update paths to get the new paths and metadata + self = self.update_paths(glob_root, pmp)?; let mut xvc_path_content_digest_map = BTreeMap::new(); - watch!(xvc_path_metadata_map); for (xvc_path, xvc_metadata) in self.xvc_path_metadata_map.iter() { let record_metadata = record.xvc_path_metadata_map.get(xvc_path); let content_digest = if XvcMetadata::diff(record_metadata, Some(xvc_metadata)).changed() @@ -105,4 +121,55 @@ persist!(GlobItemsDep, "glob-dependency"); impl Diffable for GlobItemsDep { type Item = Self; + + fn diff(record: Option<&Self::Item>, actual: Option<&Self::Item>) -> xvc_core::Diff { + watch!(record); + watch!(actual); + match (record, actual) { + (None, None) => std::unreachable!("Both record and actual are None"), + (None, Some(actual)) => xvc_core::Diff::RecordMissing { + actual: actual.clone(), + }, + (Some(record), None) => xvc_core::Diff::ActualMissing { + record: record.clone(), + }, + (Some(record), Some(actual)) => { + match Self::diff_superficial(record, actual) { + Diff::Identical => Diff::Identical, + Diff::Skipped => Diff::Skipped, + Diff::ActualMissing { .. } => std::unreachable!("We already checked this conditions above"), + Diff::RecordMissing { .. } => std::unreachable!("We already checked this conditions above"), + Diff::Different { record, actual } => Self::diff_thorough(&record, &actual), + } + } + } + } + + /// Just compares the xvc_path_metadata_map field. + fn diff_superficial(record: &Self::Item, actual: &Self::Item) -> xvc_core::Diff { + if record.xvc_path_metadata_map == actual.xvc_path_metadata_map + { + Diff::Identical + } else { + Diff::Different { + record: record.clone(), + actual: actual.clone(), + } + } + } + + /// Just compares the xvc_content_digest_map field. + fn diff_thorough(record: &Self::Item, actual: &Self::Item) -> xvc_core::Diff { + if record.xvc_path_content_digest_map == actual.xvc_path_content_digest_map + { + Diff::Identical + } else { + Diff::Different { + record: record.clone(), + actual: actual.clone(), + } + } + } + + } From 4f73961c746971c8e2fccdb2bf9c73b0ccab4ce5 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sat, 3 Aug 2024 18:37:58 +0300 Subject: [PATCH 125/257] remove some traces --- pipeline/src/pipeline/mod.rs | 6 ------ 1 file changed, 6 deletions(-) diff --git a/pipeline/src/pipeline/mod.rs b/pipeline/src/pipeline/mod.rs index 36c7c5263..68c576138 100644 --- a/pipeline/src/pipeline/mod.rs +++ b/pipeline/src/pipeline/mod.rs @@ -293,16 +293,10 @@ pub fn the_grand_pipeline_loop( watch!(consider_changed); let all_deps = xvc_root.load_r1nstore::()?; - watch!(all_deps.parents.len()); - watch!(all_deps.children.len()); let all_outs = xvc_root.load_r1nstore::()?; - watch!(all_outs.parents.len()); - watch!(all_outs.children.len()); let pmp = XvcPathMetadataProvider::new(output_snd, xvc_root)?; - watch!(&pmp); let pipeline_len = pipeline_steps.len(); - watch!(pipeline_len); let mut dependency_graph = DiGraphMap::::with_capacity( pipeline_len, From 8a803455aaa30a1b57ef60737f1134a3c175dce9 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sat, 3 Aug 2024 18:38:24 +0300 Subject: [PATCH 126/257] update superficial and thorough comparisons for glob_items --- pipeline/src/pipeline/deps/compare.rs | 25 +++++++++---------------- 1 file changed, 9 insertions(+), 16 deletions(-) diff --git a/pipeline/src/pipeline/deps/compare.rs b/pipeline/src/pipeline/deps/compare.rs index d8c5bc2e8..2fd824fef 100644 --- a/pipeline/src/pipeline/deps/compare.rs +++ b/pipeline/src/pipeline/deps/compare.rs @@ -311,14 +311,13 @@ fn thorough_compare_glob_items( cmp_params: &StepStateParams, record: &GlobItemsDep, ) -> Result> { - let actual = GlobItemsDep::from_pmp( - cmp_params.xvc_root, - cmp_params.pipeline_rundir, - record.glob.clone(), - cmp_params.pmp, - )? - .update_changed_paths_digests(record, cmp_params.xvc_root, cmp_params.algorithm)?; + let glob_root = cmp_params.pipeline_rundir; + let xvc_root = cmp_params.xvc_root; + let pmp = cmp_params.pmp; + let algorithm = cmp_params.algorithm; + // Calls update_paths in the update_changed_paths_digests method + let actual = record.clone().update_changed_paths_digests(record, xvc_root, glob_root, pmp, algorithm)?; Ok(GlobItemsDep::diff(Some(record), Some(&actual))) } @@ -343,7 +342,7 @@ fn thorough_compare_regex( ) -> Result> { let actual = RegexDep::new(record.path.clone(), record.regex.clone()) .update_metadata(cmp_params.pmp.get(&record.path)); - // Shortcircuit if the metadata is identical + // Short circuit if the metadata is identical match RegexDep::diff_superficial(record, &actual) { Diff::Different { record, actual } => { let actual = actual.update_digest(cmp_params.xvc_root, cmp_params.algorithm); @@ -535,14 +534,8 @@ fn superficial_compare_glob_items( cmp_params: &StepStateParams, record: &GlobItemsDep, ) -> Result> { - let actual = GlobItemsDep::from_pmp( - cmp_params.xvc_root, - cmp_params.pipeline_rundir, - record.glob.clone(), - cmp_params.pmp, - )? - .update_changed_paths_digests(record, cmp_params.xvc_root, cmp_params.algorithm)?; - + // We just compare the file list + let actual = record.clone().update_paths(cmp_params.pipeline_rundir, cmp_params.pmp)?; Ok(GlobItemsDep::diff_superficial(record, &actual)) } From 13dba315b0a336b711df57882d455609c594899a Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sat, 3 Aug 2024 18:40:18 +0300 Subject: [PATCH 127/257] Updated docs and changelog --- CHANGELOG.md | 7 ++++++- pipeline/src/pipeline/step.rs | 5 ++++- 2 files changed, 10 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6b7c76fb9..914cd4084 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,12 +5,17 @@ tree, and subtasks are marked with indentation. ## Unreleased +- Removed caching for globs that caused bugs in Python bindings. +- Documentation updates + +## 0.6.9 (2024-07-22) + - Added sqlite-query dependency to xvc pipelines. A step is invalidated to run when the output from an SQLite query changes. - Moved bundled sqlite behind a feature flag, bundled_sqlite. - Merged xvc-workflow-tests crate to the main xvc crate. - Added compiling Xvc with non-default features document -### 0.6.8 (2024-07-15) +## 0.6.8 (2024-07-15) - Modifications for Python bindings diff --git a/pipeline/src/pipeline/step.rs b/pipeline/src/pipeline/step.rs index 174d35ccb..4a508615a 100644 --- a/pipeline/src/pipeline/step.rs +++ b/pipeline/src/pipeline/step.rs @@ -130,7 +130,10 @@ pub enum StepSubCommand { #[arg(long = "glob", aliases=&["globs"])] globs: Option>, - /// Add a parameter dependency to the step in the form filename.yaml::model.units . Can be used multiple times. + /// Add a parameter dependency to the step in the form filename.yaml::model.units + /// + /// The file can be a JSON, TOML, or YAML file. You can specify hierarchical keys like + /// my.dict.key #[arg(long = "param", aliases = &["params"])] params: Option>, From b057fedc5a75159e2de6b646f318288a3a025a7c Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sat, 3 Aug 2024 18:52:57 +0300 Subject: [PATCH 128/257] Bump version --- config/Cargo.toml | 6 +++--- core/Cargo.toml | 12 ++++++------ ecs/Cargo.toml | 4 ++-- file/Cargo.toml | 16 ++++++++-------- lib/Cargo.toml | 20 ++++++++++---------- logging/Cargo.toml | 2 +- pipeline/Cargo.toml | 16 ++++++++-------- storage/Cargo.toml | 14 +++++++------- test_helper/Cargo.toml | 4 ++-- walker/Cargo.toml | 6 +++--- 10 files changed, 50 insertions(+), 50 deletions(-) diff --git a/config/Cargo.toml b/config/Cargo.toml index dd21e133c..fb2d90a5e 100644 --- a/config/Cargo.toml +++ b/config/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-config" -version = "0.6.9" +version = "0.6.10-alpha.0" edition = "2021" description = "Xvc configuration management" authors = ["Emre Şahin "] @@ -16,8 +16,8 @@ name = "xvc_config" crate-type = ["rlib"] [dependencies] -xvc-logging = { version = "0.6.9", path = "../logging" } -xvc-walker = { version = "0.6.9", path = "../walker" } +xvc-logging = { version = "0.6.10-alpha.0", path = "../logging" } +xvc-walker = { version = "0.6.10-alpha.0", path = "../walker" } ## Cli and config diff --git a/core/Cargo.toml b/core/Cargo.toml index fffaf198e..00f7315ad 100644 --- a/core/Cargo.toml +++ b/core/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-core" -version = "0.6.9" +version = "0.6.10-alpha.0" edition = "2021" description = "Xvc core for common elements for all commands" authors = ["Emre Şahin "] @@ -16,10 +16,10 @@ name = "xvc_core" crate-type = ["rlib"] [dependencies] -xvc-config = { version = "0.6.9", path = "../config" } -xvc-logging = { version = "0.6.9", path = "../logging" } -xvc-ecs = { version = "0.6.9", path = "../ecs" } -xvc-walker = { version = "0.6.9", path = "../walker" } +xvc-config = { version = "0.6.10-alpha.0", path = "../config" } +xvc-logging = { version = "0.6.10-alpha.0", path = "../logging" } +xvc-ecs = { version = "0.6.10-alpha.0", path = "../ecs" } +xvc-walker = { version = "0.6.10-alpha.0", path = "../walker" } ## Cli and config clap = { version = "^4.4", features = ["derive"] } @@ -80,6 +80,6 @@ itertools = "^0.13" [dev-dependencies] -xvc-test-helper = { version = "0.6.9", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.10-alpha.0", path = "../test_helper/" } proptest = "^1.4" test-case = "^3.3" diff --git a/ecs/Cargo.toml b/ecs/Cargo.toml index 16c435fa1..454eac0fe 100644 --- a/ecs/Cargo.toml +++ b/ecs/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-ecs" -version = "0.6.9" +version = "0.6.10-alpha.0" edition = "2021" description = "Entity-Component System for Xvc" authors = ["Emre Şahin "] @@ -16,7 +16,7 @@ name = "xvc_ecs" crate-type = ["rlib"] [dependencies] -xvc-logging = { version = "0.6.9", path = "../logging" } +xvc-logging = { version = "0.6.10-alpha.0", path = "../logging" } ## Serialization serde = { version = "^1.0", features = ["derive"] } diff --git a/file/Cargo.toml b/file/Cargo.toml index 858269c7e..1d257fbce 100644 --- a/file/Cargo.toml +++ b/file/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-file" -version = "0.6.9" +version = "0.6.10-alpha.0" edition = "2021" description = "File tracking, versioning, upload and download functions for Xvc" authors = ["Emre Şahin "] @@ -21,12 +21,12 @@ test = true bench = true [dependencies] -xvc-logging = { version = "0.6.9", path = "../logging" } -xvc-config = { version = "0.6.9", path = "../config" } -xvc-core = { version = "0.6.9", path = "../core" } -xvc-ecs = { version = "0.6.9", path = "../ecs" } -xvc-walker = { version = "0.6.9", path = "../walker" } -xvc-storage = { version = "0.6.9", path = "../storage" } +xvc-logging = { version = "0.6.10-alpha.0", path = "../logging" } +xvc-config = { version = "0.6.10-alpha.0", path = "../config" } +xvc-core = { version = "0.6.10-alpha.0", path = "../core" } +xvc-ecs = { version = "0.6.10-alpha.0", path = "../ecs" } +xvc-walker = { version = "0.6.10-alpha.0", path = "../walker" } +xvc-storage = { version = "0.6.10-alpha.0", path = "../storage" } ## Cli and config @@ -94,5 +94,5 @@ default = ["reflink"] reflink = ["dep:reflink"] [dev-dependencies] -xvc-test-helper = { version = "0.6.9", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.10-alpha.0", path = "../test_helper/" } shellfn = "^0.1" diff --git a/lib/Cargo.toml b/lib/Cargo.toml index d245b2d6d..734f46e4b 100644 --- a/lib/Cargo.toml +++ b/lib/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc" -version = "0.6.9" +version = "0.6.10-alpha.0" edition = "2021" description = "An MLOps tool to manage data files and pipelines on top of Git" authors = ["Emre Şahin "] @@ -20,14 +20,14 @@ name = "xvc" path = "src/main.rs" [dependencies] -xvc-config = { version = "0.6.9", path = "../config" } -xvc-core = { version = "0.6.9", path = "../core" } -xvc-logging = { version = "0.6.9", path = "../logging" } -xvc-ecs = { version = "0.6.9", path = "../ecs" } -xvc-file = { version = "0.6.9", path = "../file" } -xvc-pipeline = { version = "0.6.9", path = "../pipeline" } -xvc-walker = { version = "0.6.9", path = "../walker" } -xvc-storage = { version = "0.6.9", path = "../storage" } +xvc-config = { version = "0.6.10-alpha.0", path = "../config" } +xvc-core = { version = "0.6.10-alpha.0", path = "../core" } +xvc-logging = { version = "0.6.10-alpha.0", path = "../logging" } +xvc-ecs = { version = "0.6.10-alpha.0", path = "../ecs" } +xvc-file = { version = "0.6.10-alpha.0", path = "../file" } +xvc-pipeline = { version = "0.6.10-alpha.0", path = "../pipeline" } +xvc-walker = { version = "0.6.10-alpha.0", path = "../walker" } +xvc-storage = { version = "0.6.10-alpha.0", path = "../storage" } ## Cli and config @@ -134,4 +134,4 @@ shellfn = "^0.1" test-case = "^3.3" trycmd = "^0.15" which = "^6.0" -xvc-test-helper = { version = "0.6.9", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.10-alpha.0", path = "../test_helper/" } diff --git a/logging/Cargo.toml b/logging/Cargo.toml index 6bf3f1795..4635162c7 100644 --- a/logging/Cargo.toml +++ b/logging/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-logging" -version = "0.6.9" +version = "0.6.10-alpha.0" edition = "2021" description = "Logging crate for Xvc" authors = ["Emre Şahin "] diff --git a/pipeline/Cargo.toml b/pipeline/Cargo.toml index 5b0409486..a3bb895b2 100644 --- a/pipeline/Cargo.toml +++ b/pipeline/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-pipeline" -version = "0.6.9" +version = "0.6.10-alpha.0" edition = "2021" description = "Xvc data pipeline management" authors = ["Emre Şahin "] @@ -19,12 +19,12 @@ default = [] bundled-sqlite = ["rusqlite/bundled"] [dependencies] -xvc-config = { version = "0.6.9", path = "../config" } -xvc-core = { version = "0.6.9", path = "../core" } -xvc-ecs = { version = "0.6.9", path = "../ecs" } -xvc-logging = { version = "0.6.9", path = "../logging" } -xvc-walker = { version = "0.6.9", path = "../walker" } -xvc-file = { version = "0.6.9", path = "../file" } +xvc-config = { version = "0.6.10-alpha.0", path = "../config" } +xvc-core = { version = "0.6.10-alpha.0", path = "../core" } +xvc-ecs = { version = "0.6.10-alpha.0", path = "../ecs" } +xvc-logging = { version = "0.6.10-alpha.0", path = "../logging" } +xvc-walker = { version = "0.6.10-alpha.0", path = "../walker" } +xvc-file = { version = "0.6.10-alpha.0", path = "../file" } ## Cli and config clap = { version = "^4.4", features = ["derive"] } @@ -98,5 +98,5 @@ itertools = "^0.13" derive_more = "^0.99" [dev-dependencies] -xvc-test-helper = { version = "0.6.9", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.10-alpha.0", path = "../test_helper/" } test-case = "^3.3" diff --git a/storage/Cargo.toml b/storage/Cargo.toml index 5fda81de3..e972b6e87 100644 --- a/storage/Cargo.toml +++ b/storage/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-storage" -version = "0.6.9" +version = "0.6.10-alpha.0" edition = "2021" description = "Xvc remote and local storage management" authors = ["Emre Şahin "] @@ -16,11 +16,11 @@ name = "xvc_storage" crate-type = ["rlib"] [dependencies] -xvc-logging = { version = "0.6.9", path = "../logging" } -xvc-config = { version = "0.6.9", path = "../config" } -xvc-core = { version = "0.6.9", path = "../core" } -xvc-ecs = { version = "0.6.9", path = "../ecs" } -xvc-walker = { version = "0.6.9", path = "../walker" } +xvc-logging = { version = "0.6.10-alpha.0", path = "../logging" } +xvc-config = { version = "0.6.10-alpha.0", path = "../config" } +xvc-core = { version = "0.6.10-alpha.0", path = "../core" } +xvc-ecs = { version = "0.6.10-alpha.0", path = "../ecs" } +xvc-walker = { version = "0.6.10-alpha.0", path = "../walker" } ## Cli and config clap = { version = "^4.4", features = ["derive"] } @@ -105,7 +105,7 @@ digital-ocean = ["async"] [dev-dependencies] -xvc-test-helper = { version = "0.6.9", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.10-alpha.0", path = "../test_helper/" } shellfn = "^0.1" [package.metadata.cargo-udeps.ignore] diff --git a/test_helper/Cargo.toml b/test_helper/Cargo.toml index 30ccc3997..4a052b53a 100644 --- a/test_helper/Cargo.toml +++ b/test_helper/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-test-helper" -version = "0.6.9" +version = "0.6.10-alpha.0" edition = "2021" description = "Unit test helper functions for Xvc" authors = ["Emre Şahin "] @@ -20,7 +20,7 @@ path = "src/main.rs" # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html [dependencies] -xvc-logging = { version = "0.6.9", path = "../logging/" } +xvc-logging = { version = "0.6.10-alpha.0", path = "../logging/" } rand = "^0.8" log = "^0.4" diff --git a/walker/Cargo.toml b/walker/Cargo.toml index 6a6c8f20a..4cd8a624f 100644 --- a/walker/Cargo.toml +++ b/walker/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-walker" -version = "0.6.9" +version = "0.6.10-alpha.0" edition = "2021" description = "Xvc parallel file system walker with ignore features" authors = ["Emre Şahin "] @@ -16,7 +16,7 @@ name = "xvc_walker" crate-type = ["rlib"] [dependencies] -xvc-logging = { version = "0.6.9", path = "../logging" } +xvc-logging = { version = "0.6.10-alpha.0", path = "../logging" } globset = "^0.4" ## Parallelization @@ -38,7 +38,7 @@ itertools = "^0.13" regex = "^1.10" [dev-dependencies] -xvc-test-helper = { path = "../test_helper/", version = "0.6.9" } +xvc-test-helper = { path = "../test_helper/", version = "0.6.10-alpha.0" } test-case = "^3.3" [package.metadata.cargo-udeps.ignore] From 1a35f5678bb6308a5a91e0929b85b46fb0dbb03d Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sat, 3 Aug 2024 18:55:14 +0300 Subject: [PATCH 129/257] make reflink optional in libn --- lib/Cargo.toml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/Cargo.toml b/lib/Cargo.toml index 734f46e4b..f72b0a189 100644 --- a/lib/Cargo.toml +++ b/lib/Cargo.toml @@ -57,7 +57,7 @@ crossbeam = "^0.8" ## File system subprocess = "^0.2" -reflink = "^0.1" +reflink = { "^0.1".optional = true } relative-path = { version = "^1.9", features = ["serde"] } path-absolutize = "^3.1" glob = "^0.3" @@ -102,7 +102,7 @@ gcs = ["xvc-storage/gcs"] wasabi = ["xvc-storage/wasabi"] digital-ocean = ["xvc-storage/digital-ocean"] bundled-sqlite = ["xvc-pipeline/bundled-sqlite"] -reflink = ["xvc-file/reflink"] +reflink = ["xvc-file/reflink", "dep:reflink"] test-s3 = ["s3"] test-minio = ["minio"] test-r2 = ["r2"] From 9b1b13de42295a27f3d0215d7fc365bce2739707 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sat, 3 Aug 2024 18:56:28 +0300 Subject: [PATCH 130/257] fix manifest --- lib/Cargo.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/Cargo.toml b/lib/Cargo.toml index f72b0a189..1a8310636 100644 --- a/lib/Cargo.toml +++ b/lib/Cargo.toml @@ -57,7 +57,7 @@ crossbeam = "^0.8" ## File system subprocess = "^0.2" -reflink = { "^0.1".optional = true } +reflink = { version = "^0.1", optional = true } relative-path = { version = "^1.9", features = ["serde"] } path-absolutize = "^3.1" glob = "^0.3" From cfa1d494472b2d1febbb9094ff3a5369d4cce7ef Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sun, 4 Aug 2024 15:24:46 +0300 Subject: [PATCH 131/257] Fix optional dependencies --- config/Cargo.toml | 6 ++-- core/Cargo.toml | 12 +++---- ecs/Cargo.toml | 4 +-- file/Cargo.toml | 16 +++++----- file/src/common/mod.rs | 71 +++++++++++++++++++++++++----------------- file/src/list/mod.rs | 1 - lib/Cargo.toml | 26 +++++++--------- logging/Cargo.toml | 2 +- pipeline/Cargo.toml | 16 +++++----- storage/Cargo.toml | 26 ++++++++-------- test_helper/Cargo.toml | 4 +-- walker/Cargo.toml | 6 ++-- 12 files changed, 100 insertions(+), 90 deletions(-) diff --git a/config/Cargo.toml b/config/Cargo.toml index fb2d90a5e..94bd90d2c 100644 --- a/config/Cargo.toml +++ b/config/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-config" -version = "0.6.10-alpha.0" +version = "0.6.10-alpha.1" edition = "2021" description = "Xvc configuration management" authors = ["Emre Şahin "] @@ -16,8 +16,8 @@ name = "xvc_config" crate-type = ["rlib"] [dependencies] -xvc-logging = { version = "0.6.10-alpha.0", path = "../logging" } -xvc-walker = { version = "0.6.10-alpha.0", path = "../walker" } +xvc-logging = { version = "0.6.10-alpha.1", path = "../logging" } +xvc-walker = { version = "0.6.10-alpha.1", path = "../walker" } ## Cli and config diff --git a/core/Cargo.toml b/core/Cargo.toml index 00f7315ad..43d216484 100644 --- a/core/Cargo.toml +++ b/core/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-core" -version = "0.6.10-alpha.0" +version = "0.6.10-alpha.1" edition = "2021" description = "Xvc core for common elements for all commands" authors = ["Emre Şahin "] @@ -16,10 +16,10 @@ name = "xvc_core" crate-type = ["rlib"] [dependencies] -xvc-config = { version = "0.6.10-alpha.0", path = "../config" } -xvc-logging = { version = "0.6.10-alpha.0", path = "../logging" } -xvc-ecs = { version = "0.6.10-alpha.0", path = "../ecs" } -xvc-walker = { version = "0.6.10-alpha.0", path = "../walker" } +xvc-config = { version = "0.6.10-alpha.1", path = "../config" } +xvc-logging = { version = "0.6.10-alpha.1", path = "../logging" } +xvc-ecs = { version = "0.6.10-alpha.1", path = "../ecs" } +xvc-walker = { version = "0.6.10-alpha.1", path = "../walker" } ## Cli and config clap = { version = "^4.4", features = ["derive"] } @@ -80,6 +80,6 @@ itertools = "^0.13" [dev-dependencies] -xvc-test-helper = { version = "0.6.10-alpha.0", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.10-alpha.1", path = "../test_helper/" } proptest = "^1.4" test-case = "^3.3" diff --git a/ecs/Cargo.toml b/ecs/Cargo.toml index 454eac0fe..eb2d38df8 100644 --- a/ecs/Cargo.toml +++ b/ecs/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-ecs" -version = "0.6.10-alpha.0" +version = "0.6.10-alpha.1" edition = "2021" description = "Entity-Component System for Xvc" authors = ["Emre Şahin "] @@ -16,7 +16,7 @@ name = "xvc_ecs" crate-type = ["rlib"] [dependencies] -xvc-logging = { version = "0.6.10-alpha.0", path = "../logging" } +xvc-logging = { version = "0.6.10-alpha.1", path = "../logging" } ## Serialization serde = { version = "^1.0", features = ["derive"] } diff --git a/file/Cargo.toml b/file/Cargo.toml index 1d257fbce..5bbbbff4b 100644 --- a/file/Cargo.toml +++ b/file/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-file" -version = "0.6.10-alpha.0" +version = "0.6.10-alpha.1" edition = "2021" description = "File tracking, versioning, upload and download functions for Xvc" authors = ["Emre Şahin "] @@ -21,12 +21,12 @@ test = true bench = true [dependencies] -xvc-logging = { version = "0.6.10-alpha.0", path = "../logging" } -xvc-config = { version = "0.6.10-alpha.0", path = "../config" } -xvc-core = { version = "0.6.10-alpha.0", path = "../core" } -xvc-ecs = { version = "0.6.10-alpha.0", path = "../ecs" } -xvc-walker = { version = "0.6.10-alpha.0", path = "../walker" } -xvc-storage = { version = "0.6.10-alpha.0", path = "../storage" } +xvc-logging = { version = "0.6.10-alpha.1", path = "../logging" } +xvc-config = { version = "0.6.10-alpha.1", path = "../config" } +xvc-core = { version = "0.6.10-alpha.1", path = "../core" } +xvc-ecs = { version = "0.6.10-alpha.1", path = "../ecs" } +xvc-walker = { version = "0.6.10-alpha.1", path = "../walker" } +xvc-storage = { version = "0.6.10-alpha.1", path = "../storage", default-features = false } ## Cli and config @@ -94,5 +94,5 @@ default = ["reflink"] reflink = ["dep:reflink"] [dev-dependencies] -xvc-test-helper = { version = "0.6.10-alpha.0", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.10-alpha.1", path = "../test_helper/" } shellfn = "^0.1" diff --git a/file/src/common/mod.rs b/file/src/common/mod.rs index 47e6ecb3a..49eedb973 100644 --- a/file/src/common/mod.rs +++ b/file/src/common/mod.rs @@ -318,16 +318,7 @@ pub fn recheck_from_cache( #[allow(clippy::permissions_set_readonly_false)] match recheck_method { RecheckMethod::Copy => { - watch!("Before copy"); - watch!(&cache_path); - watch!(&path); - fs::copy(&cache_path, &path)?; - info!(output_snd, "[COPY] {} -> {}", cache_path, path); - let mut perm = path.metadata()?.permissions(); - watch!(&perm); - perm.set_readonly(false); - watch!(&perm); - fs::set_permissions(&path, perm)?; + copy_file(output_snd, cache_path, path)?; } RecheckMethod::Hardlink => { fs::hard_link(&cache_path, &path)?; @@ -337,26 +328,8 @@ pub fn recheck_from_cache( make_symlink(&cache_path, &path)?; info!(output_snd, "[SYMLINK] {} -> {}", cache_path, path); } - #[cfg(feature="reflink")] RecheckMethod::Reflink => { - match reflink::reflink_or_copy(&cache_path, &path) { - Ok(None) => { - info!(output_snd, "[REFLINK] {} -> {}", cache_path, path); - } - Ok(Some(_)) => { - warn!( - output_snd, - "File system doesn't support reflink. Copying instead." - ); - info!(output_snd, "[COPY] {} -> {}", cache_path, path); - let mut perm = path.metadata()?.permissions(); - perm.set_readonly(false); - fs::set_permissions(&path, perm)?; - } - Err(source) => { - Error::IoError { source }.error(); - } - }; + reflink(output_snd, cache_path, path)?; } } uwr!( @@ -369,6 +342,46 @@ pub fn recheck_from_cache( Ok(()) } +#[cfg(feature="reflink")] +fn reflink(output_snd: &XvcOutputSender, cache_path: AbsolutePath, path: AbsolutePath) -> Result<()> { + match reflink::reflink(&cache_path, &path) { + Ok(_) => { + info!(output_snd, "[REFLINK] {} -> {}", cache_path, path); + Ok(()) + } + Err(e) => { + warn!( + output_snd, + "File system doesn't support reflink. {e}. Copying instead." + ); + copy_file(output_snd, cache_path, path) + } + } +} + +fn copy_file(output_snd: &XvcOutputSender, cache_path: AbsolutePath, path: AbsolutePath) -> Result<()> { + watch!("Before copy"); + watch!(&cache_path); + watch!(&path); + fs::copy(&cache_path, &path)?; + info!(output_snd, "[COPY] {} -> {}", cache_path, path); + let mut perm = path.metadata()?.permissions(); + watch!(&perm); + perm.set_readonly(false); + watch!(&perm); + fs::set_permissions(&path, perm)?; + Ok(()) +} + +#[cfg(not(feature="reflink"))] +fn reflink(output_snd: &XvcOutputSender, cache_path: AbsolutePath, path: AbsolutePath) -> Result<()> { + warn!( + output_snd, + "Xvc isn't compiled with reflink support. Copying the file." + ); + copy_file(output_snd, cache_path, path) +} + /// All cache paths for all xvc paths. /// There are extracted from the event logs. pub fn cache_paths_for_xvc_paths( diff --git a/file/src/list/mod.rs b/file/src/list/mod.rs index 38aacbec0..e4f45aa2c 100644 --- a/file/src/list/mod.rs +++ b/file/src/list/mod.rs @@ -250,7 +250,6 @@ fn format_recheck_method(recheck_method: RecheckMethod) -> String { RecheckMethod::Copy => "C".to_string(), RecheckMethod::Symlink => "S".to_string(), RecheckMethod::Hardlink => "H".to_string(), - #[cfg(feature="reflink")] RecheckMethod::Reflink => "R".to_string(), } } diff --git a/lib/Cargo.toml b/lib/Cargo.toml index 1a8310636..029db7606 100644 --- a/lib/Cargo.toml +++ b/lib/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc" -version = "0.6.10-alpha.0" +version = "0.6.10-alpha.1" edition = "2021" description = "An MLOps tool to manage data files and pipelines on top of Git" authors = ["Emre Şahin "] @@ -20,18 +20,16 @@ name = "xvc" path = "src/main.rs" [dependencies] -xvc-config = { version = "0.6.10-alpha.0", path = "../config" } -xvc-core = { version = "0.6.10-alpha.0", path = "../core" } -xvc-logging = { version = "0.6.10-alpha.0", path = "../logging" } -xvc-ecs = { version = "0.6.10-alpha.0", path = "../ecs" } -xvc-file = { version = "0.6.10-alpha.0", path = "../file" } -xvc-pipeline = { version = "0.6.10-alpha.0", path = "../pipeline" } -xvc-walker = { version = "0.6.10-alpha.0", path = "../walker" } -xvc-storage = { version = "0.6.10-alpha.0", path = "../storage" } - - +xvc-config = { version = "0.6.10-alpha.1", path = "../config" } +xvc-core = { version = "0.6.10-alpha.1", path = "../core" } +xvc-logging = { version = "0.6.10-alpha.1", path = "../logging" } +xvc-ecs = { version = "0.6.10-alpha.1", path = "../ecs" } +xvc-file = { version = "0.6.10-alpha.1", path = "../file", default-features = false } +xvc-pipeline = { version = "0.6.10-alpha.1", path = "../pipeline" } +xvc-walker = { version = "0.6.10-alpha.1", path = "../walker" } ## Cli and config clap = { version = "^4.4", features = ["derive", "cargo"] } +xvc-storage = { version = "0.6.10-alpha.1", path = "../storage", default-features = false } directories-next = "2.0" ## Hashing @@ -57,7 +55,7 @@ crossbeam = "^0.8" ## File system subprocess = "^0.2" -reflink = { version = "^0.1", optional = true } +# reflink = { version = "^0.1", optional = true } relative-path = { version = "^1.9", features = ["serde"] } path-absolutize = "^3.1" glob = "^0.3" @@ -102,7 +100,7 @@ gcs = ["xvc-storage/gcs"] wasabi = ["xvc-storage/wasabi"] digital-ocean = ["xvc-storage/digital-ocean"] bundled-sqlite = ["xvc-pipeline/bundled-sqlite"] -reflink = ["xvc-file/reflink", "dep:reflink"] +reflink = ["xvc-file/reflink"] test-s3 = ["s3"] test-minio = ["minio"] test-r2 = ["r2"] @@ -134,4 +132,4 @@ shellfn = "^0.1" test-case = "^3.3" trycmd = "^0.15" which = "^6.0" -xvc-test-helper = { version = "0.6.10-alpha.0", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.10-alpha.1", path = "../test_helper/" } diff --git a/logging/Cargo.toml b/logging/Cargo.toml index 4635162c7..51248a19e 100644 --- a/logging/Cargo.toml +++ b/logging/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-logging" -version = "0.6.10-alpha.0" +version = "0.6.10-alpha.1" edition = "2021" description = "Logging crate for Xvc" authors = ["Emre Şahin "] diff --git a/pipeline/Cargo.toml b/pipeline/Cargo.toml index a3bb895b2..a0bcfb208 100644 --- a/pipeline/Cargo.toml +++ b/pipeline/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-pipeline" -version = "0.6.10-alpha.0" +version = "0.6.10-alpha.1" edition = "2021" description = "Xvc data pipeline management" authors = ["Emre Şahin "] @@ -19,12 +19,12 @@ default = [] bundled-sqlite = ["rusqlite/bundled"] [dependencies] -xvc-config = { version = "0.6.10-alpha.0", path = "../config" } -xvc-core = { version = "0.6.10-alpha.0", path = "../core" } -xvc-ecs = { version = "0.6.10-alpha.0", path = "../ecs" } -xvc-logging = { version = "0.6.10-alpha.0", path = "../logging" } -xvc-walker = { version = "0.6.10-alpha.0", path = "../walker" } -xvc-file = { version = "0.6.10-alpha.0", path = "../file" } +xvc-config = { version = "0.6.10-alpha.1", path = "../config" } +xvc-core = { version = "0.6.10-alpha.1", path = "../core" } +xvc-ecs = { version = "0.6.10-alpha.1", path = "../ecs" } +xvc-logging = { version = "0.6.10-alpha.1", path = "../logging" } +xvc-walker = { version = "0.6.10-alpha.1", path = "../walker" } +xvc-file = { version = "0.6.10-alpha.1", path = "../file", default-features = false } ## Cli and config clap = { version = "^4.4", features = ["derive"] } @@ -98,5 +98,5 @@ itertools = "^0.13" derive_more = "^0.99" [dev-dependencies] -xvc-test-helper = { version = "0.6.10-alpha.0", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.10-alpha.1", path = "../test_helper/" } test-case = "^3.3" diff --git a/storage/Cargo.toml b/storage/Cargo.toml index e972b6e87..2de77d7d2 100644 --- a/storage/Cargo.toml +++ b/storage/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-storage" -version = "0.6.10-alpha.0" +version = "0.6.10-alpha.1" edition = "2021" description = "Xvc remote and local storage management" authors = ["Emre Şahin "] @@ -16,11 +16,11 @@ name = "xvc_storage" crate-type = ["rlib"] [dependencies] -xvc-logging = { version = "0.6.10-alpha.0", path = "../logging" } -xvc-config = { version = "0.6.10-alpha.0", path = "../config" } -xvc-core = { version = "0.6.10-alpha.0", path = "../core" } -xvc-ecs = { version = "0.6.10-alpha.0", path = "../ecs" } -xvc-walker = { version = "0.6.10-alpha.0", path = "../walker" } +xvc-logging = { version = "0.6.10-alpha.1", path = "../logging" } +xvc-config = { version = "0.6.10-alpha.1", path = "../config" } +xvc-core = { version = "0.6.10-alpha.1", path = "../core" } +xvc-ecs = { version = "0.6.10-alpha.1", path = "../ecs" } +xvc-walker = { version = "0.6.10-alpha.1", path = "../walker" } ## Cli and config clap = { version = "^4.4", features = ["derive"] } @@ -95,17 +95,17 @@ openssl = { version = "^0.10", features = ["vendored"] } [features] default = ["s3", "minio", "gcs", "wasabi", "r2", "digital-ocean"] -async = ["dep:rust-s3", "dep:futures", "dep:tokio"] +async = ["rust-s3", "futures", "tokio"] s3 = ["async"] -minio = ["async"] -gcs = ["async"] -wasabi = ["async"] -r2 = ["async"] -digital-ocean = ["async"] +minio = ["s3"] +gcs = ["s3"] +wasabi = ["s3"] +r2 = ["s3"] +digital-ocean = ["s3"] [dev-dependencies] -xvc-test-helper = { version = "0.6.10-alpha.0", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.10-alpha.1", path = "../test_helper/" } shellfn = "^0.1" [package.metadata.cargo-udeps.ignore] diff --git a/test_helper/Cargo.toml b/test_helper/Cargo.toml index 4a052b53a..140308b99 100644 --- a/test_helper/Cargo.toml +++ b/test_helper/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-test-helper" -version = "0.6.10-alpha.0" +version = "0.6.10-alpha.1" edition = "2021" description = "Unit test helper functions for Xvc" authors = ["Emre Şahin "] @@ -20,7 +20,7 @@ path = "src/main.rs" # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html [dependencies] -xvc-logging = { version = "0.6.10-alpha.0", path = "../logging/" } +xvc-logging = { version = "0.6.10-alpha.1", path = "../logging/" } rand = "^0.8" log = "^0.4" diff --git a/walker/Cargo.toml b/walker/Cargo.toml index 4cd8a624f..34e2b1c51 100644 --- a/walker/Cargo.toml +++ b/walker/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-walker" -version = "0.6.10-alpha.0" +version = "0.6.10-alpha.1" edition = "2021" description = "Xvc parallel file system walker with ignore features" authors = ["Emre Şahin "] @@ -16,7 +16,7 @@ name = "xvc_walker" crate-type = ["rlib"] [dependencies] -xvc-logging = { version = "0.6.10-alpha.0", path = "../logging" } +xvc-logging = { version = "0.6.10-alpha.1", path = "../logging" } globset = "^0.4" ## Parallelization @@ -38,7 +38,7 @@ itertools = "^0.13" regex = "^1.10" [dev-dependencies] -xvc-test-helper = { path = "../test_helper/", version = "0.6.10-alpha.0" } +xvc-test-helper = { path = "../test_helper/", version = "0.6.10-alpha.1" } test-case = "^3.3" [package.metadata.cargo-udeps.ignore] From 634a8aa75ec780895be3933425dd7cab969763db Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sun, 4 Aug 2024 15:33:14 +0300 Subject: [PATCH 132/257] Added bundled-openssl feature --- .github/workflows/rust.yml | 2 +- CHANGELOG.md | 3 ++- config/Cargo.toml | 6 +++--- core/Cargo.toml | 12 ++++++------ ecs/Cargo.toml | 4 ++-- file/Cargo.toml | 16 ++++++++-------- lib/Cargo.toml | 21 +++++++++++---------- logging/Cargo.toml | 2 +- pipeline/Cargo.toml | 16 ++++++++-------- storage/Cargo.toml | 21 ++++++++++----------- test_helper/Cargo.toml | 4 ++-- walker/Cargo.toml | 6 +++--- 12 files changed, 57 insertions(+), 56 deletions(-) diff --git a/.github/workflows/rust.yml b/.github/workflows/rust.yml index d380a996c..491735356 100644 --- a/.github/workflows/rust.yml +++ b/.github/workflows/rust.yml @@ -158,7 +158,7 @@ jobs: with: use-cross: false command: build - args: --release --target ${{ matrix.target }} + args: --release --target ${{ matrix.target }} --features=bundled-openssl - name: Package shell: bash diff --git a/CHANGELOG.md b/CHANGELOG.md index 525dbb5a3..731fe849c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,8 +7,9 @@ tree, and subtasks are marked with indentation. - Removed caching for globs that caused bugs in Python bindings. - Documentation updates +- Fix optional features. Now inter-workspace dependencies are defined by `default-features = false` on Cargo.toml +- Added `bundled-openssl` feature to use `vendored` feature of `openssl` crate optionally. This is turned on for Windows. -## 0.6.9 (2024-07-22) ## 0.6.9 (2024-07-29) - Added sqlite-query dependency to xvc pipelines. A step is invalidated to run when the output from an SQLite query changes. diff --git a/config/Cargo.toml b/config/Cargo.toml index 94bd90d2c..61b2c966e 100644 --- a/config/Cargo.toml +++ b/config/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-config" -version = "0.6.10-alpha.1" +version = "0.6.10-alpha.2" edition = "2021" description = "Xvc configuration management" authors = ["Emre Şahin "] @@ -16,8 +16,8 @@ name = "xvc_config" crate-type = ["rlib"] [dependencies] -xvc-logging = { version = "0.6.10-alpha.1", path = "../logging" } -xvc-walker = { version = "0.6.10-alpha.1", path = "../walker" } +xvc-logging = { version = "0.6.10-alpha.2", path = "../logging" } +xvc-walker = { version = "0.6.10-alpha.2", path = "../walker" } ## Cli and config diff --git a/core/Cargo.toml b/core/Cargo.toml index 43d216484..fe62ea751 100644 --- a/core/Cargo.toml +++ b/core/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-core" -version = "0.6.10-alpha.1" +version = "0.6.10-alpha.2" edition = "2021" description = "Xvc core for common elements for all commands" authors = ["Emre Şahin "] @@ -16,10 +16,10 @@ name = "xvc_core" crate-type = ["rlib"] [dependencies] -xvc-config = { version = "0.6.10-alpha.1", path = "../config" } -xvc-logging = { version = "0.6.10-alpha.1", path = "../logging" } -xvc-ecs = { version = "0.6.10-alpha.1", path = "../ecs" } -xvc-walker = { version = "0.6.10-alpha.1", path = "../walker" } +xvc-config = { version = "0.6.10-alpha.2", path = "../config" } +xvc-logging = { version = "0.6.10-alpha.2", path = "../logging" } +xvc-ecs = { version = "0.6.10-alpha.2", path = "../ecs" } +xvc-walker = { version = "0.6.10-alpha.2", path = "../walker" } ## Cli and config clap = { version = "^4.4", features = ["derive"] } @@ -80,6 +80,6 @@ itertools = "^0.13" [dev-dependencies] -xvc-test-helper = { version = "0.6.10-alpha.1", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.10-alpha.2", path = "../test_helper/" } proptest = "^1.4" test-case = "^3.3" diff --git a/ecs/Cargo.toml b/ecs/Cargo.toml index eb2d38df8..aaaa30e49 100644 --- a/ecs/Cargo.toml +++ b/ecs/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-ecs" -version = "0.6.10-alpha.1" +version = "0.6.10-alpha.2" edition = "2021" description = "Entity-Component System for Xvc" authors = ["Emre Şahin "] @@ -16,7 +16,7 @@ name = "xvc_ecs" crate-type = ["rlib"] [dependencies] -xvc-logging = { version = "0.6.10-alpha.1", path = "../logging" } +xvc-logging = { version = "0.6.10-alpha.2", path = "../logging" } ## Serialization serde = { version = "^1.0", features = ["derive"] } diff --git a/file/Cargo.toml b/file/Cargo.toml index 5bbbbff4b..e9f75b452 100644 --- a/file/Cargo.toml +++ b/file/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-file" -version = "0.6.10-alpha.1" +version = "0.6.10-alpha.2" edition = "2021" description = "File tracking, versioning, upload and download functions for Xvc" authors = ["Emre Şahin "] @@ -21,12 +21,12 @@ test = true bench = true [dependencies] -xvc-logging = { version = "0.6.10-alpha.1", path = "../logging" } -xvc-config = { version = "0.6.10-alpha.1", path = "../config" } -xvc-core = { version = "0.6.10-alpha.1", path = "../core" } -xvc-ecs = { version = "0.6.10-alpha.1", path = "../ecs" } -xvc-walker = { version = "0.6.10-alpha.1", path = "../walker" } -xvc-storage = { version = "0.6.10-alpha.1", path = "../storage", default-features = false } +xvc-logging = { version = "0.6.10-alpha.2", path = "../logging" } +xvc-config = { version = "0.6.10-alpha.2", path = "../config" } +xvc-core = { version = "0.6.10-alpha.2", path = "../core" } +xvc-ecs = { version = "0.6.10-alpha.2", path = "../ecs" } +xvc-walker = { version = "0.6.10-alpha.2", path = "../walker" } +xvc-storage = { version = "0.6.10-alpha.2", path = "../storage", default-features = false } ## Cli and config @@ -94,5 +94,5 @@ default = ["reflink"] reflink = ["dep:reflink"] [dev-dependencies] -xvc-test-helper = { version = "0.6.10-alpha.1", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.10-alpha.2", path = "../test_helper/" } shellfn = "^0.1" diff --git a/lib/Cargo.toml b/lib/Cargo.toml index 029db7606..ad6b84d26 100644 --- a/lib/Cargo.toml +++ b/lib/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc" -version = "0.6.10-alpha.1" +version = "0.6.10-alpha.2" edition = "2021" description = "An MLOps tool to manage data files and pipelines on top of Git" authors = ["Emre Şahin "] @@ -20,16 +20,16 @@ name = "xvc" path = "src/main.rs" [dependencies] -xvc-config = { version = "0.6.10-alpha.1", path = "../config" } -xvc-core = { version = "0.6.10-alpha.1", path = "../core" } -xvc-logging = { version = "0.6.10-alpha.1", path = "../logging" } -xvc-ecs = { version = "0.6.10-alpha.1", path = "../ecs" } -xvc-file = { version = "0.6.10-alpha.1", path = "../file", default-features = false } -xvc-pipeline = { version = "0.6.10-alpha.1", path = "../pipeline" } -xvc-walker = { version = "0.6.10-alpha.1", path = "../walker" } +xvc-config = { version = "0.6.10-alpha.2", path = "../config" } +xvc-core = { version = "0.6.10-alpha.2", path = "../core" } +xvc-logging = { version = "0.6.10-alpha.2", path = "../logging" } +xvc-ecs = { version = "0.6.10-alpha.2", path = "../ecs" } +xvc-file = { version = "0.6.10-alpha.2", path = "../file", default-features = false } +xvc-pipeline = { version = "0.6.10-alpha.2", path = "../pipeline" } +xvc-walker = { version = "0.6.10-alpha.2", path = "../walker" } ## Cli and config clap = { version = "^4.4", features = ["derive", "cargo"] } -xvc-storage = { version = "0.6.10-alpha.1", path = "../storage", default-features = false } +xvc-storage = { version = "0.6.10-alpha.2", path = "../storage", default-features = false } directories-next = "2.0" ## Hashing @@ -100,6 +100,7 @@ gcs = ["xvc-storage/gcs"] wasabi = ["xvc-storage/wasabi"] digital-ocean = ["xvc-storage/digital-ocean"] bundled-sqlite = ["xvc-pipeline/bundled-sqlite"] +bundled-openssl = ["xvc-storage/bundled-openssl"] reflink = ["xvc-file/reflink"] test-s3 = ["s3"] test-minio = ["minio"] @@ -132,4 +133,4 @@ shellfn = "^0.1" test-case = "^3.3" trycmd = "^0.15" which = "^6.0" -xvc-test-helper = { version = "0.6.10-alpha.1", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.10-alpha.2", path = "../test_helper/" } diff --git a/logging/Cargo.toml b/logging/Cargo.toml index 51248a19e..25e14145d 100644 --- a/logging/Cargo.toml +++ b/logging/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-logging" -version = "0.6.10-alpha.1" +version = "0.6.10-alpha.2" edition = "2021" description = "Logging crate for Xvc" authors = ["Emre Şahin "] diff --git a/pipeline/Cargo.toml b/pipeline/Cargo.toml index a0bcfb208..b10513491 100644 --- a/pipeline/Cargo.toml +++ b/pipeline/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-pipeline" -version = "0.6.10-alpha.1" +version = "0.6.10-alpha.2" edition = "2021" description = "Xvc data pipeline management" authors = ["Emre Şahin "] @@ -19,12 +19,12 @@ default = [] bundled-sqlite = ["rusqlite/bundled"] [dependencies] -xvc-config = { version = "0.6.10-alpha.1", path = "../config" } -xvc-core = { version = "0.6.10-alpha.1", path = "../core" } -xvc-ecs = { version = "0.6.10-alpha.1", path = "../ecs" } -xvc-logging = { version = "0.6.10-alpha.1", path = "../logging" } -xvc-walker = { version = "0.6.10-alpha.1", path = "../walker" } -xvc-file = { version = "0.6.10-alpha.1", path = "../file", default-features = false } +xvc-config = { version = "0.6.10-alpha.2", path = "../config" } +xvc-core = { version = "0.6.10-alpha.2", path = "../core" } +xvc-ecs = { version = "0.6.10-alpha.2", path = "../ecs" } +xvc-logging = { version = "0.6.10-alpha.2", path = "../logging" } +xvc-walker = { version = "0.6.10-alpha.2", path = "../walker" } +xvc-file = { version = "0.6.10-alpha.2", path = "../file", default-features = false } ## Cli and config clap = { version = "^4.4", features = ["derive"] } @@ -98,5 +98,5 @@ itertools = "^0.13" derive_more = "^0.99" [dev-dependencies] -xvc-test-helper = { version = "0.6.10-alpha.1", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.10-alpha.2", path = "../test_helper/" } test-case = "^3.3" diff --git a/storage/Cargo.toml b/storage/Cargo.toml index 2de77d7d2..1f057bc09 100644 --- a/storage/Cargo.toml +++ b/storage/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-storage" -version = "0.6.10-alpha.1" +version = "0.6.10-alpha.2" edition = "2021" description = "Xvc remote and local storage management" authors = ["Emre Şahin "] @@ -16,11 +16,11 @@ name = "xvc_storage" crate-type = ["rlib"] [dependencies] -xvc-logging = { version = "0.6.10-alpha.1", path = "../logging" } -xvc-config = { version = "0.6.10-alpha.1", path = "../config" } -xvc-core = { version = "0.6.10-alpha.1", path = "../core" } -xvc-ecs = { version = "0.6.10-alpha.1", path = "../ecs" } -xvc-walker = { version = "0.6.10-alpha.1", path = "../walker" } +xvc-logging = { version = "0.6.10-alpha.2", path = "../logging" } +xvc-config = { version = "0.6.10-alpha.2", path = "../config" } +xvc-core = { version = "0.6.10-alpha.2", path = "../core" } +xvc-ecs = { version = "0.6.10-alpha.2", path = "../ecs" } +xvc-walker = { version = "0.6.10-alpha.2", path = "../walker" } ## Cli and config clap = { version = "^4.4", features = ["derive"] } @@ -87,10 +87,8 @@ tokio = { version = "^1.35", optional = true, features = ["rt-multi-thread"] } rust-s3 = { version = "^0.34", optional = true } futures = { version = "^0.3", optional = true } -# Although we don't use openssl directly, we use its vendored version for cross compilation -# Cannot compile on Windows with vendored -[target.'cfg(target_os = "linux")'.dependencies] -openssl = { version = "^0.10", features = ["vendored"] } +# On Linux we use "vendored" feature and on Windows we don't use that feature. +openssl = { version = "^0.10", optional = true } [features] @@ -102,10 +100,11 @@ gcs = ["s3"] wasabi = ["s3"] r2 = ["s3"] digital-ocean = ["s3"] +bundled-openssl = ["openssl/vendored"] [dev-dependencies] -xvc-test-helper = { version = "0.6.10-alpha.1", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.10-alpha.2", path = "../test_helper/" } shellfn = "^0.1" [package.metadata.cargo-udeps.ignore] diff --git a/test_helper/Cargo.toml b/test_helper/Cargo.toml index 140308b99..78d311f8b 100644 --- a/test_helper/Cargo.toml +++ b/test_helper/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-test-helper" -version = "0.6.10-alpha.1" +version = "0.6.10-alpha.2" edition = "2021" description = "Unit test helper functions for Xvc" authors = ["Emre Şahin "] @@ -20,7 +20,7 @@ path = "src/main.rs" # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html [dependencies] -xvc-logging = { version = "0.6.10-alpha.1", path = "../logging/" } +xvc-logging = { version = "0.6.10-alpha.2", path = "../logging/" } rand = "^0.8" log = "^0.4" diff --git a/walker/Cargo.toml b/walker/Cargo.toml index 34e2b1c51..42b486afe 100644 --- a/walker/Cargo.toml +++ b/walker/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-walker" -version = "0.6.10-alpha.1" +version = "0.6.10-alpha.2" edition = "2021" description = "Xvc parallel file system walker with ignore features" authors = ["Emre Şahin "] @@ -16,7 +16,7 @@ name = "xvc_walker" crate-type = ["rlib"] [dependencies] -xvc-logging = { version = "0.6.10-alpha.1", path = "../logging" } +xvc-logging = { version = "0.6.10-alpha.2", path = "../logging" } globset = "^0.4" ## Parallelization @@ -38,7 +38,7 @@ itertools = "^0.13" regex = "^1.10" [dev-dependencies] -xvc-test-helper = { path = "../test_helper/", version = "0.6.10-alpha.1" } +xvc-test-helper = { path = "../test_helper/", version = "0.6.10-alpha.2" } test-case = "^3.3" [package.metadata.cargo-udeps.ignore] From c63f52584e436dff4f69d61f89b55eb157d288bd Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sun, 4 Aug 2024 17:45:22 +0300 Subject: [PATCH 133/257] Update ref docs showing pipeline outputs --- CHANGELOG.md | 2 +- book/src/ref/xvc-pipeline-run.md | 4 ++-- book/src/ref/xvc-pipeline-step-dependency-file.md | 6 +++--- book/src/ref/xvc-pipeline-step-dependency-generic.md | 6 +++--- book/src/ref/xvc-pipeline-step-dependency-glob-items.md | 6 +++--- book/src/ref/xvc-pipeline-step-dependency-glob.md | 4 ++-- book/src/ref/xvc-pipeline-step-dependency-line-items.md | 4 ++-- book/src/ref/xvc-pipeline-step-dependency-lines.md | 4 ++-- book/src/ref/xvc-pipeline-step-dependency-param.md | 6 +++--- book/src/ref/xvc-pipeline-step-dependency-regex-items.md | 6 +++--- book/src/ref/xvc-pipeline-step-dependency-regex.md | 4 ++-- book/src/ref/xvc-pipeline-step-dependency-sqlite-query.md | 6 +++--- book/src/ref/xvc-pipeline-step-dependency-step.md | 6 +++--- book/src/ref/xvc-pipeline-step-dependency-url.md | 2 +- book/src/ref/xvc-pipeline-step-dependency.md | 4 +++- 15 files changed, 36 insertions(+), 34 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 731fe849c..8b741b579 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,7 +8,7 @@ tree, and subtasks are marked with indentation. - Removed caching for globs that caused bugs in Python bindings. - Documentation updates - Fix optional features. Now inter-workspace dependencies are defined by `default-features = false` on Cargo.toml -- Added `bundled-openssl` feature to use `vendored` feature of `openssl` crate optionally. This is turned on for Windows. +- Added `bundled-openssl` feature to use `vendored` feature of `openssl` crate optionally. This is turned on for Windows builds on GA. ## 0.6.9 (2024-07-29) diff --git a/book/src/ref/xvc-pipeline-run.md b/book/src/ref/xvc-pipeline-run.md index 0e6e46adf..6671792d4 100644 --- a/book/src/ref/xvc-pipeline-run.md +++ b/book/src/ref/xvc-pipeline-run.md @@ -56,7 +56,7 @@ You can run the default pipeline without specifying its name. $ xvc pipeline run [OUT] [hello] hello -[DONE] hello (echo hello) +[DONE] [hello] (echo hello) ``` @@ -85,7 +85,7 @@ $ xvc pipeline --pipeline-name my-pipeline step new --step-name my-hello --comma $ xvc pipeline run --pipeline-name my-pipeline [OUT] [my-hello] hello from my-pipeline -[DONE] my-hello (echo 'hello from my-pipeline') +[DONE] [my-hello] (echo 'hello from my-pipeline') ``` diff --git a/book/src/ref/xvc-pipeline-step-dependency-file.md b/book/src/ref/xvc-pipeline-step-dependency-file.md index 2617e4868..947af167f 100644 --- a/book/src/ref/xvc-pipeline-step-dependency-file.md +++ b/book/src/ref/xvc-pipeline-step-dependency-file.md @@ -27,7 +27,7 @@ When you run the command, it will print `data.txt has changed` if the file `data $ xvc pipeline run [OUT] [file-dependency] data.txt has changed -[DONE] file-dependency (echo data.txt has changed) +[DONE] [file-dependency] (echo data.txt has changed) ``` @@ -45,7 +45,7 @@ A step will run if any of its dependencies have changed. $ xvc pipeline run [OUT] [file-dependency] data.txt has changed -[DONE] file-dependency (echo data.txt has changed) +[DONE] [file-dependency] (echo data.txt has changed) ``` @@ -69,7 +69,7 @@ Now the step will run even if none of the dependencies have changed. $ xvc pipeline run [OUT] [file-dependency] data.txt has changed -[DONE] file-dependency (echo data.txt has changed) +[DONE] [file-dependency] (echo data.txt has changed) ``` diff --git a/book/src/ref/xvc-pipeline-step-dependency-generic.md b/book/src/ref/xvc-pipeline-step-dependency-generic.md index 3dfaa0c02..44689a574 100644 --- a/book/src/ref/xvc-pipeline-step-dependency-generic.md +++ b/book/src/ref/xvc-pipeline-step-dependency-generic.md @@ -36,7 +36,7 @@ The step won't run until tomorrow, when `date +%F` changes. $ xvc pipeline run [OUT] [morning-message] Good Morning! -[DONE] morning-message (echo 'Good Morning!') +[DONE] [morning-message] (echo 'Good Morning!') ``` @@ -52,7 +52,7 @@ $ xvc pipeline step dependency --step-name directory-contents --generic 'ls' $ xvc pipeline run [OUT] [directory-contents] Files changed -[DONE] directory-contents (echo 'Files changed') +[DONE] [directory-contents] (echo 'Files changed') ``` @@ -66,7 +66,7 @@ $ xvc-test-helper generate-random-file new-file.txt $ xvc pipeline run [OUT] [directory-contents] Files changed -[DONE] directory-contents (echo 'Files changed') +[DONE] [directory-contents] (echo 'Files changed') ``` diff --git a/book/src/ref/xvc-pipeline-step-dependency-glob-items.md b/book/src/ref/xvc-pipeline-step-dependency-glob-items.md index 7abb83cde..7e0a3cdca 100644 --- a/book/src/ref/xvc-pipeline-step-dependency-glob-items.md +++ b/book/src/ref/xvc-pipeline-step-dependency-glob-items.md @@ -63,7 +63,7 @@ dir-0002/file-0003.bin ### Changed Files: -[DONE] files-changed (echo "### Added Files:/n${XVC_ADDED_GLOB_ITEMS}/n### Removed Files:/n${XVC_REMOVED_GLOB_ITEMS}/n### Changed Files:/n${XVC_CHANGED_GLOB_ITEMS}") +[DONE] [files-changed] (echo "### Added Files:/n${XVC_ADDED_GLOB_ITEMS}/n### Removed Files:/n${XVC_REMOVED_GLOB_ITEMS}/n### Changed Files:/n${XVC_CHANGED_GLOB_ITEMS}") $ xvc pipeline run @@ -83,7 +83,7 @@ dir-0001/file-0001.bin ### Changed Files: -[DONE] files-changed (echo "### Added Files:/n${XVC_ADDED_GLOB_ITEMS}/n### Removed Files:/n${XVC_REMOVED_GLOB_ITEMS}/n### Changed Files:/n${XVC_CHANGED_GLOB_ITEMS}") +[DONE] [files-changed] (echo "### Added Files:/n${XVC_ADDED_GLOB_ITEMS}/n### Removed Files:/n${XVC_REMOVED_GLOB_ITEMS}/n### Changed Files:/n${XVC_CHANGED_GLOB_ITEMS}") ``` @@ -101,7 +101,7 @@ $ xvc pipeline run ### Changed Files: dir-0001/file-0002.bin -[DONE] files-changed (echo "### Added Files:/n${XVC_ADDED_GLOB_ITEMS}/n### Removed Files:/n${XVC_REMOVED_GLOB_ITEMS}/n### Changed Files:/n${XVC_CHANGED_GLOB_ITEMS}") +[DONE] [files-changed] (echo "### Added Files:/n${XVC_ADDED_GLOB_ITEMS}/n### Removed Files:/n${XVC_REMOVED_GLOB_ITEMS}/n### Changed Files:/n${XVC_CHANGED_GLOB_ITEMS}") ``` diff --git a/book/src/ref/xvc-pipeline-step-dependency-glob.md b/book/src/ref/xvc-pipeline-step-dependency-glob.md index 9a3b101aa..e3d2103aa 100644 --- a/book/src/ref/xvc-pipeline-step-dependency-glob.md +++ b/book/src/ref/xvc-pipeline-step-dependency-glob.md @@ -48,7 +48,7 @@ The step is invalidated when a file described by the glob is added, removed or c $ xvc pipeline run [OUT] [files-changed] Files have changed. -[DONE] files-changed (echo 'Files have changed.') +[DONE] [files-changed] (echo 'Files have changed.') $ xvc pipeline run @@ -62,7 +62,7 @@ $ rm dir-0001/file-0001.bin $ xvc pipeline run [OUT] [files-changed] Files have changed. -[DONE] files-changed (echo 'Files have changed.') +[DONE] [files-changed] (echo 'Files have changed.') ``` diff --git a/book/src/ref/xvc-pipeline-step-dependency-line-items.md b/book/src/ref/xvc-pipeline-step-dependency-line-items.md index daab4b15e..7311478aa 100644 --- a/book/src/ref/xvc-pipeline-step-dependency-line-items.md +++ b/book/src/ref/xvc-pipeline-step-dependency-line-items.md @@ -77,7 +77,7 @@ $ xvc pipeline run Removed Lines: -[DONE] print-top-10 (echo "Added Lines:/n ${XVC_ADDED_LINE_ITEMS}/nRemoved Lines:/n${XVC_REMOVED_LINE_ITEMS}") +[DONE] [print-top-10] (echo "Added Lines:/n ${XVC_ADDED_LINE_ITEMS}/nRemoved Lines:/n${XVC_REMOVED_LINE_ITEMS}") ``` @@ -105,7 +105,7 @@ $ xvc pipeline run Removed Lines: "Hank", "M", 30, 71, 158 -[DONE] print-top-10 (echo "Added Lines:/n ${XVC_ADDED_LINE_ITEMS}/nRemoved Lines:/n${XVC_REMOVED_LINE_ITEMS}") +[DONE] [print-top-10] (echo "Added Lines:/n ${XVC_ADDED_LINE_ITEMS}/nRemoved Lines:/n${XVC_REMOVED_LINE_ITEMS}") ``` diff --git a/book/src/ref/xvc-pipeline-step-dependency-lines.md b/book/src/ref/xvc-pipeline-step-dependency-lines.md index 951d571bf..a947f2acd 100644 --- a/book/src/ref/xvc-pipeline-step-dependency-lines.md +++ b/book/src/ref/xvc-pipeline-step-dependency-lines.md @@ -67,7 +67,7 @@ $ xvc pipeline run "Hank", "M", 30, 71, 158 "Ivan", "M", 53, 72, 175 -[DONE] print-top-10 (head people.csv) +[DONE] [print-top-10] (head people.csv) `````` @@ -101,7 +101,7 @@ $ xvc pipeline run "Ferzan", "M", 30, 71, 158 "Ivan", "M", 53, 72, 175 -[DONE] print-top-10 (head people.csv) +[DONE] [print-top-10] (head people.csv) ``` diff --git a/book/src/ref/xvc-pipeline-step-dependency-param.md b/book/src/ref/xvc-pipeline-step-dependency-param.md index bb7f1a65f..fbcb75200 100644 --- a/book/src/ref/xvc-pipeline-step-dependency-param.md +++ b/book/src/ref/xvc-pipeline-step-dependency-param.md @@ -50,11 +50,11 @@ Run for the first time, as initially all dependencies are invalid: $ xvc pipeline run [OUT] [read-hyperparams] Update Hyperparameters -[DONE] read-hyperparams (echo "Update Hyperparameters") +[DONE] [read-hyperparams] (echo "Update Hyperparameters") [OUT] [read-database-config] Updated Database Configuration -[DONE] read-database-config (echo "Updated Database Configuration") +[DONE] [read-database-config] (echo "Updated Database Configuration") ``` @@ -77,7 +77,7 @@ $ perl -pi -e 's/5432/9876/g' myparams.yaml $ xvc pipeline run [OUT] [read-database-config] Updated Database Configuration -[DONE] read-database-config (echo "Updated Database Configuration") +[DONE] [read-database-config] (echo "Updated Database Configuration") ``` diff --git a/book/src/ref/xvc-pipeline-step-dependency-regex-items.md b/book/src/ref/xvc-pipeline-step-dependency-regex-items.md index 2b0dd1a6c..90a6a9fbc 100644 --- a/book/src/ref/xvc-pipeline-step-dependency-regex-items.md +++ b/book/src/ref/xvc-pipeline-step-dependency-regex-items.md @@ -77,7 +77,7 @@ $ xvc pipeline run "Omar", "M", 38, 70, 145 "Quin", "M", 29, 71, 176 -[DONE] new-males (echo "New Males:/n ${XVC_ADDED_REGEX_ITEMS}") +[DONE] [new-males] (echo "New Males:/n ${XVC_ADDED_REGEX_ITEMS}") [OUT] [new-females] New Females: "Elly", "F", 30, 66, 124 @@ -88,7 +88,7 @@ $ xvc pipeline run "Page", "F", 31, 67, 135 "Ruth", "F", 28, 65, 131 -[DONE] new-females (echo "New Females:/n ${XVC_ADDED_REGEX_ITEMS}") +[DONE] [new-females] (echo "New Females:/n ${XVC_ADDED_REGEX_ITEMS}") ``` @@ -132,7 +132,7 @@ $ xvc pipeline run [OUT] [new-females] New Females: "Asude", "F", 12, 55, 110 -[DONE] new-females (echo "New Females:/n ${XVC_ADDED_REGEX_ITEMS}") +[DONE] [new-females] (echo "New Females:/n ${XVC_ADDED_REGEX_ITEMS}") ``` diff --git a/book/src/ref/xvc-pipeline-step-dependency-regex.md b/book/src/ref/xvc-pipeline-step-dependency-regex.md index ac63d4563..3bff6b7a0 100644 --- a/book/src/ref/xvc-pipeline-step-dependency-regex.md +++ b/book/src/ref/xvc-pipeline-step-dependency-regex.md @@ -57,7 +57,7 @@ When you run the pipeline initially, the steps are run. $ xvc pipeline run [OUT] [count-females] 7 -[DONE] count-females (grep -c '"F",' people.csv) +[DONE] [count-females] (grep -c '"F",' people.csv) `````` @@ -100,7 +100,7 @@ $ cat people.csv $ xvc pipeline run [OUT] [count-females] 8 -[DONE] count-females (grep -c '"F",' people.csv) +[DONE] [count-females] (grep -c '"F",' people.csv) ``` diff --git a/book/src/ref/xvc-pipeline-step-dependency-sqlite-query.md b/book/src/ref/xvc-pipeline-step-dependency-sqlite-query.md index 4df59fbd8..9d7d7894d 100644 --- a/book/src/ref/xvc-pipeline-step-dependency-sqlite-query.md +++ b/book/src/ref/xvc-pipeline-step-dependency-sqlite-query.md @@ -61,7 +61,7 @@ Let's run the step without a dependency first. $ xvc pipeline run [OUT] [average-age] 34.6666666666667 -[DONE] average-age (sqlite3 people.db 'SELECT AVG(Age) FROM People;') +[DONE] [average-age] (sqlite3 people.db 'SELECT AVG(Age) FROM People;') ``` @@ -83,7 +83,7 @@ So, when the number of people in the table changes, the step will run. Initially $ xvc pipeline run [OUT] [average-age] 34.6666666666667 -[DONE] average-age (sqlite3 people.db 'SELECT AVG(Age) FROM People;') +[DONE] [average-age] (sqlite3 people.db 'SELECT AVG(Age) FROM People;') ``` @@ -107,7 +107,7 @@ This time, the step will run again as the result from dependency query (`SELECT $ xvc pipeline run [OUT] [average-age] 33.3684210526316 -[DONE] average-age (sqlite3 people.db 'SELECT AVG(Age) FROM People;') +[DONE] [average-age] (sqlite3 people.db 'SELECT AVG(Age) FROM People;') ``` diff --git a/book/src/ref/xvc-pipeline-step-dependency-step.md b/book/src/ref/xvc-pipeline-step-dependency-step.md index 8f7493c87..4d01a33d5 100644 --- a/book/src/ref/xvc-pipeline-step-dependency-step.md +++ b/book/src/ref/xvc-pipeline-step-dependency-step.md @@ -25,11 +25,11 @@ When run, the dependency will be run first and the step will be run after. $ xvc pipeline run [OUT] [hello] hello -[DONE] hello (echo hello) +[DONE] [hello] (echo hello) [OUT] [world] world -[DONE] world (echo world) +[DONE] [world] (echo world) ``` @@ -49,7 +49,7 @@ $ xvc pipeline step update --step-name world --when always $ xvc pipeline run [OUT] [world] world -[DONE] world (echo world) +[DONE] [world] (echo world) ``` diff --git a/book/src/ref/xvc-pipeline-step-dependency-url.md b/book/src/ref/xvc-pipeline-step-dependency-url.md index da120083b..4df471bf2 100644 --- a/book/src/ref/xvc-pipeline-step-dependency-url.md +++ b/book/src/ref/xvc-pipeline-step-dependency-url.md @@ -26,7 +26,7 @@ The step is invalidated when the page is updated. $ xvc pipeline run [OUT] [xvc-docs-update] Xvc docs updated! -[DONE] xvc-docs-update (echo 'Xvc docs updated!') +[DONE] [xvc-docs-update] (echo 'Xvc docs updated!') ``` diff --git a/book/src/ref/xvc-pipeline-step-dependency.md b/book/src/ref/xvc-pipeline-step-dependency.md index 83f5df1b9..804678211 100644 --- a/book/src/ref/xvc-pipeline-step-dependency.md +++ b/book/src/ref/xvc-pipeline-step-dependency.md @@ -43,7 +43,9 @@ Options: The difference between this and the glob-items option is that the glob-items option keeps track of all matching files individually, but this option only keeps track of the matched files' digest. This dependency uses considerably less disk space. --param - Add a parameter dependency to the step in the form filename.yaml::model.units . Can be used multiple times + Add a parameter dependency to the step in the form filename.yaml::model.units + + The file can be a JSON, TOML, or YAML file. You can specify hierarchical keys like my.dict.key --regex_items Add a regex dependency in the form filename.txt:/^regex/ . Can be used multiple times. From f38388148bd303a6a27744ad53838d5b7599d0e4 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sun, 4 Aug 2024 17:48:37 +0300 Subject: [PATCH 134/257] Bump version --- CHANGELOG.md | 9 +++------ config/Cargo.toml | 6 +++--- core/Cargo.toml | 12 ++++++------ ecs/Cargo.toml | 4 ++-- file/Cargo.toml | 16 ++++++++-------- lib/Cargo.toml | 20 ++++++++++---------- logging/Cargo.toml | 2 +- pipeline/Cargo.toml | 16 ++++++++-------- storage/Cargo.toml | 14 +++++++------- test_helper/Cargo.toml | 4 ++-- walker/Cargo.toml | 6 +++--- 11 files changed, 53 insertions(+), 56 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8b741b579..7312a6184 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,11 +1,8 @@ -# Introduction - -This document is a change log that I write for the project as I develop. It's a -tree, and subtasks are marked with indentation. - ## Unreleased -- Removed caching for globs that caused bugs in Python bindings. +## 0.6.10 (2024-08-04) + +- Removed caching for globs that caused bugs in Python bindings and long running processes. - Documentation updates - Fix optional features. Now inter-workspace dependencies are defined by `default-features = false` on Cargo.toml - Added `bundled-openssl` feature to use `vendored` feature of `openssl` crate optionally. This is turned on for Windows builds on GA. diff --git a/config/Cargo.toml b/config/Cargo.toml index 61b2c966e..e768e7804 100644 --- a/config/Cargo.toml +++ b/config/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-config" -version = "0.6.10-alpha.2" +version = "0.6.10" edition = "2021" description = "Xvc configuration management" authors = ["Emre Şahin "] @@ -16,8 +16,8 @@ name = "xvc_config" crate-type = ["rlib"] [dependencies] -xvc-logging = { version = "0.6.10-alpha.2", path = "../logging" } -xvc-walker = { version = "0.6.10-alpha.2", path = "../walker" } +xvc-logging = { version = "0.6.10", path = "../logging" } +xvc-walker = { version = "0.6.10", path = "../walker" } ## Cli and config diff --git a/core/Cargo.toml b/core/Cargo.toml index fe62ea751..80ca436ce 100644 --- a/core/Cargo.toml +++ b/core/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-core" -version = "0.6.10-alpha.2" +version = "0.6.10" edition = "2021" description = "Xvc core for common elements for all commands" authors = ["Emre Şahin "] @@ -16,10 +16,10 @@ name = "xvc_core" crate-type = ["rlib"] [dependencies] -xvc-config = { version = "0.6.10-alpha.2", path = "../config" } -xvc-logging = { version = "0.6.10-alpha.2", path = "../logging" } -xvc-ecs = { version = "0.6.10-alpha.2", path = "../ecs" } -xvc-walker = { version = "0.6.10-alpha.2", path = "../walker" } +xvc-config = { version = "0.6.10", path = "../config" } +xvc-logging = { version = "0.6.10", path = "../logging" } +xvc-ecs = { version = "0.6.10", path = "../ecs" } +xvc-walker = { version = "0.6.10", path = "../walker" } ## Cli and config clap = { version = "^4.4", features = ["derive"] } @@ -80,6 +80,6 @@ itertools = "^0.13" [dev-dependencies] -xvc-test-helper = { version = "0.6.10-alpha.2", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.10", path = "../test_helper/" } proptest = "^1.4" test-case = "^3.3" diff --git a/ecs/Cargo.toml b/ecs/Cargo.toml index aaaa30e49..3be065b20 100644 --- a/ecs/Cargo.toml +++ b/ecs/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-ecs" -version = "0.6.10-alpha.2" +version = "0.6.10" edition = "2021" description = "Entity-Component System for Xvc" authors = ["Emre Şahin "] @@ -16,7 +16,7 @@ name = "xvc_ecs" crate-type = ["rlib"] [dependencies] -xvc-logging = { version = "0.6.10-alpha.2", path = "../logging" } +xvc-logging = { version = "0.6.10", path = "../logging" } ## Serialization serde = { version = "^1.0", features = ["derive"] } diff --git a/file/Cargo.toml b/file/Cargo.toml index e9f75b452..cf3f138e0 100644 --- a/file/Cargo.toml +++ b/file/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-file" -version = "0.6.10-alpha.2" +version = "0.6.10" edition = "2021" description = "File tracking, versioning, upload and download functions for Xvc" authors = ["Emre Şahin "] @@ -21,12 +21,12 @@ test = true bench = true [dependencies] -xvc-logging = { version = "0.6.10-alpha.2", path = "../logging" } -xvc-config = { version = "0.6.10-alpha.2", path = "../config" } -xvc-core = { version = "0.6.10-alpha.2", path = "../core" } -xvc-ecs = { version = "0.6.10-alpha.2", path = "../ecs" } -xvc-walker = { version = "0.6.10-alpha.2", path = "../walker" } -xvc-storage = { version = "0.6.10-alpha.2", path = "../storage", default-features = false } +xvc-logging = { version = "0.6.10", path = "../logging" } +xvc-config = { version = "0.6.10", path = "../config" } +xvc-core = { version = "0.6.10", path = "../core" } +xvc-ecs = { version = "0.6.10", path = "../ecs" } +xvc-walker = { version = "0.6.10", path = "../walker" } +xvc-storage = { version = "0.6.10", path = "../storage", default-features = false } ## Cli and config @@ -94,5 +94,5 @@ default = ["reflink"] reflink = ["dep:reflink"] [dev-dependencies] -xvc-test-helper = { version = "0.6.10-alpha.2", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.10", path = "../test_helper/" } shellfn = "^0.1" diff --git a/lib/Cargo.toml b/lib/Cargo.toml index ad6b84d26..78636363b 100644 --- a/lib/Cargo.toml +++ b/lib/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc" -version = "0.6.10-alpha.2" +version = "0.6.10" edition = "2021" description = "An MLOps tool to manage data files and pipelines on top of Git" authors = ["Emre Şahin "] @@ -20,16 +20,16 @@ name = "xvc" path = "src/main.rs" [dependencies] -xvc-config = { version = "0.6.10-alpha.2", path = "../config" } -xvc-core = { version = "0.6.10-alpha.2", path = "../core" } -xvc-logging = { version = "0.6.10-alpha.2", path = "../logging" } -xvc-ecs = { version = "0.6.10-alpha.2", path = "../ecs" } -xvc-file = { version = "0.6.10-alpha.2", path = "../file", default-features = false } -xvc-pipeline = { version = "0.6.10-alpha.2", path = "../pipeline" } -xvc-walker = { version = "0.6.10-alpha.2", path = "../walker" } +xvc-config = { version = "0.6.10", path = "../config" } +xvc-core = { version = "0.6.10", path = "../core" } +xvc-logging = { version = "0.6.10", path = "../logging" } +xvc-ecs = { version = "0.6.10", path = "../ecs" } +xvc-file = { version = "0.6.10", path = "../file", default-features = false } +xvc-pipeline = { version = "0.6.10", path = "../pipeline" } +xvc-walker = { version = "0.6.10", path = "../walker" } ## Cli and config clap = { version = "^4.4", features = ["derive", "cargo"] } -xvc-storage = { version = "0.6.10-alpha.2", path = "../storage", default-features = false } +xvc-storage = { version = "0.6.10", path = "../storage", default-features = false } directories-next = "2.0" ## Hashing @@ -133,4 +133,4 @@ shellfn = "^0.1" test-case = "^3.3" trycmd = "^0.15" which = "^6.0" -xvc-test-helper = { version = "0.6.10-alpha.2", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.10", path = "../test_helper/" } diff --git a/logging/Cargo.toml b/logging/Cargo.toml index 25e14145d..a928aa413 100644 --- a/logging/Cargo.toml +++ b/logging/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-logging" -version = "0.6.10-alpha.2" +version = "0.6.10" edition = "2021" description = "Logging crate for Xvc" authors = ["Emre Şahin "] diff --git a/pipeline/Cargo.toml b/pipeline/Cargo.toml index b10513491..c7cfd3ecb 100644 --- a/pipeline/Cargo.toml +++ b/pipeline/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-pipeline" -version = "0.6.10-alpha.2" +version = "0.6.10" edition = "2021" description = "Xvc data pipeline management" authors = ["Emre Şahin "] @@ -19,12 +19,12 @@ default = [] bundled-sqlite = ["rusqlite/bundled"] [dependencies] -xvc-config = { version = "0.6.10-alpha.2", path = "../config" } -xvc-core = { version = "0.6.10-alpha.2", path = "../core" } -xvc-ecs = { version = "0.6.10-alpha.2", path = "../ecs" } -xvc-logging = { version = "0.6.10-alpha.2", path = "../logging" } -xvc-walker = { version = "0.6.10-alpha.2", path = "../walker" } -xvc-file = { version = "0.6.10-alpha.2", path = "../file", default-features = false } +xvc-config = { version = "0.6.10", path = "../config" } +xvc-core = { version = "0.6.10", path = "../core" } +xvc-ecs = { version = "0.6.10", path = "../ecs" } +xvc-logging = { version = "0.6.10", path = "../logging" } +xvc-walker = { version = "0.6.10", path = "../walker" } +xvc-file = { version = "0.6.10", path = "../file", default-features = false } ## Cli and config clap = { version = "^4.4", features = ["derive"] } @@ -98,5 +98,5 @@ itertools = "^0.13" derive_more = "^0.99" [dev-dependencies] -xvc-test-helper = { version = "0.6.10-alpha.2", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.10", path = "../test_helper/" } test-case = "^3.3" diff --git a/storage/Cargo.toml b/storage/Cargo.toml index 1f057bc09..73d1132fa 100644 --- a/storage/Cargo.toml +++ b/storage/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-storage" -version = "0.6.10-alpha.2" +version = "0.6.10" edition = "2021" description = "Xvc remote and local storage management" authors = ["Emre Şahin "] @@ -16,11 +16,11 @@ name = "xvc_storage" crate-type = ["rlib"] [dependencies] -xvc-logging = { version = "0.6.10-alpha.2", path = "../logging" } -xvc-config = { version = "0.6.10-alpha.2", path = "../config" } -xvc-core = { version = "0.6.10-alpha.2", path = "../core" } -xvc-ecs = { version = "0.6.10-alpha.2", path = "../ecs" } -xvc-walker = { version = "0.6.10-alpha.2", path = "../walker" } +xvc-logging = { version = "0.6.10", path = "../logging" } +xvc-config = { version = "0.6.10", path = "../config" } +xvc-core = { version = "0.6.10", path = "../core" } +xvc-ecs = { version = "0.6.10", path = "../ecs" } +xvc-walker = { version = "0.6.10", path = "../walker" } ## Cli and config clap = { version = "^4.4", features = ["derive"] } @@ -104,7 +104,7 @@ bundled-openssl = ["openssl/vendored"] [dev-dependencies] -xvc-test-helper = { version = "0.6.10-alpha.2", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.10", path = "../test_helper/" } shellfn = "^0.1" [package.metadata.cargo-udeps.ignore] diff --git a/test_helper/Cargo.toml b/test_helper/Cargo.toml index 78d311f8b..0bd4cc62f 100644 --- a/test_helper/Cargo.toml +++ b/test_helper/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-test-helper" -version = "0.6.10-alpha.2" +version = "0.6.10" edition = "2021" description = "Unit test helper functions for Xvc" authors = ["Emre Şahin "] @@ -20,7 +20,7 @@ path = "src/main.rs" # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html [dependencies] -xvc-logging = { version = "0.6.10-alpha.2", path = "../logging/" } +xvc-logging = { version = "0.6.10", path = "../logging/" } rand = "^0.8" log = "^0.4" diff --git a/walker/Cargo.toml b/walker/Cargo.toml index 42b486afe..7eb5c786f 100644 --- a/walker/Cargo.toml +++ b/walker/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-walker" -version = "0.6.10-alpha.2" +version = "0.6.10" edition = "2021" description = "Xvc parallel file system walker with ignore features" authors = ["Emre Şahin "] @@ -16,7 +16,7 @@ name = "xvc_walker" crate-type = ["rlib"] [dependencies] -xvc-logging = { version = "0.6.10-alpha.2", path = "../logging" } +xvc-logging = { version = "0.6.10", path = "../logging" } globset = "^0.4" ## Parallelization @@ -38,7 +38,7 @@ itertools = "^0.13" regex = "^1.10" [dev-dependencies] -xvc-test-helper = { path = "../test_helper/", version = "0.6.10-alpha.2" } +xvc-test-helper = { path = "../test_helper/", version = "0.6.10" } test-case = "^3.3" [package.metadata.cargo-udeps.ignore] From 8d7ad805d67709d12c88f283f42979ec75ad22b1 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sun, 4 Aug 2024 17:56:49 +0300 Subject: [PATCH 135/257] Update to use nightly proper --- .github/workflows/rust.yml | 3 ++- CHANGELOG.md | 1 + 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/.github/workflows/rust.yml b/.github/workflows/rust.yml index 491735356..704406749 100644 --- a/.github/workflows/rust.yml +++ b/.github/workflows/rust.yml @@ -26,7 +26,8 @@ jobs: # - build: beta # rust: beta - build: nightly - rust: nightly-2024-01-01 + rust: nightly + # rust: nightly-2024-01-01 test-args: --features test-ci --no-fail-fast ## for submitters other than me, I'll add another job here. # test-args: --no-fail-fast diff --git a/CHANGELOG.md b/CHANGELOG.md index 7312a6184..52188f90f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## 0.6.10 (2024-08-04) +- PR: - Removed caching for globs that caused bugs in Python bindings and long running processes. - Documentation updates - Fix optional features. Now inter-workspace dependencies are defined by `default-features = false` on Cargo.toml From dd08d0dc730fd3ffad61f99e5d81fbe18d283da2 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sun, 4 Aug 2024 18:46:01 +0300 Subject: [PATCH 136/257] Add publish github action --- .github/workflows/publish.yml | 41 +++++++++++++++++++++++++++++++++++ CHANGELOG.md | 1 + 2 files changed, 42 insertions(+) create mode 100644 .github/workflows/publish.yml diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml new file mode 100644 index 000000000..e0987a661 --- /dev/null +++ b/.github/workflows/publish.yml @@ -0,0 +1,41 @@ +name: Publish Crates + +on: + push: + tags: + - "v*.*.*" # Matches tags like v0.6.10 but also v0.6.10-alpha.5 + +jobs: + publish: + # Don't publish alpha releases + # if: startsWith(github.ref, 'refs/tags/v') && !contains(github.ref, '-') + runs-on: ubuntu-latest + strategy: + matrix: + package: + [ + xvc-logging, + xvc-test-helper, + xvc-walker, + xvc-ecs, + xvc-config, + xvc-core, + xvc-storage, + xvc-file, + xvc-pipelines, + xvc, + ] + + steps: + - name: Checkout code + uses: actions/checkout@v2 + + - name: Set up Rust + uses: actions-rs/toolchain@v1 + with: + toolchain: stable + + - name: Publish to crates.io + env: + CARGO_REGISTRY_TOKEN: ${{ secrets.CARGO_REGISTRY_TOKEN }} + run: cargo publish -p {{ matrix.package }} diff --git a/CHANGELOG.md b/CHANGELOG.md index 52188f90f..3449cfc90 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,7 @@ - Documentation updates - Fix optional features. Now inter-workspace dependencies are defined by `default-features = false` on Cargo.toml - Added `bundled-openssl` feature to use `vendored` feature of `openssl` crate optionally. This is turned on for Windows builds on GA. +- Crates are published from Github Actions ## 0.6.9 (2024-07-29) From 4b5d6a2250c503c2767aedef37ef9a56faae18a8 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sun, 4 Aug 2024 18:51:50 +0300 Subject: [PATCH 137/257] Fix publish.yml --- .github/workflows/publish.yml | 2 +- CHANGELOG.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index e0987a661..5e7879a67 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -38,4 +38,4 @@ jobs: - name: Publish to crates.io env: CARGO_REGISTRY_TOKEN: ${{ secrets.CARGO_REGISTRY_TOKEN }} - run: cargo publish -p {{ matrix.package }} + run: cargo publish -p ${{ matrix.package }} diff --git a/CHANGELOG.md b/CHANGELOG.md index 3449cfc90..7fce520fe 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -80,7 +80,7 @@ - Major overhaul in pipelines - PR: - - Added `--generic`, `--url`, `--regex-items`, `--line-items`, `--glob-items` + - Added `--generic`, `--url`, `--regex-items`, `--line-items`, `--glob-items` dependencies - as dependency types to `xvc pipeline step dependency` command - Updated `xvc pipelines` to run the pipeline by creating a thread for each step. - Updated pipelines state machine From ac0be749373a1132eaa371b548ee5751e9dfcdf5 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sun, 4 Aug 2024 19:06:00 +0300 Subject: [PATCH 138/257] Use an action for publish crate --- .github/workflows/publish.yml | 28 +++------------------------- 1 file changed, 3 insertions(+), 25 deletions(-) diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index 5e7879a67..bcfb1f0d7 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -10,32 +10,10 @@ jobs: # Don't publish alpha releases # if: startsWith(github.ref, 'refs/tags/v') && !contains(github.ref, '-') runs-on: ubuntu-latest - strategy: - matrix: - package: - [ - xvc-logging, - xvc-test-helper, - xvc-walker, - xvc-ecs, - xvc-config, - xvc-core, - xvc-storage, - xvc-file, - xvc-pipelines, - xvc, - ] - steps: - name: Checkout code uses: actions/checkout@v2 - - - name: Set up Rust - uses: actions-rs/toolchain@v1 + - name: publish crates workspace + uses: jio-gl/publish-crates-workspace@v4 with: - toolchain: stable - - - name: Publish to crates.io - env: - CARGO_REGISTRY_TOKEN: ${{ secrets.CARGO_REGISTRY_TOKEN }} - run: cargo publish -p ${{ matrix.package }} + registry-token: ${{ secrets.CARGO_REGISTRY_TOKEN }} From 62d3bf3d6753d428d2e4a7b3e75437b45c4bc76f Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sun, 4 Aug 2024 19:11:50 +0300 Subject: [PATCH 139/257] fix another fix --- .github/workflows/publish.yml | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index bcfb1f0d7..2fe81c08e 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -11,9 +11,12 @@ jobs: # if: startsWith(github.ref, 'refs/tags/v') && !contains(github.ref, '-') runs-on: ubuntu-latest steps: - - name: Checkout code - uses: actions/checkout@v2 - - name: publish crates workspace - uses: jio-gl/publish-crates-workspace@v4 + - uses: actions/checkout@v3 + - uses: actions-rs/toolchain@v1 + with: + toolchain: stable + override: true + - uses: katyo/publish-crates@v2 with: registry-token: ${{ secrets.CARGO_REGISTRY_TOKEN }} + ignore-unpublished-changes: true From 49d3c6899158d547f471b2cb9dc2c22c9bc4378e Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sun, 4 Aug 2024 19:16:20 +0300 Subject: [PATCH 140/257] just use a for loop --- .github/workflows/publish.yml | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index 2fe81c08e..1e9f5958d 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -8,15 +8,18 @@ on: jobs: publish: # Don't publish alpha releases - # if: startsWith(github.ref, 'refs/tags/v') && !contains(github.ref, '-') + if: startsWith(github.ref, 'refs/tags/v') && !contains(github.ref, '-') runs-on: ubuntu-latest steps: - - uses: actions/checkout@v3 - - uses: actions-rs/toolchain@v1 + - name: Checkout code + uses: actions/checkout@v2 + + - name: Set up Rust + uses: actions-rs/toolchain@v1 with: toolchain: stable - override: true - - uses: katyo/publish-crates@v2 - with: - registry-token: ${{ secrets.CARGO_REGISTRY_TOKEN }} - ignore-unpublished-changes: true + + - name: Publish to crates.io + env: + CARGO_REGISTRY_TOKEN: ${{ secrets.CARGO_REGISTRY_TOKEN }} + run: for p in xvc-logging xvc-test-helper xvc-walker xvc-ecs xvc-config xvc-core xvc-storage xvc-file xvc-pipelines xvc ; do cargo publish -p $p ; done From 63d964fa26e1c4e8611a6d6b5367f9a3104bb0ab Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sun, 4 Aug 2024 19:22:53 +0300 Subject: [PATCH 141/257] Add keep going when the packages are uploaded --- .github/workflows/publish.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index 1e9f5958d..a89335759 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -22,4 +22,4 @@ jobs: - name: Publish to crates.io env: CARGO_REGISTRY_TOKEN: ${{ secrets.CARGO_REGISTRY_TOKEN }} - run: for p in xvc-logging xvc-test-helper xvc-walker xvc-ecs xvc-config xvc-core xvc-storage xvc-file xvc-pipelines xvc ; do cargo publish -p $p ; done + run: for p in xvc-logging xvc-test-helper xvc-walker xvc-ecs xvc-config xvc-core xvc-storage xvc-file xvc-pipelines xvc ; do cargo publish --package $p ; done From 0492632cbec37c38f0ec5453ca2bc7fe885d0e82 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sun, 4 Aug 2024 19:26:52 +0300 Subject: [PATCH 142/257] Fix Windows build feature --- .github/workflows/rust.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/rust.yml b/.github/workflows/rust.yml index 704406749..bfa3b31bd 100644 --- a/.github/workflows/rust.yml +++ b/.github/workflows/rust.yml @@ -200,7 +200,7 @@ jobs: with: use-cross: false command: build - args: --release --target ${{ matrix.target }} --features bundled_sqlite + args: --release --target ${{ matrix.target }} --features bundled-sqlite - name: Package shell: bash From 15811a0e715a25e037c85ee6af8f3b84653ab2a6 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Mon, 5 Aug 2024 20:57:28 +0300 Subject: [PATCH 143/257] Update lib Cargo --- lib/Cargo.toml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/Cargo.toml b/lib/Cargo.toml index 78636363b..4c9acfa58 100644 --- a/lib/Cargo.toml +++ b/lib/Cargo.toml @@ -27,9 +27,10 @@ xvc-ecs = { version = "0.6.10", path = "../ecs" } xvc-file = { version = "0.6.10", path = "../file", default-features = false } xvc-pipeline = { version = "0.6.10", path = "../pipeline" } xvc-walker = { version = "0.6.10", path = "../walker" } +xvc-storage = { version = "0.6.10", path = "../storage", default-features = false } + ## Cli and config clap = { version = "^4.4", features = ["derive", "cargo"] } -xvc-storage = { version = "0.6.10", path = "../storage", default-features = false } directories-next = "2.0" ## Hashing @@ -55,7 +56,6 @@ crossbeam = "^0.8" ## File system subprocess = "^0.2" -# reflink = { version = "^0.1", optional = true } relative-path = { version = "^1.9", features = ["serde"] } path-absolutize = "^3.1" glob = "^0.3" From b96659134f0a435ccf055d2e1bf18c1d1f3f8da4 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Mon, 5 Aug 2024 21:15:07 +0300 Subject: [PATCH 144/257] bump dependencies --- config/Cargo.toml | 4 ++-- core/Cargo.toml | 14 +++++++------- ecs/Cargo.toml | 6 +++--- file/Cargo.toml | 12 ++++++------ lib/Cargo.toml | 18 +++++++++--------- pipeline/Cargo.toml | 12 ++++++------ storage/Cargo.toml | 14 +++++++------- test_helper/Cargo.toml | 2 +- 8 files changed, 41 insertions(+), 41 deletions(-) diff --git a/config/Cargo.toml b/config/Cargo.toml index e768e7804..30add05d0 100644 --- a/config/Cargo.toml +++ b/config/Cargo.toml @@ -33,7 +33,7 @@ crossbeam-channel = "^0.5" crossbeam = "^0.8" ## File system -walkdir = "^2.4" +walkdir = "^2.5" ## Logging and errors thiserror = "^1.0" @@ -45,4 +45,4 @@ fern = { version = "^0.6", features = ["colored"] } regex = "^1.10" strum = "^0.26" strum_macros = "^0.26" -lazy_static = "^1.4" +lazy_static = "^1.5" diff --git a/core/Cargo.toml b/core/Cargo.toml index 80ca436ce..cd4f28c04 100644 --- a/core/Cargo.toml +++ b/core/Cargo.toml @@ -22,7 +22,7 @@ xvc-ecs = { version = "0.6.10", path = "../ecs" } xvc-walker = { version = "0.6.10", path = "../walker" } ## Cli and config -clap = { version = "^4.4", features = ["derive"] } +clap = { version = "^4.5", features = ["derive"] } directories-next = "2.0" ## Hashing @@ -38,20 +38,20 @@ serde = { version = "^1.0", features = ["derive"] } serde_yaml = "^0.9" serde_json = "^1.0" rmp = "^0.8" -rmp-serde = "1.1.2" +rmp-serde = "1.3.0" toml = "^0.8" ## Network reqwest = { version = "^0.11", features = ["blocking", "json", "gzip"] } ## Parallelization -rayon = "^1.8" +rayon = "^1.10" crossbeam-channel = "^0.5" crossbeam = "^0.8" ## File system jwalk = "^0.8" -walkdir = "^2.4" +walkdir = "^2.5" relative-path = { version = "^1.9", features = ["serde"] } path-absolutize = "^3.1" glob = "^0.3" @@ -71,8 +71,8 @@ paste = "1.0" regex = "^1.10" strum = "^0.26" strum_macros = "^0.26" -lazy_static = "^1.4" -uuid = { version = "^1.6", features = ["serde", "v4", "fast-rng"] } +lazy_static = "^1.5" +uuid = { version = "^1.10", features = ["serde", "v4", "fast-rng"] } hex = { version = "^0.4", features = ["serde"] } cached = "^0.53" derive_more = "^0.99" @@ -81,5 +81,5 @@ itertools = "^0.13" [dev-dependencies] xvc-test-helper = { version = "0.6.10", path = "../test_helper/" } -proptest = "^1.4" +proptest = "^1.5" test-case = "^3.3" diff --git a/ecs/Cargo.toml b/ecs/Cargo.toml index 3be065b20..211b701db 100644 --- a/ecs/Cargo.toml +++ b/ecs/Cargo.toml @@ -22,10 +22,10 @@ xvc-logging = { version = "0.6.10", path = "../logging" } serde = { version = "^1.0", features = ["derive"] } serde_json = "^1.0" rmp = "^0.8" -rmp-serde = "1.1.2" +rmp-serde = "1.3.0" ## Parallelization -rayon = "^1.8" +rayon = "^1.10" crossbeam-channel = "^0.5" ## Logging and errors @@ -35,7 +35,7 @@ fern = { version = "^0.6", features = ["colored"] } thiserror = "^1.0" ## Misc -lazy_static = "^1.4" +lazy_static = "^1.5" rand = "^0.8" diff --git a/file/Cargo.toml b/file/Cargo.toml index cf3f138e0..cab53c52a 100644 --- a/file/Cargo.toml +++ b/file/Cargo.toml @@ -30,7 +30,7 @@ xvc-storage = { version = "0.6.10", path = "../storage", default-features = fals ## Cli and config -clap = { version = "^4.4", features = ["derive"] } +clap = { version = "^4.5", features = ["derive"] } directories-next = "2.0" ## Hashing @@ -46,18 +46,18 @@ serde = { version = "^1.0", features = ["derive"] } serde_yaml = "^0.9" serde_json = "^1.0" rmp = "^0.8" -rmp-serde = "1.1.2" +rmp-serde = "1.3.0" toml = "^0.8" ## Parallelization -rayon = "^1.8" +rayon = "^1.10" crossbeam-channel = "^0.5" crossbeam = "^0.8" dashmap = { version = "^6.0", features = ["serde", "rayon"] } ## File system jwalk = "^0.8" -walkdir = "^2.4" +walkdir = "^2.5" reflink = { version = "^0.1", optional = true } relative-path = { version = "^1.9", features = ["serde"] } path-absolutize = "^3.1" @@ -81,8 +81,8 @@ regex = "^1.10" chrono = "^0.4" strum = "^0.26" strum_macros = "^0.26" -lazy_static = "^1.4" -uuid = { version = "^1.6", features = ["serde", "v4", "fast-rng"] } +lazy_static = "^1.5" +uuid = { version = "^1.10", features = ["serde", "v4", "fast-rng"] } hex = { version = "^0.4", features = ["serde"] } url = { version = "^2.5", features = ["serde"] } itertools = "^0.13" diff --git a/lib/Cargo.toml b/lib/Cargo.toml index 4c9acfa58..ea5688936 100644 --- a/lib/Cargo.toml +++ b/lib/Cargo.toml @@ -30,7 +30,7 @@ xvc-walker = { version = "0.6.10", path = "../walker" } xvc-storage = { version = "0.6.10", path = "../storage", default-features = false } ## Cli and config -clap = { version = "^4.4", features = ["derive", "cargo"] } +clap = { version = "^4.5", features = ["derive", "cargo"] } directories-next = "2.0" ## Hashing @@ -46,11 +46,11 @@ serde = { version = "^1.0", features = ["derive"] } serde_yaml = "^0.9" serde_json = "^1.0" rmp = "^0.8" -rmp-serde = "1.1.2" +rmp-serde = "1.3.0" toml = "^0.8" ## Parallelization -rayon = "^1.8" +rayon = "^1.10" crossbeam-channel = "^0.5" crossbeam = "^0.8" @@ -69,7 +69,7 @@ fern = { version = "^0.6", features = ["colored"] } anyhow = "^1.0" ## UI -comfy-table = "7.1.0" +comfy-table = "7.1.1" ## macros paste = "1.0" @@ -79,8 +79,8 @@ regex = "^1.10" chrono = "^0.4" strum = "^0.26" strum_macros = "^0.26" -lazy_static = "^1.4" -uuid = { version = "^1.6", features = ["serde", "v4", "fast-rng"] } +lazy_static = "^1.5" +uuid = { version = "^1.10", features = ["serde", "v4", "fast-rng"] } hex = { version = "^0.4", features = ["serde"] } url = { version = "^2.5", features = ["serde"] } git-version = "^0.3" @@ -88,7 +88,7 @@ git-version = "^0.3" ## Mostly for testing assert_cmd = "^2.0" assert_fs = "^1.1" -predicates = "^3.0" +predicates = "^3.1" [features] @@ -127,8 +127,8 @@ escargot = "^0.5" fs_extra = "^1.3" globset = "^0.4" jwalk = "^0.8" -predicates = "^3.0" -proptest = "^1.4" +predicates = "^3.1" +proptest = "^1.5" shellfn = "^0.1" test-case = "^3.3" trycmd = "^0.15" diff --git a/pipeline/Cargo.toml b/pipeline/Cargo.toml index c7cfd3ecb..81be11eae 100644 --- a/pipeline/Cargo.toml +++ b/pipeline/Cargo.toml @@ -27,7 +27,7 @@ xvc-walker = { version = "0.6.10", path = "../walker" } xvc-file = { version = "0.6.10", path = "../file", default-features = false } ## Cli and config -clap = { version = "^4.4", features = ["derive"] } +clap = { version = "^4.5", features = ["derive"] } directories-next = "2.0" ## Hashing @@ -43,14 +43,14 @@ serde = { version = "^1.0", features = ["derive"] } serde_yaml = "^0.9" serde_json = "^1.0" rmp = "^0.8" -rmp-serde = "1.1.2" +rmp-serde = "1.3.0" toml = "^0.8" ## Caching cached = "^0.53" ## Parallelization -rayon = "^1.8" +rayon = "^1.10" crossbeam-channel = "^0.5" crossbeam = "^0.8" @@ -76,7 +76,7 @@ fallible-iterator = "^0.3" reqwest = { version = "^0.11", features = ["blocking", "json", "gzip"] } ## UI -comfy-table = "7.1.0" +comfy-table = "7.1.1" ## Graphs petgraph = "^0.6" @@ -90,8 +90,8 @@ regex = "^1.10" chrono = "^0.4" strum = "^0.26" strum_macros = "^0.26" -lazy_static = "^1.4" -uuid = { version = "^1.6", features = ["serde", "v4", "fast-rng"] } +lazy_static = "^1.5" +uuid = { version = "^1.10", features = ["serde", "v4", "fast-rng"] } hex = { version = "^0.4", features = ["serde"] } url = { version = "^2.5", features = ["serde"] } itertools = "^0.13" diff --git a/storage/Cargo.toml b/storage/Cargo.toml index 73d1132fa..65fc37319 100644 --- a/storage/Cargo.toml +++ b/storage/Cargo.toml @@ -23,7 +23,7 @@ xvc-ecs = { version = "0.6.10", path = "../ecs" } xvc-walker = { version = "0.6.10", path = "../walker" } ## Cli and config -clap = { version = "^4.4", features = ["derive"] } +clap = { version = "^4.5", features = ["derive"] } directories-next = "2.0" ## Hashing @@ -39,17 +39,17 @@ serde = { version = "^1.0", features = ["derive"] } serde_yaml = "^0.9" serde_json = "^1.0" rmp = "^0.8" -rmp-serde = "1.1.2" +rmp-serde = "1.3.0" toml = "^0.8" ## Parallelization -rayon = "^1.8" +rayon = "^1.10" crossbeam-channel = "^0.5" crossbeam = "^0.8" ## File system jwalk = "^0.8" -walkdir = "^2.4" +walkdir = "^2.5" relative-path = { version = "^1.9", features = ["serde"] } path-absolutize = "^3.1" glob = "^0.3" @@ -73,17 +73,17 @@ regex = "^1.10" strum = "^0.26" strum_macros = "^0.26" lazy_static = "^1.5" -uuid = { version = "^1.6", features = ["serde", "v4", "fast-rng"] } +uuid = { version = "^1.10", features = ["serde", "v4", "fast-rng"] } hex = { version = "^0.4", features = ["serde"] } url = { version = "^2.5", features = ["serde"] } itertools = "^0.13" derive_more = "^0.99" -tempfile = "^3.9" +tempfile = "^3.11" ## Networking & Async -tokio = { version = "^1.35", optional = true, features = ["rt-multi-thread"] } +tokio = { version = "^1.39", optional = true, features = ["rt-multi-thread"] } rust-s3 = { version = "^0.34", optional = true } futures = { version = "^0.3", optional = true } diff --git a/test_helper/Cargo.toml b/test_helper/Cargo.toml index 0bd4cc62f..4684c719f 100644 --- a/test_helper/Cargo.toml +++ b/test_helper/Cargo.toml @@ -25,4 +25,4 @@ xvc-logging = { version = "0.6.10", path = "../logging/" } rand = "^0.8" log = "^0.4" anyhow = "^1.0" -clap = { version = "^4.4", features = ["derive"] } +clap = { version = "^4.5", features = ["derive"] } From c43cdb12b54ffce8a2ee8a5f6ab1f8b407489a54 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Mon, 5 Aug 2024 21:16:00 +0300 Subject: [PATCH 145/257] Changelog --- CHANGELOG.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7fce520fe..8d087545f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,7 @@ ## Unreleased +- Bump dependencies + ## 0.6.10 (2024-08-04) - PR: From 58f55cffa778cf03abded88fce541a9f243614cc Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Tue, 6 Aug 2024 11:14:11 +0300 Subject: [PATCH 146/257] Fix check_ignore not reading the config --- core/src/check_ignore/mod.rs | 4 ++-- core/src/lib.rs | 7 ++++++- 2 files changed, 8 insertions(+), 3 deletions(-) diff --git a/core/src/check_ignore/mod.rs b/core/src/check_ignore/mod.rs index ceb313c62..e64146771 100644 --- a/core/src/check_ignore/mod.rs +++ b/core/src/check_ignore/mod.rs @@ -43,8 +43,8 @@ pub struct CheckIgnoreCLI { impl UpdateFromXvcConfig for CheckIgnoreCLI { fn update_from_conf(self, conf: &XvcConfig) -> xvc_config::error::Result> { - let details = self.details || conf.get_bool("check_ignore.details")?.option; - let non_matching = self.non_matching || conf.get_bool("check_ignore.non_matching")?.option; + let details = self.details || conf.get_bool("check-ignore.details")?.option; + let non_matching = self.non_matching; let ignore_filename = self.ignore_filename.clone(); Ok(Box::new(Self { details, diff --git a/core/src/lib.rs b/core/src/lib.rs index 207280777..2009cdb4a 100755 --- a/core/src/lib.rs +++ b/core/src/lib.rs @@ -48,6 +48,8 @@ pub use util::file::{all_paths_and_metadata, dir_includes, glob_includes, glob_p pub use util::pmp::XvcPathMetadataProvider; pub use util::XvcPathMetadataMap; +pub use xvc_walker::PEAK_ALLOC; + /// Channel size for [crossbeam_channel::bounded] used across the library. /// TODO: This can be configurable for smaller/larger RAM sizes. pub const CHANNEL_BOUND: usize = 1000000; @@ -222,7 +224,10 @@ default = "default" default_params_file = "params.yaml" # Number of command processes to run concurrently process_pool_size = 4 -# + +[check-ignore] +# Show details by default +details = false "##, guid = guid, From d01edab7166417d76ffa368ce0062f49f3a7959f Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Tue, 6 Aug 2024 11:14:52 +0300 Subject: [PATCH 147/257] Bump version --- config/Cargo.toml | 6 +++--- core/Cargo.toml | 13 +++++++------ ecs/Cargo.toml | 4 ++-- file/Cargo.toml | 17 +++++++++-------- lib/Cargo.toml | 20 ++++++++++---------- logging/Cargo.toml | 3 ++- pipeline/Cargo.toml | 16 ++++++++-------- storage/Cargo.toml | 14 +++++++------- test_helper/Cargo.toml | 4 ++-- walker/Cargo.toml | 7 ++++--- 10 files changed, 54 insertions(+), 50 deletions(-) diff --git a/config/Cargo.toml b/config/Cargo.toml index 30add05d0..80b4d6677 100644 --- a/config/Cargo.toml +++ b/config/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-config" -version = "0.6.10" +version = "0.6.11-alpha.5" edition = "2021" description = "Xvc configuration management" authors = ["Emre Şahin "] @@ -16,8 +16,8 @@ name = "xvc_config" crate-type = ["rlib"] [dependencies] -xvc-logging = { version = "0.6.10", path = "../logging" } -xvc-walker = { version = "0.6.10", path = "../walker" } +xvc-logging = { version = "0.6.11-alpha.5", path = "../logging" } +xvc-walker = { version = "0.6.11-alpha.5", path = "../walker" } ## Cli and config diff --git a/core/Cargo.toml b/core/Cargo.toml index cd4f28c04..fe919e8c2 100644 --- a/core/Cargo.toml +++ b/core/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-core" -version = "0.6.10" +version = "0.6.11-alpha.5" edition = "2021" description = "Xvc core for common elements for all commands" authors = ["Emre Şahin "] @@ -16,10 +16,10 @@ name = "xvc_core" crate-type = ["rlib"] [dependencies] -xvc-config = { version = "0.6.10", path = "../config" } -xvc-logging = { version = "0.6.10", path = "../logging" } -xvc-ecs = { version = "0.6.10", path = "../ecs" } -xvc-walker = { version = "0.6.10", path = "../walker" } +xvc-config = { version = "0.6.11-alpha.5", path = "../config" } +xvc-logging = { version = "0.6.11-alpha.5", path = "../logging" } +xvc-ecs = { version = "0.6.11-alpha.5", path = "../ecs" } +xvc-walker = { version = "0.6.11-alpha.5", path = "../walker" } ## Cli and config clap = { version = "^4.5", features = ["derive"] } @@ -60,6 +60,7 @@ glob = "^0.3" thiserror = "^1.0" anyhow = "^1.0" log = "^0.4" +peak_alloc = "^0.2" ### meta-logging-in-format is required for sled: https://github.com/spacejam/sled/issues/1384 fern = { version = "^0.6", features = ["colored"] } @@ -80,6 +81,6 @@ itertools = "^0.13" [dev-dependencies] -xvc-test-helper = { version = "0.6.10", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.11-alpha.5", path = "../test_helper/" } proptest = "^1.5" test-case = "^3.3" diff --git a/ecs/Cargo.toml b/ecs/Cargo.toml index 211b701db..1b9d58717 100644 --- a/ecs/Cargo.toml +++ b/ecs/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-ecs" -version = "0.6.10" +version = "0.6.11-alpha.5" edition = "2021" description = "Entity-Component System for Xvc" authors = ["Emre Şahin "] @@ -16,7 +16,7 @@ name = "xvc_ecs" crate-type = ["rlib"] [dependencies] -xvc-logging = { version = "0.6.10", path = "../logging" } +xvc-logging = { version = "0.6.11-alpha.5", path = "../logging" } ## Serialization serde = { version = "^1.0", features = ["derive"] } diff --git a/file/Cargo.toml b/file/Cargo.toml index cab53c52a..d4b671eea 100644 --- a/file/Cargo.toml +++ b/file/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-file" -version = "0.6.10" +version = "0.6.11-alpha.5" edition = "2021" description = "File tracking, versioning, upload and download functions for Xvc" authors = ["Emre Şahin "] @@ -21,12 +21,12 @@ test = true bench = true [dependencies] -xvc-logging = { version = "0.6.10", path = "../logging" } -xvc-config = { version = "0.6.10", path = "../config" } -xvc-core = { version = "0.6.10", path = "../core" } -xvc-ecs = { version = "0.6.10", path = "../ecs" } -xvc-walker = { version = "0.6.10", path = "../walker" } -xvc-storage = { version = "0.6.10", path = "../storage", default-features = false } +xvc-logging = { version = "0.6.11-alpha.5", path = "../logging" } +xvc-config = { version = "0.6.11-alpha.5", path = "../config" } +xvc-core = { version = "0.6.11-alpha.5", path = "../core" } +xvc-ecs = { version = "0.6.11-alpha.5", path = "../ecs" } +xvc-walker = { version = "0.6.11-alpha.5", path = "../walker" } +xvc-storage = { version = "0.6.11-alpha.5", path = "../storage", default-features = false } ## Cli and config @@ -67,6 +67,7 @@ glob = "^0.3" thiserror = "^1.0" log = "^0.4" anyhow = "^1.0" +peak_alloc = "^0.2" ### meta-logging-in-format is required for sled: https://github.com/spacejam/sled/issues/1384 fern = { version = "^0.6", features = ["colored"] } @@ -94,5 +95,5 @@ default = ["reflink"] reflink = ["dep:reflink"] [dev-dependencies] -xvc-test-helper = { version = "0.6.10", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.11-alpha.5", path = "../test_helper/" } shellfn = "^0.1" diff --git a/lib/Cargo.toml b/lib/Cargo.toml index ea5688936..592a4398b 100644 --- a/lib/Cargo.toml +++ b/lib/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc" -version = "0.6.10" +version = "0.6.11-alpha.5" edition = "2021" description = "An MLOps tool to manage data files and pipelines on top of Git" authors = ["Emre Şahin "] @@ -20,14 +20,14 @@ name = "xvc" path = "src/main.rs" [dependencies] -xvc-config = { version = "0.6.10", path = "../config" } -xvc-core = { version = "0.6.10", path = "../core" } -xvc-logging = { version = "0.6.10", path = "../logging" } -xvc-ecs = { version = "0.6.10", path = "../ecs" } -xvc-file = { version = "0.6.10", path = "../file", default-features = false } -xvc-pipeline = { version = "0.6.10", path = "../pipeline" } -xvc-walker = { version = "0.6.10", path = "../walker" } -xvc-storage = { version = "0.6.10", path = "../storage", default-features = false } +xvc-config = { version = "0.6.11-alpha.5", path = "../config" } +xvc-core = { version = "0.6.11-alpha.5", path = "../core" } +xvc-logging = { version = "0.6.11-alpha.5", path = "../logging" } +xvc-ecs = { version = "0.6.11-alpha.5", path = "../ecs" } +xvc-file = { version = "0.6.11-alpha.5", path = "../file", default-features = false } +xvc-pipeline = { version = "0.6.11-alpha.5", path = "../pipeline" } +xvc-walker = { version = "0.6.11-alpha.5", path = "../walker" } +xvc-storage = { version = "0.6.11-alpha.5", path = "../storage", default-features = false } ## Cli and config clap = { version = "^4.5", features = ["derive", "cargo"] } @@ -133,4 +133,4 @@ shellfn = "^0.1" test-case = "^3.3" trycmd = "^0.15" which = "^6.0" -xvc-test-helper = { version = "0.6.10", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.11-alpha.5", path = "../test_helper/" } diff --git a/logging/Cargo.toml b/logging/Cargo.toml index a928aa413..c253bac5c 100644 --- a/logging/Cargo.toml +++ b/logging/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-logging" -version = "0.6.10" +version = "0.6.11-alpha.5" edition = "2021" description = "Logging crate for Xvc" authors = ["Emre Şahin "] @@ -18,6 +18,7 @@ crate-type = ["rlib"] [dependencies] ## Logging and errors +peak_alloc = "^0.2" log = "^0.4" ### meta-logging-in-format is required for sled: https://github.com/spacejam/sled/issues/1384 # We don't use sled anymore, keep the above note for future reference diff --git a/pipeline/Cargo.toml b/pipeline/Cargo.toml index 81be11eae..169c6971b 100644 --- a/pipeline/Cargo.toml +++ b/pipeline/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-pipeline" -version = "0.6.10" +version = "0.6.11-alpha.5" edition = "2021" description = "Xvc data pipeline management" authors = ["Emre Şahin "] @@ -19,12 +19,12 @@ default = [] bundled-sqlite = ["rusqlite/bundled"] [dependencies] -xvc-config = { version = "0.6.10", path = "../config" } -xvc-core = { version = "0.6.10", path = "../core" } -xvc-ecs = { version = "0.6.10", path = "../ecs" } -xvc-logging = { version = "0.6.10", path = "../logging" } -xvc-walker = { version = "0.6.10", path = "../walker" } -xvc-file = { version = "0.6.10", path = "../file", default-features = false } +xvc-config = { version = "0.6.11-alpha.5", path = "../config" } +xvc-core = { version = "0.6.11-alpha.5", path = "../core" } +xvc-ecs = { version = "0.6.11-alpha.5", path = "../ecs" } +xvc-logging = { version = "0.6.11-alpha.5", path = "../logging" } +xvc-walker = { version = "0.6.11-alpha.5", path = "../walker" } +xvc-file = { version = "0.6.11-alpha.5", path = "../file", default-features = false } ## Cli and config clap = { version = "^4.5", features = ["derive"] } @@ -98,5 +98,5 @@ itertools = "^0.13" derive_more = "^0.99" [dev-dependencies] -xvc-test-helper = { version = "0.6.10", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.11-alpha.5", path = "../test_helper/" } test-case = "^3.3" diff --git a/storage/Cargo.toml b/storage/Cargo.toml index 65fc37319..8f471fdd9 100644 --- a/storage/Cargo.toml +++ b/storage/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-storage" -version = "0.6.10" +version = "0.6.11-alpha.5" edition = "2021" description = "Xvc remote and local storage management" authors = ["Emre Şahin "] @@ -16,11 +16,11 @@ name = "xvc_storage" crate-type = ["rlib"] [dependencies] -xvc-logging = { version = "0.6.10", path = "../logging" } -xvc-config = { version = "0.6.10", path = "../config" } -xvc-core = { version = "0.6.10", path = "../core" } -xvc-ecs = { version = "0.6.10", path = "../ecs" } -xvc-walker = { version = "0.6.10", path = "../walker" } +xvc-logging = { version = "0.6.11-alpha.5", path = "../logging" } +xvc-config = { version = "0.6.11-alpha.5", path = "../config" } +xvc-core = { version = "0.6.11-alpha.5", path = "../core" } +xvc-ecs = { version = "0.6.11-alpha.5", path = "../ecs" } +xvc-walker = { version = "0.6.11-alpha.5", path = "../walker" } ## Cli and config clap = { version = "^4.5", features = ["derive"] } @@ -104,7 +104,7 @@ bundled-openssl = ["openssl/vendored"] [dev-dependencies] -xvc-test-helper = { version = "0.6.10", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.11-alpha.5", path = "../test_helper/" } shellfn = "^0.1" [package.metadata.cargo-udeps.ignore] diff --git a/test_helper/Cargo.toml b/test_helper/Cargo.toml index 4684c719f..e6bb6da3f 100644 --- a/test_helper/Cargo.toml +++ b/test_helper/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-test-helper" -version = "0.6.10" +version = "0.6.11-alpha.5" edition = "2021" description = "Unit test helper functions for Xvc" authors = ["Emre Şahin "] @@ -20,7 +20,7 @@ path = "src/main.rs" # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html [dependencies] -xvc-logging = { version = "0.6.10", path = "../logging/" } +xvc-logging = { version = "0.6.11-alpha.5", path = "../logging/" } rand = "^0.8" log = "^0.4" diff --git a/walker/Cargo.toml b/walker/Cargo.toml index 7eb5c786f..198971721 100644 --- a/walker/Cargo.toml +++ b/walker/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-walker" -version = "0.6.10" +version = "0.6.11-alpha.5" edition = "2021" description = "Xvc parallel file system walker with ignore features" authors = ["Emre Şahin "] @@ -16,7 +16,7 @@ name = "xvc_walker" crate-type = ["rlib"] [dependencies] -xvc-logging = { version = "0.6.10", path = "../logging" } +xvc-logging = { version = "0.6.11-alpha.5", path = "../logging" } globset = "^0.4" ## Parallelization @@ -32,13 +32,14 @@ dashmap = "^6" thiserror = "^1.0" anyhow = "^1.0" log = "^0.4" +peak_alloc = "^0.2" ## Misc itertools = "^0.13" regex = "^1.10" [dev-dependencies] -xvc-test-helper = { path = "../test_helper/", version = "0.6.10" } +xvc-test-helper = { path = "../test_helper/", version = "0.6.11-alpha.5" } test-case = "^3.3" [package.metadata.cargo-udeps.ignore] From 66586fd5966f5f548025e8d2c93089d3e0d11dd4 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Tue, 6 Aug 2024 11:16:18 +0300 Subject: [PATCH 148/257] Bump version --- config/Cargo.toml | 6 +++--- core/Cargo.toml | 12 ++++++------ ecs/Cargo.toml | 4 ++-- file/Cargo.toml | 16 ++++++++-------- lib/Cargo.toml | 20 ++++++++++---------- logging/Cargo.toml | 2 +- pipeline/Cargo.toml | 16 ++++++++-------- storage/Cargo.toml | 14 +++++++------- test_helper/Cargo.toml | 4 ++-- walker/Cargo.toml | 6 +++--- 10 files changed, 50 insertions(+), 50 deletions(-) diff --git a/config/Cargo.toml b/config/Cargo.toml index 80b4d6677..f1d647b6e 100644 --- a/config/Cargo.toml +++ b/config/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-config" -version = "0.6.11-alpha.5" +version = "0.6.11-alpha.6" edition = "2021" description = "Xvc configuration management" authors = ["Emre Şahin "] @@ -16,8 +16,8 @@ name = "xvc_config" crate-type = ["rlib"] [dependencies] -xvc-logging = { version = "0.6.11-alpha.5", path = "../logging" } -xvc-walker = { version = "0.6.11-alpha.5", path = "../walker" } +xvc-logging = { version = "0.6.11-alpha.6", path = "../logging" } +xvc-walker = { version = "0.6.11-alpha.6", path = "../walker" } ## Cli and config diff --git a/core/Cargo.toml b/core/Cargo.toml index fe919e8c2..49c63231e 100644 --- a/core/Cargo.toml +++ b/core/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-core" -version = "0.6.11-alpha.5" +version = "0.6.11-alpha.6" edition = "2021" description = "Xvc core for common elements for all commands" authors = ["Emre Şahin "] @@ -16,10 +16,10 @@ name = "xvc_core" crate-type = ["rlib"] [dependencies] -xvc-config = { version = "0.6.11-alpha.5", path = "../config" } -xvc-logging = { version = "0.6.11-alpha.5", path = "../logging" } -xvc-ecs = { version = "0.6.11-alpha.5", path = "../ecs" } -xvc-walker = { version = "0.6.11-alpha.5", path = "../walker" } +xvc-config = { version = "0.6.11-alpha.6", path = "../config" } +xvc-logging = { version = "0.6.11-alpha.6", path = "../logging" } +xvc-ecs = { version = "0.6.11-alpha.6", path = "../ecs" } +xvc-walker = { version = "0.6.11-alpha.6", path = "../walker" } ## Cli and config clap = { version = "^4.5", features = ["derive"] } @@ -81,6 +81,6 @@ itertools = "^0.13" [dev-dependencies] -xvc-test-helper = { version = "0.6.11-alpha.5", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.11-alpha.6", path = "../test_helper/" } proptest = "^1.5" test-case = "^3.3" diff --git a/ecs/Cargo.toml b/ecs/Cargo.toml index 1b9d58717..ba691160a 100644 --- a/ecs/Cargo.toml +++ b/ecs/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-ecs" -version = "0.6.11-alpha.5" +version = "0.6.11-alpha.6" edition = "2021" description = "Entity-Component System for Xvc" authors = ["Emre Şahin "] @@ -16,7 +16,7 @@ name = "xvc_ecs" crate-type = ["rlib"] [dependencies] -xvc-logging = { version = "0.6.11-alpha.5", path = "../logging" } +xvc-logging = { version = "0.6.11-alpha.6", path = "../logging" } ## Serialization serde = { version = "^1.0", features = ["derive"] } diff --git a/file/Cargo.toml b/file/Cargo.toml index d4b671eea..372147ba4 100644 --- a/file/Cargo.toml +++ b/file/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-file" -version = "0.6.11-alpha.5" +version = "0.6.11-alpha.6" edition = "2021" description = "File tracking, versioning, upload and download functions for Xvc" authors = ["Emre Şahin "] @@ -21,12 +21,12 @@ test = true bench = true [dependencies] -xvc-logging = { version = "0.6.11-alpha.5", path = "../logging" } -xvc-config = { version = "0.6.11-alpha.5", path = "../config" } -xvc-core = { version = "0.6.11-alpha.5", path = "../core" } -xvc-ecs = { version = "0.6.11-alpha.5", path = "../ecs" } -xvc-walker = { version = "0.6.11-alpha.5", path = "../walker" } -xvc-storage = { version = "0.6.11-alpha.5", path = "../storage", default-features = false } +xvc-logging = { version = "0.6.11-alpha.6", path = "../logging" } +xvc-config = { version = "0.6.11-alpha.6", path = "../config" } +xvc-core = { version = "0.6.11-alpha.6", path = "../core" } +xvc-ecs = { version = "0.6.11-alpha.6", path = "../ecs" } +xvc-walker = { version = "0.6.11-alpha.6", path = "../walker" } +xvc-storage = { version = "0.6.11-alpha.6", path = "../storage", default-features = false } ## Cli and config @@ -95,5 +95,5 @@ default = ["reflink"] reflink = ["dep:reflink"] [dev-dependencies] -xvc-test-helper = { version = "0.6.11-alpha.5", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.11-alpha.6", path = "../test_helper/" } shellfn = "^0.1" diff --git a/lib/Cargo.toml b/lib/Cargo.toml index 592a4398b..bd7faa780 100644 --- a/lib/Cargo.toml +++ b/lib/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc" -version = "0.6.11-alpha.5" +version = "0.6.11-alpha.6" edition = "2021" description = "An MLOps tool to manage data files and pipelines on top of Git" authors = ["Emre Şahin "] @@ -20,14 +20,14 @@ name = "xvc" path = "src/main.rs" [dependencies] -xvc-config = { version = "0.6.11-alpha.5", path = "../config" } -xvc-core = { version = "0.6.11-alpha.5", path = "../core" } -xvc-logging = { version = "0.6.11-alpha.5", path = "../logging" } -xvc-ecs = { version = "0.6.11-alpha.5", path = "../ecs" } -xvc-file = { version = "0.6.11-alpha.5", path = "../file", default-features = false } -xvc-pipeline = { version = "0.6.11-alpha.5", path = "../pipeline" } -xvc-walker = { version = "0.6.11-alpha.5", path = "../walker" } -xvc-storage = { version = "0.6.11-alpha.5", path = "../storage", default-features = false } +xvc-config = { version = "0.6.11-alpha.6", path = "../config" } +xvc-core = { version = "0.6.11-alpha.6", path = "../core" } +xvc-logging = { version = "0.6.11-alpha.6", path = "../logging" } +xvc-ecs = { version = "0.6.11-alpha.6", path = "../ecs" } +xvc-file = { version = "0.6.11-alpha.6", path = "../file", default-features = false } +xvc-pipeline = { version = "0.6.11-alpha.6", path = "../pipeline" } +xvc-walker = { version = "0.6.11-alpha.6", path = "../walker" } +xvc-storage = { version = "0.6.11-alpha.6", path = "../storage", default-features = false } ## Cli and config clap = { version = "^4.5", features = ["derive", "cargo"] } @@ -133,4 +133,4 @@ shellfn = "^0.1" test-case = "^3.3" trycmd = "^0.15" which = "^6.0" -xvc-test-helper = { version = "0.6.11-alpha.5", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.11-alpha.6", path = "../test_helper/" } diff --git a/logging/Cargo.toml b/logging/Cargo.toml index c253bac5c..708ee08d8 100644 --- a/logging/Cargo.toml +++ b/logging/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-logging" -version = "0.6.11-alpha.5" +version = "0.6.11-alpha.6" edition = "2021" description = "Logging crate for Xvc" authors = ["Emre Şahin "] diff --git a/pipeline/Cargo.toml b/pipeline/Cargo.toml index 169c6971b..105fd1513 100644 --- a/pipeline/Cargo.toml +++ b/pipeline/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-pipeline" -version = "0.6.11-alpha.5" +version = "0.6.11-alpha.6" edition = "2021" description = "Xvc data pipeline management" authors = ["Emre Şahin "] @@ -19,12 +19,12 @@ default = [] bundled-sqlite = ["rusqlite/bundled"] [dependencies] -xvc-config = { version = "0.6.11-alpha.5", path = "../config" } -xvc-core = { version = "0.6.11-alpha.5", path = "../core" } -xvc-ecs = { version = "0.6.11-alpha.5", path = "../ecs" } -xvc-logging = { version = "0.6.11-alpha.5", path = "../logging" } -xvc-walker = { version = "0.6.11-alpha.5", path = "../walker" } -xvc-file = { version = "0.6.11-alpha.5", path = "../file", default-features = false } +xvc-config = { version = "0.6.11-alpha.6", path = "../config" } +xvc-core = { version = "0.6.11-alpha.6", path = "../core" } +xvc-ecs = { version = "0.6.11-alpha.6", path = "../ecs" } +xvc-logging = { version = "0.6.11-alpha.6", path = "../logging" } +xvc-walker = { version = "0.6.11-alpha.6", path = "../walker" } +xvc-file = { version = "0.6.11-alpha.6", path = "../file", default-features = false } ## Cli and config clap = { version = "^4.5", features = ["derive"] } @@ -98,5 +98,5 @@ itertools = "^0.13" derive_more = "^0.99" [dev-dependencies] -xvc-test-helper = { version = "0.6.11-alpha.5", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.11-alpha.6", path = "../test_helper/" } test-case = "^3.3" diff --git a/storage/Cargo.toml b/storage/Cargo.toml index 8f471fdd9..95c23f935 100644 --- a/storage/Cargo.toml +++ b/storage/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-storage" -version = "0.6.11-alpha.5" +version = "0.6.11-alpha.6" edition = "2021" description = "Xvc remote and local storage management" authors = ["Emre Şahin "] @@ -16,11 +16,11 @@ name = "xvc_storage" crate-type = ["rlib"] [dependencies] -xvc-logging = { version = "0.6.11-alpha.5", path = "../logging" } -xvc-config = { version = "0.6.11-alpha.5", path = "../config" } -xvc-core = { version = "0.6.11-alpha.5", path = "../core" } -xvc-ecs = { version = "0.6.11-alpha.5", path = "../ecs" } -xvc-walker = { version = "0.6.11-alpha.5", path = "../walker" } +xvc-logging = { version = "0.6.11-alpha.6", path = "../logging" } +xvc-config = { version = "0.6.11-alpha.6", path = "../config" } +xvc-core = { version = "0.6.11-alpha.6", path = "../core" } +xvc-ecs = { version = "0.6.11-alpha.6", path = "../ecs" } +xvc-walker = { version = "0.6.11-alpha.6", path = "../walker" } ## Cli and config clap = { version = "^4.5", features = ["derive"] } @@ -104,7 +104,7 @@ bundled-openssl = ["openssl/vendored"] [dev-dependencies] -xvc-test-helper = { version = "0.6.11-alpha.5", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.11-alpha.6", path = "../test_helper/" } shellfn = "^0.1" [package.metadata.cargo-udeps.ignore] diff --git a/test_helper/Cargo.toml b/test_helper/Cargo.toml index e6bb6da3f..085a1600c 100644 --- a/test_helper/Cargo.toml +++ b/test_helper/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-test-helper" -version = "0.6.11-alpha.5" +version = "0.6.11-alpha.6" edition = "2021" description = "Unit test helper functions for Xvc" authors = ["Emre Şahin "] @@ -20,7 +20,7 @@ path = "src/main.rs" # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html [dependencies] -xvc-logging = { version = "0.6.11-alpha.5", path = "../logging/" } +xvc-logging = { version = "0.6.11-alpha.6", path = "../logging/" } rand = "^0.8" log = "^0.4" diff --git a/walker/Cargo.toml b/walker/Cargo.toml index 198971721..a93d909cc 100644 --- a/walker/Cargo.toml +++ b/walker/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-walker" -version = "0.6.11-alpha.5" +version = "0.6.11-alpha.6" edition = "2021" description = "Xvc parallel file system walker with ignore features" authors = ["Emre Şahin "] @@ -16,7 +16,7 @@ name = "xvc_walker" crate-type = ["rlib"] [dependencies] -xvc-logging = { version = "0.6.11-alpha.5", path = "../logging" } +xvc-logging = { version = "0.6.11-alpha.6", path = "../logging" } globset = "^0.4" ## Parallelization @@ -39,7 +39,7 @@ itertools = "^0.13" regex = "^1.10" [dev-dependencies] -xvc-test-helper = { path = "../test_helper/", version = "0.6.11-alpha.5" } +xvc-test-helper = { path = "../test_helper/", version = "0.6.11-alpha.6" } test-case = "^3.3" [package.metadata.cargo-udeps.ignore] From 52c687009cb4a69081c6e686343d0af55f7e22f8 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Tue, 6 Aug 2024 11:20:03 +0300 Subject: [PATCH 149/257] Bump version --- config/Cargo.toml | 6 +++--- core/Cargo.toml | 12 ++++++------ ecs/Cargo.toml | 4 ++-- file/Cargo.toml | 16 ++++++++-------- lib/Cargo.toml | 20 ++++++++++---------- logging/Cargo.toml | 2 +- pipeline/Cargo.toml | 16 ++++++++-------- storage/Cargo.toml | 14 +++++++------- test_helper/Cargo.toml | 4 ++-- walker/Cargo.toml | 6 +++--- 10 files changed, 50 insertions(+), 50 deletions(-) diff --git a/config/Cargo.toml b/config/Cargo.toml index f1d647b6e..8bdeaaebe 100644 --- a/config/Cargo.toml +++ b/config/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-config" -version = "0.6.11-alpha.6" +version = "0.6.11-alpha.7" edition = "2021" description = "Xvc configuration management" authors = ["Emre Şahin "] @@ -16,8 +16,8 @@ name = "xvc_config" crate-type = ["rlib"] [dependencies] -xvc-logging = { version = "0.6.11-alpha.6", path = "../logging" } -xvc-walker = { version = "0.6.11-alpha.6", path = "../walker" } +xvc-logging = { version = "0.6.11-alpha.7", path = "../logging" } +xvc-walker = { version = "0.6.11-alpha.7", path = "../walker" } ## Cli and config diff --git a/core/Cargo.toml b/core/Cargo.toml index 49c63231e..e760970ec 100644 --- a/core/Cargo.toml +++ b/core/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-core" -version = "0.6.11-alpha.6" +version = "0.6.11-alpha.7" edition = "2021" description = "Xvc core for common elements for all commands" authors = ["Emre Şahin "] @@ -16,10 +16,10 @@ name = "xvc_core" crate-type = ["rlib"] [dependencies] -xvc-config = { version = "0.6.11-alpha.6", path = "../config" } -xvc-logging = { version = "0.6.11-alpha.6", path = "../logging" } -xvc-ecs = { version = "0.6.11-alpha.6", path = "../ecs" } -xvc-walker = { version = "0.6.11-alpha.6", path = "../walker" } +xvc-config = { version = "0.6.11-alpha.7", path = "../config" } +xvc-logging = { version = "0.6.11-alpha.7", path = "../logging" } +xvc-ecs = { version = "0.6.11-alpha.7", path = "../ecs" } +xvc-walker = { version = "0.6.11-alpha.7", path = "../walker" } ## Cli and config clap = { version = "^4.5", features = ["derive"] } @@ -81,6 +81,6 @@ itertools = "^0.13" [dev-dependencies] -xvc-test-helper = { version = "0.6.11-alpha.6", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.11-alpha.7", path = "../test_helper/" } proptest = "^1.5" test-case = "^3.3" diff --git a/ecs/Cargo.toml b/ecs/Cargo.toml index ba691160a..4f0d87584 100644 --- a/ecs/Cargo.toml +++ b/ecs/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-ecs" -version = "0.6.11-alpha.6" +version = "0.6.11-alpha.7" edition = "2021" description = "Entity-Component System for Xvc" authors = ["Emre Şahin "] @@ -16,7 +16,7 @@ name = "xvc_ecs" crate-type = ["rlib"] [dependencies] -xvc-logging = { version = "0.6.11-alpha.6", path = "../logging" } +xvc-logging = { version = "0.6.11-alpha.7", path = "../logging" } ## Serialization serde = { version = "^1.0", features = ["derive"] } diff --git a/file/Cargo.toml b/file/Cargo.toml index 372147ba4..4dfbc5743 100644 --- a/file/Cargo.toml +++ b/file/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-file" -version = "0.6.11-alpha.6" +version = "0.6.11-alpha.7" edition = "2021" description = "File tracking, versioning, upload and download functions for Xvc" authors = ["Emre Şahin "] @@ -21,12 +21,12 @@ test = true bench = true [dependencies] -xvc-logging = { version = "0.6.11-alpha.6", path = "../logging" } -xvc-config = { version = "0.6.11-alpha.6", path = "../config" } -xvc-core = { version = "0.6.11-alpha.6", path = "../core" } -xvc-ecs = { version = "0.6.11-alpha.6", path = "../ecs" } -xvc-walker = { version = "0.6.11-alpha.6", path = "../walker" } -xvc-storage = { version = "0.6.11-alpha.6", path = "../storage", default-features = false } +xvc-logging = { version = "0.6.11-alpha.7", path = "../logging" } +xvc-config = { version = "0.6.11-alpha.7", path = "../config" } +xvc-core = { version = "0.6.11-alpha.7", path = "../core" } +xvc-ecs = { version = "0.6.11-alpha.7", path = "../ecs" } +xvc-walker = { version = "0.6.11-alpha.7", path = "../walker" } +xvc-storage = { version = "0.6.11-alpha.7", path = "../storage", default-features = false } ## Cli and config @@ -95,5 +95,5 @@ default = ["reflink"] reflink = ["dep:reflink"] [dev-dependencies] -xvc-test-helper = { version = "0.6.11-alpha.6", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.11-alpha.7", path = "../test_helper/" } shellfn = "^0.1" diff --git a/lib/Cargo.toml b/lib/Cargo.toml index bd7faa780..98c79309d 100644 --- a/lib/Cargo.toml +++ b/lib/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc" -version = "0.6.11-alpha.6" +version = "0.6.11-alpha.7" edition = "2021" description = "An MLOps tool to manage data files and pipelines on top of Git" authors = ["Emre Şahin "] @@ -20,14 +20,14 @@ name = "xvc" path = "src/main.rs" [dependencies] -xvc-config = { version = "0.6.11-alpha.6", path = "../config" } -xvc-core = { version = "0.6.11-alpha.6", path = "../core" } -xvc-logging = { version = "0.6.11-alpha.6", path = "../logging" } -xvc-ecs = { version = "0.6.11-alpha.6", path = "../ecs" } -xvc-file = { version = "0.6.11-alpha.6", path = "../file", default-features = false } -xvc-pipeline = { version = "0.6.11-alpha.6", path = "../pipeline" } -xvc-walker = { version = "0.6.11-alpha.6", path = "../walker" } -xvc-storage = { version = "0.6.11-alpha.6", path = "../storage", default-features = false } +xvc-config = { version = "0.6.11-alpha.7", path = "../config" } +xvc-core = { version = "0.6.11-alpha.7", path = "../core" } +xvc-logging = { version = "0.6.11-alpha.7", path = "../logging" } +xvc-ecs = { version = "0.6.11-alpha.7", path = "../ecs" } +xvc-file = { version = "0.6.11-alpha.7", path = "../file", default-features = false } +xvc-pipeline = { version = "0.6.11-alpha.7", path = "../pipeline" } +xvc-walker = { version = "0.6.11-alpha.7", path = "../walker" } +xvc-storage = { version = "0.6.11-alpha.7", path = "../storage", default-features = false } ## Cli and config clap = { version = "^4.5", features = ["derive", "cargo"] } @@ -133,4 +133,4 @@ shellfn = "^0.1" test-case = "^3.3" trycmd = "^0.15" which = "^6.0" -xvc-test-helper = { version = "0.6.11-alpha.6", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.11-alpha.7", path = "../test_helper/" } diff --git a/logging/Cargo.toml b/logging/Cargo.toml index 708ee08d8..7dfd62727 100644 --- a/logging/Cargo.toml +++ b/logging/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-logging" -version = "0.6.11-alpha.6" +version = "0.6.11-alpha.7" edition = "2021" description = "Logging crate for Xvc" authors = ["Emre Şahin "] diff --git a/pipeline/Cargo.toml b/pipeline/Cargo.toml index 105fd1513..fe0e8fd9a 100644 --- a/pipeline/Cargo.toml +++ b/pipeline/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-pipeline" -version = "0.6.11-alpha.6" +version = "0.6.11-alpha.7" edition = "2021" description = "Xvc data pipeline management" authors = ["Emre Şahin "] @@ -19,12 +19,12 @@ default = [] bundled-sqlite = ["rusqlite/bundled"] [dependencies] -xvc-config = { version = "0.6.11-alpha.6", path = "../config" } -xvc-core = { version = "0.6.11-alpha.6", path = "../core" } -xvc-ecs = { version = "0.6.11-alpha.6", path = "../ecs" } -xvc-logging = { version = "0.6.11-alpha.6", path = "../logging" } -xvc-walker = { version = "0.6.11-alpha.6", path = "../walker" } -xvc-file = { version = "0.6.11-alpha.6", path = "../file", default-features = false } +xvc-config = { version = "0.6.11-alpha.7", path = "../config" } +xvc-core = { version = "0.6.11-alpha.7", path = "../core" } +xvc-ecs = { version = "0.6.11-alpha.7", path = "../ecs" } +xvc-logging = { version = "0.6.11-alpha.7", path = "../logging" } +xvc-walker = { version = "0.6.11-alpha.7", path = "../walker" } +xvc-file = { version = "0.6.11-alpha.7", path = "../file", default-features = false } ## Cli and config clap = { version = "^4.5", features = ["derive"] } @@ -98,5 +98,5 @@ itertools = "^0.13" derive_more = "^0.99" [dev-dependencies] -xvc-test-helper = { version = "0.6.11-alpha.6", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.11-alpha.7", path = "../test_helper/" } test-case = "^3.3" diff --git a/storage/Cargo.toml b/storage/Cargo.toml index 95c23f935..461accf44 100644 --- a/storage/Cargo.toml +++ b/storage/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-storage" -version = "0.6.11-alpha.6" +version = "0.6.11-alpha.7" edition = "2021" description = "Xvc remote and local storage management" authors = ["Emre Şahin "] @@ -16,11 +16,11 @@ name = "xvc_storage" crate-type = ["rlib"] [dependencies] -xvc-logging = { version = "0.6.11-alpha.6", path = "../logging" } -xvc-config = { version = "0.6.11-alpha.6", path = "../config" } -xvc-core = { version = "0.6.11-alpha.6", path = "../core" } -xvc-ecs = { version = "0.6.11-alpha.6", path = "../ecs" } -xvc-walker = { version = "0.6.11-alpha.6", path = "../walker" } +xvc-logging = { version = "0.6.11-alpha.7", path = "../logging" } +xvc-config = { version = "0.6.11-alpha.7", path = "../config" } +xvc-core = { version = "0.6.11-alpha.7", path = "../core" } +xvc-ecs = { version = "0.6.11-alpha.7", path = "../ecs" } +xvc-walker = { version = "0.6.11-alpha.7", path = "../walker" } ## Cli and config clap = { version = "^4.5", features = ["derive"] } @@ -104,7 +104,7 @@ bundled-openssl = ["openssl/vendored"] [dev-dependencies] -xvc-test-helper = { version = "0.6.11-alpha.6", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.11-alpha.7", path = "../test_helper/" } shellfn = "^0.1" [package.metadata.cargo-udeps.ignore] diff --git a/test_helper/Cargo.toml b/test_helper/Cargo.toml index 085a1600c..a7f125cf2 100644 --- a/test_helper/Cargo.toml +++ b/test_helper/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-test-helper" -version = "0.6.11-alpha.6" +version = "0.6.11-alpha.7" edition = "2021" description = "Unit test helper functions for Xvc" authors = ["Emre Şahin "] @@ -20,7 +20,7 @@ path = "src/main.rs" # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html [dependencies] -xvc-logging = { version = "0.6.11-alpha.6", path = "../logging/" } +xvc-logging = { version = "0.6.11-alpha.7", path = "../logging/" } rand = "^0.8" log = "^0.4" diff --git a/walker/Cargo.toml b/walker/Cargo.toml index a93d909cc..5ef1af67b 100644 --- a/walker/Cargo.toml +++ b/walker/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-walker" -version = "0.6.11-alpha.6" +version = "0.6.11-alpha.7" edition = "2021" description = "Xvc parallel file system walker with ignore features" authors = ["Emre Şahin "] @@ -16,7 +16,7 @@ name = "xvc_walker" crate-type = ["rlib"] [dependencies] -xvc-logging = { version = "0.6.11-alpha.6", path = "../logging" } +xvc-logging = { version = "0.6.11-alpha.7", path = "../logging" } globset = "^0.4" ## Parallelization @@ -39,7 +39,7 @@ itertools = "^0.13" regex = "^1.10" [dev-dependencies] -xvc-test-helper = { path = "../test_helper/", version = "0.6.11-alpha.6" } +xvc-test-helper = { path = "../test_helper/", version = "0.6.11-alpha.7" } test-case = "^3.3" [package.metadata.cargo-udeps.ignore] From eda16588622f5de1664ca2836a25f18500bc800c Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Tue, 6 Aug 2024 21:43:54 +0300 Subject: [PATCH 150/257] debug with PEAK_ALLOC --- core/src/util/git.rs | 7 +++++ file/src/track/mod.rs | 15 +++++++++- logging/src/lib.rs | 1 + logging/src/logging/mod.rs | 56 -------------------------------------- walker/src/ignore_rules.rs | 28 ++++++++++++++++++- walker/src/lib.rs | 30 ++++++++++++++++++-- 6 files changed, 76 insertions(+), 61 deletions(-) delete mode 100644 logging/src/logging/mod.rs diff --git a/core/src/util/git.rs b/core/src/util/git.rs index c3258d4a0..1bf4652e1 100644 --- a/core/src/util/git.rs +++ b/core/src/util/git.rs @@ -1,11 +1,14 @@ //! Git operations use std::path::{Path, PathBuf}; +use xvc_logging::watch; use xvc_walker::{build_ignore_rules, AbsolutePath, IgnoreRules}; use crate::error::Result; use crate::GIT_DIR; +use crate::PEAK_ALLOC; + /// Check whether a path is inside a Git repository. /// It returns `None` if not, otherwise returns the closest directory with `.git`. /// It works by checking `.git` directories in parents, until no more parent left. @@ -27,9 +30,13 @@ pub fn inside_git(path: &Path) -> Option { /// Returns [xvc_walker::IgnoreRules] for `.gitignore` /// It's used to check whether a path is already ignored by Git. pub fn build_gitignore(git_root: &AbsolutePath) -> Result { + + watch!(PEAK_ALLOC.current_usage_as_mb()); let initial_rules = IgnoreRules::empty(git_root); + watch!(PEAK_ALLOC.current_usage_as_mb()); let rules = build_ignore_rules(initial_rules, git_root, ".gitignore")?; + watch!(PEAK_ALLOC.current_usage_as_mb()); Ok(rules) } diff --git a/file/src/track/mod.rs b/file/src/track/mod.rs index d26f9295f..6c4ca4f13 100644 --- a/file/src/track/mod.rs +++ b/file/src/track/mod.rs @@ -14,7 +14,7 @@ use xvc_config::FromConfigKey; use xvc_config::{UpdateFromXvcConfig, XvcConfig}; use xvc_core::util::git::build_gitignore; -use xvc_core::{ContentDigest, HashAlgorithm, XvcCachePath, XvcFileType, XvcMetadata, XvcRoot}; +use xvc_core::{ContentDigest, HashAlgorithm, XvcCachePath, XvcFileType, XvcMetadata, XvcRoot, PEAK_ALLOC}; use xvc_logging::{watch, XvcOutputSender}; use crate::carry_in::carry_in; @@ -32,6 +32,7 @@ use xvc_core::RecheckMethod; use xvc_core::XvcPath; use xvc_ecs::{HStore, XvcEntity}; + /// Add files for tracking with Xvc #[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, From, Parser)] #[command(rename_all = "kebab-case")] @@ -219,15 +220,26 @@ pub fn cmd_track( }) .collect(); + watch!(PEAK_ALLOC.current_usage_as_mb()); let current_gitignore = build_gitignore(xvc_root)?; + watch!(PEAK_ALLOC.current_usage_as_mb()); watch!(file_targets); + watch!(PEAK_ALLOC.current_usage_as_mb()); watch!(dir_targets); + watch!(PEAK_ALLOC.current_usage_as_mb()); update_dir_gitignores(xvc_root, ¤t_gitignore, &dir_targets)?; + watch!(PEAK_ALLOC.current_usage_as_mb()); // We reload gitignores here to make sure we ignore the given dirs + + watch!(PEAK_ALLOC.current_usage_as_mb()); let current_gitignore = build_gitignore(xvc_root)?; + watch!(PEAK_ALLOC.current_usage_as_mb()); + update_file_gitignores(xvc_root, ¤t_gitignore, &file_targets)?; + watch!(PEAK_ALLOC.current_usage_as_mb()); + if !opts.no_commit { let current_xvc_path_store = xvc_root.load_store::()?; @@ -285,5 +297,6 @@ pub fn cmd_track( opts.force, )?; } + Ok(()) } diff --git a/logging/src/lib.rs b/logging/src/lib.rs index 5e5f16424..4a5a8a56e 100755 --- a/logging/src/lib.rs +++ b/logging/src/lib.rs @@ -11,6 +11,7 @@ use std::fmt::Display; use std::path::Path; use std::sync::Once; + /// Debugging macro to print the given expression and its value, with the module, function and line number #[macro_export] macro_rules! watch { diff --git a/logging/src/logging/mod.rs b/logging/src/logging/mod.rs deleted file mode 100644 index db224278f..000000000 --- a/logging/src/logging/mod.rs +++ /dev/null @@ -1,56 +0,0 @@ -use log::LevelFilter; -use log::{error, info}; -use std::env; -use std::path::Path; -use std::sync::Once; - -// Logging - -static INIT: Once = Once::new(); - -pub fn setup_logging(term_level: Option, file_level: Option) { - INIT.call_once(|| init_logging(term_level, file_level)); -} - -fn init_logging(term_level: Option, file_level: Option) { - let logfilename = &format!( - "{}/xvc-{}.log", - env::temp_dir().to_string_lossy(), - chrono::Local::now().format("%Y%m%d-%H%M%S") - ); - - let logfile = Path::new(&logfilename); - - let mut dispatch = fern::Dispatch::new().format(|out, message, record| { - out.finish(format_args!( - "[{}][{}:{}] {}", - record.level(), - // chrono::Local::now().format("[%H:%M:%S]"), - // record.target(), - record.file().get_or_insert("None"), - record.line().get_or_insert(0), - message - )) - }); - - if let Some(level) = term_level { - dispatch = dispatch.level(level).chain(std::io::stderr()); - } - - match dispatch.apply() { - Ok(_) => { - if let Some(level) = term_level { - debug!("Terminal logger enabled with level: {:?}", level); - }; - if let Some(level) = file_level { - debug!( - "File logger enabled with level: {:?} to {:?}", - level, logfile - ); - }; - } - Err(err) => { - error!("Error enabling logger: {:?}", err); - } - }; -} diff --git a/walker/src/ignore_rules.rs b/walker/src/ignore_rules.rs index 9db7d5b8b..4f429aca6 100644 --- a/walker/src/ignore_rules.rs +++ b/walker/src/ignore_rules.rs @@ -4,9 +4,10 @@ use std::sync::{Arc, RwLock}; use globset::GlobSet; use itertools::Itertools; - use crate::{build_globset, content_to_patterns, GlobPattern, PatternEffect, Result}; +use xvc_logging::watch; +use crate::PEAK_ALLOC; /// Complete set of ignore rules for a directory and its child directories. #[derive(Debug, Clone)] pub struct IgnoreRules { @@ -41,12 +42,16 @@ impl IgnoreRules { /// Compiles patterns as [Source::Global] and initializes the elements. pub fn try_from_patterns(root: &Path, patterns: &str) -> Result { + + watch!(PEAK_ALLOC.current_usage_as_mb()); let patterns = content_to_patterns(root, None, patterns) .into_iter() .map(|pat_res_g| pat_res_g.map(|res_g| res_g.unwrap())) .collect(); + watch!(PEAK_ALLOC.current_usage_as_mb()); let mut initialized = Self::empty(&PathBuf::from(root)); + watch!(PEAK_ALLOC.current_usage_as_mb()); initialized.update(patterns)?; Ok(initialized) } @@ -54,29 +59,39 @@ impl IgnoreRules { /// Adds `new_patterns` to the list of patterns and recompiles ignore and /// whitelist [GlobSet]s. pub fn update(&mut self, new_patterns: Vec) -> Result<()> { + watch!(PEAK_ALLOC.current_usage_as_mb()); let (new_ignore_patterns, new_whitelist_patterns): (Vec<_>, Vec<_>) = new_patterns .into_iter() .partition(|p| matches!(p.effect, PatternEffect::Ignore)); + watch!(PEAK_ALLOC.current_usage_as_mb()); self.update_ignore(&new_ignore_patterns)?; + watch!(PEAK_ALLOC.current_usage_as_mb()); self.update_whitelist(&new_whitelist_patterns)?; + watch!(PEAK_ALLOC.current_usage_as_mb()); Ok(()) } /// Merge with other ignore rules, extending this one's patterns and rebuilding glob sets. pub fn merge_with(&mut self, other: &IgnoreRules) -> Result<()> { assert_eq!(self.root, other.root); + watch!(PEAK_ALLOC.current_usage_as_mb()); self.update_ignore(&other.ignore_patterns.read().unwrap())?; + watch!(PEAK_ALLOC.current_usage_as_mb()); self.update_whitelist(&other.whitelist_patterns.read().unwrap())?; + watch!(PEAK_ALLOC.current_usage_as_mb()); Ok(()) } fn update_whitelist(&mut self, new_whitelist_patterns: &[GlobPattern]) -> Result<()> { + watch!(PEAK_ALLOC.current_usage_as_mb()); assert!(new_whitelist_patterns .iter() .all(|p| matches!(p.effect, PatternEffect::Whitelist))); + watch!(PEAK_ALLOC.current_usage_as_mb()); { let mut whitelist_patterns = self.whitelist_patterns.write()?; + watch!(PEAK_ALLOC.current_usage_as_mb()); *whitelist_patterns = whitelist_patterns .iter() @@ -85,26 +100,34 @@ impl IgnoreRules { .cloned() .collect(); } + watch!(PEAK_ALLOC.current_usage_as_mb()); { + watch!(PEAK_ALLOC.current_usage_as_mb()); let whitelist_globs = self .whitelist_patterns .read()? .iter() .map(|g| g.pattern.clone()) .collect(); + watch!(PEAK_ALLOC.current_usage_as_mb()); let whitelist_set = build_globset(whitelist_globs)?; + watch!(PEAK_ALLOC.current_usage_as_mb()); *self.whitelist_set.write()? = whitelist_set; } + watch!(PEAK_ALLOC.current_usage_as_mb()); Ok(()) } fn update_ignore(&mut self, new_ignore_patterns: &[GlobPattern]) -> Result<()> { + watch!(PEAK_ALLOC.current_usage_as_mb()); assert!(new_ignore_patterns .iter() .all(|p| matches!(p.effect, PatternEffect::Ignore))); + watch!(PEAK_ALLOC.current_usage_as_mb()); { let mut ignore_patterns = self.ignore_patterns.write()?; + watch!(PEAK_ALLOC.current_usage_as_mb()); *ignore_patterns = ignore_patterns .iter() @@ -113,6 +136,7 @@ impl IgnoreRules { .cloned() .collect(); } + watch!(PEAK_ALLOC.current_usage_as_mb()); { let ignore_globs = self @@ -123,7 +147,9 @@ impl IgnoreRules { .collect(); let ignore_set = build_globset(ignore_globs)?; *self.ignore_set.write()? = ignore_set; + watch!(PEAK_ALLOC.current_usage_as_mb()); } + watch!(PEAK_ALLOC.current_usage_as_mb()); Ok(()) } diff --git a/walker/src/lib.rs b/walker/src/lib.rs index f2b49efa4..0a0803c53 100755 --- a/walker/src/lib.rs +++ b/walker/src/lib.rs @@ -33,6 +33,11 @@ pub use notify::RecommendedWatcher; use xvc_logging::watch; +use peak_alloc::PeakAlloc; + +#[global_allocator] +pub static PEAK_ALLOC: PeakAlloc = PeakAlloc; + use crossbeam_channel::Sender; // use glob::{MatchOptions, Pattern, PatternError}; pub use globset::{self, Glob, GlobSet, GlobSetBuilder}; @@ -479,17 +484,26 @@ pub fn build_ignore_rules( dir: &Path, ignore_filename: &str, ) -> Result { + + watch!(PEAK_ALLOC.current_usage_as_mb()); let elements = dir .read_dir() .map_err(|e| anyhow!("Error reading directory: {:?}, {:?}", dir, e))?; + watch!(PEAK_ALLOC.current_usage_as_mb()); let mut child_dirs = Vec::::new(); + watch!(PEAK_ALLOC.current_usage_as_mb()); let ignore_fn = OsString::from(ignore_filename); + watch!(PEAK_ALLOC.current_usage_as_mb()); xvc_logging::watch!(ignore_fn); + watch!(PEAK_ALLOC.current_usage_as_mb()); let ignore_root = given.root.clone(); + watch!(PEAK_ALLOC.current_usage_as_mb()); xvc_logging::watch!(ignore_root); let mut ignore_rules = given; + watch!(PEAK_ALLOC.current_usage_as_mb()); let mut new_patterns: Option> = None; + watch!(PEAK_ALLOC.current_usage_as_mb()); for entry in elements { match entry { @@ -521,19 +535,25 @@ pub fn build_ignore_rules( } } + watch!(PEAK_ALLOC.current_usage_as_mb()); if let Some(new_patterns) = new_patterns { ignore_rules.update(new_patterns)?; } + watch!(PEAK_ALLOC.current_usage_as_mb()); for child_dir in child_dirs { + watch!(PEAK_ALLOC.current_usage_as_mb()); match check_ignore(&ignore_rules, &child_dir) { MatchResult::NoMatch | MatchResult::Whitelist => { + watch!(PEAK_ALLOC.current_usage_as_mb()); ignore_rules = build_ignore_rules(ignore_rules, &child_dir, ignore_filename)?; + watch!(PEAK_ALLOC.current_usage_as_mb()); } MatchResult::Ignore => {} } } + watch!(PEAK_ALLOC.current_usage_as_mb()); Ok(ignore_rules) } @@ -752,8 +772,9 @@ pub fn check_ignore(ignore_rules: &IgnoreRules, path: &Path) -> MatchResult { watch!(path_str); let final_slash = path_str.ends_with('/'); watch!(final_slash); - + let path = if is_abs { + if final_slash { format!( "/{}/", @@ -774,13 +795,16 @@ pub fn check_ignore(ignore_rules: &IgnoreRules, path: &Path) -> MatchResult { }; watch!(path); - if ignore_rules.whitelist_set.read().unwrap().is_match(&path) { + watch!(PEAK_ALLOC.current_usage_as_mb()); + let result = if ignore_rules.whitelist_set.read().unwrap().is_match(&path) { MatchResult::Whitelist } else if ignore_rules.ignore_set.read().unwrap().is_match(&path) { MatchResult::Ignore } else { MatchResult::NoMatch - } + }; + watch!(PEAK_ALLOC.current_usage_as_mb()); + result } /// Return all childs of a directory regardless of any ignore rules From b111877efe677de9f7901ab62c618c16ccda6f74 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Tue, 6 Aug 2024 22:04:39 +0300 Subject: [PATCH 151/257] Bump version --- config/Cargo.toml | 6 +++--- core/Cargo.toml | 12 ++++++------ ecs/Cargo.toml | 4 ++-- file/Cargo.toml | 16 ++++++++-------- lib/Cargo.toml | 20 ++++++++++---------- logging/Cargo.toml | 2 +- pipeline/Cargo.toml | 16 ++++++++-------- storage/Cargo.toml | 14 +++++++------- test_helper/Cargo.toml | 4 ++-- walker/Cargo.toml | 6 +++--- 10 files changed, 50 insertions(+), 50 deletions(-) diff --git a/config/Cargo.toml b/config/Cargo.toml index 8bdeaaebe..ff953f129 100644 --- a/config/Cargo.toml +++ b/config/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-config" -version = "0.6.11-alpha.7" +version = "0.6.11-alpha.8" edition = "2021" description = "Xvc configuration management" authors = ["Emre Şahin "] @@ -16,8 +16,8 @@ name = "xvc_config" crate-type = ["rlib"] [dependencies] -xvc-logging = { version = "0.6.11-alpha.7", path = "../logging" } -xvc-walker = { version = "0.6.11-alpha.7", path = "../walker" } +xvc-logging = { version = "0.6.11-alpha.8", path = "../logging" } +xvc-walker = { version = "0.6.11-alpha.8", path = "../walker" } ## Cli and config diff --git a/core/Cargo.toml b/core/Cargo.toml index e760970ec..d1600f1ed 100644 --- a/core/Cargo.toml +++ b/core/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-core" -version = "0.6.11-alpha.7" +version = "0.6.11-alpha.8" edition = "2021" description = "Xvc core for common elements for all commands" authors = ["Emre Şahin "] @@ -16,10 +16,10 @@ name = "xvc_core" crate-type = ["rlib"] [dependencies] -xvc-config = { version = "0.6.11-alpha.7", path = "../config" } -xvc-logging = { version = "0.6.11-alpha.7", path = "../logging" } -xvc-ecs = { version = "0.6.11-alpha.7", path = "../ecs" } -xvc-walker = { version = "0.6.11-alpha.7", path = "../walker" } +xvc-config = { version = "0.6.11-alpha.8", path = "../config" } +xvc-logging = { version = "0.6.11-alpha.8", path = "../logging" } +xvc-ecs = { version = "0.6.11-alpha.8", path = "../ecs" } +xvc-walker = { version = "0.6.11-alpha.8", path = "../walker" } ## Cli and config clap = { version = "^4.5", features = ["derive"] } @@ -81,6 +81,6 @@ itertools = "^0.13" [dev-dependencies] -xvc-test-helper = { version = "0.6.11-alpha.7", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.11-alpha.8", path = "../test_helper/" } proptest = "^1.5" test-case = "^3.3" diff --git a/ecs/Cargo.toml b/ecs/Cargo.toml index 4f0d87584..9eafda11f 100644 --- a/ecs/Cargo.toml +++ b/ecs/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-ecs" -version = "0.6.11-alpha.7" +version = "0.6.11-alpha.8" edition = "2021" description = "Entity-Component System for Xvc" authors = ["Emre Şahin "] @@ -16,7 +16,7 @@ name = "xvc_ecs" crate-type = ["rlib"] [dependencies] -xvc-logging = { version = "0.6.11-alpha.7", path = "../logging" } +xvc-logging = { version = "0.6.11-alpha.8", path = "../logging" } ## Serialization serde = { version = "^1.0", features = ["derive"] } diff --git a/file/Cargo.toml b/file/Cargo.toml index 4dfbc5743..24e3a3fe9 100644 --- a/file/Cargo.toml +++ b/file/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-file" -version = "0.6.11-alpha.7" +version = "0.6.11-alpha.8" edition = "2021" description = "File tracking, versioning, upload and download functions for Xvc" authors = ["Emre Şahin "] @@ -21,12 +21,12 @@ test = true bench = true [dependencies] -xvc-logging = { version = "0.6.11-alpha.7", path = "../logging" } -xvc-config = { version = "0.6.11-alpha.7", path = "../config" } -xvc-core = { version = "0.6.11-alpha.7", path = "../core" } -xvc-ecs = { version = "0.6.11-alpha.7", path = "../ecs" } -xvc-walker = { version = "0.6.11-alpha.7", path = "../walker" } -xvc-storage = { version = "0.6.11-alpha.7", path = "../storage", default-features = false } +xvc-logging = { version = "0.6.11-alpha.8", path = "../logging" } +xvc-config = { version = "0.6.11-alpha.8", path = "../config" } +xvc-core = { version = "0.6.11-alpha.8", path = "../core" } +xvc-ecs = { version = "0.6.11-alpha.8", path = "../ecs" } +xvc-walker = { version = "0.6.11-alpha.8", path = "../walker" } +xvc-storage = { version = "0.6.11-alpha.8", path = "../storage", default-features = false } ## Cli and config @@ -95,5 +95,5 @@ default = ["reflink"] reflink = ["dep:reflink"] [dev-dependencies] -xvc-test-helper = { version = "0.6.11-alpha.7", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.11-alpha.8", path = "../test_helper/" } shellfn = "^0.1" diff --git a/lib/Cargo.toml b/lib/Cargo.toml index 98c79309d..5927acc49 100644 --- a/lib/Cargo.toml +++ b/lib/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc" -version = "0.6.11-alpha.7" +version = "0.6.11-alpha.8" edition = "2021" description = "An MLOps tool to manage data files and pipelines on top of Git" authors = ["Emre Şahin "] @@ -20,14 +20,14 @@ name = "xvc" path = "src/main.rs" [dependencies] -xvc-config = { version = "0.6.11-alpha.7", path = "../config" } -xvc-core = { version = "0.6.11-alpha.7", path = "../core" } -xvc-logging = { version = "0.6.11-alpha.7", path = "../logging" } -xvc-ecs = { version = "0.6.11-alpha.7", path = "../ecs" } -xvc-file = { version = "0.6.11-alpha.7", path = "../file", default-features = false } -xvc-pipeline = { version = "0.6.11-alpha.7", path = "../pipeline" } -xvc-walker = { version = "0.6.11-alpha.7", path = "../walker" } -xvc-storage = { version = "0.6.11-alpha.7", path = "../storage", default-features = false } +xvc-config = { version = "0.6.11-alpha.8", path = "../config" } +xvc-core = { version = "0.6.11-alpha.8", path = "../core" } +xvc-logging = { version = "0.6.11-alpha.8", path = "../logging" } +xvc-ecs = { version = "0.6.11-alpha.8", path = "../ecs" } +xvc-file = { version = "0.6.11-alpha.8", path = "../file", default-features = false } +xvc-pipeline = { version = "0.6.11-alpha.8", path = "../pipeline" } +xvc-walker = { version = "0.6.11-alpha.8", path = "../walker" } +xvc-storage = { version = "0.6.11-alpha.8", path = "../storage", default-features = false } ## Cli and config clap = { version = "^4.5", features = ["derive", "cargo"] } @@ -133,4 +133,4 @@ shellfn = "^0.1" test-case = "^3.3" trycmd = "^0.15" which = "^6.0" -xvc-test-helper = { version = "0.6.11-alpha.7", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.11-alpha.8", path = "../test_helper/" } diff --git a/logging/Cargo.toml b/logging/Cargo.toml index 7dfd62727..6f935ea75 100644 --- a/logging/Cargo.toml +++ b/logging/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-logging" -version = "0.6.11-alpha.7" +version = "0.6.11-alpha.8" edition = "2021" description = "Logging crate for Xvc" authors = ["Emre Şahin "] diff --git a/pipeline/Cargo.toml b/pipeline/Cargo.toml index fe0e8fd9a..2dcfa5bf4 100644 --- a/pipeline/Cargo.toml +++ b/pipeline/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-pipeline" -version = "0.6.11-alpha.7" +version = "0.6.11-alpha.8" edition = "2021" description = "Xvc data pipeline management" authors = ["Emre Şahin "] @@ -19,12 +19,12 @@ default = [] bundled-sqlite = ["rusqlite/bundled"] [dependencies] -xvc-config = { version = "0.6.11-alpha.7", path = "../config" } -xvc-core = { version = "0.6.11-alpha.7", path = "../core" } -xvc-ecs = { version = "0.6.11-alpha.7", path = "../ecs" } -xvc-logging = { version = "0.6.11-alpha.7", path = "../logging" } -xvc-walker = { version = "0.6.11-alpha.7", path = "../walker" } -xvc-file = { version = "0.6.11-alpha.7", path = "../file", default-features = false } +xvc-config = { version = "0.6.11-alpha.8", path = "../config" } +xvc-core = { version = "0.6.11-alpha.8", path = "../core" } +xvc-ecs = { version = "0.6.11-alpha.8", path = "../ecs" } +xvc-logging = { version = "0.6.11-alpha.8", path = "../logging" } +xvc-walker = { version = "0.6.11-alpha.8", path = "../walker" } +xvc-file = { version = "0.6.11-alpha.8", path = "../file", default-features = false } ## Cli and config clap = { version = "^4.5", features = ["derive"] } @@ -98,5 +98,5 @@ itertools = "^0.13" derive_more = "^0.99" [dev-dependencies] -xvc-test-helper = { version = "0.6.11-alpha.7", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.11-alpha.8", path = "../test_helper/" } test-case = "^3.3" diff --git a/storage/Cargo.toml b/storage/Cargo.toml index 461accf44..0bbc85d77 100644 --- a/storage/Cargo.toml +++ b/storage/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-storage" -version = "0.6.11-alpha.7" +version = "0.6.11-alpha.8" edition = "2021" description = "Xvc remote and local storage management" authors = ["Emre Şahin "] @@ -16,11 +16,11 @@ name = "xvc_storage" crate-type = ["rlib"] [dependencies] -xvc-logging = { version = "0.6.11-alpha.7", path = "../logging" } -xvc-config = { version = "0.6.11-alpha.7", path = "../config" } -xvc-core = { version = "0.6.11-alpha.7", path = "../core" } -xvc-ecs = { version = "0.6.11-alpha.7", path = "../ecs" } -xvc-walker = { version = "0.6.11-alpha.7", path = "../walker" } +xvc-logging = { version = "0.6.11-alpha.8", path = "../logging" } +xvc-config = { version = "0.6.11-alpha.8", path = "../config" } +xvc-core = { version = "0.6.11-alpha.8", path = "../core" } +xvc-ecs = { version = "0.6.11-alpha.8", path = "../ecs" } +xvc-walker = { version = "0.6.11-alpha.8", path = "../walker" } ## Cli and config clap = { version = "^4.5", features = ["derive"] } @@ -104,7 +104,7 @@ bundled-openssl = ["openssl/vendored"] [dev-dependencies] -xvc-test-helper = { version = "0.6.11-alpha.7", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.11-alpha.8", path = "../test_helper/" } shellfn = "^0.1" [package.metadata.cargo-udeps.ignore] diff --git a/test_helper/Cargo.toml b/test_helper/Cargo.toml index a7f125cf2..a2e15d3cf 100644 --- a/test_helper/Cargo.toml +++ b/test_helper/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-test-helper" -version = "0.6.11-alpha.7" +version = "0.6.11-alpha.8" edition = "2021" description = "Unit test helper functions for Xvc" authors = ["Emre Şahin "] @@ -20,7 +20,7 @@ path = "src/main.rs" # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html [dependencies] -xvc-logging = { version = "0.6.11-alpha.7", path = "../logging/" } +xvc-logging = { version = "0.6.11-alpha.8", path = "../logging/" } rand = "^0.8" log = "^0.4" diff --git a/walker/Cargo.toml b/walker/Cargo.toml index 5ef1af67b..4bbbba2e0 100644 --- a/walker/Cargo.toml +++ b/walker/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-walker" -version = "0.6.11-alpha.7" +version = "0.6.11-alpha.8" edition = "2021" description = "Xvc parallel file system walker with ignore features" authors = ["Emre Şahin "] @@ -16,7 +16,7 @@ name = "xvc_walker" crate-type = ["rlib"] [dependencies] -xvc-logging = { version = "0.6.11-alpha.7", path = "../logging" } +xvc-logging = { version = "0.6.11-alpha.8", path = "../logging" } globset = "^0.4" ## Parallelization @@ -39,7 +39,7 @@ itertools = "^0.13" regex = "^1.10" [dev-dependencies] -xvc-test-helper = { path = "../test_helper/", version = "0.6.11-alpha.7" } +xvc-test-helper = { path = "../test_helper/", version = "0.6.11-alpha.8" } test-case = "^3.3" [package.metadata.cargo-udeps.ignore] From 950c431738f8ba035fb6657d7ced438e546fb9f8 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Thu, 8 Aug 2024 12:28:54 +0300 Subject: [PATCH 152/257] ws --- .github/workflows/publish.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index a89335759..c2b52ce07 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -22,4 +22,4 @@ jobs: - name: Publish to crates.io env: CARGO_REGISTRY_TOKEN: ${{ secrets.CARGO_REGISTRY_TOKEN }} - run: for p in xvc-logging xvc-test-helper xvc-walker xvc-ecs xvc-config xvc-core xvc-storage xvc-file xvc-pipelines xvc ; do cargo publish --package $p ; done + run: for p in xvc-logging xvc-test-helper xvc-walker xvc-ecs xvc-config xvc-core xvc-storage xvc-file xvc-pipelines xvc ; do cargo publish --package $p ; done From aae241fb932be98d2dda30cc87f1ab36980a9a2b Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Thu, 8 Aug 2024 12:29:02 +0300 Subject: [PATCH 153/257] remove GlobSetError --- core/src/error.rs | 5 ----- 1 file changed, 5 deletions(-) diff --git a/core/src/error.rs b/core/src/error.rs index f821dc3ea..2f9bdab72 100644 --- a/core/src/error.rs +++ b/core/src/error.rs @@ -130,11 +130,6 @@ pub enum Error { #[from] source: relative_path::FromPathError, }, - #[error("Glob error: {source}")] - GlobSetError { - #[from] - source: xvc_walker::globset::Error, - }, #[error("Cannot find parent path")] CannotFindParentPath { path: PathBuf }, From 70fc0d6b8ce74b1e1f1f3b4818bc19cf9c8a72c6 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Thu, 8 Aug 2024 12:29:37 +0300 Subject: [PATCH 154/257] minor changes --- file/src/common/gitignore.rs | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/file/src/common/gitignore.rs b/file/src/common/gitignore.rs index 8ea79be10..4c6db75ed 100644 --- a/file/src/common/gitignore.rs +++ b/file/src/common/gitignore.rs @@ -7,11 +7,11 @@ use std::fs::OpenOptions; use std::io::Write; use std::thread::JoinHandle; -use xvc_core::util::git::build_gitignore; +use xvc_core::{util::git::build_gitignore}; -use crate::Result; +use crate::{Result, CHANNEL_CAPACITY}; use xvc_core::{XvcPath, XvcRoot}; -use xvc_logging::{debug, error, info, uwr, XvcOutputSender}; +use xvc_logging::{debug, error, info, uwr, watch, XvcOutputSender}; use xvc_walker::{check_ignore, AbsolutePath, IgnoreRules, MatchResult}; /// Used to signal ignored files and directories to the ignore handler @@ -39,11 +39,12 @@ pub fn make_ignore_handler( output_snd: &XvcOutputSender, xvc_root: &XvcRoot, ) -> Result<(Sender, JoinHandle<()>)> { - let (sender, receiver) = crossbeam_channel::unbounded(); + let (sender, receiver) = crossbeam_channel::bounded(CHANNEL_CAPACITY); let output_snd = output_snd.clone(); let xvc_root = xvc_root.absolute_path().clone(); let handle = std::thread::spawn(move || { + let mut ignore_dirs = Vec::::new(); let mut ignore_files = Vec::::new(); @@ -75,10 +76,12 @@ pub fn make_ignore_handler( } } debug!(output_snd, "Writing directories to .gitignore"); + uwr!( update_dir_gitignores(&xvc_root, &gitignore, &ignore_dirs), output_snd ); + // Load again to get ignored directories let gitignore = build_gitignore(&xvc_root).unwrap(); debug!(output_snd, "Writing files to .gitignore"); @@ -101,6 +104,7 @@ pub fn update_dir_gitignores( current_gitignore: &IgnoreRules, dirs: &[XvcPath], ) -> Result<()> { + // Check if dirs are already ignored let dirs: Vec = dirs .iter() @@ -127,7 +131,7 @@ pub fn update_dir_gitignores( None } }}).collect(); - + // Check if files are already ignored let mut changes = HashMap::>::new(); From 0d1173bcef991dc5dd4fd050b922c06d76ff0e7d Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Thu, 8 Aug 2024 12:29:56 +0300 Subject: [PATCH 155/257] use fast-glob --- lib/Cargo.toml | 2 +- lib/tests/test_walker_parallel.rs | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/Cargo.toml b/lib/Cargo.toml index 5927acc49..2b74501a8 100644 --- a/lib/Cargo.toml +++ b/lib/Cargo.toml @@ -125,7 +125,7 @@ assert_cmd = "^2.0" assert_fs = "^1.1" escargot = "^0.5" fs_extra = "^1.3" -globset = "^0.4" +fast-glob = "^0.3" jwalk = "^0.8" predicates = "^3.1" proptest = "^1.5" diff --git a/lib/tests/test_walker_parallel.rs b/lib/tests/test_walker_parallel.rs index 2c75e73db..756997fc9 100644 --- a/lib/tests/test_walker_parallel.rs +++ b/lib/tests/test_walker_parallel.rs @@ -3,7 +3,7 @@ use std::{ path::{Path, PathBuf}, }; -use globset::Glob; +use fast_glob::Glob; use log::LevelFilter; use xvc_walker::*; From 9def90eda5fbc8ce533f5482c5f7a8cecb76f21a Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Thu, 8 Aug 2024 12:32:31 +0300 Subject: [PATCH 156/257] add build_ignore_glob --- walker/src/lib.rs | 96 +++++++++++++++++++++++++++++++++++++---------- 1 file changed, 76 insertions(+), 20 deletions(-) diff --git a/walker/src/lib.rs b/walker/src/lib.rs index 0a0803c53..533fabd1c 100755 --- a/walker/src/lib.rs +++ b/walker/src/lib.rs @@ -18,8 +18,11 @@ pub use abspath::AbsolutePath; use crossbeam::queue::SegQueue; pub use error::{Error, Result}; pub use ignore_rules::IgnoreRules; +use jwalk::WalkDirGeneric; pub use notify::make_watcher; +use std::borrow::BorrowMut; pub use std::hash::Hash; +use std::ops::DerefMut; use std::sync::Arc; use std::sync::Mutex; use std::sync::RwLock; @@ -40,7 +43,7 @@ pub static PEAK_ALLOC: PeakAlloc = PeakAlloc; use crossbeam_channel::Sender; // use glob::{MatchOptions, Pattern, PatternError}; -pub use globset::{self, Glob, GlobSet, GlobSetBuilder}; +pub use fast_glob::Glob; use std::{ ffi::OsString, fmt::Debug, @@ -158,26 +161,8 @@ impl Pattern { } } -impl Pattern> { - /// Convert from `Pattern>` to `Result>` to get the result from - /// [Self::map] - fn transpose(self) -> Result> { - match self.pattern { - Ok(p) => Ok(Pattern:: { - pattern: p, - original: self.original, - source: self.source, - effect: self.effect, - relativity: self.relativity, - path_kind: self.path_kind, - }), - Err(e) => Err(e), - } - } -} - /// One of the concrete types that can represent a pattern. -type GlobPattern = Pattern; +type GlobPattern = Pattern; /// What's the ignore file name and should we add directories to the result? #[derive(Debug, Clone)] @@ -478,6 +463,77 @@ pub fn walk_serial( Ok((res_paths, ignore_rules)) } +/// FIXME: This should be Arc>> when Glob.is_match doesn't require mutable self +type IgnoreGlob = Arc>; + +/// Builds a [fast_glob::Glob] used to ignore files in they match +/// +/// This doesn't keep track of where the ignore rules are gathered. It's used for basic operations. +pub fn build_ignore_glob( + given: &str, + ignore_root: &Path, + ignore_filename: &str, +) -> Result { + + let mut glob = IgnoreGlob::new(Mutex::new(Glob::new(given))); + + let ignore_file_walker = WalkDirGeneric::<((usize),(bool))>::new(ignore_root) + .process_read_dir(|depth, path, read_dir_state, children| { + // Keep only ignore files + children.retain(|dir_entry_result| { + dir_entry_result.as_ref().map(|dir_entry| { + match dir_entry.file_type { + FileType::Dir => { + // Don't read ignored directories + let ignored_dir = glob.borrow_mut().deref_mut().is_match( + &format!( + "/{}/", + dir_entry.path().strip_prefix(&ignore_root).unwrap().to_string_lossy() + ) + ); + if ignored_dir { + dir_entry.read_children_path = None; + return false; + } + } + FileType::File => { + // Read only ignore files + dir_entry.file_name() + .to_str() + .map(|s| s == ignore_filename) + .unwrap_or(false) + } + } + }).unwrap_or(false) + }); + }); + for entry in walk_dir { + if let Ok(entry) = entry { + assert!(entry.file_type.is_dir() || entry.file_name.to_str() == Some(ignore_filename)); + + let content = fs::read_to_string(entry.path())?; + let entry_root = entry.path().strip_prefix(&ignore_root).unwrap(); + + let ignore_rules = content.lines().filter_map(|line| { + if line.trim().is_empty() || line.trim().starts_with('#') { + None + } else { + Some(format!("{}/{}", entry_root, line)) + } + }).for_each(|line| { + // FIXME: A small optimization is to add all patterns at once with a single + // unlock + glob.borrow_mut().deref_mut().add(&line); + }); + } else { + debug!("Error reading ignore file: {:?}", entry.unwrap_err()); + } + } + + Ok(glob) +} + + /// Just build the ignore rules with the given directory pub fn build_ignore_rules( given: IgnoreRules, From 09e349b71f1f0cffd031f49130e68b139ca190c2 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Thu, 8 Aug 2024 12:33:05 +0300 Subject: [PATCH 157/257] remove memory watches --- walker/src/lib.rs | 14 ++++---------- 1 file changed, 4 insertions(+), 10 deletions(-) diff --git a/walker/src/lib.rs b/walker/src/lib.rs index 533fabd1c..103a6d67e 100755 --- a/walker/src/lib.rs +++ b/walker/src/lib.rs @@ -541,25 +541,19 @@ pub fn build_ignore_rules( ignore_filename: &str, ) -> Result { - watch!(PEAK_ALLOC.current_usage_as_mb()); let elements = dir .read_dir() .map_err(|e| anyhow!("Error reading directory: {:?}, {:?}", dir, e))?; - watch!(PEAK_ALLOC.current_usage_as_mb()); let mut child_dirs = Vec::::new(); - watch!(PEAK_ALLOC.current_usage_as_mb()); let ignore_fn = OsString::from(ignore_filename); - watch!(PEAK_ALLOC.current_usage_as_mb()); - xvc_logging::watch!(ignore_fn); - watch!(PEAK_ALLOC.current_usage_as_mb()); + watch!(ignore_fn); let ignore_root = given.root.clone(); - watch!(PEAK_ALLOC.current_usage_as_mb()); - xvc_logging::watch!(ignore_root); + watch!(ignore_root); + let mut ignore_rules = given; - watch!(PEAK_ALLOC.current_usage_as_mb()); + let mut new_patterns: Option> = None; - watch!(PEAK_ALLOC.current_usage_as_mb()); for entry in elements { match entry { From 1b05248e5929afd1312dbf8274c7e6ed63f48fee Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Thu, 8 Aug 2024 12:33:41 +0300 Subject: [PATCH 158/257] cleanup --- walker/src/lib.rs | 27 +-------------------------- 1 file changed, 1 insertion(+), 26 deletions(-) diff --git a/walker/src/lib.rs b/walker/src/lib.rs index 103a6d67e..139176a80 100755 --- a/walker/src/lib.rs +++ b/walker/src/lib.rs @@ -559,7 +559,7 @@ pub fn build_ignore_rules( match entry { Ok(entry) => { if entry.path().is_dir() { - xvc_logging::watch!(entry.path()); + watch!(entry.path()); child_dirs.push(entry.path()); } if entry.file_name() == ignore_fn && entry.path().exists() { @@ -585,47 +585,22 @@ pub fn build_ignore_rules( } } - watch!(PEAK_ALLOC.current_usage_as_mb()); if let Some(new_patterns) = new_patterns { ignore_rules.update(new_patterns)?; } - watch!(PEAK_ALLOC.current_usage_as_mb()); for child_dir in child_dirs { - watch!(PEAK_ALLOC.current_usage_as_mb()); match check_ignore(&ignore_rules, &child_dir) { MatchResult::NoMatch | MatchResult::Whitelist => { - watch!(PEAK_ALLOC.current_usage_as_mb()); ignore_rules = build_ignore_rules(ignore_rules, &child_dir, ignore_filename)?; - watch!(PEAK_ALLOC.current_usage_as_mb()); } MatchResult::Ignore => {} } } - watch!(PEAK_ALLOC.current_usage_as_mb()); Ok(ignore_rules) } -fn clear_glob_errors( - sender: &Sender>, - new_patterns: Vec>>, -) -> Vec> { - let new_glob_patterns: Vec> = new_patterns - .into_iter() - .filter_map(|p| match p.transpose() { - Ok(p) => Some(p), - Err(e) => { - sender - .send(Err(Error::from(anyhow!("Error in glob pattern: {:?}", e)))) - .expect("Error in channel"); - None - } - }) - .collect(); - new_glob_patterns -} - fn transform_pattern_for_glob(pattern: Pattern) -> Pattern { let anything_anywhere = |p| format!("**/{p}"); let anything_relative = |p, directory| format!("{directory}/**/{p}"); From e778c16d38c2144dc72ac2c9cd63920be5ec4231 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Thu, 8 Aug 2024 12:33:58 +0300 Subject: [PATCH 159/257] update to use glob --- walker/src/lib.rs | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/walker/src/lib.rs b/walker/src/lib.rs index 139176a80..df6937e57 100755 --- a/walker/src/lib.rs +++ b/walker/src/lib.rs @@ -624,15 +624,15 @@ fn transform_pattern_for_glob(pattern: Pattern) -> Pattern { } } -fn build_globset(patterns: Vec) -> Result { - let mut gs_builder = GlobSetBuilder::new(); +fn build_globset(patterns: &Vec) -> Result { + let mut glob = Glob::new("").expect("Error building glob set"); for p in patterns { - gs_builder.add(p.clone()); + if !glob.add(&p) { + return Err(anyhow!("Error adding pattern {} to glob set", p).into()); + } } - gs_builder - .build() - .map_err(|e| anyhow!("Error building glob set: {:?}", e).into()) + Ok(glob) } fn patterns_from_file( From a1ac0021d7fc67467c354dc057cddd9644a9f4de Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Thu, 8 Aug 2024 12:34:25 +0300 Subject: [PATCH 160/257] simplify --- walker/src/lib.rs | 22 ++++++++++------------ 1 file changed, 10 insertions(+), 12 deletions(-) diff --git a/walker/src/lib.rs b/walker/src/lib.rs index df6937e57..1dd6bbfeb 100755 --- a/walker/src/lib.rs +++ b/walker/src/lib.rs @@ -638,7 +638,7 @@ fn build_globset(patterns: &Vec) -> Result { fn patterns_from_file( ignore_root: &Path, ignore_path: &Path, -) -> Result>>> { +) -> Result> { watch!(ignore_root); watch!(ignore_path); let content = fs::read_to_string(ignore_path).with_context(|| { @@ -663,8 +663,8 @@ pub fn content_to_patterns( ignore_root: &Path, source: Option<&Path>, content: &str, -) -> Vec>> { - let patterns: Vec>> = content +) -> Vec { + let patterns: Vec = content .lines() .enumerate() // A line starting with # serves as a comment. Put a backslash ("\") in front of the first hash for patterns that begin with a hash. @@ -695,7 +695,6 @@ pub fn content_to_patterns( }) .map(|(line, source)| build_pattern(source, line)) .map(transform_pattern_for_glob) - .map(|pc| pc.map(|s| Glob::new(&s).map_err(Error::from))) .collect(); patterns @@ -819,16 +818,18 @@ pub fn check_ignore(ignore_rules: &IgnoreRules, path: &Path) -> MatchResult { path_str.to_string() }; + // FIXME: Current fast_glob implementation requires &mut self in is_match, possibly to avoid to + // clone internal state. This requires to lock the ignore_rules for each call. It's possible to + // keep a clone of the ignore_rules for each thread with a design change -- or fork the glob + // lib. watch!(path); - watch!(PEAK_ALLOC.current_usage_as_mb()); - let result = if ignore_rules.whitelist_set.read().unwrap().is_match(&path) { + let result = if ignore_rules.whitelist_set.write().unwrap().is_match(&path) { MatchResult::Whitelist - } else if ignore_rules.ignore_set.read().unwrap().is_match(&path) { + } else if ignore_rules.ignore_set.write().unwrap().is_match(&path) { MatchResult::Ignore } else { MatchResult::NoMatch }; - watch!(PEAK_ALLOC.current_usage_as_mb()); result } @@ -948,11 +949,8 @@ mod tests { patterns.len() } - fn create_patterns(root: &str, dir: Option<&str>, patterns: &str) -> Vec> { + fn create_patterns(root: &str, dir: Option<&str>, patterns: &str) -> Vec { content_to_patterns(Path::new(root), dir.map(Path::new), patterns) - .into_iter() - .map(|pat_res_g| pat_res_g.map(|res_g| res_g.unwrap())) - .collect() } fn new_dir_with_ignores( From 15ed2b90daf0735ba9910ee7a4bb3af1aaacc5f8 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Thu, 8 Aug 2024 12:34:53 +0300 Subject: [PATCH 161/257] remove watches and change to fast_glob --- walker/src/ignore_rules.rs | 34 +++++----------------------------- 1 file changed, 5 insertions(+), 29 deletions(-) diff --git a/walker/src/ignore_rules.rs b/walker/src/ignore_rules.rs index 4f429aca6..f70d40eab 100644 --- a/walker/src/ignore_rules.rs +++ b/walker/src/ignore_rules.rs @@ -2,7 +2,7 @@ use std::path::{Path, PathBuf}; use std::sync::{Arc, RwLock}; -use globset::GlobSet; +use fast_glob::Glob; use itertools::Itertools; use crate::{build_globset, content_to_patterns, GlobPattern, PatternEffect, Result}; @@ -22,10 +22,10 @@ pub struct IgnoreRules { pub whitelist_patterns: Arc>>, /// Compiled [GlobSet] for whitelisted paths. - pub whitelist_set: Arc>, + pub whitelist_set: Arc>, /// Compiled [GlobSet] for ignored paths. - pub ignore_set: Arc>, + pub ignore_set: Arc>, } impl IgnoreRules { @@ -35,23 +35,20 @@ impl IgnoreRules { root: PathBuf::from(dir), ignore_patterns: Arc::new(RwLock::new(Vec::::new())), whitelist_patterns: Arc::new(RwLock::new(Vec::::new())), - ignore_set: Arc::new(RwLock::new(GlobSet::empty())), - whitelist_set: Arc::new(RwLock::new(GlobSet::empty())), + ignore_set: Arc::new(RwLock::new(Glob::new("").expect("Empty glob is valid"))), + whitelist_set: Arc::new(RwLock::new(Glob::new("").expect("Empty glob is valid"))), } } /// Compiles patterns as [Source::Global] and initializes the elements. pub fn try_from_patterns(root: &Path, patterns: &str) -> Result { - watch!(PEAK_ALLOC.current_usage_as_mb()); let patterns = content_to_patterns(root, None, patterns) .into_iter() .map(|pat_res_g| pat_res_g.map(|res_g| res_g.unwrap())) .collect(); - watch!(PEAK_ALLOC.current_usage_as_mb()); let mut initialized = Self::empty(&PathBuf::from(root)); - watch!(PEAK_ALLOC.current_usage_as_mb()); initialized.update(patterns)?; Ok(initialized) } @@ -59,39 +56,29 @@ impl IgnoreRules { /// Adds `new_patterns` to the list of patterns and recompiles ignore and /// whitelist [GlobSet]s. pub fn update(&mut self, new_patterns: Vec) -> Result<()> { - watch!(PEAK_ALLOC.current_usage_as_mb()); let (new_ignore_patterns, new_whitelist_patterns): (Vec<_>, Vec<_>) = new_patterns .into_iter() .partition(|p| matches!(p.effect, PatternEffect::Ignore)); - watch!(PEAK_ALLOC.current_usage_as_mb()); self.update_ignore(&new_ignore_patterns)?; - watch!(PEAK_ALLOC.current_usage_as_mb()); self.update_whitelist(&new_whitelist_patterns)?; - watch!(PEAK_ALLOC.current_usage_as_mb()); Ok(()) } /// Merge with other ignore rules, extending this one's patterns and rebuilding glob sets. pub fn merge_with(&mut self, other: &IgnoreRules) -> Result<()> { assert_eq!(self.root, other.root); - watch!(PEAK_ALLOC.current_usage_as_mb()); self.update_ignore(&other.ignore_patterns.read().unwrap())?; - watch!(PEAK_ALLOC.current_usage_as_mb()); self.update_whitelist(&other.whitelist_patterns.read().unwrap())?; - watch!(PEAK_ALLOC.current_usage_as_mb()); Ok(()) } fn update_whitelist(&mut self, new_whitelist_patterns: &[GlobPattern]) -> Result<()> { - watch!(PEAK_ALLOC.current_usage_as_mb()); assert!(new_whitelist_patterns .iter() .all(|p| matches!(p.effect, PatternEffect::Whitelist))); - watch!(PEAK_ALLOC.current_usage_as_mb()); { let mut whitelist_patterns = self.whitelist_patterns.write()?; - watch!(PEAK_ALLOC.current_usage_as_mb()); *whitelist_patterns = whitelist_patterns .iter() @@ -100,34 +87,26 @@ impl IgnoreRules { .cloned() .collect(); } - watch!(PEAK_ALLOC.current_usage_as_mb()); { - watch!(PEAK_ALLOC.current_usage_as_mb()); let whitelist_globs = self .whitelist_patterns .read()? .iter() .map(|g| g.pattern.clone()) .collect(); - watch!(PEAK_ALLOC.current_usage_as_mb()); let whitelist_set = build_globset(whitelist_globs)?; - watch!(PEAK_ALLOC.current_usage_as_mb()); *self.whitelist_set.write()? = whitelist_set; } - watch!(PEAK_ALLOC.current_usage_as_mb()); Ok(()) } fn update_ignore(&mut self, new_ignore_patterns: &[GlobPattern]) -> Result<()> { - watch!(PEAK_ALLOC.current_usage_as_mb()); assert!(new_ignore_patterns .iter() .all(|p| matches!(p.effect, PatternEffect::Ignore))); - watch!(PEAK_ALLOC.current_usage_as_mb()); { let mut ignore_patterns = self.ignore_patterns.write()?; - watch!(PEAK_ALLOC.current_usage_as_mb()); *ignore_patterns = ignore_patterns .iter() @@ -136,7 +115,6 @@ impl IgnoreRules { .cloned() .collect(); } - watch!(PEAK_ALLOC.current_usage_as_mb()); { let ignore_globs = self @@ -147,9 +125,7 @@ impl IgnoreRules { .collect(); let ignore_set = build_globset(ignore_globs)?; *self.ignore_set.write()? = ignore_set; - watch!(PEAK_ALLOC.current_usage_as_mb()); } - watch!(PEAK_ALLOC.current_usage_as_mb()); Ok(()) } From a0bfe41b7da320ff05bfa1fcaed749133e64f4ff Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Thu, 8 Aug 2024 12:35:09 +0300 Subject: [PATCH 162/257] remove glob erro --- walker/src/error.rs | 6 ------ 1 file changed, 6 deletions(-) diff --git a/walker/src/error.rs b/walker/src/error.rs index fa4f92de1..8fffe1e8a 100644 --- a/walker/src/error.rs +++ b/walker/src/error.rs @@ -18,12 +18,6 @@ pub enum Error { #[error("Ignore rules poisoned")] LockPoisonError { t: String, cause: String }, - #[error("Glob error: {source}")] - GlobError { - #[from] - source: globset::Error, - }, - #[error("File System Notify Error: {source:?}")] NotifyError { #[from] From f5d5b54608e6e2a00423a4c9585a6a4d88ee75e6 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Thu, 8 Aug 2024 12:35:18 +0300 Subject: [PATCH 163/257] use jwalk and fast-glob --- walker/Cargo.toml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/walker/Cargo.toml b/walker/Cargo.toml index 4bbbba2e0..bd0f13e34 100644 --- a/walker/Cargo.toml +++ b/walker/Cargo.toml @@ -17,7 +17,7 @@ crate-type = ["rlib"] [dependencies] xvc-logging = { version = "0.6.11-alpha.8", path = "../logging" } -globset = "^0.4" +fast-glob = "^0.3" ## Parallelization crossbeam-channel = "^0.5" @@ -25,6 +25,7 @@ crossbeam = "^0.8" ## File system notify = "6.1.1" +jwalk = "^0.8" dashmap = "^6" From 8ad1ffb95e4c4a5085a59c5fcd3b1d31a0eaa308 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Mon, 12 Aug 2024 21:00:17 +0300 Subject: [PATCH 164/257] Remove globset and use fast_glob in ignore --- core/src/check_ignore/mod.rs | 14 +- core/src/lib.rs | 4 +- core/src/util/git.rs | 23 +- core/src/util/pmp.rs | 9 +- core/src/util/xvcignore.rs | 21 +- file/src/common/gitignore.rs | 13 +- file/src/common/mod.rs | 83 ++- file/src/list/mod.rs | 2 +- file/src/track/mod.rs | 12 +- lib/src/git.rs | 13 + lib/tests/test_core_util_file_walker.rs | 4 +- lib/tests/test_core_util_notify.rs | 8 +- lib/tests/test_walker_parallel.rs | 14 +- lib/tests/test_walker_serial.rs | 34 +- walker/src/ignore_rules.rs | 246 +++++--- walker/src/lib.rs | 759 +++--------------------- walker/src/notify.rs | 7 +- walker/src/pattern.rs | 231 ++++++++ walker/src/walk_parallel.rs | 109 ++++ walker/src/walk_serial.rs | 89 +++ 20 files changed, 799 insertions(+), 896 deletions(-) create mode 100644 walker/src/pattern.rs create mode 100644 walker/src/walk_parallel.rs create mode 100644 walker/src/walk_serial.rs diff --git a/core/src/check_ignore/mod.rs b/core/src/check_ignore/mod.rs index e64146771..5d2d538c7 100644 --- a/core/src/check_ignore/mod.rs +++ b/core/src/check_ignore/mod.rs @@ -10,7 +10,7 @@ use std::io::BufRead; use std::path::{Path, PathBuf}; use xvc_config::{UpdateFromXvcConfig, XvcConfig}; use xvc_logging::{output, XvcOutputSender}; -use xvc_walker::{build_ignore_rules, check_ignore, IgnoreRules, MatchResult, WalkOptions}; +use xvc_walker::{build_ignore_patterns, IgnoreRules, MatchResult, WalkOptions}; // DIFFERENCES from DVC // merged --all and --details, they are the same now @@ -77,13 +77,13 @@ pub fn cmd_check_ignore( let current_dir = conf.current_dir()?; let walk_options = WalkOptions { - ignore_filename: Some(opts.ignore_filename.clone()), + ignore_filename: Some(opts.ignore_filename.clone().into()), include_dirs: true, }; - let initial_rules = IgnoreRules::try_from_patterns(xvc_root, COMMON_IGNORE_PATTERNS)?; - let ignore_rules = build_ignore_rules( - initial_rules, - current_dir, + + let ignore_rules = build_ignore_patterns( + COMMON_IGNORE_PATTERNS, + &xvc_root, &walk_options.ignore_filename.unwrap_or_default(), )?; if !opts.targets.is_empty() { @@ -149,7 +149,7 @@ fn check_ignore_line( absolute_path: &Path, show_no_match: bool, ) -> String { - match check_ignore(ignore_rules, absolute_path) { + match ignore_rules.check(absolute_path) { MatchResult::NoMatch => { if show_no_match { format!("[NO MATCH] {}", absolute_path.to_string_lossy()) diff --git a/core/src/lib.rs b/core/src/lib.rs index 2009cdb4a..8daa78477 100755 --- a/core/src/lib.rs +++ b/core/src/lib.rs @@ -8,6 +8,8 @@ pub mod root; pub mod types; pub mod util; +use std::ffi::OsStr; + pub use types::hashalgorithm::HashAlgorithm; pub use types::recheckmethod::RecheckMethod; @@ -48,8 +50,6 @@ pub use util::file::{all_paths_and_metadata, dir_includes, glob_includes, glob_p pub use util::pmp::XvcPathMetadataProvider; pub use util::XvcPathMetadataMap; -pub use xvc_walker::PEAK_ALLOC; - /// Channel size for [crossbeam_channel::bounded] used across the library. /// TODO: This can be configurable for smaller/larger RAM sizes. pub const CHANNEL_BOUND: usize = 1000000; diff --git a/core/src/util/git.rs b/core/src/util/git.rs index 1bf4652e1..88d4fa192 100644 --- a/core/src/util/git.rs +++ b/core/src/util/git.rs @@ -2,13 +2,11 @@ use std::path::{Path, PathBuf}; use xvc_logging::watch; -use xvc_walker::{build_ignore_rules, AbsolutePath, IgnoreRules}; +use xvc_walker::{build_ignore_patterns, AbsolutePath, IgnoreRules}; use crate::error::Result; use crate::GIT_DIR; -use crate::PEAK_ALLOC; - /// Check whether a path is inside a Git repository. /// It returns `None` if not, otherwise returns the closest directory with `.git`. /// It works by checking `.git` directories in parents, until no more parent left. @@ -31,12 +29,7 @@ pub fn inside_git(path: &Path) -> Option { /// It's used to check whether a path is already ignored by Git. pub fn build_gitignore(git_root: &AbsolutePath) -> Result { - watch!(PEAK_ALLOC.current_usage_as_mb()); - let initial_rules = IgnoreRules::empty(git_root); - watch!(PEAK_ALLOC.current_usage_as_mb()); - - let rules = build_ignore_rules(initial_rules, git_root, ".gitignore")?; - watch!(PEAK_ALLOC.current_usage_as_mb()); + let rules = build_ignore_patterns("", git_root, ".gitignore".to_owned().as_ref())?; Ok(rules) } @@ -44,11 +37,13 @@ pub fn build_gitignore(git_root: &AbsolutePath) -> Result { #[cfg(test)] mod test { use super::*; + use std::ffi::OsStr; + use std::ffi::OsString; use std::fs; use test_case::test_case; use xvc_logging::watch; use xvc_test_helper::*; - use xvc_walker::check_ignore; + use xvc_walker::IgnoreRules; use xvc_walker::MatchResult as M; #[test_case("myfile.txt" , ".gitignore", "/myfile.txt" => matches M::Ignore ; "myfile.txt")] @@ -70,15 +65,15 @@ mod test { } fs::write(&gitignore_path, format!("{}\n", ignore_line)).unwrap(); - let gitignore = build_ignore_rules( - IgnoreRules::empty(&git_root), + let gitignore = build_ignore_patterns( + "", gitignore_path.parent().unwrap(), - ".gitignore", + OsString::from(".gitignore").as_ref() ) .unwrap(); watch!(gitignore); - check_ignore(&gitignore, &path) + gitignore.check(&path) } } diff --git a/core/src/util/pmp.rs b/core/src/util/pmp.rs index 80718d9e2..e493f8c36 100644 --- a/core/src/util/pmp.rs +++ b/core/src/util/pmp.rs @@ -2,18 +2,18 @@ use std::collections::HashMap; +use std::ffi::OsString; use std::sync::{Arc, Mutex, RwLock}; use std::thread::{self, JoinHandle}; use std::time::Duration; use xvc_logging::{error, uwr, watch, XvcOutputSender}; -use xvc_walker::{build_ignore_rules, make_watcher, IgnoreRules, MatchResult, PathEvent}; +use xvc_walker::{build_ignore_patterns, make_watcher, IgnoreRules, MatchResult, PathEvent}; use crate::error::Error; use crate::error::Result; use crate::util::xvcignore::COMMON_IGNORE_PATTERNS; use crate::{XvcFileType, XVCIGNORE_FILENAME}; use crossbeam_channel::{bounded, RecvError, Select, Sender}; -use xvc_walker::check_ignore; use crate::types::{xvcpath::XvcPath, xvcroot::XvcRoot}; use crate::XvcMetadata; @@ -36,8 +36,7 @@ pub struct XvcPathMetadataProvider { impl XvcPathMetadataProvider { /// Create a new PathMetadataProvider pub fn new(output_sender: &XvcOutputSender, xvc_root: &XvcRoot) -> Result { - let initial_rules = IgnoreRules::try_from_patterns(xvc_root, COMMON_IGNORE_PATTERNS)?; - let ignore_rules = build_ignore_rules(initial_rules, xvc_root, XVCIGNORE_FILENAME)?; + let ignore_rules = build_ignore_patterns(COMMON_IGNORE_PATTERNS, xvc_root, OsString::from(XVCIGNORE_FILENAME).as_ref())?; let path_map = Arc::new(RwLock::new(HashMap::new())); let (watcher, event_receiver) = make_watcher(ignore_rules.clone())?; @@ -195,7 +194,7 @@ impl XvcPathMetadataProvider { match entry { Ok(entry) => { if matches!( - check_ignore(&self.ignore_rules, &entry), + &self.ignore_rules.check(&entry), MatchResult::Ignore ) { continue; diff --git a/core/src/util/xvcignore.rs b/core/src/util/xvcignore.rs index 90a79e44b..2b1f60895 100644 --- a/core/src/util/xvcignore.rs +++ b/core/src/util/xvcignore.rs @@ -6,7 +6,8 @@ use crate::{XvcMetadata, XvcPathMetadataMap, CHANNEL_BOUND, XVCIGNORE_FILENAME}; use crate::error::{Error, Result}; use crossbeam_channel::{bounded, Sender}; -use std::sync::{Arc, RwLock}; +use std::ffi::OsString; +use std::sync::{Arc, Mutex, RwLock}; use std::thread; use xvc_logging::{warn, XvcOutputSender}; use xvc_walker::Result as XvcWalkerResult; @@ -39,14 +40,12 @@ pub fn walk_serial( xvc_root: &XvcRoot, include_dirs: bool, ) -> Result<(XvcPathMetadataMap, IgnoreRules)> { - // We assume ignore_src is among the directories created - let initial_rules = IgnoreRules::try_from_patterns(xvc_root, COMMON_IGNORE_PATTERNS)?; let walk_options = WalkOptions { - ignore_filename: Some(XVCIGNORE_FILENAME.to_string()), + ignore_filename: Some(OsString::from(XVCIGNORE_FILENAME)), include_dirs, }; let (res_paths, ignore_rules) = - xvc_walker::walk_serial(output_snd, initial_rules, xvc_root, &walk_options)?; + xvc_walker::walk_serial::walk_serial(output_snd, COMMON_IGNORE_PATTERNS, xvc_root, &walk_options)?; let pmp: XvcPathMetadataMap = res_paths .iter() .filter_map(|pm| { @@ -110,7 +109,7 @@ pub fn walk_parallel( pmm }); - let mut ignore_rules = IgnoreRules::empty(xvc_root); + let ignore_rules = IgnoreRules::empty(xvc_root); let ignore_rules_thread = thread::spawn(move || { for ignore_rule in ignore_receiver { if let Ok(ignore_rule) = ignore_rule { @@ -168,16 +167,16 @@ pub fn walk_channel( xpm_upstream: Sender<(XvcPath, XvcMetadata)>, ignore_upstream: Sender>, ) -> Result<()> { - let initial_rules = IgnoreRules::try_from_patterns(xvc_root, initial_patterns)?; + let initial_rules = IgnoreRules::from_global_patterns(xvc_root, initial_patterns); let walk_options = WalkOptions { - ignore_filename, + ignore_filename: ignore_filename.map(OsString::from), include_dirs, }; let (path_sender, path_receiver) = bounded::>(CHANNEL_BOUND); let (ignore_sender, ignore_receiver) = - bounded::>>>(CHANNEL_BOUND); + bounded::>>>(CHANNEL_BOUND); - xvc_walker::walk_parallel( + xvc_walker::walk_parallel::walk_parallel( initial_rules, xvc_root, walk_options, @@ -216,7 +215,7 @@ pub fn walk_channel( for ignore_rule in ignore_receiver { match ignore_rule { Ok(ir) => { - ir.read() + ir.lock() .map(|ir| { ignore_upstream .send(Ok(ir.clone())) diff --git a/file/src/common/gitignore.rs b/file/src/common/gitignore.rs index 4c6db75ed..43cb6ab0d 100644 --- a/file/src/common/gitignore.rs +++ b/file/src/common/gitignore.rs @@ -12,7 +12,7 @@ use xvc_core::{util::git::build_gitignore}; use crate::{Result, CHANNEL_CAPACITY}; use xvc_core::{XvcPath, XvcRoot}; use xvc_logging::{debug, error, info, uwr, watch, XvcOutputSender}; -use xvc_walker::{check_ignore, AbsolutePath, IgnoreRules, MatchResult}; +use xvc_walker::{AbsolutePath, IgnoreRules, MatchResult}; /// Used to signal ignored files and directories to the ignore handler pub enum IgnoreOperation { @@ -42,6 +42,8 @@ pub fn make_ignore_handler( let (sender, receiver) = crossbeam_channel::bounded(CHANNEL_CAPACITY); let output_snd = output_snd.clone(); let xvc_root = xvc_root.absolute_path().clone(); + let xvc_directory = xvc_root.as_path().to_str().unwrap(); + let handle = std::thread::spawn(move || { @@ -56,7 +58,8 @@ pub fn make_ignore_handler( let path = dir.to_absolute_path(&xvc_root).to_path_buf(); if !ignore_dirs.contains(&dir) - && matches!(check_ignore(&gitignore, &path), MatchResult::NoMatch) + + && matches!(gitignore.check(&path), MatchResult::NoMatch) { ignore_dirs.push(dir); } @@ -64,7 +67,7 @@ pub fn make_ignore_handler( IgnoreOperation::IgnoreFile { file } => { let path = file.to_absolute_path(&xvc_root).to_path_buf(); if !ignore_files.contains(&file) - && matches!(check_ignore(&gitignore, &path), MatchResult::NoMatch) + && matches!(gitignore.check(&path), MatchResult::NoMatch) { ignore_files.push(file); } @@ -115,7 +118,7 @@ pub fn update_dir_gitignores( xvc_root.join(format!("{}/", dir)) }; - let ignore_res = check_ignore(current_gitignore, &abs_path); + let ignore_res = current_gitignore.check(&abs_path); match ignore_res { MatchResult::Ignore => { @@ -180,7 +183,7 @@ pub fn update_file_gitignores( files: &[XvcPath], ) -> Result<()> { // Filter already ignored files - let files: Vec = files.iter().filter_map(|f| match check_ignore(current_gitignore, &f.to_absolute_path(xvc_root)) { + let files: Vec = files.iter().filter_map(|f| match current_gitignore.check(&f.to_absolute_path(xvc_root)) { MatchResult::NoMatch => { Some(f.clone()) } diff --git a/file/src/common/mod.rs b/file/src/common/mod.rs index 49eedb973..4ccce0a98 100644 --- a/file/src/common/mod.rs +++ b/file/src/common/mod.rs @@ -28,7 +28,7 @@ use xvc_logging::{error, info, uwr, warn, watch, XvcOutputSender}; use xvc_ecs::{persist, HStore, Storable, XvcStore}; -use xvc_walker::{AbsolutePath, Error as XvcWalkerError, Glob, GlobSetBuilder, PathSync}; +use xvc_walker::{AbsolutePath, Error as XvcWalkerError, Glob, IgnoreRules, PathSync, Source}; use self::gitignore::IgnoreOp; @@ -146,30 +146,65 @@ pub fn filter_targets_from_store( /// Filter a set of paths by a set of globs. The globs are compiled into a /// GlobSet and paths are checked against the set. pub fn filter_paths_by_globs(paths: &HStore, globs: &[String]) -> Result> { - let mut glob_matcher = GlobSetBuilder::new(); - globs.iter().for_each(|t| { + if globs.is_empty() { + return Ok(paths.to_owned()) + } + + let mut glob_matcher = globs.first().and_then(|glob| Glob::new(glob)).unwrap(); + globs.iter().skip(1).for_each(|t| { watch!(t); if t.ends_with('/') { - glob_matcher.add(Glob::new(&format!("{t}**")).expect("Error in glob: {t}**")); + glob_matcher.add(&format!("{t}**")); } else { - glob_matcher.add(Glob::new(&format!("{t}/**")).expect("Error in glob: {t}/**")); + glob_matcher.add(&format!("{t}/**")); } - glob_matcher.add(Glob::new(t).expect("Error in glob: {t}")); }); - let glob_matcher = glob_matcher.build().map_err(XvcWalkerError::from)?; let paths = paths - .filter(|_, p| { + .iter() + .filter_map(|(e, p)| { let str_path = &p.as_relative_path().as_str(); - glob_matcher.is_match(str_path) + if glob_matcher.is_match(str_path) { + Some((*e, p.clone())) + } else { + None + } }) - .cloned(); - + .collect(); watch!(paths); + Ok(paths) } +pub fn build_glob_matcher(output_snd: &XvcOutputSender, dir: &Path, globs: &[String]) -> Result { + + let source = Source::Global; + + let mut glob_matcher = globs.first().and_then(|glob| Glob::new(glob)).unwrap(); + globs.iter().skip(1).for_each(|t| { + globs.iter().for_each(|t| { + if t.ends_with('/') { + if !glob_matcher.add(&format!("{t}**")) { + error!(output_snd, "Error in glob: {t}"); + } + } else if !t.contains('*') { + let abs_target = dir.join(Path::new(t)); + if abs_target.is_dir() { + if !glob_matcher.add(&format!("{t}/**")) { + error!(output_snd, "Error in glob: {t}") + } + } else if !glob_matcher.add(t) { + error!(output_snd, "Error in glob: {t}") + } + } else if !glob_matcher.add(t) { + error!(output_snd, "Error in glob: {t}") + } + }); + }); + Ok(glob_matcher) +} + /// Converts targets to a map of XvcPaths and their metadata. It walks the file /// system with [`all_paths_and_metadata`]. This is aimed towards `xvc file /// track`, `xvc file hash` and similar commands where we work with the existing @@ -186,6 +221,7 @@ pub fn filter_paths_by_globs(paths: &HStore, globs: &[String]) -> Resul /// repositories. pub fn targets_from_disk( + output_snd: &XvcOutputSender, xvc_root: &XvcRoot, current_dir: &AbsolutePath, targets: &Option>, @@ -203,29 +239,20 @@ pub fn targets_from_disk( None => vec![cwd.to_string()], }; - return targets_from_disk(xvc_root, xvc_root.absolute_path(), &Some(targets)); + return targets_from_disk(output_snd, xvc_root, xvc_root.absolute_path(), &Some(targets)); } + // FIXME: If there are no globs/directories in the targets, no need to retrieve all the paths + // here. let (all_paths, _) = all_paths_and_metadata(xvc_root); watch!(all_paths); if let Some(targets) = targets { - let mut glob_matcher = GlobSetBuilder::new(); - targets.iter().for_each(|t| { - if t.ends_with('/') { - glob_matcher.add(Glob::new(&format!("{t}**")).expect("Error in glob: {t}**")); - } else if !t.contains('*') { - let abs_target = current_dir.join(Path::new(t)); - if abs_target.is_dir() { - glob_matcher.add(Glob::new(&format!("{t}/**")).expect("Error in glob: {t}/**")); - } else { - glob_matcher.add(Glob::new(t).expect("Error in glob: {t}")); - } - } else { - glob_matcher.add(Glob::new(t).expect("Error in glob: {t}")); - } - }); - let glob_matcher = glob_matcher.build().map_err(XvcWalkerError::from)?; + if targets.is_empty() { + return Ok(XvcPathMetadataMap::new()); + } + + let mut glob_matcher = build_glob_matcher(output_snd, &xvc_root, targets)?; watch!(glob_matcher); Ok(all_paths .into_iter() diff --git a/file/src/list/mod.rs b/file/src/list/mod.rs index e4f45aa2c..1901ac10c 100644 --- a/file/src/list/mod.rs +++ b/file/src/list/mod.rs @@ -530,7 +530,7 @@ pub fn cmd_list(output_snd: &XvcOutputSender, xvc_root: &XvcRoot, cli_opts: List let current_dir = conf.current_dir()?; - let all_from_disk = targets_from_disk(xvc_root, current_dir, &opts.targets)?; + let all_from_disk = targets_from_disk(output_snd, xvc_root, current_dir, &opts.targets)?; let from_disk = if opts.show_dot_files { all_from_disk } else { diff --git a/file/src/track/mod.rs b/file/src/track/mod.rs index 6c4ca4f13..794ed8925 100644 --- a/file/src/track/mod.rs +++ b/file/src/track/mod.rs @@ -14,7 +14,7 @@ use xvc_config::FromConfigKey; use xvc_config::{UpdateFromXvcConfig, XvcConfig}; use xvc_core::util::git::build_gitignore; -use xvc_core::{ContentDigest, HashAlgorithm, XvcCachePath, XvcFileType, XvcMetadata, XvcRoot, PEAK_ALLOC}; +use xvc_core::{ContentDigest, HashAlgorithm, XvcCachePath, XvcFileType, XvcMetadata, XvcRoot}; use xvc_logging::{watch, XvcOutputSender}; use crate::carry_in::carry_in; @@ -118,7 +118,7 @@ pub fn cmd_track( let conf = xvc_root.config(); let opts = cli_opts.update_from_conf(conf)?; let current_dir = conf.current_dir()?; - let targets = targets_from_disk(xvc_root, current_dir, &opts.targets)?; + let targets = targets_from_disk(output_snd, xvc_root, current_dir, &opts.targets)?; watch!(targets); let requested_recheck_method = opts.recheck_method; let text_or_binary = opts.text_or_binary.unwrap_or_default(); @@ -220,25 +220,17 @@ pub fn cmd_track( }) .collect(); - watch!(PEAK_ALLOC.current_usage_as_mb()); let current_gitignore = build_gitignore(xvc_root)?; - watch!(PEAK_ALLOC.current_usage_as_mb()); watch!(file_targets); - watch!(PEAK_ALLOC.current_usage_as_mb()); watch!(dir_targets); - watch!(PEAK_ALLOC.current_usage_as_mb()); update_dir_gitignores(xvc_root, ¤t_gitignore, &dir_targets)?; - watch!(PEAK_ALLOC.current_usage_as_mb()); // We reload gitignores here to make sure we ignore the given dirs - watch!(PEAK_ALLOC.current_usage_as_mb()); let current_gitignore = build_gitignore(xvc_root)?; - watch!(PEAK_ALLOC.current_usage_as_mb()); update_file_gitignores(xvc_root, ¤t_gitignore, &file_targets)?; - watch!(PEAK_ALLOC.current_usage_as_mb()); if !opts.no_commit { diff --git a/lib/src/git.rs b/lib/src/git.rs index 2e739eeb3..a1ee653f3 100644 --- a/lib/src/git.rs +++ b/lib/src/git.rs @@ -233,3 +233,16 @@ pub fn git_auto_stage( debug!(output_snd, "Staging .xvc/ to git: {res_git_add}"); Ok(()) } + +pub fn git_ignored(output_snd: &XvcOutputSender, + git_command: &str, + xvc_root_str: &str, + path: &str) -> Result { + let command_res = exec_git(git_command, xvc_root_str, &["check-ignore", path])?; + + if command_res.trim().is_empty() { + Ok(false) + } else { + Ok(true) + } +} diff --git a/lib/tests/test_core_util_file_walker.rs b/lib/tests/test_core_util_file_walker.rs index 25e29967c..59ff0a205 100644 --- a/lib/tests/test_core_util_file_walker.rs +++ b/lib/tests/test_core_util_file_walker.rs @@ -30,9 +30,11 @@ fn test_walk() -> Result<()> { assert!(!pmp1.contains_key(&xp), "Result Contains {:?}", xp) } - common::test_logging(LevelFilter::Trace); let (pmp2, _) = walk_parallel(&xvc_root, true)?; + watch!(pmp1); + watch!(pmp2); + let mut diff1 = Vec::<&XvcPath>::new(); for k in pmp1.keys() { if !pmp2.contains_key(k) { diff --git a/lib/tests/test_core_util_notify.rs b/lib/tests/test_core_util_notify.rs index 22cc0ccaf..eeb85c452 100644 --- a/lib/tests/test_core_util_notify.rs +++ b/lib/tests/test_core_util_notify.rs @@ -4,6 +4,7 @@ use assert_fs::prelude::{FileTouch, FileWriteBin, PathChild}; use assert_fs::TempDir; use common::*; use std::env; +use std::ffi::OsString; use std::fs::remove_file; use std::path::PathBuf; use std::thread::{self, sleep}; @@ -16,7 +17,7 @@ use xvc_core::util::xvcignore::COMMON_IGNORE_PATTERNS; use xvc_core::XVCIGNORE_FILENAME; use xvc_walker::notify::{make_polling_watcher, PathEvent}; -use xvc_walker::{walk_serial, IgnoreRules, WalkOptions}; +use xvc_walker::{walk_serial, WalkOptions}; #[test] fn test_notify() -> Result<()> { @@ -24,9 +25,8 @@ fn test_notify() -> Result<()> { env::set_current_dir(&temp_dir)?; watch!(temp_dir); test_logging(log::LevelFilter::Trace); - let initial_rules = IgnoreRules::try_from_patterns(&temp_dir, COMMON_IGNORE_PATTERNS)?; let walk_options = WalkOptions { - ignore_filename: Some(XVCIGNORE_FILENAME.to_string()), + ignore_filename: Some(OsString::from(XVCIGNORE_FILENAME)), include_dirs: true, }; let (created_paths_snd, created_paths_rec) = crossbeam_channel::unbounded(); @@ -48,7 +48,7 @@ fn test_notify() -> Result<()> { let (output_sender, output_receiver) = crossbeam_channel::unbounded(); let (initial_paths, all_rules) = - walk_serial(&output_sender, initial_rules, &temp_dir, &walk_options)?; + walk_serial(&output_sender, COMMON_IGNORE_PATTERNS, &temp_dir, &walk_options)?; watch!(all_rules); assert!(output_receiver.is_empty()); let (watcher, receiver) = make_polling_watcher(all_rules)?; diff --git a/lib/tests/test_walker_parallel.rs b/lib/tests/test_walker_parallel.rs index 756997fc9..0457647a5 100644 --- a/lib/tests/test_walker_parallel.rs +++ b/lib/tests/test_walker_parallel.rs @@ -1,6 +1,5 @@ use std::{ - fs, - path::{Path, PathBuf}, + ffi::OsString, fs, path::{Path, PathBuf} }; use fast_glob::Glob; @@ -45,17 +44,14 @@ fn new_dir_with_ignores( initial_patterns: &str, ) -> Result { let patterns = create_patterns(root, dir, initial_patterns); - let mut initialized = IgnoreRules::empty(&PathBuf::from(root)); + let initialized = IgnoreRules::empty(&PathBuf::from(root)); watch!(patterns); - initialized.update(patterns).unwrap(); + initialized.add_patterns(patterns).unwrap(); Ok(initialized) } -fn create_patterns(root: &str, dir: Option<&str>, patterns: &str) -> Vec> { +fn create_patterns(root: &str, dir: Option<&str>, patterns: &str) -> Vec { xvc_walker::content_to_patterns(Path::new(root), dir.map(Path::new), patterns) - .into_iter() - .map(|pat_res_g| pat_res_g.map(|res_g| res_g.unwrap())) - .collect() } #[test_case("", "" => it contains "dir-0002/file-0001.bin" ; "t3733909666")] @@ -85,7 +81,7 @@ fn test_walk_parallel(ignore_src: &str, ignore_content: &str) -> Vec { .unwrap(); let initial_rules = new_dir_with_ignores(root.to_string_lossy().as_ref(), None, "").unwrap(); let walk_options = WalkOptions { - ignore_filename: Some(".gitignore".to_string()), + ignore_filename: Some(OsString::from(".gitignore")), include_dirs: true, }; walk_parallel( diff --git a/lib/tests/test_walker_serial.rs b/lib/tests/test_walker_serial.rs index 4e1fe6f36..789b85455 100644 --- a/lib/tests/test_walker_serial.rs +++ b/lib/tests/test_walker_serial.rs @@ -1,9 +1,7 @@ use std::{ - fs, - path::{Path, PathBuf}, + ffi::OsString, fs, path::{Path, PathBuf} }; -use globset::Glob; use xvc_walker::*; use test_case::test_case; @@ -13,27 +11,6 @@ use xvc_logging::watch; use xvc_test_helper::*; use xvc_walker::AbsolutePath; -fn new_dir_with_ignores( - root: &str, - dir: Option<&str>, - initial_patterns: &str, -) -> Result { - let patterns = create_patterns(root, dir, initial_patterns); - let mut initialized = IgnoreRules::empty(&PathBuf::from(root)); - watch!(patterns); - initialized.update(patterns).unwrap(); - watch!(initialized.ignore_patterns.read().unwrap()); - watch!(initialized.whitelist_patterns.read().unwrap()); - Ok(initialized) -} - -fn create_patterns(root: &str, dir: Option<&str>, patterns: &str) -> Vec> { - xvc_walker::content_to_patterns(Path::new(root), dir.map(Path::new), patterns) - .into_iter() - .map(|pat_res_g| pat_res_g.map(|res_g| res_g.unwrap())) - .collect() -} - // TODO: Patterns shouldn't have / prefix, but an appropriate PathKind #[test_case(true => matches Ok(_); "this is to refresh the dir for each test run")] // This builds a directory hierarchy to run the tests @@ -82,16 +59,15 @@ fn test_walk_serial(ignore_src: &str, ignore_content: &str) -> Vec { ignore_content, ) .unwrap(); - let initial_rules = new_dir_with_ignores(root.to_string_lossy().as_ref(), None, "").unwrap(); let walk_options = WalkOptions { - ignore_filename: Some(".gitignore".to_string()), + ignore_filename: Some(OsString::from(".gitignore")), include_dirs: true, }; let (output_sender, output_receiver) = crossbeam_channel::unbounded(); let (res_paths, ignore_rules) = - walk_serial(&output_sender, initial_rules, &root, &walk_options).unwrap(); - watch!(ignore_rules.ignore_patterns.read().unwrap()); - watch!(ignore_rules.whitelist_patterns.read().unwrap()); + walk_serial(&output_sender, "", &root, &walk_options).unwrap(); + watch!(ignore_rules.ignore_patterns.lock().unwrap()); + watch!(ignore_rules.whitelist_patterns.lock().unwrap()); watch!(output_receiver); watch!(res_paths); fs::remove_dir_all(&root).unwrap(); diff --git a/walker/src/ignore_rules.rs b/walker/src/ignore_rules.rs index f70d40eab..0c1323ba9 100644 --- a/walker/src/ignore_rules.rs +++ b/walker/src/ignore_rules.rs @@ -1,13 +1,12 @@ //! Ignore patterns for a directory and its child directories. use std::path::{Path, PathBuf}; -use std::sync::{Arc, RwLock}; +use std::sync::{Arc, Mutex}; +use crate::{Result, Source}; -use fast_glob::Glob; -use itertools::Itertools; -use crate::{build_globset, content_to_patterns, GlobPattern, PatternEffect, Result}; +use crate::pattern::{MatchResult, Pattern}; use xvc_logging::watch; -use crate::PEAK_ALLOC; + /// Complete set of ignore rules for a directory and its child directories. #[derive(Debug, Clone)] pub struct IgnoreRules { @@ -16,16 +15,13 @@ pub struct IgnoreRules { pub root: PathBuf, /// All ignore patterns collected from ignore files or specified in code. - pub ignore_patterns: Arc>>, + // FIXME: This should be RwLock instead but as [fast_glob::Glob.is_match] requires &mut self, + // we use a simpler approach. + pub ignore_patterns: Arc>>, /// All whitelist patterns collected from ignore files or specified in code - pub whitelist_patterns: Arc>>, - - /// Compiled [GlobSet] for whitelisted paths. - pub whitelist_set: Arc>, + pub whitelist_patterns: Arc>>, - /// Compiled [GlobSet] for ignored paths. - pub ignore_set: Arc>, } impl IgnoreRules { @@ -33,100 +29,172 @@ impl IgnoreRules { pub fn empty(dir: &Path) -> Self { IgnoreRules { root: PathBuf::from(dir), - ignore_patterns: Arc::new(RwLock::new(Vec::::new())), - whitelist_patterns: Arc::new(RwLock::new(Vec::::new())), - ignore_set: Arc::new(RwLock::new(Glob::new("").expect("Empty glob is valid"))), - whitelist_set: Arc::new(RwLock::new(Glob::new("").expect("Empty glob is valid"))), + ignore_patterns: Arc::new(Mutex::new(Vec::::new())), + whitelist_patterns: Arc::new(Mutex::new(Vec::::new())), } } - /// Compiles patterns as [Source::Global] and initializes the elements. - pub fn try_from_patterns(root: &Path, patterns: &str) -> Result { - - let patterns = content_to_patterns(root, None, patterns) - .into_iter() - .map(|pat_res_g| pat_res_g.map(|res_g| res_g.unwrap())) - .collect(); - let mut initialized = Self::empty(&PathBuf::from(root)); + pub fn from_global_patterns(ignore_root: &Path, given: &str) -> Self { - initialized.update(patterns)?; - Ok(initialized) - } - - /// Adds `new_patterns` to the list of patterns and recompiles ignore and - /// whitelist [GlobSet]s. - pub fn update(&mut self, new_patterns: Vec) -> Result<()> { - let (new_ignore_patterns, new_whitelist_patterns): (Vec<_>, Vec<_>) = new_patterns - .into_iter() - .partition(|p| matches!(p.effect, PatternEffect::Ignore)); - self.update_ignore(&new_ignore_patterns)?; - self.update_whitelist(&new_whitelist_patterns)?; - Ok(()) + let mut given_patterns = Vec::::new(); + // Add given patterns to ignore_patterns + for line in given.lines() { + let pattern = Pattern::new(Source::Global, line); + given_patterns.push(pattern); } + IgnoreRules::from_patterns(given_patterns, ignore_root) + } - /// Merge with other ignore rules, extending this one's patterns and rebuilding glob sets. - pub fn merge_with(&mut self, other: &IgnoreRules) -> Result<()> { - assert_eq!(self.root, other.root); - self.update_ignore(&other.ignore_patterns.read().unwrap())?; - self.update_whitelist(&other.whitelist_patterns.read().unwrap())?; - Ok(()) +/// Constructs a new `IgnoreRules` instance from a vector of patterns and a root path. +/// +/// This function separates the patterns into ignore patterns and whitelist patterns +/// based on their `PatternEffect`. It then stores these patterns and the root path +/// in a new `IgnoreRules` instance. +/// +/// # Arguments +/// +/// * `patterns` - A vector of `Pattern` instances to be used for creating the `IgnoreRules`. +/// * `ignore_root` - A reference to the root path for the ignore rules. +/// +/// # Returns +/// +/// A new `IgnoreRules` instance containing the given patterns and root path. + pub fn from_patterns(mut patterns: Vec, ignore_root: &Path) -> Self { + let mut ignore_patterns = Vec::new(); + let mut whitelist_patterns = Vec::new(); + patterns.drain(0..patterns.len()).for_each(|pattern| { + match pattern.effect { + crate::PatternEffect::Ignore => ignore_patterns.push(pattern), + crate::PatternEffect::Whitelist => whitelist_patterns.push(pattern), + } + }); + IgnoreRules { + root: PathBuf::from(ignore_root), + ignore_patterns: Arc::new(Mutex::new(ignore_patterns)), + whitelist_patterns: Arc::new(Mutex::new(whitelist_patterns)), + } } - fn update_whitelist(&mut self, new_whitelist_patterns: &[GlobPattern]) -> Result<()> { - assert!(new_whitelist_patterns - .iter() - .all(|p| matches!(p.effect, PatternEffect::Whitelist))); - { - let mut whitelist_patterns = self.whitelist_patterns.write()?; - - *whitelist_patterns = whitelist_patterns - .iter() - .chain(new_whitelist_patterns.iter()) - .unique() - .cloned() - .collect(); + + +/// Checks if a given path matches any of the whitelist or ignore patterns. +/// +/// The function first checks if the path matches any of the whitelist patterns. +/// If a match is found, it returns `MatchResult::Whitelist`. +/// +/// If the path does not match any of the whitelist patterns, the function then checks +/// if the path matches any of the ignore patterns. If a match is found, it returns +/// `MatchResult::Ignore`. +/// +/// If the path does not match any of the whitelist or ignore patterns, the function +/// returns `MatchResult::NoMatch`. +/// +/// # Arguments +/// +/// * `path` - A reference to the path to check. +/// +/// # Returns +/// +/// * `MatchResult::Whitelist` if the path matches a whitelist pattern. +/// * `MatchResult::Ignore` if the path matches an ignore pattern. +/// * `MatchResult::NoMatch` if the path does not match any pattern. + pub fn check(&self, path: &Path) -> MatchResult { + let is_abs = path.is_absolute(); + watch!(is_abs); + // strip_prefix eats the final slash, and ends_with behave differently than str, so we work + // around here + let path_str = path.to_string_lossy(); + watch!(path_str); + let final_slash = path_str.ends_with('/'); + watch!(final_slash); + + let path = if is_abs { + + if final_slash { + format!( + "/{}/", + path.strip_prefix(&self.root) + .expect("path must be within root") + .to_string_lossy() + ) + } else { + format!( + "/{}", + path.strip_prefix(&self.root) + .expect("path must be within root") + .to_string_lossy() + ) + } + } else { + path_str.to_string() + }; + + + { + let mut whitelist_patterns = self.whitelist_patterns.lock().unwrap(); + + for pattern in whitelist_patterns.iter_mut() { + let glob = &mut pattern.glob; + if glob.is_match(&path) { + return MatchResult::Whitelist; + } + } } - { - let whitelist_globs = self - .whitelist_patterns - .read()? - .iter() - .map(|g| g.pattern.clone()) - .collect(); - let whitelist_set = build_globset(whitelist_globs)?; - *self.whitelist_set.write()? = whitelist_set; + { + let mut ignore_patterns = self.ignore_patterns.lock().unwrap(); + for pattern in ignore_patterns.iter_mut() { + if pattern.glob.is_match(&path) { + return MatchResult::Ignore; + } + } } - Ok(()) + MatchResult::NoMatch } - fn update_ignore(&mut self, new_ignore_patterns: &[GlobPattern]) -> Result<()> { - assert!(new_ignore_patterns - .iter() - .all(|p| matches!(p.effect, PatternEffect::Ignore))); - { - let mut ignore_patterns = self.ignore_patterns.write()?; - - *ignore_patterns = ignore_patterns - .iter() - .chain(new_ignore_patterns.iter()) - .unique() - .cloned() - .collect(); + +/// Merges the ignore and whitelist patterns of another `IgnoreRules` instance into this one. +/// +/// This function locks the ignore and whitelist patterns of both `IgnoreRules` instances, +/// drains the patterns from the other instance, and pushes them into this instance. +/// The other instance is left empty after this operation. +/// +/// # Arguments +/// +/// * `other` - A reference to the other `IgnoreRules` instance to merge with. +/// +/// # Returns +/// +/// * `Ok(())` if the merge operation was successful. +/// * `Err` if the merge operation failed. +/// +/// # Panics +/// +/// This function will panic if the roots of the two `IgnoreRules` instances are not equal. + pub fn merge_with(&self, other: &IgnoreRules) -> Result<()> { + assert_eq!(self.root, other.root); + + { + let mut ignore_patterns = self.ignore_patterns.lock().unwrap(); + let mut other_ignore_patterns = other.ignore_patterns.lock().unwrap(); + let len = other_ignore_patterns.len(); + other_ignore_patterns.drain(0..len).for_each(|p| ignore_patterns.push(p.into())); } - { - let ignore_globs = self - .ignore_patterns - .read()? - .iter() - .map(|g| g.pattern.clone()) - .collect(); - let ignore_set = build_globset(ignore_globs)?; - *self.ignore_set.write()? = ignore_set; + { + let mut whitelist_patterns = self.whitelist_patterns.lock().unwrap(); + let mut other_whitelist_patterns = other.whitelist_patterns.lock().unwrap(); + let len = other_whitelist_patterns.len(); + other_whitelist_patterns.drain(0..len).for_each(|p| whitelist_patterns.push(p.into())); + } Ok(()) } -} + + pub fn add_patterns(&self, patterns: Vec) -> Result<()> { + let other = IgnoreRules::from_patterns(patterns, &self.root); + self.merge_with(&other) + } + } diff --git a/walker/src/lib.rs b/walker/src/lib.rs index 1dd6bbfeb..5110e93a9 100755 --- a/walker/src/lib.rs +++ b/walker/src/lib.rs @@ -13,35 +13,34 @@ pub mod error; pub mod ignore_rules; pub mod notify; pub mod sync; +pub mod pattern; +pub mod walk_serial; +pub mod walk_parallel; + +pub use pattern::Pattern; +pub use pattern::PatternRelativity; +pub use pattern::PathKind; +pub use pattern::PatternEffect; +pub use pattern::Source; +pub use pattern::MatchResult; + +pub use walk_parallel::walk_parallel; +pub use walk_serial::walk_serial; pub use abspath::AbsolutePath; -use crossbeam::queue::SegQueue; pub use error::{Error, Result}; pub use ignore_rules::IgnoreRules; -use jwalk::WalkDirGeneric; pub use notify::make_watcher; -use std::borrow::BorrowMut; +use std::ffi::OsStr; pub use std::hash::Hash; -use std::ops::DerefMut; -use std::sync::Arc; -use std::sync::Mutex; -use std::sync::RwLock; pub use sync::{PathSync, PathSyncSingleton}; -use xvc_logging::debug; use xvc_logging::warn; -use xvc_logging::XvcOutputSender; pub use notify::PathEvent; pub use notify::RecommendedWatcher; use xvc_logging::watch; -use peak_alloc::PeakAlloc; - -#[global_allocator] -pub static PEAK_ALLOC: PeakAlloc = PeakAlloc; - -use crossbeam_channel::Sender; // use glob::{MatchOptions, Pattern, PatternError}; pub use fast_glob::Glob; use std::{ @@ -64,112 +63,12 @@ pub struct PathMetadata { pub metadata: Metadata, } -/// Show whether a path matches to a glob rule -#[derive(Debug, Clone)] -pub enum MatchResult { - /// There is no match between glob(s) and path - NoMatch, - /// Path matches to ignored glob(s) - Ignore, - /// Path matches to whitelisted glob(s) - Whitelist, -} - -/// Is the pattern matches anywhere or only relative to a directory? -#[derive(Debug, Clone, Hash, PartialEq, Eq)] -pub enum PatternRelativity { - /// Match the path regardless of the directory prefix - Anywhere, - /// Match the path if it only starts with `directory` - RelativeTo { - /// The directory that the pattern must have as prefix to be considered a match - directory: String, - }, -} - -/// Is the path only a directory, or could it be directory or file? -#[derive(Debug, Clone, Hash, PartialEq, Eq)] -pub enum PathKind { - /// Path matches to directory or file - Any, - /// Path matches only to directory - Directory, -} - -/// Is this pattern a ignore or whitelist patter? -#[derive(Debug, Clone, Eq, PartialEq, Hash)] -pub enum PatternEffect { - /// This is an ignore pattern - Ignore, - /// This is a whitelist pattern - Whitelist, -} - -/// Do we get this pattern from a file (.gitignore, .xvcignore, ...) or specify it directly in -/// code? -#[derive(Debug, Clone, Hash, PartialEq, Eq)] -pub enum Source { - /// Pattern is obtained from file - File { - /// Path of the pattern file - path: PathBuf, - /// (1-based) line number the pattern retrieved - line: usize, - }, - /// Pattern is globally defined in code - Global, -} - -/// Pattern is generic and could be an instance of String, Glob, Regex or any other object. -/// The type is evolved by compiling. -/// A pattern can start its life as `Pattern` and can be compiled into `Pattern` or -/// `Pattern`. -#[derive(Debug, Clone, Eq, PartialEq, Hash)] -pub struct Pattern -where - T: PartialEq + Hash, -{ - /// The pattern type - pub pattern: T, - /// The original string that defines the pattern - original: String, - /// Where did we get this pattern? - source: Source, - /// Is this ignore or whitelist pattern? - effect: PatternEffect, - /// Does it have an implied prefix? - relativity: PatternRelativity, - /// Is the path a directory or anything? - path_kind: PathKind, -} - -impl Pattern { - /// Runs a function (like `compile`) on `pattern` to get a new pattern. - pub fn map(self, f: F) -> Pattern - where - U: PartialEq + Hash, - F: FnOnce(T) -> U, - { - Pattern:: { - pattern: f(self.pattern), - original: self.original, - source: self.source, - effect: self.effect, - relativity: self.relativity, - path_kind: self.path_kind, - } - } -} - -/// One of the concrete types that can represent a pattern. -type GlobPattern = Pattern; - /// What's the ignore file name and should we add directories to the result? #[derive(Debug, Clone)] pub struct WalkOptions { /// The ignore filename (`.gitignore`, `.xvcignore`, `.ignore`, etc.) or `None` for not /// ignoring anything. - pub ignore_filename: Option, + pub ignore_filename: Option, /// Should the results include directories themselves? /// Note that they are always traversed, but may not be listed if we're only interested in /// actual files. @@ -181,7 +80,7 @@ impl WalkOptions { /// directories in results. pub fn gitignore() -> Self { Self { - ignore_filename: Some(".gitignore".to_owned()), + ignore_filename: Some(".gitignore".into()), include_dirs: true, } } @@ -190,7 +89,7 @@ impl WalkOptions { /// directories in results. pub fn xvcignore() -> Self { Self { - ignore_filename: Some(".xvcignore".to_owned()), + ignore_filename: Some(".xvcignore".into()), include_dirs: true, } } @@ -213,447 +112,68 @@ impl WalkOptions { } } -fn walk_parallel_inner( - ignore_rules: Arc>, - dir: &Path, - walk_options: WalkOptions, - path_sender: Sender>, - ignore_sender: Sender>>>, -) -> Result> { - let child_paths: Vec = directory_list(dir)? - .into_iter() - .filter_map(|pm_res| match pm_res { - Ok(pm) => Some(pm), - Err(e) => { - path_sender - .send(Err(e)) - .expect("Channel error in walk_parallel"); - None - } - }) - .collect(); - - let dir_with_ignores = if let Some(ignore_filename) = walk_options.ignore_filename.clone() { - let ignore_filename = OsString::from(ignore_filename); - if let Some(ignore_path_metadata) = child_paths - .iter() - .find(|pm| pm.path.file_name() == Some(&ignore_filename)) - { - let ignore_path = dir.join(&ignore_path_metadata.path); - let new_patterns = clear_glob_errors( - &path_sender, - patterns_from_file(&ignore_rules.read()?.root, &ignore_path)?, - ); - watch!(new_patterns); - ignore_rules.write()?.update(new_patterns)?; - watch!(ignore_rules); - ignore_sender.send(Ok(ignore_rules.clone()))?; - ignore_rules - } else { - ignore_rules - } - } else { - ignore_rules - }; - - let mut child_dirs = Vec::::new(); - watch!(child_paths); - - for child_path in child_paths { - match check_ignore(&(*dir_with_ignores.read()?), child_path.path.as_ref()) { - MatchResult::NoMatch | MatchResult::Whitelist => { - watch!(child_path.path); - if child_path.metadata.is_dir() { - if walk_options.include_dirs { - path_sender.send(Ok(child_path.clone()))?; - } - child_dirs.push(child_path); - } else { - path_sender.send(Ok(child_path.clone()))?; - } - } - // We can return anyhow! error here to notice the user that the path is ignored - MatchResult::Ignore => { - watch!(child_path.path); - } - } - } - - Ok(child_dirs) -} - -/// Walk all child paths under `dir` and send non-ignored paths to `path_sender`. -/// Newly found ignore rules are sent through `ignore_sender`. -/// The ignore file name (`.xvcignore`, `.gitignore`, `.ignore`, ...) is set by `walk_options`. -/// -/// It lists elements of a directory, then creates a new crossbeam scope for each child directory and -/// calls itself recursively. It may not be feasible for small directories to create threads. -pub fn walk_parallel( - ignore_rules: IgnoreRules, - dir: &Path, - walk_options: WalkOptions, - path_sender: Sender>, - ignore_sender: Sender>>>, -) -> Result<()> { - let dir_queue = Arc::new(SegQueue::::new()); - - let ignore_rules = Arc::new(RwLock::new(ignore_rules.clone())); - - let child_dirs = walk_parallel_inner( - ignore_rules.clone(), - dir, - walk_options.clone(), - path_sender.clone(), - ignore_sender.clone(), - )?; - - child_dirs.into_iter().for_each(|pm| { - dir_queue.push(pm); - }); - - if dir_queue.is_empty() { - return Ok(()); - } - - crossbeam::scope(|s| { - for thread_i in 0..MAX_THREADS_PARALLEL_WALK { - let path_sender = path_sender.clone(); - let ignore_sender = ignore_sender.clone(); - let walk_options = walk_options.clone(); - let ignore_rules = ignore_rules.clone(); - let dir_queue = dir_queue.clone(); - - s.spawn(move |_| { - watch!(path_sender); - watch!(ignore_sender); - while let Some(pm) = dir_queue.pop() { - let child_dirs = walk_parallel_inner( - ignore_rules.clone(), - &pm.path, - walk_options.clone(), - path_sender.clone(), - ignore_sender.clone(), - ) - .unwrap_or_else(|e| { - path_sender - .send(Err(e)) - .expect("Channel error in walk_parallel"); - Vec::::new() - }); - for child_dir in child_dirs { - dir_queue.push(child_dir); - } - } - watch!("End of thread {}", thread_i); - }); - } - }) - .expect("Error in crossbeam scope in walk_parallel"); - - watch!("End of walk_parallel"); - - Ok(()) -} - -/// Walk `dir` with `walk_options`, with the given _initial_ `ignore_rules`. -/// Note that ignore rules are expanded with the rules given in the `ignore_filename` in -/// `walk_options`. -/// The result is added to given `res_paths` to reduce the number of memory inits for vec. -/// -/// It collects all [`PathMetadata`] of the child paths. -/// Filters paths with the rules found in child directories and the given `ignore_rules`. -pub fn walk_serial( - output_snd: &XvcOutputSender, - ignore_rules: IgnoreRules, - dir: &Path, - walk_options: &WalkOptions, -) -> Result<(Vec, IgnoreRules)> { - let ignore_filename = walk_options.ignore_filename.clone().map(OsString::from); - let ignore_rules = Arc::new(Mutex::new(ignore_rules.clone())); - let dir_stack = crossbeam::queue::SegQueue::new(); - let res_paths = Arc::new(Mutex::new(Vec::::new())); - - dir_stack.push(dir.to_path_buf()); - - let get_child_paths = |dir: &Path| -> Result> { - Ok(directory_list(dir)? - .into_iter() - .filter_map(|pm_res| match pm_res { - Ok(pm) => Some(pm), - Err(e) => { - warn!(output_snd, "{}", e); - None - } - }) - .collect()) - }; - - let update_ignore_rules = |child_paths: &Vec| -> Result<()> { - if let Some(ref ignore_filename) = &ignore_filename { - watch!(ignore_filename); - if let Some(ignore_path_metadata) = child_paths - .iter() - .find(|pm| pm.path.file_name() == Some(ignore_filename)) - { - let ignore_path = dir.join(&ignore_path_metadata.path); - let new_patterns: Vec = - patterns_from_file(&ignore_rules.lock()?.root, &ignore_path)? - .into_iter() - .filter_map(|res_p| match res_p.pattern { - Ok(_) => Some(res_p.map(|p| p.unwrap())), - Err(e) => { - warn!(output_snd, "{}", e); - None - } - }) - .collect(); - - ignore_rules.lock()?.update(new_patterns)?; - } - } - Ok(()) - }; - - let filter_child_paths = |child_paths: &Vec| -> Result<()> { - for child_path in child_paths { - watch!(child_path.path); - let ignore_res = check_ignore(&(*ignore_rules.lock()?), child_path.path.as_ref()); - watch!(ignore_res); - match ignore_res { - MatchResult::NoMatch | MatchResult::Whitelist => { - watch!(child_path); - if child_path.metadata.is_dir() { - watch!("here"); - if walk_options.include_dirs { - watch!("here2"); - res_paths.lock()?.push(child_path.clone()); - } - watch!("here3"); - dir_stack.push(child_path.path.clone()); - watch!("here4"); - } else { - watch!("here5"); - res_paths.lock()?.push(child_path.clone()); - watch!("here6"); - } - } - // We can return anyhow! error here to notice the user that the path is ignored - MatchResult::Ignore => { - debug!(output_snd, "Ignored: {:?}", child_path.path); - } - } - watch!(child_path); - } - Ok(()) - }; - - while let Some(dir) = { dir_stack.pop().clone() } { - watch!(dir); - let dir = dir.clone(); - watch!(dir); - let child_paths = get_child_paths(&dir)?; - watch!(child_paths); - update_ignore_rules(&child_paths)?; - filter_child_paths(&child_paths)?; - } - - let res_paths: Vec = res_paths.lock()?.clone(); - let ignore_rules = ignore_rules.lock()?.clone(); - - Ok((res_paths, ignore_rules)) -} - /// FIXME: This should be Arc>> when Glob.is_match doesn't require mutable self -type IgnoreGlob = Arc>; +type IgnorePatterns = Vec; -/// Builds a [fast_glob::Glob] used to ignore files in they match -/// -/// This doesn't keep track of where the ignore rules are gathered. It's used for basic operations. -pub fn build_ignore_glob( +/// Build the ignore rules with the given directory +pub fn build_ignore_patterns( given: &str, ignore_root: &Path, - ignore_filename: &str, -) -> Result { - - let mut glob = IgnoreGlob::new(Mutex::new(Glob::new(given))); - - let ignore_file_walker = WalkDirGeneric::<((usize),(bool))>::new(ignore_root) - .process_read_dir(|depth, path, read_dir_state, children| { - // Keep only ignore files - children.retain(|dir_entry_result| { - dir_entry_result.as_ref().map(|dir_entry| { - match dir_entry.file_type { - FileType::Dir => { - // Don't read ignored directories - let ignored_dir = glob.borrow_mut().deref_mut().is_match( - &format!( - "/{}/", - dir_entry.path().strip_prefix(&ignore_root).unwrap().to_string_lossy() - ) - ); - if ignored_dir { - dir_entry.read_children_path = None; - return false; - } - } - FileType::File => { - // Read only ignore files - dir_entry.file_name() - .to_str() - .map(|s| s == ignore_filename) - .unwrap_or(false) - } - } - }).unwrap_or(false) - }); - }); - for entry in walk_dir { - if let Ok(entry) = entry { - assert!(entry.file_type.is_dir() || entry.file_name.to_str() == Some(ignore_filename)); - - let content = fs::read_to_string(entry.path())?; - let entry_root = entry.path().strip_prefix(&ignore_root).unwrap(); - - let ignore_rules = content.lines().filter_map(|line| { - if line.trim().is_empty() || line.trim().starts_with('#') { - None - } else { - Some(format!("{}/{}", entry_root, line)) - } - }).for_each(|line| { - // FIXME: A small optimization is to add all patterns at once with a single - // unlock - glob.borrow_mut().deref_mut().add(&line); - }); - } else { - debug!("Error reading ignore file: {:?}", entry.unwrap_err()); - } - } - - Ok(glob) -} - - -/// Just build the ignore rules with the given directory -pub fn build_ignore_rules( - given: IgnoreRules, - dir: &Path, - ignore_filename: &str, + ignore_filename: &OsStr, ) -> Result { - let elements = dir - .read_dir() - .map_err(|e| anyhow!("Error reading directory: {:?}, {:?}", dir, e))?; - - let mut child_dirs = Vec::::new(); - let ignore_fn = OsString::from(ignore_filename); - watch!(ignore_fn); - let ignore_root = given.root.clone(); + watch!(ignore_filename); watch!(ignore_root); - let mut ignore_rules = given; + let elements = ignore_root + .read_dir() + .map_err(|e| anyhow!("Error reading directory: {:?}, {:?}", ignore_root, e))?; - let mut new_patterns: Option> = None; + let mut dir_stack = Vec::::new(); + let ignore_fn = OsString::from(ignore_filename); - for entry in elements { - match entry { - Ok(entry) => { - if entry.path().is_dir() { - watch!(entry.path()); - child_dirs.push(entry.path()); + let ignore_rules = IgnoreRules::from_global_patterns(ignore_root, given); + + while let Some(dir) = dir_stack.pop() { + let mut new_dirs = Vec::::new(); + let mut new_patterns: Vec = Vec::new(); + let elements = dir.read_dir()?; + for entry in elements { + match entry { + Ok(entry) => { + if entry.path().is_dir() { + watch!(entry.path()); + new_dirs.push(entry.path()); + } + if entry.file_name() == ignore_fn && entry.path().exists() { + let ignore_path = entry.path(); + watch!(ignore_path); + let ignore_content = fs::read_to_string(&ignore_path)?; + new_patterns.extend( + content_to_patterns(&ignore_root, Some(&ignore_path), &ignore_content), + ); + } } - if entry.file_name() == ignore_fn && entry.path().exists() { - let ignore_path = entry.path(); - watch!(ignore_path); - new_patterns = Some( - patterns_from_file(&ignore_root, &ignore_path)? - .into_iter() - .filter_map(|p| match p.transpose() { - Ok(p) => Some(p), - Err(e) => { - warn!("{:?}", e); - None - } - }) - .collect(), - ); + Err(e) => { + warn!("{}", e); } } - Err(e) => { - warn!("{}", e); - } } - } - if let Some(new_patterns) = new_patterns { - ignore_rules.update(new_patterns)?; - } + ignore_rules.add_patterns(new_patterns)?; - for child_dir in child_dirs { - match check_ignore(&ignore_rules, &child_dir) { + new_dirs.drain(0..new_dirs.len()).for_each(|new_dir| { + match ignore_rules.check(&new_dir) { MatchResult::NoMatch | MatchResult::Whitelist => { - ignore_rules = build_ignore_rules(ignore_rules, &child_dir, ignore_filename)?; + dir_stack.push(new_dir); } MatchResult::Ignore => {} - } - } - - Ok(ignore_rules) -} - -fn transform_pattern_for_glob(pattern: Pattern) -> Pattern { - let anything_anywhere = |p| format!("**/{p}"); - let anything_relative = |p, directory| format!("{directory}/**/{p}"); - let directory_anywhere = |p| format!("**{p}/**"); - let directory_relative = |p, directory| format!("{directory}/**/{p}/**"); - - let transformed_pattern = match (&pattern.path_kind, &pattern.relativity) { - (PathKind::Any, PatternRelativity::Anywhere) => anything_anywhere(pattern.pattern), - (PathKind::Any, PatternRelativity::RelativeTo { directory }) => { - anything_relative(pattern.pattern, directory) - } - (PathKind::Directory, PatternRelativity::Anywhere) => directory_anywhere(pattern.pattern), - (PathKind::Directory, PatternRelativity::RelativeTo { directory }) => { - directory_relative(pattern.pattern, directory) - } - }; - - Pattern { - pattern: transformed_pattern, - ..pattern + } + }); } -} -fn build_globset(patterns: &Vec) -> Result { - let mut glob = Glob::new("").expect("Error building glob set"); - for p in patterns { - if !glob.add(&p) { - return Err(anyhow!("Error adding pattern {} to glob set", p).into()); - } - } - Ok(glob) -} - -fn patterns_from_file( - ignore_root: &Path, - ignore_path: &Path, -) -> Result> { - watch!(ignore_root); - watch!(ignore_path); - let content = fs::read_to_string(ignore_path).with_context(|| { - format!( - "Cannot read file: {:?}\n - If the file is present, it may be an encoding issue. Please check if it's UTF-8 encoded.", - ignore_path - ) - })?; - watch!(&content); - Ok(content_to_patterns( - ignore_root, - Some(ignore_path), - &content, - )) + Ok(ignore_rules) } /// convert a set of rules in `content` to glob patterns. @@ -663,8 +183,8 @@ pub fn content_to_patterns( ignore_root: &Path, source: Option<&Path>, content: &str, -) -> Vec { - let patterns: Vec = content +) -> Vec { + let patterns: Vec = content .lines() .enumerate() // A line starting with # serves as a comment. Put a backslash ("\") in front of the first hash for patterns that begin with a hash. @@ -693,146 +213,31 @@ pub fn content_to_patterns( }, ) }) - .map(|(line, source)| build_pattern(source, line)) - .map(transform_pattern_for_glob) + .map(|(line, source)| Pattern::new(source, line)) .collect(); patterns } -fn build_pattern(source: Source, original: &str) -> Pattern { - let current_dir = match &source { - Source::Global => "".to_string(), - Source::File { path, .. } => { - let path = path - .parent() - .expect("Pattern source file doesn't have parent") - .to_string_lossy() - .to_string(); - if path.starts_with('/') { - path - } else { - format!("/{path}") - } - } - }; - // if Pattern starts with ! it's whitelist, if ends with / it's dir only, if it contains - // non final slash, it should be considered under the current dir only, otherwise it - // matches +pub fn update_ignore_rules(ignore_filename: &Option, + dir: &Path, + ignore_rules: &IgnoreRules) -> Result<()> { - let begin_exclamation = original.starts_with('!'); - let mut line = if begin_exclamation || original.starts_with(r"\!") { - original[1..].to_owned() - } else { - original.to_owned() - }; - - // TODO: We should handle filenames with trailing spaces better, with regex match and removing - // the \\ from the name - if !line.ends_with("\\ ") { - line = line.trim_end().to_string(); - } + if let Some(ignore_filename) = ignore_filename { + let ignore_root = &ignore_rules.root; + let ignore_path = dir.join(ignore_filename); + if ignore_path.is_file() { + let new_patterns: Vec = { + let content = fs::read_to_string(&ignore_path)?; + content_to_patterns(ignore_root, Some(ignore_path).as_deref(), &content) + }; - let end_slash = line.ends_with('/'); - if end_slash { - line = line[..line.len() - 1].to_string() - } - - let begin_slash = line.starts_with('/'); - let non_final_slash = if !line.is_empty() { - line[..line.len() - 1].chars().any(|c| c == '/') - } else { - false - }; - - if begin_slash { - line = line[1..].to_string(); - } - - let current_dir = if current_dir.ends_with('/') { - ¤t_dir[..current_dir.len() - 1] - } else { - ¤t_dir - }; - - let effect = if begin_exclamation { - PatternEffect::Whitelist - } else { - PatternEffect::Ignore - }; - - let path_kind = if end_slash { - PathKind::Directory - } else { - PathKind::Any - }; - - let relativity = if non_final_slash { - PatternRelativity::RelativeTo { - directory: current_dir.to_owned(), + ignore_rules.add_patterns(new_patterns)?; } - } else { - PatternRelativity::Anywhere - }; - - Pattern:: { - pattern: line, - original: original.to_owned(), - source, - effect, - relativity, - path_kind, } + Ok(()) } - -/// Check whether `path` is whitelisted or ignored with `ignore_rules` -pub fn check_ignore(ignore_rules: &IgnoreRules, path: &Path) -> MatchResult { - let is_abs = path.is_absolute(); - watch!(is_abs); - // strip_prefix eats the final slash, and ends_with behave differently than str, so we work - // around here - let path_str = path.to_string_lossy(); - watch!(path_str); - let final_slash = path_str.ends_with('/'); - watch!(final_slash); - - let path = if is_abs { - - if final_slash { - format!( - "/{}/", - path.strip_prefix(&ignore_rules.root) - .expect("path must be within root") - .to_string_lossy() - ) - } else { - format!( - "/{}", - path.strip_prefix(&ignore_rules.root) - .expect("path must be within root") - .to_string_lossy() - ) - } - } else { - path_str.to_string() - }; - - // FIXME: Current fast_glob implementation requires &mut self in is_match, possibly to avoid to - // clone internal state. This requires to lock the ignore_rules for each call. It's possible to - // keep a clone of the ignore_rules for each thread with a design change -- or fork the glob - // lib. - watch!(path); - let result = if ignore_rules.whitelist_set.write().unwrap().is_match(&path) { - MatchResult::Whitelist - } else if ignore_rules.ignore_set.write().unwrap().is_match(&path) { - MatchResult::Ignore - } else { - MatchResult::NoMatch - }; - result -} - /// Return all childs of a directory regardless of any ignore rules /// If there is an error to obtain the metadata, error is added to the element instead pub fn directory_list(dir: &Path) -> Result>> { @@ -889,7 +294,7 @@ mod tests { #[test_case("myfile*" => matches PatternEffect::Ignore ; "t3367706249")] #[test_case("myfile/" => matches PatternEffect::Ignore ; "t1204466627")] fn test_pattern_effect(line: &str) -> PatternEffect { - let pat = build_pattern(Source::Global, line); + let pat = Pattern::new(Source::Global, line); pat.effect } @@ -913,7 +318,7 @@ mod tests { path: PathBuf::from(dir).join(".gitignore"), line: 1, }; - let pat = build_pattern(source, line); + let pat = Pattern::new(source, line); pat.relativity } @@ -932,7 +337,7 @@ mod tests { path: PathBuf::from(dir).join(".gitignore"), line: 1, }; - let pat = build_pattern(source, line); + let pat = Pattern::new(source, line); pat.path_kind } @@ -949,7 +354,7 @@ mod tests { patterns.len() } - fn create_patterns(root: &str, dir: Option<&str>, patterns: &str) -> Vec { + fn create_patterns(root: &str, dir: Option<&str>, patterns: &str) -> Vec { content_to_patterns(Path::new(root), dir.map(Path::new), patterns) } @@ -959,9 +364,9 @@ mod tests { initial_patterns: &str, ) -> Result { let patterns = create_patterns(root, dir, initial_patterns); - let mut initialized = IgnoreRules::empty(&PathBuf::from(root)); + let initialized = IgnoreRules::empty(&PathBuf::from(root)); - initialized.update(patterns)?; + initialized.add_patterns(patterns)?; Ok(initialized) } @@ -989,7 +394,7 @@ mod tests { path: PathBuf::from(format!("{dir}/.gitignore")), line: 1, }; - let pattern = build_pattern(source, pattern); + let pattern = Pattern::new(source, pattern); pattern.relativity } @@ -1010,7 +415,7 @@ mod tests { path: PathBuf::from(format!("{dir}.gitignore")), line: 1, }; - let pattern = build_pattern(source, pattern); + let pattern = Pattern::new(source, pattern); pattern.pattern } @@ -1093,7 +498,7 @@ mod tests { let dwi = new_dir_with_ignores(root.to_str().unwrap(), Some(&source_file), contents).unwrap(); - check_ignore(&dwi, &path) + dwi.check(&path) } // TODO: Patterns shouldn't have / prefix, but an appropriate PathKind diff --git a/walker/src/notify.rs b/walker/src/notify.rs index fed0ca152..92098895c 100644 --- a/walker/src/notify.rs +++ b/walker/src/notify.rs @@ -5,7 +5,6 @@ //! It defines [PathEvent] as a simple version of [notify::EventKind]. //! It defines [PathEventHandler] that handles events from [notify::EventHandler]. use crate::{ - check_ignore, error::{Error, Result}, IgnoreRules, MatchResult, }; @@ -89,7 +88,7 @@ impl EventHandler for PathEventHandler { impl PathEventHandler { fn write_event(&mut self, path: PathBuf) { - match check_ignore(&self.ignore_rules, &path) { + match self.ignore_rules.check(&path) { MatchResult::Whitelist | MatchResult::NoMatch => { if let Ok(metadata) = path.metadata() { self.sender @@ -111,7 +110,7 @@ impl PathEventHandler { } fn create_event(&mut self, path: PathBuf) { - match check_ignore(&self.ignore_rules, &path) { + match self.ignore_rules.check(&path) { MatchResult::Whitelist | MatchResult::NoMatch => { if let Ok(metadata) = path.metadata() { self.sender @@ -133,7 +132,7 @@ impl PathEventHandler { } fn remove_event(&mut self, path: PathBuf) { - match check_ignore(&self.ignore_rules, &path) { + match self.ignore_rules.check(&path) { MatchResult::Whitelist | MatchResult::NoMatch => { self.sender .send(Some(PathEvent::Delete { path })) diff --git a/walker/src/pattern.rs b/walker/src/pattern.rs new file mode 100644 index 000000000..0b9b8fab5 --- /dev/null +++ b/walker/src/pattern.rs @@ -0,0 +1,231 @@ +//! Pattern describes a single line in an ignore file and its semantics +//! It is used to match a path with the given pattern +pub use error::{Error, Result}; +pub use ignore_rules::IgnoreRules; +use crate::sync; +pub use std::hash::Hash; +pub use sync::{PathSync, PathSyncSingleton}; + +pub use crate::notify::{make_watcher, PathEvent, RecommendedWatcher}; + + +// use glob::{MatchOptions, Pattern, PatternError}; +pub use fast_glob::Glob; +use std::{ + fmt::Debug, + path::PathBuf, +}; + +use crate::error; +use crate::ignore_rules; + +/// Show whether a path matches to a glob rule +#[derive(Debug, Clone)] +pub enum MatchResult { + /// There is no match between glob(s) and path + NoMatch, + /// Path matches to ignored glob(s) + Ignore, + /// Path matches to whitelisted glob(s) + Whitelist, +} + +/// Is the pattern matches anywhere or only relative to a directory? +#[derive(Debug, Clone, Hash, PartialEq, Eq)] +pub enum PatternRelativity { + /// Match the path regardless of the directory prefix + Anywhere, + /// Match the path if it only starts with `directory` + RelativeTo { + /// The directory that the pattern must have as prefix to be considered a match + directory: String, + }, +} + +/// Is the path only a directory, or could it be directory or file? +#[derive(Debug, Clone, Hash, PartialEq, Eq)] +pub enum PathKind { + /// Path matches to directory or file + Any, + /// Path matches only to directory + Directory, +} + +/// Is this pattern a ignore or whitelist pattern? +#[derive(Debug, Clone, Eq, PartialEq, Hash)] +pub enum PatternEffect { + /// This is an ignore pattern + Ignore, + /// This is a whitelist pattern + Whitelist, +} + +/// Do we get this pattern from a file (.gitignore, .xvcignore, ...) or specify it directly in +/// code? +#[derive(Debug, Clone, Hash, PartialEq, Eq)] +pub enum Source { + /// Pattern is globally defined in code + Global, + + /// Pattern is obtained from file + File { + /// Path of the pattern file + path: PathBuf, + /// (1-based) line number the pattern retrieved + line: usize, + }, + + /// Pattern is from CLI + CommandLine { + current_dir: PathBuf + } +} + +/// Pattern is generic and could be an instance of String, Glob, Regex or any other object. +/// The type is evolved by compiling. +/// A pattern can start its life as `Pattern` and can be compiled into `Pattern` or +/// `Pattern`. +#[derive(Debug)] +pub struct Pattern +{ + /// The pattern type + pub glob: Glob, + /// The line that we used for glob + pub pattern: String, + /// The original string that defines the pattern + pub original: String, + /// Where did we get this pattern? + pub source: Source, + /// Is this ignore or whitelist pattern? + pub effect: PatternEffect, + /// Does it have an implied prefix? + pub relativity: PatternRelativity, + /// Is the path a directory or anything? + pub path_kind: PathKind, +} + +impl Pattern { + +pub fn new(source: Source, original: &str) -> Self { + + let original = original.to_owned(); + let current_dir = match &source { + Source::Global => "".to_string(), + Source::File { path, .. } => { + let path = path + .parent() + .expect("Pattern source file doesn't have parent") + .to_string_lossy() + .to_string(); + if path.starts_with('/') { + path + } else { + format!("/{path}") + } + } + Source::CommandLine { current_dir } => current_dir.to_string_lossy().to_string(), + }; + + // if Pattern starts with ! it's whitelist, if ends with / it's dir only, if it contains + // non final slash, it should be considered under the current dir only, otherwise it + // matches + + let begin_exclamation = original.starts_with('!'); + let mut line = if begin_exclamation || original.starts_with(r"\!") { + original[1..].to_owned() + } else { + original.to_owned() + }; + + // TODO: We should handle filenames with trailing spaces better, with regex match and removing + // the \\ from the name + if !line.ends_with("\\ ") { + line = line.trim_end().to_string(); + } + + let end_slash = line.ends_with('/'); + if end_slash { + line = line[..line.len() - 1].to_string() + } + + let begin_slash = line.starts_with('/'); + let non_final_slash = if !line.is_empty() { + line[..line.len() - 1].chars().any(|c| c == '/') + } else { + false + }; + + if begin_slash { + line = line[1..].to_string(); + } + + let current_dir = if current_dir.ends_with('/') { + ¤t_dir[..current_dir.len() - 1] + } else { + ¤t_dir + }; + + let effect = if begin_exclamation { + PatternEffect::Whitelist + } else { + PatternEffect::Ignore + }; + + let path_kind = if end_slash { + PathKind::Directory + } else { + PathKind::Any + }; + + let relativity = if non_final_slash { + PatternRelativity::RelativeTo { + directory: current_dir.to_owned(), + } + } else { + PatternRelativity::Anywhere + }; + + let pattern = transform_pattern_for_glob(&line, relativity.clone(), path_kind.clone()); + let glob = Glob::new(&pattern).unwrap(); + + Pattern { + glob, + pattern, + original, + source, + effect, + relativity, + path_kind, + } +} + + +} + + +fn transform_pattern_for_glob(original: &str, relativity: PatternRelativity, path_kind: PathKind) -> String { + let anything_anywhere = |p| format!("**/{p}"); + let anything_relative = |p, directory| format!("{directory}/**/{p}"); + let directory_anywhere = |p| format!("**{p}/**"); + let directory_relative = |p, directory| format!("{directory}/**/{p}/**"); + + let transformed_pattern = match (path_kind, relativity) { + (PathKind::Any, PatternRelativity::Anywhere) => anything_anywhere(original), + (PathKind::Any, PatternRelativity::RelativeTo { directory }) => { + anything_relative(original, directory) + } + (PathKind::Directory, PatternRelativity::Anywhere) => directory_anywhere(original), + (PathKind::Directory, PatternRelativity::RelativeTo { directory }) => { + directory_relative(original, directory) + } + }; + + transformed_pattern +} + +pub fn build_pattern_list(patterns: Vec, source: Source) -> Vec { + patterns + .iter() + .map(|p| Pattern::new(source.clone(), p)) + .collect() +} diff --git a/walker/src/walk_parallel.rs b/walker/src/walk_parallel.rs new file mode 100644 index 000000000..623ef890a --- /dev/null +++ b/walker/src/walk_parallel.rs @@ -0,0 +1,109 @@ +use std::{path::Path, sync::{Arc, Mutex}}; + +use crossbeam::queue::SegQueue; +use crossbeam_channel::Sender; +use xvc_logging::{uwr, watch}; + +use crate::{directory_list, update_ignore_rules, IgnoreRules, MatchResult, PathMetadata, Result, WalkOptions, MAX_THREADS_PARALLEL_WALK}; + + +fn walk_parallel_inner( + ignore_rules: Arc>, + dir: &Path, + walk_options: WalkOptions, + path_sender: Sender>, +) -> Result> { + let ignore_filename = walk_options.ignore_filename; + update_ignore_rules(&ignore_filename, dir, &ignore_rules.lock().unwrap())?; + + Ok(directory_list(dir)? + .drain(..) + .filter_map(|pm_res| match pm_res { + Ok(pm) => Some(pm), + Err(e) => { + path_sender + .send(Err(e)) + .expect("Channel error in walk_parallel"); + None + } + }).filter_map(|pm| { + let ignore_res = ignore_rules.lock().unwrap().check(pm.path.as_ref()); + match ignore_res { + MatchResult::NoMatch | MatchResult::Whitelist => Some(pm), + MatchResult::Ignore => { + watch!(pm.path); + if pm.metadata.is_file() || (pm.metadata.is_dir() && walk_options.include_dirs) { + path_sender.send(Ok(pm)).expect("Channel error in walk_parallel"); + } + None + } + }}).filter(|pm| { pm.metadata.is_dir() }).collect::>()) + +} + +/// Walk all child paths under `dir` and send non-ignored paths to `path_sender`. +/// Newly found ignore rules are sent through `ignore_sender`. +/// The ignore file name (`.xvcignore`, `.gitignore`, `.ignore`, ...) is set by `walk_options`. +/// +/// It lists elements of a directory, then creates a new crossbeam scope for each child directory and +/// calls itself recursively. It may not be feasible for small directories to create threads. +pub fn walk_parallel( + ignore_rules: IgnoreRules, + dir: &Path, + walk_options: WalkOptions, + path_sender: Sender>, + ignore_sender: Sender>>>, +) -> Result<()> { + let dir_queue = Arc::new(SegQueue::::new()); + + let ignore_rules = Arc::new(Mutex::new(ignore_rules.clone())); + + let child_dirs = walk_parallel_inner( + ignore_rules.clone(), + dir, + walk_options.clone(), + path_sender.clone(), + )?; + + child_dirs.into_iter().for_each(|pm| { + dir_queue.push(pm); + }); + + if dir_queue.is_empty() { + return Ok(()); + } + + crossbeam::scope(|s| { + for thread_i in 0..MAX_THREADS_PARALLEL_WALK { + let path_sender = path_sender.clone(); + let ignore_sender = ignore_sender.clone(); + let walk_options = walk_options.clone(); + let ignore_rules = ignore_rules.clone(); + let dir_queue = dir_queue.clone(); + + s.spawn(move |_| { + watch!(path_sender); + watch!(ignore_sender); + while let Some(pm) = dir_queue.pop() { + let child_dirs = walk_parallel_inner( + ignore_rules.clone(), + &pm.path, + walk_options.clone(), + path_sender.clone(), + ).unwrap(); + + for child_dir in child_dirs { + dir_queue.push(child_dir); + } + } + watch!("End of thread {}", thread_i); + }); + } + }) + .expect("Error in crossbeam scope in walk_parallel"); + + watch!("End of walk_parallel"); + + Ok(()) +} + diff --git a/walker/src/walk_serial.rs b/walker/src/walk_serial.rs new file mode 100644 index 000000000..1bb817461 --- /dev/null +++ b/walker/src/walk_serial.rs @@ -0,0 +1,89 @@ +use std::{ffi::OsString, fs, path::Path, sync::{Arc, Mutex}}; + +use xvc_logging::{debug, warn, watch, XvcOutputSender}; + +use crate::{content_to_patterns, directory_list, pattern::MatchResult, update_ignore_rules, IgnoreRules, PathMetadata, Pattern, Result, WalkOptions}; + +/// Walk `dir` with `walk_options`, with the given _initial_ `ignore_rules`. +/// Note that ignore rules are expanded with the rules given in the `ignore_filename` in +/// `walk_options`. +/// The result is added to given `res_paths` to reduce the number of memory inits for vec. +/// +/// It collects all [`PathMetadata`] of the child paths. +/// Filters paths with the rules found in child directories and the given `ignore_rules`. +pub fn walk_serial( + output_snd: &XvcOutputSender, + global_ignore_rules: &str, + dir: &Path, + walk_options: &WalkOptions, +) -> Result<(Vec, IgnoreRules)> { + let ignore_filename = walk_options.ignore_filename.clone().map(OsString::from); + let ignore_rules = IgnoreRules::from_global_patterns(dir, global_ignore_rules); + let ignore_rules = Arc::new(Mutex::new(ignore_rules)); + let dir_stack = crossbeam::queue::SegQueue::new(); + let res_paths = Arc::new(Mutex::new(Vec::::new())); + let ignore_root = dir.to_path_buf(); + + dir_stack.push(dir.to_path_buf()); + + let get_child_paths = |dir: &Path| -> Result> { + Ok(directory_list(dir)? + .into_iter() + .filter_map(|pm_res| match pm_res { + Ok(pm) => Some(pm), + Err(e) => { + warn!(output_snd, "{}", e); + None + } + }) + .collect()) + }; + + let filter_child_paths = |child_paths: &Vec| -> Result<()> { + for child_path in child_paths { + watch!(child_path.path); + let ignore_res = ignore_rules.lock()?.check(child_path.path.as_ref()); + match ignore_res { + MatchResult::NoMatch | MatchResult::Whitelist => { + if child_path.metadata.is_dir() { + if walk_options.include_dirs { + res_paths.lock()?.push(child_path.clone()); + } + dir_stack.push(child_path.path.clone()); + } else { + res_paths.lock()?.push(child_path.clone()); + } + } + // We can return anyhow! error here to notice the user that the path is ignored + MatchResult::Ignore => { + debug!(output_snd, "Ignored: {:?}", child_path.path); + } + } + watch!(child_path); + } + Ok(()) + }; + + while let Some(dir) = dir_stack.pop() { + watch!(dir); + update_ignore_rules(&ignore_filename, &dir, &ignore_rules.lock().unwrap())?; + let mut res_paths = res_paths.lock()?; + get_child_paths(&dir)?.drain(..).filter_map(|p| + match ignore_rules.lock().unwrap().check(p.path.as_ref()) { + MatchResult::NoMatch | MatchResult::Whitelist => Some(p), + MatchResult::Ignore => { + debug!(output_snd, "Ignored: {:?}", p.path); + None + } + }).for_each(|p| res_paths.push(p)); + } + + let res_paths: Vec = res_paths.lock()?.clone(); + let ignore_rules = ignore_rules.lock()?.clone(); + + Ok((res_paths, ignore_rules)) +} + + + + From 1fb26ce3d555113f72c8eeb4074aae24509577c0 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Wed, 14 Aug 2024 21:08:53 +0300 Subject: [PATCH 165/257] Update SharedIgnoreRules to use RwLock --- book/src/ref/xvc-check-ignore.md | 13 ++- core/src/check_ignore/mod.rs | 4 +- .../types/xvcdigest/xvc_metadata_digest.rs | 6 +- core/src/util/file.rs | 24 +++-- core/src/util/git.rs | 5 +- core/src/util/pmp.rs | 2 +- core/src/util/xvcignore.rs | 90 ++++++------------- lib/tests/test_core_util_file_walker.rs | 6 +- lib/tests/test_core_util_notify.rs | 2 +- lib/tests/test_walker_parallel.rs | 10 +-- lib/tests/test_walker_serial.rs | 6 +- walker/Cargo.toml | 1 + walker/src/ignore_rules.rs | 70 +++++++-------- walker/src/lib.rs | 20 +++-- walker/src/pattern.rs | 8 +- walker/src/walk_parallel.rs | 49 ++++++---- walker/src/walk_serial.rs | 35 ++------ 17 files changed, 149 insertions(+), 202 deletions(-) diff --git a/book/src/ref/xvc-check-ignore.md b/book/src/ref/xvc-check-ignore.md index 039a7d5dc..7f0366007 100644 --- a/book/src/ref/xvc-check-ignore.md +++ b/book/src/ref/xvc-check-ignore.md @@ -4,7 +4,7 @@ Check whether a path is ignored or whitelisted by Xvc. -## Synopsis +## Synopsis ```console $ xvc check-ignore --help @@ -22,9 +22,9 @@ Options: --ignore-filename Filename that contains ignore rules - + This can be set to .gitignore to test whether Git and Xvc work the same way. - + [default: .xvcignore] -n, --non-matching @@ -44,7 +44,7 @@ $ xvc check-ignore my-dir/my-file ``` -If you supply paths from the CLI, they are checked instead. +If you supply paths from the CLI, they are checked instead. ```shell $ xvc check-ignore my-dir/my-file another-dir/another-file @@ -62,8 +62,5 @@ You can specify an alternative ignore filename with `--ignore-filename` option. The below command is identical to `git check-ignore` and should give the same results. ```shell -$ xvc check-ignore --ignore-filename .gitignore +$ xvc check-ignore --ignore-filename .gitignore ``` - - - diff --git a/core/src/check_ignore/mod.rs b/core/src/check_ignore/mod.rs index 5d2d538c7..d4b9215c1 100644 --- a/core/src/check_ignore/mod.rs +++ b/core/src/check_ignore/mod.rs @@ -77,13 +77,13 @@ pub fn cmd_check_ignore( let current_dir = conf.current_dir()?; let walk_options = WalkOptions { - ignore_filename: Some(opts.ignore_filename.clone().into()), + ignore_filename: Some(opts.ignore_filename.clone()), include_dirs: true, }; let ignore_rules = build_ignore_patterns( COMMON_IGNORE_PATTERNS, - &xvc_root, + xvc_root, &walk_options.ignore_filename.unwrap_or_default(), )?; if !opts.targets.is_empty() { diff --git a/core/src/types/xvcdigest/xvc_metadata_digest.rs b/core/src/types/xvcdigest/xvc_metadata_digest.rs index 47148b04b..f9bd6959e 100644 --- a/core/src/types/xvcdigest/xvc_metadata_digest.rs +++ b/core/src/types/xvcdigest/xvc_metadata_digest.rs @@ -39,11 +39,7 @@ impl XvcMetadataDigest { 0u64 }; - let size = if let Some(size) = xvc_metadata.size { - size - } else { - 0u64 - }; + let size = xvc_metadata.size.unwrap_or_default(); let mut bytes: [u8; 32] = [0; 32]; bytes[..8].clone_from_slice(&ft.to_le_bytes()); diff --git a/core/src/util/file.rs b/core/src/util/file.rs index 36d833dfa..47f00a296 100644 --- a/core/src/util/file.rs +++ b/core/src/util/file.rs @@ -11,20 +11,27 @@ use std::os::unix::fs as unix_fs; use std::os::windows::fs as windows_fs; use std::path::{Path, PathBuf}; +use std::sync::{Arc, RwLock}; use xvc_logging::watch; -use xvc_walker::{IgnoreRules, PathMetadata, WalkOptions}; +use xvc_walker::{IgnoreRules, PathMetadata, SharedIgnoreRules, WalkOptions}; use crate::error::Error; use crate::error::Result; use crate::CHANNEL_BOUND; + use crossbeam_channel::{bounded, Receiver, Sender}; + + + use crate::types::{xvcpath::XvcPath, xvcroot::XvcRoot}; use super::pmp::XvcPathMetadataProvider; -use super::xvcignore::walk_parallel; +use super::xvcignore::{walk_parallel, COMMON_IGNORE_PATTERNS}; use super::XvcPathMetadataMap; +///w +/// /// A parallel directory walker. /// It starts from `start_dir` and sends [PathMetadata] by traversing all child directories. /// It uses [xvc_walker::walk_parallel] after building an empty [IgnoreRules]. @@ -33,19 +40,18 @@ use super::XvcPathMetadataMap; /// It doesn't check any ignore files. /// It even returns `.git` and `.xvc` directory contents. pub fn path_metadata_channel(sender: Sender>, start_dir: &Path) -> Result<()> { - let initial_rules = IgnoreRules::empty(start_dir); let walk_options = WalkOptions { ignore_filename: None, include_dirs: true, }; + let ignore_rules = Arc::new(RwLock::new(IgnoreRules::empty(start_dir, None))); let (w_sender, w_receiver) = bounded(CHANNEL_BOUND); - let (ignore_sender, _ignore_receiver) = bounded(CHANNEL_BOUND); - xvc_walker::walk_parallel( - initial_rules, + + xvc_walker::walk_parallel::walk_parallel( + ignore_rules, start_dir, walk_options, w_sender, - ignore_sender, )?; for pm in w_receiver { sender.send(Ok(pm?))?; @@ -73,8 +79,8 @@ pub fn pipe_filter_path_errors( /// NOTE: /// This function only returns a snapshot of the repository. /// If you want to handle events after this initial snapshot, see [xvc_walker::notify::make_watcher]. -pub fn all_paths_and_metadata(xvc_root: &XvcRoot) -> (XvcPathMetadataMap, IgnoreRules) { - walk_parallel(xvc_root, true).unwrap() +pub fn all_paths_and_metadata(xvc_root: &XvcRoot) -> (XvcPathMetadataMap, SharedIgnoreRules) { + walk_parallel(xvc_root, COMMON_IGNORE_PATTERNS, true).unwrap() } /// Returns a compiled [glob::Pattern] by prepending it with `pipeline_rundir`. diff --git a/core/src/util/git.rs b/core/src/util/git.rs index 88d4fa192..c03b85290 100644 --- a/core/src/util/git.rs +++ b/core/src/util/git.rs @@ -37,13 +37,10 @@ pub fn build_gitignore(git_root: &AbsolutePath) -> Result { #[cfg(test)] mod test { use super::*; - use std::ffi::OsStr; - use std::ffi::OsString; use std::fs; use test_case::test_case; use xvc_logging::watch; use xvc_test_helper::*; - use xvc_walker::IgnoreRules; use xvc_walker::MatchResult as M; #[test_case("myfile.txt" , ".gitignore", "/myfile.txt" => matches M::Ignore ; "myfile.txt")] @@ -68,7 +65,7 @@ mod test { let gitignore = build_ignore_patterns( "", gitignore_path.parent().unwrap(), - OsString::from(".gitignore").as_ref() + ".gitignore" ) .unwrap(); diff --git a/core/src/util/pmp.rs b/core/src/util/pmp.rs index e493f8c36..d1ce4c680 100644 --- a/core/src/util/pmp.rs +++ b/core/src/util/pmp.rs @@ -36,7 +36,7 @@ pub struct XvcPathMetadataProvider { impl XvcPathMetadataProvider { /// Create a new PathMetadataProvider pub fn new(output_sender: &XvcOutputSender, xvc_root: &XvcRoot) -> Result { - let ignore_rules = build_ignore_patterns(COMMON_IGNORE_PATTERNS, xvc_root, OsString::from(XVCIGNORE_FILENAME).as_ref())?; + let ignore_rules = build_ignore_patterns(COMMON_IGNORE_PATTERNS, xvc_root, XVCIGNORE_FILENAME)?; let path_map = Arc::new(RwLock::new(HashMap::new())); let (watcher, event_receiver) = make_watcher(ignore_rules.clone())?; diff --git a/core/src/util/xvcignore.rs b/core/src/util/xvcignore.rs index 2b1f60895..809024187 100644 --- a/core/src/util/xvcignore.rs +++ b/core/src/util/xvcignore.rs @@ -9,8 +9,8 @@ use crossbeam_channel::{bounded, Sender}; use std::ffi::OsString; use std::sync::{Arc, Mutex, RwLock}; use std::thread; -use xvc_logging::{warn, XvcOutputSender}; -use xvc_walker::Result as XvcWalkerResult; +use xvc_logging::{warn, watch, XvcOutputSender}; +use xvc_walker::{Result as XvcWalkerResult, SharedIgnoreRules}; use xvc_walker::{self, IgnoreRules, PathMetadata, WalkOptions}; /// We ignore `.git` directories even we are not using `.git` @@ -41,7 +41,7 @@ pub fn walk_serial( include_dirs: bool, ) -> Result<(XvcPathMetadataMap, IgnoreRules)> { let walk_options = WalkOptions { - ignore_filename: Some(OsString::from(XVCIGNORE_FILENAME)), + ignore_filename: Some(XVCIGNORE_FILENAME.to_owned()), include_dirs, }; let (res_paths, ignore_rules) = @@ -77,56 +77,43 @@ pub fn walk_serial( /// /// - `xvc_root`: The root structure for Xvc /// - `include_dirs`: Whether to include directories themselves. -/// If `false`, only the actual files in the repository are listed. +/// If `false`, only the actual files in the repository are listed. /// /// ## Returns /// /// - `XvcPathMetadataMap`: A hash map of files. Keys are [XvcPath], values are their -/// [XvcMetadata]. +/// [XvcMetadata]. /// - `IgnoreRules`: The rules that were produced while reading the directories. -/// This is returned here to prevent a second traversal for ignores. +/// This is returned here to prevent a second traversal for ignores. pub fn walk_parallel( xvc_root: &XvcRoot, + global_ignore_rules: &str, include_dirs: bool, -) -> Result<(XvcPathMetadataMap, IgnoreRules)> { +) -> Result<(XvcPathMetadataMap, SharedIgnoreRules)> { let (sender, receiver) = bounded::<(XvcPath, XvcMetadata)>(CHANNEL_BOUND); - let (ignore_sender, ignore_receiver) = bounded::>(CHANNEL_BOUND); + watch!(sender); + let ignore_rules = Arc::new(RwLock::new(IgnoreRules::from_global_patterns( + xvc_root, + Some(XVCIGNORE_FILENAME), + global_ignore_rules))); + + watch!(ignore_rules); walk_channel( xvc_root, - COMMON_IGNORE_PATTERNS, - Some(XVCIGNORE_FILENAME.to_string()), + ignore_rules.clone(), include_dirs, sender, - ignore_sender, )?; - let pusher = thread::spawn(move || { + let pmm = thread::spawn(move || { let mut pmm = XvcPathMetadataMap::new(); for (path, md) in receiver.iter() { + watch!(path); pmm.insert(path, md); } pmm - }); - - let ignore_rules = IgnoreRules::empty(xvc_root); - let ignore_rules_thread = thread::spawn(move || { - for ignore_rule in ignore_receiver { - if let Ok(ignore_rule) = ignore_rule { - assert!(ignore_rules.root == ignore_rule.root); - ignore_rules.merge_with(&ignore_rule).unwrap(); - } else { - warn!("Error while collecting ignore rules"); - } - } - ignore_rules - }); - - let pmm = pusher.join().map_err(|e| Error::FSWalkerError { - error: format!("{:?}", e), - })?; - - let ignore_rules = ignore_rules_thread.join()?; + }).join().map_err(Error::from)?; Ok((pmm, ignore_rules)) } @@ -144,7 +131,7 @@ pub fn walk_parallel( /// - `xvc_root`: The repository root /// - `initial_patterns`: A set of patterns arranged similar to an `.xvcignore` (`.gitignore`) content. /// - `ignore_filename`: The name of the ignore files to be loaded for ignore rules. -/// (ex: `.xvcignore`, `.ignore`, or `.gitignore`) +/// (ex: `.xvcignore`, `.ignore`, or `.gitignore`) /// - `include_dirs`: Whether to send directory records themselves. /// If `false`, only the files in directories are sent. /// - `xpm_upstream`: The channel this function sends the paths and metadata. @@ -161,28 +148,24 @@ pub fn walk_parallel( /// These overlapping rules can be merged with [merge_ignores]. pub fn walk_channel( xvc_root: &XvcRoot, - initial_patterns: &str, - ignore_filename: Option, + ignore_rules: SharedIgnoreRules, include_dirs: bool, xpm_upstream: Sender<(XvcPath, XvcMetadata)>, - ignore_upstream: Sender>, ) -> Result<()> { - let initial_rules = IgnoreRules::from_global_patterns(xvc_root, initial_patterns); + let walk_options = WalkOptions { - ignore_filename: ignore_filename.map(OsString::from), + ignore_filename: ignore_rules.read()?.ignore_filename.clone(), include_dirs, }; let (path_sender, path_receiver) = bounded::>(CHANNEL_BOUND); - let (ignore_sender, ignore_receiver) = - bounded::>>>(CHANNEL_BOUND); - + xvc_walker::walk_parallel::walk_parallel( - initial_rules, + ignore_rules, xvc_root, walk_options, path_sender, - ignore_sender, )?; + crossbeam::scope(|s| { s.spawn(|_| { for result in path_receiver { @@ -211,27 +194,6 @@ pub fn walk_channel( } }); - s.spawn(|_| { - for ignore_rule in ignore_receiver { - match ignore_rule { - Ok(ir) => { - ir.lock() - .map(|ir| { - ignore_upstream - .send(Ok(ir.clone())) - .map_err(|e| { - Error::from(e).warn(); - }) - .unwrap_or_default(); - }) - .unwrap_or_default(); - } - Err(e) => { - e.warn(); - } - } - } - }); }) .map_err(Error::from)?; Ok(()) diff --git a/lib/tests/test_core_util_file_walker.rs b/lib/tests/test_core_util_file_walker.rs index 59ff0a205..4db85f594 100644 --- a/lib/tests/test_core_util_file_walker.rs +++ b/lib/tests/test_core_util_file_walker.rs @@ -2,7 +2,7 @@ mod common; use common::*; use log::LevelFilter; use xvc_core::{ - util::xvcignore::{walk_parallel, walk_serial}, + util::xvcignore::{walk_parallel, walk_serial, COMMON_IGNORE_PATTERNS}, XvcPath, }; use xvc_test_helper::test_logging; @@ -29,8 +29,8 @@ fn test_walk() -> Result<()> { let xp = XvcPath::new(&xvc_root, xvc_root.absolute_path(), Path::new(skipped))?; assert!(!pmp1.contains_key(&xp), "Result Contains {:?}", xp) } - - let (pmp2, _) = walk_parallel(&xvc_root, true)?; + + let (pmp2, _) = walk_parallel(&xvc_root, COMMON_IGNORE_PATTERNS, true)?; watch!(pmp1); watch!(pmp2); diff --git a/lib/tests/test_core_util_notify.rs b/lib/tests/test_core_util_notify.rs index eeb85c452..5afefecfe 100644 --- a/lib/tests/test_core_util_notify.rs +++ b/lib/tests/test_core_util_notify.rs @@ -26,7 +26,7 @@ fn test_notify() -> Result<()> { watch!(temp_dir); test_logging(log::LevelFilter::Trace); let walk_options = WalkOptions { - ignore_filename: Some(OsString::from(XVCIGNORE_FILENAME)), + ignore_filename: Some(XVCIGNORE_FILENAME.to_owned()), include_dirs: true, }; let (created_paths_snd, created_paths_rec) = crossbeam_channel::unbounded(); diff --git a/lib/tests/test_walker_parallel.rs b/lib/tests/test_walker_parallel.rs index 0457647a5..720deda0f 100644 --- a/lib/tests/test_walker_parallel.rs +++ b/lib/tests/test_walker_parallel.rs @@ -1,5 +1,5 @@ use std::{ - ffi::OsString, fs, path::{Path, PathBuf} + ffi::OsString, fs, path::{Path, PathBuf}, sync::{Arc, RwLock} }; use fast_glob::Glob; @@ -44,7 +44,7 @@ fn new_dir_with_ignores( initial_patterns: &str, ) -> Result { let patterns = create_patterns(root, dir, initial_patterns); - let initialized = IgnoreRules::empty(&PathBuf::from(root)); + let initialized = IgnoreRules::empty(&PathBuf::from(root), None); watch!(patterns); initialized.add_patterns(patterns).unwrap(); Ok(initialized) @@ -72,16 +72,15 @@ fn test_walk_parallel(ignore_src: &str, ignore_content: &str) -> Vec { test_logging(LevelFilter::Trace); let root = create_directory_hierarchy(true).unwrap(); let (path_sender, path_receiver) = crossbeam_channel::unbounded(); - let (ignore_sender, _ignore_receiver) = crossbeam_channel::unbounded(); // We assume ignore_src is among the directories created fs::write( format!("{}/{ignore_src}.gitignore", root.to_string_lossy()), ignore_content, ) .unwrap(); - let initial_rules = new_dir_with_ignores(root.to_string_lossy().as_ref(), None, "").unwrap(); + let initial_rules = Arc::new(RwLock::new(new_dir_with_ignores(root.to_string_lossy().as_ref(), None, "").unwrap())); let walk_options = WalkOptions { - ignore_filename: Some(OsString::from(".gitignore")), + ignore_filename: Some(".gitignore".to_owned()), include_dirs: true, }; walk_parallel( @@ -89,7 +88,6 @@ fn test_walk_parallel(ignore_src: &str, ignore_content: &str) -> Vec { &root, walk_options, path_sender, - ignore_sender, ) .unwrap(); let paths = path_receiver diff --git a/lib/tests/test_walker_serial.rs b/lib/tests/test_walker_serial.rs index 789b85455..8b68f73b4 100644 --- a/lib/tests/test_walker_serial.rs +++ b/lib/tests/test_walker_serial.rs @@ -60,14 +60,14 @@ fn test_walk_serial(ignore_src: &str, ignore_content: &str) -> Vec { ) .unwrap(); let walk_options = WalkOptions { - ignore_filename: Some(OsString::from(".gitignore")), + ignore_filename: Some(".gitignore".to_owned()), include_dirs: true, }; let (output_sender, output_receiver) = crossbeam_channel::unbounded(); let (res_paths, ignore_rules) = walk_serial(&output_sender, "", &root, &walk_options).unwrap(); - watch!(ignore_rules.ignore_patterns.lock().unwrap()); - watch!(ignore_rules.whitelist_patterns.lock().unwrap()); + watch!(ignore_rules.ignore_patterns.read().unwrap()); + watch!(ignore_rules.whitelist_patterns.read().unwrap()); watch!(output_receiver); watch!(res_paths); fs::remove_dir_all(&root).unwrap(); diff --git a/walker/Cargo.toml b/walker/Cargo.toml index bd0f13e34..ed88f5426 100644 --- a/walker/Cargo.toml +++ b/walker/Cargo.toml @@ -22,6 +22,7 @@ fast-glob = "^0.3" ## Parallelization crossbeam-channel = "^0.5" crossbeam = "^0.8" +rayon = "^1.10" ## File system notify = "6.1.1" diff --git a/walker/src/ignore_rules.rs b/walker/src/ignore_rules.rs index 0c1323ba9..4b7501e28 100644 --- a/walker/src/ignore_rules.rs +++ b/walker/src/ignore_rules.rs @@ -1,10 +1,12 @@ //! Ignore patterns for a directory and its child directories. use std::path::{Path, PathBuf}; -use std::sync::{Arc, Mutex}; +use std::sync::{Arc, Mutex, RwLock}; use crate::{Result, Source}; +use rayon::prelude::*; use crate::pattern::{MatchResult, Pattern}; +use fast_glob::glob_match; use xvc_logging::watch; /// Complete set of ignore rules for a directory and its child directories. @@ -14,27 +16,31 @@ pub struct IgnoreRules { /// Typically this is the root directory of Git or Xvc repository. pub root: PathBuf, + /// The name of the ignore file (e.g. `.xvcignore`, `.gitignore`) to be loaded for ignore rules. + pub ignore_filename: Option, + /// All ignore patterns collected from ignore files or specified in code. - // FIXME: This should be RwLock instead but as [fast_glob::Glob.is_match] requires &mut self, - // we use a simpler approach. - pub ignore_patterns: Arc>>, + pub ignore_patterns: Arc>>, /// All whitelist patterns collected from ignore files or specified in code - pub whitelist_patterns: Arc>>, + pub whitelist_patterns: Arc>>, } +pub type SharedIgnoreRules = Arc>; + impl IgnoreRules { /// An empty set of ignore rules that neither ignores nor whitelists any path. - pub fn empty(dir: &Path) -> Self { + pub fn empty(dir: &Path, ignore_filename: Option<&str>) -> Self { IgnoreRules { root: PathBuf::from(dir), - ignore_patterns: Arc::new(Mutex::new(Vec::::new())), - whitelist_patterns: Arc::new(Mutex::new(Vec::::new())), + ignore_filename: ignore_filename.map(|s| s.to_string()), + ignore_patterns: Arc::new(RwLock::new(Vec::::new())), + whitelist_patterns: Arc::new(RwLock::new(Vec::::new())), } } - pub fn from_global_patterns(ignore_root: &Path, given: &str) -> Self { + pub fn from_global_patterns(ignore_root: &Path, ignore_filename: Option<&str>, given: &str) -> Self { let mut given_patterns = Vec::::new(); // Add given patterns to ignore_patterns @@ -42,7 +48,7 @@ impl IgnoreRules { let pattern = Pattern::new(Source::Global, line); given_patterns.push(pattern); } - IgnoreRules::from_patterns(given_patterns, ignore_root) + IgnoreRules::from_patterns(ignore_root, ignore_filename, given_patterns) } @@ -60,7 +66,7 @@ impl IgnoreRules { /// # Returns /// /// A new `IgnoreRules` instance containing the given patterns and root path. - pub fn from_patterns(mut patterns: Vec, ignore_root: &Path) -> Self { + pub fn from_patterns(ignore_root: &Path, ignore_filename: Option<&str>, mut patterns: Vec) -> Self { let mut ignore_patterns = Vec::new(); let mut whitelist_patterns = Vec::new(); patterns.drain(0..patterns.len()).for_each(|pattern| { @@ -71,8 +77,9 @@ impl IgnoreRules { }); IgnoreRules { root: PathBuf::from(ignore_root), - ignore_patterns: Arc::new(Mutex::new(ignore_patterns)), - whitelist_patterns: Arc::new(Mutex::new(whitelist_patterns)), + ignore_filename: ignore_filename.map(|s| s.to_string()), + ignore_patterns: Arc::new(RwLock::new(ignore_patterns)), + whitelist_patterns: Arc::new(RwLock::new(whitelist_patterns)), } } @@ -101,13 +108,10 @@ impl IgnoreRules { /// * `MatchResult::NoMatch` if the path does not match any pattern. pub fn check(&self, path: &Path) -> MatchResult { let is_abs = path.is_absolute(); - watch!(is_abs); // strip_prefix eats the final slash, and ends_with behave differently than str, so we work // around here let path_str = path.to_string_lossy(); - watch!(path_str); let final_slash = path_str.ends_with('/'); - watch!(final_slash); let path = if is_abs { @@ -132,22 +136,18 @@ impl IgnoreRules { { - let mut whitelist_patterns = self.whitelist_patterns.lock().unwrap(); - - for pattern in whitelist_patterns.iter_mut() { - let glob = &mut pattern.glob; - if glob.is_match(&path) { - return MatchResult::Whitelist; - } + let whitelist_patterns = self.whitelist_patterns.read().unwrap(); + if let Some(p) = whitelist_patterns.par_iter().find_any(|pattern| glob_match(&pattern.glob, &path)) { + watch!(p); + return MatchResult::Whitelist; } } { - let mut ignore_patterns = self.ignore_patterns.lock().unwrap(); - for pattern in ignore_patterns.iter_mut() { - if pattern.glob.is_match(&path) { - return MatchResult::Ignore; - } + let ignore_patterns = self.ignore_patterns.read().unwrap(); + if let Some(p) = ignore_patterns.par_iter().find_any(|pattern| glob_match(&pattern.glob, &path)) { + watch!(p); + return MatchResult::Ignore; } } @@ -176,17 +176,17 @@ impl IgnoreRules { assert_eq!(self.root, other.root); { - let mut ignore_patterns = self.ignore_patterns.lock().unwrap(); - let mut other_ignore_patterns = other.ignore_patterns.lock().unwrap(); + let mut ignore_patterns = self.ignore_patterns.write().unwrap(); + let mut other_ignore_patterns = other.ignore_patterns.write().unwrap(); let len = other_ignore_patterns.len(); - other_ignore_patterns.drain(0..len).for_each(|p| ignore_patterns.push(p.into())); + other_ignore_patterns.drain(0..len).for_each(|p| ignore_patterns.push(p)); } { - let mut whitelist_patterns = self.whitelist_patterns.lock().unwrap(); - let mut other_whitelist_patterns = other.whitelist_patterns.lock().unwrap(); + let mut whitelist_patterns = self.whitelist_patterns.write().unwrap(); + let mut other_whitelist_patterns = other.whitelist_patterns.write().unwrap(); let len = other_whitelist_patterns.len(); - other_whitelist_patterns.drain(0..len).for_each(|p| whitelist_patterns.push(p.into())); + other_whitelist_patterns.drain(0..len).for_each(|p| whitelist_patterns.push(p)); } @@ -194,7 +194,7 @@ impl IgnoreRules { } pub fn add_patterns(&self, patterns: Vec) -> Result<()> { - let other = IgnoreRules::from_patterns(patterns, &self.root); + let other = IgnoreRules::from_patterns(&self.root, None, patterns); self.merge_with(&other) } } diff --git a/walker/src/lib.rs b/walker/src/lib.rs index 5110e93a9..5b61f5874 100755 --- a/walker/src/lib.rs +++ b/walker/src/lib.rs @@ -29,7 +29,10 @@ pub use walk_serial::walk_serial; pub use abspath::AbsolutePath; pub use error::{Error, Result}; + pub use ignore_rules::IgnoreRules; +pub use ignore_rules::SharedIgnoreRules; + pub use notify::make_watcher; use std::ffi::OsStr; pub use std::hash::Hash; @@ -68,7 +71,7 @@ pub struct PathMetadata { pub struct WalkOptions { /// The ignore filename (`.gitignore`, `.xvcignore`, `.ignore`, etc.) or `None` for not /// ignoring anything. - pub ignore_filename: Option, + pub ignore_filename: Option, /// Should the results include directories themselves? /// Note that they are always traversed, but may not be listed if we're only interested in /// actual files. @@ -119,7 +122,7 @@ type IgnorePatterns = Vec; pub fn build_ignore_patterns( given: &str, ignore_root: &Path, - ignore_filename: &OsStr, + ignore_filename: &str, ) -> Result { watch!(ignore_filename); @@ -130,9 +133,10 @@ pub fn build_ignore_patterns( .map_err(|e| anyhow!("Error reading directory: {:?}, {:?}", ignore_root, e))?; let mut dir_stack = Vec::::new(); - let ignore_fn = OsString::from(ignore_filename); - let ignore_rules = IgnoreRules::from_global_patterns(ignore_root, given); + let ignore_rules = IgnoreRules::from_global_patterns(ignore_root, Some(ignore_filename), given); + + let ignore_fn = ignore_rules.ignore_filename.as_deref().unwrap(); while let Some(dir) = dir_stack.pop() { let mut new_dirs = Vec::::new(); @@ -220,11 +224,11 @@ pub fn content_to_patterns( } -pub fn update_ignore_rules(ignore_filename: &Option, +pub fn update_ignore_rules( dir: &Path, ignore_rules: &IgnoreRules) -> Result<()> { - if let Some(ignore_filename) = ignore_filename { + if let Some(ref ignore_filename) = ignore_rules.ignore_filename { let ignore_root = &ignore_rules.root; let ignore_path = dir.join(ignore_filename); if ignore_path.is_file() { @@ -364,7 +368,7 @@ mod tests { initial_patterns: &str, ) -> Result { let patterns = create_patterns(root, dir, initial_patterns); - let initialized = IgnoreRules::empty(&PathBuf::from(root)); + let initialized = IgnoreRules::empty(&PathBuf::from(root), None); initialized.add_patterns(patterns)?; Ok(initialized) @@ -416,7 +420,7 @@ mod tests { line: 1, }; let pattern = Pattern::new(source, pattern); - pattern.pattern + pattern.glob } // Blank file tests diff --git a/walker/src/pattern.rs b/walker/src/pattern.rs index 0b9b8fab5..b8d3db8f8 100644 --- a/walker/src/pattern.rs +++ b/walker/src/pattern.rs @@ -89,9 +89,7 @@ pub enum Source { pub struct Pattern { /// The pattern type - pub glob: Glob, - /// The line that we used for glob - pub pattern: String, + pub glob: String, /// The original string that defines the pattern pub original: String, /// Where did we get this pattern? @@ -185,12 +183,10 @@ pub fn new(source: Source, original: &str) -> Self { PatternRelativity::Anywhere }; - let pattern = transform_pattern_for_glob(&line, relativity.clone(), path_kind.clone()); - let glob = Glob::new(&pattern).unwrap(); + let glob = transform_pattern_for_glob(&line, relativity.clone(), path_kind.clone()); Pattern { glob, - pattern, original, source, effect, diff --git a/walker/src/walk_parallel.rs b/walker/src/walk_parallel.rs index 623ef890a..c7394f88b 100644 --- a/walker/src/walk_parallel.rs +++ b/walker/src/walk_parallel.rs @@ -4,22 +4,24 @@ use crossbeam::queue::SegQueue; use crossbeam_channel::Sender; use xvc_logging::{uwr, watch}; -use crate::{directory_list, update_ignore_rules, IgnoreRules, MatchResult, PathMetadata, Result, WalkOptions, MAX_THREADS_PARALLEL_WALK}; +use crate::{directory_list, update_ignore_rules, IgnoreRules, MatchResult, PathMetadata, Result, SharedIgnoreRules, WalkOptions, MAX_THREADS_PARALLEL_WALK}; fn walk_parallel_inner( - ignore_rules: Arc>, + ignore_rules: SharedIgnoreRules, dir: &Path, walk_options: WalkOptions, path_sender: Sender>, ) -> Result> { - let ignore_filename = walk_options.ignore_filename; - update_ignore_rules(&ignore_filename, dir, &ignore_rules.lock().unwrap())?; + update_ignore_rules(dir, &ignore_rules.write().unwrap())?; Ok(directory_list(dir)? .drain(..) .filter_map(|pm_res| match pm_res { - Ok(pm) => Some(pm), + Ok(pm) => { + watch!(pm); + Some(pm) + }, Err(e) => { path_sender .send(Err(e)) @@ -27,17 +29,32 @@ fn walk_parallel_inner( None } }).filter_map(|pm| { - let ignore_res = ignore_rules.lock().unwrap().check(pm.path.as_ref()); + let ignore_res = ignore_rules.read().unwrap().check(pm.path.as_ref()); + watch!(ignore_res); match ignore_res { - MatchResult::NoMatch | MatchResult::Whitelist => Some(pm), + MatchResult::NoMatch | MatchResult::Whitelist => { + + // If the path is a file, don't send it to caller, just send it to the channel. + // If the path is a directory, send it to the channel if `include_dirs` is true. + // The caller expects a list of directories to recurse into. + + if pm.metadata.is_file() { + path_sender.send(Ok(pm.clone())).expect("Channel error in walk_parallel"); + None + } else if pm.metadata.is_dir() { + path_sender.send(Ok(pm.clone())).expect("Channel error in walk_parallel"); + + if walk_options.include_dirs { + Some(pm) + } else { + None + } } else { None } }, + MatchResult::Ignore => { watch!(pm.path); - if pm.metadata.is_file() || (pm.metadata.is_dir() && walk_options.include_dirs) { - path_sender.send(Ok(pm)).expect("Channel error in walk_parallel"); - } None } - }}).filter(|pm| { pm.metadata.is_dir() }).collect::>()) + }}).collect::>()) } @@ -48,27 +65,27 @@ fn walk_parallel_inner( /// It lists elements of a directory, then creates a new crossbeam scope for each child directory and /// calls itself recursively. It may not be feasible for small directories to create threads. pub fn walk_parallel( - ignore_rules: IgnoreRules, + ignore_rules: SharedIgnoreRules, dir: &Path, walk_options: WalkOptions, path_sender: Sender>, - ignore_sender: Sender>>>, ) -> Result<()> { let dir_queue = Arc::new(SegQueue::::new()); - let ignore_rules = Arc::new(Mutex::new(ignore_rules.clone())); - let child_dirs = walk_parallel_inner( ignore_rules.clone(), dir, walk_options.clone(), path_sender.clone(), )?; + watch!(child_dirs); child_dirs.into_iter().for_each(|pm| { dir_queue.push(pm); }); + watch!(dir_queue); + if dir_queue.is_empty() { return Ok(()); } @@ -76,14 +93,12 @@ pub fn walk_parallel( crossbeam::scope(|s| { for thread_i in 0..MAX_THREADS_PARALLEL_WALK { let path_sender = path_sender.clone(); - let ignore_sender = ignore_sender.clone(); let walk_options = walk_options.clone(); let ignore_rules = ignore_rules.clone(); let dir_queue = dir_queue.clone(); s.spawn(move |_| { watch!(path_sender); - watch!(ignore_sender); while let Some(pm) = dir_queue.pop() { let child_dirs = walk_parallel_inner( ignore_rules.clone(), diff --git a/walker/src/walk_serial.rs b/walker/src/walk_serial.rs index 1bb817461..2fdd3ce40 100644 --- a/walker/src/walk_serial.rs +++ b/walker/src/walk_serial.rs @@ -1,8 +1,10 @@ +//! Serial directory walker without parallelization +//! See [`walk_parallel`] for parallel version. use std::{ffi::OsString, fs, path::Path, sync::{Arc, Mutex}}; use xvc_logging::{debug, warn, watch, XvcOutputSender}; -use crate::{content_to_patterns, directory_list, pattern::MatchResult, update_ignore_rules, IgnoreRules, PathMetadata, Pattern, Result, WalkOptions}; +use crate::{content_to_patterns, directory_list, pattern::MatchResult, update_ignore_rules, IgnoreRules, PathMetadata, Result, WalkOptions}; /// Walk `dir` with `walk_options`, with the given _initial_ `ignore_rules`. /// Note that ignore rules are expanded with the rules given in the `ignore_filename` in @@ -17,12 +19,10 @@ pub fn walk_serial( dir: &Path, walk_options: &WalkOptions, ) -> Result<(Vec, IgnoreRules)> { - let ignore_filename = walk_options.ignore_filename.clone().map(OsString::from); - let ignore_rules = IgnoreRules::from_global_patterns(dir, global_ignore_rules); + let ignore_rules = IgnoreRules::from_global_patterns(dir, walk_options.ignore_filename.as_deref(), global_ignore_rules); let ignore_rules = Arc::new(Mutex::new(ignore_rules)); let dir_stack = crossbeam::queue::SegQueue::new(); let res_paths = Arc::new(Mutex::new(Vec::::new())); - let ignore_root = dir.to_path_buf(); dir_stack.push(dir.to_path_buf()); @@ -39,34 +39,9 @@ pub fn walk_serial( .collect()) }; - let filter_child_paths = |child_paths: &Vec| -> Result<()> { - for child_path in child_paths { - watch!(child_path.path); - let ignore_res = ignore_rules.lock()?.check(child_path.path.as_ref()); - match ignore_res { - MatchResult::NoMatch | MatchResult::Whitelist => { - if child_path.metadata.is_dir() { - if walk_options.include_dirs { - res_paths.lock()?.push(child_path.clone()); - } - dir_stack.push(child_path.path.clone()); - } else { - res_paths.lock()?.push(child_path.clone()); - } - } - // We can return anyhow! error here to notice the user that the path is ignored - MatchResult::Ignore => { - debug!(output_snd, "Ignored: {:?}", child_path.path); - } - } - watch!(child_path); - } - Ok(()) - }; - while let Some(dir) = dir_stack.pop() { watch!(dir); - update_ignore_rules(&ignore_filename, &dir, &ignore_rules.lock().unwrap())?; + update_ignore_rules(&dir, &ignore_rules.lock().unwrap())?; let mut res_paths = res_paths.lock()?; get_child_paths(&dir)?.drain(..).filter_map(|p| match ignore_rules.lock().unwrap().check(p.path.as_ref()) { From e5461ef8f3bb8601705289d23c6911e99b948441 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Wed, 14 Aug 2024 22:53:34 +0300 Subject: [PATCH 166/257] fixed walker_serial --- core/src/util/xvcignore.rs | 43 ++--- lib/tests/test_core_util_file_walker.rs | 33 ++-- lib/tests/test_walker_serial.rs | 7 +- walker/src/ignore_rules.rs | 234 ++++++++++++------------ walker/src/pattern.rs | 186 +++++++++---------- walker/src/walk_parallel.rs | 64 ++++--- walker/src/walk_serial.rs | 66 ++++--- 7 files changed, 320 insertions(+), 313 deletions(-) diff --git a/core/src/util/xvcignore.rs b/core/src/util/xvcignore.rs index 809024187..659cea01c 100644 --- a/core/src/util/xvcignore.rs +++ b/core/src/util/xvcignore.rs @@ -10,8 +10,8 @@ use std::ffi::OsString; use std::sync::{Arc, Mutex, RwLock}; use std::thread; use xvc_logging::{warn, watch, XvcOutputSender}; -use xvc_walker::{Result as XvcWalkerResult, SharedIgnoreRules}; use xvc_walker::{self, IgnoreRules, PathMetadata, WalkOptions}; +use xvc_walker::{Result as XvcWalkerResult, SharedIgnoreRules}; /// We ignore `.git` directories even we are not using `.git` pub const COMMON_IGNORE_PATTERNS: &str = ".xvc\n.git\n"; @@ -44,8 +44,12 @@ pub fn walk_serial( ignore_filename: Some(XVCIGNORE_FILENAME.to_owned()), include_dirs, }; - let (res_paths, ignore_rules) = - xvc_walker::walk_serial::walk_serial(output_snd, COMMON_IGNORE_PATTERNS, xvc_root, &walk_options)?; + let (res_paths, ignore_rules) = xvc_walker::walk_serial::walk_serial( + output_snd, + COMMON_IGNORE_PATTERNS, + xvc_root, + &walk_options, + )?; let pmp: XvcPathMetadataMap = res_paths .iter() .filter_map(|pm| { @@ -77,14 +81,14 @@ pub fn walk_serial( /// /// - `xvc_root`: The root structure for Xvc /// - `include_dirs`: Whether to include directories themselves. -/// If `false`, only the actual files in the repository are listed. +/// If `false`, only the actual files in the repository are listed. /// /// ## Returns /// /// - `XvcPathMetadataMap`: A hash map of files. Keys are [XvcPath], values are their -/// [XvcMetadata]. +/// [XvcMetadata]. /// - `IgnoreRules`: The rules that were produced while reading the directories. -/// This is returned here to prevent a second traversal for ignores. +/// This is returned here to prevent a second traversal for ignores. pub fn walk_parallel( xvc_root: &XvcRoot, global_ignore_rules: &str, @@ -95,16 +99,12 @@ pub fn walk_parallel( let ignore_rules = Arc::new(RwLock::new(IgnoreRules::from_global_patterns( xvc_root, Some(XVCIGNORE_FILENAME), - global_ignore_rules))); + global_ignore_rules, + ))); watch!(ignore_rules); - walk_channel( - xvc_root, - ignore_rules.clone(), - include_dirs, - sender, - )?; + walk_channel(xvc_root, ignore_rules.clone(), include_dirs, sender)?; let pmm = thread::spawn(move || { let mut pmm = XvcPathMetadataMap::new(); @@ -113,7 +113,9 @@ pub fn walk_parallel( pmm.insert(path, md); } pmm - }).join().map_err(Error::from)?; + }) + .join() + .map_err(Error::from)?; Ok((pmm, ignore_rules)) } @@ -131,7 +133,7 @@ pub fn walk_parallel( /// - `xvc_root`: The repository root /// - `initial_patterns`: A set of patterns arranged similar to an `.xvcignore` (`.gitignore`) content. /// - `ignore_filename`: The name of the ignore files to be loaded for ignore rules. -/// (ex: `.xvcignore`, `.ignore`, or `.gitignore`) +/// (ex: `.xvcignore`, `.ignore`, or `.gitignore`) /// - `include_dirs`: Whether to send directory records themselves. /// If `false`, only the files in directories are sent. /// - `xpm_upstream`: The channel this function sends the paths and metadata. @@ -152,19 +154,13 @@ pub fn walk_channel( include_dirs: bool, xpm_upstream: Sender<(XvcPath, XvcMetadata)>, ) -> Result<()> { - let walk_options = WalkOptions { ignore_filename: ignore_rules.read()?.ignore_filename.clone(), include_dirs, }; let (path_sender, path_receiver) = bounded::>(CHANNEL_BOUND); - - xvc_walker::walk_parallel::walk_parallel( - ignore_rules, - xvc_root, - walk_options, - path_sender, - )?; + + xvc_walker::walk_parallel::walk_parallel(ignore_rules, xvc_root, walk_options, path_sender)?; crossbeam::scope(|s| { s.spawn(|_| { @@ -193,7 +189,6 @@ pub fn walk_channel( } } }); - }) .map_err(Error::from)?; Ok(()) diff --git a/lib/tests/test_core_util_file_walker.rs b/lib/tests/test_core_util_file_walker.rs index 4db85f594..9480d9f42 100644 --- a/lib/tests/test_core_util_file_walker.rs +++ b/lib/tests/test_core_util_file_walker.rs @@ -7,7 +7,11 @@ use xvc_core::{ }; use xvc_test_helper::test_logging; -use std::{path::Path, time::Duration}; +use std::{ + collections::{BTreeSet, HashSet}, + path::Path, + time::Duration, +}; use xvc_logging::watch; @@ -22,34 +26,23 @@ fn test_walk() -> Result<()> { let (pmp1, _) = walk_serial(&output_sender, &xvc_root, true)?; - assert!(!pmp1.is_empty()); + let path_set1 = HashSet::::from_iter(pmp1.keys().cloned()); + assert!(!path_set1.is_empty()); // Test skip list for skipped in [".dvc", ".xvc", ".git"] { let xp = XvcPath::new(&xvc_root, xvc_root.absolute_path(), Path::new(skipped))?; - assert!(!pmp1.contains_key(&xp), "Result Contains {:?}", xp) + assert!(!path_set1.contains(&xp), "Result Contains {:?}", xp) } - - let (pmp2, _) = walk_parallel(&xvc_root, COMMON_IGNORE_PATTERNS, true)?; - - watch!(pmp1); - watch!(pmp2); - let mut diff1 = Vec::<&XvcPath>::new(); - for k in pmp1.keys() { - if !pmp2.contains_key(k) { - diff1.push(k); - } - } + let (pmp2, _) = walk_parallel(&xvc_root, COMMON_IGNORE_PATTERNS, true)?; + let path_set2 = HashSet::::from_iter(pmp2.keys().cloned()); + assert!(!path_set2.is_empty()); + let diff1: HashSet<&XvcPath> = path_set1.difference(&path_set2).collect(); watch!(diff1); - let mut diff2 = Vec::<&XvcPath>::new(); - for k in pmp2.keys() { - if !pmp1.contains_key(k) { - diff2.push(k); - } - } + let diff2: HashSet<&XvcPath> = path_set2.difference(&path_set1).collect(); watch!(diff2); diff --git a/lib/tests/test_walker_serial.rs b/lib/tests/test_walker_serial.rs index 8b68f73b4..211fcee02 100644 --- a/lib/tests/test_walker_serial.rs +++ b/lib/tests/test_walker_serial.rs @@ -1,6 +1,4 @@ -use std::{ - ffi::OsString, fs, path::{Path, PathBuf} -}; +use std::{fs, path::PathBuf}; use xvc_walker::*; @@ -64,8 +62,7 @@ fn test_walk_serial(ignore_src: &str, ignore_content: &str) -> Vec { include_dirs: true, }; let (output_sender, output_receiver) = crossbeam_channel::unbounded(); - let (res_paths, ignore_rules) = - walk_serial(&output_sender, "", &root, &walk_options).unwrap(); + let (res_paths, ignore_rules) = walk_serial(&output_sender, "", &root, &walk_options).unwrap(); watch!(ignore_rules.ignore_patterns.read().unwrap()); watch!(ignore_rules.whitelist_patterns.read().unwrap()); watch!(output_receiver); diff --git a/walker/src/ignore_rules.rs b/walker/src/ignore_rules.rs index 4b7501e28..8fdd73151 100644 --- a/walker/src/ignore_rules.rs +++ b/walker/src/ignore_rules.rs @@ -1,10 +1,10 @@ //! Ignore patterns for a directory and its child directories. +use crate::{Result, Source}; use std::path::{Path, PathBuf}; use std::sync::{Arc, Mutex, RwLock}; -use crate::{Result, Source}; -use rayon::prelude::*; use crate::pattern::{MatchResult, Pattern}; +use rayon::prelude::*; use fast_glob::glob_match; use xvc_logging::watch; @@ -24,7 +24,6 @@ pub struct IgnoreRules { /// All whitelist patterns collected from ignore files or specified in code pub whitelist_patterns: Arc>>, - } pub type SharedIgnoreRules = Arc>; @@ -40,41 +39,47 @@ impl IgnoreRules { } } - pub fn from_global_patterns(ignore_root: &Path, ignore_filename: Option<&str>, given: &str) -> Self { - - let mut given_patterns = Vec::::new(); - // Add given patterns to ignore_patterns - for line in given.lines() { - let pattern = Pattern::new(Source::Global, line); - given_patterns.push(pattern); - } - IgnoreRules::from_patterns(ignore_root, ignore_filename, given_patterns) + pub fn from_global_patterns( + ignore_root: &Path, + ignore_filename: Option<&str>, + given: &str, + ) -> Self { + let mut given_patterns = Vec::::new(); + // Add given patterns to ignore_patterns + for line in given.lines() { + let pattern = Pattern::new(Source::Global, line); + given_patterns.push(pattern); } + IgnoreRules::from_patterns(ignore_root, ignore_filename, given_patterns) + } - -/// Constructs a new `IgnoreRules` instance from a vector of patterns and a root path. -/// -/// This function separates the patterns into ignore patterns and whitelist patterns -/// based on their `PatternEffect`. It then stores these patterns and the root path -/// in a new `IgnoreRules` instance. -/// -/// # Arguments -/// -/// * `patterns` - A vector of `Pattern` instances to be used for creating the `IgnoreRules`. -/// * `ignore_root` - A reference to the root path for the ignore rules. -/// -/// # Returns -/// -/// A new `IgnoreRules` instance containing the given patterns and root path. - pub fn from_patterns(ignore_root: &Path, ignore_filename: Option<&str>, mut patterns: Vec) -> Self { + /// Constructs a new `IgnoreRules` instance from a vector of patterns and a root path. + /// + /// This function separates the patterns into ignore patterns and whitelist patterns + /// based on their `PatternEffect`. It then stores these patterns and the root path + /// in a new `IgnoreRules` instance. + /// + /// # Arguments + /// + /// * `patterns` - A vector of `Pattern` instances to be used for creating the `IgnoreRules`. + /// * `ignore_root` - A reference to the root path for the ignore rules. + /// + /// # Returns + /// + /// A new `IgnoreRules` instance containing the given patterns and root path. + pub fn from_patterns( + ignore_root: &Path, + ignore_filename: Option<&str>, + mut patterns: Vec, + ) -> Self { let mut ignore_patterns = Vec::new(); let mut whitelist_patterns = Vec::new(); - patterns.drain(0..patterns.len()).for_each(|pattern| { - match pattern.effect { - crate::PatternEffect::Ignore => ignore_patterns.push(pattern), - crate::PatternEffect::Whitelist => whitelist_patterns.push(pattern), - } - }); + patterns + .drain(0..patterns.len()) + .for_each(|pattern| match pattern.effect { + crate::PatternEffect::Ignore => ignore_patterns.push(pattern), + crate::PatternEffect::Whitelist => whitelist_patterns.push(pattern), + }); IgnoreRules { root: PathBuf::from(ignore_root), ignore_filename: ignore_filename.map(|s| s.to_string()), @@ -83,69 +88,71 @@ impl IgnoreRules { } } - - -/// Checks if a given path matches any of the whitelist or ignore patterns. -/// -/// The function first checks if the path matches any of the whitelist patterns. -/// If a match is found, it returns `MatchResult::Whitelist`. -/// -/// If the path does not match any of the whitelist patterns, the function then checks -/// if the path matches any of the ignore patterns. If a match is found, it returns -/// `MatchResult::Ignore`. -/// -/// If the path does not match any of the whitelist or ignore patterns, the function -/// returns `MatchResult::NoMatch`. -/// -/// # Arguments -/// -/// * `path` - A reference to the path to check. -/// -/// # Returns -/// -/// * `MatchResult::Whitelist` if the path matches a whitelist pattern. -/// * `MatchResult::Ignore` if the path matches an ignore pattern. -/// * `MatchResult::NoMatch` if the path does not match any pattern. + /// Checks if a given path matches any of the whitelist or ignore patterns. + /// + /// The function first checks if the path matches any of the whitelist patterns. + /// If a match is found, it returns `MatchResult::Whitelist`. + /// + /// If the path does not match any of the whitelist patterns, the function then checks + /// if the path matches any of the ignore patterns. If a match is found, it returns + /// `MatchResult::Ignore`. + /// + /// If the path does not match any of the whitelist or ignore patterns, the function + /// returns `MatchResult::NoMatch`. + /// + /// # Arguments + /// + /// * `path` - A reference to the path to check. + /// + /// # Returns + /// + /// * `MatchResult::Whitelist` if the path matches a whitelist pattern. + /// * `MatchResult::Ignore` if the path matches an ignore pattern. + /// * `MatchResult::NoMatch` if the path does not match any pattern. pub fn check(&self, path: &Path) -> MatchResult { - let is_abs = path.is_absolute(); - // strip_prefix eats the final slash, and ends_with behave differently than str, so we work - // around here - let path_str = path.to_string_lossy(); - let final_slash = path_str.ends_with('/'); - - let path = if is_abs { - - if final_slash { - format!( - "/{}/", - path.strip_prefix(&self.root) - .expect("path must be within root") - .to_string_lossy() - ) + let is_abs = path.is_absolute(); + // strip_prefix eats the final slash, and ends_with behave differently than str, so we work + // around here + let path_str = path.to_string_lossy(); + let final_slash = path_str.ends_with('/'); + + let path = if is_abs { + if final_slash { + format!( + "/{}/", + path.strip_prefix(&self.root) + .expect("path must be within root") + .to_string_lossy() + ) + } else { + format!( + "/{}", + path.strip_prefix(&self.root) + .expect("path must be within root") + .to_string_lossy() + ) + } } else { - format!( - "/{}", - path.strip_prefix(&self.root) - .expect("path must be within root") - .to_string_lossy() - ) - } - } else { - path_str.to_string() - }; - + path_str.to_string() + }; - { + { let whitelist_patterns = self.whitelist_patterns.read().unwrap(); - if let Some(p) = whitelist_patterns.par_iter().find_any(|pattern| glob_match(&pattern.glob, &path)) { + if let Some(p) = whitelist_patterns + .par_iter() + .find_any(|pattern| glob_match(&pattern.glob, &path)) + { watch!(p); return MatchResult::Whitelist; } } - { + { let ignore_patterns = self.ignore_patterns.read().unwrap(); - if let Some(p) = ignore_patterns.par_iter().find_any(|pattern| glob_match(&pattern.glob, &path)) { + if let Some(p) = ignore_patterns + .par_iter() + .find_any(|pattern| glob_match(&pattern.glob, &path)) + { watch!(p); return MatchResult::Ignore; } @@ -154,47 +161,50 @@ impl IgnoreRules { MatchResult::NoMatch } -/// Merges the ignore and whitelist patterns of another `IgnoreRules` instance into this one. -/// -/// This function locks the ignore and whitelist patterns of both `IgnoreRules` instances, -/// drains the patterns from the other instance, and pushes them into this instance. -/// The other instance is left empty after this operation. -/// -/// # Arguments -/// -/// * `other` - A reference to the other `IgnoreRules` instance to merge with. -/// -/// # Returns -/// -/// * `Ok(())` if the merge operation was successful. -/// * `Err` if the merge operation failed. -/// -/// # Panics -/// -/// This function will panic if the roots of the two `IgnoreRules` instances are not equal. + /// Merges the ignore and whitelist patterns of another `IgnoreRules` instance into this one. + /// + /// This function locks the ignore and whitelist patterns of both `IgnoreRules` instances, + /// drains the patterns from the other instance, and pushes them into this instance. + /// The other instance is left empty after this operation. + /// + /// # Arguments + /// + /// * `other` - A reference to the other `IgnoreRules` instance to merge with. + /// + /// # Returns + /// + /// * `Ok(())` if the merge operation was successful. + /// * `Err` if the merge operation failed. + /// + /// # Panics + /// + /// This function will panic if the roots of the two `IgnoreRules` instances are not equal. pub fn merge_with(&self, other: &IgnoreRules) -> Result<()> { assert_eq!(self.root, other.root); - { + { let mut ignore_patterns = self.ignore_patterns.write().unwrap(); let mut other_ignore_patterns = other.ignore_patterns.write().unwrap(); let len = other_ignore_patterns.len(); - other_ignore_patterns.drain(0..len).for_each(|p| ignore_patterns.push(p)); + other_ignore_patterns + .drain(0..len) + .for_each(|p| ignore_patterns.push(p)); } - { + { let mut whitelist_patterns = self.whitelist_patterns.write().unwrap(); let mut other_whitelist_patterns = other.whitelist_patterns.write().unwrap(); let len = other_whitelist_patterns.len(); - other_whitelist_patterns.drain(0..len).for_each(|p| whitelist_patterns.push(p)); - + other_whitelist_patterns + .drain(0..len) + .for_each(|p| whitelist_patterns.push(p)); } Ok(()) } - pub fn add_patterns(&self, patterns: Vec) -> Result<()> { + pub fn add_patterns(&self, patterns: Vec) -> Result<()> { let other = IgnoreRules::from_patterns(&self.root, None, patterns); self.merge_with(&other) - } } +} diff --git a/walker/src/pattern.rs b/walker/src/pattern.rs index b8d3db8f8..0feaa0602 100644 --- a/walker/src/pattern.rs +++ b/walker/src/pattern.rs @@ -1,20 +1,16 @@ //! Pattern describes a single line in an ignore file and its semantics //! It is used to match a path with the given pattern +use crate::sync; pub use error::{Error, Result}; pub use ignore_rules::IgnoreRules; -use crate::sync; pub use std::hash::Hash; pub use sync::{PathSync, PathSyncSingleton}; pub use crate::notify::{make_watcher, PathEvent, RecommendedWatcher}; - // use glob::{MatchOptions, Pattern, PatternError}; pub use fast_glob::Glob; -use std::{ - fmt::Debug, - path::PathBuf, -}; +use std::{fmt::Debug, path::PathBuf}; use crate::error; use crate::ignore_rules; @@ -66,7 +62,7 @@ pub enum PatternEffect { pub enum Source { /// Pattern is globally defined in code Global, - + /// Pattern is obtained from file File { /// Path of the pattern file @@ -76,9 +72,7 @@ pub enum Source { }, /// Pattern is from CLI - CommandLine { - current_dir: PathBuf - } + CommandLine { current_dir: PathBuf }, } /// Pattern is generic and could be an instance of String, Glob, Regex or any other object. @@ -86,8 +80,7 @@ pub enum Source { /// A pattern can start its life as `Pattern` and can be compiled into `Pattern` or /// `Pattern`. #[derive(Debug)] -pub struct Pattern -{ +pub struct Pattern { /// The pattern type pub glob: String, /// The original string that defines the pattern @@ -103,106 +96,105 @@ pub struct Pattern } impl Pattern { - -pub fn new(source: Source, original: &str) -> Self { - - let original = original.to_owned(); - let current_dir = match &source { - Source::Global => "".to_string(), - Source::File { path, .. } => { - let path = path - .parent() - .expect("Pattern source file doesn't have parent") - .to_string_lossy() - .to_string(); - if path.starts_with('/') { - path - } else { - format!("/{path}") + pub fn new(source: Source, original: &str) -> Self { + let original = original.to_owned(); + let current_dir = match &source { + Source::Global => "".to_string(), + Source::File { path, .. } => { + let path = path + .parent() + .expect("Pattern source file doesn't have parent") + .to_string_lossy() + .to_string(); + if path.starts_with('/') { + path + } else { + format!("/{path}") + } } + Source::CommandLine { current_dir } => current_dir.to_string_lossy().to_string(), + }; + + // if Pattern starts with ! it's whitelist, if ends with / it's dir only, if it contains + // non final slash, it should be considered under the current dir only, otherwise it + // matches + + let begin_exclamation = original.starts_with('!'); + let mut line = if begin_exclamation || original.starts_with(r"\!") { + original[1..].to_owned() + } else { + original.to_owned() + }; + + // TODO: We should handle filenames with trailing spaces better, with regex match and removing + // the \\ from the name + if !line.ends_with("\\ ") { + line = line.trim_end().to_string(); } - Source::CommandLine { current_dir } => current_dir.to_string_lossy().to_string(), - }; - - // if Pattern starts with ! it's whitelist, if ends with / it's dir only, if it contains - // non final slash, it should be considered under the current dir only, otherwise it - // matches - - let begin_exclamation = original.starts_with('!'); - let mut line = if begin_exclamation || original.starts_with(r"\!") { - original[1..].to_owned() - } else { - original.to_owned() - }; - - // TODO: We should handle filenames with trailing spaces better, with regex match and removing - // the \\ from the name - if !line.ends_with("\\ ") { - line = line.trim_end().to_string(); - } - - let end_slash = line.ends_with('/'); - if end_slash { - line = line[..line.len() - 1].to_string() - } - - let begin_slash = line.starts_with('/'); - let non_final_slash = if !line.is_empty() { - line[..line.len() - 1].chars().any(|c| c == '/') - } else { - false - }; - if begin_slash { - line = line[1..].to_string(); - } - - let current_dir = if current_dir.ends_with('/') { - ¤t_dir[..current_dir.len() - 1] - } else { - ¤t_dir - }; - - let effect = if begin_exclamation { - PatternEffect::Whitelist - } else { - PatternEffect::Ignore - }; + let end_slash = line.ends_with('/'); + if end_slash { + line = line[..line.len() - 1].to_string() + } - let path_kind = if end_slash { - PathKind::Directory - } else { - PathKind::Any - }; + let begin_slash = line.starts_with('/'); + let non_final_slash = if !line.is_empty() { + line[..line.len() - 1].chars().any(|c| c == '/') + } else { + false + }; - let relativity = if non_final_slash { - PatternRelativity::RelativeTo { - directory: current_dir.to_owned(), + if begin_slash { + line = line[1..].to_string(); } - } else { - PatternRelativity::Anywhere - }; + + let current_dir = if current_dir.ends_with('/') { + ¤t_dir[..current_dir.len() - 1] + } else { + ¤t_dir + }; + + let effect = if begin_exclamation { + PatternEffect::Whitelist + } else { + PatternEffect::Ignore + }; + + let path_kind = if end_slash { + PathKind::Directory + } else { + PathKind::Any + }; + + let relativity = if non_final_slash { + PatternRelativity::RelativeTo { + directory: current_dir.to_owned(), + } + } else { + PatternRelativity::Anywhere + }; let glob = transform_pattern_for_glob(&line, relativity.clone(), path_kind.clone()); - Pattern { + Pattern { glob, - original, - source, - effect, - relativity, - path_kind, + original, + source, + effect, + relativity, + path_kind, + } } } - -} - - -fn transform_pattern_for_glob(original: &str, relativity: PatternRelativity, path_kind: PathKind) -> String { +fn transform_pattern_for_glob( + original: &str, + relativity: PatternRelativity, + path_kind: PathKind, +) -> String { let anything_anywhere = |p| format!("**/{p}"); let anything_relative = |p, directory| format!("{directory}/**/{p}"); - let directory_anywhere = |p| format!("**{p}/**"); + let directory_anywhere = |p| format!("**/{p}/**"); let directory_relative = |p, directory| format!("{directory}/**/{p}/**"); let transformed_pattern = match (path_kind, relativity) { diff --git a/walker/src/walk_parallel.rs b/walker/src/walk_parallel.rs index c7394f88b..a3d94d7f9 100644 --- a/walker/src/walk_parallel.rs +++ b/walker/src/walk_parallel.rs @@ -1,11 +1,16 @@ -use std::{path::Path, sync::{Arc, Mutex}}; +use std::{ + path::Path, + sync::{Arc, Mutex}, +}; use crossbeam::queue::SegQueue; use crossbeam_channel::Sender; use xvc_logging::{uwr, watch}; -use crate::{directory_list, update_ignore_rules, IgnoreRules, MatchResult, PathMetadata, Result, SharedIgnoreRules, WalkOptions, MAX_THREADS_PARALLEL_WALK}; - +use crate::{ + directory_list, update_ignore_rules, IgnoreRules, MatchResult, PathMetadata, Result, + SharedIgnoreRules, WalkOptions, MAX_THREADS_PARALLEL_WALK, +}; fn walk_parallel_inner( ignore_rules: SharedIgnoreRules, @@ -13,49 +18,58 @@ fn walk_parallel_inner( walk_options: WalkOptions, path_sender: Sender>, ) -> Result> { - update_ignore_rules(dir, &ignore_rules.write().unwrap())?; + update_ignore_rules(dir, &ignore_rules.write().unwrap())?; - Ok(directory_list(dir)? + Ok(directory_list(dir)? .drain(..) .filter_map(|pm_res| match pm_res { Ok(pm) => { watch!(pm); Some(pm) - }, + } Err(e) => { path_sender .send(Err(e)) .expect("Channel error in walk_parallel"); None } - }).filter_map(|pm| { - let ignore_res = ignore_rules.read().unwrap().check(pm.path.as_ref()); - watch!(ignore_res); - match ignore_res { - MatchResult::NoMatch | MatchResult::Whitelist => { - + }) + .filter_map(|pm| { + let ignore_res = ignore_rules.read().unwrap().check(pm.path.as_ref()); + watch!(ignore_res); + match ignore_res { + MatchResult::NoMatch | MatchResult::Whitelist => { // If the path is a file, don't send it to caller, just send it to the channel. // If the path is a directory, send it to the channel if `include_dirs` is true. // The caller expects a list of directories to recurse into. - - if pm.metadata.is_file() { - path_sender.send(Ok(pm.clone())).expect("Channel error in walk_parallel"); + + if pm.metadata.is_file() { + path_sender + .send(Ok(pm.clone())) + .expect("Channel error in walk_parallel"); None - } else if pm.metadata.is_dir() { - path_sender.send(Ok(pm.clone())).expect("Channel error in walk_parallel"); + } else if pm.metadata.is_dir() { + path_sender + .send(Ok(pm.clone())) + .expect("Channel error in walk_parallel"); if walk_options.include_dirs { Some(pm) } else { - None - } } else { None } }, - - MatchResult::Ignore => { - watch!(pm.path); + None + } + } else { None } - }}).collect::>()) + } + MatchResult::Ignore => { + watch!(pm.path); + None + } + } + }) + .collect::>()) } /// Walk all child paths under `dir` and send non-ignored paths to `path_sender`. @@ -105,7 +119,8 @@ pub fn walk_parallel( &pm.path, walk_options.clone(), path_sender.clone(), - ).unwrap(); + ) + .unwrap(); for child_dir in child_dirs { dir_queue.push(child_dir); @@ -121,4 +136,3 @@ pub fn walk_parallel( Ok(()) } - diff --git a/walker/src/walk_serial.rs b/walker/src/walk_serial.rs index 2fdd3ce40..c4ac49f06 100644 --- a/walker/src/walk_serial.rs +++ b/walker/src/walk_serial.rs @@ -1,10 +1,13 @@ //! Serial directory walker without parallelization //! See [`walk_parallel`] for parallel version. -use std::{ffi::OsString, fs, path::Path, sync::{Arc, Mutex}}; +use std::path::Path; use xvc_logging::{debug, warn, watch, XvcOutputSender}; -use crate::{content_to_patterns, directory_list, pattern::MatchResult, update_ignore_rules, IgnoreRules, PathMetadata, Result, WalkOptions}; +use crate::{ + directory_list, pattern::MatchResult, update_ignore_rules, IgnoreRules, PathMetadata, Result, + WalkOptions, +}; /// Walk `dir` with `walk_options`, with the given _initial_ `ignore_rules`. /// Note that ignore rules are expanded with the rules given in the `ignore_filename` in @@ -19,46 +22,49 @@ pub fn walk_serial( dir: &Path, walk_options: &WalkOptions, ) -> Result<(Vec, IgnoreRules)> { - let ignore_rules = IgnoreRules::from_global_patterns(dir, walk_options.ignore_filename.as_deref(), global_ignore_rules); - let ignore_rules = Arc::new(Mutex::new(ignore_rules)); - let dir_stack = crossbeam::queue::SegQueue::new(); - let res_paths = Arc::new(Mutex::new(Vec::::new())); + let ignore_rules = IgnoreRules::from_global_patterns( + dir, + walk_options.ignore_filename.as_deref(), + global_ignore_rules, + ); + + let mut dir_stack = Vec::new(); dir_stack.push(dir.to_path_buf()); let get_child_paths = |dir: &Path| -> Result> { - Ok(directory_list(dir)? - .into_iter() - .filter_map(|pm_res| match pm_res { - Ok(pm) => Some(pm), - Err(e) => { - warn!(output_snd, "{}", e); - None - } - }) - .collect()) + Ok(directory_list(dir)? + .into_iter() + .filter_map(|pm_res| match pm_res { + Ok(pm) => Some(pm), + Err(e) => { + warn!(output_snd, "{}", e); + None + } + }) + .collect()) }; + let mut res_paths = Vec::new(); while let Some(dir) = dir_stack.pop() { - watch!(dir); - update_ignore_rules(&dir, &ignore_rules.lock().unwrap())?; - let mut res_paths = res_paths.lock()?; - get_child_paths(&dir)?.drain(..).filter_map(|p| - match ignore_rules.lock().unwrap().check(p.path.as_ref()) { - MatchResult::NoMatch | MatchResult::Whitelist => Some(p), + update_ignore_rules(&dir, &ignore_rules)?; + + res_paths.extend(get_child_paths(&dir)?.drain(..).filter_map(|p| { + let ignore_result = ignore_rules.check(p.path.as_ref()); + match ignore_result { + MatchResult::NoMatch | MatchResult::Whitelist => { + if p.metadata.is_dir() { + dir_stack.push(p.path.clone()); + } + Some(p) + } MatchResult::Ignore => { debug!(output_snd, "Ignored: {:?}", p.path); None } - }).for_each(|p| res_paths.push(p)); + } + })); } - let res_paths: Vec = res_paths.lock()?.clone(); - let ignore_rules = ignore_rules.lock()?.clone(); - Ok((res_paths, ignore_rules)) } - - - - From 49e3da72d3d05088bd3e2caebffd1cfb1ec5f3c2 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Wed, 14 Aug 2024 22:58:00 +0300 Subject: [PATCH 167/257] fix warnings --- core/src/lib.rs | 2 -- core/src/util/git.rs | 10 ++-------- core/src/util/pmp.rs | 14 +++++--------- core/src/util/xvcignore.rs | 17 ++++++++--------- 4 files changed, 15 insertions(+), 28 deletions(-) diff --git a/core/src/lib.rs b/core/src/lib.rs index 8daa78477..3a8efe036 100755 --- a/core/src/lib.rs +++ b/core/src/lib.rs @@ -8,8 +8,6 @@ pub mod root; pub mod types; pub mod util; -use std::ffi::OsStr; - pub use types::hashalgorithm::HashAlgorithm; pub use types::recheckmethod::RecheckMethod; diff --git a/core/src/util/git.rs b/core/src/util/git.rs index c03b85290..b984ea712 100644 --- a/core/src/util/git.rs +++ b/core/src/util/git.rs @@ -1,7 +1,6 @@ //! Git operations use std::path::{Path, PathBuf}; -use xvc_logging::watch; use xvc_walker::{build_ignore_patterns, AbsolutePath, IgnoreRules}; use crate::error::Result; @@ -28,7 +27,6 @@ pub fn inside_git(path: &Path) -> Option { /// Returns [xvc_walker::IgnoreRules] for `.gitignore` /// It's used to check whether a path is already ignored by Git. pub fn build_gitignore(git_root: &AbsolutePath) -> Result { - let rules = build_ignore_patterns("", git_root, ".gitignore".to_owned().as_ref())?; Ok(rules) @@ -62,12 +60,8 @@ mod test { } fs::write(&gitignore_path, format!("{}\n", ignore_line)).unwrap(); - let gitignore = build_ignore_patterns( - "", - gitignore_path.parent().unwrap(), - ".gitignore" - ) - .unwrap(); + let gitignore = + build_ignore_patterns("", gitignore_path.parent().unwrap(), ".gitignore").unwrap(); watch!(gitignore); diff --git a/core/src/util/pmp.rs b/core/src/util/pmp.rs index d1ce4c680..1e5c3816b 100644 --- a/core/src/util/pmp.rs +++ b/core/src/util/pmp.rs @@ -2,7 +2,6 @@ use std::collections::HashMap; -use std::ffi::OsString; use std::sync::{Arc, Mutex, RwLock}; use std::thread::{self, JoinHandle}; use std::time::Duration; @@ -36,7 +35,8 @@ pub struct XvcPathMetadataProvider { impl XvcPathMetadataProvider { /// Create a new PathMetadataProvider pub fn new(output_sender: &XvcOutputSender, xvc_root: &XvcRoot) -> Result { - let ignore_rules = build_ignore_patterns(COMMON_IGNORE_PATTERNS, xvc_root, XVCIGNORE_FILENAME)?; + let ignore_rules = + build_ignore_patterns(COMMON_IGNORE_PATTERNS, xvc_root, XVCIGNORE_FILENAME)?; let path_map = Arc::new(RwLock::new(HashMap::new())); let (watcher, event_receiver) = make_watcher(ignore_rules.clone())?; @@ -56,8 +56,7 @@ impl XvcPathMetadataProvider { let watcher = watcher; watch!(watcher); - let handle_fs_event = |fs_event, pmm: Arc>| { - match fs_event { + let handle_fs_event = |fs_event, pmm: Arc>| match fs_event { PathEvent::Create { path, metadata } => { let xvc_path = XvcPath::new(&xvc_root, &xvc_root, &path).unwrap(); let xvc_md = XvcMetadata::from(metadata); @@ -80,7 +79,7 @@ impl XvcPathMetadataProvider { let mut pmm = pmm.write().unwrap(); pmm.insert(xvc_path, xvc_md); } - } }; + }; let mut sel = Select::new(); let fs_event_index = sel.recv(&fs_receiver); @@ -193,10 +192,7 @@ impl XvcPathMetadataProvider { for entry in glob::glob(glob)? { match entry { Ok(entry) => { - if matches!( - &self.ignore_rules.check(&entry), - MatchResult::Ignore - ) { + if matches!(&self.ignore_rules.check(&entry), MatchResult::Ignore) { continue; } else { let xvc_path = XvcPath::new(&self.xvc_root, &self.xvc_root, &entry)?; diff --git a/core/src/util/xvcignore.rs b/core/src/util/xvcignore.rs index 659cea01c..9d9119eec 100644 --- a/core/src/util/xvcignore.rs +++ b/core/src/util/xvcignore.rs @@ -6,8 +6,7 @@ use crate::{XvcMetadata, XvcPathMetadataMap, CHANNEL_BOUND, XVCIGNORE_FILENAME}; use crate::error::{Error, Result}; use crossbeam_channel::{bounded, Sender}; -use std::ffi::OsString; -use std::sync::{Arc, Mutex, RwLock}; +use std::sync::{Arc, RwLock}; use std::thread; use xvc_logging::{warn, watch, XvcOutputSender}; use xvc_walker::{self, IgnoreRules, PathMetadata, WalkOptions}; @@ -27,14 +26,14 @@ pub const COMMON_IGNORE_PATTERNS: &str = ".xvc\n.git\n"; /// /// - `xvc_root`: The root structure for Xvc /// - `include_dirs`: Whether to include directories themselves. -/// If `false`, only the actual files in the repository are listed. +/// If `false`, only the actual files in the repository are listed. /// /// ## Returns /// /// - `XvcPathMetadataMap`: A hash map of files. Keys are [XvcPath], values are their -/// [XvcMetadata]. +/// [XvcMetadata]. /// - `IgnoreRules`: The rules that were produced while reading the directories. -/// This is returned here to prevent a second traversal for ignores. +/// This is returned here to prevent a second traversal for ignores. pub fn walk_serial( output_snd: &XvcOutputSender, xvc_root: &XvcRoot, @@ -81,14 +80,14 @@ pub fn walk_serial( /// /// - `xvc_root`: The root structure for Xvc /// - `include_dirs`: Whether to include directories themselves. -/// If `false`, only the actual files in the repository are listed. +/// If `false`, only the actual files in the repository are listed. /// /// ## Returns /// /// - `XvcPathMetadataMap`: A hash map of files. Keys are [XvcPath], values are their -/// [XvcMetadata]. +/// [XvcMetadata]. /// - `IgnoreRules`: The rules that were produced while reading the directories. -/// This is returned here to prevent a second traversal for ignores. +/// This is returned here to prevent a second traversal for ignores. pub fn walk_parallel( xvc_root: &XvcRoot, global_ignore_rules: &str, @@ -133,7 +132,7 @@ pub fn walk_parallel( /// - `xvc_root`: The repository root /// - `initial_patterns`: A set of patterns arranged similar to an `.xvcignore` (`.gitignore`) content. /// - `ignore_filename`: The name of the ignore files to be loaded for ignore rules. -/// (ex: `.xvcignore`, `.ignore`, or `.gitignore`) +/// (ex: `.xvcignore`, `.ignore`, or `.gitignore`) /// - `include_dirs`: Whether to send directory records themselves. /// If `false`, only the files in directories are sent. /// - `xpm_upstream`: The channel this function sends the paths and metadata. From e995543d2a092c02cfb481d29ecb8f60f984302f Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Thu, 15 Aug 2024 11:19:38 +0300 Subject: [PATCH 168/257] fix warning --- file/src/bring/mod.rs | 4 +-- file/src/common/mod.rs | 79 ++++++++++++++++++++++++++---------------- 2 files changed, 51 insertions(+), 32 deletions(-) diff --git a/file/src/bring/mod.rs b/file/src/bring/mod.rs index 56208d465..b3a90c87f 100644 --- a/file/src/bring/mod.rs +++ b/file/src/bring/mod.rs @@ -3,8 +3,8 @@ //! - [BringCLI] defines the command line options. //! //! - [cmd_bring] is the entry point for the command. -//! Uses [fetch] and [crate::recheck::cmd_recheck] to bring the file and copy/link it to the -//! workspace. +//! Uses [fetch] and [crate::recheck::cmd_recheck] to bring the file and copy/link it to the +//! workspace. use crate::common::{load_targets_from_store, move_to_cache}; diff --git a/file/src/common/mod.rs b/file/src/common/mod.rs index 4ccce0a98..349ff69f5 100644 --- a/file/src/common/mod.rs +++ b/file/src/common/mod.rs @@ -28,7 +28,7 @@ use xvc_logging::{error, info, uwr, warn, watch, XvcOutputSender}; use xvc_ecs::{persist, HStore, Storable, XvcStore}; -use xvc_walker::{AbsolutePath, Error as XvcWalkerError, Glob, IgnoreRules, PathSync, Source}; +use xvc_walker::{AbsolutePath, Glob, PathSync}; use self::gitignore::IgnoreOp; @@ -147,7 +147,7 @@ pub fn filter_targets_from_store( /// GlobSet and paths are checked against the set. pub fn filter_paths_by_globs(paths: &HStore, globs: &[String]) -> Result> { if globs.is_empty() { - return Ok(paths.to_owned()) + return Ok(paths.to_owned()); } let mut glob_matcher = globs.first().and_then(|glob| Glob::new(glob)).unwrap(); @@ -167,20 +167,21 @@ pub fn filter_paths_by_globs(paths: &HStore, globs: &[String]) -> Resul if glob_matcher.is_match(str_path) { Some((*e, p.clone())) - } else { - None - } + } else { + None + } }) .collect(); watch!(paths); - + Ok(paths) } -pub fn build_glob_matcher(output_snd: &XvcOutputSender, dir: &Path, globs: &[String]) -> Result { - - let source = Source::Global; - +pub fn build_glob_matcher( + output_snd: &XvcOutputSender, + dir: &Path, + globs: &[String], +) -> Result { let mut glob_matcher = globs.first().and_then(|glob| Glob::new(glob)).unwrap(); globs.iter().skip(1).for_each(|t| { globs.iter().for_each(|t| { @@ -191,8 +192,8 @@ pub fn build_glob_matcher(output_snd: &XvcOutputSender, dir: &Path, globs: &[Str } else if !t.contains('*') { let abs_target = dir.join(Path::new(t)); if abs_target.is_dir() { - if !glob_matcher.add(&format!("{t}/**")) { - error!(output_snd, "Error in glob: {t}") + if !glob_matcher.add(&format!("{t}/**")) { + error!(output_snd, "Error in glob: {t}") } } else if !glob_matcher.add(t) { error!(output_snd, "Error in glob: {t}") @@ -239,10 +240,15 @@ pub fn targets_from_disk( None => vec![cwd.to_string()], }; - return targets_from_disk(output_snd, xvc_root, xvc_root.absolute_path(), &Some(targets)); + return targets_from_disk( + output_snd, + xvc_root, + xvc_root.absolute_path(), + &Some(targets), + ); } // FIXME: If there are no globs/directories in the targets, no need to retrieve all the paths - // here. + // here. let (all_paths, _) = all_paths_and_metadata(xvc_root); watch!(all_paths); @@ -252,7 +258,7 @@ pub fn targets_from_disk( return Ok(XvcPathMetadataMap::new()); } - let mut glob_matcher = build_glob_matcher(output_snd, &xvc_root, targets)?; + let mut glob_matcher = build_glob_matcher(output_snd, xvc_root, targets)?; watch!(glob_matcher); Ok(all_paths .into_iter() @@ -369,8 +375,12 @@ pub fn recheck_from_cache( Ok(()) } -#[cfg(feature="reflink")] -fn reflink(output_snd: &XvcOutputSender, cache_path: AbsolutePath, path: AbsolutePath) -> Result<()> { +#[cfg(feature = "reflink")] +fn reflink( + output_snd: &XvcOutputSender, + cache_path: AbsolutePath, + path: AbsolutePath, +) -> Result<()> { match reflink::reflink(&cache_path, &path) { Ok(_) => { info!(output_snd, "[REFLINK] {} -> {}", cache_path, path); @@ -386,22 +396,31 @@ fn reflink(output_snd: &XvcOutputSender, cache_path: AbsolutePath, path: Absolut } } -fn copy_file(output_snd: &XvcOutputSender, cache_path: AbsolutePath, path: AbsolutePath) -> Result<()> { - watch!("Before copy"); - watch!(&cache_path); - watch!(&path); - fs::copy(&cache_path, &path)?; - info!(output_snd, "[COPY] {} -> {}", cache_path, path); - let mut perm = path.metadata()?.permissions(); - watch!(&perm); - perm.set_readonly(false); - watch!(&perm); - fs::set_permissions(&path, perm)?; +fn copy_file( + output_snd: &XvcOutputSender, + cache_path: AbsolutePath, + path: AbsolutePath, +) -> Result<()> { + watch!("Before copy"); + watch!(&cache_path); + watch!(&path); + fs::copy(&cache_path, &path)?; + info!(output_snd, "[COPY] {} -> {}", cache_path, path); + let mut perm = path.metadata()?.permissions(); + watch!(&perm); + // FIXME: Fix the clippy warning in the following line + perm.set_readonly(false); + watch!(&perm); + fs::set_permissions(&path, perm)?; Ok(()) } -#[cfg(not(feature="reflink"))] -fn reflink(output_snd: &XvcOutputSender, cache_path: AbsolutePath, path: AbsolutePath) -> Result<()> { +#[cfg(not(feature = "reflink"))] +fn reflink( + output_snd: &XvcOutputSender, + cache_path: AbsolutePath, + path: AbsolutePath, +) -> Result<()> { warn!( output_snd, "Xvc isn't compiled with reflink support. Copying the file." From cadebbe5aa7a4ce9e16f18f1e2500f0f028e9510 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Thu, 15 Aug 2024 11:38:07 +0300 Subject: [PATCH 169/257] fix watch --- file/src/common/mod.rs | 12 +++++++----- file/src/list/mod.rs | 1 + lib/tests/test_file_list.rs | 5 ----- 3 files changed, 8 insertions(+), 10 deletions(-) diff --git a/file/src/common/mod.rs b/file/src/common/mod.rs index 349ff69f5..92f097671 100644 --- a/file/src/common/mod.rs +++ b/file/src/common/mod.rs @@ -150,8 +150,8 @@ pub fn filter_paths_by_globs(paths: &HStore, globs: &[String]) -> Resul return Ok(paths.to_owned()); } - let mut glob_matcher = globs.first().and_then(|glob| Glob::new(glob)).unwrap(); - globs.iter().skip(1).for_each(|t| { + let mut glob_matcher = Glob::default(); + globs.iter().for_each(|t| { watch!(t); if t.ends_with('/') { glob_matcher.add(&format!("{t}**")); @@ -182,15 +182,17 @@ pub fn build_glob_matcher( dir: &Path, globs: &[String], ) -> Result { - let mut glob_matcher = globs.first().and_then(|glob| Glob::new(glob)).unwrap(); - globs.iter().skip(1).for_each(|t| { + let mut glob_matcher = Glob::default(); + globs.iter().for_each(|t| { globs.iter().for_each(|t| { + watch!(t); if t.ends_with('/') { if !glob_matcher.add(&format!("{t}**")) { error!(output_snd, "Error in glob: {t}"); } } else if !t.contains('*') { let abs_target = dir.join(Path::new(t)); + watch!(abs_target); if abs_target.is_dir() { if !glob_matcher.add(&format!("{t}/**")) { error!(output_snd, "Error in glob: {t}") @@ -239,7 +241,7 @@ pub fn targets_from_disk( Some(targets) => targets.iter().map(|t| format!("{cwd}{t}")).collect(), None => vec![cwd.to_string()], }; - + watch!(targets); return targets_from_disk( output_snd, xvc_root, diff --git a/file/src/list/mod.rs b/file/src/list/mod.rs index 1901ac10c..41b92523c 100644 --- a/file/src/list/mod.rs +++ b/file/src/list/mod.rs @@ -531,6 +531,7 @@ pub fn cmd_list(output_snd: &XvcOutputSender, xvc_root: &XvcRoot, cli_opts: List let current_dir = conf.current_dir()?; let all_from_disk = targets_from_disk(output_snd, xvc_root, current_dir, &opts.targets)?; + watch!(&all_from_disk); let from_disk = if opts.show_dot_files { all_from_disk } else { diff --git a/lib/tests/test_file_list.rs b/lib/tests/test_file_list.rs index c408cff8f..537457268 100644 --- a/lib/tests/test_file_list.rs +++ b/lib/tests/test_file_list.rs @@ -6,7 +6,6 @@ use std::path::Path; use log::LevelFilter; use xvc::error::Result; -use xvc::watch; use xvc_config::XvcVerbosity; use xvc_test_helper::{create_directory_tree, test_logging}; use xvc_walker::AbsolutePath; @@ -55,13 +54,9 @@ fn test_file_list() -> Result<()> { common::run_xvc(Some(&xvc_root), &c, XvcVerbosity::Trace) }; - watch!("begin"); let list_all = x(&["list", "--format", "{{name}}", "--show-dot-files"])?; - watch!(list_all); - let count_all = list_all.trim().lines().count(); - watch!(count_all); // There must be 33 elements in total. 6 x 5: directories, 1 for .gitignore, // 1 for .xvcignore, another line for the summary. assert!(count_all == 33); From 5fa24ccedc38abe122865d7216f36c8932437223 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Thu, 15 Aug 2024 12:02:59 +0300 Subject: [PATCH 170/257] fixed file_recheck --- file/src/common/mod.rs | 29 ++++++++++++++++++++++++++++- file/src/list/mod.rs | 4 ++-- file/src/recheck/mod.rs | 5 +++++ 3 files changed, 35 insertions(+), 3 deletions(-) diff --git a/file/src/common/mod.rs b/file/src/common/mod.rs index 92f097671..f02d15e0b 100644 --- a/file/src/common/mod.rs +++ b/file/src/common/mod.rs @@ -145,6 +145,8 @@ pub fn filter_targets_from_store( /// Filter a set of paths by a set of globs. The globs are compiled into a /// GlobSet and paths are checked against the set. +/// +/// If a target ends with /, it's considered a directory and all its children are also selected. pub fn filter_paths_by_globs(paths: &HStore, globs: &[String]) -> Result> { if globs.is_empty() { return Ok(paths.to_owned()); @@ -156,7 +158,7 @@ pub fn filter_paths_by_globs(paths: &HStore, globs: &[String]) -> Resul if t.ends_with('/') { glob_matcher.add(&format!("{t}**")); } else { - glob_matcher.add(&format!("{t}/**")); + glob_matcher.add(t); } }); @@ -177,6 +179,31 @@ pub fn filter_paths_by_globs(paths: &HStore, globs: &[String]) -> Resul Ok(paths) } +/// Builds a glob matcher based on the provided directory and glob patterns. +/// +/// # Arguments +/// +/// * `output_snd`: A sender for output messages. +/// * `dir`: The directory to which the glob patterns will be applied. +/// * `globs`: A slice of glob patterns as strings. +/// +/// # Returns +/// +/// * `Result`: A `Result` that contains the `Glob` matcher if successful, or an error if not. +/// +/// # Errors +/// +/// This function will return an error if any of the glob patterns are invalid. +/// +/// # Example +/// +/// ``` +/// let output_snd = ...; // Some XvcOutputSender +/// let dir = Path::new("/path/to/dir"); +/// let globs = vec!["*.rs", "src/"]; +/// let matcher = build_glob_matcher(&output_snd, &dir, &globs); +/// ``` + pub fn build_glob_matcher( output_snd: &XvcOutputSender, dir: &Path, diff --git a/file/src/list/mod.rs b/file/src/list/mod.rs index 41b92523c..605647f13 100644 --- a/file/src/list/mod.rs +++ b/file/src/list/mod.rs @@ -192,12 +192,12 @@ impl ListRow { watch!(&path_prefix); let name = if let Some(ap) = path_match.actual_path { watch!(ap); - ap.strip_prefix(&path_prefix.to_string_lossy().to_string()) + ap.strip_prefix(path_prefix.to_string_lossy().as_ref()) .map_err(|e| Error::RelativeStripPrefixError { e })? .to_string() } else if let Some(rp) = path_match.recorded_path { watch!(rp); - rp.strip_prefix(&path_prefix.to_string_lossy().to_string()) + rp.strip_prefix(path_prefix.to_string_lossy().as_ref()) .map_err(|e| Error::RelativeStripPrefixError { e })? .to_string() } else { diff --git a/file/src/recheck/mod.rs b/file/src/recheck/mod.rs index 37d9c07b2..399a5f863 100644 --- a/file/src/recheck/mod.rs +++ b/file/src/recheck/mod.rs @@ -98,11 +98,16 @@ pub fn cmd_recheck( watch!(targets); let stored_xvc_path_store = xvc_root.load_store::()?; + watch!(stored_xvc_path_store); let stored_xvc_metadata_store = xvc_root.load_store::()?; + watch!(stored_xvc_metadata_store); let target_files = only_file_targets(&stored_xvc_metadata_store, &targets)?; + watch!(target_files); let target_xvc_path_metadata_map = xvc_path_metadata_map_from_disk(xvc_root, &target_files); + watch!(target_xvc_path_metadata_map); let stored_recheck_method_store = xvc_root.load_store::()?; + watch!(stored_recheck_method_store); let stored_content_digest_store = xvc_root.load_store::()?; let entities: HashSet = target_files.keys().copied().collect(); let default_recheck_method = RecheckMethod::from_conf(xvc_root.config()); From b0d9ec22cac7782692d740f34ce2532b3e023c02 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Thu, 15 Aug 2024 12:55:14 +0300 Subject: [PATCH 171/257] fixed gitignore handler --- core/src/util/git.rs | 8 ++- file/src/common/gitignore.rs | 9 +-- file/src/common/mod.rs | 28 ++++---- lib/tests/test_file_track_issue_104.rs | 6 +- walker/src/lib.rs | 97 +++++++++++--------------- walker/src/walk_serial.rs | 2 +- 6 files changed, 68 insertions(+), 82 deletions(-) diff --git a/core/src/util/git.rs b/core/src/util/git.rs index b984ea712..2a1cfd274 100644 --- a/core/src/util/git.rs +++ b/core/src/util/git.rs @@ -6,6 +6,8 @@ use xvc_walker::{build_ignore_patterns, AbsolutePath, IgnoreRules}; use crate::error::Result; use crate::GIT_DIR; +use super::xvcignore::COMMON_IGNORE_PATTERNS; + /// Check whether a path is inside a Git repository. /// It returns `None` if not, otherwise returns the closest directory with `.git`. /// It works by checking `.git` directories in parents, until no more parent left. @@ -27,7 +29,11 @@ pub fn inside_git(path: &Path) -> Option { /// Returns [xvc_walker::IgnoreRules] for `.gitignore` /// It's used to check whether a path is already ignored by Git. pub fn build_gitignore(git_root: &AbsolutePath) -> Result { - let rules = build_ignore_patterns("", git_root, ".gitignore".to_owned().as_ref())?; + let rules = build_ignore_patterns( + COMMON_IGNORE_PATTERNS, + git_root, + ".gitignore".to_owned().as_ref(), + )?; Ok(rules) } diff --git a/file/src/common/gitignore.rs b/file/src/common/gitignore.rs index 43cb6ab0d..9d97c2db7 100644 --- a/file/src/common/gitignore.rs +++ b/file/src/common/gitignore.rs @@ -7,7 +7,7 @@ use std::fs::OpenOptions; use std::io::Write; use std::thread::JoinHandle; -use xvc_core::{util::git::build_gitignore}; +use xvc_core::util::git::build_gitignore; use crate::{Result, CHANNEL_CAPACITY}; use xvc_core::{XvcPath, XvcRoot}; @@ -42,11 +42,8 @@ pub fn make_ignore_handler( let (sender, receiver) = crossbeam_channel::bounded(CHANNEL_CAPACITY); let output_snd = output_snd.clone(); let xvc_root = xvc_root.absolute_path().clone(); - let xvc_directory = xvc_root.as_path().to_str().unwrap(); - let handle = std::thread::spawn(move || { - let mut ignore_dirs = Vec::::new(); let mut ignore_files = Vec::::new(); @@ -58,7 +55,6 @@ pub fn make_ignore_handler( let path = dir.to_absolute_path(&xvc_root).to_path_buf(); if !ignore_dirs.contains(&dir) - && matches!(gitignore.check(&path), MatchResult::NoMatch) { ignore_dirs.push(dir); @@ -107,7 +103,6 @@ pub fn update_dir_gitignores( current_gitignore: &IgnoreRules, dirs: &[XvcPath], ) -> Result<()> { - // Check if dirs are already ignored let dirs: Vec = dirs .iter() @@ -134,7 +129,7 @@ pub fn update_dir_gitignores( None } }}).collect(); - + // Check if files are already ignored let mut changes = HashMap::>::new(); diff --git a/file/src/common/mod.rs b/file/src/common/mod.rs index f02d15e0b..3ed1779e7 100644 --- a/file/src/common/mod.rs +++ b/file/src/common/mod.rs @@ -211,26 +211,24 @@ pub fn build_glob_matcher( ) -> Result { let mut glob_matcher = Glob::default(); globs.iter().for_each(|t| { - globs.iter().for_each(|t| { - watch!(t); - if t.ends_with('/') { - if !glob_matcher.add(&format!("{t}**")) { - error!(output_snd, "Error in glob: {t}"); - } - } else if !t.contains('*') { - let abs_target = dir.join(Path::new(t)); - watch!(abs_target); - if abs_target.is_dir() { - if !glob_matcher.add(&format!("{t}/**")) { - error!(output_snd, "Error in glob: {t}") - } - } else if !glob_matcher.add(t) { + watch!(t); + if t.ends_with('/') { + if !glob_matcher.add(&format!("{t}**")) { + error!(output_snd, "Error in glob: {t}"); + } + } else if !t.contains('*') { + let abs_target = dir.join(Path::new(t)); + watch!(abs_target); + if abs_target.is_dir() { + if !glob_matcher.add(&format!("{t}/**")) { error!(output_snd, "Error in glob: {t}") } } else if !glob_matcher.add(t) { error!(output_snd, "Error in glob: {t}") } - }); + } else if !glob_matcher.add(t) { + error!(output_snd, "Error in glob: {t}") + } }); Ok(glob_matcher) } diff --git a/lib/tests/test_file_track_issue_104.rs b/lib/tests/test_file_track_issue_104.rs index 8dfd735bf..b7d1d46c4 100644 --- a/lib/tests/test_file_track_issue_104.rs +++ b/lib/tests/test_file_track_issue_104.rs @@ -16,7 +16,7 @@ fn create_directory_hierarchy() -> Result { Ok(temp_dir) } -/// When a directory is added to projects, its child files are also ignored. +/// When a directory is added to projects, its child files should also ignored. /// #[test] fn test_file_track_issue_104() -> Result<()> { @@ -37,6 +37,10 @@ fn test_file_track_issue_104() -> Result<()> { let root_gitignore = fs::read_to_string(xvc_root.join(Path::new(".gitignore")))?; watch!(root_gitignore); let dir_ignore = xvc_root.join(Path::new("dir-0001/.gitignore")); + watch!(dir_ignore); + + watch!(std::fs::read_dir(dir_1)?); + assert!(!dir_ignore.exists()); assert!( diff --git a/walker/src/lib.rs b/walker/src/lib.rs index 5b61f5874..c8e9ff15d 100755 --- a/walker/src/lib.rs +++ b/walker/src/lib.rs @@ -12,17 +12,17 @@ pub mod abspath; pub mod error; pub mod ignore_rules; pub mod notify; -pub mod sync; pub mod pattern; -pub mod walk_serial; +pub mod sync; pub mod walk_parallel; +pub mod walk_serial; -pub use pattern::Pattern; -pub use pattern::PatternRelativity; +pub use pattern::MatchResult; pub use pattern::PathKind; +pub use pattern::Pattern; pub use pattern::PatternEffect; +pub use pattern::PatternRelativity; pub use pattern::Source; -pub use pattern::MatchResult; pub use walk_parallel::walk_parallel; pub use walk_serial::walk_serial; @@ -115,68 +115,55 @@ impl WalkOptions { } } -/// FIXME: This should be Arc>> when Glob.is_match doesn't require mutable self -type IgnorePatterns = Vec; - /// Build the ignore rules with the given directory pub fn build_ignore_patterns( given: &str, ignore_root: &Path, ignore_filename: &str, ) -> Result { - watch!(ignore_filename); watch!(ignore_root); - let elements = ignore_root - .read_dir() - .map_err(|e| anyhow!("Error reading directory: {:?}, {:?}", ignore_root, e))?; + let ignore_rules = IgnoreRules::from_global_patterns(ignore_root, Some(ignore_filename), given); - let mut dir_stack = Vec::::new(); + let dirs_under = |p: &Path| -> Vec { + p.read_dir() + .unwrap() + .filter_map(|p| { + if let Ok(p) = p { + if p.path().is_dir() { + Some(p.path()) + } else { + None + } + } else { + None + } + }) + .filter_map(|p| match ignore_rules.check(&p) { + MatchResult::NoMatch | MatchResult::Whitelist => Some(p), + MatchResult::Ignore => None, + }) + .collect() + }; - let ignore_rules = IgnoreRules::from_global_patterns(ignore_root, Some(ignore_filename), given); + let mut dir_stack: Vec = vec![ignore_root.to_path_buf()]; let ignore_fn = ignore_rules.ignore_filename.as_deref().unwrap(); while let Some(dir) = dir_stack.pop() { - let mut new_dirs = Vec::::new(); - let mut new_patterns: Vec = Vec::new(); - let elements = dir.read_dir()?; - for entry in elements { - match entry { - Ok(entry) => { - if entry.path().is_dir() { - watch!(entry.path()); - new_dirs.push(entry.path()); - } - if entry.file_name() == ignore_fn && entry.path().exists() { - let ignore_path = entry.path(); - watch!(ignore_path); - let ignore_content = fs::read_to_string(&ignore_path)?; - new_patterns.extend( - content_to_patterns(&ignore_root, Some(&ignore_path), &ignore_content), - ); - } - } - Err(e) => { - warn!("{}", e); - } - } + watch!(dir); + let ignore_filename = dir.join(ignore_fn); + watch!(ignore_filename); + if ignore_filename.is_file() { + let ignore_content = fs::read_to_string(&ignore_filename)?; + let new_patterns = + content_to_patterns(ignore_root, Some(&ignore_filename), &ignore_content); + ignore_rules.add_patterns(new_patterns)?; + dir_stack.extend(dirs_under(&dir)); } - - ignore_rules.add_patterns(new_patterns)?; - - new_dirs.drain(0..new_dirs.len()).for_each(|new_dir| { - match ignore_rules.check(&new_dir) { - MatchResult::NoMatch | MatchResult::Whitelist => { - dir_stack.push(new_dir); - } - MatchResult::Ignore => {} - } - }); } - Ok(ignore_rules) } @@ -223,15 +210,11 @@ pub fn content_to_patterns( patterns } - -pub fn update_ignore_rules( - dir: &Path, - ignore_rules: &IgnoreRules) -> Result<()> { - - if let Some(ref ignore_filename) = ignore_rules.ignore_filename { +pub fn update_ignore_rules(dir: &Path, ignore_rules: &IgnoreRules) -> Result<()> { + if let Some(ref ignore_filename) = ignore_rules.ignore_filename { let ignore_root = &ignore_rules.root; - let ignore_path = dir.join(ignore_filename); - if ignore_path.is_file() { + let ignore_path = dir.join(ignore_filename); + if ignore_path.is_file() { let new_patterns: Vec = { let content = fs::read_to_string(&ignore_path)?; content_to_patterns(ignore_root, Some(ignore_path).as_deref(), &content) diff --git a/walker/src/walk_serial.rs b/walker/src/walk_serial.rs index c4ac49f06..bedb66b2c 100644 --- a/walker/src/walk_serial.rs +++ b/walker/src/walk_serial.rs @@ -2,7 +2,7 @@ //! See [`walk_parallel`] for parallel version. use std::path::Path; -use xvc_logging::{debug, warn, watch, XvcOutputSender}; +use xvc_logging::{debug, warn, XvcOutputSender}; use crate::{ directory_list, pattern::MatchResult, update_ignore_rules, IgnoreRules, PathMetadata, Result, From 5e2145ba62a5fb2896140f12990a7f2025621396 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Thu, 15 Aug 2024 13:03:28 +0300 Subject: [PATCH 172/257] bump version --- CHANGELOG.md | 1 + config/Cargo.toml | 6 +++--- core/Cargo.toml | 12 ++++++------ ecs/Cargo.toml | 4 ++-- file/Cargo.toml | 16 ++++++++-------- lib/Cargo.toml | 20 ++++++++++---------- logging/Cargo.toml | 2 +- pipeline/Cargo.toml | 16 ++++++++-------- storage/Cargo.toml | 14 +++++++------- test_helper/Cargo.toml | 4 ++-- walker/Cargo.toml | 6 +++--- 11 files changed, 51 insertions(+), 50 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8d087545f..e27686bfa 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,7 @@ ## Unreleased - Bump dependencies +- Replace globset with fast-glob for memory usage ## 0.6.10 (2024-08-04) diff --git a/config/Cargo.toml b/config/Cargo.toml index ff953f129..0cce62bd7 100644 --- a/config/Cargo.toml +++ b/config/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-config" -version = "0.6.11-alpha.8" +version = "0.6.11-alpha.9" edition = "2021" description = "Xvc configuration management" authors = ["Emre Şahin "] @@ -16,8 +16,8 @@ name = "xvc_config" crate-type = ["rlib"] [dependencies] -xvc-logging = { version = "0.6.11-alpha.8", path = "../logging" } -xvc-walker = { version = "0.6.11-alpha.8", path = "../walker" } +xvc-logging = { version = "0.6.11-alpha.9", path = "../logging" } +xvc-walker = { version = "0.6.11-alpha.9", path = "../walker" } ## Cli and config diff --git a/core/Cargo.toml b/core/Cargo.toml index d1600f1ed..a7b5c19a9 100644 --- a/core/Cargo.toml +++ b/core/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-core" -version = "0.6.11-alpha.8" +version = "0.6.11-alpha.9" edition = "2021" description = "Xvc core for common elements for all commands" authors = ["Emre Şahin "] @@ -16,10 +16,10 @@ name = "xvc_core" crate-type = ["rlib"] [dependencies] -xvc-config = { version = "0.6.11-alpha.8", path = "../config" } -xvc-logging = { version = "0.6.11-alpha.8", path = "../logging" } -xvc-ecs = { version = "0.6.11-alpha.8", path = "../ecs" } -xvc-walker = { version = "0.6.11-alpha.8", path = "../walker" } +xvc-config = { version = "0.6.11-alpha.9", path = "../config" } +xvc-logging = { version = "0.6.11-alpha.9", path = "../logging" } +xvc-ecs = { version = "0.6.11-alpha.9", path = "../ecs" } +xvc-walker = { version = "0.6.11-alpha.9", path = "../walker" } ## Cli and config clap = { version = "^4.5", features = ["derive"] } @@ -81,6 +81,6 @@ itertools = "^0.13" [dev-dependencies] -xvc-test-helper = { version = "0.6.11-alpha.8", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.11-alpha.9", path = "../test_helper/" } proptest = "^1.5" test-case = "^3.3" diff --git a/ecs/Cargo.toml b/ecs/Cargo.toml index 9eafda11f..2bde7192b 100644 --- a/ecs/Cargo.toml +++ b/ecs/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-ecs" -version = "0.6.11-alpha.8" +version = "0.6.11-alpha.9" edition = "2021" description = "Entity-Component System for Xvc" authors = ["Emre Şahin "] @@ -16,7 +16,7 @@ name = "xvc_ecs" crate-type = ["rlib"] [dependencies] -xvc-logging = { version = "0.6.11-alpha.8", path = "../logging" } +xvc-logging = { version = "0.6.11-alpha.9", path = "../logging" } ## Serialization serde = { version = "^1.0", features = ["derive"] } diff --git a/file/Cargo.toml b/file/Cargo.toml index 24e3a3fe9..9e0b8190f 100644 --- a/file/Cargo.toml +++ b/file/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-file" -version = "0.6.11-alpha.8" +version = "0.6.11-alpha.9" edition = "2021" description = "File tracking, versioning, upload and download functions for Xvc" authors = ["Emre Şahin "] @@ -21,12 +21,12 @@ test = true bench = true [dependencies] -xvc-logging = { version = "0.6.11-alpha.8", path = "../logging" } -xvc-config = { version = "0.6.11-alpha.8", path = "../config" } -xvc-core = { version = "0.6.11-alpha.8", path = "../core" } -xvc-ecs = { version = "0.6.11-alpha.8", path = "../ecs" } -xvc-walker = { version = "0.6.11-alpha.8", path = "../walker" } -xvc-storage = { version = "0.6.11-alpha.8", path = "../storage", default-features = false } +xvc-logging = { version = "0.6.11-alpha.9", path = "../logging" } +xvc-config = { version = "0.6.11-alpha.9", path = "../config" } +xvc-core = { version = "0.6.11-alpha.9", path = "../core" } +xvc-ecs = { version = "0.6.11-alpha.9", path = "../ecs" } +xvc-walker = { version = "0.6.11-alpha.9", path = "../walker" } +xvc-storage = { version = "0.6.11-alpha.9", path = "../storage", default-features = false } ## Cli and config @@ -95,5 +95,5 @@ default = ["reflink"] reflink = ["dep:reflink"] [dev-dependencies] -xvc-test-helper = { version = "0.6.11-alpha.8", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.11-alpha.9", path = "../test_helper/" } shellfn = "^0.1" diff --git a/lib/Cargo.toml b/lib/Cargo.toml index 2b74501a8..536297e0b 100644 --- a/lib/Cargo.toml +++ b/lib/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc" -version = "0.6.11-alpha.8" +version = "0.6.11-alpha.9" edition = "2021" description = "An MLOps tool to manage data files and pipelines on top of Git" authors = ["Emre Şahin "] @@ -20,14 +20,14 @@ name = "xvc" path = "src/main.rs" [dependencies] -xvc-config = { version = "0.6.11-alpha.8", path = "../config" } -xvc-core = { version = "0.6.11-alpha.8", path = "../core" } -xvc-logging = { version = "0.6.11-alpha.8", path = "../logging" } -xvc-ecs = { version = "0.6.11-alpha.8", path = "../ecs" } -xvc-file = { version = "0.6.11-alpha.8", path = "../file", default-features = false } -xvc-pipeline = { version = "0.6.11-alpha.8", path = "../pipeline" } -xvc-walker = { version = "0.6.11-alpha.8", path = "../walker" } -xvc-storage = { version = "0.6.11-alpha.8", path = "../storage", default-features = false } +xvc-config = { version = "0.6.11-alpha.9", path = "../config" } +xvc-core = { version = "0.6.11-alpha.9", path = "../core" } +xvc-logging = { version = "0.6.11-alpha.9", path = "../logging" } +xvc-ecs = { version = "0.6.11-alpha.9", path = "../ecs" } +xvc-file = { version = "0.6.11-alpha.9", path = "../file", default-features = false } +xvc-pipeline = { version = "0.6.11-alpha.9", path = "../pipeline" } +xvc-walker = { version = "0.6.11-alpha.9", path = "../walker" } +xvc-storage = { version = "0.6.11-alpha.9", path = "../storage", default-features = false } ## Cli and config clap = { version = "^4.5", features = ["derive", "cargo"] } @@ -133,4 +133,4 @@ shellfn = "^0.1" test-case = "^3.3" trycmd = "^0.15" which = "^6.0" -xvc-test-helper = { version = "0.6.11-alpha.8", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.11-alpha.9", path = "../test_helper/" } diff --git a/logging/Cargo.toml b/logging/Cargo.toml index 6f935ea75..60eec1622 100644 --- a/logging/Cargo.toml +++ b/logging/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-logging" -version = "0.6.11-alpha.8" +version = "0.6.11-alpha.9" edition = "2021" description = "Logging crate for Xvc" authors = ["Emre Şahin "] diff --git a/pipeline/Cargo.toml b/pipeline/Cargo.toml index 2dcfa5bf4..8b0ba7acf 100644 --- a/pipeline/Cargo.toml +++ b/pipeline/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-pipeline" -version = "0.6.11-alpha.8" +version = "0.6.11-alpha.9" edition = "2021" description = "Xvc data pipeline management" authors = ["Emre Şahin "] @@ -19,12 +19,12 @@ default = [] bundled-sqlite = ["rusqlite/bundled"] [dependencies] -xvc-config = { version = "0.6.11-alpha.8", path = "../config" } -xvc-core = { version = "0.6.11-alpha.8", path = "../core" } -xvc-ecs = { version = "0.6.11-alpha.8", path = "../ecs" } -xvc-logging = { version = "0.6.11-alpha.8", path = "../logging" } -xvc-walker = { version = "0.6.11-alpha.8", path = "../walker" } -xvc-file = { version = "0.6.11-alpha.8", path = "../file", default-features = false } +xvc-config = { version = "0.6.11-alpha.9", path = "../config" } +xvc-core = { version = "0.6.11-alpha.9", path = "../core" } +xvc-ecs = { version = "0.6.11-alpha.9", path = "../ecs" } +xvc-logging = { version = "0.6.11-alpha.9", path = "../logging" } +xvc-walker = { version = "0.6.11-alpha.9", path = "../walker" } +xvc-file = { version = "0.6.11-alpha.9", path = "../file", default-features = false } ## Cli and config clap = { version = "^4.5", features = ["derive"] } @@ -98,5 +98,5 @@ itertools = "^0.13" derive_more = "^0.99" [dev-dependencies] -xvc-test-helper = { version = "0.6.11-alpha.8", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.11-alpha.9", path = "../test_helper/" } test-case = "^3.3" diff --git a/storage/Cargo.toml b/storage/Cargo.toml index 0bbc85d77..8224a4b6d 100644 --- a/storage/Cargo.toml +++ b/storage/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-storage" -version = "0.6.11-alpha.8" +version = "0.6.11-alpha.9" edition = "2021" description = "Xvc remote and local storage management" authors = ["Emre Şahin "] @@ -16,11 +16,11 @@ name = "xvc_storage" crate-type = ["rlib"] [dependencies] -xvc-logging = { version = "0.6.11-alpha.8", path = "../logging" } -xvc-config = { version = "0.6.11-alpha.8", path = "../config" } -xvc-core = { version = "0.6.11-alpha.8", path = "../core" } -xvc-ecs = { version = "0.6.11-alpha.8", path = "../ecs" } -xvc-walker = { version = "0.6.11-alpha.8", path = "../walker" } +xvc-logging = { version = "0.6.11-alpha.9", path = "../logging" } +xvc-config = { version = "0.6.11-alpha.9", path = "../config" } +xvc-core = { version = "0.6.11-alpha.9", path = "../core" } +xvc-ecs = { version = "0.6.11-alpha.9", path = "../ecs" } +xvc-walker = { version = "0.6.11-alpha.9", path = "../walker" } ## Cli and config clap = { version = "^4.5", features = ["derive"] } @@ -104,7 +104,7 @@ bundled-openssl = ["openssl/vendored"] [dev-dependencies] -xvc-test-helper = { version = "0.6.11-alpha.8", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.11-alpha.9", path = "../test_helper/" } shellfn = "^0.1" [package.metadata.cargo-udeps.ignore] diff --git a/test_helper/Cargo.toml b/test_helper/Cargo.toml index a2e15d3cf..c8994a3b0 100644 --- a/test_helper/Cargo.toml +++ b/test_helper/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-test-helper" -version = "0.6.11-alpha.8" +version = "0.6.11-alpha.9" edition = "2021" description = "Unit test helper functions for Xvc" authors = ["Emre Şahin "] @@ -20,7 +20,7 @@ path = "src/main.rs" # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html [dependencies] -xvc-logging = { version = "0.6.11-alpha.8", path = "../logging/" } +xvc-logging = { version = "0.6.11-alpha.9", path = "../logging/" } rand = "^0.8" log = "^0.4" diff --git a/walker/Cargo.toml b/walker/Cargo.toml index ed88f5426..016b7b5c9 100644 --- a/walker/Cargo.toml +++ b/walker/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-walker" -version = "0.6.11-alpha.8" +version = "0.6.11-alpha.9" edition = "2021" description = "Xvc parallel file system walker with ignore features" authors = ["Emre Şahin "] @@ -16,7 +16,7 @@ name = "xvc_walker" crate-type = ["rlib"] [dependencies] -xvc-logging = { version = "0.6.11-alpha.8", path = "../logging" } +xvc-logging = { version = "0.6.11-alpha.9", path = "../logging" } fast-glob = "^0.3" ## Parallelization @@ -41,7 +41,7 @@ itertools = "^0.13" regex = "^1.10" [dev-dependencies] -xvc-test-helper = { path = "../test_helper/", version = "0.6.11-alpha.8" } +xvc-test-helper = { path = "../test_helper/", version = "0.6.11-alpha.9" } test-case = "^3.3" [package.metadata.cargo-udeps.ignore] From df81ce084797cdfc7264efbf683c7059fe1f2ce3 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Thu, 15 Aug 2024 22:18:33 +0300 Subject: [PATCH 173/257] Fix test_walker_parallel --- lib/tests/test_walker_parallel.rs | 19 +++++++++---------- 1 file changed, 9 insertions(+), 10 deletions(-) diff --git a/lib/tests/test_walker_parallel.rs b/lib/tests/test_walker_parallel.rs index 720deda0f..548bbd902 100644 --- a/lib/tests/test_walker_parallel.rs +++ b/lib/tests/test_walker_parallel.rs @@ -1,5 +1,8 @@ use std::{ - ffi::OsString, fs, path::{Path, PathBuf}, sync::{Arc, RwLock} + ffi::OsString, + fs, + path::{Path, PathBuf}, + sync::{Arc, RwLock}, }; use fast_glob::Glob; @@ -44,7 +47,7 @@ fn new_dir_with_ignores( initial_patterns: &str, ) -> Result { let patterns = create_patterns(root, dir, initial_patterns); - let initialized = IgnoreRules::empty(&PathBuf::from(root), None); + let initialized = IgnoreRules::empty(&PathBuf::from(root), Some(".gitignore")); watch!(patterns); initialized.add_patterns(patterns).unwrap(); Ok(initialized) @@ -78,18 +81,14 @@ fn test_walk_parallel(ignore_src: &str, ignore_content: &str) -> Vec { ignore_content, ) .unwrap(); - let initial_rules = Arc::new(RwLock::new(new_dir_with_ignores(root.to_string_lossy().as_ref(), None, "").unwrap())); + let initial_rules = Arc::new(RwLock::new( + new_dir_with_ignores(root.to_string_lossy().as_ref(), None, "").unwrap(), + )); let walk_options = WalkOptions { ignore_filename: Some(".gitignore".to_owned()), include_dirs: true, }; - walk_parallel( - initial_rules, - &root, - walk_options, - path_sender, - ) - .unwrap(); + walk_parallel(initial_rules, &root, walk_options, path_sender).unwrap(); let paths = path_receiver .iter() .filter_map(|e| match e { From decef74b162d142978948396860865dc67f171a8 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sat, 24 Aug 2024 18:50:59 +0300 Subject: [PATCH 174/257] added symlink check --- walker/src/walk_parallel.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/walker/src/walk_parallel.rs b/walker/src/walk_parallel.rs index a3d94d7f9..6b921ba2c 100644 --- a/walker/src/walk_parallel.rs +++ b/walker/src/walk_parallel.rs @@ -43,7 +43,7 @@ fn walk_parallel_inner( // If the path is a directory, send it to the channel if `include_dirs` is true. // The caller expects a list of directories to recurse into. - if pm.metadata.is_file() { + if pm.metadata.is_file() || pm.metadata.is_symlink() { path_sender .send(Ok(pm.clone())) .expect("Channel error in walk_parallel"); From 62b7eaef590a6b405e4a80277cf2cc88212210c8 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sat, 24 Aug 2024 19:01:43 +0300 Subject: [PATCH 175/257] debug --- book/src/ref/xvc-file-list.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/book/src/ref/xvc-file-list.md b/book/src/ref/xvc-file-list.md index 1df07397d..2dec50657 100644 --- a/book/src/ref/xvc-file-list.md +++ b/book/src/ref/xvc-file-list.md @@ -278,7 +278,7 @@ print the second letter as `H`. ```console $ xvc file track dir-0002 --recheck-method hardlink -$ xvc file list dir-0002 +$ xvc -vvvv file list dir-0002 FH 2005 [..] 447933dc 447933dc dir-0002/file-0005.bin FH 2004 [..] 63535612 63535612 dir-0002/file-0004.bin FH 2003 [..] d2432259 d2432259 dir-0002/file-0003.bin From 5110d002b471291c70aac35b5583e2c498188187 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sat, 24 Aug 2024 19:05:21 +0300 Subject: [PATCH 176/257] final / --- book/src/ref/xvc-file-list.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/book/src/ref/xvc-file-list.md b/book/src/ref/xvc-file-list.md index 2dec50657..1b20d0d6c 100644 --- a/book/src/ref/xvc-file-list.md +++ b/book/src/ref/xvc-file-list.md @@ -276,7 +276,7 @@ If you add another set of files as hardlinks to the cached copies, it will print the second letter as `H`. ```console -$ xvc file track dir-0002 --recheck-method hardlink +$ xvc file track dir-0002/ --recheck-method hardlink $ xvc -vvvv file list dir-0002 FH 2005 [..] 447933dc 447933dc dir-0002/file-0005.bin From 0ac2237bfce0523798607e94113f00f7f416be11 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sat, 24 Aug 2024 19:21:29 +0300 Subject: [PATCH 177/257] recheck --- book/src/ref/xvc-file-list.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/book/src/ref/xvc-file-list.md b/book/src/ref/xvc-file-list.md index 1b20d0d6c..1f02db68c 100644 --- a/book/src/ref/xvc-file-list.md +++ b/book/src/ref/xvc-file-list.md @@ -276,7 +276,7 @@ If you add another set of files as hardlinks to the cached copies, it will print the second letter as `H`. ```console -$ xvc file track dir-0002/ --recheck-method hardlink +$ xvc -vvvv file track dir-0002/ --recheck-method hardlink $ xvc -vvvv file list dir-0002 FH 2005 [..] 447933dc 447933dc dir-0002/file-0005.bin From 3d8eebda62cbad4fa0ba2be466a826c1f781edd9 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sat, 24 Aug 2024 19:51:47 +0300 Subject: [PATCH 178/257] debug --- book/src/ref/xvc-file-list.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/book/src/ref/xvc-file-list.md b/book/src/ref/xvc-file-list.md index 1f02db68c..1b20d0d6c 100644 --- a/book/src/ref/xvc-file-list.md +++ b/book/src/ref/xvc-file-list.md @@ -276,7 +276,7 @@ If you add another set of files as hardlinks to the cached copies, it will print the second letter as `H`. ```console -$ xvc -vvvv file track dir-0002/ --recheck-method hardlink +$ xvc file track dir-0002/ --recheck-method hardlink $ xvc -vvvv file list dir-0002 FH 2005 [..] 447933dc 447933dc dir-0002/file-0005.bin From 75802ecafa7d6f194fa21cb3e2d89bc44d0cd265 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sat, 24 Aug 2024 20:14:42 +0300 Subject: [PATCH 179/257] update store filtering --- file/src/common/mod.rs | 33 ++++++++++++++++----------------- 1 file changed, 16 insertions(+), 17 deletions(-) diff --git a/file/src/common/mod.rs b/file/src/common/mod.rs index 3ed1779e7..ba9fc4cb2 100644 --- a/file/src/common/mod.rs +++ b/file/src/common/mod.rs @@ -93,12 +93,14 @@ pub fn pipe_path_digest( /// /// If `targets` is `None`, all paths in the store are returned. pub fn load_targets_from_store( + output_snd: &XvcOutputSender, xvc_root: &XvcRoot, current_dir: &AbsolutePath, targets: &Option>, ) -> Result> { let store: XvcStore = xvc_root.load_store()?; - filter_targets_from_store(xvc_root, &store, current_dir, targets) + watch!(store); + filter_targets_from_store(output_snd, xvc_root, &store, current_dir, targets) } /// Filters the paths in the store by given globs. @@ -107,6 +109,7 @@ pub fn load_targets_from_store( /// /// If `current_dir` is not the root, all targets are prefixed with it. pub fn filter_targets_from_store( + output_snd: &XvcOutputSender, xvc_root: &XvcRoot, store: &XvcStore, current_dir: &AbsolutePath, @@ -124,6 +127,7 @@ pub fn filter_targets_from_store( }; return filter_targets_from_store( + output_snd, xvc_root, store, xvc_root.absolute_path(), @@ -135,7 +139,7 @@ pub fn filter_targets_from_store( let hstore = HStore::::from(store); if let Some(targets) = targets { - let paths = filter_paths_by_globs(&hstore, targets.as_slice())?; + let paths = filter_paths_by_globs(output_snd, xvc_root, &hstore, targets.as_slice())?; watch!(paths); Ok(paths) } else { @@ -147,35 +151,30 @@ pub fn filter_targets_from_store( /// GlobSet and paths are checked against the set. /// /// If a target ends with /, it's considered a directory and all its children are also selected. -pub fn filter_paths_by_globs(paths: &HStore, globs: &[String]) -> Result> { +pub fn filter_paths_by_globs( + output_snd: &XvcOutputSender, + xvc_root: &XvcRoot, + paths: &HStore, + globs: &[String], +) -> Result> { if globs.is_empty() { return Ok(paths.to_owned()); } - let mut glob_matcher = Glob::default(); - globs.iter().for_each(|t| { - watch!(t); - if t.ends_with('/') { - glob_matcher.add(&format!("{t}**")); - } else { - glob_matcher.add(t); - } - }); - + let mut glob_matcher = build_glob_matcher(output_snd, xvc_root, globs)?; + watch!(glob_matcher); let paths = paths .iter() .filter_map(|(e, p)| { - let str_path = &p.as_relative_path().as_str(); - - if glob_matcher.is_match(str_path) { + if glob_matcher.is_match(p.as_str()) { Some((*e, p.clone())) } else { None } }) .collect(); - watch!(paths); + watch!(paths); Ok(paths) } From db4a27ddbcc4f8cd6430c6f6e9a668248a92f277 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sat, 24 Aug 2024 20:18:17 +0300 Subject: [PATCH 180/257] update sender --- file/src/bring/mod.rs | 2 +- file/src/carry_in/mod.rs | 2 +- file/src/copy/mod.rs | 3 +++ file/src/list/mod.rs | 4 +++- file/src/mv/mod.rs | 1 + file/src/recheck/mod.rs | 2 +- file/src/remove/mod.rs | 9 +++++++-- file/src/send/mod.rs | 2 +- file/src/share/mod.rs | 3 ++- file/src/untrack/mod.rs | 9 +++++++-- 10 files changed, 27 insertions(+), 10 deletions(-) diff --git a/file/src/bring/mod.rs b/file/src/bring/mod.rs index b3a90c87f..5cdb51075 100644 --- a/file/src/bring/mod.rs +++ b/file/src/bring/mod.rs @@ -65,7 +65,7 @@ pub fn fetch(output_snd: &XvcOutputSender, xvc_root: &XvcRoot, opts: &BringCLI) let storage = get_storage_record(output_snd, xvc_root, &opts.storage)?; let current_dir = xvc_root.config().current_dir()?; - let targets = load_targets_from_store(xvc_root, current_dir, &opts.targets)?; + let targets = load_targets_from_store(output_snd, xvc_root, current_dir, &opts.targets)?; let force = opts.force; watch!(targets); diff --git a/file/src/carry_in/mod.rs b/file/src/carry_in/mod.rs index 46ce65e06..b14931a7f 100644 --- a/file/src/carry_in/mod.rs +++ b/file/src/carry_in/mod.rs @@ -113,7 +113,7 @@ pub fn cmd_carry_in( let opts = cli_opts.update_from_conf(conf)?; watch!(opts); let current_dir = conf.current_dir()?; - let targets = load_targets_from_store(xvc_root, current_dir, &opts.targets)?; + let targets = load_targets_from_store(output_snd, xvc_root, current_dir, &opts.targets)?; watch!(targets); let stored_xvc_path_store = xvc_root.load_store::()?; diff --git a/file/src/copy/mod.rs b/file/src/copy/mod.rs index 9bfb48461..a12bc326c 100644 --- a/file/src/copy/mod.rs +++ b/file/src/copy/mod.rs @@ -60,6 +60,7 @@ pub struct CopyCLI { } pub(crate) fn get_source_path_metadata( + output_snd: &XvcOutputSender, xvc_root: &XvcRoot, stored_xvc_path_store: &XvcStore, stored_xvc_metadata_store: &XvcStore, @@ -76,6 +77,7 @@ pub(crate) fn get_source_path_metadata( let current_dir = xvc_root.config().current_dir()?; let all_sources = filter_targets_from_store( + output_snd, xvc_root, stored_xvc_path_store, current_dir, @@ -327,6 +329,7 @@ pub fn cmd_copy(output_snd: &XvcOutputSender, xvc_root: &XvcRoot, opts: CopyCLI) let stored_metadata_store = xvc_root.load_store::()?; let stored_xvc_path_store = xvc_root.load_store::()?; let (source_xvc_paths, source_metadata) = get_source_path_metadata( + output_snd, xvc_root, &stored_xvc_path_store, &stored_metadata_store, diff --git a/file/src/list/mod.rs b/file/src/list/mod.rs index 605647f13..779ef4c12 100644 --- a/file/src/list/mod.rs +++ b/file/src/list/mod.rs @@ -530,6 +530,8 @@ pub fn cmd_list(output_snd: &XvcOutputSender, xvc_root: &XvcRoot, cli_opts: List let current_dir = conf.current_dir()?; + // If targets are directories on disk, make sure they end with / + let all_from_disk = targets_from_disk(output_snd, xvc_root, current_dir, &opts.targets)?; watch!(&all_from_disk); let from_disk = if opts.show_dot_files { @@ -549,7 +551,7 @@ pub fn cmd_list(output_snd: &XvcOutputSender, xvc_root: &XvcRoot, cli_opts: List }; watch!(from_disk); - let from_store = load_targets_from_store(xvc_root, current_dir, &opts.targets)?; + let from_store = load_targets_from_store(output_snd, xvc_root, current_dir, &opts.targets)?; watch!(from_store); let stored_xvc_metadata = xvc_root.load_store::()?; let stored_recheck_method = xvc_root.load_store::()?; diff --git a/file/src/mv/mod.rs b/file/src/mv/mod.rs index b1c87a107..acc9cdcb7 100644 --- a/file/src/mv/mod.rs +++ b/file/src/mv/mod.rs @@ -163,6 +163,7 @@ pub fn cmd_move(output_snd: &XvcOutputSender, xvc_root: &XvcRoot, opts: MoveCLI) let stored_metadata_store = xvc_root.load_store::()?; let stored_xvc_path_store = xvc_root.load_store::()?; let (source_xvc_paths, source_metadata) = get_source_path_metadata( + output_snd, xvc_root, &stored_xvc_path_store, &stored_metadata_store, diff --git a/file/src/recheck/mod.rs b/file/src/recheck/mod.rs index 399a5f863..f5046f35a 100644 --- a/file/src/recheck/mod.rs +++ b/file/src/recheck/mod.rs @@ -94,7 +94,7 @@ pub fn cmd_recheck( let opts = cli_opts.update_from_conf(conf)?; let current_dir = conf.current_dir()?; - let targets = load_targets_from_store(xvc_root, current_dir, &opts.targets)?; + let targets = load_targets_from_store(output_snd, xvc_root, current_dir, &opts.targets)?; watch!(targets); let stored_xvc_path_store = xvc_root.load_store::()?; diff --git a/file/src/remove/mod.rs b/file/src/remove/mod.rs index f4699aaa4..3a3e00b94 100644 --- a/file/src/remove/mod.rs +++ b/file/src/remove/mod.rs @@ -68,8 +68,13 @@ pub fn cmd_remove(output_snd: &XvcOutputSender, xvc_root: &XvcRoot, opts: Remove let all_paths = xvc_root.load_store()?; let all_content_digests = xvc_root.load_store()?; - let remove_targets = - filter_targets_from_store(xvc_root, &all_paths, current_dir, &Some(opts.targets))?; + let remove_targets = filter_targets_from_store( + output_snd, + xvc_root, + &all_paths, + current_dir, + &Some(opts.targets), + )?; let all_cache_paths = cache_paths_for_xvc_paths(output_snd, &all_paths, &all_content_digests)?; diff --git a/file/src/send/mod.rs b/file/src/send/mod.rs index 6c71246a6..2fef1700f 100644 --- a/file/src/send/mod.rs +++ b/file/src/send/mod.rs @@ -37,7 +37,7 @@ pub fn cmd_send(output_snd: &XvcOutputSender, xvc_root: &XvcRoot, opts: SendCLI) let remote = get_storage_record(output_snd, xvc_root, &opts.storage)?; watch!(remote); let current_dir = xvc_root.config().current_dir()?; - let targets = load_targets_from_store(xvc_root, current_dir, &opts.targets)?; + let targets = load_targets_from_store(output_snd, xvc_root, current_dir, &opts.targets)?; watch!(targets); let target_file_xvc_metadata = xvc_root diff --git a/file/src/share/mod.rs b/file/src/share/mod.rs index 8079b32e9..f7ac131da 100644 --- a/file/src/share/mod.rs +++ b/file/src/share/mod.rs @@ -33,7 +33,8 @@ pub fn cmd_share(output_snd: &XvcOutputSender, xvc_root: &XvcRoot, opts: ShareCL let storage = get_storage_record(output_snd, xvc_root, &opts.storage)?; watch!(storage); let current_dir = xvc_root.config().current_dir()?; - let targets = load_targets_from_store(xvc_root, current_dir, &Some(vec![opts.target]))?; + let targets = + load_targets_from_store(output_snd, xvc_root, current_dir, &Some(vec![opts.target]))?; watch!(targets); let target_file_xvc_metadata = xvc_root diff --git a/file/src/untrack/mod.rs b/file/src/untrack/mod.rs index 0e7c45a66..e2081cff6 100644 --- a/file/src/untrack/mod.rs +++ b/file/src/untrack/mod.rs @@ -43,8 +43,13 @@ pub fn cmd_untrack( let current_dir = xvc_root.config().current_dir()?; let all_paths = xvc_root.load_store()?; let all_content_digests = xvc_root.load_store()?; - let untrack_targets = - filter_targets_from_store(xvc_root, &all_paths, current_dir, &Some(opts.targets))?; + let untrack_targets = filter_targets_from_store( + output_snd, + xvc_root, + &all_paths, + current_dir, + &Some(opts.targets), + )?; let all_cache_paths = cache_paths_for_xvc_paths(output_snd, &all_paths, &all_content_digests)?; From f6e1c6710d7f4233d04643c5d057fccf6541e50e Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sat, 24 Aug 2024 20:25:20 +0300 Subject: [PATCH 181/257] list rerun --- book/src/ref/xvc-file-list.md | 23 +++++++++++------------ 1 file changed, 11 insertions(+), 12 deletions(-) diff --git a/book/src/ref/xvc-file-list.md b/book/src/ref/xvc-file-list.md index 1b20d0d6c..1dfe7b75d 100644 --- a/book/src/ref/xvc-file-list.md +++ b/book/src/ref/xvc-file-list.md @@ -11,15 +11,15 @@ Usage: xvc file list [OPTIONS] [TARGETS]... Arguments: [TARGETS]... Files/directories to list. - + If not supplied, lists all files under the current directory. Options: -f, --format A string for each row of the output table - + The following are the keys for each row: - + - {{acd8}}: actual content digest from the workspace file. First 8 digits. - {{acd64}}: actual content digest. All 64 digits. - {{aft}}: actual file type. Whether the entry is a file (F), directory (D), @@ -38,24 +38,24 @@ Options: - {{rsz}}: recorded size. The size of the cached content in bytes. It uses MB, GB and TB to represent sizes larged than 1MB. - {{rts}}: recorded timestamp. The timestamp of the cached content. - + The default format can be set with file.list.format in the config file. -s, --sort Sort criteria. - + It can be one of none (default), name-asc, name-desc, size-asc, size-desc, ts-asc, ts-desc. - + The default option can be set with file.list.sort in the config file. --no-summary Don't show total number and size of the listed files. - + The default option can be set with file.list.no_summary in the config file. -a, --show-dot-files Don't hide dot files - + If not supplied, hides dot files like .gitignore and .xvcignore -h, --help @@ -168,7 +168,7 @@ Total #: 30 Workspace Size: 51195 Cached Size: 0 ``` -By default the command hides dotfiles. If you also want to show them, you can use `--show-dot-files`/`-a` flag. +By default the command hides dotfiles. If you also want to show them, you can use `--show-dot-files`/`-a` flag. ```console $ xvc file list --sort name-asc --show-dot-files @@ -209,7 +209,7 @@ Total #: 32 Workspace Size: 51443 Cached Size: 0 ``` -You can also hide the summary below the list to get only the list of files. +You can also hide the summary below the list to get only the list of files. ```console $ xvc file list --sort name-asc --no-summary @@ -278,7 +278,7 @@ print the second letter as `H`. ```console $ xvc file track dir-0002/ --recheck-method hardlink -$ xvc -vvvv file list dir-0002 +$ xvc file list dir-0002 FH 2005 [..] 447933dc 447933dc dir-0002/file-0005.bin FH 2004 [..] 63535612 63535612 dir-0002/file-0004.bin FH 2003 [..] d2432259 d2432259 dir-0002/file-0003.bin @@ -286,7 +286,6 @@ FH 2002 [..] 7e807161 7e807161 dir-0002/file-0002.bin FH 2001 [..] 1953f05d 1953f05d dir-0002/file-0001.bin Total #: 5 Workspace Size: 10015 Cached Size: 10015 - ``` Note, as hardlinks are files with the same inode in the file system From 30015f4354894c58fdd1aa089bd1e37ab1969d2b Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sat, 24 Aug 2024 20:26:22 +0300 Subject: [PATCH 182/257] Update --- book/src/ref/xvc-file-list.md | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/book/src/ref/xvc-file-list.md b/book/src/ref/xvc-file-list.md index 1dfe7b75d..5b415b7e2 100644 --- a/book/src/ref/xvc-file-list.md +++ b/book/src/ref/xvc-file-list.md @@ -11,15 +11,15 @@ Usage: xvc file list [OPTIONS] [TARGETS]... Arguments: [TARGETS]... Files/directories to list. - + If not supplied, lists all files under the current directory. Options: -f, --format A string for each row of the output table - + The following are the keys for each row: - + - {{acd8}}: actual content digest from the workspace file. First 8 digits. - {{acd64}}: actual content digest. All 64 digits. - {{aft}}: actual file type. Whether the entry is a file (F), directory (D), @@ -38,24 +38,24 @@ Options: - {{rsz}}: recorded size. The size of the cached content in bytes. It uses MB, GB and TB to represent sizes larged than 1MB. - {{rts}}: recorded timestamp. The timestamp of the cached content. - + The default format can be set with file.list.format in the config file. -s, --sort Sort criteria. - + It can be one of none (default), name-asc, name-desc, size-asc, size-desc, ts-asc, ts-desc. - + The default option can be set with file.list.sort in the config file. --no-summary Don't show total number and size of the listed files. - + The default option can be set with file.list.no_summary in the config file. -a, --show-dot-files Don't hide dot files - + If not supplied, hides dot files like .gitignore and .xvcignore -h, --help @@ -286,6 +286,7 @@ FH 2002 [..] 7e807161 7e807161 dir-0002/file-0002.bin FH 2001 [..] 1953f05d 1953f05d dir-0002/file-0001.bin Total #: 5 Workspace Size: 10015 Cached Size: 10015 + ``` Note, as hardlinks are files with the same inode in the file system From ed76f8b3b0b20e1f39a0672485db9f71ef43cd62 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sat, 24 Aug 2024 20:41:09 +0300 Subject: [PATCH 183/257] update --- book/src/ref/xvc-check-ignore.md | 23 ++++++++++++++++++----- 1 file changed, 18 insertions(+), 5 deletions(-) diff --git a/book/src/ref/xvc-check-ignore.md b/book/src/ref/xvc-check-ignore.md index 7f0366007..be2b65047 100644 --- a/book/src/ref/xvc-check-ignore.md +++ b/book/src/ref/xvc-check-ignore.md @@ -39,21 +39,33 @@ Options: By default it checks the files supplied from `stdin`. -```shell +```console +$ git init +... +$ xvc init +``` + +```console +$ cat .xvcignore +``` + +```console $ xvc check-ignore -my-dir/my-file + ``` If you supply paths from the CLI, they are checked instead. -```shell +```console $ xvc check-ignore my-dir/my-file another-dir/another-file + ``` If you're looking which `.xvcignore` file ignores (or whitelists) a certain path, you can use `--details`. -```shell +```console $ xvc check-ignore --details my-dir/my-file another-dir/another-file + ``` `.xvcignore` file format is identical to [`.gitignore` file format](https://git-scm.com/docs/gitignore). @@ -61,6 +73,7 @@ This utility can be used to check any other ignore rules in other files as well. You can specify an alternative ignore filename with `--ignore-filename` option. The below command is identical to `git check-ignore` and should give the same results. -```shell +```console $ xvc check-ignore --ignore-filename .gitignore + ``` From 9e1ee9a2d4cb5ae68f6ff30ad8d267fc7bc93e93 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sat, 24 Aug 2024 21:05:13 +0300 Subject: [PATCH 184/257] add patterns to xvcignore --- book/src/ref/xvc-check-ignore.md | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/book/src/ref/xvc-check-ignore.md b/book/src/ref/xvc-check-ignore.md index be2b65047..f8d2fa182 100644 --- a/book/src/ref/xvc-check-ignore.md +++ b/book/src/ref/xvc-check-ignore.md @@ -46,15 +46,20 @@ $ xvc init ``` ```console -$ cat .xvcignore +$ zsh -cl "echo 'my-dir/my-file' > .xvcignore" ``` ```console -$ xvc check-ignore + +# Add patterns of files xvc should ignore, which could improve +# the performance. +# It's in the same format as .gitignore files. + +.DS_Store ``` -If you supply paths from the CLI, they are checked instead. +If you supply paths from the CLI, they are checked against the ignore rules in `.xvcignore`. ```console $ xvc check-ignore my-dir/my-file another-dir/another-file From 5228f720bb22627c012d48014dafcbf6f257c5bc Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Mon, 26 Aug 2024 11:52:09 +0300 Subject: [PATCH 185/257] check ignore --- book/src/ref/xvc-check-ignore.md | 11 ++++++++--- core/src/check_ignore/mod.rs | 34 +++++++++++++++++++------------- lib/src/cli/mod.rs | 31 +++++++++++------------------ 3 files changed, 40 insertions(+), 36 deletions(-) diff --git a/book/src/ref/xvc-check-ignore.md b/book/src/ref/xvc-check-ignore.md index f8d2fa182..44462bf0a 100644 --- a/book/src/ref/xvc-check-ignore.md +++ b/book/src/ref/xvc-check-ignore.md @@ -37,18 +37,24 @@ Options: ## Examples -By default it checks the files supplied from `stdin`. - ```console $ git init ... $ xvc init ``` +The `.xvcignore` file format is identical to [`.gitignore` file format](https://git-scm.com/docs/gitignore). + ```console $ zsh -cl "echo 'my-dir/my-file' > .xvcignore" ``` +By default it checks the files supplied from `stdin`. + +```console +$ echo my-dir/my-file | xvc check-ignore +``` + ```console # Add patterns of files xvc should ignore, which could improve @@ -73,7 +79,6 @@ $ xvc check-ignore --details my-dir/my-file another-dir/another-file ``` -`.xvcignore` file format is identical to [`.gitignore` file format](https://git-scm.com/docs/gitignore). This utility can be used to check any other ignore rules in other files as well. You can specify an alternative ignore filename with `--ignore-filename` option. The below command is identical to `git check-ignore` and should give the same results. diff --git a/core/src/check_ignore/mod.rs b/core/src/check_ignore/mod.rs index d4b9215c1..86811c49f 100644 --- a/core/src/check_ignore/mod.rs +++ b/core/src/check_ignore/mod.rs @@ -9,7 +9,7 @@ use log::trace; use std::io::BufRead; use std::path::{Path, PathBuf}; use xvc_config::{UpdateFromXvcConfig, XvcConfig}; -use xvc_logging::{output, XvcOutputSender}; +use xvc_logging::{output, watch, XvcOutputSender}; use xvc_walker::{build_ignore_patterns, IgnoreRules, MatchResult, WalkOptions}; // DIFFERENCES from DVC @@ -86,6 +86,9 @@ pub fn cmd_check_ignore( xvc_root, &walk_options.ignore_filename.unwrap_or_default(), )?; + + watch!(ignore_rules); + if !opts.targets.is_empty() { let path_bufs = expand_globs_to_paths(current_dir, &opts.targets)?; @@ -110,19 +113,22 @@ fn check_ignore_stdin( let current_dir = conf.current_dir()?; let mut buffer = String::new(); let lines_iter = input.lines(); - lines_iter.map_while(|line| { - if let Ok(line) = line{ - XvcPath::new(xvc_root, current_dir, &PathBuf::from(line)).ok() - } else { - None - }}).for_each(|xvc_path| { - let absolute_path = xvc_path.to_absolute_path(xvc_root); - let res = check_ignore_line(ignore_rules, &absolute_path, opts.non_matching); - if !res.trim().is_empty() { - output!(output_snd, "{}", res); - } - buffer.clear(); - }); + lines_iter + .map_while(|line| { + if let Ok(line) = line { + XvcPath::new(xvc_root, current_dir, &PathBuf::from(line)).ok() + } else { + None + } + }) + .for_each(|xvc_path| { + let absolute_path = xvc_path.to_absolute_path(xvc_root); + let res = check_ignore_line(ignore_rules, &absolute_path, opts.non_matching); + if !res.trim().is_empty() { + output!(output_snd, "{}", res); + } + buffer.clear(); + }); Ok(()) } diff --git a/lib/src/cli/mod.rs b/lib/src/cli/mod.rs index 3b82af6f1..f77184501 100644 --- a/lib/src/cli/mod.rs +++ b/lib/src/cli/mod.rs @@ -167,7 +167,7 @@ impl XvcCLI { } } -// Implement FromStr for XvcCLI +// Implement FromStr for XvcCLI impl FromStr for XvcCLI { type Err = Error; @@ -212,7 +212,6 @@ pub fn run(args: &[&str]) -> Result { /// Run the supplied command within the optional [XvcRoot]. If xvc_root is None, it will be tried /// to be loaded from `cli_opts.workdir`. pub fn dispatch_with_root(cli_opts: cli::XvcCLI, xvc_root_opt: XvcRootOpt) -> Result { - // XvcRoot should be kept per repository and shouldn't change directory across runs assert!( xvc_root_opt.as_ref().is_none() @@ -309,22 +308,22 @@ pub fn dispatch_with_root(cli_opts: cli::XvcCLI, xvc_root_opt: XvcRootOpt) -> Re Some(xvc_root) } - XvcSubCommand::Aliases(opts) => { + XvcSubCommand::Aliases(opts) => { aliases::run(&output_snd, opts)?; xvc_root_opt - }, + } // following commands can only be run inside a repository - XvcSubCommand::Root(opts) => { root::run( - &output_snd, + XvcSubCommand::Root(opts) => { + root::run( + &output_snd, + xvc_root_opt + .as_ref() + .ok_or_else(|| Error::RequiresXvcRepository)?, + opts, + )?; xvc_root_opt - .as_ref() - .ok_or_else(|| Error::RequiresXvcRepository)?, - opts, - )?; - xvc_root_opt - }, - + } XvcSubCommand::File(opts) => { file::run(&output_snd, xvc_root_opt.as_ref(), opts)?; @@ -342,7 +341,6 @@ pub fn dispatch_with_root(cli_opts: cli::XvcCLI, xvc_root_opt: XvcRootOpt) -> Re )?; xvc_root_opt - } XvcSubCommand::CheckIgnore(opts) => { @@ -357,7 +355,6 @@ pub fn dispatch_with_root(cli_opts: cli::XvcCLI, xvc_root_opt: XvcRootOpt) -> Re )?; xvc_root_opt - } XvcSubCommand::Storage(opts) => { @@ -372,7 +369,6 @@ pub fn dispatch_with_root(cli_opts: cli::XvcCLI, xvc_root_opt: XvcRootOpt) -> Re xvc_root_opt } - }; watch!("Before handle_git_automation"); @@ -414,8 +410,6 @@ pub fn dispatch_with_root(cli_opts: cli::XvcCLI, xvc_root_opt: XvcRootOpt) -> Re .unwrap(); xvc_root_opt - - } /// Dispatch commands to respective functions in the API @@ -454,7 +448,6 @@ pub fn dispatch(cli_opts: cli::XvcCLI) -> Result { }; dispatch_with_root(cli_opts, xvc_root_opt) - } fn get_xvc_config_params(cli_opts: &XvcCLI) -> XvcConfigParams { From 672332603177c01399a77b431d22f2a2678da2e2 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Mon, 26 Aug 2024 12:08:59 +0300 Subject: [PATCH 186/257] Update --- book/src/ref/xvc-check-ignore.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/book/src/ref/xvc-check-ignore.md b/book/src/ref/xvc-check-ignore.md index 44462bf0a..927c1dffd 100644 --- a/book/src/ref/xvc-check-ignore.md +++ b/book/src/ref/xvc-check-ignore.md @@ -43,10 +43,10 @@ $ git init $ xvc init ``` -The `.xvcignore` file format is identical to [`.gitignore` file format](https://git-scm.com/docs/gitignore). +You can add files and directories to be ignored by Xvc to `.xvcignore` files. ```console -$ zsh -cl "echo 'my-dir/my-file' > .xvcignore" +$ zsh -cl "echo 'my-dir/my-file' >> .xvcignore" ``` By default it checks the files supplied from `stdin`. From 8ab26bcf8b11df9df6ba5791ca7f03d87b786694 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Mon, 26 Aug 2024 12:11:42 +0300 Subject: [PATCH 187/257] debug --- book/src/ref/xvc-check-ignore.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/book/src/ref/xvc-check-ignore.md b/book/src/ref/xvc-check-ignore.md index 927c1dffd..9109027a1 100644 --- a/book/src/ref/xvc-check-ignore.md +++ b/book/src/ref/xvc-check-ignore.md @@ -52,16 +52,22 @@ $ zsh -cl "echo 'my-dir/my-file' >> .xvcignore" By default it checks the files supplied from `stdin`. ```console -$ echo my-dir/my-file | xvc check-ignore +$ zsh -cl 'echo my-dir/my-file | xvc check-ignore' +[IGNORE] [CWD]/my-dir/my-file + ``` +The `.xvcignore` file format is identical to [`.gitignore` file format](https://git-scm.com/docs/gitignore). + ```console +$ cat .xvcignore # Add patterns of files xvc should ignore, which could improve # the performance. # It's in the same format as .gitignore files. .DS_Store +my-dir/my-file ``` From 737edd84a0e2e1891f5460f0980b747ca46d82e5 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Mon, 26 Aug 2024 12:12:07 +0300 Subject: [PATCH 188/257] debug --- book/src/ref/xvc-check-ignore.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/book/src/ref/xvc-check-ignore.md b/book/src/ref/xvc-check-ignore.md index 9109027a1..7b050d153 100644 --- a/book/src/ref/xvc-check-ignore.md +++ b/book/src/ref/xvc-check-ignore.md @@ -74,7 +74,7 @@ my-dir/my-file If you supply paths from the CLI, they are checked against the ignore rules in `.xvcignore`. ```console -$ xvc check-ignore my-dir/my-file another-dir/another-file +$ xvc -vvvv check-ignore my-dir/my-file another-dir/another-file ``` From 9082eda5e855bdda955a3eb5337ef7a4fef677be Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Mon, 26 Aug 2024 12:15:04 +0300 Subject: [PATCH 189/257] debug --- core/src/check_ignore/mod.rs | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/core/src/check_ignore/mod.rs b/core/src/check_ignore/mod.rs index 86811c49f..6cd7b245f 100644 --- a/core/src/check_ignore/mod.rs +++ b/core/src/check_ignore/mod.rs @@ -88,14 +88,17 @@ pub fn cmd_check_ignore( )?; watch!(ignore_rules); + watch!(opts.targets); if !opts.targets.is_empty() { let path_bufs = expand_globs_to_paths(current_dir, &opts.targets)?; + watch!(path_bufs); - let mut xvc_paths = Vec::::new(); - for p in path_bufs { - xvc_paths.push(XvcPath::new(xvc_root, current_dir, &p)?); - } + let xvc_paths = path_bufs + .iter() + .map(|p| XvcPath::new(xvc_root, current_dir, p)) + .collect::>>()?; + watch!(xvc_paths); check_ignore_paths(xvc_root, &opts, &ignore_rules, &xvc_paths) } else { check_ignore_stdin(input, output_snd, xvc_root, &opts, &ignore_rules) From 7b7669a69f7fb29640fdd6053a3f5fdbecd524ee Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Mon, 26 Aug 2024 12:55:20 +0300 Subject: [PATCH 190/257] fix --- core/src/check_ignore/mod.rs | 8 +++----- core/src/util/mod.rs | 33 --------------------------------- 2 files changed, 3 insertions(+), 38 deletions(-) diff --git a/core/src/check_ignore/mod.rs b/core/src/check_ignore/mod.rs index 6cd7b245f..7d44c42b5 100644 --- a/core/src/check_ignore/mod.rs +++ b/core/src/check_ignore/mod.rs @@ -91,12 +91,10 @@ pub fn cmd_check_ignore( watch!(opts.targets); if !opts.targets.is_empty() { - let path_bufs = expand_globs_to_paths(current_dir, &opts.targets)?; - watch!(path_bufs); - - let xvc_paths = path_bufs + let xvc_paths = opts + .targets .iter() - .map(|p| XvcPath::new(xvc_root, current_dir, p)) + .map(|p| XvcPath::new(xvc_root, current_dir, &PathBuf::from(p))) .collect::>>()?; watch!(xvc_paths); check_ignore_paths(xvc_root, &opts, &ignore_rules, &xvc_paths) diff --git a/core/src/util/mod.rs b/core/src/util/mod.rs index 0af4d545a..d44c67071 100644 --- a/core/src/util/mod.rs +++ b/core/src/util/mod.rs @@ -23,39 +23,6 @@ use crate::{XvcMetadata, XvcPath, CHANNEL_BOUND}; /// A hashmap to store [XvcMetadata] for [XvcPath] pub type XvcPathMetadataMap = HashMap; -/// Expands the given glob `targets` to paths under `current_dir`. -/// It uses [glob::glob] to travers and expand the paths. -/// WARNING: -/// This function doesn't consider any ignore rules in traversal. -/// It may be better to use `xvc_walker::walk_parallel` first and -/// [crate::util::file::glob_paths] to filter the paths. -pub fn expand_globs_to_paths(current_dir: &Path, targets: &[T]) -> Result> -where - T: AsRef + Display, -{ - let current_dir = current_dir.to_str().ok_or(Error::UnicodeError { - cause: current_dir.as_os_str().to_os_string(), - })?; - let mut paths = Vec::::new(); - for t in targets { - let glob_t = format!("{}/{}", current_dir, t); - match glob(&glob_t) { - Ok(glob_path_it) => { - for p in glob_path_it { - match p { - Ok(path) => paths.push(path), - Err(source) => { - Error::GlobError { source }.error(); - } - } - } - } - Err(source) => return Err(Error::GlobPatternError { source }.error()), - } - } - Ok(paths) -} - /// Converts stdin input to a channel. /// /// It works by creating a thread inside. From 5fc7e20420e086c2694ce787e1a7c1a6650254bf Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Mon, 26 Aug 2024 12:55:29 +0300 Subject: [PATCH 191/257] fix --- core/src/check_ignore/mod.rs | 1 - 1 file changed, 1 deletion(-) diff --git a/core/src/check_ignore/mod.rs b/core/src/check_ignore/mod.rs index 7d44c42b5..b4316b0e0 100644 --- a/core/src/check_ignore/mod.rs +++ b/core/src/check_ignore/mod.rs @@ -1,6 +1,5 @@ //! xvc check ignore CLI handling module use crate::error::Result; -use crate::util::expand_globs_to_paths; use crate::util::xvcignore::COMMON_IGNORE_PATTERNS; use crate::{XvcPath, XvcRoot, XVCIGNORE_FILENAME}; use clap::Parser; From 12a1b14baf1a9c49fc08c218cf5839f47021f83b Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Mon, 26 Aug 2024 12:56:53 +0300 Subject: [PATCH 192/257] details --- book/src/ref/xvc-check-ignore.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/book/src/ref/xvc-check-ignore.md b/book/src/ref/xvc-check-ignore.md index 7b050d153..bcc2edcb4 100644 --- a/book/src/ref/xvc-check-ignore.md +++ b/book/src/ref/xvc-check-ignore.md @@ -74,7 +74,7 @@ my-dir/my-file If you supply paths from the CLI, they are checked against the ignore rules in `.xvcignore`. ```console -$ xvc -vvvv check-ignore my-dir/my-file another-dir/another-file +$ xvc check-ignore my-dir/my-file another-dir/another-file ``` @@ -82,6 +82,8 @@ If you're looking which `.xvcignore` file ignores (or whitelists) a certain path ```console $ xvc check-ignore --details my-dir/my-file another-dir/another-file +[IGNORE] [CWD]/my-dir/my-file + ``` From 85efa229b803a86d9b5f43f38ff3a85c6fa0a304 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Mon, 26 Aug 2024 13:01:57 +0300 Subject: [PATCH 193/257] remove details flag from check ignore --- core/src/check_ignore/mod.rs | 47 ++++++------------------------------ 1 file changed, 7 insertions(+), 40 deletions(-) diff --git a/core/src/check_ignore/mod.rs b/core/src/check_ignore/mod.rs index b4316b0e0..b0bf6bb92 100644 --- a/core/src/check_ignore/mod.rs +++ b/core/src/check_ignore/mod.rs @@ -7,7 +7,6 @@ use clap::Parser; use log::trace; use std::io::BufRead; use std::path::{Path, PathBuf}; -use xvc_config::{UpdateFromXvcConfig, XvcConfig}; use xvc_logging::{output, watch, XvcOutputSender}; use xvc_walker::{build_ignore_patterns, IgnoreRules, MatchResult, WalkOptions}; @@ -18,10 +17,6 @@ use xvc_walker::{build_ignore_patterns, IgnoreRules, MatchResult, WalkOptions}; #[command()] /// Check whether xvcignore files ignore a path in an XVC repository pub struct CheckIgnoreCLI { - #[arg(short, long, alias = "all")] - /// Show the exclude patterns along with each target path. - /// A series of lines are printed in this format: :: - details: bool, #[arg( long, default_value = XVCIGNORE_FILENAME, @@ -30,30 +25,13 @@ pub struct CheckIgnoreCLI { /// /// This can be set to .gitignore to test whether Git and Xvc work the same way. ignore_filename: String, - #[arg(short, long)] - /// Include the target paths which don’t match any pattern in the --details list. - /// All fields in each line, except for , will be empty. Has no effect without --details. - non_matching: bool, + #[arg()] /// Targets to check. /// If no targets are provided, they are read from stdin. targets: Vec, } -impl UpdateFromXvcConfig for CheckIgnoreCLI { - fn update_from_conf(self, conf: &XvcConfig) -> xvc_config::error::Result> { - let details = self.details || conf.get_bool("check-ignore.details")?.option; - let non_matching = self.non_matching; - let ignore_filename = self.ignore_filename.clone(); - Ok(Box::new(Self { - details, - non_matching, - ignore_filename, - targets: self.targets.clone(), - })) - } -} - /// # `xvc check_ignore` /// /// Check whether paths are ignored by Xvc or not. @@ -72,7 +50,6 @@ pub fn cmd_check_ignore( opts: CheckIgnoreCLI, ) -> Result<()> { let conf = xvc_root.config(); - let opts = opts.update_from_conf(conf)?; let current_dir = conf.current_dir()?; let walk_options = WalkOptions { @@ -96,9 +73,9 @@ pub fn cmd_check_ignore( .map(|p| XvcPath::new(xvc_root, current_dir, &PathBuf::from(p))) .collect::>>()?; watch!(xvc_paths); - check_ignore_paths(xvc_root, &opts, &ignore_rules, &xvc_paths) + check_ignore_paths(xvc_root, &ignore_rules, &xvc_paths) } else { - check_ignore_stdin(input, output_snd, xvc_root, &opts, &ignore_rules) + check_ignore_stdin(input, output_snd, xvc_root, &ignore_rules) } } @@ -106,7 +83,6 @@ fn check_ignore_stdin( input: R, output_snd: &XvcOutputSender, xvc_root: &XvcRoot, - opts: &CheckIgnoreCLI, ignore_rules: &IgnoreRules, ) -> Result<()> { let conf = xvc_root.config(); @@ -123,7 +99,7 @@ fn check_ignore_stdin( }) .for_each(|xvc_path| { let absolute_path = xvc_path.to_absolute_path(xvc_root); - let res = check_ignore_line(ignore_rules, &absolute_path, opts.non_matching); + let res = check_ignore_line(ignore_rules, &absolute_path); if !res.trim().is_empty() { output!(output_snd, "{}", res); } @@ -134,13 +110,12 @@ fn check_ignore_stdin( fn check_ignore_paths( xvc_root: &XvcRoot, - opts: &CheckIgnoreCLI, ignore_rules: &IgnoreRules, xvc_paths: &[XvcPath], ) -> Result<()> { for path in xvc_paths { let absolute_path = path.to_absolute_path(xvc_root); - let output = check_ignore_line(ignore_rules, &absolute_path, opts.non_matching); + let output = check_ignore_line(ignore_rules, &absolute_path); trace!("output: {}", output); println!("{}", output) } @@ -150,18 +125,10 @@ fn check_ignore_paths( /// Check whether the records match to the full_path. It reports the details if /// set true. Non_matching inverts the reporting. -fn check_ignore_line( - ignore_rules: &IgnoreRules, - absolute_path: &Path, - show_no_match: bool, -) -> String { +fn check_ignore_line(ignore_rules: &IgnoreRules, absolute_path: &Path) -> String { match ignore_rules.check(absolute_path) { MatchResult::NoMatch => { - if show_no_match { - format!("[NO MATCH] {}", absolute_path.to_string_lossy()) - } else { - String::new() - } + format!("[NO MATCH] {}", absolute_path.to_string_lossy()) } MatchResult::Ignore => { format!("[IGNORE] {}", absolute_path.to_string_lossy()) From e83083a3e8c054f7ddbc0f07150b575b5c7164e7 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Mon, 26 Aug 2024 13:04:34 +0300 Subject: [PATCH 194/257] Added whitelist example --- book/src/ref/xvc-check-ignore.md | 17 +++++++---------- 1 file changed, 7 insertions(+), 10 deletions(-) diff --git a/book/src/ref/xvc-check-ignore.md b/book/src/ref/xvc-check-ignore.md index bcc2edcb4..186275129 100644 --- a/book/src/ref/xvc-check-ignore.md +++ b/book/src/ref/xvc-check-ignore.md @@ -17,9 +17,6 @@ Arguments: Targets to check. If no targets are provided, they are read from stdin Options: - -d, --details - Show the exclude patterns along with each target path. A series of lines are printed in this format: :: - --ignore-filename Filename that contains ignore rules @@ -27,9 +24,6 @@ Options: [default: .xvcignore] - -n, --non-matching - Include the target paths which don’t match any pattern in the --details list. All fields in each line, except for , will be empty. Has no effect without --details - -h, --help Print help (see a summary with '-h') @@ -75,16 +69,19 @@ If you supply paths from the CLI, they are checked against the ignore rules in ` ```console $ xvc check-ignore my-dir/my-file another-dir/another-file +[IGNORE] [CWD]/my-dir/my-file +[NO MATCH] [CWD]/another-dir/another-file ``` -If you're looking which `.xvcignore` file ignores (or whitelists) a certain path, you can use `--details`. +You can also add whitelist patterns to `,.xvcignore` files. ```console -$ xvc check-ignore --details my-dir/my-file another-dir/another-file -[IGNORE] [CWD]/my-dir/my-file - +$ zsh -cl "echo 'another-dir/*' >> .xvcignore" +``` +```console +$ xvc check-ignore my-dir/my-file another-dir/another-file ``` This utility can be used to check any other ignore rules in other files as well. From 2a38eee9c3e3c779f56b36c1f1b33672769b820d Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Mon, 26 Aug 2024 13:05:17 +0300 Subject: [PATCH 195/257] fix whitelist example --- book/src/ref/xvc-check-ignore.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/book/src/ref/xvc-check-ignore.md b/book/src/ref/xvc-check-ignore.md index 186275129..82cb8d9ec 100644 --- a/book/src/ref/xvc-check-ignore.md +++ b/book/src/ref/xvc-check-ignore.md @@ -77,7 +77,7 @@ $ xvc check-ignore my-dir/my-file another-dir/another-file You can also add whitelist patterns to `,.xvcignore` files. ```console -$ zsh -cl "echo 'another-dir/*' >> .xvcignore" +$ zsh -cl "echo '!another-dir/*' >> .xvcignore" ``` ```console From 921afbb103dbd24f5ab1feefad9b4154e86a2c2d Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Mon, 26 Aug 2024 13:06:10 +0300 Subject: [PATCH 196/257] Fixed check ignore --- book/src/ref/xvc-check-ignore.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/book/src/ref/xvc-check-ignore.md b/book/src/ref/xvc-check-ignore.md index 82cb8d9ec..62c072a5f 100644 --- a/book/src/ref/xvc-check-ignore.md +++ b/book/src/ref/xvc-check-ignore.md @@ -19,9 +19,9 @@ Arguments: Options: --ignore-filename Filename that contains ignore rules - + This can be set to .gitignore to test whether Git and Xvc work the same way. - + [default: .xvcignore] -h, --help @@ -82,6 +82,9 @@ $ zsh -cl "echo '!another-dir/*' >> .xvcignore" ```console $ xvc check-ignore my-dir/my-file another-dir/another-file +[IGNORE] [CWD]/my-dir/my-file +[WHITELIST] [CWD]/another-dir/another-file + ``` This utility can be used to check any other ignore rules in other files as well. From e8114e29f5d44536d816838f4c16e3401271897c Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Mon, 26 Aug 2024 13:08:12 +0300 Subject: [PATCH 197/257] Update changelog --- CHANGELOG.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index e27686bfa..21b3fafa6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ - Bump dependencies - Replace globset with fast-glob for memory usage +- Remove buggy --details option from xvc check-ignore +- Fixed xvc check-ignore to work with supplied paths +- Fixed loading store targets from xvc file list ## 0.6.10 (2024-08-04) From 392ddd1f5bfd027f2c6c7d60e0c9984f93b033e8 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Tue, 27 Aug 2024 10:16:47 +0300 Subject: [PATCH 198/257] debug --- book/src/ref/xvc-storage-new-local.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/book/src/ref/xvc-storage-new-local.md b/book/src/ref/xvc-storage-new-local.md index 9869f51af..1d4135b42 100644 --- a/book/src/ref/xvc-storage-new-local.md +++ b/book/src/ref/xvc-storage-new-local.md @@ -94,7 +94,7 @@ $ rm -rf dir-0001/ Then get back them from the storage. ```console -$ xvc file bring --from backup dir-0001 +$ xvc -vvvv file bring --from backup dir-0001 $ tree dir-0001 dir-0001 From b672e3d1fa16918a581dadadc8fcdcad3e978463 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Tue, 27 Aug 2024 10:56:43 +0300 Subject: [PATCH 199/257] Handle directories in store differently --- file/src/common/mod.rs | 64 ++++++++++++++++++++++++++++++++++-------- 1 file changed, 53 insertions(+), 11 deletions(-) diff --git a/file/src/common/mod.rs b/file/src/common/mod.rs index ba9fc4cb2..9ea298861 100644 --- a/file/src/common/mod.rs +++ b/file/src/common/mod.rs @@ -98,9 +98,17 @@ pub fn load_targets_from_store( current_dir: &AbsolutePath, targets: &Option>, ) -> Result> { - let store: XvcStore = xvc_root.load_store()?; - watch!(store); - filter_targets_from_store(output_snd, xvc_root, &store, current_dir, targets) + let xvc_path_store: XvcStore = xvc_root.load_store()?; + let md_store: XvcStore = xvc_root.load_store()?; + watch!(xvc_path_store); + filter_targets_from_store( + output_snd, + xvc_root, + &xvc_path_store, + &md_store, + current_dir, + targets, + ) } /// Filters the paths in the store by given globs. @@ -111,7 +119,8 @@ pub fn load_targets_from_store( pub fn filter_targets_from_store( output_snd: &XvcOutputSender, xvc_root: &XvcRoot, - store: &XvcStore, + xvc_path_store: &XvcStore, + md_store: &XvcStore, current_dir: &AbsolutePath, targets: &Option>, ) -> Result> { @@ -129,7 +138,8 @@ pub fn filter_targets_from_store( return filter_targets_from_store( output_snd, xvc_root, - store, + xvc_path_store, + md_store, xvc_root.absolute_path(), &Some(targets), ); @@ -137,13 +147,18 @@ pub fn filter_targets_from_store( watch!(targets); - let hstore = HStore::::from(store); if let Some(targets) = targets { - let paths = filter_paths_by_globs(output_snd, xvc_root, &hstore, targets.as_slice())?; + let paths = filter_paths_by_globs( + output_snd, + xvc_root, + &xvc_path_store, + md_store, + targets.as_slice(), + )?; watch!(paths); Ok(paths) } else { - Ok(hstore) + Ok(xvc_path_store.into()) } } @@ -154,14 +169,41 @@ pub fn filter_targets_from_store( pub fn filter_paths_by_globs( output_snd: &XvcOutputSender, xvc_root: &XvcRoot, - paths: &HStore, + paths: &XvcStore, + md: &XvcStore, globs: &[String], ) -> Result> { if globs.is_empty() { - return Ok(paths.to_owned()); + return Ok(paths.into()); } - let mut glob_matcher = build_glob_matcher(output_snd, xvc_root, globs)?; + // Ensure directories end with / + let globs = globs + .into_iter() + .map(|g| { + if !g.ends_with('/') && !g.contains('*') { + let xvc_path = XvcPath::new(xvc_root, xvc_root, &PathBuf::from(g)).unwrap(); + paths + .entity_by_value(&xvc_path) + .map(|e| { + md.get(&e).map(|xmd| { + if xmd.is_dir() { + // We convert these to dir/** in build_glob_matcher + format!("{g}/") + } else { + g.clone() + } + }) + }) + .flatten() + .unwrap_or_else(|| g.clone()) + } else { + g.clone() + } + }) + .collect::>(); + + let mut glob_matcher = build_glob_matcher(output_snd, xvc_root, &globs)?; watch!(glob_matcher); let paths = paths .iter() From 4c67cef90df1cc121507cd7956ee76c49995dbdd Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Tue, 27 Aug 2024 10:59:05 +0300 Subject: [PATCH 200/257] modify all store selectors --- file/src/copy/mod.rs | 1 + file/src/remove/mod.rs | 2 ++ file/src/untrack/mod.rs | 2 ++ 3 files changed, 5 insertions(+) diff --git a/file/src/copy/mod.rs b/file/src/copy/mod.rs index a12bc326c..4bf31a916 100644 --- a/file/src/copy/mod.rs +++ b/file/src/copy/mod.rs @@ -80,6 +80,7 @@ pub(crate) fn get_source_path_metadata( output_snd, xvc_root, stored_xvc_path_store, + stored_xvc_metadata_store, current_dir, &Some(source_targets), )?; diff --git a/file/src/remove/mod.rs b/file/src/remove/mod.rs index 3a3e00b94..0df37b98b 100644 --- a/file/src/remove/mod.rs +++ b/file/src/remove/mod.rs @@ -67,11 +67,13 @@ pub fn cmd_remove(output_snd: &XvcOutputSender, xvc_root: &XvcRoot, opts: Remove let current_dir = xvc_root.config().current_dir()?; let all_paths = xvc_root.load_store()?; + let all_md = xvc_root.load_store()?; let all_content_digests = xvc_root.load_store()?; let remove_targets = filter_targets_from_store( output_snd, xvc_root, &all_paths, + &all_md, current_dir, &Some(opts.targets), )?; diff --git a/file/src/untrack/mod.rs b/file/src/untrack/mod.rs index e2081cff6..1261f306c 100644 --- a/file/src/untrack/mod.rs +++ b/file/src/untrack/mod.rs @@ -42,11 +42,13 @@ pub fn cmd_untrack( // Load targets from store let current_dir = xvc_root.config().current_dir()?; let all_paths = xvc_root.load_store()?; + let all_metadata = xvc_root.load_store()?; let all_content_digests = xvc_root.load_store()?; let untrack_targets = filter_targets_from_store( output_snd, xvc_root, &all_paths, + &all_metadata, current_dir, &Some(opts.targets), )?; From e9f42c2e661e76180be4a9198fe1be547c9cae98 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Tue, 27 Aug 2024 11:02:37 +0300 Subject: [PATCH 201/257] debug --- file/src/common/mod.rs | 3 +++ 1 file changed, 3 insertions(+) diff --git a/file/src/common/mod.rs b/file/src/common/mod.rs index 9ea298861..e2feb53fb 100644 --- a/file/src/common/mod.rs +++ b/file/src/common/mod.rs @@ -173,6 +173,7 @@ pub fn filter_paths_by_globs( md: &XvcStore, globs: &[String], ) -> Result> { + watch!(globs); if globs.is_empty() { return Ok(paths.into()); } @@ -181,6 +182,7 @@ pub fn filter_paths_by_globs( let globs = globs .into_iter() .map(|g| { + watch!(g); if !g.ends_with('/') && !g.contains('*') { let xvc_path = XvcPath::new(xvc_root, xvc_root, &PathBuf::from(g)).unwrap(); paths @@ -203,6 +205,7 @@ pub fn filter_paths_by_globs( }) .collect::>(); + watch!(globs); let mut glob_matcher = build_glob_matcher(output_snd, xvc_root, &globs)?; watch!(glob_matcher); let paths = paths From f3ddb36ba905aea6663d02befeb77d3c531fd5cb Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Tue, 27 Aug 2024 11:03:43 +0300 Subject: [PATCH 202/257] debug --- file/src/common/mod.rs | 1 + 1 file changed, 1 insertion(+) diff --git a/file/src/common/mod.rs b/file/src/common/mod.rs index e2feb53fb..aa2488552 100644 --- a/file/src/common/mod.rs +++ b/file/src/common/mod.rs @@ -185,6 +185,7 @@ pub fn filter_paths_by_globs( watch!(g); if !g.ends_with('/') && !g.contains('*') { let xvc_path = XvcPath::new(xvc_root, xvc_root, &PathBuf::from(g)).unwrap(); + watch!(xvc_path); paths .entity_by_value(&xvc_path) .map(|e| { From 8fc86827efb912cd0e7fac216f18e8f39ed373e5 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Tue, 27 Aug 2024 11:04:30 +0300 Subject: [PATCH 203/257] debug --- file/src/common/mod.rs | 1 + 1 file changed, 1 insertion(+) diff --git a/file/src/common/mod.rs b/file/src/common/mod.rs index aa2488552..9d60a6a46 100644 --- a/file/src/common/mod.rs +++ b/file/src/common/mod.rs @@ -190,6 +190,7 @@ pub fn filter_paths_by_globs( .entity_by_value(&xvc_path) .map(|e| { md.get(&e).map(|xmd| { + watch!(e); if xmd.is_dir() { // We convert these to dir/** in build_glob_matcher format!("{g}/") From d86b8ab2d324f24ff102fec619911fdff7544be6 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Tue, 27 Aug 2024 11:05:43 +0300 Subject: [PATCH 204/257] debug --- file/src/common/mod.rs | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/file/src/common/mod.rs b/file/src/common/mod.rs index 9d60a6a46..53c3fccf4 100644 --- a/file/src/common/mod.rs +++ b/file/src/common/mod.rs @@ -189,8 +189,9 @@ pub fn filter_paths_by_globs( paths .entity_by_value(&xvc_path) .map(|e| { + watch!(e); md.get(&e).map(|xmd| { - watch!(e); + watch!(xmd); if xmd.is_dir() { // We convert these to dir/** in build_glob_matcher format!("{g}/") From 6bf17035aa70bfb2b362c693f266fdcf73aa7707 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Tue, 27 Aug 2024 11:08:07 +0300 Subject: [PATCH 205/257] debug --- file/src/common/mod.rs | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/file/src/common/mod.rs b/file/src/common/mod.rs index 53c3fccf4..bc6b45703 100644 --- a/file/src/common/mod.rs +++ b/file/src/common/mod.rs @@ -184,7 +184,8 @@ pub fn filter_paths_by_globs( .map(|g| { watch!(g); if !g.ends_with('/') && !g.contains('*') { - let xvc_path = XvcPath::new(xvc_root, xvc_root, &PathBuf::from(g)).unwrap(); + let slashed = format!("{g}/"); + let xvc_path = XvcPath::new(xvc_root, xvc_root, &PathBuf::from(&slashed)).unwrap(); watch!(xvc_path); paths .entity_by_value(&xvc_path) @@ -194,7 +195,7 @@ pub fn filter_paths_by_globs( watch!(xmd); if xmd.is_dir() { // We convert these to dir/** in build_glob_matcher - format!("{g}/") + slashed } else { g.clone() } From fbaf049a2fb303974e4811ed27243bd1a1c592d2 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Tue, 27 Aug 2024 11:09:30 +0300 Subject: [PATCH 206/257] debug --- file/src/common/mod.rs | 1 + 1 file changed, 1 insertion(+) diff --git a/file/src/common/mod.rs b/file/src/common/mod.rs index bc6b45703..6a16c2133 100644 --- a/file/src/common/mod.rs +++ b/file/src/common/mod.rs @@ -185,6 +185,7 @@ pub fn filter_paths_by_globs( watch!(g); if !g.ends_with('/') && !g.contains('*') { let slashed = format!("{g}/"); + watch!(slashed); let xvc_path = XvcPath::new(xvc_root, xvc_root, &PathBuf::from(&slashed)).unwrap(); watch!(xvc_path); paths From be7cdc58de6f579c2d417cbe577df73d7d49bb5b Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Tue, 27 Aug 2024 11:11:45 +0300 Subject: [PATCH 207/257] debug --- ecs/src/ecs/xvcstore.rs | 2 ++ 1 file changed, 2 insertions(+) diff --git a/ecs/src/ecs/xvcstore.rs b/ecs/src/ecs/xvcstore.rs index 2c2186456..38963842a 100644 --- a/ecs/src/ecs/xvcstore.rs +++ b/ecs/src/ecs/xvcstore.rs @@ -289,6 +289,8 @@ where where T: PartialEq, { + watch!(self.entity_index); + watch!(value); self.entity_index.get(value) } From 7960fec6c952a1ac645ff93576dbd35053221e29 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Tue, 27 Aug 2024 11:22:30 +0300 Subject: [PATCH 208/257] debug --- ecs/src/ecs/xvcstore.rs | 15 +++++++++++++++ file/src/common/mod.rs | 26 +++++++------------------- 2 files changed, 22 insertions(+), 19 deletions(-) diff --git a/ecs/src/ecs/xvcstore.rs b/ecs/src/ecs/xvcstore.rs index 38963842a..1ef0e3e66 100644 --- a/ecs/src/ecs/xvcstore.rs +++ b/ecs/src/ecs/xvcstore.rs @@ -274,6 +274,21 @@ where HStore::from(s) } + /// Runs `predicate` for all elements and returns true if one of them is true. + /// + /// `predicate` must be a function or closure that returns `bool`. + pub fn any(&self, predicate: F) -> bool + where + F: Fn(&XvcEntity, &T) -> bool, + { + for (e, v) in self.map.iter() { + if predicate(e, v) { + return true; + } + } + false + } + /// Returns the first element of the map /// /// This is useful when there is only one element after [Self::filter] diff --git a/file/src/common/mod.rs b/file/src/common/mod.rs index 6a16c2133..8757bc81b 100644 --- a/file/src/common/mod.rs +++ b/file/src/common/mod.rs @@ -180,30 +180,18 @@ pub fn filter_paths_by_globs( // Ensure directories end with / let globs = globs - .into_iter() + .iter() .map(|g| { watch!(g); if !g.ends_with('/') && !g.contains('*') { let slashed = format!("{g}/"); watch!(slashed); - let xvc_path = XvcPath::new(xvc_root, xvc_root, &PathBuf::from(&slashed)).unwrap(); - watch!(xvc_path); - paths - .entity_by_value(&xvc_path) - .map(|e| { - watch!(e); - md.get(&e).map(|xmd| { - watch!(xmd); - if xmd.is_dir() { - // We convert these to dir/** in build_glob_matcher - slashed - } else { - g.clone() - } - }) - }) - .flatten() - .unwrap_or_else(|| g.clone()) + // We don't track directories. Instead we look for files that start with the directory. + if paths.any(|_, p| p.as_str().starts_with(&slashed)) { + slashed + } else { + g.clone() + } } else { g.clone() } From 54be1726adae927d45a35f8d9e937c65dafa16b8 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Tue, 27 Aug 2024 11:25:00 +0300 Subject: [PATCH 209/257] fixed? --- book/src/ref/xvc-storage-new-local.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/book/src/ref/xvc-storage-new-local.md b/book/src/ref/xvc-storage-new-local.md index 1d4135b42..eb1da5b9a 100644 --- a/book/src/ref/xvc-storage-new-local.md +++ b/book/src/ref/xvc-storage-new-local.md @@ -21,7 +21,7 @@ Options: -n, --name Name of the storage. - + Recommended to keep this name unique to refer easily. -h, --help @@ -94,7 +94,7 @@ $ rm -rf dir-0001/ Then get back them from the storage. ```console -$ xvc -vvvv file bring --from backup dir-0001 +$ xvc file bring --from backup dir-0001 $ tree dir-0001 dir-0001 @@ -117,7 +117,7 @@ $ xvc file remove --from-storage backup dir-0001/ `--name NAME` is not checked to be unique but you should use unique storage names to refer them later. -`--path PATH` should be accessible for writing and shouldn't already exist. +`--path PATH` should be accessible for writing and shouldn't already exist. ## Technical Details From 8d4c63798b77e9f7ee4ed37bab77fd9b6b23f11f Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Tue, 27 Aug 2024 11:25:29 +0300 Subject: [PATCH 210/257] ws --- book/src/ref/xvc-storage-new-local.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/book/src/ref/xvc-storage-new-local.md b/book/src/ref/xvc-storage-new-local.md index eb1da5b9a..e5dc4911e 100644 --- a/book/src/ref/xvc-storage-new-local.md +++ b/book/src/ref/xvc-storage-new-local.md @@ -21,7 +21,7 @@ Options: -n, --name Name of the storage. - + Recommended to keep this name unique to refer easily. -h, --help From fddd0e28eee797c244e46ec9e90a939db2aa6dd3 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Tue, 27 Aug 2024 11:32:12 +0300 Subject: [PATCH 211/257] fixed some warnings --- core/src/util/mod.rs | 5 +---- ecs/src/ecs/xvcstore.rs | 2 -- file/src/common/gitignore.rs | 2 +- file/src/common/mod.rs | 23 +++-------------------- file/src/copy/mod.rs | 1 - file/src/remove/mod.rs | 2 -- file/src/untrack/mod.rs | 3 +-- lib/src/init/mod.rs | 1 + lib/tests/test_walker_parallel.rs | 1 - walker/src/ignore_rules.rs | 8 +++++++- 10 files changed, 14 insertions(+), 34 deletions(-) diff --git a/core/src/util/mod.rs b/core/src/util/mod.rs index d44c67071..ad93e05c4 100644 --- a/core/src/util/mod.rs +++ b/core/src/util/mod.rs @@ -7,17 +7,14 @@ pub mod store; pub mod xvcignore; use std::collections::HashMap; -use std::fmt::Display; use std::io::{self, Read}; -use std::path::{Path, PathBuf}; use std::thread::sleep; use std::time::Duration; use crossbeam_channel::{bounded, Receiver}; -use glob::glob; use xvc_logging::watch; -use crate::error::{Error, Result}; +use crate::error::Result; use crate::{XvcMetadata, XvcPath, CHANNEL_BOUND}; /// A hashmap to store [XvcMetadata] for [XvcPath] diff --git a/ecs/src/ecs/xvcstore.rs b/ecs/src/ecs/xvcstore.rs index 1ef0e3e66..bb7b504ea 100644 --- a/ecs/src/ecs/xvcstore.rs +++ b/ecs/src/ecs/xvcstore.rs @@ -304,8 +304,6 @@ where where T: PartialEq, { - watch!(self.entity_index); - watch!(value); self.entity_index.get(value) } diff --git a/file/src/common/gitignore.rs b/file/src/common/gitignore.rs index 9d97c2db7..3c85b9e00 100644 --- a/file/src/common/gitignore.rs +++ b/file/src/common/gitignore.rs @@ -11,7 +11,7 @@ use xvc_core::util::git::build_gitignore; use crate::{Result, CHANNEL_CAPACITY}; use xvc_core::{XvcPath, XvcRoot}; -use xvc_logging::{debug, error, info, uwr, watch, XvcOutputSender}; +use xvc_logging::{debug, error, info, uwr, XvcOutputSender}; use xvc_walker::{AbsolutePath, IgnoreRules, MatchResult}; /// Used to signal ignored files and directories to the ignore handler diff --git a/file/src/common/mod.rs b/file/src/common/mod.rs index 8757bc81b..67fc54e29 100644 --- a/file/src/common/mod.rs +++ b/file/src/common/mod.rs @@ -99,16 +99,7 @@ pub fn load_targets_from_store( targets: &Option>, ) -> Result> { let xvc_path_store: XvcStore = xvc_root.load_store()?; - let md_store: XvcStore = xvc_root.load_store()?; - watch!(xvc_path_store); - filter_targets_from_store( - output_snd, - xvc_root, - &xvc_path_store, - &md_store, - current_dir, - targets, - ) + filter_targets_from_store(output_snd, xvc_root, &xvc_path_store, current_dir, targets) } /// Filters the paths in the store by given globs. @@ -120,7 +111,6 @@ pub fn filter_targets_from_store( output_snd: &XvcOutputSender, xvc_root: &XvcRoot, xvc_path_store: &XvcStore, - md_store: &XvcStore, current_dir: &AbsolutePath, targets: &Option>, ) -> Result> { @@ -139,7 +129,6 @@ pub fn filter_targets_from_store( output_snd, xvc_root, xvc_path_store, - md_store, xvc_root.absolute_path(), &Some(targets), ); @@ -148,13 +137,8 @@ pub fn filter_targets_from_store( watch!(targets); if let Some(targets) = targets { - let paths = filter_paths_by_globs( - output_snd, - xvc_root, - &xvc_path_store, - md_store, - targets.as_slice(), - )?; + let paths = + filter_paths_by_globs(output_snd, xvc_root, xvc_path_store, targets.as_slice())?; watch!(paths); Ok(paths) } else { @@ -170,7 +154,6 @@ pub fn filter_paths_by_globs( output_snd: &XvcOutputSender, xvc_root: &XvcRoot, paths: &XvcStore, - md: &XvcStore, globs: &[String], ) -> Result> { watch!(globs); diff --git a/file/src/copy/mod.rs b/file/src/copy/mod.rs index 4bf31a916..a12bc326c 100644 --- a/file/src/copy/mod.rs +++ b/file/src/copy/mod.rs @@ -80,7 +80,6 @@ pub(crate) fn get_source_path_metadata( output_snd, xvc_root, stored_xvc_path_store, - stored_xvc_metadata_store, current_dir, &Some(source_targets), )?; diff --git a/file/src/remove/mod.rs b/file/src/remove/mod.rs index 0df37b98b..3a3e00b94 100644 --- a/file/src/remove/mod.rs +++ b/file/src/remove/mod.rs @@ -67,13 +67,11 @@ pub fn cmd_remove(output_snd: &XvcOutputSender, xvc_root: &XvcRoot, opts: Remove let current_dir = xvc_root.config().current_dir()?; let all_paths = xvc_root.load_store()?; - let all_md = xvc_root.load_store()?; let all_content_digests = xvc_root.load_store()?; let remove_targets = filter_targets_from_store( output_snd, xvc_root, &all_paths, - &all_md, current_dir, &Some(opts.targets), )?; diff --git a/file/src/untrack/mod.rs b/file/src/untrack/mod.rs index 1261f306c..99081605e 100644 --- a/file/src/untrack/mod.rs +++ b/file/src/untrack/mod.rs @@ -42,13 +42,12 @@ pub fn cmd_untrack( // Load targets from store let current_dir = xvc_root.config().current_dir()?; let all_paths = xvc_root.load_store()?; - let all_metadata = xvc_root.load_store()?; let all_content_digests = xvc_root.load_store()?; + let untrack_targets = filter_targets_from_store( output_snd, xvc_root, &all_paths, - &all_metadata, current_dir, &Some(opts.targets), )?; diff --git a/lib/src/init/mod.rs b/lib/src/init/mod.rs index e10c3309c..7ec06c225 100644 --- a/lib/src/init/mod.rs +++ b/lib/src/init/mod.rs @@ -39,6 +39,7 @@ pub struct InitCLI { /// # Arguments /// /// - `xvc_root_opt`: Optional [xvc_core::XvcRoot] +/// /// It's an error to reinit inside an Xvc repository (with `Some(xvc_root)`) normally. /// It's possible to force reinit with `opts.force` though. /// diff --git a/lib/tests/test_walker_parallel.rs b/lib/tests/test_walker_parallel.rs index 548bbd902..1769b5f79 100644 --- a/lib/tests/test_walker_parallel.rs +++ b/lib/tests/test_walker_parallel.rs @@ -5,7 +5,6 @@ use std::{ sync::{Arc, RwLock}, }; -use fast_glob::Glob; use log::LevelFilter; use xvc_walker::*; diff --git a/walker/src/ignore_rules.rs b/walker/src/ignore_rules.rs index 8fdd73151..5ae2f280b 100644 --- a/walker/src/ignore_rules.rs +++ b/walker/src/ignore_rules.rs @@ -1,7 +1,7 @@ //! Ignore patterns for a directory and its child directories. use crate::{Result, Source}; use std::path::{Path, PathBuf}; -use std::sync::{Arc, Mutex, RwLock}; +use std::sync::{Arc, RwLock}; use crate::pattern::{MatchResult, Pattern}; use rayon::prelude::*; @@ -26,6 +26,7 @@ pub struct IgnoreRules { pub whitelist_patterns: Arc>>, } +/// IgnoreRules shared across threads. pub type SharedIgnoreRules = Arc>; impl IgnoreRules { @@ -202,6 +203,11 @@ impl IgnoreRules { Ok(()) } + /// Adds a list of patterns to the current ignore rules. + /// + /// # Arguments + /// + /// * `patterns` - A vector of patterns to be added to the ignore rules. pub fn add_patterns(&self, patterns: Vec) -> Result<()> { let other = IgnoreRules::from_patterns(&self.root, None, patterns); From 291a20c5ece40b76ffd3be1c8627b3ffce8db930 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Tue, 27 Aug 2024 11:33:09 +0300 Subject: [PATCH 212/257] changelog --- CHANGELOG.md | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 21b3fafa6..8781c4b3c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,7 @@ - Remove buggy --details option from xvc check-ignore - Fixed xvc check-ignore to work with supplied paths - Fixed loading store targets from xvc file list +- Directory targets in various commands doesn't require / at the end ## 0.6.10 (2024-08-04) From da9e51c464bc891a4d16cb14c04cec0bc4be3ca5 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Tue, 27 Aug 2024 11:56:37 +0300 Subject: [PATCH 213/257] bump version --- CHANGELOG.md | 6 ++++-- config/Cargo.toml | 6 +++--- core/Cargo.toml | 12 ++++++------ ecs/Cargo.toml | 4 ++-- file/Cargo.toml | 16 ++++++++-------- lib/Cargo.toml | 20 ++++++++++---------- logging/Cargo.toml | 2 +- pipeline/Cargo.toml | 16 ++++++++-------- storage/Cargo.toml | 14 +++++++------- test_helper/Cargo.toml | 4 ++-- walker/Cargo.toml | 6 +++--- 11 files changed, 54 insertions(+), 52 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8781c4b3c..82aa887c0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,11 +1,13 @@ ## Unreleased +## 0.6.11 (2024-08-26) + - Bump dependencies - Replace globset with fast-glob for memory usage -- Remove buggy --details option from xvc check-ignore +- Remove --details option from xvc check-ignore - Fixed xvc check-ignore to work with supplied paths - Fixed loading store targets from xvc file list -- Directory targets in various commands doesn't require / at the end +- Directory targets in various commands doesn't require / at the end when they only exist in the cache ## 0.6.10 (2024-08-04) diff --git a/config/Cargo.toml b/config/Cargo.toml index 0cce62bd7..57e2c258c 100644 --- a/config/Cargo.toml +++ b/config/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-config" -version = "0.6.11-alpha.9" +version = "0.6.11" edition = "2021" description = "Xvc configuration management" authors = ["Emre Şahin "] @@ -16,8 +16,8 @@ name = "xvc_config" crate-type = ["rlib"] [dependencies] -xvc-logging = { version = "0.6.11-alpha.9", path = "../logging" } -xvc-walker = { version = "0.6.11-alpha.9", path = "../walker" } +xvc-logging = { version = "0.6.11", path = "../logging" } +xvc-walker = { version = "0.6.11", path = "../walker" } ## Cli and config diff --git a/core/Cargo.toml b/core/Cargo.toml index a7b5c19a9..3190d1763 100644 --- a/core/Cargo.toml +++ b/core/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-core" -version = "0.6.11-alpha.9" +version = "0.6.11" edition = "2021" description = "Xvc core for common elements for all commands" authors = ["Emre Şahin "] @@ -16,10 +16,10 @@ name = "xvc_core" crate-type = ["rlib"] [dependencies] -xvc-config = { version = "0.6.11-alpha.9", path = "../config" } -xvc-logging = { version = "0.6.11-alpha.9", path = "../logging" } -xvc-ecs = { version = "0.6.11-alpha.9", path = "../ecs" } -xvc-walker = { version = "0.6.11-alpha.9", path = "../walker" } +xvc-config = { version = "0.6.11", path = "../config" } +xvc-logging = { version = "0.6.11", path = "../logging" } +xvc-ecs = { version = "0.6.11", path = "../ecs" } +xvc-walker = { version = "0.6.11", path = "../walker" } ## Cli and config clap = { version = "^4.5", features = ["derive"] } @@ -81,6 +81,6 @@ itertools = "^0.13" [dev-dependencies] -xvc-test-helper = { version = "0.6.11-alpha.9", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.11", path = "../test_helper/" } proptest = "^1.5" test-case = "^3.3" diff --git a/ecs/Cargo.toml b/ecs/Cargo.toml index 2bde7192b..938b7b819 100644 --- a/ecs/Cargo.toml +++ b/ecs/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-ecs" -version = "0.6.11-alpha.9" +version = "0.6.11" edition = "2021" description = "Entity-Component System for Xvc" authors = ["Emre Şahin "] @@ -16,7 +16,7 @@ name = "xvc_ecs" crate-type = ["rlib"] [dependencies] -xvc-logging = { version = "0.6.11-alpha.9", path = "../logging" } +xvc-logging = { version = "0.6.11", path = "../logging" } ## Serialization serde = { version = "^1.0", features = ["derive"] } diff --git a/file/Cargo.toml b/file/Cargo.toml index 9e0b8190f..552ab8611 100644 --- a/file/Cargo.toml +++ b/file/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-file" -version = "0.6.11-alpha.9" +version = "0.6.11" edition = "2021" description = "File tracking, versioning, upload and download functions for Xvc" authors = ["Emre Şahin "] @@ -21,12 +21,12 @@ test = true bench = true [dependencies] -xvc-logging = { version = "0.6.11-alpha.9", path = "../logging" } -xvc-config = { version = "0.6.11-alpha.9", path = "../config" } -xvc-core = { version = "0.6.11-alpha.9", path = "../core" } -xvc-ecs = { version = "0.6.11-alpha.9", path = "../ecs" } -xvc-walker = { version = "0.6.11-alpha.9", path = "../walker" } -xvc-storage = { version = "0.6.11-alpha.9", path = "../storage", default-features = false } +xvc-logging = { version = "0.6.11", path = "../logging" } +xvc-config = { version = "0.6.11", path = "../config" } +xvc-core = { version = "0.6.11", path = "../core" } +xvc-ecs = { version = "0.6.11", path = "../ecs" } +xvc-walker = { version = "0.6.11", path = "../walker" } +xvc-storage = { version = "0.6.11", path = "../storage", default-features = false } ## Cli and config @@ -95,5 +95,5 @@ default = ["reflink"] reflink = ["dep:reflink"] [dev-dependencies] -xvc-test-helper = { version = "0.6.11-alpha.9", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.11", path = "../test_helper/" } shellfn = "^0.1" diff --git a/lib/Cargo.toml b/lib/Cargo.toml index 536297e0b..bf05bebd6 100644 --- a/lib/Cargo.toml +++ b/lib/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc" -version = "0.6.11-alpha.9" +version = "0.6.11" edition = "2021" description = "An MLOps tool to manage data files and pipelines on top of Git" authors = ["Emre Şahin "] @@ -20,14 +20,14 @@ name = "xvc" path = "src/main.rs" [dependencies] -xvc-config = { version = "0.6.11-alpha.9", path = "../config" } -xvc-core = { version = "0.6.11-alpha.9", path = "../core" } -xvc-logging = { version = "0.6.11-alpha.9", path = "../logging" } -xvc-ecs = { version = "0.6.11-alpha.9", path = "../ecs" } -xvc-file = { version = "0.6.11-alpha.9", path = "../file", default-features = false } -xvc-pipeline = { version = "0.6.11-alpha.9", path = "../pipeline" } -xvc-walker = { version = "0.6.11-alpha.9", path = "../walker" } -xvc-storage = { version = "0.6.11-alpha.9", path = "../storage", default-features = false } +xvc-config = { version = "0.6.11", path = "../config" } +xvc-core = { version = "0.6.11", path = "../core" } +xvc-logging = { version = "0.6.11", path = "../logging" } +xvc-ecs = { version = "0.6.11", path = "../ecs" } +xvc-file = { version = "0.6.11", path = "../file", default-features = false } +xvc-pipeline = { version = "0.6.11", path = "../pipeline" } +xvc-walker = { version = "0.6.11", path = "../walker" } +xvc-storage = { version = "0.6.11", path = "../storage", default-features = false } ## Cli and config clap = { version = "^4.5", features = ["derive", "cargo"] } @@ -133,4 +133,4 @@ shellfn = "^0.1" test-case = "^3.3" trycmd = "^0.15" which = "^6.0" -xvc-test-helper = { version = "0.6.11-alpha.9", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.11", path = "../test_helper/" } diff --git a/logging/Cargo.toml b/logging/Cargo.toml index 60eec1622..797ba00cb 100644 --- a/logging/Cargo.toml +++ b/logging/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-logging" -version = "0.6.11-alpha.9" +version = "0.6.11" edition = "2021" description = "Logging crate for Xvc" authors = ["Emre Şahin "] diff --git a/pipeline/Cargo.toml b/pipeline/Cargo.toml index 8b0ba7acf..23f4d8816 100644 --- a/pipeline/Cargo.toml +++ b/pipeline/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-pipeline" -version = "0.6.11-alpha.9" +version = "0.6.11" edition = "2021" description = "Xvc data pipeline management" authors = ["Emre Şahin "] @@ -19,12 +19,12 @@ default = [] bundled-sqlite = ["rusqlite/bundled"] [dependencies] -xvc-config = { version = "0.6.11-alpha.9", path = "../config" } -xvc-core = { version = "0.6.11-alpha.9", path = "../core" } -xvc-ecs = { version = "0.6.11-alpha.9", path = "../ecs" } -xvc-logging = { version = "0.6.11-alpha.9", path = "../logging" } -xvc-walker = { version = "0.6.11-alpha.9", path = "../walker" } -xvc-file = { version = "0.6.11-alpha.9", path = "../file", default-features = false } +xvc-config = { version = "0.6.11", path = "../config" } +xvc-core = { version = "0.6.11", path = "../core" } +xvc-ecs = { version = "0.6.11", path = "../ecs" } +xvc-logging = { version = "0.6.11", path = "../logging" } +xvc-walker = { version = "0.6.11", path = "../walker" } +xvc-file = { version = "0.6.11", path = "../file", default-features = false } ## Cli and config clap = { version = "^4.5", features = ["derive"] } @@ -98,5 +98,5 @@ itertools = "^0.13" derive_more = "^0.99" [dev-dependencies] -xvc-test-helper = { version = "0.6.11-alpha.9", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.11", path = "../test_helper/" } test-case = "^3.3" diff --git a/storage/Cargo.toml b/storage/Cargo.toml index 8224a4b6d..78fa2e7d3 100644 --- a/storage/Cargo.toml +++ b/storage/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-storage" -version = "0.6.11-alpha.9" +version = "0.6.11" edition = "2021" description = "Xvc remote and local storage management" authors = ["Emre Şahin "] @@ -16,11 +16,11 @@ name = "xvc_storage" crate-type = ["rlib"] [dependencies] -xvc-logging = { version = "0.6.11-alpha.9", path = "../logging" } -xvc-config = { version = "0.6.11-alpha.9", path = "../config" } -xvc-core = { version = "0.6.11-alpha.9", path = "../core" } -xvc-ecs = { version = "0.6.11-alpha.9", path = "../ecs" } -xvc-walker = { version = "0.6.11-alpha.9", path = "../walker" } +xvc-logging = { version = "0.6.11", path = "../logging" } +xvc-config = { version = "0.6.11", path = "../config" } +xvc-core = { version = "0.6.11", path = "../core" } +xvc-ecs = { version = "0.6.11", path = "../ecs" } +xvc-walker = { version = "0.6.11", path = "../walker" } ## Cli and config clap = { version = "^4.5", features = ["derive"] } @@ -104,7 +104,7 @@ bundled-openssl = ["openssl/vendored"] [dev-dependencies] -xvc-test-helper = { version = "0.6.11-alpha.9", path = "../test_helper/" } +xvc-test-helper = { version = "0.6.11", path = "../test_helper/" } shellfn = "^0.1" [package.metadata.cargo-udeps.ignore] diff --git a/test_helper/Cargo.toml b/test_helper/Cargo.toml index c8994a3b0..7bab90fca 100644 --- a/test_helper/Cargo.toml +++ b/test_helper/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-test-helper" -version = "0.6.11-alpha.9" +version = "0.6.11" edition = "2021" description = "Unit test helper functions for Xvc" authors = ["Emre Şahin "] @@ -20,7 +20,7 @@ path = "src/main.rs" # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html [dependencies] -xvc-logging = { version = "0.6.11-alpha.9", path = "../logging/" } +xvc-logging = { version = "0.6.11", path = "../logging/" } rand = "^0.8" log = "^0.4" diff --git a/walker/Cargo.toml b/walker/Cargo.toml index 016b7b5c9..0eecee147 100644 --- a/walker/Cargo.toml +++ b/walker/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "xvc-walker" -version = "0.6.11-alpha.9" +version = "0.6.11" edition = "2021" description = "Xvc parallel file system walker with ignore features" authors = ["Emre Şahin "] @@ -16,7 +16,7 @@ name = "xvc_walker" crate-type = ["rlib"] [dependencies] -xvc-logging = { version = "0.6.11-alpha.9", path = "../logging" } +xvc-logging = { version = "0.6.11", path = "../logging" } fast-glob = "^0.3" ## Parallelization @@ -41,7 +41,7 @@ itertools = "^0.13" regex = "^1.10" [dev-dependencies] -xvc-test-helper = { path = "../test_helper/", version = "0.6.11-alpha.9" } +xvc-test-helper = { path = "../test_helper/", version = "0.6.11" } test-case = "^3.3" [package.metadata.cargo-udeps.ignore] From fea9f27d62e54a335fdb28e9b6e8ef34e534f175 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Wed, 28 Aug 2024 10:42:15 +0300 Subject: [PATCH 214/257] add run-tests to gitignore --- .gitignore | 1 + 1 file changed, 1 insertion(+) diff --git a/.gitignore b/.gitignore index d576b81b5..0fa54ffa8 100644 --- a/.gitignore +++ b/.gitignore @@ -17,3 +17,4 @@ Cargo.lock flamegraph.svg workflow_tests/docs lib/docs +run-tests.zsh From 98cdeb0d385b64e4dd9338fecbaf63d0df5b9c58 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Wed, 28 Aug 2024 10:43:47 +0300 Subject: [PATCH 215/257] debug --- file/src/list/mod.rs | 1 + 1 file changed, 1 insertion(+) diff --git a/file/src/list/mod.rs b/file/src/list/mod.rs index 779ef4c12..25887ae7b 100644 --- a/file/src/list/mod.rs +++ b/file/src/list/mod.rs @@ -522,6 +522,7 @@ impl UpdateFromXvcConfig for ListCLI { /// - =: Recorded and actual file have the same timestamp /// - >: Cached file is newer, xvc recheck to update the file /// - <: File is newer, xvc carry-in to update the cache +/// /// TODO: - I: File is ignored pub fn cmd_list(output_snd: &XvcOutputSender, xvc_root: &XvcRoot, cli_opts: ListCLI) -> Result<()> { From e004399ad7ce24961c8d6e6be2bce59a3e94654f Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Wed, 28 Aug 2024 11:32:17 +0300 Subject: [PATCH 216/257] Updated example url --- lib/tests/common/mod.rs | 2 +- lib/tests/test_core_util_file_walker.rs | 6 +----- 2 files changed, 2 insertions(+), 6 deletions(-) diff --git a/lib/tests/common/mod.rs b/lib/tests/common/mod.rs index 7fc9003c0..64da31934 100644 --- a/lib/tests/common/mod.rs +++ b/lib/tests/common/mod.rs @@ -61,7 +61,7 @@ pub fn run_xvc(cwd: Option<&Path>, args: &[&str], verbosity: XvcVerbosity) -> Re } pub fn example_project_url() -> Result { - Ok(format!("http://one.emresult.com/~iex/{EXAMPLE_PROJECT_NAME}.tgz").to_string()) + Ok("https://drive.google.com/file/d/14dtbH_rGspGW_m4MiNUN6AGuu2lS8E4z/view".to_string()) } pub fn example_project_template_path() -> Result { diff --git a/lib/tests/test_core_util_file_walker.rs b/lib/tests/test_core_util_file_walker.rs index 9480d9f42..6e4aeb1dd 100644 --- a/lib/tests/test_core_util_file_walker.rs +++ b/lib/tests/test_core_util_file_walker.rs @@ -7,11 +7,7 @@ use xvc_core::{ }; use xvc_test_helper::test_logging; -use std::{ - collections::{BTreeSet, HashSet}, - path::Path, - time::Duration, -}; +use std::{collections::HashSet, path::Path, time::Duration}; use xvc_logging::watch; From b6a5dfb3535e274486929ed057484b2c6d740661 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Wed, 28 Aug 2024 11:40:56 +0300 Subject: [PATCH 217/257] minor --- lib/tests/common/mod.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/tests/common/mod.rs b/lib/tests/common/mod.rs index 64da31934..e58a1f9c3 100644 --- a/lib/tests/common/mod.rs +++ b/lib/tests/common/mod.rs @@ -91,7 +91,7 @@ pub fn download_example_project() -> Result<()> { .arg("xzf") .arg(curl_output_filename) .arg("--directory") - .arg(&env::temp_dir().to_string_lossy().to_string()) + .arg(env::temp_dir().to_string_lossy().as_ref()) .output()?; if !tar_output.status.success() { From e617be63d9104d3081e57dc994c9438fc532dcd5 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Fri, 30 Aug 2024 13:42:39 +0300 Subject: [PATCH 218/257] update test server urls --- book/src/ref/xvc-storage-new-rsync.md | 4 ++-- lib/tests/common/mod.rs | 2 +- lib/tests/test_storage_new_generic_rsync.rs | 2 +- lib/tests/test_storage_new_minio.rs | 4 ++-- lib/tests/test_storage_new_rsync.rs | 4 ++-- 5 files changed, 8 insertions(+), 8 deletions(-) diff --git a/book/src/ref/xvc-storage-new-rsync.md b/book/src/ref/xvc-storage-new-rsync.md index 6bb2af925..109768b66 100644 --- a/book/src/ref/xvc-storage-new-rsync.md +++ b/book/src/ref/xvc-storage-new-rsync.md @@ -17,7 +17,7 @@ Usage: xvc storage new rsync [OPTIONS] --name --host --storage-dir Options: -n, --name Name of the storage. - + Recommended to keep this name unique to refer easily. --host @@ -69,7 +69,7 @@ $ xvc file track dir-0001 You can define a storage bucket as storage and begin to use it. ```console -$ xvc storage new rsync --name backup --host one.emresult.com --user iex --storage-dir /tmp/xvc-backup/ +$ xvc storage new rsync --name backup --host e1.xvc.dev --user iex --storage-dir /tmp/xvc-backup/ ``` diff --git a/lib/tests/common/mod.rs b/lib/tests/common/mod.rs index e58a1f9c3..882793745 100644 --- a/lib/tests/common/mod.rs +++ b/lib/tests/common/mod.rs @@ -61,7 +61,7 @@ pub fn run_xvc(cwd: Option<&Path>, args: &[&str], verbosity: XvcVerbosity) -> Re } pub fn example_project_url() -> Result { - Ok("https://drive.google.com/file/d/14dtbH_rGspGW_m4MiNUN6AGuu2lS8E4z/view".to_string()) + Ok("http://e1.xvc.dev/example-xvc.tgz".to_string()) } pub fn example_project_template_path() -> Result { diff --git a/lib/tests/test_storage_new_generic_rsync.rs b/lib/tests/test_storage_new_generic_rsync.rs index f42df3512..8eff19d37 100644 --- a/lib/tests/test_storage_new_generic_rsync.rs +++ b/lib/tests/test_storage_new_generic_rsync.rs @@ -37,7 +37,7 @@ fn test_storage_new_generic_rsync() -> Result<()> { "/tmp/{}/", common::random_dir_name("xvc-storage", Some(111)) ); - let test_host = "xvc-test@one.emresult.com"; + let test_host = "iex@e1.xvc.dev"; let url = test_host.to_string(); let local_test_dir = env::temp_dir().join(common::random_dir_name("xvc-storage-copy", None)); diff --git a/lib/tests/test_storage_new_minio.rs b/lib/tests/test_storage_new_minio.rs index 72814cfbe..48c703605 100644 --- a/lib/tests/test_storage_new_minio.rs +++ b/lib/tests/test_storage_new_minio.rs @@ -51,8 +51,8 @@ gpg_decrypt = %(gpg_command)s -d --verbose --no-use-agent --batch --yes --passph gpg_encrypt = %(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s gpg_passphrase = guess_mime_type = True -host_base = one.emresult.com:9000 -host_bucket = one.emresult.com:9000 +host_base = e1.xvc.dev:9000 +host_bucket = e1.xvc.dev:9000 human_readable_sizes = False invalidate_default_index_on_cf = False invalidate_default_index_root_on_cf = True diff --git a/lib/tests/test_storage_new_rsync.rs b/lib/tests/test_storage_new_rsync.rs index 0786cd93d..1412fdd9f 100644 --- a/lib/tests/test_storage_new_rsync.rs +++ b/lib/tests/test_storage_new_rsync.rs @@ -37,8 +37,8 @@ fn test_storage_new_rsync() -> Result<()> { "/tmp/{}/", common::random_dir_name("xvc-storage", Some(111)) ); - let test_user = "xvc-test"; - let test_host = "one.emresult.com"; + let test_user = "iex"; + let test_host = "e1.xvc.dev"; let url = format!("{test_user}@{test_host}"); let local_test_dir = env::temp_dir().join(common::random_dir_name("xvc-storage-copy", None)); From a28b030a0679fb9cae33e1e7adb5318cbe4156e1 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Fri, 30 Aug 2024 13:44:21 +0300 Subject: [PATCH 219/257] Added run-tests script --- .gitignore | 1 - run-tests.zsh | 1 + 2 files changed, 1 insertion(+), 1 deletion(-) create mode 100755 run-tests.zsh diff --git a/.gitignore b/.gitignore index 0fa54ffa8..d576b81b5 100644 --- a/.gitignore +++ b/.gitignore @@ -17,4 +17,3 @@ Cargo.lock flamegraph.svg workflow_tests/docs lib/docs -run-tests.zsh diff --git a/run-tests.zsh b/run-tests.zsh new file mode 100755 index 000000000..d3f169171 --- /dev/null +++ b/run-tests.zsh @@ -0,0 +1 @@ +cargo test -p xvc --test test_pipeline_dag # -p xvc --test test_pipeline_run -p xvc --test test_storage_new_minio -p xvc-core --lib -p xvc-walker --lib -p xvc-file --doc From d26def3f1c7c5f43d481f105eaeb24348f88923f Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Fri, 30 Aug 2024 13:45:13 +0300 Subject: [PATCH 220/257] test --- run-tests.zsh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/run-tests.zsh b/run-tests.zsh index d3f169171..8ffddb3c6 100755 --- a/run-tests.zsh +++ b/run-tests.zsh @@ -1 +1 @@ -cargo test -p xvc --test test_pipeline_dag # -p xvc --test test_pipeline_run -p xvc --test test_storage_new_minio -p xvc-core --lib -p xvc-walker --lib -p xvc-file --doc +cargo test -p xvc --test test_pipeline_run # -p xvc --test test_storage_new_minio -p xvc-core --lib -p xvc-walker --lib -p xvc-file --doc From 90da278eaeb1d3b9c3d1c1782f93f4aad356a995 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Fri, 30 Aug 2024 13:50:38 +0300 Subject: [PATCH 221/257] test --- run-tests.zsh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/run-tests.zsh b/run-tests.zsh index 8ffddb3c6..6a771b9cf 100755 --- a/run-tests.zsh +++ b/run-tests.zsh @@ -1 +1 @@ -cargo test -p xvc --test test_pipeline_run # -p xvc --test test_storage_new_minio -p xvc-core --lib -p xvc-walker --lib -p xvc-file --doc +cargo test -p xvc --test test_storage_new_minio # -p xvc-core --lib -p xvc-walker --lib -p xvc-file --doc From 3ed4a27e37f778e8f64e251a3f535d8026892876 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Fri, 30 Aug 2024 13:51:42 +0300 Subject: [PATCH 222/257] minio --- run-tests.zsh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/run-tests.zsh b/run-tests.zsh index 6a771b9cf..5ae1853f1 100755 --- a/run-tests.zsh +++ b/run-tests.zsh @@ -1 +1 @@ -cargo test -p xvc --test test_storage_new_minio # -p xvc-core --lib -p xvc-walker --lib -p xvc-file --doc +cargo test --features=test-minio -p xvc --test test_storage_new_minio # -p xvc-core --lib -p xvc-walker --lib -p xvc-file --doc From 80f8d0ae371b029c1aa7993f5482c7cf6cc104a4 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Fri, 30 Aug 2024 13:56:14 +0300 Subject: [PATCH 223/257] fix urls --- book/src/ref/xvc-storage-new-minio.md | 5 +---- lib/tests/test_storage_new_minio.rs | 4 ++-- 2 files changed, 3 insertions(+), 6 deletions(-) diff --git a/book/src/ref/xvc-storage-new-minio.md b/book/src/ref/xvc-storage-new-minio.md index 5110776a7..d2688adde 100644 --- a/book/src/ref/xvc-storage-new-minio.md +++ b/book/src/ref/xvc-storage-new-minio.md @@ -78,7 +78,7 @@ $ xvc file track dir-0001 You can define a storage bucket as storage and begin to use it. ```console,ignore -$ xvc storage new minio --name backup --endpoint http://emresult.com:9000 --bucket-name one --region us-east-1 --storage-prefix xvc +$ xvc storage new minio --name backup --endpoint http://e1.xvc.dev:9000 --bucket-name xvc-tests --region us-east-1 --storage-prefix xvc ``` @@ -147,7 +147,6 @@ You may need to consider this when you have servers running in exact URLs. If you have a `http://minio.example.com:9001` as a Minio server, you may want to supply `http://example.com:9001` as the endpoint, and `minio` as the bucket name to form the correct URL. This behavior may change in the future. - ## Technical Details This command requires Xvc to be compiled with `minio` feature, which is _on_ by default. @@ -162,5 +161,3 @@ A file that's found in `.xvc/{{HASH_PREFIX}}/{{CACHE_PATH}}` is saved to `http:/ `{{REPO_ID}}` is the unique identifier for the repository created during `xvc init`. Hence if you use a common storage for different Xvc projects, their files are kept under different directories. There is no inter-project deduplication. - - diff --git a/lib/tests/test_storage_new_minio.rs b/lib/tests/test_storage_new_minio.rs index 48c703605..5ea698a2d 100644 --- a/lib/tests/test_storage_new_minio.rs +++ b/lib/tests/test_storage_new_minio.rs @@ -134,8 +134,8 @@ fn sh(cmd: String) -> String { fn test_storage_new_minio() -> Result<()> { common::test_logging(LevelFilter::Trace); let xvc_root = create_directory_hierarchy()?; - let endpoint = "http://emresult.com:9000"; - let bucket_name = "one"; + let endpoint = "http://e1.xvc.dev:9000"; + let bucket_name = "xvc-tests"; let storage_prefix = common::random_dir_name("xvc", None); let region = "us-east-1"; let local_test_dir = env::temp_dir().join(common::random_dir_name("xvc-storage-copy", None)); From 365c0865fdfd2295d418c1a4e81d493421472e8f Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Fri, 30 Aug 2024 13:57:51 +0300 Subject: [PATCH 224/257] test --- run-tests.zsh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/run-tests.zsh b/run-tests.zsh index 5ae1853f1..020c5d423 100755 --- a/run-tests.zsh +++ b/run-tests.zsh @@ -1 +1 @@ -cargo test --features=test-minio -p xvc --test test_storage_new_minio # -p xvc-core --lib -p xvc-walker --lib -p xvc-file --doc +cargo test --features=test-minio -p xvc --test test_storage_new_minio # -p xvc-core --lib -p xvc-walker --lib -p xvc-file --doc # From 953400e5bcdb870ce03d2282d4734aa07dbe0b23 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Fri, 30 Aug 2024 13:59:26 +0300 Subject: [PATCH 225/257] test --- run-tests.zsh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/run-tests.zsh b/run-tests.zsh index 020c5d423..5ae1853f1 100755 --- a/run-tests.zsh +++ b/run-tests.zsh @@ -1 +1 @@ -cargo test --features=test-minio -p xvc --test test_storage_new_minio # -p xvc-core --lib -p xvc-walker --lib -p xvc-file --doc # +cargo test --features=test-minio -p xvc --test test_storage_new_minio # -p xvc-core --lib -p xvc-walker --lib -p xvc-file --doc From 28efd73e8fe0ce6b7e709dcf0b4e5896d5396de1 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Fri, 30 Aug 2024 14:00:53 +0300 Subject: [PATCH 226/257] fix bucket name --- lib/tests/test_storage_new_minio.rs | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/lib/tests/test_storage_new_minio.rs b/lib/tests/test_storage_new_minio.rs index 5ea698a2d..608c1e0b3 100644 --- a/lib/tests/test_storage_new_minio.rs +++ b/lib/tests/test_storage_new_minio.rs @@ -196,7 +196,10 @@ fn test_storage_new_minio() -> Result<()> { let push_result = x(&["file", "send", "--to", "minio-storage", the_file])?; watch!(push_result); - let file_list = s3cmd(&format!("ls -r s3://one/{storage_prefix}"), "| rg 0.bin"); + let file_list = s3cmd( + &format!("ls -r s3://xvc-tests/{storage_prefix}"), + "| rg 0.bin", + ); watch!(file_list); // The file should be in: From cfbfe3e7c7d9c9d9b8a1de1be38f166934411cbf Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Fri, 30 Aug 2024 14:01:32 +0300 Subject: [PATCH 227/257] test --- run-tests.zsh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/run-tests.zsh b/run-tests.zsh index 5ae1853f1..8d6495f8d 100755 --- a/run-tests.zsh +++ b/run-tests.zsh @@ -1 +1 @@ -cargo test --features=test-minio -p xvc --test test_storage_new_minio # -p xvc-core --lib -p xvc-walker --lib -p xvc-file --doc +cargo test --features=test-minio -p xvc-core --lib # -p xvc-walker --lib -p xvc-file --doc From 236f3fad7c2c2e7a79df6f2a0d1f25da193e8f93 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Fri, 30 Aug 2024 14:01:52 +0300 Subject: [PATCH 228/257] features --- run-tests.zsh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/run-tests.zsh b/run-tests.zsh index 8d6495f8d..fa151820b 100755 --- a/run-tests.zsh +++ b/run-tests.zsh @@ -1 +1 @@ -cargo test --features=test-minio -p xvc-core --lib # -p xvc-walker --lib -p xvc-file --doc +cargo test -p xvc-core --lib # -p xvc-walker --lib -p xvc-file --doc From ea8544ca6851f0fbfb95ca95ba94b8b7aa2fc448 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Fri, 30 Aug 2024 14:49:44 +0300 Subject: [PATCH 229/257] fix tests --- core/src/util/git.rs | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/core/src/util/git.rs b/core/src/util/git.rs index 2a1cfd274..2fac41111 100644 --- a/core/src/util/git.rs +++ b/core/src/util/git.rs @@ -55,7 +55,7 @@ mod test { #[test_case("mydir/myfile.txt" , ".gitignore", "mydir/*" => matches M::Ignore ; "ignore dir star")] #[test_case("mydir/yourdir/myfile.txt" , "mydir/.gitignore", "yourdir/**" => matches M::Ignore ; "ignore deep dir star 2")] #[test_case("mydir/yourdir/myfile.txt" , "mydir/.gitignore", "yourdir/*" => matches M::Ignore ; "ignore deep dir star")] - #[test_case("mydir/myfile.txt" , "hebelep/.gitignore", "hebelep/myfile.txt" => matches M::NoMatch ; "non ignore from dir")] + #[test_case("mydir/myfile.txt" , "another-dir/.gitignore", "another-dir/myfile.txt" => matches M::NoMatch ; "non ignore from dir")] fn test_gitignore(path: &str, gitignore_path: &str, ignore_line: &str) -> M { test_logging(log::LevelFilter::Trace); let git_root = temp_git_dir(); @@ -66,8 +66,7 @@ mod test { } fs::write(&gitignore_path, format!("{}\n", ignore_line)).unwrap(); - let gitignore = - build_ignore_patterns("", gitignore_path.parent().unwrap(), ".gitignore").unwrap(); + let gitignore = build_ignore_patterns("", &git_root, ".gitignore").unwrap(); watch!(gitignore); From 265e35d22f0776a960f6e7704f7c54e6cd86bfc3 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Fri, 30 Aug 2024 15:50:28 +0300 Subject: [PATCH 230/257] fix gitignore traversal --- core/src/util/git.rs | 8 +++++++- walker/src/lib.rs | 5 ++++- 2 files changed, 11 insertions(+), 2 deletions(-) diff --git a/core/src/util/git.rs b/core/src/util/git.rs index 2fac41111..5f738c4ca 100644 --- a/core/src/util/git.rs +++ b/core/src/util/git.rs @@ -53,18 +53,24 @@ mod test { #[test_case("mydir/myfile.txt" , ".gitignore", "" => matches M::NoMatch ; "non ignore")] #[test_case("mydir/myfile.txt" , ".gitignore", "mydir/**" => matches M::Ignore ; "ignore dir star 2")] #[test_case("mydir/myfile.txt" , ".gitignore", "mydir/*" => matches M::Ignore ; "ignore dir star")] - #[test_case("mydir/yourdir/myfile.txt" , "mydir/.gitignore", "yourdir/**" => matches M::Ignore ; "ignore deep dir star 2")] #[test_case("mydir/yourdir/myfile.txt" , "mydir/.gitignore", "yourdir/*" => matches M::Ignore ; "ignore deep dir star")] + #[test_case("mydir/yourdir/myfile.txt" , "mydir/.gitignore", "yourdir/**" => matches M::Ignore ; "ignore deep dir star 2")] #[test_case("mydir/myfile.txt" , "another-dir/.gitignore", "another-dir/myfile.txt" => matches M::NoMatch ; "non ignore from dir")] fn test_gitignore(path: &str, gitignore_path: &str, ignore_line: &str) -> M { test_logging(log::LevelFilter::Trace); let git_root = temp_git_dir(); + watch!(git_root); let path = git_root.join(PathBuf::from(path)); + watch!(path); let gitignore_path = git_root.join(PathBuf::from(gitignore_path)); + watch!(gitignore_path); if let Some(ignore_dir) = gitignore_path.parent() { + watch!(ignore_dir); fs::create_dir_all(ignore_dir).unwrap(); + watch!(ignore_dir.exists()); } fs::write(&gitignore_path, format!("{}\n", ignore_line)).unwrap(); + watch!(gitignore_path.exists()); let gitignore = build_ignore_patterns("", &git_root, ".gitignore").unwrap(); diff --git a/walker/src/lib.rs b/walker/src/lib.rs index c8e9ff15d..57f00a92b 100755 --- a/walker/src/lib.rs +++ b/walker/src/lib.rs @@ -160,8 +160,11 @@ pub fn build_ignore_patterns( let new_patterns = content_to_patterns(ignore_root, Some(&ignore_filename), &ignore_content); ignore_rules.add_patterns(new_patterns)?; - dir_stack.extend(dirs_under(&dir)); } + let mut new_dirs = dirs_under(&dir); + watch!(new_dirs); + dir_stack.append(&mut new_dirs); + watch!(dir_stack); } Ok(ignore_rules) From 070f4e84b3c567938b33e3446d3beb65eb308dff Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Fri, 30 Aug 2024 15:50:57 +0300 Subject: [PATCH 231/257] test --- run-tests.zsh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/run-tests.zsh b/run-tests.zsh index fa151820b..33f99614e 100755 --- a/run-tests.zsh +++ b/run-tests.zsh @@ -1 +1 @@ -cargo test -p xvc-core --lib # -p xvc-walker --lib -p xvc-file --doc +cargo test -p xvc-walker --lib # -p xvc-file --doc From e27d57bdfd085eb631041d43ec68db221869574c Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Fri, 30 Aug 2024 16:19:06 +0300 Subject: [PATCH 232/257] remove some unnecessary tests --- walker/src/lib.rs | 54 ++++++++++++++++++++++++++++++++++++----------- 1 file changed, 42 insertions(+), 12 deletions(-) diff --git a/walker/src/lib.rs b/walker/src/lib.rs index 57f00a92b..1ae5939aa 100755 --- a/walker/src/lib.rs +++ b/walker/src/lib.rs @@ -387,18 +387,48 @@ mod tests { let pattern = Pattern::new(source, pattern); pattern.relativity } - - #[test_case("", "myfile" => "myfile" ; "t1142345310")] - #[test_case("", "/myfile" => "myfile" ; "t1427001291")] - #[test_case("", "myfile/" => "myfile" ; "t789151905")] - #[test_case("", "mydir/myfile" => "mydir/myfile" ; "t21199018162")] - #[test_case("", "myfile.*" => "myfile.*" ; "t31199018162")] - #[test_case("", "mydir/**.*" => "mydir/**.*" ; "t41199018162")] - #[test_case("dir", "myfile" => "myfile" ; "t1242345310")] - #[test_case("dir", "/myfile" => "myfile" ; "t3427001291")] - #[test_case("dir", "myfile/" => "myfile" ; "t759151905")] - #[test_case("dir", "mydir/myfile" => "mydir/myfile" ; "t21199018562")] - #[test_case("dir", "/my/file.*" => "my/file.*" ; "t61199018162")] + // ---- tests::test_pattern_line::t1242345310 stdout ---- + // thread 'tests::test_pattern_line::t1242345310' panicked at walker/src/lib.rs:391:5: + // assertion `left == right` failed + // left: "myfile" + // right: "**/myfile" + // + // ---- tests::test_pattern_line::t1142345310 stdout ---- + // thread 'tests::test_pattern_line::t1142345310' panicked at walker/src/lib.rs:391:5: + // assertion `left == right` failed + // left: "myfile" + // right: "**/myfile" + // note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace + // + // ---- tests::test_pattern_line::t1427001291 stdout ---- + // thread 'tests::test_pattern_line::t1427001291' panicked at walker/src/lib.rs:391:5: + // assertion `left == right` failed + // left: "myfile" + // right: "/**/myfile" + // + // ---- tests::test_pattern_line::t21199018562 stdout ---- + // thread 'tests::test_pattern_line::t21199018562' panicked at walker/src/lib.rs:391:5: + // assertion `left == right` failed + // left: "mydir/myfile" + // right: "/**/mydir/myfile" + // + // ---- tests::test_pattern_line::t21199018162 stdout ---- + // thread 'tests::test_pattern_line::t21199018162' panicked at walker/src/lib.rs:391:5: + // assertion `left == right` failed + // left: "mydir/myfile" + // right: "/**/mydir/myfile" + // + // ---- tests::test_pattern_line::t31199018162 stdout ---- + // thread 'tests::test_pattern_line::t31199018162' panicked at walker/src/lib.rs:391:5: + // assertion `left == right` failed + // left: "myfile.*" + // right: "**/myfile.*" + // + #[test_case("dir", "myfile" => "**/myfile" ; "t1242345310")] + #[test_case("dir", "/myfile" => "/**/myfile" ; "t3427001291")] + #[test_case("dir", "myfile/" => "**/myfile/**" ; "t759151905")] + #[test_case("dir", "mydir/myfile" => "/**/mydir/myfile" ; "t21199018562")] + #[test_case("dir", "/my/file.*" => "/**/my/file.*" ; "t61199018162")] #[test_case("dir", "/mydir/**.*" => "mydir/**.*" ; "t47199018162")] fn test_pattern_line(dir: &str, pattern: &str) -> String { let source = Source::File { From ca5d53417c06c502fce34ff9cf3d8d16355f4943 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Fri, 30 Aug 2024 16:19:35 +0300 Subject: [PATCH 233/257] fix test --- walker/src/lib.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/walker/src/lib.rs b/walker/src/lib.rs index 1ae5939aa..e87a222cb 100755 --- a/walker/src/lib.rs +++ b/walker/src/lib.rs @@ -429,7 +429,7 @@ mod tests { #[test_case("dir", "myfile/" => "**/myfile/**" ; "t759151905")] #[test_case("dir", "mydir/myfile" => "/**/mydir/myfile" ; "t21199018562")] #[test_case("dir", "/my/file.*" => "/**/my/file.*" ; "t61199018162")] - #[test_case("dir", "/mydir/**.*" => "mydir/**.*" ; "t47199018162")] + #[test_case("dir", "/mydir/**.*" => "/**/mydir/**.*" ; "t47199018162")] fn test_pattern_line(dir: &str, pattern: &str) -> String { let source = Source::File { path: PathBuf::from(format!("{dir}.gitignore")), From 539b79a7d273cac5224767b3df2dbc78b4282dd4 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Fri, 30 Aug 2024 16:20:49 +0300 Subject: [PATCH 234/257] tests --- run-tests.zsh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/run-tests.zsh b/run-tests.zsh index 33f99614e..d283be909 100755 --- a/run-tests.zsh +++ b/run-tests.zsh @@ -1 +1 @@ -cargo test -p xvc-walker --lib # -p xvc-file --doc +cargo test -p xvc-file --doc From 5efb6338042721c71d83585125d118e3cec0f18c Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Fri, 30 Aug 2024 16:28:37 +0300 Subject: [PATCH 235/257] update --- file/src/common/mod.rs | 8 -------- 1 file changed, 8 deletions(-) diff --git a/file/src/common/mod.rs b/file/src/common/mod.rs index 67fc54e29..617eae0f6 100644 --- a/file/src/common/mod.rs +++ b/file/src/common/mod.rs @@ -215,14 +215,6 @@ pub fn filter_paths_by_globs( /// /// This function will return an error if any of the glob patterns are invalid. /// -/// # Example -/// -/// ``` -/// let output_snd = ...; // Some XvcOutputSender -/// let dir = Path::new("/path/to/dir"); -/// let globs = vec!["*.rs", "src/"]; -/// let matcher = build_glob_matcher(&output_snd, &dir, &globs); -/// ``` pub fn build_glob_matcher( output_snd: &XvcOutputSender, From 9d006b6e6904744fa237e00d84d749113944eb09 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Fri, 30 Aug 2024 16:30:17 +0300 Subject: [PATCH 236/257] changelog --- CHANGELOG.md | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 82aa887c0..9e66adf3d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,7 @@ - Fixed xvc check-ignore to work with supplied paths - Fixed loading store targets from xvc file list - Directory targets in various commands doesn't require / at the end when they only exist in the cache +- Removed some duplicate tests from ignore ## 0.6.10 (2024-08-04) From 6749395e56e47af6113d2bb1952cba52c02b19ec Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sun, 1 Sep 2024 16:09:47 +0300 Subject: [PATCH 237/257] fail fast in tests --- .github/workflows/rust.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/rust.yml b/.github/workflows/rust.yml index bfa3b31bd..e1a05ee76 100644 --- a/.github/workflows/rust.yml +++ b/.github/workflows/rust.yml @@ -28,7 +28,7 @@ jobs: - build: nightly rust: nightly # rust: nightly-2024-01-01 - test-args: --features test-ci --no-fail-fast + test-args: --features test-ci # --no-fail-fast ## for submitters other than me, I'll add another job here. # test-args: --no-fail-fast # test-args: --all-features From 66e6033a9c22b903903cfd9b9bf35421382a4418 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sun, 1 Sep 2024 16:11:31 +0300 Subject: [PATCH 238/257] test local --- run-tests.zsh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/run-tests.zsh b/run-tests.zsh index d283be909..1c6c754be 100755 --- a/run-tests.zsh +++ b/run-tests.zsh @@ -1 +1 @@ -cargo test -p xvc-file --doc +cargo test --features test-ci From b499b187f277bf230260e24c55a82c5da4473f6c Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sun, 1 Sep 2024 16:20:48 +0300 Subject: [PATCH 239/257] updated doc tests --- book/src/ref/xvc-storage-new-rsync.md | 2 +- run-tests.zsh | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/book/src/ref/xvc-storage-new-rsync.md b/book/src/ref/xvc-storage-new-rsync.md index 109768b66..4535fb3a7 100644 --- a/book/src/ref/xvc-storage-new-rsync.md +++ b/book/src/ref/xvc-storage-new-rsync.md @@ -17,7 +17,7 @@ Usage: xvc storage new rsync [OPTIONS] --name --host --storage-dir Options: -n, --name Name of the storage. - + Recommended to keep this name unique to refer easily. --host diff --git a/run-tests.zsh b/run-tests.zsh index 1c6c754be..d6eef43a0 100755 --- a/run-tests.zsh +++ b/run-tests.zsh @@ -1 +1 @@ -cargo test --features test-ci +XVC_TRYCMD_TESTS=storage,file,pipeline,core,start TRYCMD=overwrite cargo test -p xvc --test z_test_docs From 97d79c25eb8e16aeac88d76a6d8e07f6c610cf9e Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sun, 1 Sep 2024 16:37:24 +0300 Subject: [PATCH 240/257] update test --- run-tests.zsh | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/run-tests.zsh b/run-tests.zsh index d6eef43a0..bcfb23fe8 100755 --- a/run-tests.zsh +++ b/run-tests.zsh @@ -1 +1,3 @@ -XVC_TRYCMD_TESTS=storage,file,pipeline,core,start TRYCMD=overwrite cargo test -p xvc --test z_test_docs +# XVC_TRYCMD_TESTS=storage,file,pipeline,core,start TRYCMD=overwrite cargo test -p xvc --test z_test_docs + +cargo test -p xvc --test test_storage_new_minio From 06c88a428523c066dacbe4282380690f6f4e9c84 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sun, 1 Sep 2024 16:37:51 +0300 Subject: [PATCH 241/257] test --- run-tests.zsh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/run-tests.zsh b/run-tests.zsh index bcfb23fe8..f84e8cece 100755 --- a/run-tests.zsh +++ b/run-tests.zsh @@ -1,3 +1,3 @@ # XVC_TRYCMD_TESTS=storage,file,pipeline,core,start TRYCMD=overwrite cargo test -p xvc --test z_test_docs -cargo test -p xvc --test test_storage_new_minio +cargo test --features test-ci -p xvc --test test_storage_new_minio From 55add131b400da1caf6af5c267fd667eba0dec39 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sun, 1 Sep 2024 16:56:53 +0300 Subject: [PATCH 242/257] config --- lib/tests/test_storage_new_minio.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/tests/test_storage_new_minio.rs b/lib/tests/test_storage_new_minio.rs index 608c1e0b3..bc9f94602 100644 --- a/lib/tests/test_storage_new_minio.rs +++ b/lib/tests/test_storage_new_minio.rs @@ -24,7 +24,7 @@ add_headers = bucket_location = us-east-1 ca_certs_file = cache_file = -check_ssl_certificate = True +check_ssl_certificate = False check_ssl_hostname = True cloudfront_host = cloudfront.amazonaws.com connection_max_age = 5 From 80e7c65dffc817a26054e89a6bca92f6d7d50d9d Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Sun, 1 Sep 2024 17:00:45 +0300 Subject: [PATCH 243/257] update --- lib/tests/test_storage_new_minio.rs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/tests/test_storage_new_minio.rs b/lib/tests/test_storage_new_minio.rs index bc9f94602..102ea0689 100644 --- a/lib/tests/test_storage_new_minio.rs +++ b/lib/tests/test_storage_new_minio.rs @@ -24,7 +24,7 @@ add_headers = bucket_location = us-east-1 ca_certs_file = cache_file = -check_ssl_certificate = False +check_ssl_certificate = True check_ssl_hostname = True cloudfront_host = cloudfront.amazonaws.com connection_max_age = 5 @@ -135,7 +135,7 @@ fn test_storage_new_minio() -> Result<()> { common::test_logging(LevelFilter::Trace); let xvc_root = create_directory_hierarchy()?; let endpoint = "http://e1.xvc.dev:9000"; - let bucket_name = "xvc-tests"; + let bucket_name = "xvctests"; let storage_prefix = common::random_dir_name("xvc", None); let region = "us-east-1"; let local_test_dir = env::temp_dir().join(common::random_dir_name("xvc-storage-copy", None)); From bca12b665bb89bcbf54ad55086659567b2903853 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Tue, 3 Sep 2024 13:04:49 +0300 Subject: [PATCH 244/257] Update minio tests --- .github/workflows/rust.yml | 1 + CHANGELOG.md | 1 + lib/tests/test_storage_new_minio.rs | 132 +++------------------------- 3 files changed, 16 insertions(+), 118 deletions(-) diff --git a/.github/workflows/rust.yml b/.github/workflows/rust.yml index e1a05ee76..e0a181342 100644 --- a/.github/workflows/rust.yml +++ b/.github/workflows/rust.yml @@ -67,6 +67,7 @@ jobs: brew install tree brew install lsd brew install python3 + brew install minio/stable/mc - name: Git config for automated Git tests run: git config --global user.name 'Xvc Rabbit' && git config --global user.email 'rabbit@xvc.dev' && git config --global init.defaultBranch main diff --git a/CHANGELOG.md b/CHANGELOG.md index 9e66adf3d..1afb0ee1c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ - Fixed loading store targets from xvc file list - Directory targets in various commands doesn't require / at the end when they only exist in the cache - Removed some duplicate tests from ignore +- Minio tests now use mc instead of s3cmd ## 0.6.10 (2024-08-04) diff --git a/lib/tests/test_storage_new_minio.rs b/lib/tests/test_storage_new_minio.rs index 102ea0689..e750b0a98 100644 --- a/lib/tests/test_storage_new_minio.rs +++ b/lib/tests/test_storage_new_minio.rs @@ -10,108 +10,6 @@ use xvc_config::XvcVerbosity; use xvc_core::XvcRoot; use xvc_test_helper::{create_directory_tree, generate_filled_file}; -fn write_s3cmd_config(access_key: &str, secret_key: &str) -> Result { - let config_file_name = env::temp_dir().join(format!( - "{}.cfg", - common::random_dir_name("minio-config", None) - )); - let config = format!( - r#"[default] -access_key = {access_key} -access_token = -add_encoding_exts = -add_headers = -bucket_location = us-east-1 -ca_certs_file = -cache_file = -check_ssl_certificate = True -check_ssl_hostname = True -cloudfront_host = cloudfront.amazonaws.com -connection_max_age = 5 -connection_pooling = True -content_disposition = -content_type = -default_mime_type = binary/octet-stream -delay_updates = False -delete_after = False -delete_after_fetch = False -delete_removed = False -dry_run = False -enable_multipart = True -encoding = UTF-8 -encrypt = False -expiry_date = -expiry_days = -expiry_prefix = -follow_symlinks = False -force = False -get_continue = False -gpg_command = None -gpg_decrypt = %(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s -gpg_encrypt = %(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s -gpg_passphrase = -guess_mime_type = True -host_base = e1.xvc.dev:9000 -host_bucket = e1.xvc.dev:9000 -human_readable_sizes = False -invalidate_default_index_on_cf = False -invalidate_default_index_root_on_cf = True -invalidate_on_cf = False -kms_key = -limit = -1 -limitrate = 0 -list_allow_unordered = False -list_md5 = False -log_target_prefix = -long_listing = False -max_delete = -1 -mime_type = -multipart_chunk_size_mb = 15 -multipart_copy_chunk_size_mb = 1024 -multipart_max_chunks = 10000 -preserve_attrs = True -progress_meter = True -proxy_host = -proxy_port = 0 -public_url_use_https = False -put_continue = False -recursive = False -recv_chunk = 65536 -reduced_redundancy = False -requester_pays = False -restore_days = 1 -restore_priority = Standard -secret_key = {secret_key} -send_chunk = 65536 -server_side_encryption = False -signature_v2 = False -signurl_use_https = False -simpledb_host = sdb.amazonaws.com -skip_existing = False -socket_timeout = 300 -ssl_client_cert_file = -ssl_client_key_file = -stats = False -stop_on_error = False -storage_class = -throttle_max = 100 -upload_id = -urlencoding_mode = normal -use_http_expect = False -use_https = False -use_mime_magic = True -verbosity = WARNING -website_endpoint = http://%(bucket)s.s3-website-%(location)s.amazonaws.com/ -website_error = -website_index = index.html -"# - ); - - fs::write(&config_file_name, config)?; - - Ok(config_file_name.to_string_lossy().to_string()) -} - fn create_directory_hierarchy() -> Result { let temp_dir: XvcRoot = run_in_temp_xvc_dir()?; // for checking the content hash @@ -141,12 +39,17 @@ fn test_storage_new_minio() -> Result<()> { let local_test_dir = env::temp_dir().join(common::random_dir_name("xvc-storage-copy", None)); let access_key = env::var("MINIO_ACCESS_KEY_ID")?; let secret_key = env::var("MINIO_SECRET_ACCESS_KEY")?; - - let config_file_name = write_s3cmd_config(&access_key, &secret_key)?; - watch!(config_file_name); - - let s3cmd = |cmd: &str, append: &str| -> String { - let sh_cmd = format!("s3cmd --config {config_file_name} {cmd} {append}"); + let alias_name = "xvc"; + + let mc_aliases = sh("mc alias list".to_owned()); + if !mc_aliases.contains(&access_key) { + let mc_alias = sh(format!( + "alias set {alias_name} {endpoint} {access_key} {secret_key}" + )); + watch!(mc_alias); + } + let mc = |cmd: &str, append: &str| -> String { + let sh_cmd = format!("mc {cmd} {alias_name} {append}"); sh(sh_cmd) }; @@ -154,10 +57,6 @@ fn test_storage_new_minio() -> Result<()> { common::run_xvc(Some(&xvc_root), cmd, XvcVerbosity::Warn) }; - let create_bucket_res = s3cmd(&format!("mb {bucket_name}"), ""); - - watch!(create_bucket_res); - let out = x(&[ "storage", "new", @@ -176,7 +75,7 @@ fn test_storage_new_minio() -> Result<()> { watch!(out); - let mc_bucket_list = s3cmd("ls xvc", &format!("| rg {bucket_name}")); + let mc_bucket_list = mc("ls", &format!("| rg {bucket_name}")); watch!(mc_bucket_list); assert!(!mc_bucket_list.is_empty()); @@ -196,10 +95,7 @@ fn test_storage_new_minio() -> Result<()> { let push_result = x(&["file", "send", "--to", "minio-storage", the_file])?; watch!(push_result); - let file_list = s3cmd( - &format!("ls -r s3://xvc-tests/{storage_prefix}"), - "| rg 0.bin", - ); + let file_list = mc("ls -r ", &format!("| rg {bucket_name}/{storage_prefix}")); watch!(file_list); // The file should be in: @@ -208,7 +104,7 @@ fn test_storage_new_minio() -> Result<()> { let n_storage_files_after = file_list.lines().count(); assert!( - n_storage_files_before + 1 == n_storage_files_after, + n_storage_files_before + 2 == n_storage_files_after, "{} - {}", n_storage_files_before, n_storage_files_after From 5dbeb341a266699b09e736bc20041b09b20d6d61 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Tue, 3 Sep 2024 19:16:47 +0300 Subject: [PATCH 245/257] fix alias command --- lib/tests/test_storage_new_minio.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/tests/test_storage_new_minio.rs b/lib/tests/test_storage_new_minio.rs index e750b0a98..2d1a8a580 100644 --- a/lib/tests/test_storage_new_minio.rs +++ b/lib/tests/test_storage_new_minio.rs @@ -44,7 +44,7 @@ fn test_storage_new_minio() -> Result<()> { let mc_aliases = sh("mc alias list".to_owned()); if !mc_aliases.contains(&access_key) { let mc_alias = sh(format!( - "alias set {alias_name} {endpoint} {access_key} {secret_key}" + "mc alias set {alias_name} {endpoint} {access_key} {secret_key}" )); watch!(mc_alias); } From 9131e85aefc2535b68d47d60467484f116ff6abb Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Wed, 4 Sep 2024 09:54:21 +0300 Subject: [PATCH 246/257] fix --- lib/tests/test_storage_new_minio.rs | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/tests/test_storage_new_minio.rs b/lib/tests/test_storage_new_minio.rs index 2d1a8a580..00a40648b 100644 --- a/lib/tests/test_storage_new_minio.rs +++ b/lib/tests/test_storage_new_minio.rs @@ -113,6 +113,7 @@ fn test_storage_new_minio() -> Result<()> { // remove all cache // let cache_dir = xvc_root.xvc_dir().join("b3"); + sh(format!("chmod -R +w {}", cache_dir.to_string_lossy())); sh(format!("rm -rf {}", cache_dir.to_string_lossy())); let fetch_result = x(&["file", "bring", "--no-recheck", "--from", "minio-storage"])?; From c0b8ecf9c7d8d8c1d3807941a189fb383d6e7141 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Wed, 4 Sep 2024 10:05:49 +0300 Subject: [PATCH 247/257] add another chmod --- lib/tests/test_storage_new_minio.rs | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/lib/tests/test_storage_new_minio.rs b/lib/tests/test_storage_new_minio.rs index 00a40648b..064a01599 100644 --- a/lib/tests/test_storage_new_minio.rs +++ b/lib/tests/test_storage_new_minio.rs @@ -129,9 +129,15 @@ fn test_storage_new_minio() -> Result<()> { }) .count(); - assert!(n_storage_files_after == n_local_files_after_fetch); + assert!( + n_storage_files_after == n_local_files_after_fetch, + "{} - {}", + n_storage_files_after, + n_local_files_after_fetch + ); let cache_dir = xvc_root.xvc_dir().join("b3"); + sh(format!("chmod -R +w {}", cache_dir.to_string_lossy())); sh(format!("rm -rf {}", cache_dir.to_string_lossy())); fs::remove_file(the_file)?; @@ -147,7 +153,7 @@ fn test_storage_new_minio() -> Result<()> { }) .count(); - assert!(n_storage_files_after == n_local_files_after_pull); + assert!(n_storage_files_after == n_local_files_after_pull, "{} - {}", n_storage_files_after, n_local_files_after_pull) assert!(PathBuf::from(the_file).exists()); // Set remote specific passwords and remove general ones From 005757067e9ca7988297ed77e5f022cdb5706334 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Wed, 4 Sep 2024 10:06:10 +0300 Subject: [PATCH 248/257] fix --- lib/tests/test_storage_new_minio.rs | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/lib/tests/test_storage_new_minio.rs b/lib/tests/test_storage_new_minio.rs index 064a01599..2f3334b88 100644 --- a/lib/tests/test_storage_new_minio.rs +++ b/lib/tests/test_storage_new_minio.rs @@ -153,7 +153,12 @@ fn test_storage_new_minio() -> Result<()> { }) .count(); - assert!(n_storage_files_after == n_local_files_after_pull, "{} - {}", n_storage_files_after, n_local_files_after_pull) + assert!( + n_storage_files_after == n_local_files_after_pull, + "{} - {}", + n_storage_files_after, + n_local_files_after_pull + ); assert!(PathBuf::from(the_file).exists()); // Set remote specific passwords and remove general ones From edcd8f1d286244599ad1cec46b7291d57c2701bc Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Wed, 4 Sep 2024 10:07:28 +0300 Subject: [PATCH 249/257] debug --- lib/tests/test_storage_new_minio.rs | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/tests/test_storage_new_minio.rs b/lib/tests/test_storage_new_minio.rs index 2f3334b88..3f82346cd 100644 --- a/lib/tests/test_storage_new_minio.rs +++ b/lib/tests/test_storage_new_minio.rs @@ -123,6 +123,7 @@ fn test_storage_new_minio() -> Result<()> { let n_local_files_after_fetch = jwalk::WalkDir::new(&cache_dir) .into_iter() .filter(|f| { + watch!(f); f.as_ref() .map(|f| f.file_type().is_file()) .unwrap_or_else(|_| false) From 6c29d346b0926cc96dedf1ff68041b1efe321f4c Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Wed, 4 Sep 2024 10:10:07 +0300 Subject: [PATCH 250/257] change counting --- lib/tests/test_storage_new_minio.rs | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/lib/tests/test_storage_new_minio.rs b/lib/tests/test_storage_new_minio.rs index 3f82346cd..1e145dab8 100644 --- a/lib/tests/test_storage_new_minio.rs +++ b/lib/tests/test_storage_new_minio.rs @@ -101,10 +101,11 @@ fn test_storage_new_minio() -> Result<()> { // The file should be in: // - storage_dir/REPO_ID/b3/ABCD...123/0.bin - let n_storage_files_after = file_list.lines().count(); + // Remove guid file from the count + let n_storage_files_after = file_list.lines().count() - 1; assert!( - n_storage_files_before + 2 == n_storage_files_after, + n_storage_files_before + 1 == n_storage_files_after, "{} - {}", n_storage_files_before, n_storage_files_after From c1b7e05d1fdca7785728046b4c6dbf9bae1fd0e5 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Wed, 4 Sep 2024 10:10:52 +0300 Subject: [PATCH 251/257] change release date --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1afb0ee1c..b9735c6ed 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,6 @@ ## Unreleased -## 0.6.11 (2024-08-26) +## 0.6.11 (2024-09-04) - Bump dependencies - Replace globset with fast-glob for memory usage From d09162b99982731c1f0646747a7403e88e60149f Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Wed, 4 Sep 2024 13:43:40 +0300 Subject: [PATCH 252/257] run in shell --- lib/tests/test_storage_new_minio.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/tests/test_storage_new_minio.rs b/lib/tests/test_storage_new_minio.rs index 1e145dab8..127ad6269 100644 --- a/lib/tests/test_storage_new_minio.rs +++ b/lib/tests/test_storage_new_minio.rs @@ -44,7 +44,7 @@ fn test_storage_new_minio() -> Result<()> { let mc_aliases = sh("mc alias list".to_owned()); if !mc_aliases.contains(&access_key) { let mc_alias = sh(format!( - "mc alias set {alias_name} {endpoint} {access_key} {secret_key}" + "zsh -c \"mc alias set {alias_name} {endpoint} {access_key} {secret_key}\"" )); watch!(mc_alias); } From c3dfc086d0ee517644d3513995a7470472d3aace Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Fri, 6 Sep 2024 19:21:30 +0300 Subject: [PATCH 253/257] Use config file instead of mc alias --- lib/tests/test_storage_new_minio.rs | 38 +++++++++++++++++++++++------ 1 file changed, 30 insertions(+), 8 deletions(-) diff --git a/lib/tests/test_storage_new_minio.rs b/lib/tests/test_storage_new_minio.rs index 127ad6269..7c5f6c3ab 100644 --- a/lib/tests/test_storage_new_minio.rs +++ b/lib/tests/test_storage_new_minio.rs @@ -22,6 +22,25 @@ fn create_directory_hierarchy() -> Result { Ok(temp_dir) } +fn mc_config(alias: &str, endpoint: &str, access_key: &str, secret_key: &str) -> String { + format!( + r##" +{{ + "version": "10", + "aliases": {{ + "{alias}": {{ + "url": "{endpoint}", + "accessKey": "{access_key}", + "secretKey": "{secret_key}", + "api": "s3v4", + "path": "auto" + }} + }} +}} +"## + ) +} + fn sh(cmd: String) -> String { watch!(cmd); Exec::shell(cmd).capture().unwrap().stdout_str() @@ -40,16 +59,19 @@ fn test_storage_new_minio() -> Result<()> { let access_key = env::var("MINIO_ACCESS_KEY_ID")?; let secret_key = env::var("MINIO_SECRET_ACCESS_KEY")?; let alias_name = "xvc"; + let mc_config = mc_config( + alias_name, + endpoint, + access_key.as_str(), + secret_key.as_str(), + ); + let mc_config_dir = xvc_root.xvc_dir().join(".mc"); + fs::create_dir_all(&mc_config_dir)?; + let mc_config_file = mc_config_dir.join("config.json"); + fs::write(mc_config_file, mc_config)?; - let mc_aliases = sh("mc alias list".to_owned()); - if !mc_aliases.contains(&access_key) { - let mc_alias = sh(format!( - "zsh -c \"mc alias set {alias_name} {endpoint} {access_key} {secret_key}\"" - )); - watch!(mc_alias); - } let mc = |cmd: &str, append: &str| -> String { - let sh_cmd = format!("mc {cmd} {alias_name} {append}"); + let sh_cmd = format!("mc --config-dir {mc_config_dir} {cmd} {alias_name} {append}"); sh(sh_cmd) }; From 94d1713833d2f1df18f17bc12f5020c025fd02bc Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Fri, 6 Sep 2024 19:27:43 +0300 Subject: [PATCH 254/257] add one-off tests to workflow --- .github/workflows/rust.yml | 3 +++ 1 file changed, 3 insertions(+) diff --git a/.github/workflows/rust.yml b/.github/workflows/rust.yml index e0a181342..175b3e68a 100644 --- a/.github/workflows/rust.yml +++ b/.github/workflows/rust.yml @@ -106,6 +106,9 @@ jobs: - name: Check if xvc is in PATH run: tree $GITHUB_WORKSPACE && xvc --help + - name: Run Current Dev Tests + run: $GITHUB_WORKSPACE/run-tests.zsh + - name: Test if: matrix.coverage uses: actions-rs/cargo@v1 From 607c7c05123ab487cc51a1d826f3cc2499148484 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Fri, 6 Sep 2024 19:28:31 +0300 Subject: [PATCH 255/257] Update changelog --- CHANGELOG.md | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index b9735c6ed..c50563a7a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,7 @@ - Directory targets in various commands doesn't require / at the end when they only exist in the cache - Removed some duplicate tests from ignore - Minio tests now use mc instead of s3cmd +- Add a step to run a subset of tests in CI for faster feedback ## 0.6.10 (2024-08-04) From 168c3e1e1330de72c28c903bea56d4290b298c48 Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Fri, 6 Sep 2024 19:38:09 +0300 Subject: [PATCH 256/257] update test logging to trace --- lib/tests/test_storage_new_minio.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/tests/test_storage_new_minio.rs b/lib/tests/test_storage_new_minio.rs index 7c5f6c3ab..1bf4f1754 100644 --- a/lib/tests/test_storage_new_minio.rs +++ b/lib/tests/test_storage_new_minio.rs @@ -76,7 +76,7 @@ fn test_storage_new_minio() -> Result<()> { }; let x = |cmd: &[&str]| -> Result { - common::run_xvc(Some(&xvc_root), cmd, XvcVerbosity::Warn) + common::run_xvc(Some(&xvc_root), cmd, XvcVerbosity::Trace) }; let out = x(&[ From 483cbe7350bcca2f00906c521f5f9b42b70aadfe Mon Sep 17 00:00:00 2001 From: Emre Sahin Date: Mon, 9 Sep 2024 11:58:55 +0300 Subject: [PATCH 257/257] Added title to Changelog --- CHANGELOG.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index c50563a7a..d89997404 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,5 @@ +# Xvc Changelog + ## Unreleased ## 0.6.11 (2024-09-04)