Mechanism to manage a directory's content but not the directory itself

[paltherr]

Is your feature request related to a problem? Please describe.

I have a directory dir whose content I would like to manage with chezmoi. However on host A that directory is a real directory while on host B it's a symbolic link to a directory somewhere else:

user@A$ ls -dla dir
drwxr-xr-x  2 user  group  64 Jul 28 14:16 dir
user@B$ ls -dla dir
lrwxr-xr-x  1 user  group   4 Jul 28 14:16 dir -> /some/where/else/dir

Creating a directory dir in my source tree, works perfectly well for host A but not quite so for host B. It does indeed allow to manage the content of dir on both hosts. Commands like chezmoi diff dir/file or chezmoi apply dir/file have the intended effect on both hosts. The problem is that on host B, chezmoi wants to replace the symbolic link dir with a real directory dir.

Describe the solution you'd like

Since the content of the directory is handled correctly on both host and that the only problem is the handling of the directory itself, I was hoping that I could use .chezmoiignore to ignore the directory itself but not its content. The documentation of .chezmoiignore states the following (emphasize mine):

Patterns can be excluded by prefixing them with a ! character. All excludes take priority over all includes.

I was hoping that adding the following lines to .chezmoiignore would do the trick:

dir
!dir/**/*

Unfortunately this has the same effect as if the second line was absent: the directory dir and its content are ignored. I have also tried more specific exclusions like !dir/* or !dir/file but the result is the same.

Describe alternatives you've considered

A possibly cleaner solution to support this use case would be to create a new file prefix (e.g., soft_) to indicate that chezmoi should only manage the content of the target directory (or file) and not its attributes. I could then create a directory named soft_dir in my source repository. Chezmoi would then only require that there is either a target directory dir or a target symbolic link dir that points to a directory. If that isn't the case, it would be an error and chezmoi could offer to create a directory or skip it.

The same prefix could also be useful to manage cases where the target directory (or file) has an unsupported set of file permissions, like for example if it's group writeable.

[twpayne]

This is really hard to implement given chezmoi's architecture. Basically, chezmoi can cope with entries having different contents across machines, but chezmoi cannot cope with entries being different types across machines (e.g. a symlink on one and a directory on another).

Instead, I suggest a different approach. Make dir a symbolic link on both machines. Symbolic links in chezmoi can be templates so you can make dir point to a subdirectory on machine A and point to /some/where/else/dir on machine B.

[paltherr]

Your workaround doesn't solve my problem, which isn't to manage the directory dir itself. What I want to be able to do is to manage the content of dir, i.e., the files and directories that it contains.

Forget about the two hosts and consider the case where dir is a symbolic link in my home directory that points to the directory /mnt/dir/:

user@host ~ $ ls -dla dir
lrwxr-xr-x  1 user  group  8 Jul 28 21:25 dir -> /mnt/dir

That link can be assumed to already exist. I don't need to manage it with chezmoi. However I want to be able to manage files under ~/dir/. This is actually already possible. In my chezmoi source directory, I can create a (real) directory dir/ and a file dir/file.txt. I can then run chezmoi apply ~/dir/file.txt. This will correctly create and/or update the file ~/dir/file.txt, which technically is the file /mnt/dir/file.txt. In other words, while updating dir/file.txt, chezmoi doesn't care that dir is a real directory in its source directory and a symbolic link in its destination directory. And that's exactly what I need/want.

My problem is that chezmoi thinks that it has to replace the symbolic link ~/dir with a real directory, which I don't want. In other words, my only problem is that if I ever accidentally run chezmoi apply it would perform that replacement. Everything else works. Chezmoi correctly handles files and directories under dir/, as long as I remember to never update dir/ itself.

A possible workaround would be to tell chezmoi to ignore changes affecting dir/itself but to keep applying changes to files and directories contained in dir/. This could be achieved by adding the following lines in the .chezmoiignore:

dir
!dir/**/*

Unfortunately, at the moment, the second line doesn't have any effect and all files and directories under dir/ are ignored because of the first line.

A cleaner solution would be to implement a new file name prefix to tell chezmoi that it should only manage the content of the target directory/file but that it shouldn't care about whether it's a real directory/file or a symbolic link. If the target directory/file doesn't already exist, chezmoi should report an error. It could offer to create a real directory/file.

I used soft_ for the new prefix in my original post but contentonly_ or allowsymlink_ might be better options.

[twpayne]

Your workaround doesn't solve my problem

With all due respect, this is your problem, not mine.

[twpayne]

My problem is that chezmoi thinks that it has to replace the symbolic link ~/dir with a real directory, which I don't want

The underlying problem here is that chezmoi is mostly declarative (with the exception of scripts, of course), and this mix of symbolic links and directories is not declarative. chezmoi gets you define what you want the contents of your home directory to be, i.e. what the contents of each file is, and what is a directory and what is a symlink, and chezmoi apply makes the minimum number of changes to make your home directory be exactly what you want it to be.

The problem with symbolic links and directories comes from walking the filesystem. chezmoi treats symbolic links as first-class citizens, i.e. it treats a symbolic link as a symbolic link, whereas you want chezmoi to treat a symbolic link as the directory that it points to, not as a symbolic link.

Treating symbolic links as their targets leads to a whole host of problems. For example, it's easy to create infinite loops (where a symbolic link points to one of its parent directories) or end up in a complex situation where a symbolic link points to a directory on a different filesystem.

This is why chezmoi treats symbolic links as symbolic links, and not the targets that they point to.

The proposed soft_ (or contentonly_, or allowsymlink_) attributes do not solve the fundamental problem here because what chezmoi really does is:

1. Read the contents of your home directory.
2. Separately, calculate the desired contents of your home directory.
3. Make the minimum changes to make your home directory match your desired contents.

Adding soft_ or similar would mean that step 1 depends on step 2, i.e. chezmoi would need to know, ahead of time, whether it should treat a symbolic link as a symbolic link or a directory.

[paltherr]

Sorry, I didn't mean to be offensive. My point was that you misunderstood my actual problem. I tried to re-explain it in a more straightforward way. In short, what would work is either a new file name prefix like contentonly_ to tell chezmoi to only care about the content of a directory/file and ignore whether it's a symlink or nor. Or, alternatively, a way to tell chezmoi to not try to update a directory itself but to keep updating/managing files and directories that it contains. Does one of these alternatives sound like something that could be eventually added to chezmoi?

[twpayne]

Thanks for #3123 (comment) ! Hopefully #3123 (comment) explains a bit why.

[halostatue]

I can see how this could work, but I’m not familiar enough with the state internals to be able to make it work, and it has some complexity on the source state, because I believe that symlink_ files in the source need to be files, and not directories, whereas this would be a directory declaring that it should merely exist.

If we were to approach this as a prerequisite concept, I think that this might be possible (although it will still require state updates). So, instead of the names proposed, let’s say direxists_ or exists_ is the prefix. Chezmoi could check to see if the intermediate target exists and is a directory or a symlink to a directory. If it does not exist, chezmoi would exit and indicate "we expected to exist as a writable path" or something like that.

I’m not sure that I like it, and this might work better with #2273 than trying to fit this into the attribute system as it exists now.

[paltherr]

The thing is that chezmoi seems to be already now smarter than your 3 steps. In fact it looks like implementing a contentonly_ prefix, at least for directories, could be as simple as removing the affected directory from the set of computed changes to apply.

Here is an example to show why I think that chezmoi is already now smarter than your 3 steps.

Content of home directory:

• L dir -> /mnt/dir

Content of /mnt/ directory:

• D dir/
• F dir/file1.txt

Content of chezmoi source directory:

• D dir/
• F dir/file1.txt
• D dir/sub/
• F dir/sub/file2.txt

If I follow your three steps, here is what I expect that chezmoi should propose to do:

• Replace L dir with a new D dir/
• Create F dir/file1.txt
• Create D dir/sub/
• Create F dir/sub/file2.txt

But here is the output of chezmoi status:

 M dir
 A dir/sub
 A dir/sub/file2.txt

Note that there is no mention of dir/file1.txt.

If I run chezmoi apply --interactive and skip the update for dir/ then the home directory remains unchanged and the content of /mnt/ is updated to match the content of the chezmoi source directory.

In other words, the contentonly_ prefix could simply remove the affected directory from the changes to apply if the directory already exists (either as a real directory or as a symlinnk) and chezmoi would behave exactly as desired (I think).

[paltherr]

If we were to approach this as a prerequisite concept, I think that this might be possible (although it will still require state updates). So, instead of the names proposed, let’s say direxists_ or exists_ is the prefix. Chezmoi could check to see if the intermediate target exists and is a directory or a symlink to a directory. If it does not exist, chezmoi would exit and indicate "we expected to exist as a writable path" or something like that.

Yep, that would work. It would also provide a solution for cases where the directory has an unconventional set of file permissions. You could still not create the directory via chezmoi (except maybe with an extra once_ script to set the permissions) but you could at least manage its content. The same could apply for regular files so exists_ should probably also be allowed for them.

I’m not sure that I like it, and this might work better with #2273 than trying to fit this into the attribute system as it exists now.

It's actually two distinct problems. It's true that I initially encountered the problem in a case similar to #2273 but if you look at my example in #3123 (comment), you can see that a single machine is involved.

[srouquette]

@twpayne I also had this problem. The latest solutions described here seem to fix my case as well.
The problem I have is that if the target directory doesn't exist, and there is a file in the source state, chezmoi exits with this error: Permission denied when reading file
is there a way to silence this error?