_______  ___   __    _  ___   _
|       ||   | |  |  | ||   | | |
|  _____||   | |   |_| ||   |_| |
| |_____ |   | |       ||      _|
|_____  ||   | |  _    ||     |_
 _____| ||   | | | |   ||    _  |
|_______||___| |_|  |__||___| |_|

Sink is a the swiss army knife for directory comparison and synchronization. With sink, you can:

• Compare multiple (2+) directories
• Take filesystem state snapshots
• Compare snapshots and states

Here are few uses cases where sink comes in handy:

• Compare multiple versions of a source tree (replacing diff -r): sink diff A B C D
• Manually merge directories outside of a revision control system: sink diff -r . ../project-otherbranch
• Synchronize files belonging to different directories: sink delta A B | sink patch B
• Track changes made to specific directory: sink snap . -o snapshot.lst; sink diff . snapshot.lst
• Implement incremental backup and restore: sink snap . -o ~/.backups/manifest.1.lst and later sink delta . ~/.backups/manifest.1.lst | sink patch backup.tar.bz2
• Detect changes made to configuration files: sink snap --store /etc and later sink diff --store /etc
• DIY version control system: …

Quickstart

sink simply requires Python (3+) to run. To install it, simply do:

python -m pip install --user sink

and then you're in business. Take a snapshot of your filesystem

$ sink snap ~ ~/home-$(date '+%Y%m%d').json

do some changes, and then compare it

$ sink diff ~/home-$(date '+%Y%m%d').json ~

Using sink is overall pretty straightforward:

• sink diff ORIGIN COMPARE… to compare two or more directories. Note that ORIGIN or COMPARE can be a snapshot JSON file (see below).
• sink snap DIR to take a snapshot (output as JSON) of the DIR state
• sink snap FILE.json lists the contents of the snapshot i

Use Case: Comparing directories

Imagine you have 3 different versions of a specific source tree. In this example, we'll simply checkout three different revisions of the sink development tree:

$ git clone git://github.com/sebastien/sink.git sink-r1
$ git clone git://github.com/sebastien/sink.git sink-r2
$ git clone git://github.com/sebastien/sink.git sink-r3

and we compare the three directories:

$ sink diff sink-r1 sink-r2 sink-r3
No differences

as the three directories are the same, we'll put them to different revisions:

cd sink-r1 ; git checkout -b older a17609eaabbf8feb3b480d46cb972df0599755fe ; cd ..
cd sink-r2 ; git checkout -b old   9e0e6a476f41f0de685d2c92e142b206b46810e4 ; cd ..

and we can now compare the directories:

sink diff sink-r1 sink-r2 sink-r3
00  ! [+][+] .hgtags
01 [=][-][-] DESIGN
02 [=][<][<] Makefile
03 [=][-][-] NOTES
04 -!--!-[+] README
05 [=][-][-] ROADMAP
06 [=][>][<] TODO
07 -!-[+][+] setup.py
08 -!-[+]-!- Documentation/DESIGN.txt
09 -!-[+]-!- Documentation/MANUAL.txt
10 [=][=][-] Resources/epydoc.css
11 -!-[+][+] Scripts/sink
12 [=][-][-] Sources/sink/Sink.py
13 [=][-][-] Sources/sink/Tracking.py
14 -!-[+][+] Sources/sink/linking.py
15 -!-[+][+] Sources/sink/main.py
16 -!--!-[+] Sources/sink/snapshot.py
17 -!-[+][+] Sources/sink/tracking.py

sink has found differences and indicates them in the form of a table, where the legend can be found in the sink --help diff command:

[=] no changes         [+] file added           [>] changed/newer
                       [-] file removed         [<] changed/older
                       -!- file missing

we can have a list of all the files that were added in sink-r2 and sink-r3 by showing only the added files:

$ sink diff +a sink-r1 sink-r2 sink-r3
00 -!-[+][+] .hgtags
01 -!--!-[+] README
02 -!-[+][+] setup.py
03 -!-[+]-!- Documentation/DESIGN.txt
04 -!-[+]-!- Documentation/MANUAL.txt
05 -!-[+][+] Scripts/sink
06 -!-[+][+] Sources/sink/linking.py
07 -!-[+][+] Sources/sink/main.py
08 -!--!-[+] Sources/sink/snapshot.py
09 -!-[+][+] Sources/sink/tracking.py

and the output is cut friendly:

$ sink diff +a sink-r1 sink-r2 sink-r3 | cut -d' ' -f3-
.hgtags
README
setup.py
Documentation/DESIGN.txt
Documentation/MANUAL.txt
Scripts/sink
Sources/sink/linking.py
Sources/sink/main.py
Sources/sink/snapshot.py
Sources/sink/tracking.py

you can also compare individual changes between all the versions. Let's see the changes made to the src/sink/tracking.py file between sink-r2 and sink-r3:

$ sink diff sink-r2 sink-r3
0 -!-[+] README
1 [=][<] TODO
2 [=][-] Documentation/DESIGN.txt
3 [=][-] Documentation/MANUAL.txt
4 [=][-] Resources/epydoc.css
5 [=][<] Sources/sink/linking.py
6 [=][<] Sources/sink/main.py
7 -!-[+] Sources/sink/snapshot.py
8 [=][<] Sources/sink/tracking.py

the number of the file in the table is 5, so we pass it to sink -d option:

$ sink diff -d5 sink-r2 sink-r3
>> gvimdiff sink-r2/Sources/sink/linking.py sink-r3/Sources/sink/linking.py

this starts up gvimdiff, showing the differences between the files. This is a great way to do a merge outside of a revision control system.

Use Case: Directory synchronization

So taking our previous example, let's imaging we'd like to synchronize sink-r1 so that it is exactly the same as sink-r3.

We start by showing the added or modified files:

$ sink +a +m sink-r1 sink-r3
00 -!-[+] .hgtags
01 [=][<] Makefile
02 -!-[+] README
03 [=][<] TODO
04 -!-[+] setup.py
05 -!-[+] Scripts/sink
06 -!-[+] Sources/sink/linking.py
07 -!-[+] Sources/sink/main.py
08 -!-[+] Sources/sink/snapshot.py
09 -!-[+] Sources/sink/tracking.py

we then pipe the result to cut and xargs, first to create the directories that may not exist, and then to copy the files

$ sink +a +m sink-r1 sink-r3 | cut -d' ' -f3- | xargs dirname | sort | uniq| xargs mkdir -p
$ sink +a +m sink-r1 sink-r3 | cut -d' ' -f3- | xargs -I FILE cp sink-r3/FILE sink-r1/FILE

and we make sure the directories are the same:

$ sink sink-r1 sink-r3
0 [=][-] DESIGN
1 [=][-] NOTES
2 [=][-] ROADMAP
3 [=][-] Resources/epydoc.css
4 [=][-] Sources/sink/Sink.py
5 [=][-] Sources/sink/Tracking.py

we see that we have files to remove from 'sink-r1', so let's do it:

$ sink +r sink-r1 sink-r3 | cut -d' ' -f3 | xargs -I FILE rm sink-r1/FILE
$ sink sink-r1 sink-r3
No changes found.

and we've done the synchronization right. This may look a little bit verbose, but it's Unix's design philosophy -- have simple tools that do thing, and then integrate with other tools to do higher-level operations.

Use Case: Tracking changes made to a directory

Let's say you've just downloaded 'fltk', a cross-platform C++ widget library and you'd like to install it in /usr/local -- but you want to keep a receipt of the installed files.

First, get fltk, and compile it

$ wget 'http://ftp.easysw.com/pub/fltk/1.1.9/fltk-1.1.9-source.tar.bz2'
$ tar fvxj fltk-1.1.9-source.tar.bz2
$ cd fltk-1.1.9 ; ./configure --prefix=/usr/local ; make

now we take a snapshot of '/usr/local'

$ sink snap /usr/local > usr-local.snap

you can now install fltk and see the changes:

$ sudo make install
$ sink usr-local.snap /usr/local
000 -!-[+] bin/fltk-config
001 -!-[+] lib/libfltk.a
002 -!-[+] lib/libfltk_forms.a
003 -!-[+] lib/libfltk_gl.a
004 -!-[+] lib/libfltk_images.a
005 -!-[+] include/FL/Enumerations.H
006 -!-[+] include/FL/Fl.H
007 -!-[+] include/FL/Fl_Adjuster.H
008 -!-[+] include/FL/Fl_BMP_Image.H
009 -!-[+] include/FL/Fl_Bitmap.H
...
434 -!-[+] share/doc/fltk/examples/pixmaps/whiteking_3.xbm
435 -!-[+] share/doc/fltk/examples/pixmaps/whiteking_4.xbm
436 -!-[+] share/doc/fltk/examples/pixmaps/yellow.xpm
437 -!-[+] share/doc/fltk/examples/pixmaps/yellow_bomb.xpm

now you could create a receipt for the installation:

$ sink +a +m usr-local.snap /usr/local | cut -d' ' -f3- | xargs -IFILE echo /usr/local/FILE > fltk.receipt

or create a tarball with the installed files:

$ tar cvfj fltk-1.1.9-i386.tar.bz2 `sink +a +m usr-local.snap /usr/local | cut -d' ' -f3- | xargs -IFILE echo /usr/local/FILE`

Use Case: Backuping your data

Let's imagine you just got a new slice on Linode, and you'd like to start doing its configuration. You'd start by creating a snapshot of etc:

$ sudo sink -s /etc > etc-`date  +'%Y%m%d'`.snap

you'd then do your modifications and list the changes you made:

$ sudo sink etc-20090929.snaphsot /etc
000 [=][>] apt/sources.list
001 [=][>] passwd
002 [=][>] group
...

and make a tarball out of your changes:

$ tar cvfj node-configured.tar.bz2 `sink +a +m etc-20090929.snap /etc | cut -d' ' -f3- | xargs -IFILE echo etc/FILE`

you'll then be able to simply apply the same configuration to a new node by doing this:

$ cd / ; sudo tar fvxj ~/node-configured.tar.bz2

Tips

You can use find to generate a snapshot that you can sink diff against, which is useful for instance if you don't have sink in a container and want to see what's available.

find . -name "*" -not -type d | cut -d/ -f2- | sort | uniq > snap.lst
sink diff . snap.lst

When troubleshooting your filters, you can compare sink snap and find:

find . -name "*" -not -type d | cut -d/ -f2- | sort | uniq > a.lst
time sink snap  -snone . | sort | uniq > b.lst

Development

Building from Source

This project requires GNU Make for building and development tasks. The Makefile will automatically detect if you're using GNU Make and provide guidance if not.

Installation:

• macOS: brew install make (then use gmake command)
• Ubuntu/Debian: apt-get install make
• CentOS/RHEL: yum install make

Building:

# If you have GNU Make as 'make':
make prep          # Install dependencies
make check         # Run all checks (security, linting, typing)
make compile       # Compile with mypyc

# If you need to use 'gmake':
gmake prep
gmake check
gmake compile

# Or use the provided wrapper:
./gmake.sh prep
./gmake.sh check
./gmake.sh compile

The build system automatically refreshes dependencies weekly to ensure you have the latest versions of development tools.

Formats

Snapshot Format

Delta Format