Linux macOS Administration Notes

Contents for Linux & macOS Administration Notes

*nix, BSD, & macOS Universal Tooling

• Backing up with rsync
  • Backing up a root partition using rsync
  • Useful rsync flags
  • Useful rsync commands
  • Useful Links rsync
• Security
• Working with find
  • Useful Examples using findcommand
  • Useful Links find
• Working with grep
  • grep Gotchas
• Working with GNU Sed
• Working with GNU Core Utilities
• Working with GNU Core Utilites split
• Working with tree
• Working with du
• Working with df
  • Working with df on macOS
• Working with the watch
• Legacy Communication
• Working with different search utilities
  • Using mdfind
  • Using GNU find
  • Using BSD locate
  • Using GNU locate
  • Using the_platinum_searcher pt
  • Using ripgrep
• Working with X11
• Working with X11 on macOS
• Working with networking

Useful Links

TODOs

💡 All macOS & Linux related notes once contained within this doc have been migrated to their own respected documents.

• ⌘ macos administration Notes
• 🐧 linux administration Notes

*nix, BSD, & macOS Universal Tooling 🔝

The concepts outlined in this section should be applicable to most if not all modern OS's derived from a UNIX platform.

Backing up with rsync 🔝

To display a progress meter when transferring a file to a remote box, use rsync and not scp

rsync -e "ssh -p4242" --info=progress2 /path/to/mr-fancy.txt remote:/path/to/

One of the many benefits of backing up with rsync is that it performs incremental backups by default, so it should not recopy files if they have already been backed up.

Backing up a root partition using rsync 🔝

The below write up will demonstrate how to backup an SD card that contains Raspbian on it.

To perform a full system backup of a root file system using rsync

rsync -aAXh --exclude=/path/to/rsync-exclude-file --info=progress2 /path/to/source/directory/or/file /path/to/destination/directory/or/file

To restore from an rsync backup reverse the source and destination paths.

A typical rsync excludes file, ie. rsync-excludes will contain

/.fseventsd/*
/boot/*
/dev/*
/lib/modules/*
/media/*
/mnt/*
/proc/*
/run/*
/sys/*
/tmp/*
/var/log/*
/lost+found
/etc/fstab
/etc/mtab
/etc/modules
/etc/network/interfaces

rsync > Backing up across a network, ie. LAN

rsync is useful for backing up files and folders across a network attached disk, ie. a large mass storage device attached to a Raspberry Pi on a LAN. However, there are some gotchas to be aware of.

1. I did not get any decent results using NFS for whatever reason, I was averaging ~ 30KB transfer speeds from my MBP to ext USB device attached to a rpi.
2. NFS is kind of a PITA to get setup and going.
3. Installing sshfs on a local box, ie. my MBP is far less painless then setting up NFS.

• make sure that the local $USER and GROUP exist on both the local and remote systems along with having the same UID and GID for best results.
• even though there is a pi user on the stock Raspbian OS, I'd suggest creating a new user that has the same name and GID as on the local box (my MBP).

After all the above criteria has been met, then one can mount the mass storage device attached to the pi using the newly created credentials. If all goes well the rsync files should match the local UID and GID (my MBP) and the files contained on the USB ext drive (mass storage device) should have the same UID and GID

❗️ As far as I can tell, any non root user can mount the USB device / filesystem / partition on the pi and as long as the local user sshfs's the remote filesystem using the same remote / local credentials all should be good.

❗️ If the remote filesystem is mounted using sshfs such as,

sshfs [email protected]:/remote/mnt/point /local/mnt/point

then issues could arise where when rsyncing file to the remote filesystem will the UID of the pi user, ie. NOT HOT DOG 🌭

Useful rsync flags 🔝

• -a, --archive archive mode; files should be archived, meaning most characteristics are preserved

the -a flag implies --recursive and also implies --links.

• -A, --acls preserve ACLs and update the destination ACLs with the source ACLs.

• --exclude exclude the directories contained within curly braces.

• --info=[FLAGS] control over the output rsync displays to STDOUT.

  • --info=help to display a list of flags that can be used with --info.
    • Ex --info=progress2

• -h, --human-readable print output in friendly human-readable format.

• -l, --links symlinks are recreated on the destination.

• --no-i-r, --no-inc-recurssive disable incremental recursion

• -P, --partial --progress keeps partially transferred files

• -r, --recursive to recurse into directories

• -S, --sparse handle sparse files efficiently, ie. this is useful when backing up files such as Docker images or virtual hard disks for emulators such as QEMU.

• --stats record a verbose set of statistics of the file transfer.

• -X update the destination with extended attributes to be the same as the source.

• -x, --one-file-system prohibits rsync from going beyond the local file system.

• -u, --update update destination file with source files if the destination file already exists.

• --delete-excluded removes files from destination that no longer exist on the source. This is flag is useful when running into the below error message when running rsync

cannot delete non-empty directory:

Useful rsync commands 🔝

dat new new new rsync command

rsync \
...
--filter 'protect /path/to/dir/to/not/delete/but/add/new/files'
/path/to/src
/path/to/destination/

To backup a file or dir without preserving UID and GID

rsync -a --no-o --no-g /path/to/local/dir /path/to/backup/

To backup a directory of files from a local disk to an external volume on macOS

rsync \
--archive \
--acls \
--info=progress2 \
--human-readable \
--stats  \
--update \
-X \
--one-file-system \
--sparse \
--delete \
--delete-excluded \
--exclude-from=/path/to/rsync-excludes-file \
/path/to/src \
/Volumes/external-disk/path/to/bkp/dir/

The above command should create a 1:1 backup the source directory to destination volume, and also delete files on the destination that are no longer present on the source, and should also update file attributes and files in modifications have occurred since last backup.

A useful command that can compliment the above rsync command when backing up files is using the watch command.

There may be a yarn / NPM package that installs a watch binary that can conflict with the watch command installed by brew that is located in /usr/local/bin, so the watch package may need to be relinked, or specify the absolute path to watch

/usr/local/Cellar/watch/3.3.15/bin/watch

Complimentary watch commands

watch -n 1 -d 'du -hs ~/path/of/dir/to/watch
echo "the below two cmds not hotdog recently"
watch -d -n1 -c -x du -hs /path/to/bkup/dir/to/watch/
watch -d -n1 -c -x du -s /path/to/bkup/dir/to/watch/

To copy files using rsync and ssh across a network

rsync -avz \
-e "ssh -o StrictHostKeyChecking=no \
-o UserKnowHostsFile=/devnull" \
--progress [user]@[hostname.local]:/path/to/remote/file_or_directory /path/to/local/file_or_directory

The above command will copy a remote file or directory using ssh authorization and using rsync for the copying / transfer process, and copy a remote file from a remote box on the local network to the computer that the rsync command was issued from.

To do a simple file / directory transfer using rsync and have it display a progress bar for each file transferred

rsync -ah --progress /path/to/src/file_or_dir[/] /path/to/dest/file_or_dir

The above command is useful for displaying a progress bar when copying large files or directories.

rsync > Copying large files over a network or between filesystems

There are times were extremely large files need to be copied across filesystems and if the rsync command does not complete or fails for whatever reason it's useful to how to resume a failed rsync operation.

To start a rsync operation that can be resumed

rsync -ah --partial --info=progress2 /path/to/lrg/src/file /path/to/lrg/dest/file

If the above rsync operation fails / stops for whatever reason run the below command to resume

rsync -ah --append-verify --info=progress2 /path/to/lrg/src/file /path/to/lrg/dest/file

If screen or tmux is installed on the box running the rsync commands run the rsync command within tmux so one can logout or exit the current terminal / shell session and bring the job back to the foreground at a later date.

❗️ If a rsync job is started in the foreground and not within a terminal multiplexer the job can be suspended and moved to the background, however if the job is disowned then there is not an easy way to associate the disowned job back with the $USER who started the job and the rsync command will more than like fail for reasons I do not yet know. All that said run rsync within a terminal multiplexer to avoid this nonsense.

Useful Links rsync 🔝

• wiki.archlinux rsync
• Using rsync a la Apple Time Machine

Security 🔝

To calculate the SHA1 of a file

sha1sum /path/to/file.tar.bz

Working with find 🔝

A quick and dirty way to delete all files and folders within the PWD and all sub directories using ls and xargs

💡 Note fish shell requires escaping the wildcard * with a backslash \

ls --hide=\*.[ext] | xargs rm -rf

echo "the above cmd will delete all files and folders within the $PWD execpt the specified file extension"

To find / search for particular information on a topic through the system man pages, ie. editor

apropos editor

apropos searches man pages reading the description for a particular program and then printing it to STDOUT within the terminal.

To find all regular files related to a particular search query

find / -iname "*silverlight*" -type f 2>/dev/null

The above command will perform a case insensitive search of all files on the local system with the text silverlight contained anywhere within a file name, and suppress any errors to STDOUT, ie. the dreaded permission denied errors and what not.

To delete all files within a particular folder / directory and preserve the folder / directory itself.

find . -path ['*/mr_fancy_pants_dir/*'] -delete

A practical example

find . -path '*/node_modules/*' -delete

will delete all files and directories within the node_modules directory.

To find all directories on a file system with a particular name

find / -type d -name "mr-fancy-dir" -ls

To find all directories on a local file system with the word .kext in the directory name

find / -type d -name "*.kext"

To find all binary / executable files along with shell scripts within a specified directory

find /opt/code -executable -type f

To set the executable bit for all binary files within a specified directory

find /opt/code -executable -type f -exec chmod a+x {} \;

VirtualBox stores its custom kernel extensions within,

/Library/Application Support/VirtualBox/

To find all regular files with a specific criteria on a system using find

find / -name '[mr-fancy-pants]' -type f

To replace all spaces ␣ with a hyphen - for files and directories

find -name "* *" -print0 | sort -rz | \
while read -d $'\0' f; do mv -v "$f" "$(dirname "$f")/$(basename "${f// /_}")"; done

The above command needs be run with a POSIX shell, ie. bash or zsh

To change permissions of files and not folders recursively

find . -name '*.jpg' -type f | xargs chmod -x

To count the total number of files and directories within a path

find /path/to/directory -type f | wc -l

The above command will output the total count of the files and directories within the path specified. An alternative, but much more verbose solution is to use the tree command with the a flag. Personally, 🙋‍♂️ I would not recommend this solution.

Useful Examples using find command 🔝

To find all .DS_Store files on a file sytem without printing errors to STDERR

find / 2>/dev/null -type f -name ".DS_Store"

The above command will redirect all error output ie. Permission denied messages to /dev/null and NOT to the screen or console, thus making STDOUT more terse.

Useful Links find 🔝

• superuser.com question
• cyberciti.biz using find
• GitHub.com fd an alt find utility written in Rust ⚙️

Working with grep 🔝

To search for expression from STDOUT, ie. searching for nerd and plex from the output of fc-list on macOS

grep -iP '^(?=.*nerd)(?=.*plex)'

To invert the matching using grep

grep -v

To search for a pattern within all text files through a directory and sub directories.

grep -r [mr-fancy-search-pattern] .

To recursively search for a pattern in all text files, ie. source code files within the current directory

grep -rnw '/path/to/mr/fancy/dir' -e 'mr-fancy-pattern'
grep -rnw . -e 'mrs-fancy-pattern'

-r search recursively through sub directories -n print the line number for returned results -w match whole word

To search and print all markdown documents recursively for the word linux

find . -name "*.md" | xargs grep -i "linux"

grep Gotchas 🔝

When searching for the phrase set showmode grep will not return any results because the search string will explicitly search for set showmode even though the config line within the .vimrc is set noshowmode

Working with GNU sed 🔝

To iterate over all files in a directory and search for a pattern and remove all occurrences of the pattern within the files.

for i in *.rb
  sed -i -e 's/<<<<<<< HEAD//g' $i
end

The above command will remove all occurances of <<<<<<< HEAD within all the files defined in the for loop.

To verify the pattern has been erased

pt "<<<<<<< HEAD"

GNU sed > Useful Links

• To see how I used GNU sed to process entries in my $PATH environment variable for fish shell, see

Working with GNU Core Utilities 🔝

Working with GNU Core Utilties dd 🔝

dd provided by GNU Core Utitlies can be used to backup an entire partition to a file if needed. For my particular use case I used dd provided by GNU Core Utilities on macOS installed via brew to install GNU Core Utilities which provides its variant of the dd command.

To backup an entire partition on macOS using dd

dd if=/dev/disk[NUM]s[PARTITION] of=./path/backedup.img status=progress bs=512 conv=noerror

The above command can take quite a bit of time if the partition is a large size, ie. > 100GB or more. Also, if using dd on macOS, Apple provides diskutil which is a CLI program for working with disks, partitions, and file systems.

To find the block size of a file system, which is useful for running the above dd command to backup a partition for a disk on macOS diskutil can be used.

diskutil info /dev/disk[NUM]

To create zero'd out file using dd which can be formatted using a file system such as ext2, ext3, ext4, or even HFS.

dd if=/dev/zero of=./path/to/zero.file bs=1024 count=1024000

The above command will create ~ 1GB file filled with zeros.

Working with GNU Core Utitilites split 🔝

To split a large file on a system that has split installed which is like 💁 99.9% all *nix systems.

split -b [SIZE_OF_SPLIT_FILE] "/path/to/large_file.ext" "/path/to/split_files.ext."

To join the split files back together

cat /path/to/split_files.ext.\* > /path/to/large_file.join.ext

To flash broken symlinks on macOS, GNU Core Utilities will need be installed, and setup for the shell environment.

Flash broken symlink with GNU Core Utilities

brew install coreutils

Out of the box macOS supports $LSCOLORS

To configure BSD based $LSCOLORS online, see

Core Utilities uses a user directive for $LS_COLORS that is generated by running the dircolors command provided by Core Utilities.

Working with tree 🔝

The tree command can be used to list a hierarchical structure of a directory(ies) and it's files.

The command I use to generate file structure for this git project / repo and the accompanying submodules, I run the below,

tree -a -I "tmux_resurrect_*|.git|undo|swap|target|tmp|*.pyc|*.vader|tests|.vscode|.netrwhist|*.weechatlog|*.gpg"

Working with du 🔝

To display the to 10 largest files and directories within a given directory

\du -a $HOME | sort -n -r | head -n 10

Applying the -h flag jacks 💪 shit up

To get a recursive size of a directory and all files and directories within that directory

du -sh /path/to/mr-fancy-dir

The above command should output a single line and not the typical du output shit.

Working with df

Working with df display free disk space on macOS

macOS can perform some heavy caching of files, ie. Spotlight.app indexes and what not, ie. local library caches for applications, ~/Library/Caches That said, rebuilding the spotlight index from time to time can free up a substantial amount of disk space. Also, note that df uses a different blocksize than does Finder.app and Disk Utility.app

Example

Finder.app

df -h

To compare the df -k to the free space calculated from Finder.app multiply the Available free space, ie. in my case 572163364 by 1024 the side of a 1K block which is what Finder.app Disk Utility.app appear to be using.

$$572163364 * 1024 = 585895284736$$

Convert 585895284736 to Gigabyte notation

$$585895284736 ÷ 1,073,741,824 = Gigabytes ~ 545.65GB$$

Understanding purgeable space on macOS

When Apple released macOS 10.13 and the APFS file system for High Sierra, APFS file systems take local snapshots of the local file system, ie. the internal disk that is running macOS. That said if a large file or directory is deleted df will not be able to reflect the new changes to file system because macOS can take a snapshot of the large deletion and supposedly recover the deleted files in a Recovery Mode. These local snapshots features are coupled with Time Machine.app ie. tmutil which allows several sub commands for working with localsnapshots. However, presently I can not figure out how to permenently disable local snapshots on a APFS volume.

For a visual understanding of what I am describing, click here

• figure out where the 40GB of discrepency space is coming from

Working with the watch command 🔝

To continously monitor a directory for file changes, ie. display the newest files at the top of the listing while continously listing results in real time.

watch -n 1 -d 'ls -lt /dev'

The above command is useful for monitoring when devices are connected / added to a system.

Legacy communication 🔝

To write a message to another user on the system

write <user>
My friendly message 8)
EOF

Working with different search utilities 🔝

Searching for files in a directory with a specific file extension 🔝

Using mdfind

To search for all files in /opt/Code/dotfiles with a fish extension using one of the below search tools, and print an integer of all files contained within the search.

mdfind is a CLI utility that interfaces with Spotlight, ie. macOS search

mdfind -onlyin /opt/Code/dotfiles -name .fish -count

The above command can report false positives, ie. a file with the name of mr-fancy.fish=

Using GNU find 🔝

find -name "*.fish" | wc -l

Useful find commands

To change permissions for a type of file or directory recursively throughout a directory, ie. useful when working with rsync to backup directories.

find /Users/mr-fancy-42 -type d -exec chmod g+wrx {} +
find /Users/mr-fancy-42 -type f -exec chmod g+wr {} +

The above command will modify files and dirs for the user mr-fancy-42 to allow users in the same group to read, write, and execute files and dirs within mr-fancy-42's $HOME dir.

Using BSD locate 🔝

sudo /usr/libexec/locate.updatedb
locate '/opt/Code/dotfiles/**.fish' | wc -l

To update the native locate database on macOS

sudo /usr/libexec/locate.updatedb

Using GNU locate 🔝

/usr/local/bin/locate --version

GNU locate allows specifying a path to a database file that can be queried from the CLI, also GNU locate supports using the $LOCATE_PATH env var. However, the macOS version of BSD locate DB, ie. /var/db/locate.database is NOT compatible with the version of GNU locate installed by brew.

To create or update the locate database using updatedb provided by GNU locate

sudo /usr/local/bin/updatedb --localpaths='/' --output='/path/to/gnu-locate-database-file'

The database file will be created by the super user on the system,

To search for all files with a fish extension using GNU locate

/usr/local/bin/locate -c '/opt/Code/dotfiles/*ME.md' --database=/path/to/gnu-locate-database-file

Using the_platinum_searcher pt 🔝

TODO

Using ripgrep rg 🔝

To search for a specific PATTERN within a specific file

rg PATTERN file

• github ripgrep user guide

Example

rg rails Gemfile

TODO

Working with X11

General notes about working with X11 and X11 forwarding in a client / server model

• the X11 server is running on the local machine
• the X11 app is the client

So when connecting to a remote linux box with X11 forwarding setup and opening an app such as xeyes the app is considered a client of the linux box, but the X11 server on the local box macOS is running the client app.

X11 / x11 forwarding

On a recent VPS i setup, i ran into difficulty with X11 forwarding from macos to debian buster TL;DR ended up add the below entry to /etc/hosts on the VPS

127.0.0.1 localhost

Adding the above entry to the /etc/hosts allowed my X11 forwarding agent to properly work with the DISPLAY environment variable specifically the localhost address.

Side Note Debian based distros are configured to use 127.0.1.1 as localhost, and do not come preconfigured to work with 127.0.0.1 as localhost.

X11 forwarding was tested by launching the xeyes application after logging into the remote server using ssh. I commented on the following stackexchange answer ↗

Working with X11 on macOS

To print a list of available resolutions supported by X11

xrandr

To print the dots per inch, ie. DPI

xdpyinfo | grep -i resolution

To print the current keymap table

xmodmap -pke

To print the current keyboard layout configuration

setxkbmap -query

Working with Networking

To print a process ID related to a networking port

lsof -n -i :[PORT_NUM] | grep LISTEN

mounting network drives from a cli on macos

• stackexchange / why smbfs are mounted with executable bit set

HOSTS files

Most if not all UNIX derivatives, ie. Linux, BSD's macOS support editing a hosts file, even Microsoft Windows supports a hosts file.

The hosts file on macOS is

/etc/hosts

and entries can be added to quickly navigate to a particular host on a intranet / LAN for experimenting with different services, ie. a web server and what not. A sample entry in /etc/hosts would look like the following.

10.0.1.42 the-meaning-of-life.exp # Local IP address domain name

Microsoft Windows based OSes have hosts file at

C:\WINDOWS\SYSTEM32\DRIVERS\ETC\HOSTS

Working with OpenSSH

To suppress verbose output when logging into a OpenSSH server that is running on a GNU+Linux box, add a .hushlogin file in the $USER $HOME directory.

openssh/troubleshooting

🚨 easiest way to troubleshoot ssh and sshd related issues is start a sshd server on a arbitrary port, ie. 2222 for test purposes, and use the -d debug flag when starting the server,

 /usr/sbin/sshd -d -p 2222

For my particular use case on getting public key auth working on iOS 9.3.5 the user mobile $HOME directory was group writeable which was preventing public key / passwordless login from working, no matter how many ssh -vvvvvvvvv i put in the command the was not a good diagnostic msg to let me know what the error was until started sshd in the foreground with debug mode enabled. sshd gave me the following output message when trying to login with public key

Authentication refused: bad ownership or modes for directory /private/var/mobile

After reading that message, and changing /var/mobile to 755 with chmod public key / passwordless login was working with ssh ✧*｡٩(ˊᗜˋ*)و✧*｡

Troubleshooting Networking related issues

A fix for the below error message

sudo: unable to resolve host [NAME_OF_HOST]

There is no entry in the /etc/hosts file that corresponds to the name in /etc/hostname. see for more info.

Working with man and man pages

To print a list of paths that will be searched for man pages

manpath

Working with man pages on macOS

To print a list of directories that can be searched for man pages

man -d man

To print the directory where man would look for the MANPATH for a binary

man -d

My man page breakthrough

On macOS if binaries, ie. rvm, who, and mv etc etc, are contained within a bin directory, then if a sibling level man directory exists macOS will be able to find the accompanying man page that corresponds to the command.

./
bin/
man/

The parent paths aren't important, and editing of system level files contained with the /etc is not required.

Working with GNU Coreutils on macOS

If GNU Coreutils are installed with default names on macOS the binaries are located within

$brew_prefix/Cellar/coreutils/[MAJOR.MINOR]/libexec/gnubin
$brew_prefix/Cellar/coreutils/[MAJOR.MINOR]/libexec/gnuman

Copy the gnubin and gnuman to bin and man within the same libexec directory.

cd $brew_prefix/Cellar/coreutils/[MAJOR.MINOR]/libexec
ln -sf ./gnubin ./bin
ln -sf ./gnuman ./man

After the above directories have been copied the man command should be able to read the accomponying man pages for the commands.

Working with compressed files

To extract a password protected file

unzip -p 4242 /path/to/file.zip

If the below error occurs when trying to extract a zip file

unsupported compression method 99

7z x -p 4242 /path/to/file.zip

All Useful Links 🔝

• DigitalOcean - Setting up a LEMP stack
• DigitalOcean - How to install an SSL cert
• linuxjourney.com - A pleasant site for learning Linux

TODOs 🔝

• Add Table of Contents to this document.