Skip to content

macos administration Notes

Jump to bottom
chris edited this page Sep 8, 2022 · 28 revisions
Contents for Linux & macOS Administration Notes
*nix, BSD, & macOS Universal Tooling
macOS
Useful Links
TODOs

*nix, BSD, & macOS Universal Tooling 🔝

The concepts outlined in this section should be applicable to most if not all modern OS's derived from a UNIX platform.

Backing up with rsync 🔝

To display a progress meter when transferring a file to a remote box, use rsync and not scp

rsync -e "ssh -p4242" --info=progress2 /path/to/mr-fancy.txt remote:/path/to/

One of the many benefits of backing up with rsync is that it performs incremental backups by default, so it should not recopy files if they have already been backed up.

Backing up a root partition using rsync 🔝

The below write up will demonstrate how to backup an SD card that contains Raspbian on it.

To perform a full system backup of a root file system using rsync

rsync -aAXh --exclude=/path/to/rsync-exclude-file --info=progress2 /path/to/source/directory/or/file /path/to/destination/directory/or/file

To restore from an rsync backup reverse the source and destination paths.

A typical rsync excludes file, ie. rsync-excludes will contain

/.fseventsd/*
/boot/*
/dev/*
/lib/modules/*
/media/*
/mnt/*
/proc/*
/run/*
/sys/*
/tmp/*
/var/log/*
/lost+found
/etc/fstab
/etc/mtab
/etc/modules
/etc/network/interfaces

macos / rsync / backing up to zfs filesystem/disk

to use rsync to backup data to a zfs formatted disk/storage medium one first has to have zfs support working on macos, then has to run the below command to mount the filesystem. learn more

sudo zpool import [NAME_OF_POOL]

to list the name of available zfs pools

diskutil list

echo "look for the NAME of the ZFS partition"

to verify the volume is mounted

mount

rsync > Backing up across a network, ie. LAN

rsync is useful for backing up files and folders across a network attached disk, ie. a large mass storage device attached to a Raspberry Pi on a LAN. However, there are some gotchas to be aware of.

  1. I did not get any decent results using NFS for whatever reason, I was averaging ~ 30KB transfer speeds from my MBP to ext USB device attached to a rpi.
  2. NFS is kind of a PITA to get setup and going.
  3. Installing sshfs on a local box, ie. my MBP is far less painless then setting up NFS.
  • make sure that the local $USER and GROUP exist on both the local and remote systems along with having the same UID and GID for best results.
  • even though there is a pi user on the stock Raspbian OS, I'd suggest creating a new user that has the same name and GID as on the local box (my MBP).

After all the above criteria has been met, then one can mount the mass storage device attached to the pi using the newly created credentials. If all goes well the rsync files should match the local UID and GID (my MBP) and the files contained on the USB ext drive (mass storage device) should have the same UID and GID

❗️ As far as I can tell, any non root user can mount the USB device / filesystem / partition on the pi and as long as the local user sshfs's the remote filesystem using the same remote / local credentials all should be good.

❗️ If the remote filesystem is mounted using sshfs such as,

sshfs [email protected]:/remote/mnt/point /local/mnt/point

then issues could arise where when rsyncing file to the remote filesystem will the UID of the pi user, ie. NOT HOT DOG 🌭

Useful rsync flags 🔝

  • -a, --archive archive mode; files should be archived, meaning most characteristics are preserved

the -a flag implies --recursive and also implies --links.

  • -A, --acls preserve ACLs and update the destination ACLs with the source ACLs.

  • --exclude exclude the directories contained within curly braces.

  • --info=[FLAGS] control over the output rsync displays to STDOUT.

    • --info=help to display a list of flags that can be used with --info.
      • Ex --info=progress2

  • -h, --human-readable print output in friendly human-readable format.

  • -l, --links symlinks are recreated on the destination.

  • --no-i-r, --no-inc-recurssive disable incremental recursion

  • -P, --partial --progress keeps partially transferred files

  • -r, --recursive to recurse into directories

  • -S, --sparse handle sparse files efficiently, ie. this is useful when backing up files such as Docker images or virtual hard disks for emulators such as QEMU.

  • --stats record a verbose set of statistics of the file transfer.

  • -X update the destination with extended attributes to be the same as the source.

  • -x, --one-file-system prohibits rsync from going beyond the local file system.

  • -u, --update update destination file with source files if the destination file already exists.

  • --delete-excluded removes files from destination that no longer exist on the source. This is flag is useful when running into the below error message when running rsync

cannot delete non-empty directory:

Useful rsync commands 🔝

dat new new new rsync command

rsync \
...
--filter 'protect /path/to/dir/to/not/delete/but/add/new/files'
/path/to/src
/path/to/destination/

To backup a file or dir without preserving UID and GID

rsync -a --no-o --no-g /path/to/local/dir /path/to/backup/

To backup a directory of files from a local disk to an external volume on macOS

rsync \
--archive \
--acls \
--info=progress2 \
--human-readable \
--stats  \
--update \
-X \
--one-file-system \
--sparse \
--delete \
--delete-excluded \
--exclude-from=/path/to/rsync-excludes-file \
/path/to/src \
/Volumes/external-disk/path/to/bkp/dir/

The above command should create a 1:1 backup the source directory to destination volume, and also delete files on the destination that are no longer present on the source, and should also update file attributes and files in modifications have occurred since last backup.

A useful command that can compliment the above rsync command when backing up files is using the watch command.

There may be a yarn / NPM package that installs a watch binary that can conflict with the watch command installed by brew that is located in /usr/local/bin, so the watch package may need to be relinked, or specify the absolute path to watch

/usr/local/Cellar/watch/3.3.15/bin/watch
Complimentary watch commands
watch -n 1 -d 'du -hs ~/path/of/dir/to/watch
echo "the below two cmds not hotdog recently"
watch -d -n1 -c -x du -hs /path/to/bkup/dir/to/watch/
watch -d -n1 -c -x du -s /path/to/bkup/dir/to/watch/

To copy files using rsync and ssh across a network

rsync -avz \
-e "ssh -o StrictHostKeyChecking=no \
-o UserKnowHostsFile=/dev/null" \
--progress [user]@[hostname.local]:/path/to/remote/file_or_directory /path/to/local/file_or_directory

The above command will copy a remote file or directory using ssh authorization and using rsync for the copying / transfer process, and copy a remote file from a remote box on the local network to the computer that the rsync command was issued from.

To do a simple file / directory transfer using rsync and have it display a progress bar for each file transferred

rsync -ah --progress /path/to/src/file_or_dir[/] /path/to/dest/file_or_dir

The above command is useful for displaying a progress bar when copying large files or directories.

rsync > Copying large files over a network or between filesystems

There are times were extremely large files need to be copied across filesystems and if the rsync command does not complete or fails for whatever reason it's useful to how to resume a failed rsync operation.

To start a rsync operation that can be resumed

rsync -ah --partial --info=progress2 /path/to/lrg/src/file /path/to/lrg/dest/file

If the above rsync operation fails / stops for whatever reason run the below command to resume

rsync -ah --append-verify --info=progress2 /path/to/lrg/src/file /path/to/lrg/dest/file

If screen or tmux is installed on the box running the rsync commands run the rsync command within tmux so one can logout or exit the current terminal / shell session and bring the job back to the foreground at a later date.

❗️ If a rsync job is started in the foreground and not within a terminal multiplexer the job can be suspended and moved to the background, however if the job is disowned then there is not an easy way to associate the disowned job back with the $USER who started the job and the rsync command will more than like fail for reasons I do not yet know. All that said run rsync within a terminal multiplexer to avoid this nonsense.

Useful Links rsync 🔝

Security 🔝

To calculate the SHA1 of a file

sha1sum /path/to/file.tar.bz

Working with find 🔝

To find / search for particular information on a topic through the system man pages, ie. editor

apropos editor

apropos searches man pages reading the description for a particular program and then printing it to STDOUT within the terminal.

To find all regular files related to a particular search query

find / -iname "*silverlight*" -type f 2>/dev/null

The above command will perform a case insensitive search of all files on the local system with the text silverlight contained anywhere within a file name, and suppress any errors to STDOUT, ie. the dreaded permission denied errors and what not.

To delete all files within a particular folder / directory and preserve the folder / directory itself.

find . -path ['*/mr_fancy_pants_dir/*'] -delete

A practical example

find . -path '*/node_modules/*' -delete

will delete all files and directories within the node_modules directory.

To find all directories on a file system with a particular name

find / -type d -name "mr-fancy-dir" -ls

To find all directories on a local file system with the word .kext in the directory name

find / -type d -name "*.kext"

To find all binary / executable files along with shell scripts within a specified directory

find /opt/code -executable -type f

To set the executable bit for all binary files within a specified directory

find /opt/code -executable -type f -exec chmod a+x {} \;

VirtualBox stores its custom kernel extensions within,

/Library/Application Support/VirtualBox/

To find all regular files with a specific criteria on a system using find

find / -name '[mr-fancy-pants]' -type f

To replace all spaces ␣ with a hyphen - for files and directories

find -name "* *" -print0 | sort -rz | \
while read -d $'\0' f; do mv -v "$f" "$(dirname "$f")/$(basename "${f// /_}")"; done

The above command needs be run with a POSIX shell, ie. bash or zsh

To change permissions of files and not folders recursively

find . -name '*.jpg' -type f | xargs chmod -x

To count the total number of files and directories within a path

find /path/to/directory -type f | wc -l

The above command will output the total count of the files and directories within the path specified. An alternative, but much more verbose solution is to use the tree command with the a flag. Personally, 🙋‍♂️ I would not recommend this solution.

Useful Examples using find command 🔝

To find all .DS_Store files on a file sytem without printing errors to STDERR

find / 2>/dev/null -type f -name ".DS_Store"

The above command will redirect all error output ie. Permission denied messages to /dev/null and NOT to the screen or console, thus making STDOUT more terse.

Useful Links find 🔝

Working with grep 🔝

To search for expression from STDOUT, ie. searching for nerd and plex from the output of fc-list on macOS

grep -iP '^(?=.*nerd)(?=.*plex)'

To invert the matching using grep

grep -v

To search for a pattern within all text files through a directory and sub directories.

grep -r [mr-fancy-search-pattern] .

To recursively search for a pattern in all text files, ie. source code files within the current directory

grep -rnw '/path/to/mr/fancy/dir' -e 'mr-fancy-pattern'
grep -rnw . -e 'mrs-fancy-pattern'

-r search recursively through sub directories -n print the line number for returned results -w match whole word

To search and print all markdown documents recursively for the word linux

find . -name "*.md" | xargs grep -i "linux"

grep Gotchas 🔝

When searching for the phrase set showmode grep will not return any results because the search string will explicitly search for set showmode even though the config line within the .vimrc is set noshowmode

Working with GNU sed 🔝

macos does not provide gnu based sed out of the box, but rather provides the bsd version of sed, which IMHO is missing some of the bells and whistles of the gnu based sed. either way, to edit a file in place with bsd flavored sed see below link,

https://stackoverflow.com/a/7573438/708807

To iterate over all files in a directory and search for a pattern and remove all occurrences of the pattern within the files.

for i in *.rb
  sed -i -e 's/<<<<<<< HEAD//g' $i
end

The above command will remove all occurances of <<<<<<< HEAD within all the files defined in the for loop.

To verify the pattern has been erased

pt "<<<<<<< HEAD"

GNU sed > Useful Links

  • To see how I used GNU sed to process entries in my $PATH environment variable for fish shell, see

Working with GNU Core Utilities 🔝

Working with GNU Core Utilties dd 🔝

dd provided by GNU Core Utitlies can be used to backup an entire partition to a file if needed. For my particular use case I used dd provided by GNU Core Utilities on macOS installed via brew to install GNU Core Utilities which provides its variant of the dd command.

To backup an entire partition on macOS using dd

dd if=/dev/disk[NUM]s[PARTITION] of=./path/backedup.img status=progress bs=512 conv=noerror

The above command can take quite a bit of time if the partition is a large size, ie. > 100GB or more. Also, if using dd on macOS, Apple provides diskutil which is a CLI program for working with disks, partitions, and file systems.

To find the block size of a file system, which is useful for running the above dd command to backup a partition for a disk on macOS diskutil can be used.

diskutil info /dev/disk[NUM]

To create zero'd out file using dd which can be formatted using a file system such as ext2, ext3, ext4, or even HFS.

dd if=/dev/zero of=./path/to/zero.file bs=1024 count=1024000

The above command will create ~ 1GB file filled with zeros.

Working with GNU Core Utitilites split 🔝

To split a large file on a system that has split installed which is like 💁 99.9% all *nix systems.

split -b [SIZE_OF_SPLIT_FILE] "/path/to/large_file.ext" "/path/to/split_files.ext."

To join the split files back together

cat /path/to/split_files.ext.\* > /path/to/large_file.join.ext

To flash broken symlinks on macOS, GNU Core Utilities will need be installed, and setup for the shell environment.

Flash broken symlink with GNU Core Utilities

Flash broken symlink

brew install coreutils

Out of the box macOS supports $LSCOLORS

To configure BSD based $LSCOLORS online, see

Core Utilities uses a user directive for $LS_COLORS that is generated by running the dircolors command provided by Core Utilities.

Working with tree 🔝

The tree command can be used to list a hierarchical structure of a directory(ies) and it's files.

The command I use to generate file structure for this git project / repo and the accompanying submodules, I run the below,

tree -a -I "tmux_resurrect_*|.git|undo|swap|target|tmp|*.pyc|*.vader|tests|.vscode|.netrwhist|*.weechatlog|*.gpg"

Working with du 🔝

To display the to 10 largest files and directories within a given directory

\du -a $HOME | sort -n -r | head -n 10

Applying the -h flag jacks 💪 shit up

To get a recursive size of a directory and all files and directories within that directory

du -sh /path/to/mr-fancy-dir

The above command should output a single line and not the typical du output shit.

Working with df

Working with df display free disk space on macOS

macOS can perform some heavy caching of files, ie. Spotlight.app indexes and what not, ie. local library caches for applications, ~/Library/Caches That said, rebuilding the spotlight index from time to time can free up a substantial amount of disk space. Also, note that df uses a different blocksize than does Finder.app and Disk Utility.app

Example

Finder.app

Finder.app-free-space

df -h

df

To compare the df -k to the free space calculated from Finder.app multiply the Available free space, ie. in my case 572163364 by 1024 the side of a 1K block which is what Finder.app Disk Utility.app appear to be using.

$$572163364 * 1024 = 585895284736$$

Convert 585895284736 to Gigabyte notation

$$585895284736 ÷ 1,073,741,824 = Gigabytes ~ 545.65GB$$
Understanding purgeable space on macOS

When Apple released macOS 10.13 and the APFS file system for High Sierra, APFS file systems take local snapshots of the local file system, ie. the internal disk that is running macOS. That said if a large file or directory is deleted df will not be able to reflect the new changes to file system because macOS can take a snapshot of the large deletion and supposedly recover the deleted files in a Recovery Mode. These local snapshots features are coupled with Time Machine.app ie. tmutil which allows several sub commands for working with localsnapshots. However, presently I can not figure out how to permenently disable local snapshots on a APFS volume.

For a visual understanding of what I am describing, click here

  • figure out where the 40GB of discrepency space is coming from

Working with the watch command 🔝

To continously monitor a directory for file changes, ie. display the newest files at the top of the listing while continously listing results in real time.

watch -n 1 -d 'ls -lt /dev'

The above command is useful for monitoring when devices are connected / added to a system.

Legacy communication 🔝

To write a message to another user on the system

write <user>
My friendly message 8)
EOF

Working with different search utilities 🔝

Searching for files in a directory with a specific file extension 🔝

Using mdfind aka. spotlight

#protip ✅

To disable the little magnifying glass from the apple menu bar on macos 10.14 learn more

UPDATE macos big sur requires an additional workaround 🙄 learn more

query terms #spotlight #menubar #disable #remove #icon

cd /System/Library/CoreServices/Spotlight.app/Contents/MacOS;

echo "# note: if running macos ≥ catalina the below cmd is required";
sudo mount -uw / ;

sudo cp Spotlight Spotlight.bak ;
sudo perl -pi -e 's|(\x00\x00\x00\x00\x00\x00\x47\x40\x00\x00\x00\x00\x00\x00)\x42\x40(\x00\x00\x80\x3f\x00\x00\x70\x42)|$1\x00\x00$2|sg' Spotlight ;

cmp -l Spotlight Spotlight.bak ;
sudo codesign -f -s - Spotlight ;
sudo killall Spotlight ;

To search for all files in /opt/Code/dotfiles with a fish extension using one of the below search tools, and print an integer of all files contained within the search.

mdfind is a CLI utility that interfaces with Spotlight, ie. macOS search

mdfind -onlyin /opt/Code/dotfiles -name .fish -count

The above command can report false positives, ie. a file with the name of mr-fancy.fish=

Using GNU find 🔝 
find -name "*.fish" | wc -l
Useful find commands

To change permissions for a type of file or directory recursively throughout a directory, ie. useful when working with rsync to backup directories.

find /Users/mr-fancy-42 -type d -exec chmod g+wrx {} +
find /Users/mr-fancy-42 -type f -exec chmod g+wr {} +

The above command will modify files and dirs for the user mr-fancy-42 to allow users in the same group to read, write, and execute files and dirs within mr-fancy-42's $HOME dir.

Using BSD locate 🔝 
sudo /usr/libexec/locate.updatedb
locate '/opt/Code/dotfiles/**.fish' | wc -l

To update the native locate database on macOS

sudo /usr/libexec/locate.updatedb

Using GNU locate 🔝 
/usr/local/bin/locate --version

GNU locate allows specifying a path to a database file that can be queried from the CLI, also GNU locate supports using the $LOCATE_PATH env var. However, the macOS version of BSD locate DB, ie. /var/db/locate.database is NOT compatible with the version of GNU locate installed by brew.

To create or update the locate database using updatedb provided by GNU locate

sudo /usr/local/bin/updatedb --localpaths='/' --output='/path/to/gnu-locate-database-file'

The database file will be created by the super user on the system,

To search for all files with a fish extension using GNU locate

/usr/local/bin/locate -c '/opt/Code/dotfiles/*ME.md' --database=/path/to/gnu-locate-database-file

Using the_platinum_searcher pt 🔝 
TODO

Using ripgrep rg 🔝 
TODO

Working with X11

Working with X11 on macOS

To print a list of available resolutions supported by X11

xrandr

To print the dots per inch, ie. DPI

xdpyinfo | grep -i resolution

To print the current keymap table

xmodmap -pke

To print the current keyboard layout configuration

setxkbmap -query

Working with Networking

To print a process ID related to a networking port

lsof -n -i :[PORT_NUM] | grep LISTEN

HOSTS files

Most if not all UNIX derivatives, ie. Linux, BSD's macOS support editing a hosts file, even Microsoft Windows supports a hosts file.

The hosts file on macOS is

/etc/hosts

and entries can be added to quickly navigate to a particular host on a intranet / LAN for experimenting with different services, ie. a web server and what not. A sample entry in /etc/hosts would look like the following.

10.0.1.42 the-meaning-of-life.exp # Local IP address domain name

Microsft Windows based OSes have hosts file at

C:\WINDOWS\SYSTEM32\DRIVERS\ETC\HOSTS

Working with OpenSSH

To suppress verbose output when logging into a OpenSSH server that is running on a GNU+Linux box, add a .hushlogin file in the $USER $HOME directory.

Troubleshooting Networking related issues

A fix for the below error message

sudo: unable to resolve host [NAME_OF_HOST]

There is no entry in the /etc/hosts file that corresponds to the name in /etc/hostname. see for more info.

Working with man and man pages

To print a list of paths that will be searched for man pages

manpath

Working with man pages on macOS

To print a list of directories that can be searched for man pages

man -d man

To print the directory where man would look for the MANPATH for a binary

man -d

My man page breakthrough

On macOS if binaries, ie. rvm, who, and mv etc etc, are contained within a bin directory, then if a sibling level man directory exists macOS will be able to find the accompanying man page that corresponds to the command.

./
bin/
man/

The parent paths aren't important, and editing of system level files contained with the /etc is not required.

Working with GNU Coreutils on macOS

If GNU Coreutils are installed with default names on macOS the binaries are located within

$brew_prefix/Cellar/coreutils/[MAJOR.MINOR]/libexec/gnubin
$brew_prefix/Cellar/coreutils/[MAJOR.MINOR]/libexec/gnuman

Copy the gnubin and gnuman to bin and man within the same libexec directory.

cd $brew_prefix/Cellar/coreutils/[MAJOR.MINOR]/libexec
ln -sf ./gnubin ./bin
ln -sf ./gnuman ./man

After the above directories have been copied the man command should be able to read the accomponying man pages for the commands.

Working with compressed files

To extract a password protected file

unzip -p 4242 /path/to/file.zip

If the below error occurs when trying to extract a zip file

unsupported compression method 99
7z x -p 4242 /path/to/file.zip

macOS 🔝

Working with security related features on macOS

macOS provides a feature know as SIP System Integrity Protection which basically prevents certain files and settings from being modified.

To disable SIP on a modern version of macOS

  1. Boot into recovery mode ⌘ + r
  2. Enter the below command to disable SIP 
csrutil disable; reboot

To check the status of SIP

csrutil status

Useful Links macOS 🔝

For a comprehensive list of changes of macOS between releases see

To download macOS High Sierra officially using App Store.app via macOS

https://itunes.apple.com/us/app/macos-high-sierra/id1246284741?mt=12

To download old versions of Apple Software, ie. iMovie

support.apple.com/downloads

Networking in macOS 🔝

Setting up a NFS Server / Share in macOS High Sierra 🔝

macOS has the NFS daemon disabled by default, but it can be enabled.

  1. Make certain that SIP, ie. System Integrity Protection has been disabled.
  2. Edit /System/Library/LaunchDaemons/com.apple.nfsd.plist to include the -N flag, so that non root users can mount nfs shares hosted on macOS.
  3. Edit /etc/exports and include a path to a directory to be shared.

Ex

/Users/mr-fany/Movies/42 -ro -mapall=mr-fancy:mr-fancy -alldirs

The above example presently works on macOS 10.13.6 🤞

  • With super user privileges load the above mentioned plist using launchctl 
sudo -E launchctl -load /System/Library/LaunchDaemons/com.apple.nfsd.plist

To verify if the service was successfully loaded into launchd service manager with launchctl

sudo -E launchctl list | grep -i "nfsd"

The above command should list the process ID of nfsd

Troubleshooting nfsd, ie. a NFS Server on macOS 🔝

To verify if the /etc/exports has properly been configured

nfsd checkexports

If there is an error the above mentioned file, nfsd will print the errors to STDOUT.

If nfsd needs to restarted, it can be restarted directly ie. launchctl is not required. 👍

nfsd restart

To get the status of nfsd

nfsd status

To display locally served file shares

showmount -e

To verify there are no syntax errors in the above mentioned .plist file

plutil -lint /System/Library/LaunchDaemons/com.apple.nfsd.plist

Useful Resources for working with nfsd 🔝 
man nfs
man nfs.conf
man nfsd
man exports
ifconfig en0

macOS > nfs > Useful Links

ncures 🔝

To print / show the current version of ncures installed on a system, ie. macOS

cat /usr/include/ncurses.h | grep -i "version"

Working with OpenSSH 🔝

🔥, for reasons above my paygrade when attempting to ssh into another local user who's $HOME dir was set to 775, ssh refused to let me login. ie. use 755 for /Users/mr-fancy

🔥 When attempting to stop a LaunchDaemon macos use sudo to unload the daemon or else it may respawn and start a new process.

useful permissions to have when working with ssh

chmod 700 .ssh
sudo chmod 640 .ssh/authorized_keys
sudo chown $USER .ssh
sudo chown $USER .ssh/authorized_keys

When troubleshooting OpenSSH, and making changes to ~/.ssh/config for particular host entries, neither the OpenSSH server or the current shell need to be restarted, ie. reinvoked as changes made to ~/.ssh/config are reflected in realtime when using ssh.

For an incompetent guide comparing LibreSSL to OpenSSL, see this

To set up password less login, ie. public key authentication

cat ~/.ssh/id_rsa.pub | ssh -vvv -p [PORT_NUM] [user@[hostname] 'cat >> $HOME.ssh/authorized_keys && echo "Key copied"'

If the id_rsa.pub file doesn't exists one can be generated using ssh-keygen

An alternative way to copy a public key from one host to another is using ssh-copy-id

ssh-copy-id [USER]@[HOSTNAME/IP].local -p[PORT]

[USER] is the name of the $USER that will contain the public key to allow loggin in of the current $USER

Working with ssh-agent 🔝

ssh-agent is a utility app that allows the OS / user session to store unencrypted SSH keys in memory.

The safest place to store unencrypted keys for SSH sessions would be in memory and NOT on disk.

Working with brewed OpenSSH 🔝

First off, grab a cup of joe or tea if it is that time of the day for you because this isn't something that your gunna why to skim through, that said, if you have your cup of covfefe 🙄 and you're ready to roll 🚗

  1. Download and install OpenSSH using homebrew
brew install openssh

If you prefer to use LibreSSL over OpenSSL 🙋 then you're gunna wanna check out no pun intended there the version of OpenSSH I rolled 🚬 for homebrew.

  1. 1.1 optional 
brew install openssh --with-libressl

OpenSSH does not need to be recompiled when upgrading LibreSSL, ie. I tested this when upgrading LibreSSL from 2.7.2 to 2.7.3 and running, ssh -V

The output from the above command is

OpenSSH_7.7p1, LibreSSL 2.7.3

However, the sshd service will need to be stopped and started because launchd doesn't have a "restart" feature per se on macOS in order for OpenSSH to load the new version of LibreSSL.

> launchctl unload homebrew.mxcl.sshd
> launchctl load homebrew.mxcl.sshd

If using the Apple provided service of OpenSSH which happens to be compiled with LibreSSL as well ...hmmm 🤔 wonder why that is then your first gunna wanna stop that launchd service, and unload it from launchd.

Recent version of macOS provide a nice little feature known as SIP that supposedly prevents third parties 🙋 from modfying certain files and services. thanks Obama

Personally, I disabled SIP on my local box a while ago because I like to pretend that I know what I'm doing when using "apps" such as DTrace.

  • To stop the Apple provided OpenSSH daemon if it is currently running, and prevent it from launching in the future

To restart the Apple provided ssh server

sudo launchctl unload /System/Library/LaunchDaemons/ssh.plist
sudo launchctl load -w /System/Library/LaunchDaemons/ssh.plist
  • To unload the service from launchd
sudo -E launchctl unload /System/Library/LaunchDaemons/ssh.plist
  • To verify the service is stopped
lsof -i | grep "sshd"

...and you shouldn't 🙈 see anything printed to STDOUT if sshd isn't running.

  • Copy or download my provided sshd_config along with my com.chrisrjones.sshd.plist from this repo

Personally I keep those two files checked into my dotfiles repo, and symlink them to their respected locations on macOS, so I can preserve a copy of the files, and not have to modify / keep track of two separate files. 👍

curl https://raw.githubusercontent.com/ipatch/dotfiles/master/config/brew/macOS/Sierra/etc/ssh/sshd_config --output /usr/local/etc/ssh/sshd_config
curl https://raw.githubusercontent.com/ipatch/dotfiles/master/config/brew/macOS/Sierra/Library/LaunchDaemons/homebrew.mxcl.sshd.plist --output /Library/LaunchDaemons/com.chrisrjones.sshd.plist
  • Create the respected log files and the PID for the OpenSSH service.
touch /usr/local/var/log/{sshd-error.log,sshd-out.log,sshd.log}
touch /usr/local/var/run/ssh/sshd.pid
  • Next load the newly created com.chrisrjones.sshd.plist into launchd
sudo -E launchctl load -w /Library/LaunchDaemons/com.chrisrjones.sshd.plist
  • Cross fingers 🤞 and if everything goes well you should be able to see your newly built and configured sshd server running with
ps aux | grep -i "sshd"

Pat yourself on the back if you made it this far and your server is up and running, and make sure to tip your DevOps engineer.

because developers need heros too.

Gotchas Working with brewed OpenSSH 🔝

Presently the OpenSSH provided by homebrew compiles with OpenSSL by default when installing 🙄 boo

The access list for allowing users who can login remotely using SSH is still maintained via System Preferences... > Sharing > Remote Login and toggling the users by adding their name to the Only theses users: list.

In order to get password auth working with OpenSSH Server UsePAM yes must be set within the respected sshd_config file. Most recent build of OpenSSH 7.x > do not allow running sshd as a standard user, in short sshd needs to be run as the super user on the system, ie. root if on a UNIX based system. To aviod permission issues make sure the respected `ssh_host_{dsa_key,ecdsa_key,ed25519_key,rsa_key} files have a root/wheel ownership to avoid

debug1: setgroups() failed: Operation not permitted

launchctl list | grep "chrisrjones"

presently does not display the sshd service 🤷‍♂️

Troubleshooting Working with brewed OpenSSH 🔝

I am avaible for children's birthday parties and bar mitzvahs.

If launchd runs into any errors with the provided plist file, when starting sshd it should dump the error messages into /usr/local/var/log/sshd-error.log.

If there is an issue with the OpenSSH server itself, ie. a bad configuration setting with the sshd_config the server should put the error message into /usr/local/var/log/sshd.log

Uncomment the -d flag in the above mentioned .plist file to start sshd in debug mode for more versobe troubleshooting.

-d support the use of -d -d -d, ie. 3 levels of d's. When sshd is run in debug mode the server will stop running after the first disconnection.

TODOs Working with brewed OpenSSH 🔝

With your newly aquired knowledge you should be able to setup ssh-agent.

  • Setup ssh-agent to work with newly built OpenSSH server
  • See if it's possible if the homebrewed version of OpenSSH server can play nice 👨 👧 👦 with macOS System Preferences. 🤔

Setting Up X11 Forwarding through SSH 🔝

Native macOS apps won't run on *nix because their not X11 apps, but rather their cocoa apps.

To allow for X11 forwarding through SSH, edit sshd_config

X11Forwarding yes

Connecting to a X11 Server with X11 forwarding enabled 🔝 
ssh -Y [user]@[host]

Useful Links OpenSSH 🔝

i was recently requested to create a new Ed25519 ssh key for a server i needed to login to, the below medium article does a decent job of setting up a Ed25519 key

Working with Locales on macOS 🔝

To print the set locales on macOS

locale

macOS > Working with Users and Groups 🔝

TL;DR after upgrading my system from high sierra to mojave, i ended with a user account that i could not delete. long store short, the only way i could delete the undeleteable account was to delete the .AppleSetupDone file from booting into single user mode, and rerunning Setup Assitant, then i was able to grant my primary user a secure access token. macos will not let you delete the last and only user who has a secure access token, so use setup assistant to create a new user with a secure access token. use the new admin user to grant other users SAT's. after than delete the undeleteable account.

to grant a user with a secureaccesstoken after a system upgrade.

  • follow the instructions here
  • the new admin user will have a secure access token.
  • use sysadminctl to grant & check the status of secure access tokens. learn more
  • a decent blog post

to add a user to the wheel group

dscl . append /Groups/wheel GroupMembership [USERNAME]

To list all users on a macOS

dscl . list /Users

To list all groups on a macOS

dscl . list /groups

To create, edit, update, or delete a group on macOS from a CLI

⚠️ When managing groups from a CLI System Preferences.app will not update the Users & Groups pane with the updated data until the System Preferences.app is closed and then reopened.

echo "macOS group editing syntax"
dseditgroup -p -o VERB "GROUP_NAME"
echo "macOS create group example"
dseditgroup -p -o create "mr-fancies"
echo "macOS delete group example"
dseditgroup -v -p -o delete -n /Local/Default "mr-fancies"

An optional -v flag can be passed to print more verbose output to STDOUT

To add a user to a group on macOS, ie,

dseditgroup -v -p -o edit -a ipatch -t user wheel

To verify if a user is a member of a group on macOS

dseditgroup -o checkmember ntfsusers

To list all users of a particular group

dscacheutil -q group -a name [mr-fancy]

The above command should print all local users who belong to the group mr-fancy

To add a user to the system

sysadminctl -addUser [ipatch]

To print the GID UID $HOME directory, and SHELL for a specific user

dscacheutil -q user -a name mr-fancy

❗️ If only the UID of the user is known but not the name of the user then try the below example

dscacheutil -q user -a uid 42

To change the UID for mr-fancy on macOS using a CLI

dscl . -change /Users/mr-fancy UniqueID 42 4242
echo "❗️ chage the Users $HOME dir so they have access to it"
chown -R 4242 /Users/pi

To rename a user from the System Preferences using the Cocoa GUI see

Useful Links working with users and groups 🔝

Image Processing 🔝

To take a screen capture using a GUI on macOS use Grab

Grab is able to take a screen capture including the mouse cursor.

Useful Links Image Processing 🔝

Working with X11 on macOS 🔝

To install X11 windowing manager on macOS using brew

brew cask install xquartz

To prevent X11, ie. XQuartz.app from polluting the $HOME dir with .serverauth.* files

  1. Edit /opt/X11/bin/startx
  2. Change
xserverauthfile=$HOME/.serverauth.$$

to

xserverauthfile=$XAUTHORITY

Credit

Xcode 🔝

To install Xcode Command Line Tools

xcode-select --install

Install Command Line Developer Tools.app can be located within /System/Library/CoreServices

The command line developer tools, ie. clang are located in as of macOS 10.13.x with Xcode 9.4 installed.

/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/

To display all the Xcode SDKs installed on a system

xcodebuild -showsdks

To purge old symbol files for unused versions iOS used by Xcode iOS version [X.X.X] can be deleted from

$USER/Developer/Xcode/iOS DeviceSupport/[MAJOR.MINOR.PATCH]

Working with Xcode from the command line 🔝

macOS 10.13 High Sierra, file tagging using color labels appears to be broken. see

To compile an Objective-C file from the CLI on macOS, use xcrun

xcrun clang /path/to/mr-fancy-42.m -c

The above will only create an object file ie. .o of the hello world source file wrtiten in ojbc

To compile a Swift source file using the Xcode toolchain from the CLI on macOS, use xcrun

xcrun swiftc -c /path/to/mr-fancy-42.swift

To compile a binary that can be run from the CLI using ./path/to/hello-world-objc

clang -x objective-c -framework Foundation hello-world.m

The above command will create an intermediate object file within the directory that the source file, ie. hello-world.m resides. The -x flag explicity tells clang the source file is an Objective-C source file.

To syntax check a source file that can be compiled with clang use the -fsyntax-only flag

-fsyntax-only

Working with files in Finder 🔝

#Finder #HideAllWindows #HideAllApplications #macOS #OnlyShowDesktop #DisplayDesktop

To toggle the visibility 🙈 of hidden files in macOS >= Sierra

⌘+shift+.

A workflow for for hiding all windows including Finder ie. to only show the desktop is to

  1. Make sure the Finder.app is in focus ie. the Finder menu bar is presented
  2. Hide Others ⌘ + ⌥ + h a. One all other application windows have been hidden but the finder window.
  3. Close the active finder window, or even Quit the Finder.app a. Then there shouldn't be any application windows visible.

See notes contained within this document for allowing to Quit Finder

To hide all icons on the desktop

defaults write com.apple.finder CreateDesktop -bool false
killall Finder

To show all files & folders including hidden files and folders in macOS Finder

defualts wirte com.apple.finder AppleShowAllFiles TRUE
killall Finder

⚠️ on MacOS >= Sierra .DS_Store files are still hidden from Finder when setting AppleShowAllFiles TRUE

To show Finder command variables

defaults read com.apple.Finder

To prevent writing .DS_Store files to network shares

defaults write com.apple.desktopservices DSDontWriteNetworkStores TRUE

To show a specific setting for a Finder variable

defaults read com.apple.Finder AppleShowAllFiles

To show full path in Finder windows

defaults write com.apple.finder _FXShowPosixPathInTitle -bool true; killall Finder

To allow text selection in Quick Look

defaults write com.apple.finder QLEnableTextSelection -bool true

Working with PreferencePanes 🔝

Custom installed Preference Panes on macOS 10.13.x are located within /Library/PrefrencePanes/

Try and avoid putting custom Preference Panes within /System/Library/PreferencePanes because macOS does not like 3rd party files in that location. The above mentioned directory is 755 with root owner and wheel group perms by default.

IMHO I put my daily driver user echo (whoami) in the wheel group and set the permissions to 775, so my $USER could create custom PreferencePanes if needed.

Working with files via CLI 🔝

To delete all .DS_Store files in the current working directory & and all sub directories recursively

find . -name '.DS_Store' -delete

Tweaking macOS settings 🔝

❗️ tags macOS osx os x settings setting customize

To disable app icons from bouncing in the dock

defaults write com.apple.dock no-bouncing -bool TRUE
killall Finder
echo "inverse `TRUE` to reenable dock bounce"

macOS menu bar icon entries located in the following path

/System/Library/CoreServices/Menu Extras

Ex

/System/Library/CoreServices/Menu Extras/Volume.menu

To automatically reopen windows for an application after it has been closed on macOS

 → System Preferences... → General

restore-app-windows-macOS

Tweaking macOS settings from the CLI 🔝

To check and verify if SIP is enabled

csrutil status

Working with macOS boot-args

❗️when updating boot-args for the NVRAM on macOS, remember that setting new boot-args will overwrite existing args so make sure that the old / existing args are printed first before adding new args or the old/existing will be deleted
separate each additional arg with a space ␣

echo "print current boot-args"
nvram boot-args
echo "add multiple boot-args"
sudo nvram boot-args="-v amfi_get_out_of_my_way=1"

To disable the macOS boot chime

nvram SystemAudioVolume=" "

To always boot macOS in verbose mode, ie. typically how a GNU/Linux displays boot messages when booting

sudo nvram boot-args="-v"

To disable boot-args ie. change macOS from booting verbosely

sudo nvram boot-args=

To check and see if verbose booting is enabled on macOS

nvram -p | grep -i "boot-args"

To always have the save dialog / panel expanded

defaults write NSGlobalDomain NSNavPanelExpandedStateForSaveMode -bool true
defaults write NSGlobalDomain NSNavPanelExpandedStateForSaveMode2 -bool true

To automatically illuminate built-in MacBook keyboard in low light

defaults write com.apple.BezelServices kDim -bool true

To turn off keyboard illumination when computer is not used for 5 minutes

defaults write com.apple.BezelServices kDimTime -int 300

To disable shadow in screenshots

defaults write com.apple.screencapture disable-shadow -bool true

When editing system settings using defaults write settings for $USER are stored in /Users/$USER/Library/Preferences as a Apple binary Plist file.

macOS Automating CLI tasks 🔝

The below directories run commands automatically

/etc/periodic/daily
/etc/periodic/weekly
/etc/periodic/monthly

To run the periodic scripts in macOS

periodic daily weekly monthly

macOS Automating tasks on macOS >= Sierra 🔝

General launchd commands 🔝

The preferred way to load a launchd services

launchctl bootstrap gui/[USER_ID] ~/Library/LaunchAgents/com.domain.service-name.plist

The preferred way to unload a launchd service

launchctl bootout gui/[USER_ID] ~/Library/LaunchAgents/com.domain.service-name.plist

To find the user id for a the current $USER operating the $SHELL

id -u

Setting up a launchd service 🔝

macOS does not support automating tasking using cron anymore. The preferred way is to use launchd to load a .plist file.

  • Create a com.chrisrjones.mr-fancy-task.plist file within,
$HOME/Library/LaunchAgents/

  • Generate a .plist file via this

  • Load the .plist file via

launchctl load -w ~/Library/LaunchAgents/com.chrisrjones.mr-fancy-task.plist

Useful Links macOS Automating tasks 🔝

Working with Audio on macOS 🔝

To determine the value of SystemAudioVolume

nvram SystemAudioVolume

Working with Power Management on macOS 🔝

To check the remaining time the system can run on battery power

pmset -g batt

To get an approximate time of how long a macOS system has been running on battery 🔋 power see Activity Monitor.app

To display all settings related to power management on macOS

pmset -g

Experimenting power management settings on macOS 🔝

To change the standbydelay time on macOS using pmset

sudo pmset -a standbydelay [INTEGER]

The above integer is in milliseconds

Ex

To set the standbydelay time on macOS for 12 hours

sudo pmset -a standbydelay 43200000

To disable standby ie. deep sleep on macOS

sudo pmset -a standby 0

Working with Disk Images 🔝

To convert an ISO file system to a macOS image file

hdiutil convert /path/to/ISO -format UDRW -o /path/to/destination

To create an ISO to be burnt as a data DVD for PC and MAC

mkisofs -J -r -o myimage.iso data_dir

mkisofs is not native on macOS 10.12.x

To copy the contents of an ISO to a USB drive

dd if=/path/to/myISO.iso of=/dev/myUSB bs=1m

Creating a Bootable Windows 10 USB Install Drive for Intel Macs 🔝

  • Rufus can be used to create a bootable USB Windows 10 install disk. Rufus provides options to create a bootable USB for both BIOS and UEFI systems. 👍

The quickest and most painless way of creating a bootable Windows 10 USB drive on macOS is to use Boot Camp Assistant provided by Apple.

🚨 A bootable Windows 10 USB Install drive created with Boot Camp Assistant will only work with a Intel Mac, ie. modern MacBook Pro, and will not work with other UEFI laptops! Etcher.app will NOT be able to create a bootable Windows 10 USB drive. 😩

Troubleshooting modern Intel Mac hardware 🔝

SMC a Intel Mac controls the the power management features of a Mac.

To reset SMC on an Intel Mac without a touch bar

  1. Press and hold shift + control + option
  2. Keep the keys pressed for 10 seconds while holding the power button.
  3. Release the power button along with the three keys
  4. Then press the power button again to turn on the computer.

To check and see if the EFI on a Intel Mac is 32bit or 64 bit

ioreg -l -p IODeviceTree | grep firmware-abi

The output from the above command should either ouput "EFI64" or "EFI32"

To boot an Intel Mac, ie. a modern MacBook Pro, ie. 8,2 version into single-user mode.

⌘+s after booting up the computer, and shortly after hearing the Mac boot chime.

To reset the NVRAM on a modern MacBook

⌘+option+p+r

To check and see if an Intel Mac is using a 32 or 64 bit UEFI system

ioreg -l -p IODeviceTree | grep firmware-abi

The above command 👆 should either print EFI64 or EFI32

To boot an Intel Mac into Internet Recovery Mode

option+⌘+R or shift+option+⌘+R

To boot an Intel Mac into Apple Diagnostic/Hardware Test mode

  1. Turn the computer completely off
  2. Hold option+D after the startup chime, select the appropiate WIFI network, and then the Mac should download the appropiate Apple Diagnosit/Hardware Test for the Mac.

Apple changed the Apple Diagnostic/Hardware Test program some time around 2013, and Macs prior to 2013 user an older blue GUI version of the test.

Working with file systems and disks, ie. storage devices 🔝

to increase an apfs partition / filesystem to fill an entire storage medium

diskutil apfs resizeContainer [DISK[NUM]s[PARTITION]] 0
echo "example"
diskutil apfs resizeContainer disk0s2 0

macOS running a APFS filesystem appears to be using a 4KB sector size for files on the filesystem, however there's some Apple magic smoke going on with the filesystem. Running the below command

diskutil list

will present two disks, internal and synthesized, and when running

diskutil info / | /usr/bin/grep "Block Size"
echo "or use the below command to determine the file system block size"
stat -f %k .

the sector size of the partition is listed as 4KB sectors. that's not to say that APFS could be doing some trickery to emulate 512 byte sectors.

running the same command on /dev/disk0

diskutil info /dev/disk0 | /usr/bin/grep "Block Size"

dispaly 512 sectors

💡 possibly a quicker way to securely wipe a macos filesystem, turn on file vault, and encrypt the filesystem, and never record the hash for the encryption key, and the data should be unrecoverable, a la ransomware.

it does take a considerably amount of time to encrypt a large filesystem with file vault, but is def less destructive to the drive than writing to it with several passes.

To securely wipe a partition on a disk, ie. storage medium

/path/to/shred --verbose --random-source=/dev/urandom -n1 /dev/disk[Number]s[Partition]

A quick way to list the file system, ie. HFS+ for a particular mount point

diskutil info -all

or

diskutil list

To display / show all file systems accessible to a macOS system

mount

No flags or arguments are required to display the currently mounted file systems on macOS, and the above command should the mounted disks, ie. a removable USB drive and their respected file systems accessible to macOS.

To unmount a file system on macOS

diskutil unmountDisk /path/to/disk

The above command will unmount all volumes associated with a disk.

Ex

diskutil unmountDisk /dev/disk2

To quick and dirty way to see the type of partition table a disk using on macOS

diskutil list

The above command should 99% of the time list volume 0 as either GUID_partition_scheme or FDISK_partition_scheme

Ex

/dev/disk3 (external, physical):
0: GUID_partition_shceme

To format a file system on macOS with a specific format

newfs_* -F /path/to/device

To format a USB disk with a MSDOS FAT 16 file system on macOS

Ex

newfs_msdos -v "MR_FANCY_VOLUME" -F 16 /dev/disk2

BSD derivatives of UNIX use newfs_[FS_TYPE] to where as GNU derivatives use mkfs.[FS_TYPE]

On macOS an alternative way to format a disk is to use diskutil

Ex

diskutil eraseDisk "MS-DOS FAT16" MR_FANCY_DISK_NAME /dev/disk2

To copy / burn the contents of an ISO image to a USB drive or other removable storage

sudo dd if=/path/to/mr-fancy-42.iso of=/dev/disk[NUMBER]

sudo will be required to write to block level device

Protip to print a status of the transfer on macOS

control+shift+T

To display an active progress bar when using a command such as dd use pv

pv /path/to/mr-fancy-42.iso | sudo dd of=/dev/disk[NUMBER]

Working with FAT-32 file systems 🔝

To the best of my knowledge the largest file a FAT-32 file system can store is 4GB.

To mount a FAT[16-32] file system on macOS using diskutility

diskutil mount /path/to/blockDevice

Ex

diskutil mount /dev/disk3s2

The above command will mount the file system within the /Volumes directory on the local macOS system with the respected volume label as the mount point.

To unmount a volume / partition on macOS with FAT[16-32] file system.

diskutil umount /path/to/mount/point

The path to the block device can also be specified.

Working with large files on a FAT-32 file system 🔝

To split a really large file into chunks 🤮 that can be copied to a FAT-32 file system

rar a -v4G /path/to/mr-fancy-42-universe.rar /path/to/mr-fancy-42-universe.qcow2

An alternative to using rar is using GNU Core Utilites to split the file using the aptly named split command.

split --bytes=3500M /path/to/mr-fancy-42-universe.qcow2 /path/to/output/splits/mr-fancy-42-universe.qcow2

When using the split command, specifying -bytes=4G will create files too large for a FAT32 file system.

Transferring large files to a FAT-32 file system 🔝

cp is great for copying files, however it lacks any sort of progress bar, but there are various bolt-on 🔩 solutions, ie. piping through pv or using some form of python, but from my personal expierence rsync is the best solution for transferring large files.

rsync -ah --progress /path/to/mr-fancy-42-universe.qcow2 /path/to/external/media/fat32/mr-fancy-42-universe.qcow2

-a will preserve file permissions 👍 FAT32 doesn't support permissions though -h will make the progres bar in human readable output --progress will transfer the file will outputting the progress to STDOUT

Building GNU GRUB v2 on macOS from source 🔝

  • Clone the grub source
git clone git://git.savannah.gnu.org/grub.git
  • Clone objconv from GitHub
git clone [email protected]:vertis/objconv.git

GRUB will not compile using Xcode tooling out of the box, thus the dependency for objconv

  • Build objconv 
g++ -o objconv -O2 src/*.cpp
  • Put objconv in the $PATH
  • Build GRUB to support x86_64 and efi 
cd /path/to/src/grub
mkdir grub-build
../configure --prefix=/path/to/put/the/grub/binary --target=x86_64 --with-platform=efi
make
make install

make should be run the grub-build dir and NOT the grub git source project root.

  • Optional Specify a font to use with GRUB, on macOS most system fonts can be found in /Library/Fonts, that said, choose a .ttf font for the below command.
/path/to/grub-mkfont -o share/grub/unicode.pf2 /Library/Fonts/Times\ New\ Roman.ttf
  • Create a build dir preferably in the git source root for building a self contained GRUB binary to work with a UEFI system, ie. a MacBook
cd /path/to/grub/git/src
mkdir build
  • Create a grub.cfg file, and put the following lines within the file
insmod part_msdos
set root=(hd0,msdos1)
  • Build a standalone GRUB binary to bootstrap an ISO, ie. an install media from a USB drive.
/path/to/bin/grub-mkstandalone -O x86_64-efi -o BOOTX64.EFI "boot/grub/grub.cfg=build/grub.cfg"

If all goes well there should be a BOOTX64.EFI file within the build/ directory.

Install GRUB on a removable USB drive 🔝

To install the GRUB bootloader on a removable USB drive using macOS

grub-install -v --target=x86_64-efi --efi-directory=/mnt/[USB-DRIVE-LABEL] --boot-directory=/Volumes/[USB-DRIVE-LABEL]/boot --removable --bootloader-id=grub

The above command will not succeed on a removable USB drive if --removable is omitted from the command. If the above command succeeds the following message will output to STDOUT

Installation finished. No error reported
grub-install alternative 🔝

To install grub on a GPT formatted removable USB disk using macOS a couple of steps will need to be performed

  • Verify the disk has been formatted with a GPT parititon table, and not an MBR partition table.
diskutil list
  • set a path to where the EFI partition can be mounted, and you have read/write access to the path.

Ex

mkdir /mnt/usb-gpt-efi
sudo chown -R $USER:wheel /mnt/usb-gpt-efi

If the above directory is created within the /Volumes directory then the mount point, ie. directory that holds the mounted file system will be deleted.

  • mount the EFI partition of the USB disk
sudo mount -t msdos /dev/disk3s1 /Volumes/usb-gpt-efi
  • Verify the partition has properly been mounted.
mount
  • Install GRUB boot loader to the USB drive to work with a UEFI system / boot rom.
grub-install -v --target=x86_64-efi --efi-directory=/Volumes/usb-gpt-efi --boot-directory=/Volumes/grub-for-mac/boot --removable --bootloader-id=grub

macOS and GRUB Useful Links 🔝

refind is the successor to refit

Working with the bless command on macOS 🔝

In order to boot from a removable media, ie. a USB flash drive on macOS a drive or Volume needs to blessed, using the native bless command from macOS or using the hfs-bless provided by mactel linux.

To see / print the currently blessed disk that will boot macOS

bless -getBoot

Working with automountd 🔝

automountd is a service provided by macOS that can automount file systems, disks, local & remote, ie. across the network.

The configuration files for automountd

/etc/autofs.conf
/etc/auto_master
/etc/auto_resources

To show the file systems / partitions that have been automatically mounted

automount -v

If a change is made to one of the above files, the cache needs to be flushed

automount -cv

Some useful man pages include

man automount
man automountd
man auto_master
man autofs.conf

Benchmarking HDD performance 🔝

To test HDD performance on a macOS / Linux system, see

A general universal command to test read / write performance to a storage media

time sh -c "dd if=/dev/zero of=/tmp/testfile bs=100k count=1k && sync"

Results running macOS 10.12.6 with a HFS+ filesystem on a 1TB Samsung SSD

╭─capin at rogue in /opt/Code/dotfiles (master ✔)
╰─λ time sh -c "dd if=/dev/zero of=/tmp/testfile bs=100k count=1k && sync"
1024+0 records in
1024+0 records out
104857600 bytes transferred in 0.212054 secs (494485334 bytes/sec)
        0.34 real         0.00 user         0.16 sys

╭─capin at rogue in /opt/Code/dotfiles (master ✔)
╰─λ ls -lah /tmp/testfile
-rw-r--r--  1 capin  wheel   100M May 19 20:24 /tmp/testfile
╭─capin at rogue in /opt/Code/dotfiles (master ✔)
╰─λ time sh -c "dd if=/dev/zero of=/tmp/testfile2 bs=100k count=1k && sync"
1024+0 records in
1024+0 records out
104857600 bytes transferred in 0.086153 secs (1217108667 bytes/sec)
        0.20 real         0.00 user         0.12 sys

╭─capin at rogue in /opt/Code/dotfiles (master ✔)
╰─λ time sh -c "dd if=/dev/zero of=/tmp/testfile3 bs=100k count=1k && sync"
1024+0 records in
1024+0 records out
104857600 bytes transferred in 0.299482 secs (350129766 bytes/sec)
        0.49 real         0.00 user         0.18 sys

Building macOS install / boot media 🔝

To create a bootable macOS High Sierra install media on a removable USB disk drive.

Make sure Install macOS High Sierra.app has been downloaded from the App Store.

sudo /Applications/Install\ macOS\ High\ Sierra.app/Contents/Resources/createinstall \
--volume /Volumes/Install-macOS-High-Sierra \
--applicationpath /Application/Install\ macOS\ High\ Sierra.app

To create a bootable macOS Mavericks install media USB

sudo /Applications/Install\ OS\ X\ Mavericks.app/Contents/Resources/createinstallmedia \
--volume /Volumes/<MEDIA> \
--applicationpath /Applications/Install\ OS\ X\ Mavericks.app --nointeraction

Building a USB drive / install media for an Intel Mac with GRUB 🔝

directory layout

EFI
  debian
    grubx64.ef
mach_kernel
System
  Library
    CoreServices
      boot.efi
      SystemVersion.plist
.VolumeIcon.icns

The contents of the SystemVersion.plist file will roughly look like the following

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
"http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>ProductBuildVersion</key>
  <string></string>
  <key>ProductName</key>
  <string>GNU+Linux</string>
  <key>ProductVersion</key>
  <string>GRUB V2</string>
</dict>
</plist>

Disk Management 🔝

To mount an ISO file from a CLI in macOS

hdiutil mount /path/to/file.iso

To eject / unmount an ISO file in macOS

hdiutil dettach /dev/device
echo "example"
hdiutil detach /dev/disk2

To enable Debug mode for Disk Utility.app

defaults write com.apple.DiskUtility DUDebugMenuEnabled 1

The above command may not be applicable for macOS High Sierra as Disk Utility by default can show all partitions.

To list the supported file systems a disk can formatted with

diskutil listFilesystems

To list the available disks attached to the system

diskutil list

To erase a Volume using diskutil and not allocate the free space with a partition.

diskutil eraseVolume free %noformat% [IDENTIFIER]

[IDENTIFIER] can be printed using diskutil list

To erase an entire disk and not just a partition / volume on a disk and allocate with free space

diskutil eraseDisk free %noformat% [IDENTIFIER]

The [IDENTIFIER] can be obtained with diskutil list, and verified with the /dev/disk[NUMBER], also using the above command will format the disk with a GUID partition map, ie. known as a Scheme in the macOS world.

Ex

diskutil list

Output

/dev/disk3 (external, physical):
...

To resize a partition for a particular disk, ie. a USB flash drive using diskutil on macOS, use the eraseDisk in conjunction with the resizeVolume command.

From my empirical evidence (could be wrong on this assessment) using diskUtil on macOS it's not possible to create a partition with a specific size from a free space drive using diskUtil on macOS. The reason for this is because there are no logical volumes, ie. partitions on the disk, that said, that's why the simplest solution I found was to create a JHFS+ partition that can be resized at later date and time.

  1. Create a JHFS+ partition that occupies all free / empty space for a disk.
diskutil eraseDisk JHFS+ grub-for-mac /dev/disk3

About the above command, eraseDisk will nuke 🤯 the entire disk ie. removing all data from it, grub-for-mac will be the volume label for the disk, and /dev/disk3 specifies the path to the disk to put the JHFS file system and partition on.

  • Validate the disk is properly partitioned and formatted.
diskutil list

Should see ~ 3 partitions indexed at 0

  1. GUID_partition_scheme
  2. EFI
  3. Apple_HFS
  • Next resize the JHFS+ partition to a smaller size
diskutil resizeVolume /dev/disk3s2 512M

To get an idea of what size a file system on a partition / logical volume can be

diskutil resizeVolume /dev/disk3s2 limits

There now should be a partition on the removable USB disk that is ~ 512MB in size and formatted with a JHFS+ file system.

To create an additional partition on the above mentioned disk with a 512MB JHFS+ partition use an external program named gdisk.

  1. gdisk can be easily obtained if Homebrew is installed.
brew cask install gdisk

⚠ To modify any attributes associated with a disk gdisk needs be launched with super user privileges.

  • Launch gdisk 
sudo gdisk
  • Specify path to disk
/dev/disk[NUMBER]
  • Create a partition with they type hex code of 0700 so a FAT file system can be formatted to the partition.

To create a partition with a specific size using gdisk ie. create a 5GB on a 32GB disk use +5G

  • Format the newly created partition with a FAT-32 file system
diskutil eraseVolume FAT32 [VOLUME_NAME] /dev/disk[ID_PARTITION_NUMBER]

To erase a disk and format with a particular file system

diskutil eraseDisk [fileSystem] [label / name] [/path/to/disk]

Ex

diskutil eraseDisk fuse-ext2 samsung-1tb /dev/disk2

To create Linux compatible file system for a particular partition on a disk

/opt/gnu/sbin/mkfs.ext2 /dev/disk2s2

🚨 The above mentioned command will create a ext2 formatted filesystem. If you'd prefer to use an ext4 filesystem, use mkfs.ext4 instead. Also mkfs.ext2 requires tools to be properly installed and setup on macOS in order to format a disk with a ext{2,3,4} file system.

To disable / turn off journaling for a hfs+ disk

sudo diskutil disableJournal /Volumes/mr-fancy-disk-label

Working with fuse-ext2 🔝

fuse-ext2 relies on osxfuse and from what I have gathered the last free open source version of osxfuse is 3.8.3 without any licensing limitations, so I will manually install that from the github releases page as oppsoed to continually updating via brew.

To mount an external disk with an ext{2,3,4} file system using fuse-ext2

- sudo -E mount -t fuse-ext2 /path/to/external/disk/partition /path/to/fancy/mnount/point
+ sudo fuse-ext2 /path/to/block/device/partition /path/to/mount/point -o rw+,allow_other,uid=[INT],gid=[INT]

Ex

sudo fuse-ext2 /dev/disk2s2 ~/mnt/sdcard -o rw+,allow_other,uid=501,gid=20

sudo must be used to mount the file system.

To verify if the file system successfully mounted using FUSE for macOS

mount

The above command will display all locally mounted file systems.

To unmount a fuse-ext2 file system

sudo -E unmount /full/path/to/mount/point

fuse-ext2 Gotchas 🔝

When working with fuse-ext2 to mount ext{2,3,4} file systems, presently sudo is required to mount a file system boo. In order to mount a file system and to allow access for a particular user on the system

sudo -E fuse-ext2 /path/to/disk/s[num] /path/to/mount/point -o rw+,allow_other,uid=[num],gid=[num]

Ex

sudo -E fuse-ext2 /dev/disk2s2 ~/mnt/ext-m2-1tb -o rw+,allow_other,uid=501,gid=20

Useful Links Disk Management 🔝

Working with Java 🔝

To completely remove / uninstall Java JRE and JDK

rm -rf /Library/Java/JavaVirtualMachines/jdk<version>.jdk

rm -rf /Library/PreferencePanes/JavaControlPanel.prefPane
rm -rf /Library/Internet\ Plug-Ins/JavaAppletPlugin.plugin
rm -rf /Library/LaunchAgents/com.oracle.java.Java-Updater.plist
rm -rf /Library/LaunchDaemons/com.oracle.java.Helper-Tool.plist
rm -rf /Library/Preferences/com.oracle.java.Helper-Tool.plist

To completely remove Android Studio, see this

Microsoft Silverlight on macOS

To remove / uninstall Silverlight on macOS, perform a search using GNU find or locate and manually remove all files.

Ex

find / -iname "*silverlight*" -type f 2>/dev/null

Working with launchd 🔝

TL;DR

  • working with launchd / launchctl is a total pain the ass, that said, tip 💵 your dev ops engineers next time you see them.
  • when loading or unloading a launchd service, ie. a .plist file within $HOME/Library/LaunchAgents refer to the .plist file, ie.
launchctl [load/unload] $HOME/Library/LaunchAgents/mr-fancy.plist

...and NOT the Label in the .plist file.

  • when testing a launchd service refer to the value in defined below the Label within the .plist, ie.
launchctl list com.example.mr-fancy-42-service
  • if StandardErrorPath & StandardOutPath or defined in the .plist file / service, launchd should create the files if not created yet, NOTE double check permissions of directory where files will be written.
  • launchd should append to the files defined by StandardErrorPath & StandardOutPath and not overwrite them each time the service is started. 👌

To get a frame of reference on how launchd service files, ie. .plist files can be written, see /System/Library/Launch{Angents,Daemons}

From all my empirical evidence, environment variables can not be set within the StandardOutPath or the StandardErrorPath

launchd General Notes 🔝

A daemon is a program that runs in the background as part of the overall system (that is, it is not tied to a particular user). A daemon cannot display any GUI; more specifically, it is not allowed to connect to the window server.

An agent is a process that runs in the background on behalf of a particular user. Agents are useful because they can do things that daemons can't, like reliably access the user's home directory or connect to the window server.

If a launchd service / .plist file calls / runs a shell script one can verify the the shell script contains no errors by passing the -n flag as an arugument to the shell script, ie.

sh -n mr_fancy_shell_script.sh

If the above command does not print any output in the terminal then their aren't any syntax errors using that particular shell interpreter.

To test / lint a .plist file for syntax errors

plutil -lint $HOME/Library/LaunchAgents/com.mr-fancy.plist

To manually / arbitrarily start a launchd service / .plist

- launchd start $HOME/Library/LaunchAgents/com.mr-fancy.plist

To start a launchd service as a specific user via a .plist file

<key>UserName</key>
<string>mr-fancy-user</string>

Working with Environment Variables in Launchd 🔝

To create an environment variable in launchd add the below to the respected .plist file.

-<array>
-  <string>/bin/launchctl</string>
-  <string>setenv</string>
-  <string>MR_FANCY</string>
-  <string>42</string>
-</array>
+<key>EnvironmentVariables</key>
<dict>
  <key>PATH</key>
  <string>/usr/local/bin:/usr/bin:/bin</string>
</dict>

To create a launchd variable from the command line

launchctl setenv FOO "BAR"

echo $FOO will output BAR

To create a environment variable in a .plist

<string>launchctl setenv FOO "BAR"</string>
<string>launchctl setenv USER "${USER}";</string>

launchd Gotchas 🔝

launchd .plist files can contain spaces in <string> values, ie.

<string>/Mr Super/fancy/path/to/special.plist</string>

When working with launchd services, MAKE sure to remove the service if it is no longer required to use it, ie.

launchctl remove com.github.asdf-vm.plist

...otherwise the service will remain loaded even if the env var is manually removed, and the service is stopped or unloaded 😡

To get a human readable explanation of a launchd error code, ie. 127

launchctl error 127

When creating a launchd service at a system level, ie. putting a service file in /Library/LaunchDaemons/, running

lc list | grep `chrisrjones`

...will not list the service, because it is running at system level.

Useful Links launchd 🔝

TODOs launchd

  • see if environment variables can be used in .sh associated with launchd.
  • see if environment variables can be used in .plist files associated with launchd.

Working with devices on macOS 🔝

To status devices, ie. devices connected via Bluetooth or USB, macOS provides ioreg and system_profiler

Either one of the tools can be used from the CLI to status devices internal or external to a running version of macOS.

To get a crude list of devices attached to a USB bus on macOS

ioreg -p IOUSB

The -p flag will scan the ioregistry over a given plane, and in the above example the plane specified is IOUSB.

To get more verbose / detailed information about USB devices connected to macOS

ioreg -p IOUSB -w0 -l

-w0 will not constrain the output to the width of the window ,and the -l will add to the verbosity of the ioreg command.

⚠️ The -l flag is useful for listing the idProduct and idVendor

An alternative to ioreg is system_profiler

To status USB devices connected to macOS using system_profiler

system_profiler SPUSBDataType

Working with Kernel, ie. Darwin boot arguments 🔝

To manually kernel boot arguments for the Darwin kernel on macOS, edit the below file.

/Library/Preferences/SystemConfiguration/com.apple.Boot.plist

Working with Graphics Programming on macOS 🔝

Working with OpenGL on macOS 🔝

macOS does not ship with OpenGL header files for building programs against OpenGL, however Xcode can be installed to build against some of the OpenGL header files.

To the best of my knowledge Apple stopped pulling in upstream features of OpenGL as of v3.2.

macos-opengl

That said, certain apps that use Vulkan, ie. the successor to OpenGL run on macOS.

Working with Postgres 🔝

To start the homebrew implementation of PostgreSQL

launchctl load ~/Library/LaunchAgents/homebrew.mxcl.postgresql.plist

The above command will load the homebrew implementation PostgreSQL as a service using launchd on macOS.

To verify if the service is running

launchctl list | grep -i "homebrew"

To stop the newly added launchd service

launchctl stop [com.mr-fancy-service]

or

launchctl unload [com.mr-fancy-service]

Troubleshooting Postgres on macOS

Make sure the Postgres data directories on macOS have a 700 permission

/usr/local/var/postgres

Working with Screen Savers on macOS 🔝

The default location for system Screen Savers on macOS

/System/Library/Screen Savers/

The default $USER location for screen savers on macOS

~/Library/Screen Savers/

Useful Links 🔝

TODO 🔝

  • Mess around with Screen Saver project template in Xcode

Working with AppleScript 🔝

Useful Links AppleScript 🔝

Working with Messages 🔝

To instert a newline or line break using Messages

option + return

Unsorted Notes 🤷‍♂️

NoMachine is the only application installer where I had to manually perform the installation from a CLI to complete the install process because the GUI based installer would do just that stall and never complete the installation process.

echo "mount the dmg"
echo "optional, kill the existing installer if required"
sudo killall -1 installd
echo "install the NoMachine app bundle from a CLI"
sudo installer -pkg NoMachine.pkg -target /

All Useful Links 🔝

TODOs 🔝

  • ~~~Add Table of Contents to this document.~~~

If you find any of this info helpful on your journey 🏍 click that 👆 ⭐️ star button. It sure makes me feel warm and fuzzy 🐻 on the inside.

Frequent Flyers ✈️

❤️ ~/.🛠🐈

Clone this wiki locally