Node OS Setup Guide Debian 8

SolarNode OS Setup Guide Debian 8

This guide describes the steps I took to create a "minimal" Debian 8 based system with base SolarNode deployment configured on an eBox 3300 (NorhTec JrMX) computer. The overall goals were these:

No X window system.
No development tools.
No daemons or servers unless required by Debian or SolarNode.
SSH daemon for network access.
NTP daemon for time synchronization.
Monit daemon for SolarNode monitoring.
Boot from SD card.
Java 8 JRE.

With these goals in mind, let's dive in. You'll need a Linux-based system to work with.

Note: Binary images for the Node OS are also available here: http://sourceforge.net/projects/solarnetwork/files/solarnode These are great if you are after a quick OS setup (but they do not always contain the latest updates), if you want the latest and greatest you should continue reading below.

Download Debian 8 and copy to USB stick

Download the Debian 8 (jessie) installer CD. This should be the full 650MB CD, not the netboot image, so it can install the base system without needing the network. You can get the image from

http://www.debian.org/CD/http-ftp/#stable

Once you have the .iso image downloaded, copy the image to a USB stick. See https://www.debian.org/releases/stable/i386/ch04s03.html.en for information on copying the ISO image to a USB stick.

Boot into the Debian installer

Insert the Debian installer USB stick into the USB port on the back of the eBox and a 2GB or larger SD card into the SD slot. You'll need a monitor and keyboard (USB or PS/2) plugged in as well, but no mouse is required. Turn the node on, and quickly press the Delete key to enter the BIOS setup utility.

Make sure the BIOS boots from the USB stick first, under the Boot menu. The BIOS should list both the USB and SD disks, as USB: and HDD: options, respectively.

Next go to the Advanced > IDE Configuration screen and make sure the Standard IDE Compatible setting is set to Enabled.

Now you can exit the BIOS configuration - F10 to save and exit - and the Debian boot menu should appear. Choose the Expert install mode, and let the Debian installer load.

Debian install options

First I manually partitioned and formatted an ext4 filesystem, using a guide. I found the estimated block size of my SD card to be 4MB, so I halved all the settings from the guide:

First sector: 16384 (8M)
Size: 1600M (divisible by 8M and 4M)
File system: mkfs.ext4 -O ^has_journal -E stride=2,stripe-width=512 -b 2048 -L SOLARNODE /dev/mmcblk0p1

Then here are some important options I chose during the Debian install:

DHCP for network configuration, with solarnode.localdomain as the host name.
No root user, with a single solar user.
No NTP time synchronization. The installer will install ntpd but we want to use systemd-timesyncd manually later, which is smaller and easier to manage.
Use the pre-existing ext4 partition created earlier, with a label SOLARNODE and the noatime option enabled. Also mark this partition bootable.
No swap partition. This is to minimize wear on the SD card.
Select the most recent available i586 kernel.
Configure a local apt mirror, without the non-free and without the contrib repositories used (they are not needed).
Install ssh server and sytem utilities but no other software at this time.
Install the Grub boot loader (but we will be replaced by extlinux later).

At this point, the installer should complete. You can reboot the node, but un-plug the USB stick before the BIOS RAM check completes, so the node boots from the newly minted SD card.

Post install tasks

Edit /etc/fstab to replace UUID=XXX with LABEL=SOLARNODE (to make it easier to share this image) so it should look like this:
```
 	LABEL=SOLARNODE / ext4 noatime,nodiratime,errors=remount-ro 0 1
```
Edit /etc/apt/sources.list to:
Remove the cdrom entry (which is really the USB stick).
Remove all -src sources.
Add deb http://ftp.nz.debian.org/debian jessie-backports main to support Java 8.
Run apt-get update.
Add solar user to dialout group, to allow use of serial ports.
```
 usermod -a -G dialout solar
```

Make /tmp mounted as a tmpfs (RAM) filesystem with:

 systemctl enable tmp.mount
 rm -rf /tmp/*
 systemctl start tmp.mount

Disable persistent history in bash, by editing /etc/bash.bashrc and adding
```
 unset HISTFILE
```

Software setup

Now I manually removed and added the software I deemed appropriate for the node.

Replace rsyslog with busybox-syslogd, to minimize writing to the SD card:

 apt-get remove --purge rsyslog
 apt-get install --no-install-recommends busybox-syslogd

Install localepurge to remove excess locale data:#!sh
```
 apt-get install localepurge
```
Configure systemd-timesyncd
Edit /etc/systemd/timesyncd.conf and uncomment the SERVERS= line, optionally editing the default server list.
Run timedatectl set-ntp true
Remove nano and docs... any anything else you can, like exim, gcc, perl, python, etc.
```
 apt-get remove --purge nano info manpages
```

Install Java (this pulls in a whopping 250MB+ of stuff):

 apt-get -t jessie-backports install --no-install-recommends openjdk-8-jre-headless

Install rsync (required for node database backups)
```
 apt-get install --no-install-recommends rsync
```
Install RXTX and JNA Java libraries, to support serial ports in Java:
```
 apt-get install --no-install-recommends librxtx-java libjna-java
```
Then follow the RXTX setup guide to configure RXTX with the JVM.

Further savings can be found by installing the deborphan and debfoster packages. Use those to identify non-essential packages and remove them.

WiFi support

If you plan to use WiFi, you'll need to install some firmware for the radio and install the wpasupplicant package.

apt-get install --no-install-recommends wpasupplicant
You need the rtl8712u.bin firmware file, which is available in git. You can clone this repository via
```
 git clone git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git
```
Then copy the linux-firmware/rtlwifi/rtl8712u.bin file to /lib/firmware/rtlwifi/rtl8712u.bin.

Use extlinux in place of Grub bootloader

The Debian installer will have configured Grub as the boot loader, but extlinux is better for our purposes here. In order to make this system easier to clone onto other SD cards, I tweaked some of the settings so that the SOLARNODE filesystem label I set up during installation is referenced instead. See http://shallowsky.com/linux/extlinux.html for more info.

Install extlinux: apt-get install extlinux
Remove grub: apt-get remove --purge grub-pc grub-common grub-pc-bin grub2-common
mkdir /boot/extlinux
extlinux --install /boot/extlinux
dd if=/usr/lib/EXTLINUX/mbr.bin of=/dev/sda bs=440 count=1

Create a simple /boot/extlinux/extlinux.conf file with the following content:

 default linux
 label linux
   kernel ../vmlinuz-3.16.0-4-586
   append initrd=../initrd.img-3.16.0-4-586 root=/dev/sda1 ro quiet

Add udev rule file to remove MAC address associations

The Debian installer will have set up a udev rule that associates the ethernet and WiFi devices with persistent device names, eth0 and wlan0, based on those devices' hardware MAC addresses. In order to make this system easier to clone onto other SD cards, I disabled the automatic generation of this file:

# Disable the automatic, MAC-based naming rules file generation
rm /etc/udev/rules.d/70-persistent-net.rules
ln -s /dev/null /etc/udev/rules.d/70-persistent-net.rules

Next I added a custom udev rules file /etc/udev/rules.d/a10-solarnode.rules with the following content:

# Rename network interfaces NOT using MAC addresses, so this image can be copied to other devices
SUBSYSTEM=="net", DRIVERS=="?*", KERNEL=="eth*", NAME="eth%n"
SUBSYSTEM=="net", DRIVERS=="?*", KERNEL=="wlan*", NAME="wlan%n"

This will map ethernet devices to ethX and WiFi to wlanX where X starts at 0. This thus provides the eth0 and wlan0 network device names.

Use systemd-networkd for network configuration

Create /etc/systemd/network/eth.network with:

[Match]
Name=eth0

[Network]
DHCP=yes

[DHCP]
RouteMetric=10

Then to support WiFi, which will have lower precedence to the wired network, create /etc/systemd/network/wlan.network with:

[Match]
Name=wlan0

[Network]
DHCP=yes

[DHCP]
RouteMetric=20

Create unit /lib/systemd/system/[email protected] with:

[Unit]
Description=WPA supplicant daemon (interface-specific version)
Requires=sys-subsystem-net-devices-%i.device
After=sys-subsystem-net-devices-%i.device

[Service]
Type=simple
ExecStart=/sbin/wpa_supplicant -c/etc/wpa_supplicant/wpa_supplicant-%I.conf -i%I -Dwext

[Install]
Alias=multi-user.target.wants/wpa_supplicant@%i.service

Create WPA configuration /etc/wpa_supplicant/wpa_supplicant-wlan0.conf with:

network={
	ssid="ssid"
	#psk="passphrase"
	psk=2b1d17284c5410ee5eaae7151290e9744af2182b0eb8af20dd4ebb415928f726

	# if the SSID is hidden, add
	#scan_ssid=1
}

Then,

systemctl enable systemd-networkd
systemctl enable wpa_supplicant\@wlan0
systemctl enable systemd-resolved
systemctl start systemd-networkd
systemctl start wpa_supplicant\@wlan0
systemctl start systemd-resolved

Now map resolve.conf to systemd-resolved:

rm /etc/resolv.conf
ln -s /run/systemd/resolve/resolv.conf /etc/resolv.conf

Finally, can remove ifupdown:

apt-get purge ifupdown

Note as the WPA settings are just placeholders, after testing the setup works the service was disabled:

systemctl disable wpa_supplicant\@wlan0

Add firewall and redirect port 80

The SolarNode web application runs on port 8080 by default, but we'd like to be able to access it via the standard HTTP port, 80. We can use iptables to both provide a firewall for the node as well as setup NAT to translate port 80 into 8080 for us. This is adapted from Arch Linux. In addition we will configure support for SSH password brute-force mitigation by dynamically blocking IP addresses after failed password login attempts via SSH.

First create a configuration file suitable for iptables-restore to read, at /etc/iptables/iptables.rules:

*filter

# Create chain for SSH brute-force password cracking mitigation
-N dropBrute

# Allows all loopback (lo0) traffic and drop all traffic to 127/8 that doesn't use lo0
-A INPUT -i lo -j ACCEPT
-A INPUT ! -i lo -d 127.0.0.0/8 -j REJECT

# Allow all established inbound connections
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

# Allow outbound traffic
-A OUTPUT -j ACCEPT

# Allows HTTP
-A INPUT -p tcp --dport 80 -j ACCEPT
-A INPUT -p tcp --dport 8080 -j ACCEPT

# Allow SSH 
-A INPUT -p tcp -m state --state NEW --dport 22 -j dropBrute
-A INPUT -p tcp -m state --state NEW --dport 22 -m limit --limit 6/min --limit-burst 6 -j ACCEPT

# Allow ping
-A INPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT

# log iptables denied calls (access via 'dmesg' command)
-A INPUT -m limit --limit 5/min -j LOG --log-prefix "iptables denied: " --log-level 7

-A INPUT -j REJECT
-A FORWARD -j REJECT

COMMIT

*nat

# Redirect port 80 to 8080 for SolarNode
-A PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 8080

COMMIT

Create /lib/systemd/system/iptables.service with:

[Unit]
Description=Packet Filtering Framework
DefaultDependencies=no
After=systemd-sysctl.service
Before=sysinit.target

[Service]
Type=oneshot
ExecStart=/sbin/iptables-restore /etc/iptables/iptables.rules
ExecReload=/sbin/iptables-restore /etc/iptables/iptables.rules
ExecStop=/lib/systemd/scripts/iptables-flush
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

Create /lib/systemd/scripts/iptables-flush with:

#!/bin/bash
#
# Usage: iptables-flush [6]
#

iptables=ip$1tables
if ! type -p "$iptables"; then
  echo "error: invalid argument"
  exit 1
fi

while read -r table; do
  tables+=("/var/lib/$iptables/empty-$table.rules")
done <"/proc/net/ip$1_tables_names"

if (( ${#tables[*]} )); then
  cat "${tables[@]}" | "$iptables-restore"
fi

Make it executable:

chmod 755 /lib/systemd/scripts/iptables-flush

Create /var/lib/iptables/empty-filter.rules with:

# Empty iptables filter table rule file
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
COMMIT

Create /var/lib/iptables/empty-nat.rules with:

# Empty iptables nat table rules file
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
COMMIT

Now enable the iptables service:

systemctl enable iptables
systemctl start iptables

You can check the filter and nat tables with:

iptables -L
iptables -L -t nat

Configure SSH password brute-force attack mitigation

Create a /usr/share/solarnode/drop-brute.sh script using the script on GitHub.

Make this executable:

chmod 755 /usr/share/solarnode/drop-brute.sh

Then add a file /etc/cron.d/drop-brute with the following content, to run the drop-brute.sh script every 2 minutes:

# SSH password brute-force mitigation
*/2 * * * * root /usr/share/solarnode/drop-brute.sh >/dev/null 2>&1

Configure systemd-tmpfiles

Create a memory-only location for application files under a /run/solar directory, by adding a /usr/lib/tmpfiles.d/solarnode.conf file with the following content:

# SolarNode tmpfile configuration

# Type Path Mode UID GID Age Argument

# Create primary work area
d /run/solar 0755 solar solar - 

# For history compatibility, create symlink
L /run/shm/solar - - - - /run/solar

# Do not clean up any files in these areas
x /run/solar/*
x /run/shm/solar/*

Configure journald, systemd

Edit /etc/systemd/journald.conf to limit the size of the systemd journal:

[Journal]
RuntimeMaxUse=10M

Edit /etc/systemd/system.conf to limit the size of the systemd journal:

[Manager]
DumpCore=no

Configure kernel to reboot on error

If the SolarNode process (or any process) becomes unresponsive, we can make the kernel panic and reboot by adding a file /etc/sysctl.d/solarnode.conf with the following content:

# Reboot 5 seconds after panic
kernel.panic = 5

# Panic if a hung task was found
kernel.hung_task_panic = 1

# Setup timeout for hung task to 300 seconds
kernel.hung_task_timeout_secs = 300

Delete SSH key and set up to get created at boot

If you're creating this as an image for many nodes to copy from, then you should delete the SSH key Debian generated for you, and add a small systemd unit to recreate the keys the next time the OS boots. First, delete the keys:

rm -f /etc/ssh/ssh_host_*

Then add a /lib/systemd/system/sshdgenkeys.service unit file with:

[Unit]
Description=SSH Key Generation
ConditionPathExists=|!/etc/ssh/ssh_host_key
ConditionPathExists=|!/etc/ssh/ssh_host_key.pub
ConditionPathExists=|!/etc/ssh/ssh_host_rsa_key
ConditionPathExists=|!/etc/ssh/ssh_host_rsa_key.pub
ConditionPathExists=|!/etc/ssh/ssh_host_dsa_key
ConditionPathExists=|!/etc/ssh/ssh_host_dsa_key.pub
ConditionPathExists=|!/etc/ssh/ssh_host_ecdsa_key
ConditionPathExists=|!/etc/ssh/ssh_host_ecdsa_key.pub
ConditionPathExists=|!/etc/ssh/ssh_host_ed25519_key
ConditionPathExists=|!/etc/ssh/ssh_host_ed25519_key.pub

[Service]
ExecStart=/usr/bin/ssh-keygen -A
Type=oneshot
RemainAfterExit=yes

Create /etc/systemd/system/ssh.service.d/local.conf to add dependency on sshdgenkeys.service:

[Unit]
Wants=sshdgenkeys.service
After=network.target auditd.service sshdgenkeys.service

Note the Wants line was added, while the After line was edited to add sshdgenkeys.service. Finally, reload the configuration:

systemctl daemon-reload

Clean Up

As a final clean up to free space, run apt-get clean. The sfill program, available in the secure-delete package, can help make compressed images created with dd smaller by forcing all unused blocks to 0. Run sfill -f -z -I -ll / to zero out all unused space. This can take some time to run.

Install base SolarNode platform

Now we'll deploy a basic SolarNode platform, and configure it to startup when the node boots. See Deploying the SolarNode application for more information.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly