CentOS7
Before you run yum update, you should probably stop services, e.g.
systemtl stop jupyterhubThen run and reboot:
yum update -y
rsc
rebootYou'll need to reconnect to the idrac_console.
After reboot, remove the old kernel:
yum remove $(rpm -qa | grep ^kernel-3 | grep -v $(uname -r))Edit /etc/dhcpd/dhcpd.conf on the dhcpd server and restart:
# Add next-server and filename lines (search in file)
systemctl restart dhcpdReboot target machine.
Run yum update:
sed -i.bak -e 's,^mirrorlist=http://mirrorlist,#mirrorlist=http://vault,' -e 's,^#baseurl=http://mirror.centos.org,baseurl=https://depot.radiasoft.org/yum,' /etc/yum.repos.d/CentOS-Base.repo
yum update -yOn host change root password & add luks key on root device (/dev/md1 or /dev/sda2):
passwd
cryptsetup luksAddKey <rootdev>On the rsconf server:
bash run.sh host init <host>This command will output some text that is copied to a shell on the freshly installed machine. The output looks something like:
install -m 600 /dev/stdin /root/.netrc <<'EOF'
machine <snip>
EOF
curl <snip>This will run rsconf on the machine and ask you to reboot (to turn off ipv6). Then you will need to return the same curl command to finish the installation.
Note (12/2/20): the network didn't start after this boot, because the network manager may not like that ifcg gets updated. ifup worked, and then the next rsconf run removed network manager.
After reboot on host:
# might do nothing
yum remove $(rpm -qa | grep ^kernel-3 | grep -v $(uname -r))
cryptsetup luksRemoveKey <rootdev>
# enter temporary key
curl https://rsconf.radiasoft.org | install_server=https://rsconf.radiasoft.org bash
exit
ssh <host>
cat /dev/null > ~/.bash_history
exitDon't forget to remove any old keys in .ssh/known_hosts files and ssh <host> true to update them.
- ImportError: No module named updates_disk_hook happens if no disks are recognized.
You will have to do this if the kickstart fails, because it
has started md on a drive(s).
mdadm --stop /dev/md/3
mdadm --zero-superblock /dev/sda
mdadm --zero-superblock /dev/sdb
dd bs=512 if=/dev/zero count=2048 of=/dev/sda
dd bs=512 if=/dev/zero count=2048 of=/dev/sda seek=$((`blockdev --getsz /dev/sda` - 2048))
dd bs=512 if=/dev/zero count=2048 of=/dev/sdb
dd bs=512 if=/dev/zero count=2048 of=/dev/sdb seek=$((`blockdev --getsz /dev/sdb` - 2048))To completely unwind a mounted LVM on top an encrypted RAID:
umount /scratch
lvremove /dev/mapper/nfs-scratch
vgremove nfs
pvremove /dev/mapper/luks-nfs
cryptsetup remove /dev/mapper/luks-nfs
mdadm --stop /dev/md/3
mdadm --zero-superblock /dev/sda
mdadm --zero-superblock /dev/sdbThis might not work:
yum install -y ledmon
ledctl locate=/dev/sdd
ledctl off=/dev/sddAnother option is:
dd if=/dev/sdd of=/dev/nullMirrors should be setup with two drives each. If you setup a mirror with --num-devices=4, say, it will be 4 copies of the same data, not 2x2. You need RAID1+0 (RAID10) for spanning multiple drives in a mirror, but this will stripe the encrypted data, which means in a catastrophe, you'll never be able to recover anything. Rather, setup mirror pairs, encrypt them, create physical volumes (PVs), and then form into a single volume group. Reference: https://linuxgazette.net/140/pfeiffer.html
Find the devices:
lsblk -o +UUID
lsscsiCreate the RAID array(s):
mdadm --create --verbose --metadata 1.2 /dev/md/3 --level=1 --raid-devices=2 /dev/sda /dev/sdb
mdadm --create --verbose --metadata 1.2 /dev/md/4 --level=1 --raid-devices=2 /dev/sdc /dev/sdd
# update /etc/mdadm.conf by hand (eliminating name=, confusing and unnecessary)
mdadm --detail --scan >> /etc/mdadm.conf
emacs /etc/mdadm.conf
cat /proc/mdstatWait for RAID array to be completely rebuilt.
Encrypt each RAID disk with the same key and create PVs:
vgname=nfs
mkdir -m 700 -p /etc/luks
keyfile=/etc/luks/$vgname
emacs $keyfile
# remove newline or create another way
tr -d '\012' < $keyfile | dd of=$keyfile
cryptsetup luksFormat --type luks2 /dev/md/3 $keyfile
cryptsetup luksFormat --type luks2 /dev/md/4 $keyfile
cryptsetup luksOpen --key-file=$keyfile /dev/md/3 luks-md3
cryptsetup luksOpen --key-file=$keyfile /dev/md/4 luks-md4
pvcreate /dev/mapper/luks-md3
pvcreate /dev/mapper/luks-md4Update /etc/crypttab with keyfile:
lsblk -o +UUID
set
echo "luks-md3 UUID=<uuid> $keyfile" >> /etc/crypttab
echo "luks-md4 UUID=<uuid> $keyfile" >> /etc/crypttabCreate the volume group and logical volumes and mount
vgname=nfs
lvname=jupyterhub
mount=/srv/$lvname
vgcreate $vgname /dev/mapper/luks-md3 /dev/mapper/luks-md4
lvcreate -L 4TB $vgname -n $lvname
mkfs -t xfs -n ftype=1 /dev/mapper/$vgname-$lvname
echo "/dev/mapper/$vgname-$lvname $mount xfs defaults 0 0" >> /etc/fstab
mkdir -m 700 -p $mount
mount $mountFigure out which devices, but physical slot (lssci) and the new mdadm device name:
lsblk -o +UUID
lsscsi
mdadm --detail --scanCreate the mirror with mdadm device name (e.g. 6) and devices (e.g. sdj and sdk):
mdadm --create --verbose --metadata 1.2 /dev/md/6 --level=1 --raid-devices=2 /dev/sdj /dev/sdkFind the UUID of the new device and add to mdadm.conf:
mdadm --detail --scan
emacs /etc/mdadm.confCreate the luks device:
keyfile=/etc/luks/bkp
cryptsetup luksOpen --key-file=/etc/luks/bkp /dev/md/6 luks-md6
cryptsetup luksFormat --type luks2 /dev/md/6 /etc/luks/bkpAdd UUID and device to /etc/crypttab:
lsblk -o +UUID|grep md6.*raid
emacs /etc/crypttabCreate pv and add to vg:
pvcreate /dev/mapper/luks-md6
vgextend bkp /dev/mapper/luks-md6You need to disable random mac addresses in /etc/NetworkManager/NetworkManager.conf:
[device]
wifi.can-rand-mac-address=no
After this, use nmtui to configure the device. You have to modify
/etc/wpa_supplicant/wpa_supplicant.conf with the network configuration:
network={
ssid="<SSID>"
key_mgmt=WPA-PSK
scan_ssid=1
psk=<long hex string>
}
The /etc/sysconfig/network-scripts/ifcfg-ssid looks something like:
ESSID=<ssid>
MODE=Managed
KEY_MGMT=WPA-PSK
MAC_ADDRESS_RANDOMIZATION=default
TYPE=Wireless
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=dhcp
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
NAME=<ssid>
UUID=bla-bl-bal
DEVICE=wlp2s0
ONBOOT=yes
USERS=root
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
You should use ctrl-alt-F1 in X11 to get to the tmux screen.
- ctrl-b 2 gets to the shell window
- /tmp/anaconda.log will help debug kickstart
An easy kickstart is to name a USB key OEMDRV and put a ks.cfg in the
root directory. This doesn't work on Mac OS, because it writes illegal
files (e.g. .Trashes) for FAT16. This can be fixed up in rescue mode.
It doesn't always work automatically so so just specify inst.ks on the on the boot line. Go to the boot menu, select "Install" (i) and then tab. Add:
inst.ks=hd:LABEL=OEMDRV
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/installation_guide/sect-kickstart-syntax
https://docs-old.fedoraproject.org/en-US/Fedora/23/html/Installation_Guide/sect-kickstart-examples.html
Here's a sample ks.cfg:
auth --enableshadow --passalgo=sha512
zerombr
text
firewall --disabled
firstboot --disable
selinux --disabled
keyboard --vckeymap=us --xlayouts='us'
lang en_US.UTF-8
network --bootproto=dhcp --device=enp3s0 -ipv6=ignore --activate
#network --bootproto=static --device=em2 --gateway=10.10.10.1 --ip=10.10.10.10 --nameserver=8.8.8.8 --netmask=255.255.255.0 --ipv6=ignore --activate
network --hostname=localhost.localdomain
rootpw --iscrypted $1$truIB6el$usCWVY0UajwNoNh1VpwHE.
services --enabled="chronyd"
timezone Etc/UTC --isUtc
bootloader --append=" crashkernel=auto" --location=mbr --boot-drive=sda
clearpart --all --initlabel --drives=sda
# Needed if UEFI enabled https://access.redhat.com/solutions/1369253
part /boot/efi --fstype="efi" --ondisk=sda --size=200
part /boot --fstype="xfs" --ondisk=sda --size=1024
part pv.1 --fstype="lvmpv" --ondisk=sda --size=1024 --grow --asprimary
volgroup centos --pesize=4096 pv.1
logvol / --fstype="xfs" --size=5121 --name=root --vgname=centos
logvol swap --fstype="swap" --size=4096 --name=swap --vgname=centos
logvol /home --fstype="xfs" --size=5120 --name=home --vgname=centos
skipx
reboot
%packages
@^minimal
@core
chrony
kexec-tools
%end
To debug, you really need to specify the config file on the boot line. Go to the boot menu, select "Install" (i) and then tab. Add:
inst.ks=hd:LABEL=OEMDRV
This will produce better error messages. The implicit OEMDRV silently fails and
goes to graphical install.
Some references:
- http://www.golinuxhub.com/p/kickstart-configuration-for-rhel6.html
- https://www.alteeve.com/files/an-cluster/ks/generic_server_rhel6.ks
Use ipmiutil:
yum install -y ipmiutilconnect to the BMC:
ipmiutil config -l -E -N $ip -U root
ipmiutil sol -E -N $ip -U root -a -V 4Set your terminal window to 80x25.
Make sure you are setup for SOL. On Dell 2950:
Remote Access >> Configuration >> Serial
- Direct Console Terminal Mode
- Baud Rate 115
- Channel Privilege Level Limit Adminstrator
Remote Access >> Configuration >> Serial over LAN Configuration
- Enable Serial Over LAN [x]
- Baud Rate 115.2kbps
- Channel Privilege Level Limit Administrator
VLAN tagging is useful segregate the same physical network, and necessary with a blade chassis like m1000e. You have to load 8021q, and setup the interface (e.g. for enp3s0 on VLAN 13):
modprobe 8021q
# permanent
echo 8021q > /etc/modules-load.d/rsconf-8021q.conf
cd /etc/sysconfig/network-scripts
cat > ifcfg-enp3s0.13 <<'EOF'
VLAN=yes
DEVICE=enp3s0.13
IPADDR=192.168.13.27
GATEWAY=""
NETMASK=255.255.255.0
BOOTPROTO=static
DEVICE=enp3s0.13
ONBOOT=yes
DEFROUTE=no
IPV6INIT="no"
EOF
ifup enp3s0.13Setting up a dhcpd:
yum install -y dhcp
cat > /etc/dhcp/dhcpd.conf <<'EOF'
option domain-name "bivio.biz";
option domain-name-servers 8.8.8.8;
default-lease-time 600;
max-lease-time 7200;
subnet 192.168.13.0 netmask 255.255.255.0 {
range 192.168.13.100 192.168.13.199;
}
EOF
systemctl start dhcpdSomething to consider is augeas.net. Here's an example:
augtool -s <<EOF
#root login needs to be enabled during initial setup so the project specific scripts can be executed
set /files/etc/ssh/sshd_config/PermitRootLogin yes
#This saves time during vm startup
set /files/etc/grub.conf/timeout 0
#Removed because otherwise user install scripts can't use sudo
rm /files/etc/sudoers/Defaults[requiretty]
EOFpvcreate /dev/mapper/luks-md6
vgextend centos /dev/mapper/luks-md6
lvextend -l 100%FREE /dev/centos/home
xfs_growfs /homeWhat ports are listening:
ss -nlput
Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port
udp UNCONN 0 0 127.0.0.1:323 *:* users:(("chronyd",pid=1290,fd=1))
udp UNCONN 0 0 ::1:323 :::* users:(("chronyd",pid=1290,fd=2))
tcp LISTEN 0 128 *:22 *:* users:(("sshd",pid=1730,fd=3))
tcp LISTEN 0 100 127.0.0.1:25 *:* users:(("master",pid=1939,fd=13))
tcp LISTEN 0 128 :::22 :::* users:(("sshd",pid=1730,fd=4))
tcp LISTEN 0 100 ::1:25 :::* users:(("master",pid=1939,fd=14))To generate an ssh key:
ssh-keygen -t ed25519Interesting details on ssh config:
https://security.stackexchange.com/questions/143442/what-are-ssh-keygen-best-practices https://stribika.github.io/2015/01/04/secure-secure-shell.html
https://alteeve.com/w/Setting_Up_a_PXE_Server_on_an_RPM-based_OS
http://stackblog.us/2016/04/18/how-to-remotely-install-centosrhel-7-with-an-idrac-express/
Installing with software raid. Can only really do with kickstart:
https://www.alteeve.com/w/Install_CentOS7_with_Software_RAID_1
It won't mount partitions so you need to madam:
mdadm --examine --scan >> /etc/mdadm.conf
mdadm --assemble --scan
# might need: pvscan; vgscan; vgchange -ay
mount /dev/mapper/centos-root /mnt/sysimageI've decided that not only is swap not useful, but it's a security issue. If a machine crashes, whatever is in swap is available unless it is encrypted, which is why people recommend encrypted swap.
Do we really need swap in modern systems on redhat.org says that we do. The article assumes you don't know what you are doing. If you really care, you'll get an email when there is "memory pressure" on the system, e.g. when 80% of RAM gets consumed, not when it swaps.
My experience with swap is that when the system has to swap, it's usually due to too many processes (due to some bug), and you can't get into the system any way.
For diskless machines, swap makes no sense, of course. The logic extends to diskful systems: just because you have a disk, doesn't mean you should use it to extend RAM.
Modern memory is 1500x faster (or more) than (fast) SSD and 100,000x faster than magnetic media. For all intents and purposes, a system cannot function with these scale factors. Our need to comply with NIST 800-171 makes swap even slower.
So, for now, I am deciding not to swap.
SYSV init scripts show up as "bad"
in systemctl status:
# systemctl status network -l
network.service - LSB: Bring up/down networking
Loaded: loaded (/etc/rc.d/init.d/network; bad; vendor preset: disabled)
Active: active (exited) since Mon 2018-01-08 00:19:46 UTC; 2 weeks 6 days ago
Docs: man:systemd-sysv-generator(8)
Memory: 0B"bad" is fine.
The boot command is rather noisy, but you have to scroll to the bottom to add the single
keyword and then type control-X to initiate the boot:
