Skip to content

Time Card installation and usage

Jump to bottom
Ahmad Byagowi edited this page Jun 21, 2024 · 71 revisions

System setup

To use the time card, you'll a system with a PCIe slot

I typically use Centos for my setup, so my instructions will be for Centos.

BIOS Setting

  1. Enable the I/O virtualization of your CPU on the BIOS:
  • For Intel CPUs: Enable Vt-D or Vt-x
  • For AMD CPUs: Enable IOMMU

In addition, use the following commands to get certain info about your system:

Use the following command to get info about your motherboard:

sudo dmidecode -t baseboard

Kernel installation

In order to build the latest ocp_tap driver and use all the features on the Time Card, I typically build the kernel from scratch. If you want to avoid this step (patching the kernel), you can use Ubuntu 20.04.3 LTS or 22.04.3 LTS or anything newer by Canonical and jump directly to step 11.

  1. Make sure system clock is current. To update your system clock from the internet use the following command:
sudo date -s "$(wget -qSO- --max-redirect=0 google.com 2>&1 | grep Date: | cut -d' ' -f5-8)Z" ; hwclock -w
  1. Copy latest github
cd ~
git clone https://github.com/opencomputeproject/Time-Appliance-Project
  1. Install the latest kernel keys and reboot for CentOS
yum -y install https://www.elrepo.org/elrepo-release-8.el8.elrepo.noarch.rpm
yum config-manager --set-enabled powertools
yum install epel-release
dnf --enablerepo=elrepo-kernel install kernel-ml
  1. Install prereqs for CentOS
yum install -y ncurses-devel make gcc bc bison flex elfutils-libelf-devel openssl-devel grub2 i2c-tools patch git dwarves

Install prereqs for Ubuntu

sudo apt-get install libncurses-dev flex bison openssl vim libssl-dev dkms libelf-dev libudev-dev libpci-dev libiberty-dev autoconf zstd
  1. Get full kernel source from online.

for CentOS

cd /usr/src/kernels
wget https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.19.16.tar.xz
tar -xvf linux-5.19.16.tar.xz
rm linux-5.19.16.tar.xz
mv linux-5.19.16/ 5.19.16/
cd 5.19.16/

for Ubuntu

cd /usr/src/
sudo wget https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.19.16.tar.xz
sudo tar -xvf linux-5.19.16.tar.xz
sudo rm linux-5.19.16.tar.xz
cd linux-5.19.16/
  1. Copy existing kernel config to kernel source, edit the /boot/ file to whatever is the latest
cp -v /boot/config-5.19.16-1.el8.elrepo.x86_64 .config
vim .config

if you are adding TGPIO support, make sure you patch the source as well

sudo patch -p1 < ~/Time-Aware-GPIO-Support-main/linux-kernel-patch/tgpio-kernel-5_19_16-static.patch

then run make prepare and press enter on all the selections

sudo make prepare
  1. Change these kernel build parameters in .config
CONFIG_I2C_XILINX=m
CONFIG_MTD=y
CONFIG_MTD_SPI_NOR=m
CONFIG_SPI=y
CONFIG_SPI_ALTERA=m
CONFIG_SPI_BITBANG=m
CONFIG_SPI_MASTER=y
CONFIG_SPI_MEM=y
CONFIG_SPI_XILINX=m
CONFIG_I2C=y
CONFIG_I2C_OCORES=m
CONFIG_IKCONFIG=y
CONFIG_EEPROM_AT24=m
CONFIG_SYSTEM_TRUSTED_KEYS=""

if you are enabling TGPIO and PTM, select the following in addition

CONFIG_PTP_INTEL_PMC_TGPIO=y
CONFIG_PCIE_PTM=y

additionally for for Ubuntu

CONFIG_SYSTEM_TRUSTED_KEYS=""
CONFIG_SYSTEM_REVOCATION_KEYS=""
CONFIG_DEBUG_INFO_BTF=n

run make prepare again and select "y" for CONFIG_IKCONFIG_PROC and select default (just press Enter) for any other question

  1. Build and install this kernel
make -j5 bzImage modules;  make INSTALL_MOD_STRIP=1 modules_install; make install;
  1. Update grubby for the new kernel, make sure version matches here.

for CentOS

grub2-mkconfig -o /boot/grub2/grub.cfg
grubby --set-default /boot/vmlinuz-5.19.16
grubby --update-kernel=ALL --args=8250.nr_uarts=8

for Ubuntu Modify the GRUB_DEFAULT in /etc/default/grub to the following

GRUB_DEFAULT="Advanced options for Ubuntu>Ubuntu, with Linux 5.19.16"

then run

sudo update-grub
  1. Reboot
reboot
  1. Now kernel is up to date. Make sure this is actually picked up by the kernel next time you boot
cat /proc/cmdline
[root@localhost Binaries]# cat /proc/cmdline
BOOT_IMAGE=(hd1,gpt2)/vmlinuz-5.19.19 root=/dev/mapper/cs-root ro crashkernel=auto resume=/dev/mapper/cs-swap rd.lvm.lv=cs/root rd.lvm.lv=cs/swap rhgb quiet 8250.nr_uarts=8
  1. Copy latest github for most up to date driver and build and install it
cd ~/Time-Appliance-Project/Time-Card/DRV/
./remake
modprobe ptp_ocp

In addition, you can add ptp_ocp as well as i2c_xiic to the /etc/modules to make them automatically load on boot

Now you can check dmesg to see if things are in a good shape. This is what you should expect:

[ 1491.440223] 0000:02:00.0: ttyS5 at MMIO 0x70161000 (irq = 180, base_baud = 3125000) is a 16550A
[ 1491.440278] 0000:02:00.0: ttyS6 at MMIO 0x70171000 (irq = 181, base_baud = 3125000) is a 16550A
[ 1491.440322] 0000:02:00.0: ttyS7 at MMIO 0x70181000 (irq = 182, base_baud = 3125000) is a 16550A
[ 1491.440362] 0000:02:00.0: ttyS0 at MMIO 0x70191000 (irq = 187, base_baud = 3125000) is a 16550A
[ 1491.441702] xilinx_spi xilinx_spi.1024: at [mem 0x70310000-0x7031ffff], irq=186
[ 1491.442678] spi-nor spi1024.0: n25q128a13 (16384 Kbytes)
[ 1491.447310] pps pps1: new PPS source ptp2
[ 1491.449374] ptp_ocp 0000:02:00.0: Version 1.2.0, clock PPS, device ptp2
[ 1491.449388] ptp_ocp 0000:02:00.0: Time: 1647548640.775286647, in-sync
[ 1491.449390] ptp_ocp 0000:02:00.0: version 8005
[ 1491.449392] ptp_ocp 0000:02:00.0: regular image, version 32773
[ 1491.449393] ptp_ocp 0000:02:00.0:  GNSS: /dev/ttyS5  @ 115200
[ 1491.449394] ptp_ocp 0000:02:00.0: GNSS2: /dev/ttyS6  @ 115200
[ 1491.449396] ptp_ocp 0000:02:00.0:   MAC: /dev/ttyS7  @  57600
[ 1491.449399] ptp_ocp 0000:02:00.0:  NMEA: /dev/ttyS0  @ 115200

Kernel installation (Raspberry Pi Compute Module) (WIP)

https://www.raspberrypi.com/documentation/computers/linux_kernel.html

CONFIG_NET_DEVLINK=y CONFIG_SERIAL_8250_NR_UARTS=8 CONFIG_IRQ_POLL=y

changed Kconfig in net to have DEVLINK enabled

make oldconfig before doing build but after manually adding DEVLINK

Time Card Usage

Time Card is interfaced with via the sysfs. Newer cards can have any SMA be an input or output.

cd /sys/class/timecard/ocp0/

Serial ports and PHC number can be found via dmesg or ls -l

Outputs

All possible outputs are available via available_sma_outputs

cat available_sma_outputs
10Mhz PHC MAC GNSS1 GNSS2 IRIG DCF GEN1 GEN2 GEN3 GEN4 GND VCC
  • Output FPGA PPS on SMA
echo OUT: PHC >> sma1
  • Output Atomic clock PPS on SMA
echo OUT: MAC >> sma1
  • Output GPS Module's PPS on SMA
echo OUT: GNSS1 >> sma1

Inputs

All possible inputs are available via available_sma_inputs

cat available_sma_inputs
10Mhz PPS1 PPS2 TS1 TS2 IRIG DCF TS3 TS4 FREQ1 FREQ2 FREQ3 FREQ4 None
  • Use an SMA for Timestamping incoming signals
echo IN: TS1 >> sma1
  • To read back Timestamps, use testptp.
    • Source available at these locations
https://www.mjmwired.net/kernel/Documentation/ptp/testptp.c
https://github.com/torvalds/linux/blob/master/tools/testing/selftests/ptp/testptp.c

https://unix.stackexchange.com/questions/725969/does-testptp-c-compile-for-anyone-else -> seems to be an issue soon

wget https://raw.githubusercontent.com/torvalds/linux/master/tools/testing/selftests/ptp/testptp.c
cp /usr/src/kernels/5.17.4/include/uapi/linux/ptp_clock.h /usr/include/linux/ptp_clock.h
gcc testptp.c -lrt -o testptp
  • Read infinite timestamps (-1 , use a positive number for fixed event count) from Timestamper 1 (-i 1) from the Time Card (/dev/ptp1)
./testptp -d /dev/ptp1 -e -1 -i 1

Check GPS is working

  • Use gpsd toolset (recommended)
stty -F /dev/ttyS5 speed 115200
gpsd /dev/ttyS5
cgps
  • Use tio to make sure the GPS is sending messages
tio -b 115200 /dev/ttyS5
  • Make sure the FPGA time is correct by checking NMEA
tio -b 115200 /dev/ttyS0

yum install python3
yum install python3-tkinter
python3 -m pip install --upgrade pip
python3 -m pip install --upgrade Pillow
python3 -m pip install pygpsclient
pygpsclient

Interact with SA5x

  • For a factory default SA5x you must run the following commands at default baud rate of 57600
\{set,Disciplining,1}
\{set,PpsWidth,80000000}
\{set,TauPps0,10000}
\{set,PpsOffset,-30}
\{set,DisciplineThresholdPps0,20}
\{store}
  • Use timetickler.py
cd ~/Time-Appliance-Project/Time-Card/MAC/SA53
python3 timetickler.py

Upgrading the card

cp TimeCard.bin /lib/firmware
devlink dev flash pci/0000:11:00.0 file TimeCard.bin

FPGA binaries with PTM Support

The FPGA binaries with PTM support require an updated Linux driver. The mainstream Linux kernel driver won't work.

For most Time Card versions FPGA images with PTM support are available:

Linux driver with PTM Support

For PTM support, the driver included in the mainstream Linux kernel will not work (host may not boot). Therefore, it makes sense to update the driver before updating the FPGA binaries.

If the system no longer boots, blacklisting ptp_ocp will help the system boot without the mainstream Linux kernel driver:

  1. Remove the Time Card and boot the system
  2. open /etc/modprobe.d/blacklist.conf
  3. add the entry blacklist ptp_ocp
  4. Install the Time Card again

With an updated and correctly loaded driver, PTM is supported:

  1. Get the latest version from git: https://github.com/opencomputeproject/Time-Appliance-Project/tree/master/Time-Card/DRV
  2. Change to the directory of the driver cd ~/Time-Appliance-Project/Time-Card/DRV/
  3. Compile the latest ptp_ocp driver with ./remake
  4. Load the driver with insmod ptp_ocp.ko

Testing PTM

To test PTM, the host must also support PTM.

Also make sure you have a newer version of linuxptp installed. (e.g. v4.3 https://github.com/richardcochran/linuxptp/releases/tag/v4.3).

Other time services should be stopped.

Following commands can be used:

sudo systemctl stop systemd-timesyncd.service
sudo phc_ctl /dev/ptp2 set
sudo phc2sys -c CLOCK_REALTIME -s /dev/ptp2 -O 0 -R 16 -u 8 -m

With correct working PTM the print out of phy2sys should show delay 0 +/-.

Clone this wiki locally