Shovel

Shovel is a web application that offers a graphical user interface to explore Suricata EVE outputs. Its primary focus is to help Capture-the-Flag players analyse network flows during stressful and time-limited attack-defense games such as FAUSTCTF, ENOWARS or ECSC. Shovel is developed in the context of ECSC Team France training.

You might also want to have a look at these other awesome traffic analyser tools:

• https://github.com/secgroup/flower (first commit in 2018)
• https://github.com/eciavatta/caronte (first commit in 2020)
• https://github.com/OpenAttackDefenseTools/tulip (fork from flower in May 2022)

Compared to these traffic analyser tools, Shovel only relies on Suricata while making opinionated choices for the frontend. This has a few nice implications:

• dissection of all application protocols supported by Suricata (HTTP2, modbus, SMB, DNS, etc),
• flows payloads and dissections are stored inside SQLite databases for fast queries,
• ingest can be a folder of pcaps for non-root CTF, or a live capture (less delay),
• tags are defined using Suricata rules (regex, libmagic match, HTTP header, etc),
• no heavy build tools needed, Shovel is easy to tweak.

Moreover, Shovel is batteries-included with some Suricata alert rules.

        ┌────────────────────────┐
device  │ Suricata with:         │   eve.db    ┌───────────────┐
or pcap │  - Eve SQLite plugin   ├────────────►│               │
───────►│  - TCP payloads plugin │ payload.db  │ Python webapp │
        │  - UDP payloads plugin ├────────────►│               │
        └────────────────────────┘             └────▲──────────┘
                                              .env  │
                                              ──────┘

Getting started

Services mapping, ticks and flag format configuration

Shovel is configured using environment variables. Copy example.env to .env and update the optional configuration parameters. You may update this file later and restart only the webapp.

Add the flag format in suricata/rules/suricata.rules if needed. If you modify this file after starting Suricata, you may reload rules using pkill -USR2 suricata.

Network capture

Suricata supports multiple capture methods. Please use a live capture with AF_PACKET when possible, or libpcap if you can't mirror the traffic (archives replay or rootless CTF).

pcap capture mode (slower)

Place pcap files in a folder such as input_pcaps/.

If you are continuously adding new pcap, add --pcap-file-continuous to Suricata command line.

Then you may start the compose using:

docker compose up -d

If you don't want to use Docker, you may manually launch Suricata and the web application using the two following commands:

./suricata/entrypoint.sh -r input_pcaps
(cd webapp && uvicorn --host 127.0.0.1 main:app)

Tip

If the CTF event does not already provide PCAP files, then you may adapt the following command for a GNU/Linux system (22 is SSH):

ssh [email protected] tcpdump -i game -n -w - 'tcp port not 22' | tcpdump -n -r - -G 30 -w input_pcaps/trace-%Y-%m-%d_%H-%M-%S.pcap

For a Microsoft Windows system, you may run the following command (3389 is RDP) inside a PowerShell console:

&'C:\Program Files\Wireshark\tshark.exe' -i game -w Z:\ -f "tcp port not 3389" -b duration:60

Warning

Please note that restarting Suricata will cause all network capture files to be loaded again. It might add some delay before observing new flows.

Live capture mode (faster)

This mode requires access to a network device with the game traffic. Here this device is named tun5.

Edit docker-compose.yml and comment option A and uncomment option B under suricata container definitions. Then, you may start the compose using:

docker compose up -d

If you don't want to use Docker, you may manually launch Suricata and the web application using the two following commands:

sudo ./suricata/entrypoint.sh -i tun5
(cd webapp && uvicorn --host 127.0.0.1 main:app)

Warning

Please note that stopping Suricata will stop network capture.

You may run sudo tcpdump -n -i tun5 -G 30 -w trace-%Y-%m-%d_%H-%M-%S.pcap for archiving purposes.

Frequently Asked Questions

Is Suricata flow_id really unique?

flow_id is derived from timestamp (ms scale) and current flow parameters (such as source and destination ports and addresses). See source code: https://github.com/OISF/suricata/blob/suricata-6.0.13/src/flow.h#L680.

How do I reload rules without restarting Suricata?

You can edit suricata rules in suricata/rules/suricata.rules, then reload the rules using:

pkill -USR2 suricata