Skip to content

Commit

Permalink
Initial public code release
Browse files Browse the repository at this point in the history
  • Loading branch information
aiooss-anssi committed Nov 9, 2023
0 parents commit 39ae83b
Show file tree
Hide file tree
Showing 38 changed files with 11,084 additions and 0 deletions.
Binary file added .github/demo.webp
Binary file not shown.
25 changes: 25 additions & 0 deletions .github/workflows/linter.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
name: Lint code

on: [ push, pull_request ]

jobs:
python-lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: chartboost/ruff-action@v1
with:
src: "./webapp"
- uses: psf/black@stable
with:
src: "./webapp"

javascript-lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/setup-node@v1
with:
node-version: '*'
- run: npm install standard
- run: npx standard ./webapp/static/assets
12 changes: 12 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
input_pcaps/*
suricata/output/filestore
suricata/output/tcpstore
suricata/output/udpstore
suricata/output/*.json
suricata/output/*.log
webapp/database/*
.env

# Python
__pycache__
.ruff_cache
674 changes: 674 additions & 0 deletions COPYING

Large diffs are not rendered by default.

100 changes: 100 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
# Shovel

Shovel is a web application that offers a graphical user interface to explore
[Suricata EVE outputs](https://docs.suricata.io/en/suricata-7.0.1/output/eve/eve-json-output.html).
Its primary focus is to help [Capture-the-Flag players](https://ctftime.org/ctf-wtf)
analyse network traffic dumps during stressful and time-limited attack-defense games such as
[FAUSTCTF](https://faustctf.net/) or [ECSC](https://ecsc.eu/).
Shovel is developed in the context of
[ECSC Team France](https://ctftime.org/team/159269/) training.

![Shovel during ENOWARS7](./.github/demo.webp)

You might also want to have a look at these other awesome traffic analyser tools:

- https://github.com/secgroup/flower (first commit in 2018)
- https://github.com/eciavatta/caronte (first commit in 2020)
- https://github.com/OpenAttackDefenseTools/tulip (fork from flower in May 2022)

Compared to these traffic analyser tools, Shovel relies on Suricata while making
some opinionated choices for the frontend. This has a few nice implications:

- dissection of all application protocols already supported by Suricata (TCP and UDP),
- use a single SQLite database,
- on disk TCP/UDP/HTTP payload deduplication,
- filters based on libmagic, e.g. quickly filter flows containing PDF documents or PNG images,
- no heavy build tools needed, Shovel is easy to tweak.

Moreover, Shovel is batteries-included with Grafana visualizations and some Suricata alert rules.

## Setup

### 0. Before the Capture-the-Flag event begins

Copy `example.env` to `.env` and tweak the configuration parameters.
Also add the flag format in `suricata/custom.rules` if needed.

If you are playing a CTF using an IPv6 network, you might want to [enable IPv6 support in Docker deamon](https://docs.docker.com/config/daemon/ipv6/) before the CTF starts.

### 1. Network capture setup

You should place network captures in `input_pcaps/` folder.
Capture files should be splitted into chunks to be progressively imported.
If the CTF event does not already provide PCAP files, then you can adapt one
of the following commands for a GNU/Linux system:
```bash
ssh [email protected] tcpdump -i wg-faustctf -n -w - 'tcp port not 22' | tcpdump -n -r - -G 30 -w input_pcaps/trace-%Y-%m-%d_%H-%M-%S.pcap
```
For a Microsoft Windows system, you may adapt the following command (3389 is RDP):
```powershell
.\tshark.exe -b duration:60 -w \\share\captures\trace -f "tcp port not 3389"
```

### 2. Launch Suricata and webapp via Docker (option A)

Start Suricata, the web application and Grafana using `docker compose up -d --build`.

Please note that restarting Suricata will cause all network capture files to be loaded again from zero.

### 2. Launch Suricata and webapp traditionally (option B)

You may launch Suricata then the web application using the following:
```bash
# Start Suricata
export $(grep -vE "^(#.*|\s*)$" .env)
./suricata/entrypoint.sh
```

```bash
# Start web app
export $(grep -vE "^(#.*|\s*)$" .env)
(cd webapp && uvicorn --host 0.0.0.0 main:app)
```

Please note that restarting Suricata will cause all network capture files to be loaded again from zero.

## Frequently Asked Questions

### Is Suricata `flow_id` really unique?

`flow_id` is derived from timestamp (ms scale) and current flow parameters (such
as source and destination ports and addresses). See source code:
<https://github.com/OISF/suricata/blob/suricata-6.0.13/src/flow.h#L680>.

### How do I reload rules without rebuilding the database?

You can edit suricata rules in `suricata/custom.rules`, then reload the rules
using:
```bash
kill -USR2 $(pidof suricata)
```

## Licensing

Copyright (C) 2023 ANSSI

Shovel is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3.

Shovel is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with Shovel. If not, see <https://www.gnu.org/licenses/>.
32 changes: 32 additions & 0 deletions docker-compose.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
version: "3"

services:
suricata:
build: ./suricata
image: shovel-suricata
volumes:
- "./input_pcaps:/input_pcaps:ro"
- "./suricata/custom.rules:/suricata/custom.rules:ro"
- "./suricata/output:/suricata/output:rw"
env_file:
- .env

webapp:
build: ./webapp
image: shovel-webapp
volumes:
- "./input_pcaps:/webapp/static/input_pcaps:ro"
- "./suricata/output:/suricata/output:ro"
- "./webapp/database:/webapp/database:rw"
ports:
- 8000:8000
env_file:
- .env

grafana:
build: ./grafana
image: shovel-grafana
volumes:
- "./webapp/database:/webapp/database:ro"
ports:
- 3000:3000
62 changes: 62 additions & 0 deletions example.env
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
# Examples from FAUSTCTF (2023-09-23)
CTF_START_DATE=2023-09-23T15:00+02:00
CTF_TICK_LENGTH=180
CTF_HOME_NET=[fd66:666::0/32]
CTF_SERVICES=chatapp,image_galoisry,jokes,rsamail,buerographie_app,tic_tac_toe,office_supplies,auction_service
CTF_SERVICE_AUCTION_SERVICE=[fd66:666:798::2]:12345,[fd66:666:798::2]:12346
CTF_SERVICE_BUEROGRAPHIE_APP=[fd66:666:798::2]:13731
CTF_SERVICE_CHATAPP=[fd66:666:798::2]:3000
CTF_SERVICE_IMAGE_GALOISRY=[fd66:666:798::2]:5005
CTF_SERVICE_JOKES=[fd66:666:798::2]:5000
CTF_SERVICE_OFFICE_SUPPLIES=[fd66:666:798::2]:1337
CTF_SERVICE_RSAMAIL=[fd66:666:798::2]:5555
CTF_SERVICE_TIC_TAC_TOE=[fd66:666:798::2]:3333

# Examples from ENOWARS7 (2023-07-22)
#CTF_START_DATE=2023-07-22T15:00+02:00
#CTF_TICK_LENGTH=60
#CTF_HOME_NET=[10.1.42.1/25]
#CTF_SERVICES=asocialnetwork,bollwerk,granulizer,oldschool,phreaking,steinsgate,yvm
#CTF_SERVICE_ASOCIALNETWORK=10.1.42.1:3000
#CTF_SERVICE_BOLLWERK=10.1.42.1:6009
#CTF_SERVICE_GRANULIZER=10.1.42.1:2345
#CTF_SERVICE_OLDSCHOOL=10.1.42.1:9080
#CTF_SERVICE_PHREAKING=10.1.42.1:3399,10.1.42.1:6060,10.1.42.1:9930,10.1.42.1:6061,10.1.42.1:9931,10.1.42.1:6062,10.1.42.1:9932,10.1.42.1:6063,10.1.42.1:9933,10.1.42.1:6064,10.1.42.1:9934,10.1.42.1:6065,10.1.42.1:9935,10.1.42.1:6066,10.1.42.1:9936,10.1.42.1:6067,10.1.42.1:9937,10.1.42.1:6068,10.1.42.1:9938,10.1.42.1:6069,10.1.42.1:9939
#CTF_SERVICE_STEINSGATE=10.1.42.1:4433,10.1.42.1:4420
#CTF_SERVICE_YVM=10.1.42.1:3165

# Examples from ICC 2023 training (2023-07-09)
#CTF_START_DATE=2023-07-09T11:00+02:00
#CTF_TICK_LENGTH=120
#CTF_HOME_NET=[10.20.9.0/24]
#CTF_SERVICES=flag_prescription,navashield,win_dc1,win_srv1
#CTF_SERVICE_FLAG_PRESCRIPTION=10.20.9.6:5001
#CTF_SERVICE_NAVASHIELD=10.20.9.6:8001,10.20.9.6:5000
#CTF_SERVICE_WIN_DC1=10.20.9.4:80,10.20.9.4:445,10.20.9.4:135
#CTF_SERVICE_WIN_SRV1=10.20.9.5:80,10.20.9.5:31337,10.20.9.5:135

# Examples from ECSC 2022 (2022-09-15)
#CTF_START_DATE=2023-01-30T13:00+02:00
#CTF_TICK_LENGTH=120
#CTF_HOME_NET=[10.20.9.0/24]
#CTF_SERVICES=dewaste,cantina,hps,aquaeductus,blinkygram,winds_of_the_past,techbay
#CTF_SERVICE_DEWASTE=10.10.10.1:10010
#CTF_SERVICE_CANTINA=10.10.10.1:10020,10.10.10.1:10021,10.10.10.1:10024
#CTF_SERVICE_HPS=10.10.10.1:10030
#CTF_SERVICE_AQUAEDUCTUS=10.10.10.1:10041
#CTF_SERVICE_BLINKYGRAM=10.10.10.1:10050
#CTF_SERVICE_WINDS_OF_THE_PAST=10.10.10.1:10060
#CTF_SERVICE_TECHBAY=10.10.10.1:10070

# Examples from FAUST CTF 2022 (2022-07-09)
#CTF_START_DATE=2023-01-30T13:00+02:00
#CTF_TICK_LENGTH=180
#CTF_HOME_NET=[fd66:666::0/32]
#CTF_SERVICES=compiler60,docsnotebook,digital_seconds_ago,fittyfit,fluxmail,notes_from_the_future,admincrashboard
#CTF_SERVICE_COMPILER60=[fd66:666:534::2]:6061,[fd66:666:534::2]:6062
#CTF_SERVICE_DOCSNOTEBOOK=[fd66:666:534::2]:9000
#CTF_SERVICE_DIGITAL_SECONDS_AGO=[fd66:666:534::2]:13731
#CTF_SERVICE_FITTYFIT=[fd66:666:534::2]:5001
#CTF_SERVICE_FLUXMAIL=[fd66:666:534::2]:4242
#CTF_SERVICE_NOTES_FROM_THE_FUTURE=[fd66:666:534::2]:1338
#CTF_SERVICE_ADMINCRASHBOARD=[fd66:666:534::2]:5000,[fd66:666:534::2]:5002
10 changes: 10 additions & 0 deletions grafana/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
FROM grafana/grafana-oss:10.1.2
ENV GF_ANALYTICS_CHECK_FOR_PLUGIN_UPDATES=false
ENV GF_ANALYTICS_CHECK_FOR_UPDATES=false
ENV GF_ANALYTICS_REPORTING_ENABLED=false
ENV GF_AUTH_ANONYMOUS_ENABLED=true
ENV GF_AUTH_ANONYMOUS_HIDE_VERSION=true
ENV GF_INSTALL_PLUGINS=frser-sqlite-datasource
ENV GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_PATH=/var/lib/grafana/dashboards/home.json
COPY ./provisioning /etc/grafana/provisioning
COPY ./dashboards /var/lib/grafana/dashboards
Loading

0 comments on commit 39ae83b

Please sign in to comment.