diff --git a/docs/index.html b/docs/index.html index 80e59ef..5d01a58 100644 --- a/docs/index.html +++ b/docs/index.html @@ -2,1170 +2,70 @@ - - - - - - Gont - A testing framework for distributed Go applications - - - - - - - - - - - - -
-
-
-
-

Gont

-

A testing framework for distributed Go applications

-
- -
-

Introduction

-
- -
-

What does Gont do?

-
    -
  • Software-defined virtual networking for testing
    • -
  • Define hosts, switches, routers in a single host
    • -
  • Reentrancy
    • -
  • Reproducibility
    • -
  • Inspired by Mininet
    • -
-
- -
-

Mininet

-
- - Mininet creates a realistic virtual network, running real kernel, switch and application code, on a single machine (VM, cloud or native) - -

-- mininet.org

-
-
- -
-

Mininet

-
    -
  • Written in Python 2
    • -
  • Lacking active maintainer
    • -
  • Focus on SDN: OpenFlow controllers
    • -
  • No SSL cert on homepage?!
    • -
-
→ We need something better
-
- -
-

Why?

-
    -
  • Describe network topologies quickly in code
    • -
  • Automate construction of complex topologies
    • -
  • Unit / CI testing -
  • Parallel test execution
    • -
  • Example use-cases -
      -
    • VPN / network tools development
      • -
    • SDN Openflow controller development
      • -
    • cunīcu: zeroconf • p2p • mesh • vpn agent (cunīcu)
      • -
    -
    • -
-
-
- -
-
-

A little detour:

-

Namespaces

-

(Feel free to skip this section if you are
only interested in learning about Gont)

-
- -
-

What are Linux namespaces?

-
- -
-

They..

-
    -
  • partition kernel resources
    from a process-perspective.
    • -
  • power most of Linux containerization tools.
    • -
  • appear in many Linux subsystems.
    • -
  • are used by many sandboxing solutions in browsers.
    • -
-
- -
-

Available Linux namespaces

-
    -
  • mnt: Mounts
    • -
  • pid: Process IDs
    • -
  • net: Networking
    • -
  • ipc: Interprocess Communication
    • -
  • uts: Unix Timesharing (hostname)
    • -
  • user: User identification & privileges
    • -
  • cgroup: Process Control Groups
    • -
  • time: System time
    • -
-
- -
-

Buts what exactly is a namespace?

-
    -
  • Can be identified by a file descriptor
    • -
  • A namespace can have multiple processes assigned to it
    • -
  • It lives as long as there is still at least one remaining process
    • -
  • Child processes inherit parent namespaces
    • -
-
- -
-

How do I create a namespace?

- -
- -
-

unshare(2)

- 

-							func main() {
-								err := syscall.Unshare(syscall.CLONE_NEWNS);
-							}
-
-
- -
-

How do I create a namespace?

- 

-							static int child(void *arg) {
-								struct utsname uts;
-
-								sethostname(arg, "ernie")
-
-								uname(&uts)
-								printf("nodename in child:  %s\n", uts.nodename);
-
-								return 0;
-							}
-
-							int main() {
-								struct utsname uts;
-
-								/* Allocate stack for child */
-								char *stack = malloc(STACK_SIZE);
-								if (stack == NULL)
-									return -1;
-
-								/* Start new kernel task in new UTS namespace */
-								pid_t child_pid = clone(child, stack + STACK_SIZE, CLONE_NEWUTS | SIGCHLD, NULL);
-
-								/* Output hostname */
-								uname(&uts)
-								printf("nodename in parent:  %s\n", uts.nodename);
-							}
-
-
- -
-

How can I share a namespace with other processes?

- -
- -
-

Joining namespace of another process by /proc/{pid}/ns/*

- 

-							fd := syscall.Open("/proc/1234/ns/uts", syscall.O_RDONLY);
-
-							err := unix.Setns(fd, syscall.CLONE_NEWUTS);
-
- Note: Can only set a single namespace per netns(2) invocation. -
- -
-

Joining namespace of another process by pidfd_open(2)

- 

-							pid_t pid = 1234;
-							int fd = pidfd_open(pid, 0);
-
-							setns(fd, CLONE_NEWUTS | CLONE_NEWNET);
-
- Note: Can only set a multiple namespaces per netns(2) invocation. -
- -
-

Persisting namespaces

- 

-							err := syscall.Mount("/proc/self/ns/uts",
-							                     "/home/acs/my_uts_namespace", "", syscall.MS_BIND, nil);
-
-
In other process
- 

-							fd := syscall.Open("/home/acs/my_uts_namespace", syscall.O_RDONLY);
-
-							err := unix.Setns(fd, syscall.CLONE_NEWUTS);
-
-
-
- -
-
-

Back to Gont...

-

First some definitions

-
- -
-
-
network
-
A set of nodes and links between them
-
-
- -
-
-
node
-
A network namespace which represents any device in the network
-
-
- -
-
-
link
-
A pair of veth network interfaces which are associated to separate nodes
-
-
- -
-
-
interface
-
an endpoint with one or more assigned IP addresses
-
-
- -
-
-
switch
-
A node containing a Linux layer 2 bridge and attached interfaces
-
-
- -
-
-
host
-
A node with several configured interfaces
-
-
- -
-
-
router
-
A host with IP forwarding enabled
-
-
- -
-
-
nat
-
A special type of router which implements network address translation between a set of south- and north-bound interfaces
-
-
- -
-

Relationships

- -
- -
-

Gont and Namespaces

-
    -
  • for each node: -
      -
    • a separate network namespace
      • -
    • a separate uts namespace -
        -
      • for a unique hostname within each node
        • -
      -
      • - -
    -
    • -
  • for each network: -
      -
    • a separate mount namespace
      • -
        -
      • for a unique /etc/hosts files between networks
        • -
      -
    -
    • -
-
-
- -
-
-

Examples

-

Show me some code!

-

- ⚠️ Warning: Please note that these examples are show
just for demonstration purposes and might not compile. -

-
- -
-

Detailed examples

-

- Have a look at the following code for full fledged test/example code: -

- -
-
- -
-
-

Basic networking

-
- -
-

Two directly connected hosts

- 

-							import g "github.com/stv0g/gont/v2/pkg"
-							import o "github.com/stv0g/gont/v2/pkg/options"
-
-							...
-
-							n, _ := g.NewNetwork("mynet")
-
-							h1, _ := n.AddHost("h1")
-							h2, _ := n.AddHost("h2")
-
-							n.AddLink(
-								g.NewInterface("eth0", h1, o.AddressIP("10.0.0.1/24")),
-								g.NewInterface("eth0", h2, o.AddressIP("10.0.0.2/24")),
-							)
-
-							h1.Ping(h2)
-
-

(We use g and o throughout these examples as import aliases)

-
- -
-

Lets add a L2 switch

- 

-							sw, _ := n.AddSwitch("sw")
-
-							h1, _ := n.AddHost("h1",
-								g.NewInterface("eth0", sw,
-									o.AddressIP("10.0.0.1/24")))
-
-							h2, _ := n.AddHost("h2",
-								g.NewInterface("eth0", sw,
-									o.AddressIP("10.0.0.2/24")))
-
-							h1.Ping(h2)
-
-
- -
-

How about a L3 router?

- 

-							sw1, _ := n.AddSwitch("sw1")
-							sw2, _ := n.AddSwitch("sw2")
-
-							h1, _ := n.AddHost("h1", g.NewInterface("eth0", sw1,
-							                    o.AddressIP("10.0.0.2/24")))
-							h2, _ := n.AddHost("h2", g.NewInterface("eth0", sw2,
-							                    o.AddressIP("10.0.1.2/24")))
-
-							n.AddRouter("r1",
-								g.NewInterface("eth0", sw, o.AddressIP("10.0.0.1/24")),
-								g.NewInterface("eth1", sw, o.AddressIP("10.0.1.1/24"))
-							)
-
-							h1.Ping(h2)
-
-
- -
-

Lets do some evil NATing 😈

- 

-							sw1, _ := n.AddSwitch("sw1")
-							sw2, _ := n.AddSwitch("sw2")
-
-							h1, _ := n.AddHost("h1", g.NewInterface("eth0", sw1,
-							                    o.AddressIP("10.0.0.2/24")))
-							h2, _ := n.AddHost("h2", g.NewInterface("eth0", sw2,
-							                    o.AddressIP("10.0.1.2/24")))
-
-							n.AddNAT("n1",
-								g.NewInterface("eth0", sw, o.SouthBound,
-								             o.AddressIP("10.0.0.1/24")),
-								g.NewInterface("eth1", sw, o.NorthBound,
-								             o.AddressIP("10.0.1.1/24")))
-
-							h1.Ping(h2)
-
-
- -
-

How about a whole chain of routers?

- 

-							var firstSwitch *g.Switch = n.AddSwitch("sw0")
-							var lastSwitch  *g.Switch = nil
-
-							for i := 1; i < 100; i++ {
-								swName := fmt.Printf("sw%d", i)
-								rtrName := fmt.Printf("r%d", i)
-
-								newSwitch, _ := n.AddSwitch(swName)
-
-								n.AddRouter(rtrName,
-									g.NewInterface("eth0", lastSwitch,
-									             o.AddressIP("10.0.0.1/24")),
-									g.NewInterface("eth1", newSwitch,
-									             o.AddressIP("10.0.1.1/24"))
-								)
-
-								lastSwitch = newSwitch
-							}
-
-							h1, _ := n.AddHost("h1", g.NewInterface("eth0", firstSwitch,
-							                    o.AddressIP("10.0.0.2/24")))
-							h2, _ := n.AddHost("h2", g.NewInterface("eth0", lastSwitch,
-							                    o.AddressIP("10.0.1.2/24")))
-
-							h1.Ping(h2)
-
-
-
- -
-
-

Running your application

-

Inside the network namespaces / hosts

-
- -
-

exec.Cmd API

- 

-							// Get a exec.Cmd-like struct
-							cmd := h1.Command("ping", "h2")
-							out, err := cmd.CombinedOutput()
-
-							// Directly run a simple process synchronously
-							cmd, err := h1.Run("ping", "h2")
-
-							// Directly start asynchronously
-							cmd, err := h1.Start("ping", "h2")
-
-							time.Sleep(5 * time.Second)
-
-							err = cmd.Process.Signal(os.Interrupt)
-							cmd.Wait()
-
-

The g.Node type implements an API similar to the one provided by Go's exec package.

-
- -
-

Pass options

- - 

-							import co "github.com/stv0g/gont/v2/pkg/options/cmd"
-
-							outb := &bytes.Buffer{}
-
-							cmd := h1.Command("ping", "1.1.1.1",
-								co.DisableASLR(true),
-								co.Dir("/custom/working/dir"),
-								co.EnvVar("DEBUG", "1"),
-								co.Stdin(...), // pass any io.Reader
-								co.Stdout(outb), // pass any io.Writer (can be repeated)
-								co.Stderr(...), // pass any io.Writer (can be repeated)
-							)
-
-							print(outb.String())
-
-
- -
-

Pass non-string arguments

- - 

-							ip := net.ParseIP("1.1.1.1")
-
-							cmd := h1.Command("ping", "-c", 10, "-i", 0.1, ip)
-
-
- -
-

Go functions

- - 

-							h1.RunFunc(func() {
-								r := http.Get("http://h2:8080")
-								io.Copy(os.Stdout, r.Body)
-							})
-
- -

- Call a function inside a network namespace - of host h1 but still
in the same process - so you can use channels and access global variables. -

- -

- ⚠️ Warning: Spawning Goroutines from within the callback
is only indirectly supported: -

- - 

-							h1.RunFunc(func() {
-								go h1.RunFunc(func() { ... })
-							})
-
-
- -
-

Go packages

- 

-							cmd, err := h1.RunGo("test/prog.go", "arg1")
-
-
-
- -
-
-

Firewall

-

by Netfilter's nftables

-
- -
-

Add some firewall rules for a host

- 

-							import fo "github.com/stv0g/gont/v2/options/filter"
-
-							_, src, _ := net.ParseCIDR("10.0.0.1/32")
-
-							h1, _ := n.AddHost("h1",
-												o.Filter(
-													g.FilterInput,
-														fo.Source(src),
-														fo.Protocol(unix.AF_INET),
-														fo.TransportProtocol(unix.IPPROTO_TCP),
-														fo.SourcePortRange(0, 1024)))
-
-
-
- -
-
-

Network Emulation

-

by Linux's Traffic Control: Netem Qdisc

-
- -
-

Attach a netem Qdisc to an interface

- 

-							import tco "github.com/stv0g/gont/v2/options/tc"
-
-							n.AddLink(
-								g.NewInterface("eth0", h1,
-									o.WithNetem(
-										tco.Latency(50 * time.Millisecond),
-										tco.Jitter(5 * time.Millisecond),
-										tco.Loss(0.1),
-									),
-									o.AddressIP("10.0.0.1/24")),
-								g.NewInterface("eth0", h2,
-									o.AddressIP("10.0.0.2/24")),
-							)
-
-							h1.Ping(h2)
-
-
-
- -
-
-

Packet captures

-

PCAP, WireShark, tshark

-
- -
-

Write to packets to

- -

Gont merges and sorts packet captures in real-time
from multiple interfaces and records them to one of the following sinks:

- -
    -
  • Using PCAPng format -
      -
    • Regular files
      • -
    • Named pipes
      • -
    • TCP / UDP / Unix listeners
      • -
    • WireShark real-time stream
      • -
    -
    • -
  • Go channels
    • -
  • Go callback functions.
    • -
-
- -
-

Filtering

-

Captured network traffic can be filtered by

-
    -
  • Selected Gont nodes and interfaces
    • -
  • eBPF filter programs
    • -
  • pcap-filter(7) expressions
    • -
  • Go callback functions (⚠ slow!)
    • -
-
- -
-

Session key logging

-

Most transport layer encryption protocols today provide perfect forward secrecy by using short-lived ephemeral session keys.

-

Gont offers a feature to log these session keys into a PCAPng file to enable a decryption of upper layer protocols with a dissector tool like Wireshark.

-
- -
-

Example

- -
-
- -
-
-

Distributed Tracing

-

User defined events, Log messages

-
- -
-

A trace event

- -

Gont supports collecting trace events from all processes running in a distributed system which can carry the following information. Gont orders trace events by time and saves them to different destinations for analysis.

- - 

-							type Event struct {
-								Timestamp time.Time	// Timestamp when the event occurred
-								Type      string    // Either: 'log', ́'trace', 'break' & ́ watchpoint'
-								Level     uint8     // Log level
-								Message   string    // A human readable description
-								Source    string    // Logger name
-								PID       int 
-								Function  string
-								File      string
-								Line      int
-								Args      []any
-								Data      any       // User defined data
-							}
-
-
- -
-

Write to trace events to

-
    -
  • JSON files
    • -
  • Go channels
    • -
  • Go callbacks
    • -
  • Packet captures
    • -
-
- -
-

Create a tracer

- - 

-							import "github.com/stv0g/gont/v2/trace"
-							import to "github.com/stv0g/gont/v2/options/trace"
-
-							c := g.NewCapture(...)
-							f, _ := os.OpenFile(...)
-							ch := make(chan trace.Event)
-
-							t := g.NewTracer(
-								to.ToFile(f)
-								to.ToFilename("trace.log"),
-								to.ToChannel(ch),
-								to.ToCapture(c),
-								to.Callback(func(e trace.Event) { ... }))
-
-							t.Start()
-
-
- -
-

Attach the tracer

- -

Trace all processes started by nodes of this network

- 

-							n, _ := g.NewNetwork("", t)
-
- -

Trace all processes started by a node

- 

-							h1 := n.NewHost("h1", t)
-
- -

Trace a single process

- 

-							h1.RunGo("test/main.go", t)
-
- -
- -
-

Trace with trace package

- 

-							import "github.com/stv0g/gont/v2/pkg/trace"
-
-							someData := map[string]string{"Hello": "World"}
-							count := 42
-
-							trace.Start(0)
-
-							trace.PrintfWithData(someData, "Count is: %d", count)
-							trace.Print("Another message")
-
-							trace.Stop()
-
- -

Works from:

-
    -
  • Gont process itself
    • -
  • Any process spawned via Gont's
    - Host.{Command,Run,RunGo,Start,StartGo}(...) functions
    • -
-
- -
-

Trace via slog structured logging package

- - 

-							import "golang.org/x/exp/slog"
-							import "github.com/stv0g/gont/v2/pkg/trace"
-
-							// Create a slog handler which emits trace events
-							handler := trace.NewTraceHandler(slog.HandlerOptions{})
-
-							// Add the tracing option which emits a trace event for each log message
-							logger := slog.New(handler)
-
- -

- Each log message emits a trace event which includes the log message, filename, line number as well function name and more.
- Any fields passed to to zap structured logger are included in the Data field of the Event structure. -

-
- -
-

Trace via zap logging package

- - 

-							import "go.uber.org/zap"
-							import "github.com/stv0g/gont/v2/pkg/trace"
-
-							// Add the tracing option which emits a trace event for each log message
-							logger := zap.NewDevelopment(trace.Log())
-
-							// Add the caller info which gets also included in the trace event
-							logger = logger.WithOptions(zap.AddCaller())
-
-							// Give the logger some name which is added as the Source field to the trace event
-							logger = logger.Named("my-test-logger")
-
- -

- Each log message emits a trace event which includes the log message, filename, line number as well function name and more.
- Any fields passed to to zap structured logger are included in the Data field of the Event structure. -

-
-
- -
-
-

Distributed Debugging

-

powered by Delve

-
- -
-

Introduction

-

- Gont can manage Delve debugger instances attached to spawned sub-processes. - These Delve instances can be used to debug or trace multiple applications in a distributed system simultaneously. - Users can either use DAP-compatible IDEs like VScode to attach to those processes, or record tracepoint data generated by Delve break- & watchpoints to a PCAPng file. -

-

- Its integration with the packet capture and event tracing feature of Gont allows the user to even streaming tracepoint data interleaved with packet and other tracing data in real-time to Wireshark. -

-
- -
-

Create a debugger

- - 

-							import do "github.com/stv0g/gont/v2/pkt/options/debug"
-
-							t := g.NewTracer(...)
-
-							d := g.NewDebugger(
-								// ... Tracepoints are defined her
-								do.BreakOnEntry(true),
-								do.ListenAddr("tcp:[::]:1234"), // DAP listening socket
-								do.ToTracer(t))
-
-
- -
-

Attach a debugger

- -

Debug all processes started by nodes of this network:

- 

-							n, _ := g.NewNetwork("", d)
-
- -

Debug all processes started by a node:

- 

-							h1 := n.NewHost("h1", d)
-
- -

Debug a single process:

- 

-							h1.RunGo("test/main.go", d)
-
- -

(Like for the event tracing)

-
- -
-

Define tracepoints

-
Break-, watchpoint location
- - 

-							import "github.com/go-delve/delve/service/api"
-							import do "github.com/stv0g/gont/v2/pkt/options/debug"
-
-							d := g.NewDebugger(
-								g.NewTracepoint(
-									do.Disabled(false),
-									do.Name("tp1"),
-									do.Message("A trace message with evaluated {placeholders}"),
-									do.Location(...), // A Delve locspec
-									do.Address(0x12312321),
-									do.File("main.go"),
-									do.Line(12),
-									do.FunctionName("main.main"),
-									do.FunctionNameRegex("main\.(main|setupLogger)"),
-									do.Condition("i % 100 == 0"),
-									do.HitCondition("> 100"),
-									do.HitConditionPerGoroutine(true),
-									do.Watch("p", api.WatchRead|api.WatchWrite),
-								),
-								...
-							)
-
-
- -
-

Define tracepoints

-
Gathering of breakpoint information
- - 

-							import do "github.com/stv0g/gont/v2/pkt/options/debug"
-
-							d := g.NewDebugger(
-								g.NewTracepoint(
-									...
-									do.Variable("j"),
-									do.Goroutine(true),
-									do.Stacktrace(10),
-									do.LoadLocals(...),
-									do.LoadArguments(
-										do.FollowPointers(true),
-										do.MaxVariableRecurse(3),
-										do.MaxStringLen(128),
-										do.MaxArrayValues(128),
-										do.MaxStructFields(32),
-									)
-								)
-							)
-
-
- -
-

VSCode Integration

- -

Gont generates VS Code launch compound configurations based on the active debugging sessions.

- - 

-							d.WriteVSCodeConfigs("", false)
-
-
-
- -
-

Topology factories (WIP)

- 

-						createHost := func(pos int) (*g.Host. error) {
-							return n.AddHost(fmt.Sprintf("h%d", pos))
-						}
-
-						linkHosts := func(a, b *g.Node) error {
-							_, err := n.AddRouter(fmt.Sprintf("r%d", pos),
-								g.NewInterface("eth0", a, o.AddressIPv4(10, 0, 0, a.Position, 24),
-								g.NewInterface("eth1", b, o.AddressIPv4(10, 0, 0, b.Position, 24)
-							)
-							return err
-						}
-
-						topo.Linear(n, 100, createHost, linkHosts)
-
-						n.Nodes["h0"].Traceroute(n.Nodes["h99"])
-
-
- -
-
-

CLI utility

-
-
-

CLI Example

-

Make network persistent

- 

-							n, _ := g.NewNetwork("mynet", o.Persistent(true))
-
- -

Introspect network after creation with gontc

- 

-							$ gontc list
-							mynet
-
-							$ gontc list mynet
-							mynet/h1
-							mynet/h2
-
-							$ gontc exec mynet/h1 hostname
-							h1.mynet.gont
-
-							$ gontc shell mynet/h1
-							$ mynet/h1: ip address show
-
-
-
-

CLI Usage

- 

-							Usage: gontc [flags] <command>
-
-								Supported <commands> are:
-
-								  identify                               return the network and node name if gontc is executed within a network namespace
-								  shell [<net>]/<node>                   get an interactive shell inside <node>
-								  exec  [<net>]/<node> <command> [args]  executes a <command> in the namespace of <node> with optional [args]
-								  list  [<net>]                          list all active Gont networks or nodes of a given network
-								  clean [<net>]                          removes the all or just the specified Gont network
-								  help                                   show this usage information
-								  version                                shows the version of Gont
-
-							   Example:
-
-								  gontc exec zorn/h1 ping h2
-
-							   Gont - The Go network tester
-								  Author Steffen Vogel <post@steffenvogel>
-
-
-
- -
-
-

How do I use it?

-
-
-

Gont ...

- -
-
-

Requirements

-
    -
  • Go 1.19
    • -
  • A moderate recent Linux kernel (>= 4.9) -
      -
    • mnt and net namespace support
      • -
    -
    • -
  • root access / NET_ADMIN caps
    • -
  • traceroute userspace tool
    • -
  • libpcap for packet captures
    • -
-
-
-

Roadmap

-
    -
  • Use pure Go implementation of traceroute
    • -
  • More topology factories
    • -
  • Design a nice logo
    • -
  • A Graphviz output for topology visualization
    • -
  • Automatic address assignment
    • -
  • Parser for network topologies from Dot, YAML or JSON description.
    • -
-
-
-

Dot example

-
-
- 

-									digraph D {
-										/* network options */
-										persistent = true
-
-										/* nodes */
-										h1 [type=host, exec="ping h2"]
-										h2 [type=host]
-										r1 [type=router]
-
-										/* links */
-										h1 -> r1 [address="10.0.0.1/24",
-										          mtu=9000]
-										h2 -> r1 [address="10.0.0.2/24",
-										          mtu=9000]
-									}
-
-
-
- -
-
-
-
- -
-

Thanks for your attention

-

Steffen Vogel

-
-

- @stv0g, - stv0g@0l.de -

-
-
- - EONERC Logo - -
-
- European Flag -

- The development of Gont has been supported by the ERIGrid 2.0 project of the H2020 Programme under Grant Agreement No. 870620. -

-
-
- -
- - - - - - - - - - - + + + + + + Gont + + + + + + + + + +
+
+
+
+
+
+ + + + + + + + + + diff --git a/docs/slides.css b/docs/slides.css new file mode 100644 index 0000000..6cea5b6 --- /dev/null +++ b/docs/slides.css @@ -0,0 +1,30 @@ +em { + font-weight: bold; +} + +code { + line-height: 1em; + font-size: 90%; +} + +li { font-size: 100% } +li li { font-size: 90% } +li li li { font-size: 80% } +li li li li { font-size: 70% } +li li li li li { font-size: 60% } + +.container{ + display: flex; +} + +.col{ + flex: 1; +} + +ul.inline-list li { + display:inline; +} + +.smaller { + font-size: 0.7em; +} diff --git a/docs/slides.md b/docs/slides.md new file mode 100644 index 0000000..cb7c993 --- /dev/null +++ b/docs/slides.md @@ -0,0 +1,1064 @@ +# Gont + +### A testing framework for distributed Go applications + +--- + +## Introduction + +... + +### What does Gont do? + +- Software-defined virtual networking for testing +- Define hosts, switches, routers in a single host +- Reentrancy +- Reproducibility +- Inspired by [Mininet](http://mininet.org/) + +... + +### Mininet + +“Mininet creates a realistic virtual network, running real kernel, +switch and application code, on a single machine (VM, cloud or native)” + +-- [mininet.org](http://mininet.org/) + +... + +### Mininet + +- Written in Python 2 +- Lacking active maintainer +- Focus on SDN: OpenFlow controllers +- No SSL cert on homepage?! + +**→** We need something better + +... + +### Why? + +- Describe network topologies quickly in code +- Automate construction of complex topologies +- Unit / CI testing +- Parallel test execution +- Example use-cases + - VPN / network tools development + - SDN Openflow controller development + - cunīcu: zeroconf • p2p • mesh • vpn agent + ([cunīcu](https://github.com/stv0g/cunicu)) + +--- + +A little detour: + +## Namespaces + +(Feel free to [skip this section](#first-some-definitions) if you are +only interested in learning about Gont) + +... + +### What are Linux namespaces? + +... + +### They.. + +- partition kernel resources + from **a process-perspective.** +- power most of Linux containerization tools. +- appear in many Linux subsystems. +- are used by many sandboxing solutions in browsers. + +... + +### Available Linux namespaces + +- **mnt:** Mounts +- **pid:** Process IDs +- **net:** Networking +- **ipc:** Interprocess Communication +- **uts:** Unix Timesharing (hostname) +- **user:** User identification & privileges +- **cgroup:** Process Control Groups +- **time:** System time + +... + +### But what exactly is a namespace? + +- Can be identified by a file descriptor +- A namespace can have multiple processes assigned to it +- It lives as long as there is still at least one remaining process +- Child processes inherit parent namespaces + +... + +### How do I create a namespace? + +- [unshare(2)](https://man7.org/linux/man-pages/man2/unshare.2.html) +- [clone(2)](https://man7.org/linux/man-pages/man2/clone.2.html) + +... + +### unshare(2) + +```go +func main() { + err := syscall.Unshare(syscall.CLONE_NEWNS); +} +``` + +... + +### How do I create a namespace? + +```go +static int child(void *arg) { + struct utsname uts; + + sethostname(arg, "ernie") + + uname(&uts) + printf("nodename in child: %s\n", uts.nodename); + + return 0; +} + +int main() { + struct utsname uts; + + /* Allocate stack for child */ + char *stack = malloc(STACK_SIZE); + if (stack == NULL) + return -1; + + /* Start new kernel task in new UTS namespace */ + pid_t child_pid = clone(child, stack + STACK_SIZE, CLONE_NEWUTS | SIGCHLD, NULL); + + /* Output hostname */ + uname(&uts) + printf("nodename in parent: %s\n", uts.nodename); +} +``` + +... + +### How can I share a namespace with other processes? + +- By forking with + [clone(2)](https://man7.org/linux/man-pages/man2/clone.2.html) +- By passing file descriptor and + [setns(2)](https://man7.org/linux/man-pages/man2/setns.2.html) + +... + +### Joining namespace of another process: /proc + +You can join the namespace of another process by using [/proc/{pid}/ns/\*](https://man7.org/linux/man-pages/man5/proc.5.html) + +```go +fd := syscall.Open("/proc/1234/ns/uts", syscall.O_RDONLY); +err := unix.Setns(fd, syscall.CLONE_NEWUTS); +``` + +**Note:** Can only set a single namespace per `netns(2)` invocation. + +... + +### Joining namespace of another process: pidfd_open + +You can join the namespace of another process by using [pidfd\_open(2)](https://man7.org/linux/man-pages/man2/pidfd_open.2.html) + +```go +pid_t pid = 1234; +int fd = pidfd_open(pid, 0); + +setns(fd, CLONE_NEWUTS | CLONE_NEWNET); + ``` + +**Note:** Can only set a multiple namespaces per `netns(2)` invocation. + +... + +### Persisting namespaces + +```go +err := syscall.Mount("/proc/self/ns/uts", + "/home/acs/my_uts_namespace", "", syscall.MS_BIND, nil); +``` + +And in another process... + +```go +fd := syscall.Open("/home/acs/my_uts_namespace", syscall.O_RDONLY); +err := unix.Setns(fd, syscall.CLONE_NEWUTS); +``` + +--- + +Back to Gont... + +## First some definitions + +... + +- **network** + + A set of *nodes* and *links* between them + +... + +- **node** + + A network namespace which represents any device in the *network* + +... + +- **link** + + A pair of `veth` network interfaces which are associated to separate *nodes* + +... + +- **interface** + + an endpoint with one or more assigned IP addresses + +... + +- **switch** + + A *node* containing a Linux layer 2 bridge and attached *interfaces* + +... + +- **host** + + A *node* with several configured *interfaces* + +... + +- **router** + + A *host* with IP forwarding enabled + +... + +- **nat** + + A special type of *router* which implements network address translation between a set of south- and north-bound *interfaces* + +... + +### Relationships + + + +... + +### Gont and Namespaces + +- for each *node*: + - a separate network namespace + - a separate uts namespace + - for a unique hostname within each *node* +- for each *network*: + - a separate mount namespace + - for a unique `/etc/hosts` files between *networks* + +--- + +## Examples + +### Show me some code! + +**⚠️ Warning:** Please note that these examples are show just for demonstration purposes and might not compile. + +... + +### Detailed examples + +Have a look at the following code for full fledged test/example code: + +- [Ping](https://github.com/stv0g/gont/blob/master/pkg/ping_test.go) +- [Run](https://github.com/stv0g/gont/blob/master/pkg/run_test.go) +- [NAT](https://github.com/stv0g/gont/blob/master/pkg/nat_test.go) +- [Switch](https://github.com/stv0g/gont/blob/master/pkg/switch_test.go) +- [Links](https://github.com/stv0g/gont/blob/master/pkg/link_test.go) +- [Firewall + Rules](https://github.com/stv0g/gont/blob/master/pkg/filter_test.go) +- [Packet + Tracing](https://github.com/stv0g/gont/blob/master/pkg/capture_test.go) +- [Event + Tracing](https://github.com/stv0g/gont/blob/master/pkg/trace_test.go) +- [Tracing with TLS + decryption](https://github.com/stv0g/gont/blob/master/pkg/capture_keylog_test.go) +- [Debugger](https://github.com/stv0g/gont/blob/master/pkg/debug_test.go) + +--- + +## Basic networking + +... + +### Two directly connected hosts + +```go +import g "github.com/stv0g/gont/v2/pkg" +import o "github.com/stv0g/gont/v2/pkg/options" + +... + +n, _ := g.NewNetwork("mynet") + +h1, _ := n.AddHost("h1") +h2, _ := n.AddHost("h2") + +n.AddLink( + g.NewInterface("eth0", h1, o.AddressIP("10.0.0.1/24")), + g.NewInterface("eth0", h2, o.AddressIP("10.0.0.2/24")), + ) + +h1.Ping(h2) + +``` + +(We use `g` and `o` throughout these examples as import aliases) + +... + +### Lets add a L2 switch + +```go +sw, _ := n.AddSwitch("sw") + +h1, _ := n.AddHost("h1", + g.NewInterface("eth0", sw, + o.AddressIP("10.0.0.1/24"))) + +h2, _ := n.AddHost("h2", + g.NewInterface("eth0", sw, + o.AddressIP("10.0.0.2/24"))) + +h1.Ping(h2) +``` + +... + +### How about a L3 router? + +```go +sw1, _ := n.AddSwitch("sw1") +sw2, _ := n.AddSwitch("sw2") + +h1, _ := n.AddHost("h1", g.NewInterface("eth0", sw1, + o.AddressIP("10.0.0.2/24"))) +h2, _ := n.AddHost("h2", g.NewInterface("eth0", sw2, + o.AddressIP("10.0.1.2/24"))) + +n.AddRouter("r1", + g.NewInterface("eth0", sw, o.AddressIP("10.0.0.1/24")), + g.NewInterface("eth1", sw, o.AddressIP("10.0.1.1/24")) +) + +h1.Ping(h2) +``` + +... + +### Lets do some evil NATing 😈 + +```go +sw1, _ := n.AddSwitch("sw1") +sw2, _ := n.AddSwitch("sw2") + +h1, _ := n.AddHost("h1", g.NewInterface("eth0", sw1, + o.AddressIP("10.0.0.2/24"))) +h2, _ := n.AddHost("h2", g.NewInterface("eth0", sw2, + o.AddressIP("10.0.1.2/24"))) + +n.AddNAT("n1", + g.NewInterface("eth0", sw, o.SouthBound, + o.AddressIP("10.0.0.1/24")), + g.NewInterface("eth1", sw, o.NorthBound, + o.AddressIP("10.0.1.1/24"))) + +h1.Ping(h2) +``` + +... + +### How about a whole chain of routers? + +```go +var firstSwitch *g.Switch = n.AddSwitch("sw0") +var lastSwitch *g.Switch = nil + +for i := 1; i < 100; i++ { + swName := fmt.Printf("sw%d", i) + rtrName := fmt.Printf("r%d", i) + + newSwitch, _ := n.AddSwitch(swName) + + n.AddRouter(rtrName, + g.NewInterface("eth0", lastSwitch, + o.AddressIP("10.0.0.1/24")), + g.NewInterface("eth1", newSwitch, + o.AddressIP("10.0.1.1/24")) + ) + + lastSwitch = newSwitch +} + +h1, _ := n.AddHost("h1", g.NewInterface("eth0", firstSwitch, + o.AddressIP("10.0.0.2/24"))) +h2, _ := n.AddHost("h2", g.NewInterface("eth0", lastSwitch, + o.AddressIP("10.0.1.2/24"))) + +h1.Ping(h2) +``` + +--- + +## Running your application + +Inside the network namespaces / hosts + +... + +### exec.Cmd API + +```go +// Get a exec.Cmd-like struct +cmd := h1.Command("ping", "h2") +out, err := cmd.CombinedOutput() + +// Directly run a simple process synchronously +cmd, err := h1.Run("ping", "h2") + +// Directly start asynchronously +cmd, err := h1.Start("ping", "h2") + +time.Sleep(5 * time.Second) + +err = cmd.Process.Signal(os.Interrupt) +cmd.Wait() +``` + +The `g.Node` type implements an API similar to the one provided by Go's +`exec` package. + +... + +### Pass options + +```go +import co "github.com/stv0g/gont/v2/pkg/options/cmd" + +outb := &bytes.Buffer{} + +cmd := h1.Command("ping", "1.1.1.1", + co.DisableASLR(true), + co.Dir("/custom/working/dir"), + co.EnvVar("DEBUG", "1"), + co.Stdin(...), // pass any io.Reader + co.Stdout(outb), // pass any io.Writer (can be repeated) + co.Stderr(...), // pass any io.Writer (can be repeated) +) + +print(outb.String()) +``` +... + +### Pass non-string arguments + +```go +ip := net.ParseIP("1.1.1.1") + +cmd := h1.Command("ping", "-c", 10, "-i", 0.1, ip) +``` + +... + +### Go functions + +```go +h1.RunFunc(func() { + r := http.Get("http://h2:8080") + io.Copy(os.Stdout, r.Body) +}) +``` + +Call a function inside a network namespace of host `h1` but still +in the same process so you can use channels and access global variables. + +**⚠️ Warning:** Spawning Goroutines from within the callback +is only indirectly supported: + +```go +h1.RunFunc(func() { + go h1.RunFunc(func() { ... }) +}) +``` + +... + +### Go packages + +```go +cmd, err := h1.RunGo("test/prog.go", "arg1") +``` + +--- + +## Firewall + +by Netfilter's nftables + +... + +### Add some firewall rules for a host + +```go +import fo "github.com/stv0g/gont/v2/options/filter" + +_, src, _ := net.ParseCIDR("10.0.0.1/32") + +h1, _ := n.AddHost("h1", + o.Filter( + g.FilterInput, + fo.Source(src), + fo.Protocol(unix.AF_INET), + fo.TransportProtocol(unix.IPPROTO_TCP), + fo.SourcePortRange(0, 1024))) +``` + +--- + +## Network Emulation + +by Linux's Traffic Control: Netem Qdisc + +... + +### Attach a netem Qdisc to an interface + +```go +import tco "github.com/stv0g/gont/v2/options/tc" + +n.AddLink( + g.NewInterface("eth0", h1, + o.WithNetem( + tco.Latency(50 * time.Millisecond), + tco.Jitter(5 * time.Millisecond), + tco.Loss(0.1), + ), + o.AddressIP("10.0.0.1/24")), + g.NewInterface("eth0", h2, + o.AddressIP("10.0.0.2/24")), +) + +h1.Ping(h2) +``` + +--- + +## Packet captures + +PCAP, WireShark, tshark + +... + +### Write to packets to + +Gont merges and sorts packet captures in real-time +from multiple interfaces and records them to one of the following sinks: + +- Using [PCAPng](https://github.com/pcapng/pcapng) format + - Regular files + - Named pipes + - TCP / UDP / Unix listeners + - WireShark real-time stream +- Go channels +- Go callback functions. + +... + +### Filtering + +Captured network traffic can be filtered by + +- Selected Gont nodes and interfaces +- eBPF filter programs +- [pcap-filter(7)](https://www.tcpdump.org/manpages/pcap-filter.7.html) + expressions +- Go callback functions (⚠ slow!) + +... + +### Session key logging + +Most transport layer encryption protocols today provide [perfect forward +secrecy](https://en.wikipedia.org/wiki/Forward_secrecy) by using +short-lived ephemeral session keys. + +Gont offers a feature to log these session keys into a PCAPng file to +enable a decryption of upper layer protocols with a dissector tool like +Wireshark. + +... + +### Example + +![](images/session-key-logging.png) + +--- + +## Distributed Tracing + +User defined events, Log messages + +... + +### A trace event + +Gont supports collecting trace events from all processes running in a +distributed system which can carry the following information. Gont +orders trace events by time and saves them to different destinations for +analysis. + +```go +type Event struct { + Timestamp time.Time // Timestamp when the event occurred + Type string // Either: 'log', ́'trace', 'break' & ́ watchpoint' + Level uint8 // Log level + Message string // A human readable description + Source string // Logger name + PID int + Function string + File string + Line int + Args []any + Data any // User defined data +} +``` + +... + +### Write to trace events to + +- JSON files +- Go channels +- Go callbacks +- Packet captures + +... + +### Create a tracer + +```go +import "github.com/stv0g/gont/v2/trace" +import to "github.com/stv0g/gont/v2/options/trace" + +c := g.NewCapture(...) +f, _ := os.OpenFile(...) +ch := make(chan trace.Event) + +t := g.NewTracer( + to.ToFile(f) + to.ToFilename("trace.log"), + to.ToChannel(ch), + to.ToCapture(c), + to.Callback(func(e trace.Event) { ... })) + +t.Start() +``` + +... + +### Attach the tracer + +Trace all processes started by nodes of this network + +```go +n, _ := g.NewNetwork("", t) +``` + +Trace all processes started by a node + +```go +h1 := n.NewHost("h1", t) +``` + +Trace a single process + +```go +h1.RunGo("test/main.go", t) +``` + +... + +### Trace with trace package + +```go +import "github.com/stv0g/gont/v2/pkg/trace" + +someData := map[string]string{"Hello": "World"} +count := 42 + +trace.Start(0) + +trace.PrintfWithData(someData, "Count is: %d", count) +trace.Print("Another message") + +trace.Stop() +``` + +Works from: + +- Gont process itself +- Any process spawned via Gont's + `Host.{Command,Run,RunGo,Start,StartGo}(...)` functions + +... + +### Trace via slog structured logging package + +```go +import "golang.org/x/exp/slog" +import "github.com/stv0g/gont/v2/pkg/trace" + +// Create a slog handler which emits trace events +handler := trace.NewTraceHandler(slog.HandlerOptions{}) + +// Add the tracing option which emits a trace event for each log message +logger := slog.New(handler) +``` + +Each log message emits a trace event which includes the log message, +filename, line number as well function name and more. +Any fields passed to to zap structured logger are included in the `Data` +field of the `Event` structure. + +... + +### Trace via zap logging package + +```go +import "go.uber.org/zap" +import "github.com/stv0g/gont/v2/pkg/trace" + +// Add the tracing option which emits a trace event for each log message +logger := zap.NewDevelopment(trace.Log()) + +// Add the caller info which gets also included in the trace event +logger = logger.WithOptions(zap.AddCaller()) + +// Give the logger some name which is added as the Source field to the trace event +logger = logger.Named("my-test-logger") +``` + +Each log message emits a trace event which includes the log message, +filename, line number as well function name and more. +Any fields passed to to zap structured logger are included in the `Data` +field of the `Event` structure. + +--- + +## Distributed Debugging + +powered by Delve + +... + +### Introduction + +Gont can manage Delve debugger instances attached to spawned +sub-processes. These Delve instances can be used to debug or trace +multiple applications in a distributed system simultaneously. Users can +either use DAP-compatible IDEs like VScode to attach to those processes, +or record tracepoint data generated by Delve break- & watchpoints to a +PCAPng file. + +Its integration with the packet capture and event tracing feature of +Gont allows the user to even streaming tracepoint data interleaved with +packet and other tracing data in real-time to Wireshark. + +... + +### Create a debugger + +```go +import do "github.com/stv0g/gont/v2/pkt/options/debug" + +t := g.NewTracer(...) + +d := g.NewDebugger( +// ... Tracepoints are defined her +do.BreakOnEntry(true), +do.ListenAddr("tcp:[::]:1234"), // DAP listening socket +do.ToTracer(t)) +``` + +... + +### Attach a debugger + +Debug all processes started by nodes of this network: + +```go +n, _ := g.NewNetwork("", d) +``` + +Debug all processes started by a node: + +```go +h1 := n.NewHost("h1", d) +``` + +Debug a single process: + +```go +h1.RunGo("test/main.go", d) +``` + +(Like for the event tracing) + +... + +### Define tracepoints + +... + +#### Break-, watchpoint location + +```go +import "github.com/go-delve/delve/service/api" +import do "github.com/stv0g/gont/v2/pkt/options/debug" + +d := g.NewDebugger( + g.NewTracepoint( + do.Disabled(false), + do.Name("tp1"), + do.Message("A trace message with evaluated {placeholders}"), + do.Location(...), // A Delve locspec + do.Address(0x12312321), + do.File("main.go"), + do.Line(12), + do.FunctionName("main.main"), + do.FunctionNameRegex("main\.(main|setupLogger)"), + do.Condition("i % 100 == 0"), + do.HitCondition("> 100"), + do.HitConditionPerGoroutine(true), + do.Watch("p", api.WatchRead|api.WatchWrite), + ), +... +) +``` + +... + +### Define tracepoints + +... + +#### Gathering of breakpoint information + +```go +import do "github.com/stv0g/gont/v2/pkt/options/debug" + +d := g.NewDebugger( + g.NewTracepoint( + ... + do.Variable("j"), + do.Goroutine(true), + do.Stacktrace(10), + do.LoadLocals(...), + do.LoadArguments( + do.FollowPointers(true), + do.MaxVariableRecurse(3), + do.MaxStringLen(128), + do.MaxArrayValues(128), + do.MaxStructFields(32), + ) + ) +) +``` + +... + +### VSCode Integration + +Gont generates VS Code launch compound configurations based on the +active debugging sessions. + +```go +d.WriteVSCodeConfigs("", false) +``` + +--- + +### Topology factories (WIP) + +```go +createHost := func(pos int) (*g.Host. error) { + return n.AddHost(fmt.Sprintf("h%d", pos)) +} + +linkHosts := func(a, b *g.Node) error { + _, err := n.AddRouter(fmt.Sprintf("r%d", pos), + g.NewInterface("eth0", a, o.AddressIPv4(10, 0, 0, a.Position, 24), + g.NewInterface("eth1", b, o.AddressIPv4(10, 0, 0, b.Position, 24) + ) + return err +} + +topo.Linear(n, 100, createHost, linkHosts) + +n.Nodes["h0"].Traceroute(n.Nodes["h99"]) +``` + +--- + +## CLI utility + +... + +### CLI Example + +Make network persistent + +```go +n, _ := g.NewNetwork("mynet", o.Persistent(true)) +``` + +Introspect network after creation with `gontc` + +```sh +$ gontc list +mynet + +$ gontc list mynet +mynet/h1 +mynet/h2 + +$ gontc exec mynet/h1 hostname +h1.mynet.gont + +$ gontc shell mynet/h1 +$ mynet/h1: ip address show +``` + +... + +### CLI Usage + +``` +Usage: gontc [flags] + + Supported are: + + identify return the network and node name if gontc is executed within a network namespace + shell []/ get an interactive shell inside + exec []/ [args] executes a in the namespace of with optional [args] + list [] list all active Gont networks or nodes of a given network + clean [] removes the all or just the specified Gont network + help show this usage information + version shows the version of Gont + + Example: + + gontc exec zorn/h1 ping h2 + + Gont - The Go network tester + Author Steffen Vogel +``` + +--- + +## How do I use it? + +... + +### Gont ... + +- can be used in Go unit & integration-tests + - on Github-powered CI runners +- is licensed under Apache-2.0 +- is available at + [github.com/stv0g/gont](https://github.com/stv0g/gont) +- is documented at + [pkg.go.dev/github.com/stv0g/gont](https://pkg.go.dev/github.com/stv0g/gont) +- has slides at [stv0g.github.io/gont](https://stv0g.github.io/gont) + +... + +### Requirements + +- Go 1.19 +- A moderate recent Linux kernel (>= 4.9) + - `mnt` and `net` namespace support +- root access / `NET_ADMIN` caps +- traceroute userspace tool +- libpcap for packet captures + +... + +### Roadmap + +- Use pure Go implementation of traceroute +- More topology factories +- Design a nice logo +- A Graphviz output for topology visualization +- Automatic address assignment +- Parser for network topologies from + [Dot](https://graphviz.org/doc/info/lang.html), + [YAML](https://yaml.org/) or [JSON](https://www.json.org/) + description. + +... + +#### Dot example + +```dot +digraph D { + /* network options */ + persistent = true + + /* nodes */ + h1 [type=host, exec="ping h2"] + h2 [type=host] + r1 [type=router] + + /* links */ + h1 -> r1 [address="10.0.0.1/24", + mtu=9000] + h2 -> r1 [address="10.0.0.2/24", + mtu=9000] +} +``` + +![](images/graphviz.svg) + +--- + +### Thanks for your attention + +Steffen Vogel + +[@stv0g](https://github.com/stv0g/gont), + +[![EONERC +Logo](https://fein-aachen.org/img/logos/eonerc.png)](https://www.acs.eonerc.rwth-aachen.de) + +European Flag + +The development of Gont has been supported by the [ERIGrid +2.0](https://erigrid2.eu) project of the H2020 Programme under [Grant +Agreement No. 870620](https://cordis.europa.eu/project/id/870620).