docs: update docs for design choices

README.md

@@ -15,6 +15,7 @@ and the underlying implementation of the asynchronous interface.
 - Fast context switches (powered by Boost.Context)
 - Multiple backends: `kqueue`, `epoll`, `io_uring` (all of these seem to be edge triggered)
 - Supports unix domain sockets
+- Support for pinning event loop to hardware threads.
 
 ## Todo
 
@@ -33,7 +34,7 @@ https://webflow.com/made-in-webflow/website/Apple-Style-Grid Can make this as th
 ## Resources
 
 - [Tiger Style](https://github.com/tigerbeetle/tigerbeetle/blob/main/docs/TIGER_STYLE.md)
--
+- [⭐Fibers](https://graphitemaster.github.io/fibers/)
 
 *`kqueue`*: https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man2/kqueue.2.html
 

docs/ARCHITECTURE.md

@@ -1,28 +1,4 @@
-Any task that is ran by the event loop MUST NOT BLOCK. Disastrous things will
-happen if any task blocks. Furthermore, once a task has started, we cannot pre-empt
-tasks as that would essentially become threading, and a whole suite of problems.
-
-In the case of CPU intensive tasks, a threaded based program would perform
-much better.
-
-We might have to use threaded workers for filesystem operations, if `io_uring`
-is not available. We can also use this to offload CPU intensive tasks. TODO: Perhaps,
-for us, the orderbook building can be offloaded? Does this mean we need a separate `submit` queue?
-
-## eventloop interface
-
-```cpp
-event_loop.write(fd, buffer, bytesToWrite, callback);
-char buffer[1024];
-event_loop.read(fd, &buffer, bytesToRead, callback);
-```
-
-`kqueue` and `io_uring` themselves have a batch size. But if that is full, we
-move the stuff into the overflow queue.
-
-Overflow queue.
-
-## Design choices
+# Design choices
 
 Callback vs Threaded workers?
 What's our problem?
@@ -33,29 +9,38 @@ be a state machine that is called every time a message is received. This functio
 
 The coroutine runs until it yields cause it waits for an event
 
-# Boost.Context
+## Cooperative multi tasking
 
-# Backends
+- The whole idea behind fibers is that instead of having time quantums, we trust that the tasks will yield within a
+  reasonable amount of time. In exchange, we get full control over scheduling.
+- The drawback of this approach is that any task ran by the event loop MUST NOT BLOCK. Disastrous things will happen if
+  any task blocks.
+    - Furthermore, once a task has started, we cannot pre-empt tasks as that would essentially become
+      threading, and a whole suite of problems.
 
-## Kqueue
+https://agraphicsguynotes.com/posts/fiber_in_cpp_understanding_the_basics/
 
-`KV_CLEAR`: Prevents kq from signalling the same event over and over again. If a socket wasn't read fully,
-then kq will signal us again, so instead of having to process the same signal multiple times, we make sure that we
-fully consume the data.
+## Memory Management for Fibers
 
-The `poll` interface requires passing one big array containing all the fds in each event loop epoch. A `kqueue`
-event loop on the other hand notifies the kernel of changes to the monitored fds by passing in a changelist. This
-can be done in two ways,
+- The allocation of stacks is delayed until the scheduler (reactor) is ready to execute the fiber.
+    - Can easily burn through a lot of memory especially if stacks are reused.
+    - This delayed stack allocation enables reuse optimisation: keep a free-list of stacks, and reuse it for new fibers
+      instead of `malloc`-ing new memory
+- Not all fibers will require the same stack size. Can take inspiration from slab allocator design, keep a free-list for
+  3-different stack sizes and allocate from that instead.
 
-1. Call `kevent` for each change to actively monitored fd list
-2. Build up a list of descriptor changes and pass it to the kernel the next event loop epoch. Reduces number of
-   sys-calls made
+## Pinning of hardware threads
 
-```cpp
-struct kevent {
-  uintptr_t ident; // Most of the time refers to a FD, but its meaning can change based on the filter. Essentially a value to identify the event
-  filter; // Determines the kernel filter to process this event, e.g `EVFILT_TIMER`
+- We are the scheduler in this case. Don't want fibers to jump across cores, since we are scheduling them ourselves.
+  Also incur cache penalties
+- Clock drift: If we use `rdtsc` for timing, measured in terms of clock cycles, different cores will have clock drift
+  and comparing cycle count is not fair. Using system calls is out of the question, too costly in terms of context
+  switch.
+- OS Specific
+    - Windows: `SetThreadAffinityMask`
+    - Linux: `pthread_setaffinity_np`
 
-};
-```
+## Overflow queue
 
+`kqueue` and `io_uring` themselves have a batch size. But if that is full, we
+move the stuff into the overflow queue.

docs/NOTES.md

@@ -0,0 +1,28 @@
+# Reactor Backends
+
+## `kqueue` (Darwin)
+
+`KV_CLEAR`: Prevents kq from signalling the same event over and over again. If a socket wasn't read fully,
+then kq will signal us again, so instead of having to process the same signal multiple times, we make sure that we
+fully consume the data.
+
+The `poll` interface requires passing one big array containing all the fds in each event loop epoch. A `kqueue`
+event loop on the other hand notifies the kernel of changes to the monitored fds by passing in a changelist. This
+can be done in two ways,
+
+1. Call `kevent` for each change to actively monitored fd list
+2. Build up a list of descriptor changes and pass it to the kernel the next event loop epoch. Reduces number of
+   sys-calls made
+
+```cpp
+struct kevent {
+  uintptr_t ident; // Most of the time refers to a FD, but its meaning can change based on the filter. Essentially a value to identify the event
+  filter; // Determines the kernel filter to process this event, e.g `EVFILT_TIMER`
+
+};
+```
+
+## `io_uring` (Linux)
+
+## `epoll` (Older Linux distributions)
+