Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add threading #100

Merged
merged 32 commits into from
Nov 6, 2019
Merged

Add threading #100

merged 32 commits into from
Nov 6, 2019

Conversation

HugoSoszynski
Copy link
Contributor

@HugoSoszynski HugoSoszynski commented Oct 28, 2019

✨ Pull Request Template

‼️ Once all the checklist is done you have to:

📃 Type of change

Breaking change: fix or feature that would cause existing functionality to not work as expected.

💡 Related Issue(s)

✒️ Description

Core

Added simple thread group to the project in order to run the asio callbacks.
The ThreadGroup class uses a simple list of threads because it is made to create threads once and join all threads once.

On stopping signal, no longer clean the sessions before the handlers
are finished running.
Now stopping io_context to prevent any callback push in the queue
during a graceful exit.
This removes memory errors, leaks, bad_weak_ptr and bad file descriptors.
Added it to monitoring thread too.

Rework of the "Send" methods of the session to fix filter closing and potential datarace on closing filters.

Some refactoring around the Generator and the Session.

Cmake

  • Updated C++ version to 14 to 17
  • Removed fend from default compilation
  • Removed freputation from default compilation

Redis Manager

  • works for heavy multithreading
  • supports individual connections for each thread
  • supports master/slave connections
  • easy query, wide response types (int, string, array)
  • automatic disconnection/reconnection timeouts (default 20 seconds)
  • keep alive thread
  • fallback to slave connection if master failed and slave exists
  • stills connects to IP:port or unix sockets

Filters

  • Updating all the filters to support threading.
  • Updating all the filters to match the refactor of Generator and Session.

Content Inspection

  • Fix memory leaks
  • Rename implementation of TCP states to avoid conflicts (GNU implementation not used as order of definitions is important)
  • Add conditional inclusion for assert()

DGA

  • Now closing tensorflow Session the proper way.
  • Tensorflow still has internal memory leaks that we cannot solve

Session (the filter)

  • Rework functions to send data to filters/clients
  • Removed cache because of potential security issues while using it (e.g. session deleted in redis but still in the in-memory cache)

Tests

RedisManager:

  • Simple master connect
  • Slave -> master connect
  • Slave -> master connect, master fail
  • Slave -> master connect, master off
  • Master connect, disconnect/reconnect after timeout
  • Multi-threading tests to the RedisManager

DGA: (The test are not using valgrind due to internal tensorflow memory leaks)

  • Token map loading tests

🎯 Test Environments

FreeBSD 12.0

  • Redis 4.0.14
  • Boost 1.71.0
  • clang++ 8.0.1
  • CMake 3.15.2
  • Python 3.6.9

Ubuntu 19.04

  • Redis 5.0.3
  • Boost 1.71.0
  • g++ 8.3.0
  • CMake 3.13.4
  • Python 3.7.3
  • Valgrind 3.14.0

✔️ Checklist:

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

  • 🙋 I certify on my honor that all the information provided are true, and I've done all I can to deliver a high quality code

HugoSoszynski and others added 8 commits October 16, 2019 09:30
Added simple thread group to the project
in order to run the asio callbacks.

The ThreadGroup class uses a simple list of threads because it is made
to create threads once and joine all threads once.

WIP: Making the whole core thread safe using mutexes.
All generators were sharing the same methods but with duplicate code.
Now the generators inherit from an abstract implementing the common code

We now use the main process ressources to process requests.
The main process is considered to be part of the processing threads.

On stopping signal, no longer clean the sessions before the handlers
are finished running.
This removes memory errors, leaks, bad_weak_ptr and bad file descriptors

Now stopping io_context to prevent any callback push in the queue
during a graceful exit.
Added it to monitoring thread too.

Getting back to accepting sessions one by one
because of strange behaviour.
Still WIP.

Applied the generator changes to the log filter.

Added a mutex to the log filter because of undefined behaviour
using filestreams in a threaded environment.
Added a mutex to access the cache in read/write because it is not
thread safe natively.

Updated fconnection generator to match prior updates
(06fd113)

Updated fhostlookup generator to match prior updates
(06fd113)

The host db of fhostlookup is thread safe for multiple reader with no
writer.
### CMAKE
- updated C++ version to 14 to 17

### Redis Manager
- works for heavy multithreading
- supports individual connections for each thread
- supports master/slave connections
- easy query, wide response types (int, string, array)
- automatic disconnection/reconnection timeouts (default 20 seconds)
- keep alive thread
- fallback to slave connection if master failed and slave exists
- stills connects to IP:port or unix sockets

### Redis Manager
- added first tests
  - simple master connect
  - slave -> master connect
  - slave -> master connect, master fail
  - slave -> master connect, master off
  - master connect, disconnect/reconnect after timeout
Correcting warnings due to worng initialization order of class members.

Updated fdga generator to match prior updates
(06fd113)

faup_hadler_t is unique for a given entry thus making it not thread safe.
Now passing options to thread because they will only be read safely.
Now faup_handler is created when we treat a new entry.

Updated fuseragent generator to match prior updates
(06fd113)

Now properly closing tensorflow session when all execution are finished.
(fdga & fuseragent)
@HugoSoszynski HugoSoszynski added this to the Version 1.1 milestone Oct 28, 2019
@HugoSoszynski HugoSoszynski self-assigned this Oct 28, 2019
Added the Workflow method to the Session class as it is
shared by all the filters.

Removed Workflow method from fdga.

Upadted Session to close on client connection interruption
only (without crashing).

Removed include of maxminddb from DGA Generator because useless.

FDGA threading hand-tested on Ubuntu 19.04.
Removed Workflow methods from filters to comply to
6cab130 :
  - fconnection
  - fhostlookup
  - flogs
Using lock_guars and unique_lock instead of raw mutex.

Reducing use of the _setMut mutex to the very minimum needed.

Removed the thread_id because it was set but never used.
Updated finspection generator to match prior updates
(06fd113)
frikilax and others added 7 commits October 31, 2019 10:22
- rename implementation of TCP states to avoid conflicts (GNU implementation not used as order of definitions is important)
- add conditional inclusion for assert()
This is a WIP because fanomaly now compiles but is not tested
thus not considered stable.

Tests and stability will come for  version 1.1.
This is a WIP because fsession now compiles but is not tested
thus not considered stable.

Tests and stability will come for  version 1.1.
frikilax and others added 4 commits November 5, 2019 14:56
Threads are useful on this filter in order to be able to process a lot of
entries simultaneously.

The workflow of the "cron" thread is not impacted.
Fixed by removing old class method from the hpp file.
As fend is going to be deleted in a close future
we remove it from default compilation.

As we no longer use freputation we remove it from default compilation.

We no longer provide support for those filters.
@HugoSoszynski HugoSoszynski marked this pull request as ready for review November 6, 2019 14:38
@HugoSoszynski HugoSoszynski merged commit 34de47c into dev Nov 6, 2019
@HugoSoszynski HugoSoszynski deleted the add_threading branch November 6, 2019 16:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants