-
Notifications
You must be signed in to change notification settings - Fork 11
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add threading #100
Merged
Merged
Add threading #100
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Added simple thread group to the project in order to run the asio callbacks. The ThreadGroup class uses a simple list of threads because it is made to create threads once and joine all threads once. WIP: Making the whole core thread safe using mutexes.
All generators were sharing the same methods but with duplicate code. Now the generators inherit from an abstract implementing the common code We now use the main process ressources to process requests. The main process is considered to be part of the processing threads. On stopping signal, no longer clean the sessions before the handlers are finished running. This removes memory errors, leaks, bad_weak_ptr and bad file descriptors Now stopping io_context to prevent any callback push in the queue during a graceful exit. Added it to monitoring thread too. Getting back to accepting sessions one by one because of strange behaviour. Still WIP. Applied the generator changes to the log filter. Added a mutex to the log filter because of undefined behaviour using filestreams in a threaded environment.
### CMAKE - updated C++ version to 14 to 17 ### Redis Manager - works for heavy multithreading - supports individual connections for each thread - supports master/slave connections - easy query, wide response types (int, string, array) - automatic disconnection/reconnection timeouts (default 20 seconds) - keep alive thread - fallback to slave connection if master failed and slave exists - stills connects to IP:port or unix sockets ### Redis Manager - added first tests - simple master connect - slave -> master connect - slave -> master connect, master fail - slave -> master connect, master off - master connect, disconnect/reconnect after timeout
Correcting warnings due to worng initialization order of class members. Updated fdga generator to match prior updates (06fd113) faup_hadler_t is unique for a given entry thus making it not thread safe. Now passing options to thread because they will only be read safely. Now faup_handler is created when we treat a new entry. Updated fuseragent generator to match prior updates (06fd113) Now properly closing tensorflow session when all execution are finished. (fdga & fuseragent)
Added the Workflow method to the Session class as it is shared by all the filters. Removed Workflow method from fdga. Upadted Session to close on client connection interruption only (without crashing). Removed include of maxminddb from DGA Generator because useless. FDGA threading hand-tested on Ubuntu 19.04.
Removed Workflow methods from filters to comply to 6cab130 : - fconnection - fhostlookup - flogs
Using lock_guars and unique_lock instead of raw mutex. Reducing use of the _setMut mutex to the very minimum needed. Removed the thread_id because it was set but never used.
Updated finspection generator to match prior updates (06fd113)
- rename implementation of TCP states to avoid conflicts (GNU implementation not used as order of definitions is important) - add conditional inclusion for assert()
This is a WIP because fanomaly now compiles but is not tested thus not considered stable. Tests and stability will come for version 1.1.
This is a WIP because fsession now compiles but is not tested thus not considered stable. Tests and stability will come for version 1.1.
df53550
to
d4ae423
Compare
Threads are useful on this filter in order to be able to process a lot of entries simultaneously. The workflow of the "cron" thread is not impacted.
Fixed by removing old class method from the hpp file.
As fend is going to be deleted in a close future we remove it from default compilation. As we no longer use freputation we remove it from default compilation. We no longer provide support for those filters.
frikilax
requested changes
Nov 6, 2019
Co-Authored-By: frikilax <[email protected]>
Co-Authored-By: frikilax <[email protected]>
Co-Authored-By: frikilax <[email protected]>
Co-Authored-By: frikilax <[email protected]>
Co-Authored-By: frikilax <[email protected]>
Co-Authored-By: frikilax <[email protected]>
frikilax
approved these changes
Nov 6, 2019
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
✨ Pull Request Template
📃 Type of change
Breaking change: fix or feature that would cause existing functionality to not work as expected.
💡 Related Issue(s)
✒️ Description
Core
Added simple thread group to the project in order to run the asio callbacks.
The ThreadGroup class uses a simple list of threads because it is made to create threads once and join all threads once.
On stopping signal, no longer clean the sessions before the handlers
are finished running.
Now stopping io_context to prevent any callback push in the queue
during a graceful exit.
This removes memory errors, leaks, bad_weak_ptr and bad file descriptors.
Added it to monitoring thread too.
Rework of the "Send" methods of the session to fix filter closing and potential datarace on closing filters.
Some refactoring around the Generator and the Session.
Cmake
Redis Manager
Filters
Content Inspection
DGA
Session (the filter)
Tests
RedisManager:
DGA: (The test are not using valgrind due to internal tensorflow memory leaks)
🎯 Test Environments
FreeBSD 12.0
Ubuntu 19.04
✔️ Checklist: