Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* not necessary for index for every single channel anymore * swap at %25 * swap at %20 * prepend HELK to all elasticsearch templates * remove appended "template" * fix accidental appending of .json to ES templates * use more common field name for ETL tagging * change original timestamp, and don't use "log" - as HELK will include more than just "logs" and should be more defined as "data" when referring to things applicable to many sources/data * move setting computer_name to single file (ie: remove duplicate/unnecessary code) * ossem for network layer 3, layer 7 and network community id * community id / ecs check * begin ecs to helk * update to use event_original_message * event_etl_processed_timestamp * update network OSSEM * use event_etl_pipeline * add naming * rever elastalert helk rules back to keyword subfield changes * revert elasticsearch template mappings back to keyword subfield changes * revert elasticsearch template mappings back to keyword subfield changes * use etl_pipeline * etl_processed_timestamp * elastalert, elasticsearch, template mappings back to keyword subfield changes * elastalert, elasticsearch, template mappings * revert elastalert helk rules back to keyword subfield changes * process_parent_id hex to dec * user_data winlog field removal * user_data winlog field removal v2 * update logstash.yml for newest versions * #346 geo fields despite schema/taxonomy * beginning of the end.. adding Zeek and NSM ossem fields * lowercase title * match new OSSEM updates * update jvm options file to 7.4 * G1GC is fixed, so switch from CMS to G1GC * add log4j2.properties for elasticsearch * cleanup logstash config so only enabled settings and very common settings that may change add full logstash config reference since we are cleaning up to only have enabled settings * further ossem to helk * Elastic 7.4 * give users path for custom logstash configs that won't be updated/modified/merged during HELK updates if they choose to have these * Elastic 7.4 * add additional logstash config options for reference and future modifications * add network and zeek index patterns in kibana * elastic 7.4.0 * logstash plugin updates * logstash plugin documentation update * build latest kafka and also elastalert (testing currently) * #314 user_target_ > target user_ * #314 user_target_ > target user_ * LogonType is always converted to logon_type, so cleanup and make global * #314 add EventID:4626 * #314 add EventID:4664 * #314 finish `process_target_` to `target_process_` * more verbose ES check * #314 * revert jvm options file * correct some process_parent fields * add 4696 #314 * correct 4688 #314 * process cleanup, corrections for #314 OSSEM, and prevent possible collisions in names * sysmon 255 #314 * cleanup * annotate works for winlogbeat 7 and above * 5168 typos * windows event IDs 4722,4723,4724,4725,4726 #314 * end everything in privilege_list * all Audit User Account Management #314 * fix duplicate key * more verbose ES check * add log rotation #354 * add log rotation #354 * Add TargetLinkedLogonId * add addition of cloned/original logs * more specific matching * g1-NOTSO-gc ... g1gc * forward to more logical layout * kafka create&consume zeek topic * make sure not duplicate original and previous message field name * add addition of cloned/original logs * singular elasticsearch output for many of the windows event logs * Audit Security Group Management and other group schema ossem<>logstash #314 * (de)increase refresh * CallerProcessId vs ProcessID * bump to elastic 7.4.1 * increase kafka partitions for greater consumption for future releases * increase elasticsearch heap checks * powershell command invocation indexed and fixed * - 7.4.1 logstash plugin updates - plugin management information update * update .gitignore * support custom logstash and kibana files * audit distribution group management HELK<>OSSEM for event id's 4744, 4745, 4746, 4747, 4748, 4749, 4750, 4751, 4752, 4753, 4759, 4760, 4761, 4762, 4763 #314 * match latest upstream jvm.options * log name case insensitive temp fix #209 * - add original/chain-of-custody log kibana index pattern - add catch all indexme kibana index pattern * logstash plugin updates for 7.4.2 * elastic stack 7.4.2 * if data sources already have IPv6 true/false define, and to keep backwards compatible HELK versions and such - add ability to check if is_ipv6 already defined and also add back the field (obviously) * if data sources already have IPv6 true/false define, and to keep backwards compatible HELK versions and such - add ability to check if is_ipv6 already defined and also add back the field (obviously) * bit of reorder * typo * disable addition of cloned/original logs for now. because small builds this will make things explode * spell words * correction for - if data sources already have IPv6 true/false define, and to keep backwards compatible HELK versions and such - add ability to check if is_ipv6 already defined and also add back the field (obviously) * keep previous Sysmon field for QueryName that was converted to dns_query_name pre-OSSEM updates - for backwards compatibility, but tag/annotate that this is happening for future * bash/shell programmability and syntax/check and info logging (ie: no major functionality changes) * move kibana objects (ie: dashboards) to a folder that can allow to users add their own custom and allow script to run for them to import without full restart and such. as well as allow dashboards, visualizations, spaces, and all other objects to be differentiated between in the future * - (kibana) startup detection finally 100 - don't "log" passwords - use more funcitons for programmability - split scripts for ability to a) troubleshoot but more importantly b) allow users to load things without doing "everything" at docker startup/restart - above gives custom object/things for users - error checking for index patterns and other things.. ie: doesn't just blindly continue * always replica kibana index :) not many do..... also same for security index.. this is huge for multi-node * update kafka and scala * bash/shell programmability and syntax/check and info logging (ie: no major functionality changes) * logstash plugins update based on 30 day interval.. stores last install of plugin and checks diff between dates and then updates/NOT according to that logic. * give user ability to set number of logstash workers/processors in docker build AND if not set, added logic to set cores in a "intelligent" fashion * elastic 7.5. also updated logstash plugins as usual * elastic 7.5.1 also updated logstash plugins as usual * move more grains of sand to further building a beach #314 * update git ignores * update sigmac * further ecs > helk * continue HELK script updates * typo network_initiated * towards OSSEM * towards OSSEM add event id 5051 * towards OSSEM scheduled tasks * towards OSSEM directory service objects * typos * typos * add HELK logstash/ETL versioning * typo * finish old "user_target_" to new "target_user_" * 4690 HELK <> OSSEM * Audit Policy Change HELK <> OSSEM * typo * Audit Policy Change HELK <> OSSEM * typo * 4964 HELK <> OSSEM * 4777 is 4776 * event_id's 4774, 4775, 4882, 4883 * begin domain/hostname enrichments, additions, processing * Audit DPAPI Activity HELK <> OSSEM * forgot logging rotation * gitignore python and compiled code stuff * update gitignore * add mapping for some ECS fields that don't/won't get renamed * forward with ossem * copy for backwards compatibility * log level warn * add jdbc integration instead of coded + input + output. tweak guide * plugins for 7.6.1 * 7.6.1 elastic * update kafka version * update kafka containers * use nginx image, to get latest version - otherwise HELK nginx is years behind * latest otrf nginx image * latest otrf zookeeper image * latest otrf zookeeper image * latest otrf kafka image * latest otrf elastalert image * Merge branch 'master' of https://github.com/Cyb3rWard0g/HELK into updates_os_and_scripts � Conflicts: � docker/helk-logstash/plugins/Gemfile.lock * make build and elk version variable in script * merge updates_os_and_scripts * update nginx to 0.3.0 * update nginx to 0.3.0 * combine main windows event outputs into one - performance and continuity reasons * update plugins * only use .conf files, so users can disable/rename things without potentially trying to have logstash load it * update plugins * only use .conf files, so users can disable/rename things without potentially trying to have logstash load it * create zeek topic * elastic 7.6.2 * elastic 7.6.2 * update in logstash allows commented conf files again * update in logstash allows commented conf files again * zeek kafka topic * enable syslog ports for cisco asa and paloalto and one for "any" * Updated dashboards 50% * renamed dashboards * bash cleanup * fix possible potential issue of max bool max_clause_count * enable syslog ports for cisco asa and paloalto and one for "any" * Converted dashboards to Kibana 7 for further support * add keyword to IP - fixes #348 * hashes as keyword only * First version (test) of the new objects import script * replaced hash_.*.keyword in all dashboards * make more readable and geared towards HELK * first version new import/export script * removed old unused script * first working version of new scripts, remodeling needed * corrected a little issue + changed permissions * small fix in export script * new saved objects structure files * prepare for next release * added logstash cloud input plugins, update to logstash 7.6.2 and other plugins updated add es-rule backend cleanup keep original IP values forward ossem keep original hex values and forward ossem some more keep original IP values better fingerprinting forward schema/etl/ossem typo typo forward schema/etl/ossem winlogbeat ecs = deuces zeek filebeat to kafka example fix plugin file path test logic (files were remove in docker build :facepalm). also split input and output due to github size limit typos fix plugin file path test logic (files were remove in docker build :facepalm). also split input and output due to github size limit * add note for #350 * small modifications on the object files * not always original time, needs to be handled elsewhere as it varies per log * normalize / provide some logic to parse failure tagging. also handle sysmon logging differently than other windows logs since it has UTC * Removed old dashboards * add kafka partition/topic/other info * move zeek to its own kafka input * touchup winlogbeat 6. tweak nxlog stuff. normalize / provide some logic to parse failure tagging. also handle sysmon logging differently than other windows logs since it has UTC * move zeek/corelight index * forward * normalize / provide some logic to parse failure tagging. also handle sysmon logging differently than other windows logs since it has UTC * Added prototype code to control sigma rule updates * Added some missing code * Fully working version of the import mechanism * Added missing visualization * Updated saved objects * Updated export script * memory example for elasticsearch and logic in script * memory and logic in scripts * ports for eventually getting to cisco-ise logs * correct time field * cleanup * small tweak of shards * dynamix template fix, ordering regression in 7.x * add description * cleanup * index patterns * setup parse failing logic to prevent critical database errors (would apply to almost any database) * beat.name to host_name * set process time in UTC * keep json as is * basic zeek pipeline * zeek temp index * manual merge of #432 * manual merge of commit #9a74d4426b734f2f3cd292345c37f3315af6628a * a Co-authored-by: neu5ron <> Co-authored-by: troplolBE <[email protected]> Co-authored-by: Dev Dua <[email protected]> Co-authored-by: tcastron <[email protected]>
- Loading branch information
4 people
authored
May 5, 2020
1 parent
9a74d44
commit ebf25b5