diff --git a/docs/captagent_architecture.md b/docs/captagent_architecture.md new file mode 100644 index 0000000..fb51957 --- /dev/null +++ b/docs/captagent_architecture.md @@ -0,0 +1,80 @@ +## Abstract +CaptAgent has been around for a while and went through several redesign phases. + +The current version (6) has been completely redesigned from the ground up and ships with new internal architecture expanding its performance range and core capabilities to handle more network protocols, the ability to relay and aggregate traffic for remote agents, parse and handle RTCP-XR and RTP statistics, RPC centralized control and much more in a modular architecture. + +## Internal Architecture +##### Sockets, Pipes and Plans + +Captagent 6 features a fully modular design enabling users to design and program their packet capture and processing logic, leveraging specialized functionality provided on-demand via loadable dynamic modules. + +Core module types include: + +| type | description | +|:--|:--| +| socket | responsible for capturing ingress packets according to settings _(ie: PCAP, RAW)_ | +| protocol | responsible for processing/dissecting/parsing protocol data _(ie: SIP, RTCP)_ | +| transport | responsible for providing egress transport for generated data _(ie: HEP, JSON)_| +| function | responsible for providing additional functionality _(ie: database, etc)_ | + +Core modules are loaded via the main ```captagent.xml``` configuration file and can be easily concatenated to create multiple, independent capture chains: + + + +In the above example: + +```SOCKET``` -> ```PROFILE``` -> ```CAPTURE PLAN``` <--> ```MODULES (functions)``` + +------------- + + +###### CAPTURE CHAINS +For each chain, the logic and functionality is managed using a "capture-plan" which defines the behavior of the packet processing pipe. Capture plans are defined within the socket configuration alongside the general capture settings. An example for PCAP socket follows: +``` + + + + + + + portrange 5060-5091 + + +``` + +In the above example, packets captured by the socket would be processed by *capture-plan* in ```sip_capture_plan.cfg```: + +``` +# PCAP socket module +capture[pcap] { + # PROTO SIP module + # Ie: check source/destination IP/port, message size, etc. + if(msg_check("size", "100")) { + # Parse SIP Protocol + if(parse_sip()) { + # use HEP TRANSPORT module (transport_hep.xml) + if(!send_hep("hepsocket")) { + clog("ERROR", "Error sending HEP!"); + } + } + } +} +``` + +The capture-plan can access all functions provided by the loaded modules globally. + + +### Main Features +* Multiple incoming sockets (PCAP, RAW, PF_RING, RX-RING, FILE) +* Multiple outgoing types (HEP, JSON, CSV) +* HTTP JSON API for statistics, config changes etc +* RTCP-XR collector module +* RTCP output module (output in raw or json format) +* Capture scenario configuration (pseudo scripting via flex, bison) +* Call transaction tracking +* TCP/UDP reassembling and defragmentation. +* applying and change capture filter on demand +* LUA scripting (JITLua) (experimental) +* V7 Javascripting sandbox (experimental) +* SIPFIX Support (experimental) +* Websocket encapsulation support diff --git a/docs/captagent_capture_plans.md b/docs/captagent_capture_plans.md new file mode 100644 index 0000000..4bdee64 --- /dev/null +++ b/docs/captagent_capture_plans.md @@ -0,0 +1,52 @@ +### Capture Plans +Capture Plans are pipelines attached to capture sockets and utilized to define processing logic, +using global functions and methods exported by the loaded modules as defined in ```captagent.xml```. + +The socket is defined in each capture plan, supporting the following `capture[...]` types: + +* pcap +* tzsp +* collector + +##### Example Configuration Chain +```socket_pcap``` -> ```{profile}``` -> ```capture_plan``` + +##### Example Pointer +/usr/local/etc/captagent/socket_pcap.xml +``` + + + + + + + + + portrange 5060-5091 + + + +``` + +##### Example Capture Plan +/usr/local/etc/captagent/captureplans/sip_capture_plan.cfg +``` +capture[pcap] { + # here we can check source/destination IP/port, message size + if(msg_check("size", "100")) { + #Do parsing + if(parse_sip()) { + # Drop unwanted methods + if(sip_check("rmethod","OPTIONS") || sip_check("rmethod","NOTIFY")) { + drop; + } + + #Multiple profiles can be defined in transport_hep.xml + if(!send_hep("hepsocket")) { + clog("ERROR", "Error sending HEP!!!!"); + } + } + } + drop; +} +``` diff --git a/docs/captagent_configuration.md b/docs/captagent_configuration.md new file mode 100644 index 0000000..1e606df --- /dev/null +++ b/docs/captagent_configuration.md @@ -0,0 +1,68 @@ +## CaptAgent: Configuration + +This section provides guidance to configure Captagent and its core modules on your system. + +#### Configuration Logic + +Understanding of CaptAgent configuration structure is key - read [this](https://github.com/sipcapture/captagent/wiki/About#sockets-pipes-and-plans) section carefully! + + +------------- + +#### Basic Configuration + +The following are the default file locations _(unless otherwise specified during configuration)_: + +* Configuration: ```/usr/local/captagent/etc``` +* Capture Plans: ```/usr/local/captagent/etc/captureplans``` +* Modules: ```/usr/local/captagent/lib/modules``` + + +##### Configuration tree +The default directory should contains the following using default profiles and plans: +``` +captagent.xml +captureplans/ + sip_capture_plan.cfg + rtcp_capture_plan.cfg + rtcpxr_capture_plan.cfg + tzsp_capture_plan.cfg +protocol_rtcp.xml +protocol_sip.xml +protocol_rtcpxr.xml +protocol_diameter.xml +socket_pcap.xml +socket_tzsp.xml +socket_collector.xml +transport_hep.xml +output_json.xml + +``` + +##### Main Configuration +To begin, edit and validate the configuration and the module paths in ```/usr/local/etc/captagent/captagent.xml``` to match your actual captagent config/lib path: + +``` + + + + + + + + + + + + + + + + +``` + +### Next: + +* Configure [Socket Modules](https://github.com/sipcapture/captagent/wiki/Socket-Modules) +* Configure [Transport Modules](https://github.com/sipcapture/captagent/wiki/Transport-Modules) +* Configure [Capture Plans](https://github.com/sipcapture/captagent/wiki/Capture-Plans) diff --git a/docs/captagent_init_scripts.md b/docs/captagent_init_scripts.md new file mode 100644 index 0000000..d4d2bee --- /dev/null +++ b/docs/captagent_init_scripts.md @@ -0,0 +1,52 @@ +# Captagent Init Scripts + +The init.d script can be used to start/stop captagent in a nicer way. + +### Installation & Configuration + +A sample of init.d script for captagent is provided at: +``` +/usr/src/captagent/init/deb/debian/captagent.init +``` + +Just copy the init file to ```/etc/init.d/captagent``` and change the permisions: + +``` + cp /usr/src/captagent/init/deb/debian/captagent.init /etc/init.d/captagent + chmod 755 /etc/init.d/captagent +``` + +then edit the file updating the $DAEMON and $CFGFILE values: + +``` + DAEMON=/usr/local/captagent/bin/captagent + CFGFILE=/usr/local/captagent/etc/captagent/captagent.xml +``` + +You need also setup a configuration file in the /etc/default/ directory: +``` + cp /usr/src/captagent/init/deb/debian/captagent.default /etc/default/captagent +``` + +When using systemd _(we all wish we were not)_ the service file might be required: +``` + cp /usr/src/captagent/init/deb/debian/captagent.service /etc/systemd/system/captagent.service +``` + +Once installed and before using, edit the default file to reflect your captagent path: +``` + RUN_CAPTAGENT=yes + CFGFILE=/usr/local/captagent/etc/captagent/captagent.xml +``` + + +### Automatic Startup +To execute automatically at startup: +``` + update-rc.d captagent defaults +``` + +To exclude the script from startup: +``` + update-rc.d -f captagent remove +``` diff --git a/docs/captagent_installtion.md b/docs/captagent_installtion.md new file mode 100644 index 0000000..6b928b8 --- /dev/null +++ b/docs/captagent_installtion.md @@ -0,0 +1,83 @@ +## CaptAgent: Installation + +This section provides guidance to download the latest code from our repository and compile it on your system. + + +### Requirements + +* Captagent 6.2+ requires ```libuv``` + + If your system does not provide ```libuv``` and ```libuv-dev```, please install from our +[repository](https://github.com/sipcapture/captagent/tree/master/dependency) or compile it from [source]( https://github.com/libuv/libuv/releases) + +##### Operating Systems +###### Debian 10 (buster): +``` +apt-get install libexpat-dev libpcap-dev libjson-c-dev libtool automake flex bison libgcrypt-dev libuv1-dev libpcre3-dev libfl-dev + +``` +###### Debian 9 (stretch): +``` +apt-get install libexpat1-dev libpcap-dev libjson-c-dev libtool automake flex bison libgcrypt11-dev libuv1-dev libpcre3-dev libfl-dev + +``` +###### Debian 8 (jessie): +``` +apt-get install libexpat-dev libpcap-dev libjson0-dev libtool automake flex bison libuv-dev libgcrypt11-dev libfl-dev +``` +###### Debian 7 (wheezy): +``` +wget https://github.com/sipcapture/captagent/raw/master/dependency/debian/wheezy/libuv_1.8.0-2_amd64.deb +dpkg -i libuv_1.8.0-2_amd64.deb + +apt-get install libexpat-dev libpcap-dev libjson0-dev libtool automake flex bison + +``` + +###### CentOS 7: +``` +yum -y install json-c-devel expat-devel libpcap-devel flex-devel automake libtool bison libuv-devel flex +``` + +###### CentOS 6: +``` +rpm -i https://github.com/sipcapture/captagent/raw/master/dependency/centos/6/libuv-1.8.0-1.el6.x86_64.rpm +rpm -i https://github.com/sipcapture/captagent/raw/master/dependency/centos/6/libuv-devel-1.8.0-1.el6.x86_64.rpm +yum -y install json-c-devel expat-devel libpcap-devel pcre-devel flex-devel automake libtool bison flex +``` + + + +### Clone & Compile +``` + cd /usr/src + git clone https://github.com/sipcapture/captagent.git captagent + cd captagent + ./build.sh + ./configure + make && make install +``` + +#### Build Options +| Name | Configure Flag | Libraries | +|--- |--- |--- | +| HEP Compression | --enable-compression | | +| IPv6 Support | --enable-ipv6 | | +| PCRE Support | --enable-pcre | libpcre | +| SSL Support | --enable-ssl | openssl | +| TLS Support | --enable-tls | libgcrypt20 openssl | +| MySQL Support | --enable-mysql | libmysqlclient | +| Redis Support | --enable-redis | libhiredis | + +-------------- + +#### TLS Support (experimental) + +To compile and enable TLS decryption features, please check the dedicated Wiki page. + +-------------- + + +Congratulations! You just installed your first basic instance of CaptAgent 6! + +#### Next: [Configure CaptAgent 6](https://github.com/sipcapture/captagent/wiki/Configuration) diff --git a/docs/captagent_nat.md b/docs/captagent_nat.md new file mode 100644 index 0000000..89e562a --- /dev/null +++ b/docs/captagent_nat.md @@ -0,0 +1,17 @@ +## NAT +When the SDP details are reporting masqueraded or private IP ranges, the ```nat-mode``` switch can be enabled to force Captagent to use the received address for matching media sessions. Settings enabled by module ```database_hash``` should be applied in file ```database_hash.xml``` : + +``` + + + + + + + + + + + + +``` diff --git a/docs/captagent_protocol_modules.md b/docs/captagent_protocol_modules.md new file mode 100644 index 0000000..0304eea --- /dev/null +++ b/docs/captagent_protocol_modules.md @@ -0,0 +1,53 @@ +### Protocol Modules +Protocol modules are used by captagent to process specific protocol pipelines. +Protocol modules are loaded at startup by the ```captagent.xml``` general configuration. + + +``` + + + + ... + + + ... + + + +``` + +### Functions to use in modules +#### protocol_sip + +**Note:** This list is probably incomplete by the time you read it. You can get all functions available here: [protocol_sip.c](https://github.com/sipcapture/captagent/blob/master/src/modules/protocol/sip/protocol_sip.c) + +| Function | Description | +| ---| --- | +| `sip_check("method", "INVITE")` | only INVITE requests | +| `sip_check("rmethod", "REGISTER")` | REGISTER requests and answers (CSeq) | +| `sip_check("response", "200")` | only 200 responses | +| `sip_check("response_gt", "400")` | only responses >= 400 | +| `sip_check("response_lt", "199")` | only responses <= 199 | +| `sip_check("from_user_suffix", "tail")` | check if from_user has "tail" as suffix | +| `sip_check("to_user_suffix", "tail")` | check if to_user has "tail" as suffix | +| `sip_check("from_user_prefix", "head")` | check if from_user has "head" as prefix | +| `sip_check("to_user_prefix", "head")` | check if to_user has "head" as prefix | +| `msg_check("size", "100")` | only packets > 100 bytes | +| `msg_check("src_ip", "10.0.0.1")` | only packets from 10.0.0.1 | +| `msg_check("destination_ip", "10.0.0.1")` | only packets to 10.0.0.1 | +| `msg_check("src_port", "10000")` | only packets from port 10000 | +| `msg_check("src_port_gt", "10000")` | only packets from ports >= 10000 | +| `msg_check("src_port_lt", "10000")` | only packets from ports <= 10000 | +| `msg_check("dst_port", "10000")` | only packets to port 10000 | +| `msg_check("dst_port_gt", "10000")` | only packets to ports >= 10000 | +| `msg_check("dst_port_lt", "10000")` | only packets to ports <= 10000 | +| `sip_is_method()` | check whether this is a request | +| `parse_sip()` | check for valid SIP message | +| `light_parse_sip()` | check whether Call-ID Header is valid | +| `parse_full_sip()` | check for valid SIP and parse Contact and Via header | +| `sip_has_sdp()` | check whether sip packet has sdp | +| `is_flag_set("1", "0")` | check whether flag named "1" has value "0" | +| `send_reply("200", "OK")` | generate a reply from Captagent | +| `clog("ERR", "foo")` | generate error log line | +| `clog("NOTICE", "foo")` | generate notice log line | +| `clog("foo", "bar")` | generate debug log line | diff --git a/docs/captagent_readme.md b/docs/captagent_readme.md new file mode 100644 index 0000000..d506426 --- /dev/null +++ b/docs/captagent_readme.md @@ -0,0 +1,654 @@ +## Abstract +CaptAgent has been around for a while and went through several redesign phases. + +The current version (6) has been completely redesigned from the ground up and ships with new internal architecture expanding its performance range and core capabilities to handle more network protocols, the ability to relay and aggregate traffic for remote agents, parse and handle RTCP-XR and RTP statistics, RPC centralized control and much more in a modular architecture. + +## Internal Architecture +##### Sockets, Pipes and Plans + +Captagent 6 features a fully modular design enabling users to design and program their packet capture and processing logic, leveraging specialized functionality provided on-demand via loadable dynamic modules. + +Core module types include: + +| type | description | +|:--|:--| +| socket | responsible for capturing ingress packets according to settings _(ie: PCAP, RAW)_ | +| protocol | responsible for processing/dissecting/parsing protocol data _(ie: SIP, RTCP)_ | +| transport | responsible for providing egress transport for generated data _(ie: HEP, JSON)_| +| function | responsible for providing additional functionality _(ie: database, etc)_ | + +Core modules are loaded via the main ```captagent.xml``` configuration file and can be easily concatenated to create multiple, independent capture chains: + + + +In the above example: + +```SOCKET``` -> ```PROFILE``` -> ```CAPTURE PLAN``` <--> ```MODULES (functions)``` + +------------- + + +###### CAPTURE CHAINS +For each chain, the logic and functionality is managed using a "capture-plan" which defines the behavior of the packet processing pipe. Capture plans are defined within the socket configuration alongside the general capture settings. An example for PCAP socket follows: +``` + + + + + + + portrange 5060-5091 + + +``` + +In the above example, packets captured by the socket would be processed by *capture-plan* in ```sip_capture_plan.cfg```: + +``` +# PCAP socket module +capture[pcap] { + # PROTO SIP module + # Ie: check source/destination IP/port, message size, etc. + if(msg_check("size", "100")) { + # Parse SIP Protocol + if(parse_sip()) { + # use HEP TRANSPORT module (transport_hep.xml) + if(!send_hep("hepsocket")) { + clog("ERROR", "Error sending HEP!"); + } + } + } +} +``` + +The capture-plan can access all functions provided by the loaded modules globally. + + +### Main Features +* Multiple incoming sockets (PCAP, RAW, PF_RING, RX-RING, FILE) +* Multiple outgoing types (HEP, JSON, CSV) +* HTTP JSON API for statistics, config changes etc +* RTCP-XR collector module +* RTCP output module (output in raw or json format) +* Capture scenario configuration (pseudo scripting via flex, bison) +* Call transaction tracking +* TCP/UDP reassembling and defragmentation. +* applying and change capture filter on demand +* LUA scripting (JITLua) (experimental) +* V7 Javascripting sandbox (experimental) +* SIPFIX Support (experimental) +* Websocket encapsulation support + +## CaptAgent: Installation + +This section provides guidance to download the latest code from our repository and compile it on your system. + + +### Requirements + +* Captagent 6.2+ requires ```libuv``` + + If your system does not provide ```libuv``` and ```libuv-dev```, please install from our +[repository](https://github.com/sipcapture/captagent/tree/master/dependency) or compile it from [source]( https://github.com/libuv/libuv/releases) + +##### Operating Systems +###### Debian 10 (buster): +``` +apt-get install libexpat-dev libpcap-dev libjson-c-dev libtool automake flex bison libgcrypt-dev libuv1-dev libpcre3-dev libfl-dev + +``` +###### Debian 9 (stretch): +``` +apt-get install libexpat1-dev libpcap-dev libjson-c-dev libtool automake flex bison libgcrypt11-dev libuv1-dev libpcre3-dev libfl-dev + +``` +###### Debian 8 (jessie): +``` +apt-get install libexpat-dev libpcap-dev libjson0-dev libtool automake flex bison libuv-dev libgcrypt11-dev libfl-dev +``` +###### Debian 7 (wheezy): +``` +wget https://github.com/sipcapture/captagent/raw/master/dependency/debian/wheezy/libuv_1.8.0-2_amd64.deb +dpkg -i libuv_1.8.0-2_amd64.deb + +apt-get install libexpat-dev libpcap-dev libjson0-dev libtool automake flex bison + +``` + +###### CentOS 7: +``` +yum -y install json-c-devel expat-devel libpcap-devel flex-devel automake libtool bison libuv-devel flex +``` + +###### CentOS 6: +``` +rpm -i https://github.com/sipcapture/captagent/raw/master/dependency/centos/6/libuv-1.8.0-1.el6.x86_64.rpm +rpm -i https://github.com/sipcapture/captagent/raw/master/dependency/centos/6/libuv-devel-1.8.0-1.el6.x86_64.rpm +yum -y install json-c-devel expat-devel libpcap-devel pcre-devel flex-devel automake libtool bison flex +``` + + + +### Clone & Compile +``` + cd /usr/src + git clone https://github.com/sipcapture/captagent.git captagent + cd captagent + ./build.sh + ./configure + make && make install +``` + +#### Build Options +| Name | Configure Flag | Libraries | +|--- |--- |--- | +| HEP Compression | --enable-compression | | +| IPv6 Support | --enable-ipv6 | | +| PCRE Support | --enable-pcre | libpcre | +| SSL Support | --enable-ssl | openssl | +| TLS Support | --enable-tls | libgcrypt20 openssl | +| MySQL Support | --enable-mysql | libmysqlclient | +| Redis Support | --enable-redis | libhiredis | + +-------------- + +#### TLS Support (experimental) + +To compile and enable TLS decryption features, please check the dedicated Wiki page. + +-------------- + + +Congratulations! You just installed your first basic instance of CaptAgent 6! + +#### Next: [Configure CaptAgent 6](https://github.com/sipcapture/captagent/wiki/Configuration) + + +# Captagent Init Scripts + +The init.d script can be used to start/stop captagent in a nicer way. + +### Installation & Configuration + +A sample of init.d script for captagent is provided at: +``` +/usr/src/captagent/init/deb/debian/captagent.init +``` + +Just copy the init file to ```/etc/init.d/captagent``` and change the permisions: + +``` + cp /usr/src/captagent/init/deb/debian/captagent.init /etc/init.d/captagent + chmod 755 /etc/init.d/captagent +``` + +then edit the file updating the $DAEMON and $CFGFILE values: + +``` + DAEMON=/usr/local/captagent/bin/captagent + CFGFILE=/usr/local/captagent/etc/captagent/captagent.xml +``` + +You need also setup a configuration file in the /etc/default/ directory: +``` + cp /usr/src/captagent/init/deb/debian/captagent.default /etc/default/captagent +``` + +When using systemd _(we all wish we were not)_ the service file might be required: +``` + cp /usr/src/captagent/init/deb/debian/captagent.service /etc/systemd/system/captagent.service +``` + +Once installed and before using, edit the default file to reflect your captagent path: +``` + RUN_CAPTAGENT=yes + CFGFILE=/usr/local/captagent/etc/captagent/captagent.xml +``` + + +### Automatic Startup +To execute automatically at startup: +``` + update-rc.d captagent defaults +``` + +To exclude the script from startup: +``` + update-rc.d -f captagent remove +``` + + +## CaptAgent: Configuration + +This section provides guidance to configure Captagent and its core modules on your system. + +#### Configuration Logic + +Understanding of CaptAgent configuration structure is key - read [this](https://github.com/sipcapture/captagent/wiki/About#sockets-pipes-and-plans) section carefully! + + +------------- + +#### Basic Configuration + +The following are the default file locations _(unless otherwise specified during configuration)_: + +* Configuration: ```/usr/local/captagent/etc``` +* Capture Plans: ```/usr/local/captagent/etc/captureplans``` +* Modules: ```/usr/local/captagent/lib/modules``` + + +##### Configuration tree +The default directory should contains the following using default profiles and plans: +``` +captagent.xml +captureplans/ + sip_capture_plan.cfg + rtcp_capture_plan.cfg + rtcpxr_capture_plan.cfg + tzsp_capture_plan.cfg +protocol_rtcp.xml +protocol_sip.xml +protocol_rtcpxr.xml +protocol_diameter.xml +socket_pcap.xml +socket_tzsp.xml +socket_collector.xml +transport_hep.xml +output_json.xml + +``` + +##### Main Configuration +To begin, edit and validate the configuration and the module paths in ```/usr/local/etc/captagent/captagent.xml``` to match your actual captagent config/lib path: + +``` + + + + + + + + + + + + + + + + +``` + +### Next: + +* Configure [Socket Modules](https://github.com/sipcapture/captagent/wiki/Socket-Modules) +* Configure [Transport Modules](https://github.com/sipcapture/captagent/wiki/Transport-Modules) +* Configure [Capture Plans](https://github.com/sipcapture/captagent/wiki/Capture-Plans) + +### Socket Modules +Socket modules are used by captagent to capture packets from the system available interfaces. By default, the ```pcap``` socket is enabled. + +#### PCAP Socket +Within each socket, multiple profiles can be defined and configured as pipelines to process captured packets. By default the ```socketspcap_sip``` pipeline is enabled defining the interface and ports utilized for capturing and processing ```SIP``` packets via the dedicated [Capture Plan](https://github.com/sipcapture/captagent/wiki/Capture-Plans) + +``` + + + + + + + + + + + portrange 5060-5091 + + + +``` +### GRE-ERSPAN Example +When capturing GRE-ERSPAN Encapsulated traffic this needs to be setup +``` + + + proto GRE and len > 50 + +``` + +### Websocket encapsulation layer +Sometimes WebSocket subprotocol it is used as a reliable transport mechanism between Session Initiation Protocol (SIP) entities to enable use of SIP in web-oriented deployments. + +Captagent provide additional parsing for Websocket layer on TCP by enable the `websocket-detection` param (default is false) +``` + +``` + +### IP-to-IP encapsulation layer +`IP-to-IP` is an IP tunneling protocol that encapsulates one IP packet in another IP packet. To encapsulate an IP packet in another IP packet, an outer header is added with source IP, the entry point of the tunnel, and the destination IP, the exit point of the tunnel. + +Captagent now has the possibility to check and correctly parse this tunnel by adding a new `capture-filter` on portrange `5060-590`: +``` + +``` +This filter perform also the internal check on portrange 5060-5090, so in the BPF filter you can simply add this +``` + + ip + +``` + +### Transport Modules +Transport modules are used by captagent to send packets and reports to collectors using different methods and protocols. By default, the *HEP* method is activated. + +#### HEP +The *HEP* module is used to define a HEP collector for captured packets, such as [HOMER](http://github.com/sipcapture/homer/wiki). + +The critical parameters are: + +* capture-host: defines the IP/hostname of the collector +* capture-port: defines the PORT to deliver HEP packets at the collector +* capture-proto: defines the transport protocol for HEP packets [udp/tcp] +* capture-id: defines a unique delivery HEP-ID to be used for filtering + +NOTE: Parameters such as ```capt-password``` and ```payload-compression``` are currently only used in advanced deployments and can be ignored for standard setups. + +##### Example HEP Configuration: +``` + + + + + + + + + + + + + + + + +``` + + + +#### JSON +The *JSON* module is used to define a collector used by captagent to deliver JSON messages and statistics, such as [HOMER](http://github.com/sipcapture/homer/wiki). + +The critical parameters are: + +* capture-host: defines the IP/hostname of the JSON API +* capture-port: defines the PORT for the JSON API +* capture-proto: defines the transport protocol for HEP packets [udp|tcp] +* capture-id: defines a unique delivery HEP-ID to be used for filtering +* payload-send: defines if full packet payload should be included in API POST [true|false] + +##### Example JSON Configuration: + +``` + + + + + + + + + + + + + + + +``` + + +### Protocol Modules +Protocol modules are used by captagent to process specific protocol pipelines. +Protocol modules are loaded at startup by the ```captagent.xml``` general configuration. + + +``` + + + + ... + + + ... + + + +``` + +### Functions to use in modules +#### protocol_sip + +**Note:** This list is probably incomplete by the time you read it. You can get all functions available here: [protocol_sip.c](https://github.com/sipcapture/captagent/blob/master/src/modules/protocol/sip/protocol_sip.c) + +| Function | Description | +| ---| --- | +| `sip_check("method", "INVITE")` | only INVITE requests | +| `sip_check("rmethod", "REGISTER")` | REGISTER requests and answers (CSeq) | +| `sip_check("response", "200")` | only 200 responses | +| `sip_check("response_gt", "400")` | only responses >= 400 | +| `sip_check("response_lt", "199")` | only responses <= 199 | +| `sip_check("from_user_suffix", "tail")` | check if from_user has "tail" as suffix | +| `sip_check("to_user_suffix", "tail")` | check if to_user has "tail" as suffix | +| `sip_check("from_user_prefix", "head")` | check if from_user has "head" as prefix | +| `sip_check("to_user_prefix", "head")` | check if to_user has "head" as prefix | +| `msg_check("size", "100")` | only packets > 100 bytes | +| `msg_check("src_ip", "10.0.0.1")` | only packets from 10.0.0.1 | +| `msg_check("destination_ip", "10.0.0.1")` | only packets to 10.0.0.1 | +| `msg_check("src_port", "10000")` | only packets from port 10000 | +| `msg_check("src_port_gt", "10000")` | only packets from ports >= 10000 | +| `msg_check("src_port_lt", "10000")` | only packets from ports <= 10000 | +| `msg_check("dst_port", "10000")` | only packets to port 10000 | +| `msg_check("dst_port_gt", "10000")` | only packets to ports >= 10000 | +| `msg_check("dst_port_lt", "10000")` | only packets to ports <= 10000 | +| `sip_is_method()` | check whether this is a request | +| `parse_sip()` | check for valid SIP message | +| `light_parse_sip()` | check whether Call-ID Header is valid | +| `parse_full_sip()` | check for valid SIP and parse Contact and Via header | +| `sip_has_sdp()` | check whether sip packet has sdp | +| `is_flag_set("1", "0")` | check whether flag named "1" has value "0" | +| `send_reply("200", "OK")` | generate a reply from Captagent | +| `clog("ERR", "foo")` | generate error log line | +| `clog("NOTICE", "foo")` | generate notice log line | +| `clog("foo", "bar")` | generate debug log line | + + +## WARNING! +*This is a work in Progress! Issues must be raised w/ full details + PCAP to reproduce.* + +## Captagent TLS +When provided with the appropriate keying material, the TCP protocol module can attempt decryption TLS connections and display the application data traffic in real-time. + +##### How decryption work +Internally, the work is divided to resolve the _asymmetric_ and _symmetric_ encryption. +1. The _encrypted pre-master secret_ is captured during the key exchange between Client and Server, and decrypted to obtain a **Pre-Master Secret (PMS)**. +2. The **Master Secret (MS)** is recreated thanks to _PMS_, and used for the symmetric encryption of the keys. The _MS_ is necessary to build the _keys block_, that is dissected and leveraged to regenerate the required decryption keys used in the asymmetric part of the encryption. +3. These keys (**MACs** or **IVs**, and **Write** keys) perform the final decryption of the _real_ data. + +Decryption can only be attempted for scenarios including the full client-server handshake. + +##### SUPPORTED: +* TLS_RSA_WITH_AES_256_GCM_SHA384 +* TLS_RSA_WITH_AES_128_GCM_SHA256 + + * RSA_PKCS1_PADDING + +##### UNSUPPORTABLE: +* TLS_DH* (Diffie-Hellman) + +------------- + +### Requirements + +* RSA PRIVATE KEY (max size _2048_ bit) +* libssl-dev +* libgcrypt 1.8 or higher + +#### Debian +``` +apt-get install -y libgcrypt20 libgcrypt20-dev libssl-dev +``` + +### Compile & Install +``` +./build.sh +./configure --enable-tls --enable-ssl +make && sudo make install +``` + +### Configure + +#### captagent.xml +Enable loading for ```protocol_tls``` under `/usr/local/captagent/etc/captagent/captagent.xml` +``` + [...] + <---- our module + + [...] +``` +Remember to put `` **before** `` in order to activate the protocol functions in the correct way. + +#### socket_pcap +Configure ```socket_pcap``` and enable the TLS profile block using the proper port or portrange: +``` + + + + + + + + + tcp port 5061 + + + +``` + +#### protocol_tls +Configure ```protocol_tls``` with the full path to the required ```private-key``` to decrypt RSA/TLS traffic: +``` + + + + + + + + + + + + +``` + +_That's all!_ + +### Dev Demo +Development [demo setup](https://github.com/lmangani/docker-opensips-hepclient/tree/tls) leverages the default [OpenSIPS rootCA](https://github.com/OpenSIPS/opensips/tree/master/etc/tls/rootCA). + +##### TODO +For a complete scenarios it will be desirable to have other supported cipher suites for the RSA family + +### Troubleshooting +The code is relatively fresh and might be impacted by TCP reassembly and many other challenging scenarios requiring fine-tuning. If your traffic is not decoded or you receive TLS parsing errors for traffic you can decode with Wireshark, please open and issue w/ full details + PCAP to reproduce. + + +### Capture Plans +Capture Plans are pipelines attached to capture sockets and utilized to define processing logic, +using global functions and methods exported by the loaded modules as defined in ```captagent.xml```. + +The socket is defined in each capture plan, supporting the following `capture[...]` types: + +* pcap +* tzsp +* collector + +##### Example Configuration Chain +```socket_pcap``` -> ```{profile}``` -> ```capture_plan``` + +##### Example Pointer +/usr/local/etc/captagent/socket_pcap.xml +``` + + + + + + + + + portrange 5060-5091 + + + +``` + +##### Example Capture Plan +/usr/local/etc/captagent/captureplans/sip_capture_plan.cfg +``` +capture[pcap] { + # here we can check source/destination IP/port, message size + if(msg_check("size", "100")) { + #Do parsing + if(parse_sip()) { + # Drop unwanted methods + if(sip_check("rmethod","OPTIONS") || sip_check("rmethod","NOTIFY")) { + drop; + } + + #Multiple profiles can be defined in transport_hep.xml + if(!send_hep("hepsocket")) { + clog("ERROR", "Error sending HEP!!!!"); + } + } + } + drop; +} +``` + + +In order for captagent to correlate and pair SIP SDP Sessions to RTCP packets, the following configuration block should be uncommented in the ```sip_capture_plan.cfg```: +``` +if(sip_has_sdp()) + { + #Activate it for RTCP checks + if(!check_rtcp_ipport()) + { + clog("ERROR", "SDP SESSION ALREADY EXISTS!"); + } + } +``` + +This will produce a match for the ```is_rtcp_exist()``` function in the ```rtcp_capture_plan.cfg``` scope to inject a correlation ID and return results in HOMER. + +## NAT +When the SDP details are reporting masqueraded or private IP ranges, the ```nat-mode``` switch can be enabled to force Captagent to use the received address for matching media sessions. Settings enabled by module ```database_hash``` should be applied in file ```database_hash.xml``` : + +``` + + + + + + + + + + + + +``` + + diff --git a/docs/captagent_rtcp.md b/docs/captagent_rtcp.md new file mode 100644 index 0000000..2cd3d8f --- /dev/null +++ b/docs/captagent_rtcp.md @@ -0,0 +1,13 @@ +In order for captagent to correlate and pair SIP SDP Sessions to RTCP packets, the following configuration block should be uncommented in the ```sip_capture_plan.cfg```: +``` +if(sip_has_sdp()) + { + #Activate it for RTCP checks + if(!check_rtcp_ipport()) + { + clog("ERROR", "SDP SESSION ALREADY EXISTS!"); + } + } +``` + +This will produce a match for the ```is_rtcp_exist()``` function in the ```rtcp_capture_plan.cfg``` scope to inject a correlation ID and return results in HOMER. diff --git a/docs/captagent_socket_modules.md b/docs/captagent_socket_modules.md new file mode 100644 index 0000000..3f66775 --- /dev/null +++ b/docs/captagent_socket_modules.md @@ -0,0 +1,52 @@ +### Socket Modules +Socket modules are used by captagent to capture packets from the system available interfaces. By default, the ```pcap``` socket is enabled. + +#### PCAP Socket +Within each socket, multiple profiles can be defined and configured as pipelines to process captured packets. By default the ```socketspcap_sip``` pipeline is enabled defining the interface and ports utilized for capturing and processing ```SIP``` packets via the dedicated [Capture Plan](https://github.com/sipcapture/captagent/wiki/Capture-Plans) + +``` + + + + + + + + + + + portrange 5060-5091 + + + +``` +### GRE-ERSPAN Example +When capturing GRE-ERSPAN Encapsulated traffic this needs to be setup +``` + + + proto GRE and len > 50 + +``` + +### Websocket encapsulation layer +Sometimes WebSocket subprotocol it is used as a reliable transport mechanism between Session Initiation Protocol (SIP) entities to enable use of SIP in web-oriented deployments. + +Captagent provide additional parsing for Websocket layer on TCP by enable the `websocket-detection` param (default is false) +``` + +``` + +### IP-to-IP encapsulation layer +`IP-to-IP` is an IP tunneling protocol that encapsulates one IP packet in another IP packet. To encapsulate an IP packet in another IP packet, an outer header is added with source IP, the entry point of the tunnel, and the destination IP, the exit point of the tunnel. + +Captagent now has the possibility to check and correctly parse this tunnel by adding a new `capture-filter` on portrange `5060-590`: +``` + +``` +This filter perform also the internal check on portrange 5060-5090, so in the BPF filter you can simply add this +``` + + ip + +``` diff --git a/docs/captagent_tls.md b/docs/captagent_tls.md new file mode 100644 index 0000000..74c6537 --- /dev/null +++ b/docs/captagent_tls.md @@ -0,0 +1,99 @@ +## WARNING! +*This is a work in Progress! Issues must be raised w/ full details + PCAP to reproduce.* + +## Captagent TLS +When provided with the appropriate keying material, the TCP protocol module can attempt decryption TLS connections and display the application data traffic in real-time. + +##### How decryption work +Internally, the work is divided to resolve the _asymmetric_ and _symmetric_ encryption. +1. The _encrypted pre-master secret_ is captured during the key exchange between Client and Server, and decrypted to obtain a **Pre-Master Secret (PMS)**. +2. The **Master Secret (MS)** is recreated thanks to _PMS_, and used for the symmetric encryption of the keys. The _MS_ is necessary to build the _keys block_, that is dissected and leveraged to regenerate the required decryption keys used in the asymmetric part of the encryption. +3. These keys (**MACs** or **IVs**, and **Write** keys) perform the final decryption of the _real_ data. + +Decryption can only be attempted for scenarios including the full client-server handshake. + +##### SUPPORTED: +* TLS_RSA_WITH_AES_256_GCM_SHA384 +* TLS_RSA_WITH_AES_128_GCM_SHA256 + + * RSA_PKCS1_PADDING + +##### UNSUPPORTABLE: +* TLS_DH* (Diffie-Hellman) + +------------- + +### Requirements + +* RSA PRIVATE KEY (max size _2048_ bit) +* libssl-dev +* libgcrypt 1.8 or higher + +#### Debian +``` +apt-get install -y libgcrypt20 libgcrypt20-dev libssl-dev +``` + +### Compile & Install +``` +./build.sh +./configure --enable-tls --enable-ssl +make && sudo make install +``` + +### Configure + +#### captagent.xml +Enable loading for ```protocol_tls``` under `/usr/local/captagent/etc/captagent/captagent.xml` +``` + [...] + <---- our module + + [...] +``` +Remember to put `` **before** `` in order to activate the protocol functions in the correct way. + +#### socket_pcap +Configure ```socket_pcap``` and enable the TLS profile block using the proper port or portrange: +``` + + + + + + + + + tcp port 5061 + + + +``` + +#### protocol_tls +Configure ```protocol_tls``` with the full path to the required ```private-key``` to decrypt RSA/TLS traffic: +``` + + + + + + + + + + + + +``` + +_That's all!_ + +### Dev Demo +Development [demo setup](https://github.com/lmangani/docker-opensips-hepclient/tree/tls) leverages the default [OpenSIPS rootCA](https://github.com/OpenSIPS/opensips/tree/master/etc/tls/rootCA). + +##### TODO +For a complete scenarios it will be desirable to have other supported cipher suites for the RSA family + +### Troubleshooting +The code is relatively fresh and might be impacted by TCP reassembly and many other challenging scenarios requiring fine-tuning. If your traffic is not decoded or you receive TLS parsing errors for traffic you can decode with Wireshark, please open and issue w/ full details + PCAP to reproduce. diff --git a/docs/captagent_transport_modules.md b/docs/captagent_transport_modules.md new file mode 100644 index 0000000..db9d1a6 --- /dev/null +++ b/docs/captagent_transport_modules.md @@ -0,0 +1,67 @@ +### Transport Modules +Transport modules are used by captagent to send packets and reports to collectors using different methods and protocols. By default, the *HEP* method is activated. + +#### HEP +The *HEP* module is used to define a HEP collector for captured packets, such as [HOMER](http://github.com/sipcapture/homer/wiki). + +The critical parameters are: + +* capture-host: defines the IP/hostname of the collector +* capture-port: defines the PORT to deliver HEP packets at the collector +* capture-proto: defines the transport protocol for HEP packets [udp/tcp] +* capture-id: defines a unique delivery HEP-ID to be used for filtering + +NOTE: Parameters such as ```capt-password``` and ```payload-compression``` are currently only used in advanced deployments and can be ignored for standard setups. + +##### Example HEP Configuration: +``` + + + + + + + + + + + + + + + + +``` + + + +#### JSON +The *JSON* module is used to define a collector used by captagent to deliver JSON messages and statistics, such as [HOMER](http://github.com/sipcapture/homer/wiki). + +The critical parameters are: + +* capture-host: defines the IP/hostname of the JSON API +* capture-port: defines the PORT for the JSON API +* capture-proto: defines the transport protocol for HEP packets [udp|tcp] +* capture-id: defines a unique delivery HEP-ID to be used for filtering +* payload-send: defines if full packet payload should be included in API POST [true|false] + +##### Example JSON Configuration: + +``` + + + + + + + + + + + + + + + +``` diff --git a/docs/heplify_installation_usage.md b/docs/heplify_installation_usage.md new file mode 100644 index 0000000..4b3d156 --- /dev/null +++ b/docs/heplify_installation_usage.md @@ -0,0 +1,262 @@ + + + + + + +heplify is captagents little brother, optimized for speed and simplicity. It's a single binary which you can run +on Linux, ARM, MIPS, Windows to capture IPv4 or IPv6 packets and send them to Homer. Heplify is able to send +SIP, correlated RTCP, RTCPXR, DNS, Logs into homer. +It's able to handle fragmented and duplicate packets out of the box. + +## Requirements + +### Linux + +None if you use the binary from the releases + +### Windows + +[WinPcap](https://www.winpcap.org/install/default.htm) + +## Installation + +### Linux + +Download [heplify](https://github.com/sipcapture/heplify/releases) and execute 'chmod +x heplify' + +### Windows + +Download [heplify.exe](https://github.com/sipcapture/heplify/releases) + +### Development build + +If you have Go 1.18+ installed, build the latest heplify binary by running `make`. + +Now you should install LUA Jit: + +* Compile from sources: + + Install luajit dev libary + + `apt-get install libluajit-5.1-dev` + + or + + `yum install luajit-devel` + + or for macOS + + ```sh + # Assuming brew installs to /usr/local/ + brew install lua@5.1 luajit + ln -s /usr/local/lib/pkgconfig/luajit.pc /usr/local/lib/pkgconfig/luajit-5.1.pc + export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig/ + ``` + + [install](https://golang.org/doc/install) Go 1.11+ + + `go build cmd/heplify/heplify.go` + + + +### Docker + +You can also build a docker image: + +```bash +docker build --no-cache -t sipcapture/heplify:latest -f docker/heplify/Dockerfile . +``` + +You can use the image using docker compose: + +``` + heplify: + image: sipcapture/heplify:latest + user: 1000:1000 + cap_add: + - CAP_NET_ADMIN + - CAP_NET_RAW + command: + ./heplify -e -hs ${HOMER_DST}:9060 -m SIP -dd -zf -l info + network_mode: host + restart: unless-stopped +``` + +## Usage + +```bash + -assembly_debug_log + If true, the github.com/google/gopacket/tcpassembly library will log verbose debugging information (at least one line per packet) + -assembly_memuse_log + If true, the github.com/google/gopacket/tcpassembly library will log information regarding its memory use every once in a while. + -b int + Interface buffersize (MB) (default 32) + -bpf string + Custom BPF to capture packets + -collectonlysip + collect only sip + -d string + Enable certain debug selectors [defrag,layer,payload,rtp,rtcp,sdp] + -dd + Deduplicate packets + -di string + Discard uninteresting packets by any string + -didip string + Discard uninteresting SIP packets by Destination IP(s) + -diip string + Discard uninteresting SIP packets by Source or Destination IP(s) + -dim string + Discard uninteresting SIP packets by Method [OPTIONS,NOTIFY] + -disip string + Discard uninteresting SIP packets by Source IP(s) + -e + Log to stderr and disable syslog/file output + -eof-exit + Exit once done reading pcap file + -erspan + erspan + -fg uint + Fanout group ID for af_packet + -fi string + Filter interesting packets by any string + -fnum int + The total num of log files to keep (default 7) + -fsize uint + The rotate size per log file based on byte (default 10485760) + -fw int + Fanout worker count for af_packet (default 4) + -hep-buffer-activate + enable buffer messages if connection to HEP server broken + -hep-buffer-debug + enable debug buffer messages + -hep-buffer-file string + filename and location for hep-buffer file (default "HEP-Buffer.dump") + -hep-buffer-max-size string + max buffer size, can be B, KB, MB, GB, TB. By default - unlimited (default "0") + -hi uint + HEP node ID (default 2002) + -hin + HEP collector listening protocol, address and port (example: "tcp:10.10.99.10:9060") + -hn string + HEP node Name + -hp string + HEP node PW + -hs string + HEP server destination address and port (default "127.0.0.1:9060") + -i string + Listen on interface (default "any") + -keepalive uint + keep alive internal - 5 seconds by default. 0 - disable (default 5) + -l string + Log level [debug, info, warning, error] (default "info") + -lp int + Loop count over ReadFile. Use 0 to loop forever (default 1) + -m string + Capture modes [SIP, SIPDNS, SIPLOG, SIPRTCP] (default "SIPRTCP") + -n string + Log filename (default "heplify.log") + -nt string + Network types are [udp, tcp, tls] (default "udp") + -bpf string + Custom bpf filter (default "") + -o + Read packet for packet + -p string + Log filepath (default "./") + -pr string + Portrange to capture SIP (default "5060-5090") + -prometheus string + prometheus metrics - ip:port. By default all IPs (default ":8090") + -protobuf + Use Protobuf on wire + -rf string + Read pcap file + -rs + Use packet timestamps with maximum pcap read speed + -rt int + Pcap rotation time in minutes (default 60) + -s int + Snaplength (default 8192) + -script-file string + Script file to execute on each packet + -script-hep-filter string + HEP filter for script, comma separated list of HEP types (default "1") + -sipassembly + If true, sipassembly will be enabled + -skipverify + skip certifcate validation + -sl + Log to syslog + -t string + Capture types are [pcap, af_packet] (default "pcap") + -tcpassembly + If true, tcpassembly will be enabled + -tcpsendretries uint + Number of retries for sending before giving up and reconnecting (default 64) + -version + Show heplify version + -vlan + vlan + -wf string + Path to write pcap file + -zf + Enable pcap compression + -script-file string + LUA script file path to execute on each packet + -script-hep-filter string + HEP Type filter for LUA script, comma separated list (default "1") + +``` + +## Examples + +```bash +# Capture SIP and RTCP packets on any interface and send them to 127.0.0.1:9060 +./heplify + +# Capture SIP and RTCP packets on any interface and send them via TLS to 192.168.1.1:9060 +./heplify -hs 192.168.1.1:9060 -nt tls + +# Capture SIP and RTCP packets on any interface and send them to 192.168.1.1:9060. Use a someNodeName +./heplify -hs 192.168.1.1:9060 -hn someNodeName + +# Capture SIP and RTCP packets on any interface and send them to 192.168.1.1:9060. Print info to stdout +./heplify -hs 192.168.1.1:9060 -e + +# Capture SIP and RTCP packets on any interface and send them to 192.168.1.1:9060 and 192.168.2.2:9060 +./heplify -hs "192.168.1.1:9060,192.168.2.2:9060" + +# Capture SIP and RTCP packets on any interface and send them to 192.168.1.1:9060. Print debug selectors +./heplify -hs 192.168.1.1:9060 -e -d fragment,payload,rtcp + +# Capture SIP and RTCP packets with custom SIP port range on eth2 and send them to 192.168.1.1:9060 +./heplify -i eth2 -pr 6000-6010 -hs 192.168.1.1:9060 + +# Capture SIP and RTCP packets on eth2, send them to homer and compressed to /srv/pcapdumps/ +./heplify -i eth2 -hs 192.168.1.1:9060 -wf /srv/pcapdumps/ -zf + +# Read example/rtp_rtcp_sip.pcap and send SIP and correlated RTCP packets to 192.168.1.1:9060 +./heplify -rf example/rtp_rtcp_sip.pcap -hs 192.168.1.1:9060 + +# Capture and send packets except SIP OPTIONS and NOTIFY to 192.168.1.1:9060 +./heplify -hs 192.168.1.1:9060 -dim OPTIONS,NOTIFY + +# Capture SIP packet with HPERM encapsulation on port 7932 and interface eth2, send to 192.168.1.1:9060 and print debug info on stdout +./heplify -i eth2 -bpf "port 7932" -hs 192.168.1.1:9060 -l debug -e + +# Capture SIP packet with VXLAN encapsulation on port 4789 and interface eth0, send to 192.168.1.1:9060 and print debug info on stdout +./heplify -i eth0 -bpf "port 4789" -hs 192.168.1.1:9060 -l debug -e + +# Run heplify in "HEP Collector" mode in order to receive HEP input via TCP on port 9060 and fork (output) to two HEP servers listening on port 9063 +./heplify -e -hs HEPServer1:9063,HEPserver2:9063 -hin tcp:1.2.3.4:9060 + + +``` +### Made by Humans + +This Open-Source project is made possible by actual Humans without corporate sponsors, angels or patreons. + +If you use this software in production, please consider supporting its development with contributions or [donations](https://www.paypal.com/cgi-bin/webscr?cmd=_donations&business=donation%40sipcapture%2eorg&lc=US&item_name=SIPCAPTURE&no_note=0¤cy_code=EUR&bn=PP%2dDonationsBF%3abtn_donateCC_LG%2egif%3aNonHostedGuest) + +[![Donate](https://www.paypalobjects.com/en_US/i/btn/btn_donateCC_LG.gif)](https://www.paypal.com/cgi-bin/webscr?cmd=_donations&business=donation%40sipcapture%2eorg&lc=US&item_name=SIPCAPTURE&no_note=0¤cy_code=EUR&bn=PP%2dDonationsBF%3abtn_donateCC_LG%2egif%3aNonHostedGuest) diff --git a/docs/homer_app_FAQs.md b/docs/homer_app_FAQs.md new file mode 100644 index 0000000..2cd7488 --- /dev/null +++ b/docs/homer_app_FAQs.md @@ -0,0 +1,60 @@ + +#### 1) What are the credentials for Homer dashboard login? + + Username: admin + Password: sipcapture + + +#### 2) how do I upgrade the UI changes alone without affecting the homer-app core functionalities? + + Homer-App UI is driven by the dist folder in the source code. to upgrade the UI part we can take the release version that we want and replace the dist folder with homer-ui-.tar.gz dist folder and files + + +#### 3) How data retention policy is taken care of within Homer setup - PostgreSQL Data? + + Currently, this is controlled from Heplify-server **DBDropDays = 14**. We maintain 2 weeks of data. The rest of the data will be purged automatically. + ### File: heplify-server.toml + + +#### 4) Can we set a retention policy for different types of data? say calls 14 days and Registration 4 days? + + Yes, The same can be achieved via Heplify-server config. Below are the params for the same which we need to change in the below-mentioned file. + + `DBDropDaysCall = 14` + `DBDropDaysRegister = 4` + + ### File: heplify-server.toml + +#### 5) How data is stored with PostgreSQL integration? How do view the data for Call and Registration SIP traces? + + Homer stores the SIP traces with respect to **`Call`** and **`Registration`** data in a seperate tables which can be partitioned by hourly or daily(upto our convenient). So in order to view the data in the dashboard we need to select respective HEP Proto. + **For Example:** + i. To view the Call SIP traces we need to select the Proto as **SIP - call**. + ii.To view the Registration SIP traces we need to select the Proto as **SIP - registration**. + +#### 6) Can you change the date format on the Widget Result tab? + +Yes, click Settings -> Advanced -> Add, create a Category called 'system' and Param 'dateTimeFormat' with this structure for JSON object: + +``` +{ + "format": "YYYY/MM/DD HH:mm:ss", + "custom": { + "dateTimeResults": "YYYY/MM/DD HH:mm:ss", + "dateTimeTransaction": "YYYY/MM/DD HH:mm:ss", + "dateTimePreferences": "YYYY/MM/DD HH:mm:ss", + "dateTimeTransactionSubTypes": { + "flow": "YYYY/MM/DD HH:mm:ss", + "recordings": "YYYY/MM/DD HH:mm:ss", + "messages": { + "date": "YYYY/MM/DD", + "time": "HH:mm:ss" + }, + "media_reports": "YYYY/MM/DD HH:mm:ss", + "message_content": "YYYY/MM/DD HH:mm:ss" + } + } +} +``` + +Every field in custom is optional. Change the fields that you're interested in, for the Widget Result tab, change property 'dateTimeResults'. Tokens used for formatting: https://momentjs.com/docs/#/parsing/string-format/ diff --git a/docs/homer_app_configure_ldap.md b/docs/homer_app_configure_ldap.md new file mode 100644 index 0000000..ea1e723 --- /dev/null +++ b/docs/homer_app_configure_ldap.md @@ -0,0 +1,30 @@ +## Configuration block for the LDAPS protocol + +``` + "ldap_config": { + "base": "dc=example,dc=local", + "host": "example.local", + "port": 636, + "usessl": true, + "skiptls": false, + "binddn": "CN=homer-svc,OU=ServiceUsers,DC=example,DC=local", + "bindpassword": "HighlySuperSecretServicePassword", + "userfilter": "(&(sAMAccountName=%s)(|(memberof:1.2.840.113556.1.4.1941:=CN=homer_user_group,OU=Resources,DC=example,DC=local)(memberof:1.2.840.113556.1.4.1941:=CN=homer_admin_group,OU=Resources,DC=example,DC=local)))", + "groupfilter": "(&(objectCategory=group)(name=homer*)(member:1.2.840.113556.1.4.1941:=%s))", + "group_attributes": [ + "cn" + ], + "admingroup": "homer_admin_group", + "adminmode": false, + "usergroup": "homer_user_group", + "usermode": false, + "attributes": ["dn", "givenName", "sn", "mail", "sAMAccountName"], + "skipverify": false, + "anonymous": false, + "userdn": "CN=%s,OU=Users,DC=example,DC=local" + } +``` + +* Authorization allowed only for users from groups homer_admin_group and homer_user_group (configured by `userfilter`); +* Active Directory should only look for user membership in groups whose names begin with `homer` (configured by `groupfilter`); +* `group_attributes` - a list of attributes by which to search for the names of the groups defined in `admingroup` and `usergroup`. diff --git a/docs/homer_app_correlation_mapping_examples.md b/docs/homer_app_correlation_mapping_examples.md new file mode 100644 index 0000000..5712d2b --- /dev/null +++ b/docs/homer_app_correlation_mapping_examples.md @@ -0,0 +1,220 @@ +Let's say you have SIP CALLS, and all SIP messages have been stored in the table **hep_proto_1_call**, and your LOGs are stored in **hep_proto_100_logs** + +For protocol `HEP ID: 1`, `Profile: Call`, you can make an additional correlation MAPPING. +Click to EDIT (the blue wrench in the following picture): + +![Screenshot from 2019-05-20 13-59-14](https://user-images.githubusercontent.com/4513061/58020275-f8c29500-7b07-11e9-9def-ab286d07da4e.png) + + + +And in the MAPPING, add your custom logic. + +In this case, to correlate SIP CALLs traffic to your LOGs which are stored in **hep_proto_100_logs**: +we extract **callid** from the JSON body of **hep_proto_1_call** (below: `"source_field": "data_header.callid",`) and do the lookup to HEP: **100** (logs), profile: **default** +in destination field: **sid**, in the time-range (original) from+=-300, to+=200. + +```javascript +[ + { + "source_field": "data_header.callid", + "lookup_id": 100, + "lookup_profile": "default", + "lookup_field": "sid", + "lookup_range": [ + -300, + 200 + ] + } +] +``` + +The SQL query will look like : +```sql +select * from hep_proto_100_default where sid = 'CALLID'; +``` + +## RTCP JSON correlation + +Here is an example of how to do correlation to another protocol: RTCP JSON, HEP: 5 (below: `"lookup_id": 5,`), destination SID (below: `"lookup_field": "sid",`) can be any header from your JSON body. Here we choose `callid` i.e. ` "source_field": "data_header.callid",`: + +```javascript +[ + { + "source_field": "data_header.callid", + "lookup_id": 5, + "lookup_profile": "default", + "lookup_field": "sid", + "lookup_range": [ + -300, + 200 + ] + } +] +``` + +![Screenshot from 2019-05-20 14-03-36](https://user-images.githubusercontent.com/4513061/58020321-1b54ae00-7b08-11e9-9b13-b65e7d71437e.png) + +and of course you can combine the mappings: +```javascript +[ + { + "source_field": "data_header.callid", + "lookup_id": 100, + "lookup_profile": "default", + "lookup_field": "sid", + "lookup_range": [ + -300, + 200 + ] + }, + { + "source_field": "data_header.callid", + "lookup_id": 5, + "lookup_profile": "default", + "lookup_field": "sid", + "lookup_range": [ + -300, + 200 + ] + } +] +``` +## Correlation-ID correlation - graphing all calls with the same Correlation-ID ;) + +For my SBC, it maps call-leg A, which has call-ID `123@abc` (it fills in the HEP packet correlation-id with `123`), and call-leg B which has call-ID `123@def` (whose HEP packet correlation-id has `123`). This means that the correlation-id for all SIP packets belonging to leg A and leg B (i.e. the same call, across the b2bua), are identical. + +In the Settings -> mapping -> `SIP-1-call` profile, if you look in the lower `Fields Mapping` section below the `Correlation mapping` section, you may have a section like so, if you add Correlation ID as a search field in your SIP Calls panel (Usually called "Home"): +```javascript +... + { + "id": "protocol_header.correlation_id", + "name": "Correlation ID", + "type": "string", + "index": "none", + "form_type": "input", + "position": 3, + "skip": false, + "hide": true, + "sid_type": true + }, +... +``` + +This defines the Correlation ID column as a field to search for. If you want it there, go to your Call Search Settings, and set (click and drag) 'Correlation ID' field to active, then save. + +![Screenshot 2020-07-24 at 02 58 18](https://user-images.githubusercontent.com/647633/88352611-a22f9700-cd5a-11ea-81bb-bf7d62f3e497.png) + +Then click the cog-wheel icon in your results panel (usually bottom left), and set 'Correlation ID' active, save. Now if you search for Correlation ID `123`, you should see only those two call legs A and B for `123` in the results, and the results should show the Correlation ID column. + +![Screenshot 2020-07-24 at 03 08 55](https://user-images.githubusercontent.com/647633/88352771-41ed2500-cd5b-11ea-8403-005dbae5c8e6.png) + + + +Now in the Settings -> mapping -> `SIP-1-call` profile, look in the upper `Correlation Mapping` section, add the following (i.e. replace existing content): + +```javascript +[ + { + "comment": "/*this graphs all calls with the same correlation-ID */", + "source_field": "protocol_header.correlation_id", + "lookup_id": 1, + "lookup_profile": "call", + "lookup_field": "protocol_header->>'correlation_id'", + "lookup_range": [ + -300, + 200 + ] + } +] +``` +Note: ensure you don't have extra commas at line endings after `lookup_range`, or lack commas after each field line, otherwise the script doesn't run properly. You can safely copy-paste the above. + +In the following image, you see the two resulting Call-ID shown at the bottom, when we click on a Call-ID/Correlation-ID displayed in the search results. + +![Screenshot 2020-07-24 at 03 17 31](https://user-images.githubusercontent.com/647633/88353401-98f3f980-cd5d-11ea-811e-30e22cf6af5b.png) + + +If, like me, you want to also graph the RTCP packets on the call-graph, you can change Settings -> mapping -> `SIP-1-call` profile to the following: +```javascript +[ + { + "comment": "/*this graphs RTCP for the respective call-ID*/", + "source_field": "data_header.callid", + "lookup_id": 5, + "lookup_profile": "default", + "lookup_field": "sid", + "lookup_range": [ + -300, + 200 + ] + }, + { + "comment": "/*this graphs all calls with the same correlation-ID */", + "source_field": "protocol_header.correlation_id", + "lookup_id": 1, + "lookup_profile": "call", + "lookup_field": "protocol_header->>'correlation_id'", + "lookup_range": [ + -300, + 200 + ] + } +] +``` + +And your call graph should also show RTCP. + +![Screenshot 2020-07-24 at 03 17 31](https://user-images.githubusercontent.com/647633/88353401-98f3f980-cd5d-11ea-811e-30e22cf6af5b.png) + + + +## SIP-ISUP correlation. + +In the below example, `input_function` will remove any leading `0` from the number and add it to the array. +`post_aggregation_field: sid` will aggregate the calls with the same `SID` (OPC:DPC:CIC) + +So that the resulting SQL query might look like: +```sql +select * from hep_proto_54_default where data_header->'calling_number' IN ('0123456', '123456', '123456') and create_date BETWEEN '2019-02-02-XXXX' AND '2019-02-02-YYYYY' +``` +and that the second SQL query might look like +```sql +select * from hep_proto_54_default where sid IN ('SID_FROM_LAST_QUERY' ) +``` + +The following correlation mapping is necessary: + +```javascript + { + "source_field": "data_header.from_user", + "lookup_id": 54, + "lookup_match_field": "data_header.method", + "lookup_match_value": [ + "INVITE" + ], + "input_function": "data.forEach(function(el) {if(el.charAt(0) === '0') data.push(el.substr(1));});return data", + "lookup_match_first": true, + "lookup_profile": "default", + "lookup_field": "data_header->>calling_number", + "post_aggregation_field": "sid", + "lookup_range": [ + -300, + 200 + ] + }, + +``` + + +### Remote Mapping +Correlation requests can be emitted to entities through the HEP pub-sub API, and dispatched by type. The following example will emit a data request to any entity providing `cdr` capabilities using the `source_field` specified in the mapping configuration: +```javascript + { + source_field: 'data_header.callid', + lookup_id: 0, + lookup_type: "pubsub", + lookup_profile: 'cdr', + lookup_field: '{"data":$source_field,"fromts":$fromts,"tots":$tots}', + lookup_range: [-300, 200], + } +``` diff --git a/docs/homer_app_correlation_using_loki_db.md b/docs/homer_app_correlation_using_loki_db.md new file mode 100644 index 0000000..7a23994 --- /dev/null +++ b/docs/homer_app_correlation_using_loki_db.md @@ -0,0 +1,74 @@ +# How to make correlation using LOKI DB + +`output_function`: + +```javascript +const returnData = []; +for (let dt of data) { + let arrayRet = { + id: 0, + sid: 'unknown', + protocol_header: { + captureId: 2001, + capturePass: 'myHep', + payloadType: 100, + timeSeconds: 1548839273, + timeUseconds: 649, + protocolFamily: 2, + method: 'LOKI' + }, + data_header: { + protocol: 17, + dstIp: '127.0.0.1', + srcIp: '127.0.0.1', + dstPort: 0, + srcPort: 0 + }, + raw: '', + create_date: 0 + }; + arrayRet.id = dt.id; + arrayRet.protocol_header.timeSeconds = new Date(dt.micro_ts).getTime() / 1000; + arrayRet.protocol_header.timeUseconds = new Date(dt.micro_ts).getTime() % 1000; + arrayRet.data_header.sid = 'unknown'; + arrayRet.data_header.srcPort = 1; + arrayRet.data_header.dstPort = 2; + arrayRet.data_header.srcIp = '127.0.0.2'; + arrayRet.data_header.payloadType = 200; + arrayRet.data_header.create_date = dt.micro_ts; + arrayRet.raw = dt.custom_1 + '\\r\\nTags:' + dt.custom_2; + arrayRet.tag = dt.custom_2; + arrayRet.create_date = new Date(dt.micro_ts).getTime(); + var n = dt.custom_1.search(' SrcIP='); + if (n) { + var ipData = dt.custom_1.substring(n + 1, dt.custom_1.length); /* split data */ + var entry = {}; + ipData.split(' ').forEach(function(pair) { + var kv = pair.split('='); + if (!kv[0] || !kv[1]) return; + entry[kv[0]] = kv[1]; + }); + arrayRet.data_header.srcIp = entry['SrcIP']; + arrayRet.data_header.srcPort = entry['SrcPort']; + arrayRet.data_header.dstIp = entry['DstIP']; + arrayRet.data_header.dstPort = entry['DstPort']; + arrayRet.sid = entry['CID']; + arrayRet.data_header.sid = entry['CID']; + } + var tagData = dt.custom_2.substring(1, dt.custom_2.length - 1); + var tagArray = {}; + tagData.split(', ').forEach(function(pair) { + var kv = pair.split('='); + if (!kv[0] || !kv[1]) return; + tagArray[kv[0]] = kv[1].replace(/\"/g, ''); + }); + if (tagArray.type) { + arrayRet.protocol_header.method = 'LOKI-' + tagArray.type; + if (tagArray.type == 'sip' && tagArray.method) arrayRet.protocol_header.method = 'LOKI-SIP-M-' + tagArray.method; + else if (tagArray.type == 'sip' && tagArray.response) arrayRet.protocol_header.method = 'LOKI-SIP-R-' + tagArray.response; + arrayRet.protocol_header.captureId = tagArray.node_id; + } + returnData.push(arrayRet); +}; +return returnData +``` diff --git a/docs/homer_app_multinode_setup.md b/docs/homer_app_multinode_setup.md new file mode 100644 index 0000000..d491ac8 --- /dev/null +++ b/docs/homer_app_multinode_setup.md @@ -0,0 +1,25 @@ +If you want use Homer in the cluster mode, please open your webapp_config.json just add a new block: + +``` + "database_data": { + "node01": { + "help": "Node #1 PGSQL Database (data)", + "host": "10.0.0.1", + "name": "homer_data", + "node": "Node01", + "pass": "homer_password", + "user": "homer_user" + }, + "node02": { + "help": "Node #2 PGSQL Database (data)", + "host": "10.0.0.2", + "name": "homer_data", + "node": "Node02", + "pass": "homer_password", + "user": "homer_user" + } + }, + +``` + +you can configure as many nodes as you want and in the UI you can select nodes where you want to search data diff --git a/docs/query_limit.md b/docs/query_limit.md new file mode 100644 index 0000000..8dfccfc --- /dev/null +++ b/docs/query_limit.md @@ -0,0 +1,42 @@ +In a default (Docker assumed) installation of Homer7, the amount of results returned for SIP Call type searches is 200. + +To change this: + +1. Edit your SIP Call search form +1. Add the `Query Limit` field to the `Active` column +1. Save the form +1. Enter e.g. `1000` in the `Query limit` field +1. Search + +Alternatively + +1 - Settings +2 - User settings +3 - find your dashboard and edit the block +```javascript +... + { + "field_name": "limit", + "hepid": 1, + "name": "1:call:limit", + "selection": "Query Limit", + "type": "string", + "value": "" + }, +... +``` + +to e.g. for 1000 results + +```javascript +... + { + "field_name": "limit", + "hepid": 1, + "name": "1:call:limit", + "selection": "Query Limit", + "type": "string", + "value": "1000" + }, +... +``` diff --git a/docs/rtcagent_installtion_usage.md b/docs/rtcagent_installtion_usage.md new file mode 100644 index 0000000..dd11447 --- /dev/null +++ b/docs/rtcagent_installtion_usage.md @@ -0,0 +1,143 @@ + +RTCagent is an HEP/eBPF powered observability tool for VoIP/WebRTC Applications. + + +About +RTCAgent is a next-generation HEP Agent developed using the latest eBPF technologies. + +RTCAgent greatly differs from any other previous HEP Agent in several ways: + +Unlike native agents, it does not require any code modifications or patches +Unlike passive agents, it does not require access to network interfaces and packets +Unlike any other agent, it traces functions used for sending/receiving data +The result is a new, lightweight and portable HEP Agent able to mirror SIP packets through eBPF hooks +from the core of supported applications bypassing manual code integrations, network encryption and complexity. + + + +Download +Download an amd64/x86 static build of rtcagent and use it immediately on modern kernels. + +curl -fsSL github.com/sipcapture/rtcagent/releases/latest/download/rtcagent -O && chmod +x rtcagent +Prefer using packages? Get the latest deb and rpm releases for amd64/x86 + +Usage + +NAME: rtcagent - Capture and debug RTC Projects. +USAGE: rtcagent [flags] + +COMMANDS: + + help Help about any command + freeswitch capture SIP messages from freeswitch (libsofia): t_port, su_recv + kamailio capture SIP messages from kamailio: recv_msg, udp_send, tcp_send. + opensips capture SIP messages from v: recv_msg, udp_send, tcp_send. + tcprtt show tcp rtt stats + monitor show advanced monitor statistics + + +DESCRIPTION: + + RTCAgent is a tool that can capture and trace SIP packets using eBPF hooks and HEP + + Usage: + rtcagent -h + +OPTIONS: + -d, --debug[=false] enable debug logging + -h, --help[=false] help for rtcagent + -P, --hep-port="9060" hep port - default 9060 + -S, --hep-server="" hep server to duplicate: i.e. 10.0.0.1 + -T, --hep-transport="udp" hep transport default udp. Can be udp, tcp, tls + --hex[=false] print byte strings as hex encoded strings + -l, --log-file="" -l save the packets to file + --nosearch[=false] no lib search + -p, --pid=0 if pid is 0 then we target all pids + -u, --uid=0 if uid is 0 then we target all users + -v, --version[=false] version for rtcagent + + +Build +Compatible with Linux/Android kernel versions >= x86_64 5.x, >= aarch64 5.5. +Linux only. Does not support Windows and macOS. + +Requirements +golang 1.18 or newer +clang 9.0 or newer +cmake 3.18.4 or newer +clang backend: llvm 9.0 or newer +kernel config:CONFIG_DEBUG_INFO_BTF=y +Instructions +Ubuntu +If you are using Ubuntu 20.04 or later versions, you can use a single command to complete the initialization of the compilation environment. + +/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com//sipcapture/rtcagent/master/builder/init_env.sh)" +Any Linux +In addition to the software listed in the 'Toolchain Version' section above, the following software is also required for the compilation environment. Please install before proceeding. + +linux-tools-common +linux-tools-generic +pkgconf +libelf-dev +Clone the repository code and compile + +git clone git@github.com:/sipcapture/rtcagent.git +cd rtcagent +make +bin/rtcagent +compile without BTF +RTCAgent support BTF disabled with command make nocore to compile at 2022/04/17 and can run on Linux systems that do not support BTF. + +make nocore +bin/rtcagent --help + +Usage +Run rtcagent on a native host attaching to a local binary target + +Kamailio +./rtcagent kamailio -T udp -P 9060 -S $HOMER -m /usr/sbin/kamailio +OpenSIPS +./rtcagent opensips -T tcp -P 9061 -S $HOMER -m /usr/sbin/opensips +FreeSwitch +./rtcagent freeswitch -T udp -P 9060 -S $HOMER -m /usr/sbin/freeswitch +Docker +Hypervisor Mode +Run rtcagent as a priviledged container on your host attaching to a local binary target + +rtcagent: + privileged: true + pid: host + image: ghcr.io/sipcapture/rtcagent + container_name: rtcagent + restart: unless-stopped + volumes: + - /sys/fs/cgroup:/host/sys/fs/cgroup:ro + - /sys/kernel/debug:/sys/kernel/debug:rw + command: --cgroupfs-root=/host/sys/fs/cgroup +Cross-Container Mode +Run rtcagent as a priviledged container attached to a target container via docker volume mounts + +Kamailio +rtcagent: + privileged: true + image: ghcr.io/sipcapture/rtcagent + container_name: rtcagent + restart: unless-stopped + volumes: + - $(docker inspect --format="{{.GraphDriver.Data.MergedDir}}" kamailio)/usr/sbin/kamailio:/kamailio:ro + command: /rtcagent kamailio -m /kamailio + depends_on: + - kamailio +OpenSIPS +rtcagent: + privileged: true + image: ghcr.io/sipcapture/rtcagent + container_name: rtcagent + restart: unless-stopped + volumes: + - $(docker inspect --format="{{.GraphDriver.Data.MergedDir}}" opensips)/usr/sbin/opensips:/opensips:ro + command: /rtcagent opensips -m /opensips + depends_on: + - opensips +Credits +RTCAgent is inspired by Cilum, Odigos, eCapture and the many eBPF guides, libraries and implementations.