Skip to content

Helper API

Jump to bottom
Ilia Filippov edited this page Mar 29, 2019 · 13 revisions

TODO...

User-Defined Function Creation

General packet parsing

All UDFs get a pointer to the packet. If it is a first UDF in a flow packet is unparsed and should be parsed. Ethernet (L2) pointer is automatically parsed, other levels should be parsed in lazy mode by user request. Following FFs will get a packet in the parsed state, which is preserved for further processing. Parsing is done in place following a no copy paradigm. Packet structure is filled with pointers to specified headers which are represented by structures of headers. After filing developer has easy access to all header fields.

The first variant of parsing: "Parse" packet methods will set appropriate packet structure fields and must be called before any "Get" methods of corresponding levels. "Get" methods check exact protocol (with performance penalty for checking) and return either this protocol header or nil. "Get_NoCheck" methods convert some packet data to required protocol header without checking (no performance penalties), however, they should be used only after condition from normal "Get" methods. Methods for L4 level should be called only after methods for L3 level.

  • ParseL3 -- Should be first called method for L3 level, set L3 field in proceeding packet structure

  • GetIPv4 -- Return IPv4 header of proceeding packet or nil if this packet is not IPv4

    GetIPv4NoCheck -- Return IPv4 header without checking, no nil, can return garbage

    • ParseL4ForIPv4 -- Should be first called method for L4 level of IPv4 packet, set L4 field

    • GetTCPForIPv4 -- Return TCP header of proceeding IPv4 packet or nil if this packet is not TCP

      GetTCPForIPv4NoCheck -- Return TCP header without checking, no nil, can return garbage

    • GetUDPForIPv4 -- Return UDP header of proceeding IPv4 packet or nil if this packet is not UDP

      GetUDPForIPv4NoCheck -- Return UDP header without checking, no nil, can return garbage

    • GetICMPForIPv4 -- Return ICMP header of proceeding IPv4 packet or nil if this packet is not ICMP

      GetICMPForIPv4NoCheck -- Return ICMP header without checking, no nil, can return garbage

  • GetIPv6 -- Return IPv6 header of proceeding packet or nil if this packet is not IPv6

    GetIPv6NoCheck -- Return IPv6 header without checking, not nil, can return garbage

    • ParseL4ForIPv6 -- Should be first called method for L4 level of IPv6 packet, set L4 field

    • GetTCPForIPv6 -- Return TCP header of proceeding IPv6 packet or nil if this packet is not TCP

      GetTCPForIPv6NoCheck -- Return TCP header without checking, not nil. Can return garbage

    • GetUDPForIPv6 -- Return UDP header of proceeding IPv6 packet or nil if this packet is not UDP

      GetUDPForIPv6NoCheck -- Return UDP header without checking, no nil, can return garbage

    • GetICMPForIPv6 -- Return ICMP header of proceeding IPv6 packet or nil if this packet is not ICMP

      GetICMPForIPv6NoCheck -- Return ICMP header without checking, no nil, can return garbage

  • GetARP -- Return ARP header of proceeding packet or nil is this is not ARP (also see ARP session below)

    GetARPNoCheck -- Return ARP header without checking, no nil, can return garbage

The second variant of parsing: Use of "ParseAll" functions. These functions return pointers to all known protocol headers, all of them will be "nil" except packet protocol:

  • ParseAllKnownL3 -- returns IPv4, IPv6 and ARP headers' pointers, non-nil of the packet is of this protocol

  • ParseAllKnownL4ForIPv4 -- returns TCP, UDP and ICMP headers or nil if the packet doesn't have this protocol

  • ParseAllKnownL4ForIPv6 -- returns TCP, UDP and ICMP headers or nil if the packet doesn't have this protocol

For higher protocols developer can use the following methods:

  • ParseL7 -- gets all supported protocol ID. Sets Data pointer to data after L4 protocol, returns nothing

  • ParseData -- sets Data pointer to data after L4 protocol. Parses the whole packet from L2. Returns 0 for success and -1 for fail

General packet creation

YANFF uses "generate" FF for the creation of flow with new packets. The idea is the following: YANFF automatically allocates future packet (or vector of packets) and give them to UDF. UDF should set the appropriate size to each packet and fill it with the required information. So the type of UDF is the same as in "handle" FF: packet pointer and current context. However here input packets are empty and developer needs to fill them. This can be done manually via encapsulate plus parse functions but it is not very efficient. YANFF provides a range of functions for this purpose. Developer can simply copy required bytes inside packet:

  • GeneratePacketFromByte - gets a slice of bytes of any size and empty packet. Fills packet with these bytes, returns success or failure.

Or there are several functions which will prepare a packet for filling. All of them get a length and empty packet and returns success or fail. Preparation means a set up all basic protocol fields and parses appropriate L3, L4 and Data pointers:

  • InitEmptyPacket -- Prepares packet for filling as Ethernet packet

  • InitEmptyIPv4Packet -- Prepares packet for filling as IPv4 packet

  • InitEmptyIPv6Packet -- Prepares packet for filling as IPv6 packet

  • InitEmptyARPPacket -- Prepares packet for filling as ARP packet (see more cases in ARP session below)

  • InitEmptyIPv4TCPPacket -- Prepares packet for filling as IPv4/TCP packet

  • InitEmptyIPv4UDPPacket -- Prepares packet for filling as IPv4/UDP packet

  • InitEmptyIPv6TCPPacket -- Prepares packet for filling as IPv6/TCP packet

  • InitEmptyIPv6UDPPacket -- Prepares packet for filling as IPv6/UDP packet

  • InitEmptyIPv4ICMPPacket -- Prepares packet for filling as IPv4/ICMP packet

  • InitEmptyIPv6ICMPPacket -- Prepares packet for filling as IPv6/ICMP packet

After these functions developer should fill custom protocol fields like addresses and ports. Initialization of specific ARP packets is described below.

Additional useful general functions

Packet methods:

  • Start -- returns pointer to processing packet data start

  • GetRawPacketBytes -- returns slice with all packet data

  • GetPacketLen -- returns full length of packet (sums of length for reassembled packets)

  • GetPacketSegmentLen -- returns legth of current packet segment (full length for non-reassembled packets)

  • EncapsulateHead -- gets start and length of added segment. Encapsulate by shifting packet head

  • EncapsulateTail -- gets start and length of added segment. Encapsulate by shifting packet tail

  • DecapsulateHead -- gets start and length of the removed segment. Decapsulate by shifting packet head

  • DecapsulateTail -- gets start and length of the removed segment. Decapsulate by shifting packet tail

  • PacketBytesChange -- gets start and slice of new bytes. Writes given bytes into the packet

Other useful functions:

  • SwapBytesUint16 -- swap uint16 for switching between big/little endian

  • SwapBytesUint32 -- swap uint32 for switching between big/little endian

  • BytesToIPv4 -- gets four bytes, returns uint32 which represent IPv4

  • IPv4ToBytes -- gets uint32 which represent IPv4, returns array of four bytes

Dealing with access control lists - ACL

The developer can compare packets manually after parsing. Besides that YANFF allows usage of automatic comparison like access control lists -- ACL. YANFF has an abstraction for rules which can be created via three functions:

  • GetL2ACLFromJSON -- gets filename of JSON structured file with L2 rules, returns created L2Rules

  • GetL2ACLFromORIG -- gets filename of tuple structured file with L2 rules, returns created L2Rules

  • GetL3ACLFromJSON -- gets filename of JSON structured file with L3 and L4 rules, returns created L3Rules

  • GetL3ACLFromORIG -- gets filename of tuple structured file with L3 and L4 rules, returns created L3Rules

JSON structured file is a simple JSON, tuple structured file uses the following structure:

  • "#" is used for commenting a whole string

  • Other strings should have four (for L2) or six (for L3/L4) fields corresponding to the source and destination addresses, next protocol ID, source and destination ports (for L3/L4) and output number

  • All fields except output number can use "ANY" for pointing that this condition should not be used

  • Output number field can be positive or empty / "0" / "Reject" - packet is treated as rejected

These construction functions can be used in a separate goroutine for dynamically changing ACLs. Four packet methods can be used after rules construction:

  • L2ACLpermit -- gets L2Rules. Returns accept or reject for proceeding packet

  • L2ACLport -- gets L2Rules. Returns output number for proceeding packet (0 for rejected packets)

  • L3ACLpermit -- gets L3Rules. Returns accept or reject for proceeding packet

  • L3ACLport -- gets L3Rules. Returns output number for proceeding packet (0 for rejected packets)

Permit functions are expected to be used in "separate" FFs, port functions are expected to be used in "split" FFs. Important note: packets cannot be parsed before using ACL functions. Parsing will be automatic.

Dealing with VLAN tagged packets

YANFF provides several packet methods for dealing with VLAN tagged packets:

  • AddVLANTag -- gets tag and inserts it in proceeding packet

  • ParseL3CheckVLAN -- parses L3 taking VLAN tag into account. If a tag is present return VLAN header structure.

  • GetIPv4CheckVLAN -- returns parsed L3 as IPv4 taking VLAN tag into account

  • GetARPCheckVLAN -- returns parsed L3 as IPv4 taking VLAN tag into account

  • GetIPv6CheckVLAN -- returns parsed L3 as IPv4 taking VLAN tag into account

  • ParseAllKnownL3CheckVLAN -- returns all known protocols taking VLAN tag into account

  • GetEtherType -- returns EtherType taking VLAN tag into account VLAN header can be taken not only via ParseL3CheckVLAN method but also with two special methods:

  • GetVLAN -- returns VLAN header structure if present

  • GetVLANNoCheck -- returns VLAN header structure or something at its place

VLAN header has itself methods:

  • GetVLANTag -- returns tag

  • SetVLANTag -- gets tag and fills VLAN header structure with it

Dealing with ARP requests

A developer can deal with ARP headers by general packet parsing functions: GetARP, GetARPNoCheck, ParseAllKnownL3 and InitEmptyARPPacket described earlier. Besides, developer can use additional functions for initiating ARP requests and answers. These functions fills empty packet inside generate UDF with appropriate information:

  • InitARPRequestPacket add description, parameters and used structures

  • InitARPReplyPacket add description, parameters and used structures

  • InitGARPAnnouncementRequestPacket add description, parameters and used structures

  • InitGARPAnnouncementReplyPacket add description, parameters and used structures

Working with PCAP files

The developer can use "read" and "write" FFs to create flow from packet trace or dump flow to packet trace automatically. Additionally if developer wants to use PCAP files in UDFs directly it is possible to use four methods:

  • WritePcapGlobalHdr -- gets output file descriptor. Writes global PCAP header into a file.

  • WritePcapOnePacket -- gets output file descriptor. Writes one packet with PCAP header into a file. Should be called after WritePcapGlobalHdr.

  • ReadPcapGlobalHdr -- gets input file descriptor and pointer to PcapGlobHdr struct (which is global PCAP header representation according to PCAP format specification). The function reads global PCAP header from a file into given struct.

  • ReadPcapOnePacket -- gets input file descriptor. Read one packet with PCAP header from a file. Returns true if the end of file is reached. Should be called after ReadPcapGlobalHdr.

Dealing with checksums

Checksums can be calculated by software implemented functions or can be offloaded to a network card (hardware offloading).

Set of functions for software checksum calculation include the following functions:

  • CalculateIPv4Checksum -- gets pointer to IPv4 header. Returns checksum of IPv4 header.

  • CalculateIPv4TCPChecksum -- gets pointers to IPv4 and TCP headers and unsafe.Pointer to packet data. Data pointer should point to end of minimal TCP header because TCP options are considered as part of data. Returns TCP checksum.

  • CalculateIPv6TCPChecksum -- gets pointers to IPv6 and TCP headers and unsafe.Pointer to packet data. Data pointer should point to end of minimal TCP header because TCP options are considered as part of data. Returns TCP checksum.

  • CalculateIPv4UDPChecksum -- gets pointers to IPv4 and UDP headers and unsafe.Pointer to packet data. Returns UDP checksum.

  • CalculateIPv6UDPChecksum -- gets pointers to IPv6 and UDP headers and unsafe.Pointer to packet data. Returns UDP checksum.

  • CalculateIPv4ICMPChecksum -- gets pointers to IPv4 and ICMP headers. Returns ICMP checksum. Before calling this function make sure that ICMP L4 checksum is set to zero, otherwise you can get a wrong calculation.

  • CalculateIPv6ICMPChecksum -- gets pointers to IPv6 and ICMP headers. Returns ICMP checksum.

Hardware checksum offloading requires pre-calculation of pseudo-header checksums. Set of functions for calculation of pseudo-header checksums include the following functions:

  • CalculatePseudoHdrIPv4TCPCksum -- gets pointer to IPv4 header. Returns separately computed checksum for TCP pseudo-header for case if L3 protocol is IPv4.

  • CalculatePseudoHdrIPv4UDPCksum -- gets pointers to IPv4 and UDP headers. Returns separately computed checksum for UDP pseudo-header for the case if L3 protocol is IPv4.

  • CalculatePseudoHdrIPv6TCPCksum -- gets pointer to IPv6 header. Returns separately computed checksum for UDP pseudo-header for case if L3 protocol is IPv6.

  • CalculatePseudoHdrIPv6UDPCksum -- gets pointers to IPv6 and UDP headers. Returns separately computed checksum for UDP pseudo-header for a case if L3 protocol is IPv6.

  • SetPseudoHdrChecksum -- gets pointer to packet. Makes pre-calculation of pseudo header checksum. Separately computes checksum for required pseudo-header and writes result to correct place.

HW checksum flags setting

[TODO]

Gopacket compatibility

YANFF library can compat with Gopacket library. There are several options:

  1. Explicitly convert YANFF Packet to gopacket.Packet inside user-defined-function.
gopacketPkt := gopacket.NewPacket(currentPacket.GetRawPacketBytes(), layers.LayerTypeEthernet, gopacket.Default)

This approach is not very performant due to a new gopacket.Packet structure is created for each received packet.

  1. Create known headers in handler UDF to avoid extra gopacket.Packet allocation. This can be used to decode known packet structure and works faster.
var eth layers.Ethernet
var ip4 layers.IPv4
var ip6 layers.IPv6
var tcp layers.TCP
var udp layers.UDP
parser := gopacket.NewDecodingLayerParser(layers.LayerTypeEthernet, &eth, &ip4, &ip6, &tcp, &udp)
decoded := []gopacket.LayerType{}
packetData := currentPacket.GetRawPacketBytes()
err := parser.DecodeLayers(packetData, &decoded)

If this snippet used ‘as is’ inside UDF “HandleFunction”, it will run for every packet in flow. Actually temporary headers and parser can be common for all packets processed by handler, so it is recommended to apply the next option:

  1. Pre-allocate known headers and parser once for each handler and then pass it as context to the handler UDF. This avoids redundant memory allocations (headers and parser creation) on each received packet. Refer to the examples/gopacket_parser_example.go for sample application.
Clone this wiki locally