VNF and VNI tables may get full under unknown condition/usage pattern

[guvenc]

On one of the systems running dpservice, the internal tables looked like the following:

hash_table_saturation{stat_name="capacity",table_name="conntrack_table"} 850000
hash_table_saturation{stat_name="capacity",table_name="dnat_table"} 4032
hash_table_saturation{stat_name="capacity",table_name="interface_table"} 128
hash_table_saturation{stat_name="capacity",table_name="loadbalancer_id_table"} 256
hash_table_saturation{stat_name="capacity",table_name="loadbalancer_table"} 256
hash_table_saturation{stat_name="capacity",table_name="nat_portmap_table"} 850000
hash_table_saturation{stat_name="capacity",table_name="nat_portoverload_table"} 850000
hash_table_saturation{stat_name="capacity",table_name="reverse_vnf_table"} 1000
hash_table_saturation{stat_name="capacity",table_name="snat_table"} 4032
hash_table_saturation{stat_name="capacity",table_name="vnf_table"} 1000
hash_table_saturation{stat_name="capacity",table_name="vni_table"} 512
hash_table_saturation{stat_name="entries",table_name="conntrack_table"} 0
hash_table_saturation{stat_name="entries",table_name="dnat_table"} 1
hash_table_saturation{stat_name="entries",table_name="interface_table"} 0
hash_table_saturation{stat_name="entries",table_name="loadbalancer_id_table"} 0
hash_table_saturation{stat_name="entries",table_name="loadbalancer_table"} 0
hash_table_saturation{stat_name="entries",table_name="nat_portmap_table"} 0
hash_table_saturation{stat_name="entries",table_name="nat_portoverload_table"} 0
hash_table_saturation{stat_name="entries",table_name="reverse_vnf_table"} 1200
hash_table_saturation{stat_name="entries",table_name="snat_table"} 0
hash_table_saturation{stat_name="entries",table_name="vnf_table"} 1200
hash_table_saturation{stat_name="entries",table_name="vni_table"} 614

The conditions are not clear how this state was reached. The logs for both dpservice and metalnet were overwritten.
When this state is reached, creating a new interface returns:
DP_GRPC_ERR_VNF_INSERT

A pattern showing this error from the logs looks like this:

{ "ts": "2024-09-17 12:06:56.947", "thread_id": 2, "thread_name": "grpc", "level": "Info", "logger": "grpc", "msg": "Getting interface info", "interface_id": "ff5fc898-daba-48fe-ac89-a58a35e0f6d4", "caller": "../src/grpc/dp_async_grpc.cpp:287:FillRequest()" }
{ "ts": "2024-09-17 12:06:56.947", "thread_id": 3, "thread_name": "worker", "level": "Warning", "logger": "grpc", "msg": "Failed request", "grpc_request": 6, "grpc_error": 201, "grpc_message": "NOT_FOUND", "caller": "../src/grpc/dp_grpc_impl.c:1111:dp_process_request()" }
{ "ts": "2024-09-17 12:06:56.947", "thread_id": 2, "thread_name": "grpc", "level": "Info", "logger": "grpc", "msg": "Adding interface", "interface_id": "ff5fc898-daba-48fe-ac89-a58a35e0f6d4", "vni": 3605718, "ipv4": "10.180.0.0", "ipv6": "::", "pci": "0000:06:00.0_representor_vf49", "meter_total": 0, "meter_public": 0, "caller": "../src/grpc/dp_async_grpc.cpp:223:FillRequest()" }
{ "ts": "2024-09-17 12:06:56.947", "thread_id": 3, "thread_name": "worker", "level": "Warning", "logger": "service", "msg": "VNF handle addition failed", "vnf_type": 6, "vni": 3605718, "port_id": 50, "ipv4": "0.0.0.0", "prefix_len": 0, "caller": "../src/dp_vnf.c:49:dp_vnf_log_warning()" }
{ "ts": "2024-09-17 12:06:56.947", "thread_id": 3, "thread_name": "worker", "level": "Warning", "logger": "grpc", "msg": "Failed request", "grpc_request": 4, "grpc_error": 401, "grpc_message": "VNF_INSERT", "caller": "../src/grpc/dp_grpc_impl.c:1111:dp_process_request()" }