Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Pcap file download hangs #8875

Open
Alisher-Nabiev opened this issue Dec 15, 2024 · 13 comments
Open

Pcap file download hangs #8875

Alisher-Nabiev opened this issue Dec 15, 2024 · 13 comments
Assignees
@cardigliano
Labels
In Progress

Comments

@Alisher-Nabiev
Copy link

Environment:

OS name: Ubuntu
OS version: 24.04.1 LTS
Architecture: amd64
ntopng version/revision: ntopng Enterprise L v.6.3.241215

What happened:

I'm trying to download a PCAP file via the API for a 24-hour time range, but it continuously attempts to download over 2GB of data—even though there is no data for that time range. However, when I narrow the time range down to 3 hours, it downloads the correct size.

How did you reproduce it?
API 24h:
https://(user):(pass)@(domain)/lua/rest/v2/get/pcap/live_extraction.lua?ifid=0&host=(IP)%3Beq&epoch_begin=1733810580&epoch_end=1733896800&aggregated=false&query_preset=&count=THROUGHPUT&ip=(IP)

3h:
https://(user):(pass)@(domain)/lua/rest/v2/get/pcap/live_extraction.lua?ifid=0&host=(IP)%3Beq&epoch_begin=1733857380&epoch_end=1733950800&aggregated=false&query_preset=&count=THROUGHPUT&ip=(IP)

I will provide the full API via support ticket
@cardigliano cardigliano self-assigned this Dec 15, 2024
@Alisher-Nabiev
Copy link
Author

Another question on the same topic:

Do we have any limitations when using the API?

We want to understand the time range and the size of the PCAP files that we can download via the API.

Our use case involves searching for a specific IP address. not interface.

@cardigliano
Copy link
Member

@Alisher-Nabiev did you check the downloaded pcap when running a query for 24h? Did you check if it matches the specified IP or actually contains unexpected traffic?

@cardigliano
Copy link
Member

As of the questions, the API is streaming the result, that can be big. However for extractions returning huge amount of data is probably more convenient to use the CLI tools.

@Alisher-Nabiev
Copy link
Author

Alisher-Nabiev commented Dec 16, 2024 •
edited
Loading

when we downloaded a specific IP PCAP file we didn't expect it to be a huge size file.
We can even download the file. We are trying on two separate PCs, one running Windows and the other Mac.
On Windows, the file is showing as 0 bytes, while on Mac, the download stops entirely.

the API we using:
https://user:pass@ourdomain/lua/rest/v2/get/pcap/live_extraction.lua?ifid=0&epoch_begin=1734255080&epoch_end=1734341480&bpf_filter=inner+host+10..0.0.0 ("NOT THE EXACT IP")

explorer_HDUa6oksFA

WhatsApp Image 2024-12-16 at 11 37 22

@cardigliano
Copy link
Member

I see you are specifying a "inner host x.x.x.x" as filter, what kind of tunneled traffic are you capturing? Are you sure it is supposed to match your filter?
Please try with just "host x.x.x.x" or "outer host x.x.x.x".
Anyway, I tried to reproduce the 0-byte download on Mac and it is correctly downloading an empty pcap.

@Alisher-Nabiev
Copy link
Author

Traffic on this interface is not tunneled as far as I know,

also tried without "inner" and with "outer"
nothing happened
explorer_j9IIT9YdaN

@cardigliano
Copy link
Member

I suggest you to verify the dumped files (.pcap) or the indexes (.idx) content connecting to the box via ssh and running the below commands:

  • tcpdump -nr /var/lib/ntopng//pcap/.. .pcap
  • npcapprintindex -i /var/lib/ntopng//pcap/.. .idx

@Alisher-Nabiev
Copy link
Author

checked and looks good :

.pcap
/mnt/new-pcap-vol/ntopng/0/pcap/1733818990.906769# tcpdump -nr 1733821771.640863.pcap
reading from file 1733821771.640863.pcap, link-type EN10MB (Ethernet), snapshot length 1536
11:09:31.640863 IP x.x.x.x.65530 > x.x.x.x.4789: VXLAN, flags [I] (0x08), vni 4446350
IP 213.136.84.130.6027 > x.x.x.x.5707: Flags [P.], seq 3134938850:3134938854, ack 3555740, win 773, length 4
11:09:31.641057 IP x.x.x.x.65439 > x.x.x.x.4789: VXLAN, flags [I] (0x08), vni 4446350
IP x.x.x.x > 100.100.100.126: GREv0, length 48: IP 213.136.84.130.6027 > x.x.x.x.56093: Flags [P.], seq 3134938850:3134938854, ack 3555740, win 773, length 4
11:09:31.641057 IP x.x.x.x.65439 > x.x.x.x.4789: VXLAN, flags [I] (0x08), vni 4446350
IP 100.100.100.126 > x.x.x.x: GREv0, length 209: IP 10.115.42.158.58161 > x.x.x.x.20633: Flags [P.], seq 177872:178037, ack 1580479760, win 31432, length 165
11:09:31.641114 IP x.x.x.x.65439 > x.x.x.x.4789: VXLAN, flags [I] (0x08), vni 4446350
IP 100.100.100.126 > x.x.x.x: GREv0, length 56: IP 10.119.45.105.38486 > x.x.x.x.443: Flags [.], ack 3953409078, win 2006, options [nop,nop,TS val 295303074 ecr 489287410], length 0
11:09:31.641136 IP x.x.x.x.65439 > x.x.x.x.4789: VXLAN, flags [I] (0x08), vni 4446350
IP 100.100.100.126 > x.x.x.x: GREv0, length 56: IP 10.119.45.105.38486 > x.x.x.x.443: Flags [.], ack 1969, win 2006, options [nop,nop,TS val 295303075 ecr 489287410], length 0
11:09:31.641164 IP x.x.x.x.65439 > x.x.x.x.4789: VXLAN, flags [I] (0x08), vni 4446350
IP 100.100.100.126 > x.x.x.x: GREv0, length 68: IP 10.119.45.105.38486 > x.x.x.x.443: Flags [.], ack 1, win 2006, options [nop,nop,TS val 295303074 ecr 489287410,nop,nop,sack 1 {1349:1969}], length 0
11:09:31.641196 IP x.x.x.x.65439 > x.x.x.x.4789: VXLAN, flags [I] (0x08), vni 4446350
IP 100.100.100.126 > x.x.x.x: GREv0, length 67: IP 10.142.160.231.8710 > 185.37.124.113.8883: Flags [P.], seq 1963944490:1963944513, ack 2091902363, win 16384, length 23
11:09:31.641305 IP x.x.x.x.65431 > x.x.x.x.4789: VXLAN, flags [I] (0x08), vni 4446350
IP x.x.x.x.58161 > x.x.x.x.20633: Flags [P.], seq 177872:178037, ack 1580479760, win 31432, length 165

pcap.idx
:/mnt/new-pcap-vol/ntopng/0/pcap/1733818990.906769# npcapprintindex -i 1733821771.640863.pcap.idx
0) len: 110, vlan: 0, vlan_qinq: 0, ipv4, proto: 17, x.x.x.x:65530 -> x.x.x.x:4789, l7proto: (null), tunneled ipv4, proto: 6, x.x.x.x:6027 -> x.x.x.x:5707

  1. len: 132, vlan: 0, vlan_qinq: 0, ipv4, proto: 17, x.x.x.x:65439 -> x.x.x.x:4789, l7proto: (null), tunneled ipv4, proto: 6, x.x.x.x:6027 -> xxx.x.xx.x.x56093
  2. len: 293, vlan: 0, vlan_qinq: 0, ipv4, proto: 17, x.x.x.x:65439 -> x.x.x.x:4789, l7proto: (null), tunneled ipv4, proto: 6, x.x.x.x:58161 ->xxx.x.xx.x.x20633
  3. len: 140, vlan: 0, vlan_qinq: 0, ipv4, proto: 17, x.x.x.x:65439 -> x.x.x.x:4789, l7proto: (null), tunneled ipv4, proto: 6, x.x.x.x:38486 ->xxx.x.xx.x.x32:443
  4. len: 140, vlan: 0, vlan_qinq: 0, ipv4, proto: 17, x.x.x.x:65439 -> x.x.x.x:4789, l7proto: (null), tunneled ipv4, proto: 6, x.x.x.x:38486 ->xxx.x.xx.x.x32:443
  5. len: 152, vlan: 0, vlan_qinq: 0, ipv4, proto: 17, x.x.x.x:65439 -> x.x.x.x:4789, l7proto: (null), tunneled ipv4, proto: 6, x.x.x.x:38486 ->xxx.x.xx.x.x32:443
  6. len: 151, vlan: 0, vlan_qinq: 0, ipv4, proto: 17, x.x.x.x:65439 -> x.x.x.x:4789, l7proto: (null), tunneled ipv4, proto: 6, x.x.x.x:8710 -> xxx.x.xx.x.x:8883
  7. len: 269, vlan: 0, vlan_qinq: 0, ipv4, proto: 17, x.x.x.x:65431 -> x.x.x.x:4789, l7proto: (null), tunneled ipv4, proto: 6, x.x.x.x:58161 ->xxx.x.xx.x.x20633
  8. len: 116, vlan: 0, vlan_qinq: 0, ipv4, proto: 17, x.x.x.x:65484 -> x.x.x.x:4789, l7proto: (null), tunneled ipv4, proto: 6, x.x.x.x:38486 ->xxx.x.xx.x.x32:443
  9. len: 116, vlan: 0, vlan_qinq: 0, ipv4, proto: 17, x.x.x.x:65484 -> x.x.x.x:4789, l7proto: (null), tunneled ipv4, proto: 6, x.x.x.x:38486 ->xxx.x.xx.x.x32:443

@Alisher-Nabiev
Copy link
Author

we capture Vxlan and gre

@cardigliano
Copy link
Member

@Alisher-Nabiev any chance we can schedule a call to check this together? Please drop me an email in case (cardigliano at ntop.org)

@Alisher-Nabiev
Copy link
Author

Time zone?

@cardigliano
Copy link
Member

CET, drop me an email

@cardigliano
Copy link
Member

Update: this works for me, debugging with the user

@cardigliano cardigliano changed the title pcap file download API Pcap file download hangs Dec 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@cardigliano cardigliano

Labels
In Progress
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@cardigliano @Alisher-Nabiev