Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Azurehound runs out of memory during data collection #32

Closed
rpgmaster280 opened this issue Mar 1, 2023 · 5 comments
Closed

Azurehound runs out of memory during data collection #32

rpgmaster280 opened this issue Mar 1, 2023 · 5 comments

Comments

@rpgmaster280
Copy link

rpgmaster280 commented Mar 1, 2023 •
edited
Loading

Performing an azurehound list command against very large azure deployments triggers oom-killer in kali linux, which causes the process to die. It dies shortly after the service principal owner enumeration (so likely the az-ad enum?). Tried to enumerate each tenant separately but couldn't figure out how to do so. Instead, I decided to enumerate based on each available method for the list command. Not sure if this will work, but the command I am attempting to use to accomplish the task is as follows:

for METHOD in $(cat methods.txt); do ./azurehound list ${METHOD} -t REDACTED -r REDACTED -o azurehound_${METHOD}.json; done

This might work as a short term solution, but a long term fix for this issue is definitely needed.
@rpgmaster280
Copy link
Author

rpgmaster280 commented Mar 1, 2023 •
edited
Loading

This issue appears to be a duplicate to #6 . However, I didn't receive any error aside from zsh reporting that the process was killed. Had to dig into the system messages in order to figure out that it was specifically an out of memory error.

@rpgmaster280
Copy link
Author

rpgmaster280 commented Mar 1, 2023 •
edited
Loading

Although that bash command didn't resolve the issue, it did help me collect a significant amount of the azurehound data. This issue is probably present with all submodules to the list command, but I suspect the az-ad and az-rm modules are the submodules that are most likely to cause the issue.

@Daniel-CS-Team
Copy link

Daniel-CS-Team commented Sep 5, 2023 •
edited
Loading

Any updates on this Issue?
We ran in a similar Problem when collecting data from our tenant.
azurehound.exe died after collecting ~ 30GB of data. Unfortunately, the JSON-File seems to be corrupt, so that BloodhoundCE is not able to parse it.

Find here the according stack-trace:

2023-09-04T14:57:06Z INF finished listing all groups count=127590
2023-09-04T14:57:06Z INF finished listing all group owners
2023-09-04T14:58:38Z INF finished listing all devices count=208208
2023-09-04T14:58:38Z INF finished listing all device owners
runtime: VirtualAlloc of 2953224192 bytes failed with errno=1455
fatal error: out of memory

runtime stack:
runtime.throw({0x6a1ba7?, 0xc4a1f9b000?})
        /opt/hostedtoolcache/go/1.18.10/x64/src/runtime/panic.go:992 +0x76
runtime.sysUsed(0xc4760a2000, 0xb006a000)
        /opt/hostedtoolcache/go/1.18.10/x64/src/runtime/mem_windows.go:83 +0x1c9
runtime.(*mheap).allocSpan(0xa90ac0, 0x58035, 0x0, 0x1)
        /opt/hostedtoolcache/go/1.18.10/x64/src/runtime/mheap.go:1288 +0x434
runtime.(*mheap).alloc.func1()
        /opt/hostedtoolcache/go/1.18.10/x64/src/runtime/mheap.go:920 +0x65
runtime.systemstack()
        /opt/hostedtoolcache/go/1.18.10/x64/src/runtime/asm_amd64.s:469 +0x4e

goroutine 1144 [running]:
runtime.systemstack_switch()
        /opt/hostedtoolcache/go/1.18.10/x64/src/runtime/asm_amd64.s:436 fp=0xc09ae77e20 sp=0xc09ae77e18
pc=0x151de0
runtime.(*mheap).alloc(0xb006a000?, 0x58035?, 0x1?)
        /opt/hostedtoolcache/go/1.18.10/x64/src/runtime/mheap.go:914 +0x65 fp=0xc09ae77e68
sp=0xc09ae77e20 pc=0x116fc5
runtime.(*mcache).allocLarge(0xc09ae77ed0?, 0xb00697e2, 0x1)
        /opt/hostedtoolcache/go/1.18.10/x64/src/runtime/mcache.go:213 +0x85 fp=0xc09ae77eb8
sp=0xc09ae77e68 pc=0x107265
runtime.mallocgc(0xb00697e2, 0x0, 0x0)
        /opt/hostedtoolcache/go/1.18.10/x64/src/runtime/malloc.go:1096 +0x5a5 fp=0xc09ae77f30
sp=0xc09ae77eb8 pc=0xfd6a5
runtime.slicebytetostring(0x62c0c0?, 0xc3c6038000, 0xb00697e2)
        /opt/hostedtoolcache/go/1.18.10/x64/src/runtime/string.go:114 +0x50 fp=0xc09ae77f68
sp=0xc09ae77f30 pc=0x13ded0
github.com/bloodhoundad/azurehound/v2/pipeline.FormatJson[...].func1()
        /home/runner/work/AzureHound/AzureHound/pipeline/pipeline.go:239 +0x87 fp=0xc09ae77fe0
sp=0xc09ae77f68 pc=0x5876e7
runtime.goexit()
        /opt/hostedtoolcache/go/1.18.10/x64/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc09ae77fe8
sp=0xc09ae77fe0 pc=0x154141
created by github.com/bloodhoundad/azurehound/v2/pipeline.FormatJson[...]
        /home/runner/work/AzureHound/AzureHound/pipeline/pipeline.go:232 +0xae

goroutine 1 [chan receive, 16 minutes]:
github.com/bloodhoundad/azurehound/v2/sinks.WriteToFile[...]({0x7892a8, 0xc000098280},
{0xc0000b2400, 0x6b8383?}, 0x2c?)
        /home/runner/work/AzureHound/AzureHound/sinks/file.go:47 +0x1b1
github.com/bloodhoundad/azurehound/v2/cmd.outputStream[...]({0x7892a8, 0xc000098280}, 0xc000028340)
        /home/runner/work/AzureHound/AzureHound/cmd/utils.go:397 +0xd6
github.com/bloodhoundad/azurehound/v2/cmd.listCmdImpl(0xa6a540?, {0xc0000bc200?, 0x8?, 0x8?})
        /home/runner/work/AzureHound/AzureHound/cmd/list-root.go:59 +0x257
github.com/spf13/cobra.(*Command).execute(0xa6a540, {0xc0000bc180, 0x8, 0x8})
        /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:860 +0x663
github.com/spf13/cobra.(*Command).ExecuteC(0xa64b40)
        /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:974 +0x3b4
github.com/spf13/cobra.(*Command).Execute(...)
        /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:902
github.com/bloodhoundad/azurehound/v2/cmd.Execute(...)
        /home/runner/work/AzureHound/AzureHound/cmd/root.go:42
main.main()
        /home/runner/work/AzureHound/AzureHound/main_windows.go:39 +0xf4

goroutine 34 [syscall, 61 minutes]:
os/signal.signal_recv()
        /opt/hostedtoolcache/go/1.18.10/x64/src/runtime/sigqueue.go:151 +0x2f
os/signal.loop()
        /opt/hostedtoolcache/go/1.18.10/x64/src/os/signal/signal_unix.go:23 +0x19
created by os/signal.Notify.func1.1
        /opt/hostedtoolcache/go/1.18.10/x64/src/os/signal/signal.go:151 +0x2a

goroutine 35 [select, 61 minutes]:
os/signal.NotifyContext.func1()
        /opt/hostedtoolcache/go/1.18.10/x64/src/os/signal/signal.go:288 +0x70
created by os/signal.NotifyContext
        /opt/hostedtoolcache/go/1.18.10/x64/src/os/signal/signal.go:287 +0x165

goroutine 188 [chan receive]:
github.com/bloodhoundad/azurehound/v2/cmd.listGroupMembers.func2()
        /home/runner/work/AzureHound/AzureHound/cmd/list-group-members.go:93 +0x3fa
created by github.com/bloodhoundad/azurehound/v2/cmd.listGroupMembers
        /home/runner/work/AzureHound/AzureHound/cmd/list-group-members.go:84 +0x14e

goroutine 190 [semacquire, 61 minutes]:
sync.runtime_Semacquire(0x0?)
        /opt/hostedtoolcache/go/1.18.10/x64/src/runtime/sema.go:56 +0x25
sync.(*WaitGroup).Wait(0x0?)
        /opt/hostedtoolcache/go/1.18.10/x64/src/sync/waitgroup.go:136 +0x52
github.com/bloodhoundad/azurehound/v2/cmd.listGroupMembers.func3()
        /home/runner/work/AzureHound/AzureHound/cmd/list-group-members.go:116 +0x26
created by github.com/bloodhoundad/azurehound/v2/cmd.listGroupMembers
        /home/runner/work/AzureHound/AzureHound/cmd/list-group-members.go:115 +0x29e

goroutine 4417405 [select]:
net/http.(*persistConn).writeLoop(0xc001a7a120)
        /opt/hostedtoolcache/go/1.18.10/x64/src/net/http/transport.go:2392 +0xf5
created by net/http.(*Transport).dialConn
        /opt/hostedtoolcache/go/1.18.10/x64/src/net/http/transport.go:1751 +0x1791

goroutine 287 [chan receive, 2 minutes]:
github.com/bloodhoundad/azurehound/v2/pipeline.Mux[...].func1()
        /home/runner/work/AzureHound/AzureHound/pipeline/pipeline.go:68 +0xba
created by github.com/bloodhoundad/azurehound/v2/pipeline.Mux[...]
        /home/runner/work/AzureHound/AzureHound/pipeline/pipeline.go:75 +0xdb

goroutine 296 [semacquire, 61 minutes]:
sync.runtime_Semacquire(0x0?)
        /opt/hostedtoolcache/go/1.18.10/x64/src/runtime/sema.go:56 +0x25
sync.(*WaitGroup).Wait(0x0?)
        /opt/hostedtoolcache/go/1.18.10/x64/src/sync/waitgroup.go:136 +0x52
github.com/bloodhoundad/azurehound/v2/pipeline.Mux[...].func2()
        /home/runner/work/AzureHound/AzureHound/pipeline/pipeline.go:79 +0x26
created by github.com/bloodhoundad/azurehound/v2/pipeline.Mux[...]
        /home/runner/work/AzureHound/AzureHound/pipeline/pipeline.go:78 +0x1b8

goroutine 3937935 [runnable]:
net/http.(*persistConn).roundTrip(0xc001a7a120, 0xc0017964c0)
        /opt/hostedtoolcache/go/1.18.10/x64/src/net/http/transport.go:2620 +0x974
net/http.(*Transport).roundTrip(0xc000074000, 0xc000158900)
        /opt/hostedtoolcache/go/1.18.10/x64/src/net/http/transport.go:594 +0x7c9
net/http.(*Transport).RoundTrip(0xfd7c5?, 0x785a40?)
        /opt/hostedtoolcache/go/1.18.10/x64/src/net/http/roundtrip.go:17 +0x19
net/http.send(0xc000158900, {0x785a40, 0xc000074000}, {0x6876c0?, 0x30001?, 0x0?})
        /opt/hostedtoolcache/go/1.18.10/x64/src/net/http/client.go:252 +0x5d8
net/http.(*Client).send(0xc000026480, 0xc000158900, {0x0?, 0x8?, 0x0?})
        /opt/hostedtoolcache/go/1.18.10/x64/src/net/http/client.go:176 +0x9b
net/http.(*Client).do(0xc000026480, 0xc000158900)
        /opt/hostedtoolcache/go/1.18.10/x64/src/net/http/client.go:725 +0x8f5
net/http.(*Client).Do(...)
        /opt/hostedtoolcache/go/1.18.10/x64/src/net/http/client.go:593
github.com/bloodhoundad/azurehound/v2/client/rest.(*restClient).send(0xc00005e500, 0xc000158900)
        /home/runner/work/AzureHound/AzureHound/client/rest/client.go:255 +0x179
github.com/bloodhoundad/azurehound/v2/client/rest.(*restClient).Send(0xc00005e500, 0xc000158900)
        /home/runner/work/AzureHound/AzureHound/client/rest/client.go:218 +0x42b
github.com/bloodhoundad/azurehound/v2/client.(*azureClient).ListAzureADGroupMembers.func1()
        /home/runner/work/AzureHound/AzureHound/client/groups.go:243 +0x413
created by github.com/bloodhoundad/azurehound/v2/client.(*azureClient).ListAzureADGroupMembers
        /home/runner/work/AzureHound/AzureHound/client/groups.go:209 +0x17f

goroutine 4416177 [select, 2 minutes]:
net/http.(*persistConn).writeLoop(0xc07e639c20)
        /opt/hostedtoolcache/go/1.18.10/x64/src/net/http/transport.go:2392 +0xf5
created by net/http.(*Transport).dialConn
        /opt/hostedtoolcache/go/1.18.10/x64/src/net/http/transport.go:1751 +0x1791

goroutine 849 [select, 2 minutes]:
github.com/bloodhoundad/azurehound/v2/pipeline.OrDone[...].func1()
        /home/runner/work/AzureHound/AzureHound/pipeline/pipeline.go:43 +0xec
created by github.com/bloodhoundad/azurehound/v2/pipeline.OrDone[...]
        /home/runner/work/AzureHound/AzureHound/pipeline/pipeline.go:40 +0xae

goroutine 4416176 [IO wait, 2 minutes]:
internal/poll.runtime_pollWait(0x21ab50deb28, 0x72)
        /opt/hostedtoolcache/go/1.18.10/x64/src/runtime/netpoll.go:302 +0x89
internal/poll.(*pollDesc).wait(0xc088f094a0?, 0x3c?, 0x0)
        /opt/hostedtoolcache/go/1.18.10/x64/src/internal/poll/fd_poll_runtime.go:83 +0x32
internal/poll.execIO(0xc06c805b98, 0x6eae28)
        /opt/hostedtoolcache/go/1.18.10/x64/src/internal/poll/fd_windows.go:175 +0xe5
internal/poll.(*FD).Read(0xc06c805b80, {0xc091680a00, 0x1482, 0x1482})
        /opt/hostedtoolcache/go/1.18.10/x64/src/internal/poll/fd_windows.go:441 +0x25f
net.(*netFD).Read(0xc06c805b80, {0xc091680a00?, 0xc092a08780?, 0xc091680a0d?})
        /opt/hostedtoolcache/go/1.18.10/x64/src/net/fd_posix.go:55 +0x29
net.(*conn).Read(0xc000095040, {0xc091680a00?, 0x58?, 0x21a8fde0108?})
        /opt/hostedtoolcache/go/1.18.10/x64/src/net/net.go:183 +0x45
crypto/tls.(*atLeastReader).Read(0xc0928964f8, {0xc091680a00?, 0x0?, 0x674940?})
        /opt/hostedtoolcache/go/1.18.10/x64/src/crypto/tls/conn.go:785 +0x3d
bytes.(*Buffer).ReadFrom(0xc08f0ee978, {0x7853c0, 0xc0928964f8})
        /opt/hostedtoolcache/go/1.18.10/x64/src/bytes/buffer.go:204 +0x98
crypto/tls.(*Conn).readFromUntil(0xc08f0ee700, {0x21ab5110070?, 0xc000095040}, 0x6bd?)
        /opt/hostedtoolcache/go/1.18.10/x64/src/crypto/tls/conn.go:807 +0xe5
crypto/tls.(*Conn).readRecordOrCCS(0xc08f0ee700, 0x0)
        /opt/hostedtoolcache/go/1.18.10/x64/src/crypto/tls/conn.go:614 +0x116
crypto/tls.(*Conn).readRecord(...)
        /opt/hostedtoolcache/go/1.18.10/x64/src/crypto/tls/conn.go:582
crypto/tls.(*Conn).Read(0xc08f0ee700, {0xc092975000, 0x1000, 0x1?})
        /opt/hostedtoolcache/go/1.18.10/x64/src/crypto/tls/conn.go:1285 +0x16f
net/http.(*persistConn).Read(0xc07e639c20, {0xc092975000?, 0xc0925ae5a0?, 0xc000621d30?})
        /opt/hostedtoolcache/go/1.18.10/x64/src/net/http/transport.go:1929 +0x4e
bufio.(*Reader).fill(0xc08f5a3aa0)
        /opt/hostedtoolcache/go/1.18.10/x64/src/bufio/bufio.go:106 +0x103
bufio.(*Reader).Peek(0xc08f5a3aa0, 0x1)
        /opt/hostedtoolcache/go/1.18.10/x64/src/bufio/bufio.go:144 +0x5d
net/http.(*persistConn).readLoop(0xc07e639c20)
        /opt/hostedtoolcache/go/1.18.10/x64/src/net/http/transport.go:2093 +0x1ac
created by net/http.(*Transport).dialConn
        /opt/hostedtoolcache/go/1.18.10/x64/src/net/http/transport.go:1750 +0x173e

goroutine 1141 [chan receive, 2 minutes]:
github.com/bloodhoundad/azurehound/v2/pipeline.Mux[...].func1()
        /home/runner/work/AzureHound/AzureHound/pipeline/pipeline.go:68 +0xba
created by github.com/bloodhoundad/azurehound/v2/pipeline.Mux[...]
        /home/runner/work/AzureHound/AzureHound/pipeline/pipeline.go:75 +0xdb

goroutine 1143 [semacquire, 61 minutes]:
sync.runtime_Semacquire(0x0?)
        /opt/hostedtoolcache/go/1.18.10/x64/src/runtime/sema.go:56 +0x25
sync.(*WaitGroup).Wait(0x0?)
        /opt/hostedtoolcache/go/1.18.10/x64/src/sync/waitgroup.go:136 +0x52
github.com/bloodhoundad/azurehound/v2/pipeline.Mux[...].func2()
        /home/runner/work/AzureHound/AzureHound/pipeline/pipeline.go:79 +0x26
created by github.com/bloodhoundad/azurehound/v2/pipeline.Mux[...]
        /home/runner/work/AzureHound/AzureHound/pipeline/pipeline.go:78 +0x1b8

goroutine 1145 [select, 16 minutes]:
github.com/bloodhoundad/azurehound/v2/pipeline.OrDone[...].func1()
        /home/runner/work/AzureHound/AzureHound/pipeline/pipeline.go:43 +0xec
created by github.com/bloodhoundad/azurehound/v2/pipeline.OrDone[...]
        /home/runner/work/AzureHound/AzureHound/pipeline/pipeline.go:40 +0xae

goroutine 1182 [select, 2 minutes]:
github.com/bloodhoundad/azurehound/v2/pipeline.OrDone[...].func1()
        /home/runner/work/AzureHound/AzureHound/pipeline/pipeline.go:43 +0xec
created by github.com/bloodhoundad/azurehound/v2/pipeline.OrDone[...]
        /home/runner/work/AzureHound/AzureHound/pipeline/pipeline.go:40 +0xae

goroutine 1184 [select, 2 minutes]:
github.com/bloodhoundad/azurehound/v2/pipeline.OrDone[...].func1()
        /home/runner/work/AzureHound/AzureHound/pipeline/pipeline.go:43 +0xec
created by github.com/bloodhoundad/azurehound/v2/pipeline.OrDone[...]
        /home/runner/work/AzureHound/AzureHound/pipeline/pipeline.go:40 +0xae

goroutine 4417404 [select]:
net/http.(*persistConn).readLoop(0xc001a7a120)
        /opt/hostedtoolcache/go/1.18.10/x64/src/net/http/transport.go:2213 +0xd99
created by net/http.(*Transport).dialConn
        /opt/hostedtoolcache/go/1.18.10/x64/src/net/http/transport.go:1750 +0x173e

@Daniel-CS-Team
Copy link

Ok, we were able to resolve the issue (kind of):
Just throw more hardware on the problem. 16GB of RAM was not enough, but 32 GB did the job...

@StephenHinck
Copy link
Contributor

We believe that this should be resolved as of https://github.com/BloodHoundAD/AzureHound/releases/tag/v2.1.5. Can you please test again and let us know if this remains an ongoing issue?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@StephenHinck @rpgmaster280 @Daniel-CS-Team