BSim 'Compare Matching Callees': {...} is outside the current listing's view

[fkelava]

Describe the bug
I apologize in advance for the unclear title and lengthy explanation: I lack the expertise to describe this issue succinctly.

I am currently reversing two binaries. One is a 32-bit executable, the other an ARMv8 64-bit (specifically, Nintendo Switch) binary, but they're both compiled from almost identical source. The latter has unstripped function names, while the former does not. I am attempting to use BSim to match functions between the two.

Consider the following two matching (minus MSVC Control-Flow Guard in the former) functions. I have manually identified them as identical, and BSim agrees.

                             **************************************************************
                             *                          FUNCTION                          *
                             **************************************************************
                             undefined FUN_00919ad0()
                               assume FS_OFFSET = 0xffdff000
             undefined         AL:1           <RETURN>
             undefined4        Stack[-0x8]:4  local_8                                 XREF[2]:     00919ae0(W), 
                                                                                                   00919b55(R)  
             undefined1        Stack[-0x18]:1 local_18                                XREF[2]:     00919b18(*), 
                                                                                                   00919b21(*)  
             undefined1        Stack[-0x38]:1 local_38                                XREF[1]:     00919b2c(*)  
             undefined1        Stack[-0x88]:1 local_88                                XREF[2]:     00919ae3(*), 
                                                                                                   00919aef(*)  
             undefined1        Stack[-0x2b8   local_2b8                               XREF[1]:     00919b3c(*)  
             undefined1        Stack[-0x2d8   local_2d8                               XREF[2]:     00919afb(*), 
                                                                                                   00919b07(*)  
                             FUN_00919ad0
        00919ad0 55              PUSH       EBP
        00919ad1 8b ec           MOV        EBP,ESP
        00919ad3 81 ec d4        SUB        ESP,0x2d4
                 02 00 00
        00919ad9 a1 d8 13        MOV        EAX,[DAT_00c613d8]                               = BB40E64Eh
                 c6 00
        00919ade 33 c5           XOR        EAX,EBP
        00919ae0 89 45 fc        MOV        dword ptr [EBP + local_8],EAX
        00919ae3 8d 85 7c        LEA        EAX=>local_88,[EBP + 0xffffff7c]
                 ff ff ff
        00919ae9 50              PUSH       EAX
        00919aea e8 f1 9c        CALL       FUN_009237e0                                     undefined FUN_009237e0(undefined
                 00 00
        00919aef 8d 85 7c        LEA        EAX=>local_88,[EBP + 0xffffff7c]
                 ff ff ff
        00919af5 50              PUSH       EAX
        00919af6 e8 75 99        CALL       FUN_00923470                                     undefined FUN_00923470(undefined
                 00 00
        00919afb 8d 85 2c        LEA        EAX=>local_2d8,[EBP + 0xfffffd2c]
                 fd ff ff
        00919b01 50              PUSH       EAX
        00919b02 e8 89 a8        CALL       FUN_00924390                                     undefined FUN_00924390(undefined
                 00 00
        00919b07 8d 85 2c        LEA        EAX=>local_2d8,[EBP + 0xfffffd2c]
                 fd ff ff
        00919b0d 50              PUSH       EAX
        00919b0e e8 dd a7        CALL       FUN_009242f0                                     undefined FUN_009242f0(undefined
                 00 00
        00919b13 e8 18 bc        CALL       FUN_00925730                                     undefined FUN_00925730()
                 00 00
        00919b18 8d 45 ec        LEA        EAX=>local_18,[EBP + -0x14]
        00919b1b 50              PUSH       EAX
        00919b1c e8 3f 78        CALL       FUN_00921360                                     undefined FUN_00921360(undefined
                 00 00
        00919b21 8d 45 ec        LEA        EAX=>local_18,[EBP + -0x14]
        00919b24 50              PUSH       EAX
        00919b25 e8 26 77        CALL       FUN_00921250                                     undefined FUN_00921250(undefined
                 00 00
        00919b2a dd d8           FSTP       ST0
        00919b2c 8d 45 cc        LEA        EAX=>local_38,[EBP + -0x34]
        00919b2f 50              PUSH       EAX
        00919b30 e8 ab 79        CALL       FUN_009214e0                                     undefined FUN_009214e0(undefined
                 00 00
        00919b35 6a 00           PUSH       0x0
        00919b37 e8 c4 62        CALL       FUN_0091fe00                                     undefined FUN_0091fe00(undefined
                 00 00
        00919b3c 8d 85 4c        LEA        EAX=>local_2b8,[EBP + 0xfffffd4c]
                 fd ff ff
        00919b42 50              PUSH       EAX
        00919b43 e8 78 b6        CALL       FUN_009251c0                                     undefined FUN_009251c0(undefined
                 00 00
        00919b48 6a 00           PUSH       0x0
        00919b4a 6a 00           PUSH       0x0
        00919b4c 6a 00           PUSH       0x0
        00919b4e 6a 00           PUSH       0x0
        00919b50 e8 cb ae        CALL       FUN_00924a20                                     undefined FUN_00924a20(undefined
                 00 00
        00919b55 8b 4d fc        MOV        ECX,dword ptr [EBP + local_8]
        00919b58 33 cd           XOR        ECX,EBP
        00919b5a 83 c4 34        ADD        ESP,0x34
        00919b5d e8 de f6        CALL       @__security_check_cookie@4                       undefined @__security_check_cook
                 02 00
        00919b62 8b e5           MOV        ESP,EBP
        00919b64 5d              POP        EBP
        00919b65 c3              RET


                             **************************************************************
                             *                          FUNCTION                          *
                             **************************************************************
                             undefined Yn_Dummy()
             undefined         w0:1           <RETURN>
                             Yn_Dummy                                        XREF[1]:     Entry Point(*)  
      7100549130 fc 0f 1e f8     str        x28,[sp, #-0x20]!
      7100549134 fd 7b 01 a9     stp        x29,x30,[sp, #0x10]
      7100549138 fd 43 00 91     add        x29,sp,#0x10
      710054913c ff c3 0d d1     sub        sp,sp,#0x370
      7100549140 a0 83 01 d1     sub        x0,x29,#0x60
      7100549144 29 da 0d 94     bl         rcdtTm2Init                                      undefined rcdtTm2Init()
      7100549148 a0 83 01 d1     sub        x0,x29,#0x60
      710054914c 2b da 0d 94     bl         rcdtTm2Fini                                      undefined rcdtTm2Fini()
      7100549150 a0 83 02 d1     sub        x0,x29,#0xa0
      7100549154 2d da 0d 94     bl         rcdtRsd2Init                                     undefined rcdtRsd2Init()
      7100549158 a0 83 02 d1     sub        x0,x29,#0xa0
      710054915c 2f da 0d 94     bl         rcdtRsd2Fini                                     undefined rcdtRsd2Fini()
      7100549160 32 da 0d 94     bl         rcbgIDLoad                                       undefined rcbgIDLoad()
      7100549164 a0 43 03 d1     sub        x0,x29,#0xd0
      7100549168 1c d8 0d 94     bl         rcVec3fNormalize                                 undefined rcVec3fNormalize()
      710054916c a0 43 03 d1     sub        x0,x29,#0xd0
      7100549170 16 d9 0d 94     bl         rcVec3fLength                                    undefined rcVec3fLength()
      7100549174 a0 03 03 d1     sub        x0,x29,#0xc0
      7100549178 b0 d8 0d 94     bl         rcBBoxInit                                       undefined rcBBoxInit()
      710054917c e0 03 1f aa     mov        x0,xzr
      7100549180 2e da 0d 94     bl         rcbgPSimInit                                     undefined rcbgPSimInit()
      7100549184 e0 03 00 91     mov        x0,sp
      7100549188 30 da 0d 94     bl         rcefMainFini                                     undefined rcefMainFini()
      710054918c e0 03 1f 2a     mov        w0,wzr
      7100549190 e1 03 1f 2a     mov        w1,wzr
      7100549194 e2 03 1f 2a     mov        w2,wzr
      7100549198 e3 03 1f 2a     mov        w3,wzr
      710054919c 2f da 0d 94     bl         rcefTargetInit                                   undefined rcefTargetInit()
      71005491a0 ff c3 0d 91     add        sp,sp,#0x370
      71005491a4 fd 7b 41 a9     ldp        x29,x30,[sp, #0x10]
      71005491a8 fc 07 42 f8     ldr        x28,[sp], #0x20
      71005491ac c0 03 5f d6     ret

Raise a comparison between these two functions, then try raising 'Compare Matching Callees' for the first pair of callees- FUN_009237e0 and rcdtTm2Init. See attached images and function listings.

                             **************************************************************
                             *                          FUNCTION                          *
                             **************************************************************
                             undefined FUN_009237e0(undefined4 param_1)
                               assume FS_OFFSET = 0xffdff000
             undefined         AL:1           <RETURN>
             undefined4        Stack[0x4]:4   param_1                                 XREF[1]:     009237e7(R)  
                             FUN_009237e0                                    XREF[3]:     FUN_00919ad0:00919aea(c), 
                                                                                          FUN_00923b40:00923b52(c), 
                                                                                          FUN_0093beb0:0093beed(c)  
        009237e0 55              PUSH       EBP
        009237e1 8b ec           MOV        EBP,ESP
        009237e3 6a 50           PUSH       0x50
        009237e5 6a 00           PUSH       0x0
        009237e7 ff 75 08        PUSH       dword ptr [EBP + param_1]
        009237ea e8 5d 5e        CALL       MSVCR110.DLL::memset                             void * memset(void * _Dst, int _
                 02 00
        009237ef 83 c4 0c        ADD        ESP,0xc
        009237f2 5d              POP        EBP
        009237f3 c3              RET

                             **************************************************************
                             *                          FUNCTION                          *
                             **************************************************************
                             undefined rcdtTm2Init()
             undefined         w0:1           <RETURN>
                             rcdtTm2Init                                     XREF[2]:     Yn_Dummy:7100549144(c), 
                                                                                          rcbgTrxTexLoadFromFile:71005613e
      71008bf9e8 90 47 00 d0     adrp       x16,0x71011b1000
      71008bf9ec 11 96 40 f9     ldr        x17,[x16, #0x128]=>->rcdtTm2Init                 = 710054aff0
      71008bf9f0 10 a2 04 91     add        x16,x16,#0x128
      71008bf9f4 20 02 1f d6     br         x17=>rcdtTm2Init

                             **************************************************************
                             *                          FUNCTION                          *
                             **************************************************************
                             undefined rcdtTm2Init()
             undefined         w0:1           <RETURN>
                             rcdtTm2Init                                     XREF[3]:     Entry Point(*), 
                                                                                          rcdtTm2Init:71008bf9f4(j), 
                                                                                          71011b1128(*)  
      710054aff0 02 0a 80 52     mov        w2,#0x50
      710054aff4 e1 03 1f 2a     mov        w1,wzr
      710054aff8 b4 b5 0c 14     b          memset                                           void * memset(void * __s, int __
                             -- Flow Override: CALL_RETURN (CALL_TERMINATOR)

These are the same function, but in the ARM executable there is another stub in-between. However, I cannot actually navigate past the stub call because "710054aff0" is outside the current listing's view. This is proving quite the blocker, because every call in the ARM executable has this prologue of sorts between the base function and the actual "contents", so to say.

I can circumvent this by having another CodeBrowser open and manually inspecting the function, but this adds quite a lot of hassle to the whole process. Is there any way 'Compare Matching Callees' can permit me to descend past this 'stub' and see the 'real' rcdtTm2Init (the one at 710054aff0) in this case?

Expected behavior
'Compare Matching Callees` does not force me to stay at the same "level" in both listings.

Environment (please complete the following information):

• OS: Windows 11
• Java Version: Temurin JDK 17
• Ghidra Version: 11.0
• Ghidra Origin: Official GitHub distro

Additional context
The executable in question is a commercially available game, Final Fantasy X/X-2 HD Remaster. I do not think it would be proper to share the executables in this case, so I hope the function listings suffice to explain the issue.

If I can provide any other relevant info, please let me know.

[mumbel]

The thunk analyzer should have picked that up and marked as such.

Ghidra/Processors/AARCH64/data/aarch64-pltThunks.xml

I don't know why this is not in the pattern/ directory like other processor modules, so there might be an analyzer you have to enable for AARCH64PltThunkAnalyzer.java

[fkelava]

The only analyzer I see that seems related is AARCH64 ELF PLT Thunks. It is enabled by default, but it does not seem to have any effect. (i.e. it results in the function listings/analysis given above on Ghidra 10.3 through 11.0)

I have found a related issue here, but it isn't clear to me what I'm missing. I tried seeing if this analyzer is available as a one-shot option or even running auto-analysis twice, first with it disabled and then with it enabled, but it still has no effect.

[fkelava]

Upon closer inspection, it seems that the analyzer attempts to run first (coincidentally, it's also first in alphabetical order), when the PLT section has not yet been disassembled.

However, it seems the analyzer requires the PLT section to be disassembled first.

ghidra/Ghidra/Processors/AARCH64/src/main/java/ghidra/app/plugin/core/analysis/AARCH64PltThunkAnalyzer.java

Lines 57 to 63 in e73d208

	public AARCH64PltThunkAnalyzer() {
	super(NAME, DESCRIPTION, AnalyzerType.BYTE_ANALYZER); // assumes ELF Loader disassembled PLT section
	setDefaultEnablement(true);
	setPriority(AnalysisPriority.FORMAT_ANALYSIS);
	}

Is there any way for me to control the analyzer run order so as to allow this to function normally?

[ghidra1]

undefined rcdtTm2Init()
      71008bf9e8 90 47 00 d0     adrp       x16,0x71011b1000
      71008bf9ec 11 96 40 f9     ldr        x17,[x16, #0x128]=>->rcdtTm2Init                 = 710054aff0
      71008bf9f0 10 a2 04 91     add        x16,x16,#0x128
      71008bf9f4 20 02 1f d6     br         x17=>rcdtTm2Init

As @mumbel pointed out, the above function should have been identified as a thunk which hopefully would remov it from consideration on your first navigation. Because of this issue the comparison panels are out-of-sync, one is showing an unidentifed thunk which should be skipped over if it been marked as such.

What section is the address 71008bf9e8 contained in? The pattern is contained with aarch64-pltThunks.xml. Unfortunetly, the AARCH64PltThunkAnalyzer is constrained to the ELF .plt section which is probably incorrect since such such thunks could occur anywhere local function linkage is required.

Could you please try to manually establish the thunk at 71008bf9e8 and identify the "thunked" function as 710054aff0. You may want to remove the label at 71008bf9e8 before you do this. Once this is done, could you repeat your comparison panel use case and navigation and convey the results. The popup menu action on the thunk's function signature field Function -> Set Thunked Function... can be used to this.

[ghidra1]

Let us know if you are running in a development enviorment where you could tryout a code change.

[fkelava]

What section is the address 71008bf9e8 contained in?

71008bf9e8 is located at .plt+47e70h. By that logic, it should still be correctly analysed. The executable also contains a .rela.plt section and a .got.plt section. However, neither seems to contain actual PLT thunks as .plt does- the latter contains a ton of regular function pointers, and auto-analysis produces no discernible output in the former. See images.

Could you please try to {...}

This does not seem to have an effect. The comparison does not descend past the thunk even when it is declared as such. I removed the label, then Function -> Set Thunked Function -> 710054aff0. See image.

Let us know if you are running in a development enviorment where you could tryout a code change.

I can set one up. The instructions in the README seem simple enough.

[fkelava]

Does declaring the function as a thunk possibly require a complete BSim generatesigs/commitsigs cycle again? It seems to me that it doesn't, as merely declaring it as a thunk and saving the executable was already reflected in the comparison.

[ghidra1]

@fkelava For some reason your images are private and I get an error when I click on them to show them in a larger readable form. This private-user-images.githubusercontent.com page can’t be found. I can't read them without first clicking them which fails for me.

Do you have default enablement for all your analyzers? I am assuming they all ran properly.

Is this a readily available binary we could try analyzer ourselves?

[ghidra1]

Does declaring the function as a thunk possibly require a complete BSim generatesigs/commitsigs cycle again? It seems to me that it doesn't, as merely declaring it as a thunk and saving the executable was already reflected in the comparison.

Specifying the thunked function can be done without additional analysis. Which file was ingested into BSim? The DLL or the ELF AARCH64 binary? If it was the ELF binary it is possible that it would need to be updated in the BSim database to update the function callgraph.

[ghidra1]

For some reason your images are private and I get an error when I click on them

Issue seems to be resolved

[ghidra1]

It is also curious why the .rela.plt is not marked-up. This section contains relocation entries which appear to have been skipped. Not sure what impact this may have, but I prefer to ensure all relocation tables are processed. I don't think it would have impacted the thunk creation issue we have been discussing but is worth looking into. Is there a .dynamic section and entry which refers to this? I see 4 references to the top of it. Hopefully it is not relying purely on hard-coded processing by the init function.

[ghidra1]

This does not seem to have an effect. The comparison does not descend past the thunk ...

This would seem to imply the ELF binary is the one which was ingested into the BSim database where the callgraph was retained. This callgraph is stale since it will still identify the thunk as the called destination. The intent is that thunks shuold be ignored in a callgraph. The BSim data would need to be updated, although we should probably understand why the thunk was not created during analysis. Our access to a similar binary is key to us resolving this. Is this binary from a AARCH64 Linux distribution you could provide a link to?

[fkelava]

Let us know if you are running in a development enviorment where you could tryout a code change.

I have set one up in case you would like me to try any changes.

Do you have default enablement for all your analyzers? I am assuming they all ran properly.

Yes. However, see comment above - I have noticed the AARCH64 ELF PLT Thunks analyzer seems to run before .plt has been disassembled.

Perhaps the Nintendo Switch binary loader being used violates the analyzer's assumption that .plt will have been disassembled before it runs? I didn't piece together until now that it might act as the "ELF loader" the analyzer is referring to in this case.

If that is indeed the case, I will raise this issue with them instead. My apologies for not realizing this sooner.

Is this a readily available binary we could try analyzer ourselves?

As I've noted above, this is a commercially available game; "readily available" may therefore be a stretch.

Which file was ingested into BSim? The DLL or the ELF AARCH64 binary?

The ELF binary. Just in case, I decided to completely regenerate the signatures after checking in a version where the thunk is marked, commit them, then try again. This, too, does not resolve the issue.

I have just noticed that I can continue using Compare Matching Callees after the first level. The result is that the comparison is always out of synch by one level. See images.

Is there a .dynamic section and entry which refers to this?

Yes. It seems to be as devoid of markings as .rela.plt itself. Image attached.

[fkelava]

Our access to a similar binary is key to us resolving this.

I will attempt to find a binary which exhibits the same issue that I can freely share and attach it as soon as I can.

[fkelava]

Okay, I've worked it out- sort of. Your comments finally led me to the right idea.

.plt/.got are indeed processed at import time for regular AARCH64 ELF files- it is the loader for Nintendo Switch NRO format binaries in use that fails to do this, leading to a conflict with the PLT thunk analyzer.

So that merits no further investigation- I will raise that issue with them instead.

What still concerns me is that even a manually set thunk, after a full regeneration of signatures, did not resolve the issue in the function comparison; even if the thunk was not detected as such by the analyzer due to the aforementioned situation, should it not have been fixed by me manually declaring the function a thunk?

[ghidra1]

Did not realize this was a game binary - they tend to be rather non-conforming and I don't have much experience with them. I imported an AARCH64 binary from an Ubuntu distribution and the .rela.plt does get processed and has a dynamic DT_JMPREL entry that identifies it.

Nintendo Switch may rely on hardcoded relocation processing within there init that bypasses the dynamic section entries that we rely on.

[ghidra1]

should it not have been fixed by me manually declaring the function a thunk?

I am not really familiar with how the BSim comparison panels are managed and how they handle navigation. However, as I mentioned above, I suspect the callgraph data that is stored within the BSim DB is not in-sync with the thunk change you applied manually. BSim still believes it is a valid called function - while thunks would not be contained with the callgraph.

[ghidra1]

Unfortunately, the Nintendo Switch Loader is out of our control. It would be its responsibility for handling all relocations properly - which it may without marking them up. Never having used it, it is difficult to say why the analysis does not properly establish the thunks within the .plt section. You could attempt to debug the AARCH64PltThunkAnalyzer to see why it fails to process the .plt. Unfortunately, we have to close this ticket since the analyzed state of the binary is unknown to us. If you would like to pass-along a Ghidra gzf file we could determine why the thunk fails to be identified during analysis.

[fkelava]

Yeah, that's fair. It took a few of your comments for me to realize it was a loader fault; thank you for the time regardless. Apologies that I didn't figure that one out sooner.

In the event that I "fix" or at least adapt the PLT analyzer to my use-case, I will update you on whether that resolved the comparison issue. In the event that it still occurs, I'll update you with a (hopefully) better repro.

[ghidra1]

Does your use case currently disable many of the analyzers? If so, did you try to run any of the analyzers as a one shot (or re-analyze with these enabled) to see if the identify the thunk(s)? i.e., AARCH64 ELF PLT Thunks, Function Start Search

[fkelava]

It does not. AARCH64 ELF PLT Thunks is not available as a one-shot analyzer, for one or another reason. Is it meant to be available?

I would hazard a guess that if it did, this could be resolved by simply re-running it once initial auto-analysis completes.

Could I make a code change to enable one-shot capability for it? I have a build environment set up.

[ghidra1]

It does not. AARCH64 ELF PLT Thunks is not available as a one-shot analyzer, for one or another reason. Is it meant to be available?

Is the analyzer enabled in the list of auto-analyzers? If it runs you can try to debug since it should get a pattern match on the thunk in question.

The one-shot simply indicates if it will appear in GUI menu. It can always be forced to run via the API :AutoAnalysisManager.scheduleOneTimeAnalysis(new AARCH64PltThunkAnalyzer(), program.getMemory())

[fkelava]

I've continued investigating this. I've managed to resolve the issue where the thunks are not marked as such. However, even once the ELF PLT Thunks analyzer properly marks the thunks as such, the BSim comparison still gets hung up on it in Compare Matching Functions.

If you would like to pass-along a Ghidra gzf file we could determine why the thunk fails to be identified during analysis.

Here is a link to GZF exports of both executables. A comparison that exhibits the issue is FUN_00919ad0 in FFX.exe with Yn_Dummy in FFX.nro.

To clarify and spare you the effort of re-reading the thread, when Compare Matching Functions is performed in a situation where said matching function is a thunk in only one executable, the dual listing won't pass through the thunk and show the actual function body.

Please let me know if you need the files uploaded in some other place or way, or if anything is missing.

If you require any output from my BSim server, please let me know. I've also updated to 11.0.1, to no avail.

[ghidra1]

We will look into the comparison panel navigation concern.

[ryanmkurtz]

Fixed by 5ee04b2