Improvements and fixes for Windows and PE

[elicn]

This PR introduces a lot of improvements and bugfixes to the Windows OS and PE Loader.

Changelog highlights:

• Refreshed handle, clipboard, fiber and thread modules
• Rewrote the registry module, fixing several fundamental bugs in the previous implementation
• Redefined many of the Windows structures to ctypes, eliminating many 32 bits vs. 64 bits layout duplications (still needs work)
• Restructured and refreshed PE Loader (still needs work)
  • Improved stability
  • Improved log messages
  • Removed unused / duplicated class members
  • Relocated some class members to OS, where it makes more sense
  • DLLs now load and relocate recusrively along with their symbols, based on t-moe's work in Fix PE loader to recursivly import symbols and call DllMain #917 and [WIP] Basic C++ support for windows #918 (see notes below)
• Extracted QlOsStats from utils to its own module
  • Incorporated Windows registry stats into a new QlWinStats object
  • Created nullified versions of stats objects to allow dropping stats transparently (e.g. when stats are not needed, like fuzzing, to save memory and time)
• Numerous styling changes, code quality fixes, comments and typing annotations

Opportunistic DLL loading: loading DLLs recursively solves many "unimplemented API" issues (such as #377), but has its own caveats. To let DLL initialize properly their DllMain function is called and executed on a best-effort basis: many of them are bound to fail sooner or later due to some unimplemented API, and Qiling will resume with loading other DLLs as soon as it happens.

Long DLL relocation time: Qiling is using the pefile module to parse and handle PE files. That module has a known issue with very long relocation durations for large DLL files (e.g. the 64 bits version of kernelbase.dll, which could take up to ~3 minutes to relocate; see erocarrera/pefile#266 and erocarrera/pefile#344). Until the pefile incorporates the fixes a waiting animation has been implemented to show Qiling is still alive - but waiting for relocation to complete (the animation will not appear on non-tty log streams).

Additional changes:

• Rewrote the OS path module, making it more robust and independent of Qiling
  • Generic virtual to host path conversion
  • Safer virtual path handling
  • POSIX host can now retrieve NT case-insensitive filenames, regardless of their case-sensitive names on the host (e.g. "KERNELBASE.DLL" will be found although it is stored as "KernelBase.dll" on the host)

[elicn]

For some reason PE tests run fine on a Linux / WSL host, but fail on Windows.
Will need to figure this out (most probably due to path normalization / canonicalization).

Edit: as expected, that was a path canonicalization error; fixed now.

[elicn]

The Windows sality test keeps failing due to a pefile error that occurs when relocating ntoskrnl.exe.

Local runs on both Ubuntu WSL and an updated Windows finish successfully, so it may indicate that the ntoskrnl.exe file on the testing VM is corrupted. @xwings can you give it a look?

[Note that ntskrnl.exe is a 64-bit PE file loaded on a 32-bit emulated system, but pefile is not aware of that.]

[xwings]

@0ssigeno maybe you have some idea ?

[elicn]

@xwings can you update the VM and re-create the snapshot? That might help.

[xwings]

@xwings can you update the VM and re-create the snapshot? That might help.

sure. @chfl4gs on his way

[chfl4gs]

@elicn the Windows 2019 server is running on fresh image each run. More info on this https://github.com/actions/virtual-environments/blob/main/images/win/Windows2019-Readme.md . Current build is OS Version: 10.0.17763 Build 2686 which is up-to-date.

I did a run on newer Windows Server 2022. Tests took nearly 2 hrs to complete.
https://github.com/chfl4gs/qiling/actions/runs/2107458175

If you need a copy of the dll from these images, I can port it out for you to test.

[elicn]

Thanks @chfl4gs.
The fact that all tests pass on Windows Server 2022 bolsters my assumption that something is wrong with the Windows Server 2019 setup that we have there, and perhaps we should replace it. The failure that we see is originated either in ntoskrnl.exe itself or in the pefile library that is used to load it (we haven't seen this till now only because this is the first time ntoskrnl.exe actually gets loaded).

As for the long running time, that is because pefile has a known issue that causes relocation of large files to take forever. See the last paragraph of this PR description.

[chfl4gs]

Sure. will move to 2022 runner while waiting for pefile get fixed.

[elicn]

@xwings:
pefile maintainer has just merged my fix to the long relocation time. Till he releases a new version (last one was in 2019, so don't hold your breath) we can set the Windows test machine to clone a fresh pefile copy and use it - instead of pip install the current release.

@chfl4gs:
If this is a trouble, could you please pack the Windows Server 2019 System32 folder for me? I want to have a closer look on that ntoskernel.exe loading problem.

[xwings]

@xwings: pefile maintainer has just merged my fix to the long relocation time. Till he releases a new version (last one was in 2019, so don't hold your breath) we can set the Windows test machine to clone a fresh pefile copy and use it - instead of pip install the current release.

@chfl4gs: If this is a trouble, could you please pack the Windows Server 2019 System32 folder for me? I want to have a closer look on that ntoskernel.exe loading problem.

@chfl4gs will you be able to change the pefile clone install and not pip3 ?

[elicn]

Once Qiling dependencies are modified to pull latest pefile, we can merge this into dev.

[xwings]

sure, i will fix it ASAP.

[xwings]

@elicn updated the pefile to latest. Do you want run a test before we merge ?

[elicn]

@xwings
Rebased and re-ran the test suites. The Windows tests now take ~20m instead of 120m.