Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add module S28_python_run #1277

Closed
wants to merge 0 commits into from
Closed

Add module S28_python_run #1277

wants to merge 0 commits into from

Conversation

B1TC0R3
Copy link

@B1TC0R3 B1TC0R3 commented Aug 20, 2024

  • What kind of change does this PR introduce? (Bug fix, feature, docs update, ...)
    Feature.

  • What is the current behavior? (You can also link to an open issue here)
    EMBA cannot run Python scripts natively as modules.

  • What is the new behavior (if this is a feature change)? If possible add a screenshot.
    This PR adds a new module ("S28_python_run") with the capability of running user-supplied python scripts as
    modules. Related to issue Python runner module #1264.

image

  • Does this PR introduce a breaking change? (What changes might users need to make in their application due to this PR?)
    No. The module S28 is disabled in the default scan profile (default-scan.emba).
    In addition, when no Python scripts are manually specified, it will start and terminate without doing anything.

  • Other information:

  1. The changes run in strict mode (-S).
  2. check_project.sh does not detect issues with the new code.
  3. shell_check does not find errors with the new code.
  4. A shell_check exception has been added in S28_python_run.sh to conform with EMBA coding standards. The exception allows the use of quotation marks in arithmetic operations. (See line 39 in S28_python_run.sh)
  5. All Python code has been verified using pyright.

Copy link

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for contributing a pull request!

Welcome to the EMBA firmware analysis community!

We are glad you are here and appreciate your contribution. Please keep in mind our contributing guidelines here and here.
Also, please check existing open issues and consider to open a discussion in the dedicated discussion area.
Additionally, we have collected a lot of details around EMBA, the installation and the usage of EMBA in our Wiki.

If you like EMBA you have the chance to support us by becoming a Sponsor or buying some beer here.

To show your love for EMBA with nice shirts or other merch you can check our Spreadshop.

This is an automatic message. Allow for time for the EMBA community to be able to read the pull request and comment on it.

@m-1-k-3
Copy link
Member

m-1-k-3 commented Aug 20, 2024

I am currently on vacation and will come back to this PR soon.

@B1TC0R3
Copy link
Author

B1TC0R3 commented Aug 20, 2024

Understood.
Enjoy your time off! :)

@m-1-k-3 m-1-k-3 self-assigned this Aug 20, 2024
@m-1-k-3 m-1-k-3 added enhancement New feature or request help wanted Extra attention is needed Core modules (Sxx) The core scanning modules (Sxx modules) EMBA labels Aug 20, 2024
@m-1-k-3
Copy link
Member

m-1-k-3 commented Aug 20, 2024

If someone can start with some initial rewiew and testing it would be highly appreciate.

Copy link
Member

@BenediktMKuehne BenediktMKuehne left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For testing: where/how do I install pip packages?( does the venv work with the scripts)

modules/S28_python_run.sh Outdated Show resolved Hide resolved
modules/S28_python_run.sh Outdated Show resolved Hide resolved
modules/S28_python_run/embamodule.py Outdated Show resolved Hide resolved
modules/S28_python_run/example_script.py Outdated Show resolved Hide resolved
scan-profiles/default-scan-python-runner.emba Outdated Show resolved Hide resolved
Copy link
Member

@m-1-k-3 m-1-k-3 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did a quick code review of the bash code. Please check my notes.
If we are going to include python code into EMBA we will use the same actions as for EMBArk (pylint/bandit/pycodestyle). This means, we need to setup further github actions and include them also in the checker script.

I am not completely sure how we should handle the logging and we need to link to the results from the python modules.

Finally, we need a way to setup a virtual environment dynamically for the container to run the python scripts. Probably we can create a virtual environment in the external directory and mount it via docker-compose into the container.

As there are currently some questions, I would recomment to draft this PR for now to work on these topics.

Good work and I think this will results in cool stuff for EMBA. Keep on going :)

modules/S28_python_run.sh Outdated Show resolved Hide resolved
local lRESULTS=()
local lCOUNT_FINDINGS=0

lPYTHON_BIN="$( find . -name python3 | head -n 1 )"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to check this in the dependency checker. I also think this find is not needed as the installer should have already installed python3 in path. So, at the end we can do something like this here

check_dep_tool "ssdeep"

modules/S28_python_run.sh Outdated Show resolved Hide resolved
modules/S28_python_run.sh Outdated Show resolved Hide resolved
if [[ ${lPYTHON_SCRIPT_COUNT} -gt 0 ]]; then
print_output "[*] ${lPYTHON_SCRIPT_COUNT} Python script/s queued for execution."

for lSCRIPT in "${PYTHON_SCRIPTS[@]}"; do
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I was missing somthing like the following:

local lSCRIPT=""

Why ist the PYTHON_SCRIPTS array not local?
Please rename it to PYTHON_SCRIPTS_ARR

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Considering the PYTHON_SCRIPTS array works essentially the same as the SELECT_MODULES array in an EMBA profile, should I still rename it?

modules/S28_python_run.sh Outdated Show resolved Hide resolved
sub_module_title "${lSCRIPT}"
print_output "[*] Executing: ${lPYTHON_BIN} ${lSCRIPT_DIR}/${lSCRIPT}.py"
mapfile -t lRESULTS < <(${lPYTHON_BIN} "${lSCRIPT_DIR}/${lSCRIPT}.py")
lCOUNT_FINDINGS=$(("${lCOUNT_FINDINGS}" + "${#lRESULTS[@]}"))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can do at least something like the following (unsure on the lRESULTS counting):

                lCOUNT_FINDINGS=$((lCOUNT_FINDINGS + "${#lRESULTS[@]}"))

lRESULTS is an array, please rename it the following lRESULTS_ARR

modules/S28_python_run.sh Outdated Show resolved Hide resolved
modules/S28_python_run.sh Outdated Show resolved Hide resolved
@m-1-k-3 m-1-k-3 marked this pull request as draft August 21, 2024 08:17
@m-1-k-3 m-1-k-3 added the Python label Aug 21, 2024
@B1TC0R3
Copy link
Author

B1TC0R3 commented Aug 21, 2024

I will begin work on the requested changes.
Since this goes out of scope of my Bachelors Thesis, it might take a bit longer tho as I will be working on it in my free time.
Initially, I did not plan on adding any Python modules to EMBA permanently, just to implement an easy method
to run custom ones when required, but I will gladly expand the module to support this properly.
All minor changes (colorization, name changes, ...) should not take very long.

I will also update the copyright notices accordingly, I did not change them due to a misunderstanding on my end.

Please feel free to contact me for any additional questions or requirements.


To clarify some things:

For testing: where/how do I install pip packages?( does the venv work with the scripts)

As it is currently, the module uses EMBAs venv. PIP packages should work when installed into this environment.
However, I have not tested this. Will confirm that it works.
I also do not know whether this venv even should be used, or whether I need to create a custom one
for this module. That decision is up to you.

Why ist the PYTHON_SCRIPTS array not local?

The PYTHON_SCRIPTS array is not local because it is set in the EMBA profile.
It's supposed to allow a user to specify what Python scripts they want to run.
I can add a local version of the array with default settings, which can then be overwritten by the profile setting.
Would be similar to the SELECT_MODULES arrays functionality.

I am not completely sure how we should handle the logging and we need to link to the results from the python modules.

Right now, the Bash module reads the results from STDOUT of the Python script (each line = 1 results).
An alternative solution would be to use grep (or a similar tool) to parse the results from the Python scripts own log file.
This would require some formatting changes in the embamodule.py output to make it grep-able, but
should not take too much work.

Finally, we need a way to setup a virtual environment dynamically for the container to run the python scripts. Probably we can create a virtual environment in the external directory and mount it via docker-compose into the container.

Instead of using EMBAs venv, this might be possible. I will look into it.

@m-1-k-3
Copy link
Member

m-1-k-3 commented Aug 25, 2024

Since this goes out of scope of my Bachelors Thesis, it might take a bit longer tho as I will be working on it in my free time. Initially, I did not plan on adding any Python modules to EMBA permanently, just to implement an easy method to run custom ones when required, but I will gladly expand the module to support this properly.

No hurry with this. I think this could be a very interesting addon for EMBA. Nevertheless, we need to design it in a way it will also fulfill the expectations into a python integration for EMBA.

@B1TC0R3
Copy link
Author

B1TC0R3 commented Sep 1, 2024

As the next big change I want to change the logging method around.

As mentioned before, I plan on making the output from the python module grep-able, so it is easier
to determine what is a finding, what is just debugging output and what is an error.
The goal is to add a better result counter that does not have to rely on the amount of lines output by a Python module.

I could either implement this by reading STDOUT from each Python module directly like I am doing at the moment, or
by parsing the module specific log file after it has finished.

Is there a preferred method to go about this?

@m-1-k-3
Copy link
Member

m-1-k-3 commented Sep 1, 2024

As the next big change I want to change the logging method around.

As mentioned before, I plan on making the output from the python module grep-able, so it is easier to determine what is a finding, what is just debugging output and what is an error. The goal is to add a better result counter that does not have to rely on the amount of lines output by a Python module.

Probably it makes sense to log from the python modules as from the main EMBA modules. All relevant output into a text file that we can then link from the main EMBA s28 module. With this we should be able to easily generate the html report page. I think this could be around line 37 of your module. There we will need a write_link after your print_output. This links to the original log file from your python module.

The grep-able thing is additionally a useful mechanism.

I could either implement this by reading STDOUT from each Python module directly like I am doing at the moment, or by parsing the module specific log file after it has finished.

Is there a preferred method to go about this?

See my notes before. I would recommend to log all relevant details in a nice way to a dedicated log file and link to this log file via write_link from the main s28 module.

@B1TC0R3
Copy link
Author

B1TC0R3 commented Sep 1, 2024

That makes sense, I did not know about the write_link function.

@m-1-k-3
Copy link
Member

m-1-k-3 commented Sep 1, 2024

That makes sense, I did not know about the write_link function.

The web reporter automatically picks it up and is doing the html stuff for you ... hopefully ;) You should see a link with the [REF] marker in the text report. See also here

write_link()

@m-1-k-3 m-1-k-3 linked an issue Sep 8, 2024 that may be closed by this pull request
@B1TC0R3 B1TC0R3 force-pushed the master branch 2 times, most recently from 817a37f to cba0296 Compare September 25, 2024 23:05
@B1TC0R3 B1TC0R3 closed this Sep 25, 2024
@B1TC0R3
Copy link
Author

B1TC0R3 commented Sep 25, 2024

I majorly screwed up my git history just now, which caused this pull request to close. :,)
All the changes are still backed up locally, so that is not too much of an issue.
I will see that I reopen a new pull request/issue as soon as I can, tho I might have to create a new fork of the repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Core modules (Sxx) The core scanning modules (Sxx modules) EMBA enhancement New feature or request help wanted Extra attention is needed Python
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Python runner module
3 participants