Clean OS, clean docker image

[y0d4a]

Hi, i just spins new vm (ubuntu 20.04) and installed docker 20.10.
Pulled this image and on first start i got in problems, grafana is up but without dashboard and data.
In log i see:

parsedmarc_1 | urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f49ef6c3890>: Failed to establish a new connection: [Errno 111] Connection refused
parsedmarc_1 | elasticsearch.exceptions.ConnectionError: ConnectionError(<urllib3.connection.HTTPConnection object at 0x7ffb51ceba90>: Failed to establish a new connection: [Errno 111] Connection refused) caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x7ffb51ceba90>: Failed to establish a new connection: [Errno 111] Connection refused)
parsedmarc_1 | FileNotFoundError: [Errno 2] No such file or directory: '/output/aggregate.json'

and somewhere i saw error that he had permission denied for dashboard file..

where i am wrong? i do like is in tutorial

[manpreets]

Hey @y0d4a

I am having a similar issue in a Mac running Docker 4.15.0 (93002) on engine 20.10.21

dmarc-visualizer-master-parsedmarc-1 | Traceback (most recent call last):
dmarc-visualizer-master-parsedmarc-1 | File "/usr/local/bin/parsedmarc", line 8, in
dmarc-visualizer-master-parsedmarc-1 | sys.exit(_main())
dmarc-visualizer-master-parsedmarc-1 | ^^^^^^^
dmarc-visualizer-master-parsedmarc-1 | File "/usr/local/lib/python3.11/site-packages/parsedmarc/cli.py", line 981, in _main
dmarc-visualizer-master-parsedmarc-1 | process_reports(results)
dmarc-visualizer-master-parsedmarc-1 | File "/usr/local/lib/python3.11/site-packages/parsedmarc/cli.py", line 81, in process_reports
dmarc-visualizer-master-parsedmarc-1 | save_output(results, output_directory=opts.output,
dmarc-visualizer-master-parsedmarc-1 | File "/usr/local/lib/python3.11/site-packages/parsedmarc/init.py", line 1325, in save_output
dmarc-visualizer-master-parsedmarc-1 | append_json(os.path.join(output_directory, aggregate_json_filename),
dmarc-visualizer-master-parsedmarc-1 | File "/usr/local/lib/python3.11/site-packages/parsedmarc/init.py", line 1265, in append_json
dmarc-visualizer-master-parsedmarc-1 | with open(filename, "r+", newline="\n", encoding="utf-8") as output:
dmarc-visualizer-master-parsedmarc-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
dmarc-visualizer-master-parsedmarc-1 | FileNotFoundError: [Errno 2] No such file or directory: '/output/aggregate.json'
dmarc-visualizer-master-parsedmarc-1 exited with code 1

[erikrandom]

Hi, i just spins new vm (ubuntu 20.04) and installed docker 20.10. Pulled this image and on first start i got in problems, grafana is up but without dashboard and data. In log i see:

parsedmarc_1 | urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f49ef6c3890>: Failed to establish a new connection: [Errno 111] Connection refused parsedmarc_1 | elasticsearch.exceptions.ConnectionError: ConnectionError(<urllib3.connection.HTTPConnection object at 0x7ffb51ceba90>: Failed to establish a new connection: [Errno 111] Connection refused) caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x7ffb51ceba90>: Failed to establish a new connection: [Errno 111] Connection refused) parsedmarc_1 | FileNotFoundError: [Errno 2] No such file or directory: '/output/aggregate.json'

and somewhere i saw error that he had permission denied for dashboard file..

where i am wrong? i do like is in tutorial

Hello,

I had to create the following empty files in the /OUTPUT folder mounted by the parsedmarc container:
aggregate.csv
aggregate.json
forensic.csv
forensic.json

..and the subfolder:
sample

Try that if you, like me, run out of patience solving the underlying permission issue.

Now I'm having problems with the grafana graphs from the elastic data. Something about an elastic interval setting that might be deprecated. Let me know if you figure that one out! "Invalid interval specified, must be non-null and non-empty"

[y0d4a]

i created but he did not bring up anything (no grafana, no elastic...) not sure how this project is working after all....

[jburgui]

Hi @y0d4a @erikrandom , I solved the problem adding this to grafana-dockerfile:
run sed 's/fixed_interval/interval/' -i /var/lib/grafana/dashboards/grafana-dmarc_reports.json
Cheers ;)

[rtlong]

I have it working, though I encountered these errors too.

The connection errors come when ElasticSearch is not yet available (it takes a short time to initialize and start listening), but parsedmarc will continue trying to connect for bit. Once it connects and starts processing, it goes silent!

I just opened a PR #42 which is the code I have working and most helpfully it adds the --verbose flag so you get some output during what may be a long process of importing your reports.

Spin it up, let it run, and eventually the parsedmarc container will stop on it's own. Then you can load the dashboard and see it working.

[erikrandom]

That did the trick!
..I had to use case sensitive file name though: /var/lib/grafana/dashboards/Grafana-DMARC_Reports.json

Thank you

[phi0x]

I'm still having issues. I cloned the git, tried to run stock and it fails. I did tweak some ES settings to my liking which is fine, ES seems to run perfectly fine and grafana also appears to be running the parsedmarc fails to establish a connection with ES.

Many failed tried attempts:

elasticsearch.exceptions.ConnectionError: ConnectionError(<urllib3.connection.HTTPConnection object at 0x7fb781086110>: Failed to establish a new connection: [Errno 111] Connection refused) caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x7fb781086110>: Failed to establish a new connection: [Errno 111] Connection refused) INFO:cli.py:753:Starting parsedmarc

the docker container exits after this and if i restart the container it just loops through these errors and exits again instantly.

I tried the potential workarounds people mentioned such as creating blank files in the output directory, updating the grafana Dockerfile to include both:
run sed 's/fixed_interval/interval/' -i /var/lib/grafana/dashboards/grafana-dmarc_reports.json
run sed 's/fixed_interval/interval/' -i /var/lib/grafana/dashboards/Grafana-DMARC_Reports.json

Nothing appears to work to fix enabling the ES connection between parsedmarc and ES. thoughts?...

[tsvane]

I had the "Invalid interval specified, must be non-null and non-empty"-problem when moving the project to a new server.
The sed command did the trick for me, though i had to format it a bit differently.

Added the following command to the end of /dmarc-visualizer/grafana/Dockerfile:
RUN sed -i 's/fixed_interval/interval/g' /var/lib/grafana/dashboards/Grafana-DMARC_Reports.json

[phi0x]

I tried now installing on windows 11 WSL ubuntu, same problem. Seems it's broken right out of the box. Seems parsedmarc can't connect to ES yet ES logs seem to be fine for showing the DB is up. any suggestions?

[Erklin]

@phi0x Have you checked the details of the parsedmarc container? Are the /input and /output mounted as expected and do you have any actual files to process in the input folder?

[phi0x]

I managed to get it working by setting my ES host configuration to 127.0.0.1:9200 instead of what's in the config as 'elasticsearch:9200'. Along with as you note, needing some files for it to parse. When I fixed the connection issue between parsedmarc and ES, I saw the container was quiting right away and the logs weren't saying much that was helpful. I decided to try adding files to the files folder and then restarted the container for parsedmarc and it worked. I then realized that the container must be restarted each time as it doesn't just stay running in the background watching the files folder continuously.

[GIYItalk]

I managed to get it working by setting my ES host configuration to 127.0.0.1:9200 instead of what's in the config as 'elasticsearch:9200'. Along with as you note, needing some files for it to parse. When I fixed the connection issue between parsedmarc and ES, I saw the container was quiting right away and the logs weren't saying much that was helpful. I decided to try adding files to the files folder and then restarted the container for parsedmarc and it worked. I then realized that the container must be restarted each time as it doesn't just stay running in the background watching the files folder continuously.

I encountered the same error as you, did you solve the problem after changing the IP of ES to 127.0.0.1? I did the same, but there is still no data in Grafana. The error is still going on, have you solved this BUG