Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cleaned picks.csv formatting with the csv library #11

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

lennijusten
Copy link
Contributor

I described issue #9 a while ago and here is the PR with the fix. The fix is a two-part solution.

First, I open the picks.csv file in the write w mode with the csv.writer from the csv library and write the header row. The write w mode will overwrite any existing file with the same filename.

I then open picks.csv in the append a mode and write the results to picks.csv by looping through each batch. The results are converted from arrays to lists with the tolist() method which removes the excess white spaces. I then append the results to the csv with the writerow() method by passing the results in list format. This automatically delimits the csv columns into fname, itp, tp_prob, its, ts_prob and adresses any issues like overfull lines automatically (see issue #6).

I have tried and tested the method and it works as hoped. When reading the csv files with pandas, one should still note that the pandas.read_csv() method will default to reading all entries as strings. This is described fully in a stackoverflow post. An easy fix is using the literal_eval method from ast.

>>> from ast import literal_eval
>>> literal_eval('[1.23, 2.34]')
[1.23, 2.34]

Best,
Lenni

I described this issue in under AI4EPS#9 and here is the PR with the fix. The fix is a two-part solution. 

First, I open the picks.csv file as fclog with the csv library and write the header row. I then open picks.csv in the append mode and write the results to picks.csv batch by batch. The results are converted from arrays to lists (this removes the empty white spaces), and then written to the row in a list of results instead of a single string formatted with the results. This also fixes the new-line issue I had opened and resolved earlier. 

I have tried and tested the method and it works as hoped. 

Best,
Lenni
@lennijusten
Copy link
Contributor Author

Here is an excel copy of an example output.

picks.xlsx

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant