Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Elias Steyaert changes for google cloud #28

Open
wants to merge 80 commits into
base: master
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
80 commits
Select commit Hold shift + click to select a range
9b1080b
test
EliasSteyaert Apr 24, 2024
3c156d4
ETN-EliasSteyaert-stage.Rmd
EliasSteyaert Apr 26, 2024
df9ecab
adding a file with the google cloud IP adress in
EliasSteyaert Apr 29, 2024
6b708cd
changing some directories
EliasSteyaert Apr 30, 2024
70a1a10
correcting the usage of a variable
EliasSteyaert Apr 30, 2024
aaca91b
adding a new variable DATAPREFIX
EliasSteyaert Apr 30, 2024
f644d90
Merge branch 'master' of github.com:EliasSteyaert/stage-VIB-irefindex
EliasSteyaert Apr 30, 2024
cecf7e1
correcting a directory
EliasSteyaert Apr 30, 2024
b5f8e4f
adding 2 markers, adding and updating (still needs automatisation f…
EliasSteyaert May 2, 2024
c7fa1b4
updating the data_dir variable and creating an empty existing_dirs array
EliasSteyaert May 2, 2024
22fa4ad
a new variable data2
EliasSteyaert May 4, 2024
85aa31f
commenting the jobs that can't be run
EliasSteyaert May 4, 2024
26fbb9e
adding code to create a venv
EliasSteyaert May 4, 2024
f100291
adding the USE_PYTHON_INTERPRETER variable
EliasSteyaert May 6, 2024
794f69c
adjusting the variable USE_PYTHON_INTERPRETER
EliasSteyaert May 6, 2024
5fa8ac1
Update ETN-EliasSteyaert-stage.Rmd
EliasSteyaert May 7, 2024
8971164
changed the cloning the github part to copying my own files to use on…
EliasSteyaert May 7, 2024
c6a2983
adding an extra backslash (since it doesn't matter)
EliasSteyaert May 7, 2024
648b8b3
added an echo message to see where the USE_PYTHON_INTERPRETER sourced…
EliasSteyaert May 7, 2024
ff3321d
changing a backslash (minor unimportant detail)
EliasSteyaert May 7, 2024
c925c35
Merge branch 'master' of github.com:EliasSteyaert/stage-VIB-irefindex
EliasSteyaert May 7, 2024
ad00f23
added the python_interpreter as an environment
EliasSteyaert May 7, 2024
d84ca62
trying to find a solution for the error msgs
EliasSteyaert May 8, 2024
7080461
Added code to put the pyhton interpreter before in the scripts
EliasSteyaert May 10, 2024
a6c7c99
adding the python interpreter before the
EliasSteyaert May 10, 2024
f94320f
adding a sudo_command and markers
EliasSteyaert May 13, 2024
4e04620
putting things in commands because they don't work anymore and use_py…
EliasSteyaert May 13, 2024
5557e0e
New block of code to give the scripts the execute privileges
EliasSteyaert May 14, 2024
fe7eb38
making it more user friendly
EliasSteyaert May 14, 2024
31514ce
adding a block of code to install pip and changing the python environ…
EliasSteyaert May 16, 2024
5d4e55c
deleting the sudo command and the markers that weren't needed
EliasSteyaert May 16, 2024
47905b5
Update ETN-EliasSteyaert-stage.Rmd
EliasSteyaert May 16, 2024
eed975b
Update ETN-EliasSteyaert-stage.Rmd
EliasSteyaert May 16, 2024
98e2fdd
changinng the urls of uniprot and innatedb so they can be downloaded …
EliasSteyaert May 17, 2024
a8d944a
deleting comments so more databases gets downloaded
EliasSteyaert May 17, 2024
e97c396
recreating a part of the playbook to do problem-solving easier/quicker
EliasSteyaert May 17, 2024
04b8a6a
changing the vars of sources to the sources2.yml one
EliasSteyaert May 17, 2024
c42db46
making a smaller sample of the jobs to make it run faster
EliasSteyaert May 17, 2024
46c7d6e
New sources file to test irparse on one file at a time
EliasSteyaert May 22, 2024
9a707bf
last changes to make the mini script work
EliasSteyaert May 22, 2024
5712b8a
defining the interpreter's variable
EliasSteyaert May 22, 2024
a67c0bb
changes to make the irparse work and definitive way to install the ir…
EliasSteyaert May 22, 2024
a8464e5
adding the definition of the variable USE_PYTHON_INTERPRETER
EliasSteyaert May 23, 2024
f9132b0
adding a ansible config file to change the directory of /tmp/.ansible
EliasSteyaert May 24, 2024
3dd481e
script to see free disk space of /home/guest/tmp
EliasSteyaert May 24, 2024
2ad57c7
putting temporary echo's into the file to see what it loops specifically
EliasSteyaert May 29, 2024
49ab22c
minor changes to replace a bit of code that didn't make sense
EliasSteyaert May 29, 2024
2c8da31
big change for the interpreter to work with the irparallel script, ne…
EliasSteyaert May 29, 2024
90dd59c
making changes to get the irparallel working together with the script
EliasSteyaert May 29, 2024
54f31d3
making changes to get the irparallel working together with the script
EliasSteyaert May 29, 2024
e6e9909
minor changes while getting the irparallel working
EliasSteyaert May 29, 2024
c75ca22
adding minor changes to get the irparallel working
EliasSteyaert May 29, 2024
d143e16
making changes to gete irparallel working with the script
EliasSteyaert May 29, 2024
f6adfd1
drop
EliasSteyaert May 31, 2024
86040ee
removing the interpreter
EliasSteyaert May 31, 2024
03a44d6
Update ETN-EliasSteyaert-stage.Rmd
EliasSteyaert Jun 1, 2024
03951d0
Update ETN-EliasSteyaert-stage.Rmd
EliasSteyaert Jun 1, 2024
9cdd8f4
Merge branch 'master' of github.com:EliasSteyaert/stage-VIB-irefindex
EliasSteyaert Jun 1, 2024
27ddbc4
Update ETN-EliasSteyaert-stage.Rmd
EliasSteyaert Jun 1, 2024
140171f
Merge branch 'master' of github.com:EliasSteyaert/stage-VIB-irefindex
EliasSteyaert Jun 1, 2024
682bbb7
minor adjustments to find the ongoing probleme, which was a wrong var…
EliasSteyaert Jun 6, 2024
b51fd24
the .ansible logs were going onto the / disk again
EliasSteyaert Jun 6, 2024
d75b2ae
minor adjustment to figure out if the parsing worked
EliasSteyaert Jun 6, 2024
2e62588
Delete internships/ETN-Elias-howest/test
EliasSteyaert Jun 8, 2024
335da85
Changing code so the data from postgresql gets written to the /mnt/di…
EliasSteyaert Jun 12, 2024
de192ad
Merge branch 'master' of github.com:EliasSteyaert/stage-VIB-irefindex
EliasSteyaert Jun 12, 2024
918a47e
removing echos that were added to understand a problem better
EliasSteyaert Jun 12, 2024
e73343e
adjusting so the .async_dir doesn't write on the / disk again
EliasSteyaert Jun 12, 2024
84e9489
adding a download_data variable
EliasSteyaert Jun 12, 2024
a6a3e43
minor or no changes when trying to find out how irimport really works
EliasSteyaert Jun 12, 2024
60a73bf
Update ETN-EliasSteyaert-stage.Rmd
EliasSteyaert Jun 12, 2024
d14366d
Update ETN-EliasSteyaert-stage.Rmd
EliasSteyaert Jun 12, 2024
0410b79
Create README.md
EliasSteyaert Jun 13, 2024
0b384b9
Update README.md
EliasSteyaert Jun 13, 2024
df8f727
extra adjustments so that the path of the data gets changed without e…
EliasSteyaert Jun 13, 2024
50d0581
Update ETN-EliasSteyaert-stage.Rmd
EliasSteyaert Jun 13, 2024
894d7bc
Update ETN-EliasSteyaert-stage.Rmd
EliasSteyaert Jun 14, 2024
668f7f1
pushing the final versions of the playbooks
EliasSteyaert Jun 14, 2024
ddc0790
Merge branch 'master' of github.com:EliasSteyaert/stage-VIB-irefindex
EliasSteyaert Jun 14, 2024
ed76b1f
deleting commented lines
EliasSteyaert Jun 19, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
putting things in commands because they don't work anymore and use_py…
…thon_interpreter variable
EliasSteyaert committed May 13, 2024
commit 4e04620302ee931b26f557e182acf90c74a9eeb5
19 changes: 10 additions & 9 deletions vsc/ansible/irparse.yml
Original file line number Diff line number Diff line change
@@ -10,6 +10,7 @@

vars:
log_location: "{{ logs_dir }}/irparse"
USE_PYTHON_INTERPRETER: "{{ python_interpreter | default('/usr/bin/python3') }}"

tasks:
- name: Set permissions on Ansible temporary directory
@@ -31,19 +32,19 @@

- name: Find files containing "TOOLS"
find:
paths: ./../usr/bin/
paths: /home/irefindex/usr/bin/
patterns: "*"
recurse: yes
register: found_files

- name: Iterate over found files
vars:
use_python_interpreter: "{{ USE_PYTHON_INTERPRETER }}"
ansible.builtin.shell: |
for filename in {{ found_files.files | map(attribute='path') | join(' ') }}; do sed "s|\(\$TOOLS/.*.py\)|{{ use_python_interpreter }}\" \"\1|g" "{{ filename }}" > "{{ filename }}.tmp" && mv "{{ filename }}.tmp" "{{ filename }}"; done
args:
executable: /bin/bash
when: found_files.matched > 0
# - name: Iterate over found files
# vars:
# use_python_interpreter: "{{ USE_PYTHON_INTERPRETER }}"
# ansible.builtin.shell: |
# for filename in {{ found_files.files | map(attribute='path') | join(' ') }}; do sed "s|\(\$TOOLS/.*.py\)|{{ use_python_interpreter }}\" \"\1|g" "{{ filename }}" > "{{ filename }}.tmp" && mv "{{ filename }}.tmp" "{{ filename }}"; done
# args:
# executable: /bin/bash
# when: found_files.matched > 0

- name: Convert the downloaded data to files for each source
ansible.builtin.command: "./irparse {{ item.name | upper }}"