Examples
To simplify when we talk about irods file it refer to data_object and irods folder to collection.
This command create a file (.irods_metadata_local_save.pkl) in your local home folder. This usefull because to have the autocompletion when working on metadata you have to loop on all your file that can be very timeconsumming when you have many or have to do it many time. This dictionary will be update when you add new metadata. The good pratice would be to create this dictionnary before puting any data on irods or if you already have data on irods with metadata associated you can run this command and wait that the dictionary is built.
## BUILD THE DICTIONARY IN LOCAL TO ACCES AUTOCOMPLETION ON METADATA
$ ./easicmd.py build_dico_meta
Dictionary have been save in /beegfs/home/gdebaecker/.irods_metadata_local_save.pkl
To put a data (file or folder) on irods you just have to give a path to the data. Then you will be asked where in irods you want to put it. When you use push to upload data the sha256 is calculated and stock in the icat (option -K in iput). After "pushing" your data on irods you will be asked if you want to add metadata to your new irods object.
## PUT THE LOCAL FOLDER "PROJECT_1" IN THE IRODS FOLDER "MY_PROJECT" AND ADD METADATA
$ ./easicmd.py push PROJECT_1
ifolder (empty = /zone/home/user ): /lbbeZone/home/gdebaecker/MY_PROJECT
/lbbeZone/home/gdebaecker/irods_test
/lbbeZone/home/gdebaecker/irods_test/raw_data
/lbbeZone/home/gdebaecker/irods_test/raw_data/fast5
/lbbeZone/home/gdebaecker/MY_PROJECT
/lbbeZone/home/gdebaecker/NeGa
Running recursive pre-scan... pre-scan complete... transferring data...
C- /lbbeZone/home/gdebaecker/MY_PROJECT/PROJECT_1:
0/2 - 0.00% of files done 0.000/0.000 MB - 0.00% of file sizes done
Processing file.fastq - 0.000 MB 2021-11-03.17:06:50
file.fastq 0.000 MB | 0.047 sec | 0 thr | 0.000 MB/s
1/2 - 50.00% of files done 0.000/0.000 MB - 0.00% of file sizes done
Processing file.fasta - 0.000 MB 2021-11-03.17:06:50
file.fasta 0.000 MB | 0.041 sec | 0 thr | 0.000 MB/s
add metadata ?(y/n): y
attribut (empty to stop) : client
value : MISTER_X
unit : Top_Secret
$ ils -r MY_PROJECT
/lbbeZone/home/gdebaecker/MY_PROJECT:
C- /lbbeZone/home/gdebaecker/MY_PROJECT/PROJECT_1
/lbbeZone/home/gdebaecker/MY_PROJECT/PROJECT_1:
file.fasta
file.fastq
With the GUI:
To get back data from irods you have to give a type ( [-f] for a file and [-C] for a folder ) and the local path where you want to download the data. If you don't give a path, the script will scan all your local data from the root-folder and ask you where you want to put it (it can take some time if you have many folder).
## PUT THE IRODS FOLDER "PROJECT_2" IN THE LOCAL FOLDER "MY_LOCAL_PROJECT"
$ ./easicmd.py pull -C MY_LOCAL_PROJECT/
ifolder (empty = /zone/home/user ): /lbbeZone/home/gdebaecker/MY_PROJECT/PROJECT_2
/lbbeZone/home/gdebaecker/irods_test
/lbbeZone/home/gdebaecker/irods_test/raw_data
/lbbeZone/home/gdebaecker/irods_test/raw_data/fast5
/lbbeZone/home/gdebaecker/MY_PROJECT
/lbbeZone/home/gdebaecker/MY_PROJECT/PROJECT_2
/lbbeZone/home/gdebaecker/NeGa
0/3 - 0.00% of files done 0.000/0.000 MB - 0.00% of file sizes done
Processing file_bis.r - 0.000 MB 2021-11-03.17:35:48
file_bis.r 0.000 MB | 0.036 sec | 0 thr | 0.000 MB/s
1/3 - 33.33% of files done 0.000/0.000 MB - 0.00% of file sizes done
Processing file.fasta - 0.000 MB 2021-11-03.17:35:48
file.fasta 0.000 MB | 0.029 sec | 0 thr | 0.000 MB/s
2/3 - 66.67% of files done 0.000/0.000 MB - 0.00% of file sizes done
Processing file.fastq - 0.000 MB 2021-11-03.17:35:48
file.fastq 0.000 MB | 0.027 sec | 0 thr | 0.000 MB/s
$ls MY_LOCAL_PROJECT
PROJECT_2/
## PUT THE CONTAIN OF IRODS FOLDER "PROJECT_2" IN THE LOCAL FOLDER "MY_LOCAL_PROJECT"
$ ./easicmd.py pull -C MY_LOCAL_PROJECT/
ifolder (empty = /zone/home/user ): /lbbeZone/home/gdebaecker/MY_PROJECT/PROJECT_2
irods file (tap tab) : %
file_bis.r 0.000 MB | 0.037 sec | 0 thr | 0.000 MB/s
file.fasta 0.000 MB | 0.030 sec | 0 thr | 0.000 MB/s
file.fastq 0.000 MB | 0.035 sec | 0 thr | 0.000 MB/s
$ls MY_LOCAL_PROJECT
file_bis.r file.fasta file.fastq
## DOWNLOAD ONE SPECIFIC FAST5 IRODS FILE IN MY CURRENT LOCAL FOLDER
$./easicmd.py pull -f .
ifolder (empty = /zone/home/user ): /lbbeZone/home/gdebaecker/irods_test/raw_data/fast5
irods file (tap tab) :FAL56006_29db37dd_251.fast5
0/1 - 0.00% of files done 0.000/181.788 MB - 0.00% of file sizes done
Processing FAL56006_29db37dd_251.fast5 - 181.788 MB 2021-11-03.17:42:38
From server: NumThreads=46, addr:lbbe-irods-local, port:20099, cookie=549953450
FAL56006_29db37dd_251.fast5 - 169.932/181.788 MB - 93.48% done 2021-11-03.17:42:38
FAL56006_29db37dd_251.fast5 - 181.788/181.788 MB - 100.00% done 2021-11-03.17:42:38
FAL56006_29db37dd_251.fas 181.788 MB | 1.709 sec | 46 thr | 106.344 MB/s
$ls FAL56006_29db37dd_251.fast5
FAL56006_29db37dd_251.fast5
With the GUI:
This command take as argument the path to a local folder you want to synchronize with irods and optionally a path in irods where you xwant to synchronise the folder. If the folder is not already on irods it will be created at the root (/zone/home/user) by default or where you give the path and synchronize using rsync. If you have already put the folder in irods (with push or synchro) you don't need to give an irods path as the script will find it and do the synchronisation . This command can be use when you create a new project and you know that it will often be modified. This command calculate the sha256 of the local file and look for their present in the icat, if they're not in the icat (the file doesn't exist yet in irods or had been edited in local) the file is sent to irods
This command has been written to be associated with when-changed : every time a change is detected in your folder by when-changed it runs the command synchro on your folder.
## SYNCHRONISE THE LOCAL FOLDER "Nanoplot" in irods folder "PROJECT_ASSELUS/ONT_READS"
$ ils
/lbbeZone/home/gdebaecker:
C- /lbbeZone/home/gdebaecker/NeGa
C- /lbbeZone/home/gdebaecker/PROJECT_ASSELUS
$ ls Nanoplot/
full_report.pdf QC_porechop
$ ./easicmd.py synchro /beegfs/home/gdebaecker/irule/Nanoplot /lbbeZone/home/gdebaecker/PROJECT_ASSELUS/ONT_READS/
$ ils ./PROJECT_ASSELUS/ONT_READS/Nanoplot/
/lbbeZone/home/gdebaecker/PROJECT_ASSELUS/ONT_READS/Nanoplot:
full_report.pdf
C- /lbbeZone/home/gdebaecker/PROJECT_ASSELUS/ONT_READS/Nanoplot/QC
## NOW WE MODIFY OUR LOCAL FOLDER AND WANT TO SYNCHRONISE IT WITH THE ONE IN IRODS
$ cp -r /beegfs/data/gdebaecker/QC* /beegfs/home/gdebaecker/irule/Nanoplot/
$ ls Nanoplot/
full_report.pdf QC QC_3kb QC_porechop
$ ./easicmd.py synchro /beegfs/home/gdebaecker/irule/Nanoplot/
$ ils ./PROJECT_ASSELUS/ONT_READS/Nanoplot/
/lbbeZone/home/gdebaecker/PROJECT_ASSELUS/ONT_READS/Nanoplot:
full_report.pdf
C- /lbbeZone/home/gdebaecker/PROJECT_ASSELUS/ONT_READS/Nanoplot/QC
C- /lbbeZone/home/gdebaecker/PROJECT_ASSELUS/ONT_READS/Nanoplot/QC_3kb
C- /lbbeZone/home/gdebaecker/PROJECT_ASSELUS/ONT_READS/Nanoplot/QC_porechop
When you want to create a new irods folder without knowing the full tree view. It had the "-p" option by default, so you can create parents folder while creating a folder.
$ ils -r
C- /lbbeZone/home/gdebaecker/irods_test
/lbbeZone/home/gdebaecker/irods_test:
final_summary_FAL56006_29db37dd.txt
C- /lbbeZone/home/gdebaecker/NeGa
C- /lbbeZone/home/gdebaecker/sr_aselus
$ ./easicmd.py imkdir
create : /lbbeZone/home/gdebaecker/irods_test/test_C-test/raw_test/raw_test2/new_folder
$ ils /lbbeZone/home/gdebaecker/irods_test/test_C-test/raw_test/raw_test2/
/lbbeZone/home/gdebaecker/irods_test/test_C-test/raw_test/raw_test2:
C- /lbbeZone/home/gdebaecker/irods_test/test_C-test/raw_test/raw_test2/new_folder
With the GUI:
When you no longer need a data on irods, or you need to make some place, you can remove them by using irm. This command take as argument the type of the irods data you want to remove and then with the autocompletion you can choose which data you want to remove. You can use "*" to remove several irods objects.
## REMOVE ALL THE ".r" file from the "MY_project" irods folder
$ ils ./MY_PROJECT/
/lbbeZone/home/gdebaecker/MY_PROJECT:
file_bis_2.r
file_bis_3.r
file_bis_4.r
file_bis_5.r
file_bis.r
file.fasta
file.fastq
C- /lbbeZone/home/gdebaecker/MY_PROJECT/PROJECT_2
$ ./easicmd.py irm -f
you can use * as wildcard
ifolder (empty = /zone/home/user ): /lbbeZone/home/gdebaecker/MY_PROJECT
irods file (tap tab) :*.r
$ ils ./MY_PROJECT/
/lbbeZone/home/gdebaecker/MY_PROJECT:
file.fasta
file.fastq
C- /lbbeZone/home/gdebaecker/MY_PROJECT/PROJECT_2
REMOVE THE "MY_PROJECT" irods folder
$ ils
/lbbeZone/home/gdebaecker:
C- /lbbeZone/home/gdebaecker/irods_test
C- /lbbeZone/home/gdebaecker/MY_PROJECT
C- /lbbeZone/home/gdebaecker/NeGa
C- /lbbeZone/home/gdebaecker/sr_aselus
$ ./easicmd.py irm -C
you can use * as wildcard
ifolder (empty = /zone/home/user ): /lbbeZone/home/gdebaecker/MY_PROJECT
$ ils
/lbbeZone/home/gdebaecker:
C- /lbbeZone/home/gdebaecker/irods_test
C- /lbbeZone/home/gdebaecker/NeGa
C- /lbbeZone/home/gdebaecker/sr_aselus
With the GUI:
Adding metadata to your irods objects is important and useful as irods is originally thought to be metadata based and not base on tree view. This metadata will be then used to search for your data on irods. If you didn't add metadata to your object when you put it in irods you can add then later on with add_meta. This command take as argument an option ([-f]/[-C]). If you don't give one, you will be asked to choose an option that can be [f] for an irods file and [C] for an irods folder, and then you can select your irods object with the autocompletion. It's a while loop, so you can add all your metadata at once, to stop it just left the attribute empty. You can use autocompletion for this command so if you reuse attribute it's more easy and less error-prone. If you want to gain time and doesn't need the autocompletion you can commant the line about attribute in ADD_META().
## ADD METADATA TO IRODS FOLDER "sr_aselus"
$ ./easicmd.py add_meta
add metadata to folder (C) or file (f) : C
ifolder (empty = /zone/home/user ): /lbbeZone/home/gdebaecker/sr_aselus
attribut (empty to stop) : technology
value : illumina
unit : 150pb
imeta add -C /lbbeZone/home/gdebaecker/sr_aselus technology illumina 150pb
attribut (empty to stop) : Species
value : Proasellus_coiffaiti
unit :
imeta add -C /lbbeZone/home/gdebaecker/sr_aselus Species Proasellus_coiffaiti
attribut (empty to stop) : Coverage
value : 200X
unit :
imeta add -C /lbbeZone/home/gdebaecker/sr_aselus Coverage 200X
attribut (empty to stop) :
$ imeta ls -C sr_aselus/
AVUs defined for collection /lbbeZone/home/gdebaecker/sr_aselus:
attribute: Species
value: Proasellus_coiffaiti
units:
----
attribute: technology
value: illumina
units: 150pb
----
attribute: Coverage
value: 200X
units:
With the GUI:
If you made a mistake while adding metadata, you can remove them with the command rm_meta. This command take as argument an option ([-f] or [-C]). If you did not give one, you will be asked to choose an option that can be [f] for an irods file and [C] for an irods folder, and then you can select your irods object with the autocompletion.
## REMOVE METADATA FROM IRODS FOLDER "sr_aselus"
## REMOVE THE "species" METADATA
$ imeta ls -C sr_aselus/
AVUs defined for collection /lbbeZone/home/gdebaecker/sr_aselus:
attribute: Species
value: Proasellus_coiffaiti
units:
----
attribute: technology
value: illumina
units: 150pb
----
attribute: Coverage
value: 200X
units:
$./easicmd.py rm_meta
remove metadata from folder (C) or file (f) : C
ifolder (empty = /zone/home/user ): /lbbeZone/home/gdebaecker/sr_aselus
attribut (empty for all): Species
$ imeta ls -C sr_aselus/
AVUs defined for collection /lbbeZone/home/gdebaecker/sr_aselus:
attribute: technology
value: illumina
units: 150pb
----
attribute: Coverage
value: 200X
units:
$./easicmd.py rm_meta
remove metadata from folder (C) or file (f) : C
ifolder (empty = /zone/home/user ): /lbbeZone/home/gdebaecker/sr_aselus
attribut (empty for all):
$ imeta ls -C sr_aselus/
AVUs defined for collection /lbbeZone/home/gdebaecker/sr_aselus:
None
With the GUI:
If you want to know the metadata associated with an irods object, you can use the command show_meta. This command takes as argument an option that can be [f] for an irods file and [C] for an irods folder, and then you can select your irods object with the autocompletion.
$ ./easicmd.py show_meta -C
ifolder (empty = /zone/home/user ): /lbbeZone/home/gdebaecker/sr_aselus
AVUs defined for collection /lbbeZone/home/gdebaecker/sr_aselus:
attribute: technology
value: illumina
units: 150pb
----
attribute: Species
value: Proassellus_coifaiti
units:
With the GUI:
SEARCH_BY_META : search for irods objects (folder/file) based on the metadata (not working yet with python api)
If you have associated metadata to your object, you can find them later by making query based on this metadata. The command search_by_meta take as argument an option that can be [f] for an irods file and [C] for an irods folder, and then ask you which attributes/values you want to query helped with the autocompletion. The script create a dictionary where the key are the attributes and the values are the values(metadata) associates with this attribute, so you can't ask for attributes that do not exist or values not associates with an attribute.
##SEARCH ALL THE IRODS FOLDER THAT HAVE THE ATTRIBUTE "technology" CREATED BEFORE 2020
./easicmd.py search_by_meta -C
attribute: technology
auteur
technology
Species
value (% as *): %
illumina
Nanopore
operation (like if your value contain % as a *): like
liaison (empty to stop): and
attribute: year
value (% as *): 2020
operation (like if your value contain % as a *): '<'
liaison (empty to stop):
collection: /lbbeZone/home/gdebaecker/irods_test/raw_data/fast5
----
collection: /lbbeZone/home/gdebaecker/sr_aselus
With the GUI:
If you forget to associated metadata to your object, you can find them later by making query based on their name or a part of it. The command search_name take as argument an option that can be [f] for an irods file and [C] for an irods folder, and then you will be asked to give your query where % can be use as a wildcard.
## FIND ALL THE ".fast5" FILES ON IRODS
$ ./easicmd.py search_name -f
your query(% as *) : %.fast5
/lbbeZone/home/gdebaecker/irods_test/raw_data/FAL56006_29db37dd_253.fast5
/lbbeZone/home/gdebaecker/irods_test/raw_data/fast5/FAL56006_29db37dd_250.fast5
/lbbeZone/home/gdebaecker/irods_test/raw_data/fast5/FAL56006_29db37dd_251.fast5
/lbbeZone/home/gdebaecker/irods_test/raw_data/fast5/FAL56006_29db37dd_252.fast5
/lbbeZone/home/gdebaecker/irods_test/raw_data/fast5/FAL56006_29db37dd_253.fast5
/lbbeZone/home/gdebaecker/irods_test/raw_data/fast5/FAL56006_29db37dd_25.fast5
/lbbeZone/home/gdebaecker/irods_test/test_C-test/raw_test/raw_test3/FAL56006_29db37dd_253.fast5
##FIND ALL THE "fastQ" FOLDER IN IRODS
$ ./easicmd.py search_name -C
your query (you can use *): *fastQ
/lbbeZone/home/gdebaecker/irods_test/raw_data/UNICORN_AND_DRAGON/fastQ
/lbbeZone/home/gdebaecker/NeGa/MY_2nd_PORJECT/TIGER/fastQ
/lbbeZone/home/gdebaecker/NeGa/MY_PROJECT/Proasellus/fastQ
With the GUI:
If you want to know the size of a folder on irods you can use idush. It makes the sum of the size of all the irods files inside an irods folder and print it. If you don't know if you will have enough place on your disk before the download, you can verify it with idush.
$ ./easicmd.py idush
ifolder (empty = /zone/home/user ): /lbbeZone/home/gdebaecker/irods_test/raw_data/fast5
/lbbeZone/home/gdebaecker/irods_test
/lbbeZone/home/gdebaecker/irods_test/raw_data
/lbbeZone/home/gdebaecker/irods_test/raw_data/fast5
913.7MiB
With the GUI:
With this command you can give (or remove with null) write/read/owner right to another iRODS user or group. You will be ask if you want to give the acces from a collection or a file and have to choose it. Then if you want to give acces to a user or a group and choose it with the autocompletion. Finally you will have to choose hich right you want to give (read/write/own/remove).
## GIVE WRITE PERMISSION OVER PROJECT_1 TO COLLABORATOR_1
$ ./easicmd.py ichmod
Give permission for a folder (C) or file (f) : C
ifolder (empty = /zone/home/user ): /lbbeZone/home/gdebaecker/Project_1
/lbbeZone/home/gdebaecker/fast5
/lbbeZone/home/gdebaecker/NeGa
/lbbeZone/home/gdebaecker/Project_1
group or user : user
give acces to : COLLABORATOR_1
MISTER_X
MJACKSON
COLLABORATOR_1
which right : write
read
write
own
remove/null
## REMOVE RIGHT OVER PROJECT_X TO TRAITOR GROUP
$ ./easicmd.py ichmod
Give permission for a folder (C) or file (f) : C
ifolder (empty = /zone/home/user ): /lbbeZone/home/gdebaecker/Project_X
/lbbeZone/home/gdebaecker/fast5
/lbbeZone/home/gdebaecker/NeGa
/lbbeZone/home/gdebaecker/Project_X
group or user : group
give acces to : TRAITOR
hufflepuff
ravenclaw
gryffindor
TRAITOR
which right : remove/null
read
write
own
remove/null
With the GUI:
The autocompletion dictionary is store in your home as a pickle file (.irods_metadata_local_save.pkl). It contain the python dictionary composed of your IRODS attributes as key and your IRODS values as ... value in a list. You can edit it with this function, you can create a new key/attribut and add values associated with it, modifying the list of value associated with an already existing key/attribute (remove or add value) and erase a key/attribut with all its associated value.
With the GUI:
CREATE A NEW KEY/ATTRIBUTEadd to the list of collection an additional path
MODIFY THE LIST OF VALUE OF AN ALREADY EXISTING ATTRIBUTE
ERASE A KEY/ATTRIBUTE AND ALL ITS VALUE
If you have several irods accounts (for example on several zones) you can add them in easicmd and switch from one to the other easily. As with the first use, you will have to fill in the configuration information for the irods account (host, port, user, zone and password).
$ ./api_easicmd.py new_user
creation d'un nouvel utilsateur
creating the irods config file in /home/gdebaeck/.easicmd_config/.easicmd.info (equivalent to irods_environment.json with icommand/iinit)
host: ccirods.in2p3.fr
port: 5596
user: gdebaeck
zone: tempZone
Checked out new branch 'gdebaeck_tempZone_5596'
ENTER YOUR IRODS PASSWORD : *************
Do you want to save it for later (the password will be encrypt and stock in /home/gdebaeck/.easicmd_config/.easicmd.psw [Y/N] : Y
create a file with all the irods folder to save time later
if you already have some irods folder it can take a bit of time
a list or your irods collection have been save
Dictionary have been save in /home/gdebaeck/.easicmd_config/.easicmd_irods_metadata_local_save.pkl
Initial commit created with specified files.
new user create
With the GUI:
$./api_easicmd.py switch_user
switch to (tab to list) :rods_tempZone_5590
gdebaeck_fits_5555
gdebaeck_tempZone_5590
rods_tempZone_5590
tga_tempZone_5590
switch to rods_tempZone_5590
With the GUI:
(this is done automatically when you use the version based on the python api, with this one all the folders/files on which you have rights are taken into account and not only those descending from your home folder when you use the script for the first time or when you click on update. )
If you're using an irods folder that isn't descendants of your irods home it won't show by default with easicmd. For that you need to add the full path to this folder by using the command easicmd.py add_path or use the add additional path wih the gui. These commands can be usefull if your working on a project with other people and have a commun folder not in your home. Additionally, if you have a shared folder, you can use this command to update the folder list. To improve performance, the folder list is stored in a .pkl file [~/.irods_collection_save.pkl] so that it doesn't have to be recalculated every time the command is run. However, this can be a limitation when multiple users need access to the same folder list. To overcome this limitation, you can create a symlink for each user from their local ~/.irods_collection_save.pkl file to a common file that can be accessed by all users.
With the GUI:
