Scripting access of django databases

[deming7h777]

Like we discussed today, perhaps we don't need to have APIs.
Here are some possible options. I would update if I have more information:

1.using python manage.py shell < myscript.py: https://stackoverflow.com/a/16853799/5363040

2.using RunScript of django-extensions.
See an example here: https://github.com/django-extensions/django-extensions/blob/master/docs/runscript.rst.

The underlying mechanism is the same as 1, but based on a python package.

3.use loaddata, dumpdata of django commands: https://www.creativesoftware.com/blog/8-useful-django-dumpdata-and-loaddata-commands. It's the mechanisms used by Han Lin before and there are lots of scripts/command weren't maintained for a long time: https://github.com/NAL-i5K/genomics-workspace/tree/e9d355bc51118d3f7a4b56b94717d740853bf6f6/util.

django customize manage.py command :https://docs.djangoproject.com/en/1.11/howto/custom-management-commands/

[deming7h777]

Current effort is saved in this PR: #216, but haven't finished yet.
by Hsiao Yi

[deming7h777]

I think this will be a good way to implement this feature:
https://docs.djangoproject.com/en/1.11/howto/custom-management-commands/
For example, we can implement a command to do something like:
python manage.py add_organism -i config.json
to add an organism (with information written in config.json)

[deming7h777]

success to make an demo.py for customize django manage.py
step 1. adding the two folder under the specific app (in this case put it in /genomics-workspace/app/)
step 2. remember to make the init.py for each folder that python can make it as the package
step 3. the customize command category must inherit BaseCommand and the function name need to be "handle"

[deming7h777]

Complete the feature : automatically generate the short_name
Now working on web scraping tutorial and finished the demo part.(scrape the S&P 500 value from bloomberg website, the problem that I have encountered was the bloomberg website rejected my request because it treated me as a robot.
I solve it by using the "fake header" in my script

[deming7h777]

Complete the whole progress to add database by script, I push to our repo and encounter several issue

1. Django 1.11.15 has known security vulnerabilities. It suggests that we should update to 1.11.19
2. it analyzes I import several module which was useless, however I indeed use those package.
3. To integrate to the CWL project it seems that we need to host the genomics-project to CWL project due to the command we use for "python manage.py" . It will make our CWL really huge so we need to figure it out if there are the other solutions.

[mpoelchau]

Closing in favor of #280.