Dockerize genomics-workspace

[deming7h777]

docker is very useful to rebuild the environment and whole configuration so I want to apply it to our genomics-workspace.

reference : 1. https://github.com/twtrubiks/docker-tutorial
2. https://docs.docker.com/get-started/

[deming7h777]

http://www.yamllint.com/

useful tool to check yml file

[deming7h777]

It's docker-compose.yml I made for docker application

[deming7h777]

I also create the other file name dockerfile, its function looks like vagrantfile while building the docker it will process the command in this file.

[deming7h777]

however it shows error after running "docker-compose up"

[bradfordcondon]

yaml can be annoying. Looking at yours, I see a whole bunch of problems, you have to be very careful with the indenting, new lines, -s etc. Compare it to the apollo one i wrote here: https://github.com/NAL-i5K/tripal_apollo/blob/master/docker-compose.yml

note for example

volumes:
 - .:/something

vs

volumes: pgdata:/something 

ie, different line, missing the dash. This is why its saying it isnt an array, its not defined as an indented set of -'d lines.

[hsiaoyi0504]

Sorry... @deming7h777 @bradfordcondon Can I ask why we are rebuilding this ? As far as I know, there is a repo based on docker for genomics-workspace in our gitlab organization. If we want to update to newer version. That will be a good starting point.

[deming7h777]

'''
genomicsworkspace_workspace_1 exited with code 2
workspace_1 | python: can't open file 'manage.py': [Errno 2] No such file or directory
'''
after build the four images for docker i encounter the problem that i can'y access manage.py

and i also push the branch call genomics-docker
https://github.com/NAL-i5K/genomics-workspace/tree/genomics_docker

[hsiaoyi0504]

Do you forget to clone the submodule ?

[deming7h777]

Do you forget to clone the submodule ?

What kind of submodule you mean ?

[hsiaoyi0504]

git submodule, check if your i5k directory is empty or not first.

[deming7h777]

It's not empty.
I also find that your i5k docker had one submodule but i don't know what's its usage.

[hsiaoyi0504]

Can you provide the result when you run the tree command ? It's hard for anyone to help you without providing further detailed information and I am confused about the error you encountered here is that on your branch or the internal i5k-docker repo.

[deming7h777]

Ah, I can try it myself first, thanks for your kindness !
The error I encountered was on my branch, what I want to do is updating the old version to docker-compose

[hsiaoyi0504]

Got it ! I am not sure what you mean old version here. Do you mean the internal i5k-docker repo ?

[deming7h777]

branch link : https://github.com/NAL-i5K/genomics-workspace/blob/genomics_docker/docker-compose.yml

[deming7h777]

/bin/sh: ./configure: No such file or directory
ERROR: Service 'workspace' failed to build: The command '/bin/sh -c ./configure --prefix=/usr/local --enable-unicode=usc4 --enable-shared LDFLAGS="-WL,-rpath /usr/local/lib" && make && make altinstall' returned a non-zero code: 127

now I'm trying to figure out linux problem when I try to build cutomize image for workspace

[deming7h777]

configure: error: in `/genomics-workspace/Python-2.7.15':
configure: error: C compiler cannot create executables
See `config.log' for more details
ERROR: Service 'workspace' failed to build: The command '/bin/sh -c cd Python-2.7.15 &&     ./configure --prefix=/usr/lo
cal --enable-unicode=usc4 --enable-shared LDFLAGS="-WL,-rpath /usr/local/lib" &&     make &&     make altinstall' return
ed a non-zero code: 77

I change the code in Dockerfile from WORKDIR Python-2.7.15 to RUN cd Python-2.7.15 to solve the above problem.

In the current issue, it seems I need to figure out the problem of c compiler

[deming7h777]

i change it to this and the build proceeds to `RUN yum groupinstall "Development tools" -y && yum install wget -y` and fails
because yum isnt installed
what exactly is your question

[childers]

I get an error on step 11:

$ docker-compose up
Creating network "genomics-workspace_default" with the default driver
Building workspace
Step 1/13 : FROM centos:6.10
 ---> 30e66b619e9f
Step 2/13 : LABEL maintainer Deming
 ---> Using cache
 ---> 2fda5d15f394
Step 3/13 : ENV PYTHONBUFFERED 1
 ---> Using cache
 ---> 52eba52b63c3
Step 4/13 : VOLUME /data
 ---> Using cache
 ---> ed9ad883dfbe
Step 5/13 : RUN mkdir /genomics-workspace
 ---> Using cache
 ---> 7a04d699d6fe
Step 6/13 : WORKDIR /genomics-workspace
 ---> Using cache
 ---> fb278825b72d
Step 7/13 : COPY requirements.txt /genomics-workspace/
 ---> Using cache
 ---> c42e9156239c
Step 8/13 : RUN yum groupinstall "Development tools" -y && yum install wget -y
 ---> Using cache
 ---> 94e2bc2416b5
Step 9/13 : RUN wget http://www.python.org/ftp/python/2.7.15/Python-2.7.15.tar.xz && tar -xf Python-2.7.15.tar.xz
 ---> Using cache
 ---> 7f26b413d79e
Step 10/13 : WORKDIR /Python-2.7.15
 ---> Using cache
 ---> a2a50cbaa3d9
Step 11/13 : RUN yum install automake && yum install autoconf && autoreconf -i && ./configure --prefix=/usr/local --enable-unicode=usc4 --enable-shared LDFLAGS="-WL,-rpath /usr/local/lib" && make && make altinstall
 ---> Running in b8fea31fcd41
Loaded plugins: fastestmirror, ovl
Setting up Install Process
Loading mirror speeds from cached hostfile
 * base: mirror.umd.edu
 * extras: mirror.umd.edu
 * updates: mirror.umd.edu
Package automake-1.11.1-4.el6.noarch already installed and latest version
Nothing to do
Loaded plugins: fastestmirror, ovl
Setting up Install Process
Loading mirror speeds from cached hostfile
 * base: mirror.umd.edu
 * extras: mirror.umd.edu
 * updates: mirror.umd.edu
Package autoconf-2.63-5.1.el6.noarch already installed and latest version
Nothing to do
autoreconf: `configure.ac' or `configure.in' is required
ERROR: Service 'workspace' failed to build: The command '/bin/sh -c yum install automake && yum install autoconf && autoreconf -i && ./configure --prefix=/usr/local --enable-unicode=usc4 --enable-shared LDFLAGS="-WL,-rpath /usr/local/lib" && make && make altinstall' returned a non-zero code: 1

[deming7h777]

Complete!
checking build system type... x86_64-pc-linux-gnu
checking host system type... x86_64-pc-linux-gnu
checking for python2.7... no
checking for python3... no
checking for python... python
checking for --enable-universalsdk... no
checking for --with-universal-archs... no
checking MACHDEP... linux2
checking EXTRAPLATDIR...
checking for --without-gcc... no
checking for --with-icc... no
checking for gcc... gcc
checking whether the C compiler works... no
configure: error: in `/genomics-workspace/Python-2.7.15':
configure: error: C compiler cannot create executables
See `config.log' for more details
ERROR: Service 'workspace' failed to build: The command '/bin/sh -c cd Python-2.7.15 &&     yum install -y gcc gcc-c++ zlib-devel gdbm-devel readline-devel libffi-devel sqlite-devel &&    ./configure --prefix=/usr/local --enable-unicode=usc4 --enable-shared LDFLAGS="-WL,-rpath /usr/local/lib" &&     make &&     make altinstall' returned a non-zero code: 77

[deming7h777]

It's great to solve the c compiler problem above then now I'm going to link

[deming7h777]

FROM centos:6.10
LABEL maintainer Deming
ENV PYTHONBUFFERED 1
RUN mkdir /genomics-workspace
WORKDIR /genomics-workspace
COPY  . /genomics-workspace
RUN yum -y groupinstall "Development tools" && \
    yum -y install zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel && \
    yum -y install readline-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-devel python-devel && \
    yum install wget -y
RUN wget http://www.python.org/ftp/python/2.7.15/Python-2.7.15.tar.xz && \
    tar -xf Python-2.7.15.tar.xz
RUN cd Python-2.7.15 && \
    ./configure --enable-optimizations  && \
    make && \
    make altinstall && \
    wget https://bootstrap.pypa.io/ez_setup.py && \
    /usr/local/bin/python2.7 ez_setup.py && \
    wget https://bootstrap.pypa.io/get-pip.py && \
    /usr/local/bin/python2.7 get-pip.py
RUN pip install -r requirements.txt

FROM python:2.7-slim

RUN mkdir /genomics-workspace

WORKDIR /genomics-workspace

COPY . /genomics-workspace/

RUN pip install -r requirements.txt

I tried these two customize dockerfile to build my own genomics-workspace image
but it turn out the same error for "can't file the file name manage.py"

[bradfordcondon]

the docker-compose runs without error on my system after:

• replacing the port mapping for the postgres container with

  expose:
        - "5432"

as port 5432 is in use on my machine.

• additionally, changing the settings.py file HOST settings from localhost to postgres as docker makes the containers available under whatever you name that as.

So I no longer get db connection errors etc from the workspace container, meaning its hooked up right... but, i get ERR_EMPTY_RESPONSE at the web address. I dont know anything about the web stack there, perhaps apache or whatever is running needs to be restarted?

[bradfordcondon]

warning: my knowledge of Django is nonexistant.

you need to supply the IP address in the run command, like so

command: python manage.py runserver 0.0.0.0:8000

now, instead of an empty response error, i get a routing type error from Django, yay!

So from this error i can see that we only define certain paths. so if i try to visit, say, http://localhost:8000/blast/ instead, i get this error that shows me stuff is definitely happening, but perhaps not configured all the way:

the admin address also doesnt work:

ProgrammingError at /admin/login/
relation "django_site" does not exist
LINE 1: ..."django_site"."domain", "django_site"."name" FROM "django_si...
                                                             ^

so looks to me like the django site isnt configured and perhaps additional setup commands or config files need to be done.

[hsiaoyi0504]

@bradfordcondon You are right. We didn’t define root path and the error message means no datatable has been created. I guess python manage.py migrate hasn’t been included into the setup steps.

[bradfordcondon]

the only command run is the server start command specified in the docker-compose. You all probably need to run a full script that runs each command necessary to setting up and launching the django site.

[deming7h777]

I didn't put migrate commend in the dockerfile because I thought we can build the container first then do the migrate commend directly in container.

[deming7h777]

Just a minor suggestion here. The file permissions are natively saved in git. Then, we can directly modify file permission in git history rather than fixing it every time we want to use them.

That's really good idea so I just need to modify the permission in git history in advance.
Thanks for the awesome suggestion!

[deming7h777]

By the way try docker toolbox rather than desktop docker :https://docs.docker.com/toolbox/toolbox_install_mac/
in mac

[bradfordcondon]

a note- docker toolbox is only recommended for machines that dont meet the system requirements for docker desktop. In general its use is discouraged I think.

[bradfordcondon]

my current status:

Starting genomics-workspace_rabbitmq_1 ... done
Starting postgres                      ... done
Starting genomics-workspace_memcache_1 ... done
Starting genomics-workspace_workspace_1 ... error

ERROR: for genomics-workspace_workspace_1  Cannot start service workspace: OCI runtime create failed: container_linux.go:344: starting container process caused "exec: \"run_web.sh\": executable file not found in $PATH": unknown

ERROR: for workspace  Cannot start service workspace: OCI runtime create failed: container_linux.go:344: starting container process caused "exec: \"run_web.sh\": executable file not found in $PATH": unknown
ERROR: Encountered errors while bringing up the project.

my version:

docker-compose version 1.23.2, build 1110ad01

theres an update available (not docker-compose, 1.23.2 is the latest full release of compose)

[bradfordcondon]

docker/compose#3876 (comment)

this looks likely to me. windows stile line endings in the script.

edit- nope. i ran dos2unix on all files in the project and that didnt resolve.

[bradfordcondon]

ok got it @deming7h777

command: sh run_web.sh works, as does command: ["sh", "run_web.sh"]

as its written, command: ['run_web.sh'] doesnt work.

[deming7h777]

For this issue, I solve it by modify the EOL covertion from CR LF to LF

[deming7h777]

Most of windows user need to install docker toolbox instead of desktop of docker due to the window's issue, ig, Windows 10 64bit: Pro, Enterprise or Education (1607 Anniversary Update, Build 14393 or later) is required.
@mpoelchau 's issue

1. use docker logs [container ID] to check db
2. try to remove [depends_on: ] in docker-compose.yml to restart
3. install using toolbox instead of desktop version

@bradfordcondon 's issue

1. I think Monica should have the same issue with you due to the same environment you have, but it didn't.

[mpoelchau]

Seems to work well for windows, possibly for Mac with some modifications. Closing this until @childers can test later in the year.

[mpoelchau]

Re-opening - we need to document this properly first.

[mpoelchau]

@deming7h777 did you change the setup so that run_web.sh is already executable once you clone the repo?

If that's not possible, can you change the mac documentation to include instructions to do this: sudo chmod +x ./run_web.sh

Once that's done, I think you can create a PR for the documentation (sphinx_doc branch) - perhaps @tony006469 or @r06942072 can test the PR.

[deming7h777]

@deming7h777 did you change the setup so that run_web.sh is already executable once you clone the repo?

If that's not possible, can you change the mac documentation to include instructions to do this: sudo chmod +x ./run_web.sh

Once that's done, I think you can create a PR for the documentation (sphinx_doc branch) - perhaps @tony006469 or @r06942072 can test the PR.

I have not only made the change to run_web.sh and waiting for test but also add the instruction "sudo chmod +x ./run_web.sh"

[deming7h777]

There are some solutions suggested: you can use a separate tool to do it for you, use a proper combination of user account and umask to set them properly by default or write a git hook yourself to do it. A hook would've to be installed on the user doing the checkout.

I make two files called post-checkout and post-merge in genomics-workspace/.git/hooks/
to fix the premission problem

Git is not really a deployment tool and should not be used as such. It is a version control system for source code

[hsiaoyi0504]

I don't quite understand the point @deming7h777 state here. Do you mean that we should not directly change the permission of the file in git repo ?

[deming7h777]

I still change the permission file in git repo used .git/hook/post-checkout and .git/hook/post-merge but I also see some suggestion that disagree to use git repo to do it

[hsiaoyi0504]

@deming7h777 Can you post your reference here ?
Even if user can change the permission of files in git repo by using same mechanism through git-hook, files under .git/hook/ are not traced by the git repo. In my opinion, having another step in setup like setting up git hook is not good.

[deming7h777]

ah I see, so what' s your opinion for "we can directly modify file permission in git history rather than fixing it every time we want to use them." you just suggest

and I think the permission issue I have encounter now is like this one : https://medium.com/@akash1233/change-file-permissions-when-working-with-git-repos-on-windows-ea22e34d5cee
for git on windows

reference :
1.https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks
2.https://stackoverflow.com/questions/14557106/git-changes-my-file-permissions-upon-checkout
3.https://stackoverflow.com/questions/14557106/git-changes-my-file-permissions-upon-checkout

[hsiaoyi0504]

@deming7h777 Thanks. I traced whole thread again and it seems to me that the fundamental reason of this issue is that on Windows (and only on Windows), git clone will lost the correct file permissions saved in the git repo (but as I remember, it can be configured to change this behavior). Then, when building docker image, files that are without correct permissions are copied into built image. In the end, the execution that relies on linux file permission will fail.

Therefore, simply change the execution command into ["sh", "run_web.sh"] should work.
That means, don't rely on shebang and file permission of the file. (and we don't need to chmod +x run_web.sh in Dockerfille, either)

@bradfordcondon What kind of OS you use for testing this ? I think you use Windows, right ?

[bradfordcondon]

hi @hsiaoyi0504 im on OSX.

Therefore, simply change the execution command into ["sh", "run_web.sh"] should work.

this is the solution i proposed above, which does work for me on OSX (when I posted it, this branch might have changed since then).

as far as a windows solution, i linked this in slack. Seems like if you are using window's git bash, there are tools for changing the file permissions (git update-index --chmod=+x 'name-of-shell-script')

[hsiaoyi0504]

Hi @bradfordcondon, thanks. Yeah, I just copy your solution :>)

By default, git clone will not lost the permission on Mac. My assumption is that, when developing on Windows, the file permissions were lost and were committed into git, or weren't commited into git initially.

[hsiaoyi0504]

As far as 592de71, the file permission of run_web.sh is -rw-r--r--.

[deming7h777]

To solve the layout issue for blast home page
try to build another customize dockerfile base on centos:6.10 image
This dockerfile version contains (1) epel-release (2) python package (3) nodejs
but encounter the "can't find manage.py module " again
and try to figure it out

[deming7h777]

1. Put the command /usr/local/bin/python2.7 manage.py [migrate/collectstatic/runserver] in the dockerfile
   *encounter the could not translate host name "postgres" to address: Name or service not known
2. Put the command into the script and configure the script to docker-compose.yml
   *encounter the manage.py missing issue

[mpoelchau]

@deming7h777 do you think it's possible to make a PR of the sphinx_docker branch, have someone test it, and merge before AGS? Would be nice to have the docker documentation available by then.

[deming7h777]

@mpoelchau I think it is ok, I will work on it