BIIGLE
Replies: 20 comments 11 replies
-
|
Can you please describe again what interface you would like to have in the end? I vaguely remember that you want to add a new button to the volume overview and that there was a simple solution using the image label features of the volume overview and a more complex solution with entirely new views. Once I know again what you want, I'll describe how you can implement it. Did you also have a look at the tutorial about module develoment? |
Beta Was this translation helpful? Give feedback.
-
|
Ok, no problem, The page layout I want is similar to the Biigle annotation interface, the left is the picture display area, the right is the label selection area, after the label selection will appear a picture backward sliding button, the page I wrote a little with Vue roughly, almost like this: https://4frcy2-5173.csb.app/ |
Beta Was this translation helpful? Give feedback.
-
|
I see. If I remember correctly then you planned to integrate a deep learning method for substrate classification into BIIGLE. It could work as simply as pressing a button (in the volume overview), then the DL model does its magic and returns classification results for each image as image labels. Is this correct? Your description now sounds more like a new kind of annotation tool. Is this right? Your description and the example sounds very similar to what's currently possible with image labels in BIIGLE (both in the volume overview and the image annotation tool). |
Beta Was this translation helpful? Give feedback.
-
|
Of course, this page in URI is just an example to explain the elements in the page, and it will use the interface style that incorporates Biigle later |
Beta Was this translation helpful? Give feedback.
-
|
Ah yes, for the completeness of the module functionality, we added a training function that allows the user to manually label the images to train the network again and improve the generalization ability of the network. This is called label selection because our CNN can only recognize the labels specified by the network. I put all these specified labels into a label tree, and when the user chooses to use the training function for the labeling session, the system will automatically retrieve the contents of this label tree to show to the user for them to select the appropriate labels from these labels. for the image annotation, as you can see in the example I provided |
Beta Was this translation helpful? Give feedback.
-
|
Should the training be also done "inside" BIIGLE (like MAIA does)? In our meeting I got the impression that a retraining is not planned/possible. Can't you use the image labelling tools of BIIGLE to generate new training data? This way you don't have to implement an entirely new tool. I suggest to target a "minimum viable product" (MVP) for the module at first. This means that you can press a button in BIIGLE that starts the classification method. The classification results are attached as image labels to the images. When the classification task is finished, the user receives an email notification. This will already be lots of work. Once the MVP is ready, you can still continue and extend it with new features/views/tools. |
Beta Was this translation helpful? Give feedback.
-
|
Yeah, I agree with you, I will start firstly with MVP. Thank you! |
Beta Was this translation helpful? Give feedback.
-
|
Sounds good! Please feel free to ask any questions about how to implement this. Most of what you need is already implemented in the MAIA module so you will be able to copy lots of code from there. |
Beta Was this translation helpful? Give feedback.
-
|
Hello, I'm currently facing an issue with integrating Laravel and Python, specifically regarding how Laravel can determine if Python scripts have completed execution. Currently, my tasks are running continuously, and I am unable to proceed to the next step. However, I can see the results in a JSON file located in the "tmp" path. |
Beta Was this translation helpful? Give feedback.
-
|
Here is an example how Python is called in MAIA: protected function python($command, $log = 'log.txt')
{
$code = 0;
$lines = [];
$python = config('maia.python', '/usr/bin/python3');
$logFile = "{$this->tmpDir}/{$log}";
exec("{$python} -u {$command} >{$logFile} 2>&1", $lines, $code);
if ($code !== 0) {
$lines = File::get($logFile);
throw new Exception("Error while executing python command '{$command}':\n{$lines}", $code);
}
return $logFile;
}The method itself does not expect any input arguments or return any ouput from the Python script. Instead, the PHP code generates input JSON files, passes the paths to the files as command line arguments and the Python script generates output JSON files at locations specified in the input JSON files, so PHP can read them when the command is finished. |
Beta Was this translation helpful? Give feedback.
-
|
The python script are very simple like this, it is just for a test. images = params['images'] |
Beta Was this translation helpful? Give feedback.
-
|
Ok, I understand, thank you |
Beta Was this translation helpful? Give feedback.
-
|
Hello, |
Beta Was this translation helpful? Give feedback.
-
|
because the program is still at "Label recognition in progress. |
Beta Was this translation helpful? Give feedback.
-
|
protected function parseLabels($images) |
Beta Was this translation helpful? Give feedback.
-
|
here is my codes to readthe file, I am not sure whether if it dosen't work, because these path for files already existed in 'tmp' |
Beta Was this translation helpful? Give feedback.
-
|
Can I see all the code in a repo somewhere? Also, please describe the steps you do (e.g. clicks in the UI, code executed manually) and what you expect as a result. While I understand that it might seem too much work for experiments and if you ae not that experienced with web development yet, I can only recommend to write tests for your (PHP) code. This way, you can test smaller parts of the codebase before you plug everything together. Errors are much easier to find in the smaller parts than in the whole thing. Also, I'm on vacation for three weeks beginning tomorrow, so unfortunately I can't answer during that time. |
Beta Was this translation helpful? Give feedback.
-
|
Hello, |
Beta Was this translation helpful? Give feedback.
-
|
Hello, I'm happy to tell you that all the code for the interface and the database is working fine now, there is just one problem left, which is how to integrate the docker for the python environment used for label recognition into the application. Also, another problem is that the weight files for this network are up to 50 gigabytes, so maybe we need to find a place on the server to store them. In my localhost biigle application, the python executor is a very simple version that doesn't even run maia properly, and I'm just using very simple python code right now to make sure to run all the interfaces and databases without problems. So I would like to ask some questions about python environment and docker integration, thank you very much for your answer. In the meantime, you can find all the updated code at the github link: https://github.com/Zuoyu2524/abysses. Wish you happy holidays! |
Beta Was this translation helpful? Give feedback.
-
|
MAIA has a storage path for weight files configured like this. The files are loaded from the "urls" and stored locally at the "paths". In the job to perform the training, the files are downloaded if they are not already there. If you want to execute the Python code locally, you can add the GPU worker image of the production configuration to your local gpu-worker:
image: biigle/gpu-worker
build:
context: ./.docker/
dockerfile: gpu-worker.dockerfile
user: ${USER_ID}:${GROUP_ID}
depends_on:
- app
- database_testing
tmpfs:
- /tmp
volumes_from:
- app
ipc: host # Required for Torch multiprocessing
environment:
- "DB_PORT=5432"
init: true
command: "php -d memory_limit=1G artisan queue:work --sleep=5 --tries=1 --timeout=0 --queue=gpu"Run Please note that the Docker Compose configuration above does not actually expose the GPU to the Docker container (because I don't have one locally). So processing will be done on the CPU. If you want to use a local GPU you have to install NVIDIA Docker and add something like this to your |
Beta Was this translation helpful? Give feedback.
-
|
Hello, ok, thank you very much, I will try this! |
Beta Was this translation helpful? Give feedback.
-
|
Hello |
Beta Was this translation helpful? Give feedback.
-
|
I got the same error just yesterday. This change will fix it: biigle/biigle@89b92ad |
Beta Was this translation helpful? Give feedback.
-
|
Hello, |
Beta Was this translation helpful? Give feedback.
-
|
That's unfortunate, as BIIGLE recently moved away from TensorFlow. Now we use PyTorch as you probably discovered. Didn't we talk about this in our video call? I think so but I don't remember if we identified this problem then. TensorFlow 1 also has lots of security vulnerabilities and is quite outdated. If you can't move away from TensorFlow you have to build your own Docker image. You also have to deploy it on a separate GPU (if you want to use the other GPU-based modules of BIIGLE, too, in your production instance). Here you can find the last MAIA Docker image that was based on TensorFlow 1 (from almost 3 years ago). Maybe you can use it as a starting point. But be warned that the TensorFlow and PHP versions as well as Python dependencies are outdated and probably have many security vulnerabilities! |
Beta Was this translation helpful? Give feedback.
-
|
Hello, |
Beta Was this translation helpful? Give feedback.
-
|
Hello, I have a little question that I can now successfully build the gpu-docker, but the peoblem is in biigle application, when running the scripts, the core didn't use the gpu-worker, so not deep learning environments, I try to make some changes to config but no success, the logs of docker show bellow: |
Beta Was this translation helpful? Give feedback.
-
|
It says that it could not detect a GPU. Is this correct? Otherwise you may have to install the NVIDIA Container Toolkit. Also make sure that you have a resource definition like this one for the service. Usually TensorFlow falls back to CPU computation without a GPU so this may or may not be the cause of the error. A failed job probably puts error messages in the |
Beta Was this translation helpful? Give feedback.
-
|
I now saw that the In some setups I used a |
Beta Was this translation helpful? Give feedback.
-
|
Yes, you're right, I really don't have a GPU configuration on my computer at the moment, that should be the problem, thank you! |
Beta Was this translation helpful? Give feedback.
-
|
I don't know, TensorFlow usually falls back to the CPU in that case. Please read my second comment regarding queues. |
Beta Was this translation helpful? Give feedback.
-
I tried many examples of common Blade templates interacting with Vue JS, but was not able to successfully bring up the desired interface, and the approach I used was as follows:
Beta Was this translation helpful? Give feedback.
All reactions