Docker image for Overpass #374

wiktorn · 2017-03-12T22:16:58Z

Based on NGINX image. Uses supervisord to run:

nginx
fastcgi wrapper
Overpass dispatcher
fetch_osc.sh
apply_osc_to_db.sh

On first run initializes itself with the data. Details provided in README_Docker.md.

This depends on changes in: #373. apply_osc_to_db.sh may start before dispatcher is ready. Thanks to changes in #373 it will die and start again, when dispatcher is fully working.

Changes in src/bin/fetch_osc.sh skips the download of files already available locally on restart of the container.

Based on NGINX image. Uses supervisord to run: * nginx * fastcgi wrapper * Overpass dispatcher * fetch_osc.sh * apply_osc_to_db.sh On first run initializes itself with the data. Details provided in README_Docker.md.

mmd-osm · 2017-03-13T08:37:48Z

Thanks! This needs some cross checking with what has already been done in #318 and the +/- 5 other Docker images proposed so far. One example is the topic of area creation and running an area dispatcher.

wiktorn · 2017-03-13T09:18:02Z

I was using some of the Dockerfiles as inspiration. The only part that I'm aware, that's missing is:

area creation
attic support

If for areas support I need two extra processes:

$EXEC_DIR/bin/dispatcher --areas --db-dir=$DB_DIR
EXEC_DIR/bin/rules_loop.sh $DB_DIR

And populate $DB_DIR with src/rules on initialization, then it looks like an easy way forward.

mmd-osm · 2017-03-13T11:31:58Z

Compression settings could also be interesting beyond area and attic support.

Add liblz4 to image so Overpass will support additional compression algorithm. Add two processes: * dispatcher_areas - for calculating areas * areas_rules - rules loop Added src/rules to databases directory Added OVERPASS_COMPRESSION environment variable Added `attic` value to OVERPASS_META variable

…ocker_image

wiktorn · 2017-03-14T00:04:41Z

I've just added area creation, attic support.

I've tried added compression but though i added --enable-lz4 and it looks like it's properly identified, bin/update_database doesn't report --compression=lz4.
configure log:

checking lz4.h usability... yes
checking lz4.h presence... yes
checking for lz4.h... yes
checking for LZ4_compress in -llz4... yes

$ cat /app/src/config.h | grep LZ4
#define HAVE_LZ4 1

$ /app/bin/update_database -h
Unkown argument: -h
Usage: /app/bin/update_database [--db-dir=DIR] [--version=VER] [--meta|--keep-attic] [--flush_size=FLUSH_SIZE] [--compression_method=(no|gz)]

mmd-osm · 2017-03-15T11:52:55Z

bin/update_database doesn't report --compression=lz4.

I think that's a bug in the coding (

Overpass-API/src/overpass_api/osm-backend/update_database.cc

Line 114 in a3ab379

    
           std::cerr<<"Usage: "<<argv[0]<<" [--db-dir=DIR] [--version=VER] [--meta|--keep-attic] [--flush_size=FLUSH_SIZE]"

): the help/usage text doesn't correctly reflect the available compression options. It should be similar to line 97 in update_database.cc.

Obviously, the parameter itself is also incorrect: --compression_method is given in the usage text, while the coding expects --compression-method . And of course, there's also a --map-compression-method parameter, which is not mentioned at all. At this time the usage text is not really that helpful.

As mentioned in drolbr#374 (comment) I'm adding missing lz4 information.

mmd-osm · 2017-03-17T14:11:51Z

docker-entrypoint.sh

+            META="--meta=$OVERPASS_META"
+        fi
+        mkdir -p /db/db \
+        && /app/bin/download_clone.sh --db-dir=/db/db --source=http://dev.overpass-api.de/api_drolbr/ $META "--compresion_method=$OVERPASS_COMPRESSION" \


download_clone.sh just downloads existing files from dev.overpass-api.de with a predefined compression setting, i.e. there's not much point to provide a compression method parameter, as it will not have any impact.

Your documentation above correctly mentions that this only works in init mode, though.

mmd-osm · 2017-03-17T14:12:51Z

docker-entrypoint.sh

+
+    if [ "$OVERPASS_MODE" = "init" ]; then
+        lftp -c "get -c \"$OVERPASS_PLANET_URL\" -o /db/planet; exit" \
+        && /app/bin/init_osm3s.sh /db/planet /db/db /app "--meta=$OVERPASS_META" \


compression setting is better suited here.

@mmd-osm

`OVERPASS_COMPRESSION` makes sense only for init_osm3s not for download_clone. Reported by: @mmd-osm

wiktorn · 2017-03-17T14:52:23Z

God catch, thanks.

mmd-osm · 2017-03-17T14:55:18Z

docker-entrypoint.sh

@@ -23,7 +23,7 @@ if [ ! -d /db/db ] ; then

    if [ "$OVERPASS_MODE" = "init" ]; then
        lftp -c "get -c \"$OVERPASS_PLANET_URL\" -o /db/planet; exit" \
-        && /app/bin/init_osm3s.sh /db/planet /db/db /app "--meta=$OVERPASS_META" \
+        && /app/bin/init_osm3s.sh /db/planet /db/db /app "--meta=$OVERPASS_META" "--compresion_method=$OVERPASS_COMPRESSION" \


... and the parameter should be --compression-method= with a dash (not an underscore, the usage text is wrong) and two "s" in compression.

Ok, I'll create a seperate pull request for usage text

commit f2560c7 Author: W <[email protected]> Date: Fri Mar 17 16:14:07 2017 +0100 Add --map-compression-method to usage message commit c71d26e Author: W <[email protected]> Date: Fri Mar 17 16:00:12 2017 +0100 Fix spelling of parameter name in usage message commit 122ea06 Author: W <[email protected]> Date: Thu Mar 16 22:56:00 2017 +0100 Fix help message for LZ4 compilation. As mentioned in drolbr#374 (comment) I'm adding missing lz4 information.

wiktorn added 2 commits March 12, 2017 23:10

Docker image for Overpass

95c2b48

Based on NGINX image. Uses supervisord to run: * nginx * fastcgi wrapper * Overpass dispatcher * fetch_osc.sh * apply_osc_to_db.sh On first run initializes itself with the data. Details provided in README_Docker.md.

Add quotations around variable name

0e6f3f7

wiktorn and others added 6 commits March 14, 2017 00:30

Merge remote-tracking branch 'upstream/master' into docker_image

c82af11

Fixed bad resource management in out:popup

0debd98

Fixed further bug in out:popup

1553a91

Merge branch 'docker_image' of github.com:wiktorn/Overpass-API into d…

083a2de

…ocker_image

Add --enable-lz4 to configure script

124bac4

wiktorn added a commit to wiktorn/Overpass-API-old that referenced this pull request Mar 16, 2017

Fix help message for LZ4 compilation.

122ea06

As mentioned in drolbr#374 (comment) I'm adding missing lz4 information.

wiktorn added 2 commits March 16, 2017 22:59

Copy areas rules to database folder

bc55c48

Add rules load to supervisord

47f2063

mmd-osm reviewed Mar 17, 2017

View reviewed changes

Fix passing compression algorithm.

92c015a

`OVERPASS_COMPRESSION` makes sense only for init_osm3s not for download_clone. Reported by: @mmd-osm

mmd-osm reviewed Mar 17, 2017

View reviewed changes

Fix spelling of parameter

9bebfc6

wiktorn added 4 commits July 27, 2017 10:13

Add gecos parameter to adduser

ab7004a

Merge remote-tracking branch 'upstream/master' into docker_image

7e3fbfd

Add documentation about OVERPASS_RULES_LOAD

96842dd

Fix typo

4736f0f

wiktorn closed this May 28, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Docker image for Overpass #374

Docker image for Overpass #374

wiktorn commented Mar 12, 2017

mmd-osm commented Mar 13, 2017 •

edited

Loading

wiktorn commented Mar 13, 2017

mmd-osm commented Mar 13, 2017

wiktorn commented Mar 14, 2017

mmd-osm commented Mar 15, 2017 •

edited

Loading

mmd-osm Mar 17, 2017 •

edited

Loading

mmd-osm Mar 17, 2017

wiktorn commented Mar 17, 2017

mmd-osm Mar 17, 2017 •

edited

Loading

wiktorn Mar 17, 2017

wiktorn Mar 17, 2017

Docker image for Overpass #374

Docker image for Overpass #374

Conversation

wiktorn commented Mar 12, 2017

mmd-osm commented Mar 13, 2017 • edited Loading

wiktorn commented Mar 13, 2017

mmd-osm commented Mar 13, 2017

wiktorn commented Mar 14, 2017

mmd-osm commented Mar 15, 2017 • edited Loading

mmd-osm Mar 17, 2017 • edited Loading

Choose a reason for hiding this comment

mmd-osm Mar 17, 2017

Choose a reason for hiding this comment

wiktorn commented Mar 17, 2017

mmd-osm Mar 17, 2017 • edited Loading

Choose a reason for hiding this comment

wiktorn Mar 17, 2017

Choose a reason for hiding this comment

wiktorn Mar 17, 2017

Choose a reason for hiding this comment

mmd-osm commented Mar 13, 2017 •

edited

Loading

mmd-osm commented Mar 15, 2017 •

edited

Loading

mmd-osm Mar 17, 2017 •

edited

Loading

mmd-osm Mar 17, 2017 •

edited

Loading