Skip to content

Commit edfd98e

Browse files
maudetesmaudetes
authored
Hook into new metrics views (#8)
1 parent 9c5f8cb commit edfd98e

32 files changed

+1090
-103
lines changed

‎.circleci/config.yml

+2-18
Original file line numberDiff line numberDiff line change
@@ -5,9 +5,8 @@ jobs:
55
build:
66
docker:
77
- image: udata/circleci:2-alpine
8-
- image: mongo:3.6
8+
- image: mongo:6.0.4
99
- image: redis:alpine
10-
- image: udata/elasticsearch:2.4.5
1110
environment:
1211
BASH_ENV: /root/.bashrc
1312
steps:
@@ -28,6 +27,7 @@ jobs:
2827
command: |
2928
virtualenv venv
3029
source venv/bin/activate
30+
pip install --force-reinstall setuptools==66.1.1
3131
pip install -e . || pip install -e .
3232
pip install -r requirements/develop.pip || pip install -r requirements/develop.pip
3333
- save_cache:
@@ -84,28 +84,12 @@ jobs:
8484
name: Publish on PyPI
8585
command: twine upload --username "${PYPI_USERNAME}" --password "${PYPI_PASSWORD}" dist/*.whl
8686

87-
github:
88-
docker:
89-
- image: udata/circleci:2-alpine
90-
environment:
91-
BASH_ENV: /root/.bashrc
92-
steps:
93-
- attach_workspace:
94-
at: .
95-
- run:
96-
name: Upload github release
97-
command: gh_release
98-
9987
workflows:
10088
version: 2
10189
build:
10290
jobs:
10391
- build:
10492
filters:
105-
branches:
106-
only:
107-
- master
108-
- /[0-9]+(\.[0-9]+)+/
10993
tags:
11094
only: /v[0-9]+(\.[0-9]+)*/
11195
- publish:

‎.gitignore

+81
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
*.py[cod]
2+
*~
3+
4+
# C extensions
5+
*.so
6+
7+
# Packages
8+
*.egg
9+
*.egg-info
10+
dist
11+
build
12+
eggs
13+
.eggs/
14+
parts
15+
bin
16+
var
17+
sdist
18+
develop-eggs
19+
.installed.cfg
20+
lib
21+
lib64
22+
__pycache__
23+
staticroot
24+
/media
25+
*.sqlite
26+
bower
27+
/youckan.ini
28+
*.db
29+
*.log
30+
*.pid
31+
.webassets-cache
32+
js-built
33+
.DS_Store
34+
*.min.css
35+
*.min.js
36+
/instance/
37+
/static/
38+
/doc/_build/
39+
/site/
40+
/node_modules/
41+
/udata/static/
42+
/jsdoc/
43+
*.rdb
44+
venv
45+
46+
# Installer logs
47+
pip-log.txt
48+
49+
# Unit test / coverage reports
50+
.cache
51+
.coverage
52+
.pytest_cache
53+
.tox
54+
cover
55+
nosetests.xml
56+
coverage.xml
57+
/coverage/
58+
/reports/
59+
60+
# Translations
61+
*.mo
62+
*.pot
63+
64+
# Mr Developer
65+
.mr.developer.cfg
66+
.project
67+
.pydevproject
68+
69+
# Force .gitignore files
70+
!.gitignore
71+
.noseids
72+
Procfile
73+
74+
# Local configuration file of udata
75+
udata.cfg
76+
77+
# Local data directory
78+
data
79+
80+
# PyCharm
81+
.idea

‎CHANGELOG.md

+4
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,10 @@
22

33
## Current (in progress)
44

5+
- :warning: New metrics logic [#8](https://github.com/opendatateam/udata-metrics/pull/8)
6+
- Add metrics views (using template hook)
7+
- Use optional metrics API (with job to update metrics on objects)
8+
- Remove Influx logic
59
- Replace mongo legacy image in CI [#6](https://github.com/opendatateam/udata-metrics/pull/6)
610

711
## 1.0.2 (2020-07-01)

‎MANIFEST.in

+1-1
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,6 @@ include LICENSE MANIFEST.in README.md CHANGELOG.md
22

33
recursive-include requirements *
44

5-
recursive-include udata-metrics *
5+
recursive-include udata_metrics *
66

77
global-exclude *~ *.egg *.pyc

‎README.md

+7-10
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,15 @@
11
# udata-metrics
22

3-
This plugin handles the connexion to an InfluxDB service for udata.
4-
It provides a client to store and to retrieve views and hits generated by users before injecting them in udata's objects metrics.
3+
This plugin adds views to display metrics on datasets, reuses, organizations and site dashboard pages.
4+
It uses hook logic based on udata-front hooks.
5+
We feed these views with data from:
6+
* an optional metrics API for traffic (views, download, external links)
7+
* mongo for stock values (with an aggregation pipeline)
8+
A job can be scheduled to inject traffic metrics in udata's objects metrics.
59

610
## Installation
711

812
Install [udata](https://github.com/opendatateam/udata).
9-
Install [udata-piwik](https://github.com/opendatateam/udata-piwik).
1013

1114
Remain in the same virtual environment (for Python).
1215

@@ -20,11 +23,5 @@ Modify your local configuration file of **udata** (typically, `udata.cfg`) as fo
2023

2124
```python
2225
PLUGINS = ['metrics']
23-
METRICS_DSN = {
24-
'host': 'localhost',
25-
'port': '8086',
26-
'username': 'johndoe',
27-
'password': 'youwillneverguess',
28-
'database': 'metrics_db'
29-
}
26+
METRICS_API = 'http://localhost:8005/api'
3027
```

‎babel.cfg

+13
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
[python: **.py]
2+
3+
[jinja2: templates/**.html]
4+
encoding = utf-8
5+
silent = false
6+
extensions= flask_caching.jinja2ext.CacheExtension
7+
8+
[jinja2: templates/comments/**.txt]
9+
encoding = utf-8
10+
silent = false
11+
extensions= flask_caching.jinja2ext.CacheExtension
12+
13+
[ignore: static/**]

‎crowdin.yml

+3
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
files:
2+
- source: /udata_metrics/translations/*.pot
3+
translation: /udata_metrics/translations/%two_letters_code%/LC_MESSAGES/udata.po

‎requirements/develop.pip

+2-3
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,4 @@
11
-e .[test]
22
flake8==3.7.8
3-
invoke==1.3.0
4-
pytest-cov==2.6.1
5-
readme-renderer[md]==29.0
3+
invoke==1.7.3
4+
pytest-cov==4.0.0

‎requirements/install.pip

+2-2
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,2 @@
1-
udata>=2.0.0
2-
influxdb==5.2.3
1+
udata>=6.0.0
2+
udata-front>=3.2.8

‎requirements/test.pip

+6-5
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
1-
httpretty==0.9.7
2-
mock==3.0.5
3-
pytest-flask==0.15.0
4-
pytest-sugar==0.9.2
5-
pytest==4.6.3
1+
httpretty==1.1.4
2+
mock==5.0.1
3+
pytest-flask==1.2.0
4+
pytest-sugar==0.9.6
5+
pytest==7.2.1
6+
requests-mock==1.10.0

‎setup.cfg

+28
Original file line numberDiff line numberDiff line change
@@ -6,3 +6,31 @@ testpaths = tests
66
python_files = test_*.py
77
python_functions = test_*
88
python_classes = *Test
9+
10+
[flake8]
11+
exclude = doc,.git
12+
max_line_length = 100
13+
14+
[compile_catalog]
15+
domain = udata_metrics
16+
directory = udata_metrics/translations
17+
statistics = true
18+
19+
[extract_messages]
20+
keywords = _ N_:1,2 P_:1c,2 L_ gettext ngettext:1,2 pgettext:1c,2 npgettext:1c,2,3 lazy_gettext lazy_pgettext:1c,2
21+
mapping_file = babel.cfg
22+
add_comments = TRANSLATORS:
23+
output_file = udata_metrics/translations/udata_metrics.pot
24+
width = 80
25+
26+
[init_catalog]
27+
domain = udata_metrics
28+
input_file = udata_metrics/translations/udata_metrics.pot
29+
output_dir = udata_metrics/translations
30+
31+
[update_catalog]
32+
domain = udata_metrics
33+
input_file = udata_metrics/translations/udata_metrics.pot
34+
output_dir = udata_metrics/translations
35+
ignore_obsolete = true
36+
previous = true

‎setup.py

+4-9
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44
import io
55
import re
66

7-
from setuptools import setup, find_packages
7+
from setuptools import setup
88

99
RE_BADGE = re.compile(r'^\[\!\[(?P<text>.*?)\]\[(?P<badge>.*?)\]\]\[(?P<target>.*?)\]$', re.M)
1010

@@ -57,17 +57,12 @@ def pip(filename):
5757
'test': tests_require,
5858
},
5959
entry_points={
60-
'udata.models': [
61-
'metrics = udata_metrics.models',
60+
'udata.views': [
61+
'metrics = udata_metrics.views',
6262
],
63-
6463
'udata.tasks': [
6564
'metrics = udata_metrics.tasks',
66-
],
67-
68-
'udata.plugins': [
69-
'metrics = udata_metrics',
70-
],
65+
]
7166
},
7267
license='AGPL',
7368
zip_safe=False,

‎tasks.py

+53-2
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,18 @@
11
import os
22

3+
from datetime import datetime
4+
5+
from babel.messages.pofile import read_po, write_po
6+
from babel.util import LOCALTZ
7+
38
from invoke import task, call
49

510
ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__)))
611

12+
PYTHON_I18N_ROOT = 'udata_metrics/translations'
13+
14+
LANGUAGES = ['fr']
15+
716
TO_CLEAN = ['build', 'dist', '**/*.pyc', 'reports']
817

918

@@ -64,7 +73,7 @@ def test(ctx, report=False):
6473
@task
6574
def cover(ctx, html=False):
6675
'''Run tests suite with coverage'''
67-
cmd = 'pytest --cov udata-metrics --cov-report term'
76+
cmd = 'pytest --cov udata_metrics --cov-report term'
6877
if html:
6978
cmd = ' '.join((cmd, '--cov-report html:reports/cover'))
7079
with ctx.cd(ROOT):
@@ -77,7 +86,7 @@ def qa(ctx):
7786
header(qa.__doc__)
7887
with ctx.cd(ROOT):
7988
info('Python Static Analysis')
80-
flake8_results = ctx.run('flake8 udata-metrics', pty=True, warn=True)
89+
flake8_results = ctx.run('flake8 udata_metrics', pty=True, warn=True)
8190
if flake8_results.failed:
8291
error('There is some lints to fix')
8392
else:
@@ -94,6 +103,48 @@ def qa(ctx):
94103
exit(flake8_results.return_code or readme_results.return_code)
95104
success('Quality check OK')
96105

106+
def set_po_metadata(filename, locale):
107+
# Fix crowdin requiring Language with `2-digit` iso code in potfile
108+
# to produce 2-digit iso code pofile
109+
# Opening the catalog also allows to set extra metadata
110+
with open(filename, 'rb') as infile:
111+
catalog = read_po(infile, locale)
112+
catalog.copyright_holder = 'Etalab'
113+
catalog.msgid_bugs_address = '[email protected]'
114+
catalog.language_team = 'Data.gouv.fr Team <[email protected]>'
115+
catalog.last_translator = 'Data.gouv.fr Team <[email protected]>'
116+
catalog.revision_date = datetime.now(LOCALTZ)
117+
with open(filename, 'wb') as outfile:
118+
write_po(outfile, catalog, width=80)
119+
120+
121+
@task
122+
def i18n(ctx, update=False):
123+
'''Extract translatable strings'''
124+
header(i18n.__doc__)
125+
126+
# Python translations
127+
info('Extract python translations')
128+
with ctx.cd(ROOT):
129+
ctx.run('python setup.py extract_messages')
130+
set_po_metadata(os.path.join(PYTHON_I18N_ROOT, 'udata_metrics.pot'), 'en')
131+
for lang in LANGUAGES:
132+
pofile = os.path.join(PYTHON_I18N_ROOT, lang, 'LC_MESSAGES', 'udata_metrics.po')
133+
if not os.path.exists(pofile):
134+
ctx.run('python setup.py init_catalog -l {}'.format(lang))
135+
set_po_metadata(pofile, lang)
136+
elif update:
137+
ctx.run('python setup.py update_catalog -l {}'.format(lang))
138+
set_po_metadata(pofile, lang)
139+
140+
141+
@task
142+
def i18nc(ctx):
143+
'''Compile translations'''
144+
header('Compiling translations')
145+
with ctx.cd(ROOT):
146+
ctx.run('python setup.py compile_catalog')
147+
97148

98149
@task
99150
def dist(ctx, buildno=None):

‎udata_metrics/models.py ‎tests/__init__.py

File renamed without changes.

‎tests/conftest.py

+6-1
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,12 @@
44
from udata.app import create_app
55

66

7+
class MetricsSettings(settings.Testing):
8+
PLUGINS = ['metrics']
9+
METRICS_API = 'http://metrics-api.fr/api'
10+
11+
712
@pytest.fixture
813
def app():
9-
app = create_app(settings.Defaults, override=settings.Testing)
14+
app = create_app(settings.Defaults, override=MetricsSettings)
1015
return app

‎tests/data/README.md

-1
This file was deleted.

0 commit comments

Comments
 (0)