Skip to content

Home

Jump to bottom
Finn Årup Nielsen edited this page Sep 17, 2024 · 10 revisions

Welcome to the ordia wiki!

Weblog statistics

Convert the uwsgi.log log file to a more manageable version:

grep "^\[pid:" uwsgi.log | grep -v "GET /ordia/static/" | grep -v "GET /ordia/images/" | grep -v "HEAD /ordia" | awk '{print $12 " " $13 " " $14 " " $15 " " $16 " " $18 }' > stats-`date -I`.txt

Read the data into Python:

from __future__ import print_function
from datetime import datetime
from dateutil.parser import parse
import re
import pandas as pd

filename = "stats-" + datetime.now().isoformat().split('T')[0] + ".txt"

PATTERN = re.compile('\[(.*?)\] /ordia(?:/(.*?)(?:\?|/))?')

data = []
for line_no, line in enumerate(open(filename)):
    try:
        datetime_string, aspect = PATTERN.findall(line)[0]
    except: 
        continue
    datetime = parse(datetime_string)
    data.append((datetime, aspect, 1))
    if not line_no % 10000:
        print(line_no)

df = pd.DataFrame(data, columns=['datetime', 'aspect', 'count'])
yearmonth = [datetime.strftime('%Y-%m') for datetime in df['datetime']]
df['yearmonth'] = yearmonth
print(df.groupby('yearmonth').sum())

print(df.groupby('aspect').sum())

Updates for wmflabs

Login to login.tools.wmflabs.org

become ordia
webservice --backend=kubernetes python3.9 shell
source ~/www/python/venv/bin/activate
cd ~/www/python/ordia ; git pull
# pip install -rrequirements.txt
logout
webservice --backend=kubernetes python3.9 restart
tail -f ~/uwsgi.log  # look at web log

Toolforge monitoring:

Clone this wiki locally