Code

Feature/add scapacra bt#536

Code

Merged

ManAnRuck merged 131 commits intomasterdemokratie-live/democracy-development:masterfrom feature/add-scapacra-btdemokratie-live/democracy-development:feature/add-scapacra-btCopy head branch name to clipboardNov 5, 2023

+13,068-5,269

Commits

Commits on Nov 5, 2023

• Initial commit

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  3caa986

  Copy full SHA for 3caa986

• First version of an DocumentEvaluator which provides information from a Plenarprotokoll via xPath.

  

  Jannik Rebmann

  authored and

  Manuel Ruck

  committed
  60341bf

  Copy full SHA for 60341bf

• fixed security issue(?), implemented nodemon

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  bc459bc

  Copy full SHA for bc459bc

• Presentation of the importer model concept with plant-uml project files.

  

  Jannik Rebmann

  authored and

  Manuel Ruck

  committed
  1a9c70b

  Copy full SHA for 1a9c70b

• add package.json

  

  ManAnRuck

  authored and

  Manuel Ruck

  committed
  91f2f20

  Copy full SHA for 91f2f20

• Add package-lock.json to .gitignore

  

  Jannik Rebmann

  authored and

  Manuel Ruck

  committed
  1611111

  Copy full SHA for 1611111

• Remove console debug message, unused imports and uncommented code lines

  

  Jannik Rebmann

  authored and

  Manuel Ruck

  committed
  662bf32

  Copy full SHA for 662bf32

• Integration of unit testing framework chai/mocha

  Show description for 5b8539a
  

  Jannik Rebmann

  authored and

  Manuel Ruck

  committed
  5b8539a

  Copy full SHA for 5b8539a

• Formating JSON schema files

  

  Jannik Rebmann

  authored and

  Manuel Ruck

  committed
  004ec06

  Copy full SHA for 004ec06

• Improve folder structure

  

  Jannik Rebmann

  authored and

  Manuel Ruck

  committed
  9dd4329

  Copy full SHA for 9dd4329

• Implemented importer structure (see doc)

  Show description for 7bfdd79
  

  Jannik Rebmann

  authored and

  Manuel Ruck

  committed
  7bfdd79

  Copy full SHA for 7bfdd79

• BugFix: Page counter was not incremented

  

  Jannik Rebmann

  authored and

  Manuel Ruck

  committed
  424868e

  Copy full SHA for 424868e

• Add abstract class BundestagListBrowser to generalize the list browsing.

  

  Jannik Rebmann

  authored and

  Manuel Ruck

  committed
  b84dab7

  Copy full SHA for b84dab7

• BugFixes: Browser paging was buggy

  

  Jannik Rebmann

  authored and

  Manuel Ruck

  committed
  05d58aa

  Copy full SHA for 05d58aa

• Indroduce async await concept

  Show description for 8d267d6
  

  Jannik Rebmann

  authored and

  Manuel Ruck

  committed
  8d267d6

  Copy full SHA for 8d267d6

• Remaining async refactoring done

  

  Jannik Rebmann

  authored and

  Manuel Ruck

  committed
  0c8f3b7

  Copy full SHA for 0c8f3b7

• Join Scraper, IBrowser and IParser into one file (Importer.ts)

  

  Jannik Rebmann

  authored and

  Manuel Ruck

  committed
  7caf1d0

  Copy full SHA for 7caf1d0

• Remove external browser URL setting. URL strongly depends to the browser implementation itself.

  

  Jannik Rebmann

  authored and

  Manuel Ruck

  committed
  f7fad4e

  Copy full SHA for f7fad4e

• Include scapacra library

  

  Jannik Rebmann

  authored and

  Manuel Ruck

  committed
  2e182a7

  Copy full SHA for 2e182a7

• Migration from Example.js to index.js

  

  Jannik Rebmann

  authored and

  Manuel Ruck

  committed
  f5304ff

  Copy full SHA for f5304ff

• Move index.ts to src folder

  

  Jannik Rebmann

  authored and

  Manuel Ruck

  committed
  e2ee02a

  Copy full SHA for e2ee02a

• first implementation of deputy scraper

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  a2801cd

  Copy full SHA for a2801cd

• cleanup

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  7d48ef3

  Copy full SHA for 7d48ef3

• more cleanup

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  cc55855

  Copy full SHA for cc55855

• BugFix: Added missing URL import. Error while runtime.

  

  Jannik Rebmann

  authored and

  Manuel Ruck

  committed
  3d164fa

  Copy full SHA for 3d164fa

• Code adaptions due to the API changes in scapacra

  

  Jannik Rebmann

  authored and

  Manuel Ruck

  committed
  f6ecd19

  Copy full SHA for f6ecd19

• cleanup for production

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  0f251bf

  Copy full SHA for 0f251bf

• referenced scapacra 1.0.0 instead of master

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  f0db883

  Copy full SHA for f0db883

• corrected package name, included prepare script, version bump

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  8b43568

  Copy full SHA for 8b43568

• adjusted compile options to not include test & run.ts

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  d4880b9

  Copy full SHA for d4880b9

• cleanup result, included directCandidate field

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  1aac653

  Copy full SHA for 1aac653

• corrected deputy link structure

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  a8dc259

  Copy full SHA for a8dc259

• use tsc command directly for build

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  b7a3412

  Copy full SHA for b7a3412

• removed unused file

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  4efe98d

  Copy full SHA for 4efe98d

• fixed package name & include for production&npm publish

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  8bd4e6e

  Copy full SHA for 8bd4e6e

• missing npm ignore to publish dist folder - gotta love this bs

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  21a18cc

  Copy full SHA for 21a18cc

• named polls (party results)

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  5f480e9

  Copy full SHA for 5f480e9

• deputy vote results

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  87f2c8d

  Copy full SHA for 87f2c8d

• version bump

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  0f4f397

  Copy full SHA for 0f4f397

• fixed several scraping mistakes, found a way to collect all deputyIds, updated scapacra reference (1.0.3), v1.0.6

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  8684b41

  Copy full SHA for 8684b41

• [NamedPoll] several fixes of structure and type

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  79b0ada

  Copy full SHA for 79b0ada

• v1.0.7

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  dcb0011

  Copy full SHA for dcb0011

• v1.0.8 [NamedPoll] fixed structur

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  5da34b6

  Copy full SHA for 5da34b6

• corrected [NamedPollDeputies] Scraper (types, structure)

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  2bc3503

  Copy full SHA for 2bc3503

• v1.0.9

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  6d44235

  Copy full SHA for 6d44235

• fixed deputy regex

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  36782f0

  Copy full SHA for 36782f0

• fixed zero votes - resolve to 0 instead null

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  7ff9786

  Copy full SHA for 7ff9786

• updated version 1.0.11

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  ca4d01a

  Copy full SHA for ca4d01a

• fixed scrapers, new version 1.0.12

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  bea6ea9

  Copy full SHA for bea6ea9

• fixed deputy id parsing

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  a43be04

  Copy full SHA for a43be04

• updated version to 1.0.13

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  d22cd2a

  Copy full SHA for d22cd2a

• another deputy id fix

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  4ee898e

  Copy full SHA for 4ee898e

• version 1.0.14

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  4b74bfe

  Copy full SHA for 4b74bfe

• fixed deputyId (confirmed)

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  b1ddf39

  Copy full SHA for b1ddf39

• Update README.md

  Show description for 9b69f3e
  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  9b69f3e

  Copy full SHA for 9b69f3e

• fixed several paths for the scrapers to work on linux

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  d6d3c0e

  Copy full SHA for d6d3c0e

• - fixed deputy name parsing to not include Partyname,

  Show description for 43432c3
  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  43432c3

  Copy full SHA for 43432c3

• fixed NamedPollScraper

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  aea0483

  Copy full SHA for aea0483

• - include run script in build

  Show description for 21d9f76
  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  21d9f76

  Copy full SHA for 21d9f76

• updated all packages

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  fba64f1

  Copy full SHA for fba64f1

• version 1.0.16

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  0bfb80f

  Copy full SHA for 0bfb80f

• added gitkeep for outfolder

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  d69da18

  Copy full SHA for d69da18

• first try with travis

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  12992af

  Copy full SHA for 12992af

• echos at the end of the script

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  00a2ff3

  Copy full SHA for 00a2ff3

• - "fixed" tests

  Show description for 3aae148
  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  3aae148

  Copy full SHA for 3aae148

• travis commit & push properly

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  aaf8ccc

  Copy full SHA for aaf8ccc

• - removed package-lock.json file

  Show description for a1fa3ee
  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  a1fa3ee

  Copy full SHA for a1fa3ee

• encrypted github key, prepare github push

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  4f79515

  Copy full SHA for 4f79515

• fixed git url to be ssh

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  548f7bb

  Copy full SHA for 548f7bb

• - individual keys for every repo

  Show description for 50eb48e
  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  50eb48e

  Copy full SHA for 50eb48e

• fixed deputyProfileParser to sort functions to prevent unnecessary commits

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  057ab8e

  Copy full SHA for 057ab8e

• fixed deputyProfileParser to sort functions to prevent unnecessary commits

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  98a8726

  Copy full SHA for 98a8726

• - fixed encoding

  Show description for 3fe1fe2
  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  3fe1fe2

  Copy full SHA for 3fe1fe2

• scrape on cronjob only

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  030d465

  Copy full SHA for 030d465

• version 1.0.17

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  48838b9

  Copy full SHA for 48838b9

• fixed travis cron check

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  d10d1c4

  Copy full SHA for d10d1c4

• v 1.0.18

  Show description for 2db55af
  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  2db55af

  Copy full SHA for 2db55af

• - time parsing

  Show description for e54d431
  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  e54d431

  Copy full SHA for e54d431

• interface fixes for unused parts

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  4528155

  Copy full SHA for 4528155

• iterate to the last Detailpage

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  d1cfa75

  Copy full SHA for d1cfa75

• v 1.0.19

  Show description for f7ba3db
  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  f7ba3db

  Copy full SHA for f7ba3db

• also commit untracked files

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  bbeccda

  Copy full SHA for bbeccda

• fixed travis to add all untracked files

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  09858e5

  Copy full SHA for 09858e5

• bio rules & heading

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  a72b0e9

  Copy full SHA for a72b0e9

• - cleanup

  Show description for 284e871
  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  284e871

  Copy full SHA for 284e871

• v1.0.20

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  9f06769

  Copy full SHA for 9f06769

• - leading zeros for weeks,

  Show description for f107466
  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  f107466

  Copy full SHA for f107466

• v1.0.21

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  45840cf

  Copy full SHA for 45840cf

• - scrape article link

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  4d3d5c6

  Copy full SHA for 4d3d5c6

• v1.0.22

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  66088ee

  Copy full SHA for 66088ee

• fixed status regex

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  8062e6e

  Copy full SHA for 8062e6e

• v 1.0.23

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  f02e802

  Copy full SHA for f02e802

• adjusted all scrapers to new scapacra interfaces

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  4185e27

  Copy full SHA for 4185e27

• v1.0.24

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  1c5186b

  Copy full SHA for 1c5186b

• updated scapacra reference, made callback async

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  50b63e5

  Copy full SHA for 50b63e5

• v1.0.25

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  c58ec2b

  Copy full SHA for c58ec2b

• handle TOP dates correctly for sessions spanning into the next day

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  35f7975

  Copy full SHA for 35f7975

• v1.0.26

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  152e475

  Copy full SHA for 152e475

• exclude out for npm package

  Show description for 2ab21ee
  

  ManAnRuck

  authored and

  Manuel Ruck

  committed
  2ab21ee

  Copy full SHA for 2ab21ee

• fix regex

  Show description for 53944ee
  

  ManAnRuck

  authored and

  Manuel Ruck

  committed
  53944ee

  Copy full SHA for 53944ee

• 1.0.27

  

  ManAnRuck

  authored and

  Manuel Ruck

  committed
  ac6a0e9

  Copy full SHA for ac6a0e9

• get all tops

  Show description for 9c2796e
  

  ManAnRuck

  authored and

  Manuel Ruck

  committed
  9c2796e

  Copy full SHA for 9c2796e

• improve conference week scraper

  Show description for cae4f97
  

  ManAnRuck

  authored and

  Manuel Ruck

  committed
  cae4f97

  Copy full SHA for cae4f97

• Fix conference week regex

  Show description for eeccb9e
  

  Manuel Ruck

  committed
  eeccb9e

  Copy full SHA for eeccb9e

• fix named polls

  Show description for ceac5e0
  

  Manuel Ruck

  committed
  ceac5e0

  Copy full SHA for ceac5e0

• upgrade version

  Show description for 1fe8107
  

  Manuel Ruck

  committed
  1fe8107

  Copy full SHA for 1fe8107

• v1.0.33

  

  Manuel Ruck

  committed
  fa50873

  Copy full SHA for fa50873

• chore(updates): 📦 update packages

  Show description for a1a6ce8
  

  Manuel Ruck

  committed
  a1a6ce8

  Copy full SHA for a1a6ce8

• Initial commit

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  ea464b1

  Copy full SHA for ea464b1

• Added importer structure and xml document evaluator helper

  

  Jannik Rebmann

  authored and

  Manuel Ruck

  committed
  3e4baa1

  Copy full SHA for 3e4baa1

• Adaptions for packing and publishing

  

  Jannik Rebmann

  authored and

  Manuel Ruck

  committed
  45713a0

  Copy full SHA for 45713a0

• fix build by including "prepare" to let the package build its dist folder on include

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  87e1e89

  Copy full SHA for 87e1e89

• remove watch flag

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  8357502

  Copy full SHA for 8357502

• Added IDataPackage (Data + Metadata)

  

  Jannik Rebmann

  authored and

  Manuel Ruck

  committed
  4fdace4

  Copy full SHA for 4fdace4

• Basic description of scapacra

  

  Jannik Rebmann

  authored and

  Manuel Ruck

  committed
  2ccdde1

  Copy full SHA for 2ccdde1

• Include UML diagrams into the documentation

  

  Jannik Rebmann

  authored and

  Manuel Ruck

  committed
  742b7e7

  Copy full SHA for 742b7e7

• Import UML diagrams as svg

  

  Jannik Rebmann

  authored and

  Manuel Ruck

  committed
  ecac9aa

  Copy full SHA for ecac9aa

• rename package to publish it to npm - v 1.0.2 since npm is the worst tool you can imagine

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  5696de1

  Copy full SHA for 5696de1

• missing v1.0.2 version - thanks npm

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  c72507b

  Copy full SHA for c72507b

• allow custom error handlers on xmldom parsing

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  1e9def8

  Copy full SHA for 1e9def8

• - separated all classes and interfaces into files

  Show description for 0eeb822
  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  0eeb822

  Copy full SHA for 0eeb822

• version 1.0.4

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  8526825

  Copy full SHA for 8526825

• apply jannik fix, refactored interfaces to allow any Meta Data, removed Datatype, defined datapackage container instead to address memory issues

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  5a110b3

  Copy full SHA for 5a110b3

• v1.0.5

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  666e2fe

  Copy full SHA for 666e2fe

• made callback async and wait for it

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  6427c77

  Copy full SHA for 6427c77

• v1.0.6

  

  ulfgebhardt

  authored and

  Manuel Ruck

  committed
  bef1ee4

  Copy full SHA for bef1ee4

• chore(cleanup): 🧹 remove yarn.lock files

  Show description for 5903869
  

  Manuel Ruck

  committed
  5903869

  Copy full SHA for 5903869

• fix: 🐛 fix scapacra build

  Show description for e6454e2
  

  Manuel Ruck

  committed
  e6454e2

  Copy full SHA for e6454e2

• fix: 🐛 fix scapacra-bt build

  Show description for 6b73754
  

  Manuel Ruck

  committed
  6b73754

  Copy full SHA for 6b73754

• chore(packages): 📦 update packages

  Show description for e9a96aa
  

  Manuel Ruck

  committed
  e9a96aa

  Copy full SHA for e9a96aa

• fix: 🐛 setting up dayjs out of loops

  Show description for 2e66186
  

  Manuel Ruck

  committed
  2e66186

  Copy full SHA for 2e66186