Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor and test gleaner more thoroughly #4

Draft
wants to merge 40 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
451c1d5
all existing tests passing.
C-Loftus Oct 22, 2024
fb3d9e4
various refactors. Github CI for go tests
C-Loftus Oct 23, 2024
cb293cb
try to rerun ci scripts
C-Loftus Oct 23, 2024
303acca
change tests
C-Loftus Oct 23, 2024
8215bb9
more clean ups. Fixfunctions to return err instead of panicing
C-Loftus Oct 23, 2024
e9352e1
more improvements with error handling
C-Loftus Oct 23, 2024
4aee556
move test helpers. expose ui
C-Loftus Oct 23, 2024
2a09f20
more clean ups
C-Loftus Oct 23, 2024
59e3fe3
move geoconnex pids test into go
C-Loftus Oct 24, 2024
dd4a4c1
get rid of global variables
C-Loftus Oct 24, 2024
4cce7eb
remove unneeded docs; sitemap tests passing.
C-Loftus Oct 28, 2024
ceb5add
tests passing again with relative paths in the gleanerconfig
C-Loftus Oct 29, 2024
26ea806
sitemap test working with mainstem page
C-Loftus Oct 30, 2024
b9458c0
cleanups and more helpers for tests. Test dead sitemap
C-Loftus Nov 7, 2024
7508a32
fix config file path and some linter errors
C-Loftus Nov 7, 2024
ca116bf
make source as a type name more semantically meaningful
C-Loftus Nov 7, 2024
ba50d78
clean testing api
C-Loftus Nov 7, 2024
b199066
org tests
C-Loftus Nov 7, 2024
feda232
remove assert replace with require
C-Loftus Nov 7, 2024
2af9bac
remove old docs info
C-Loftus Nov 7, 2024
f0a111c
more linter fixes
C-Loftus Nov 7, 2024
820f067
test what happens with two different configs one after another.
C-Loftus Nov 18, 2024
900cd74
tests for what happens when the sitemap changes
C-Loftus Nov 19, 2024
cd6908f
test for bad jsonld
C-Loftus Nov 19, 2024
75d7d55
backup docs
C-Loftus Nov 19, 2024
b33ebff
Rerun test with no changes
C-Loftus Nov 19, 2024
41f2ded
change to use mix instead of an http handler which cant be unregistered
C-Loftus Nov 19, 2024
7b0f00b
separated github actions. fix typo
C-Loftus Nov 19, 2024
d43dada
add archive for old docs
C-Loftus Nov 19, 2024
de6f1df
lint fixes
C-Loftus Nov 20, 2024
564fa83
fix lint errors
C-Loftus Nov 20, 2024
42e9f0b
add docs
C-Loftus Nov 20, 2024
bcc0681
diff isnt supported
C-Loftus Nov 20, 2024
fb7cbc7
docs updated
C-Loftus Nov 20, 2024
4d4e051
backup
C-Loftus Nov 20, 2024
6b0d47e
add vscode test setting
C-Loftus Nov 25, 2024
47ee600
TestSameSitemapWithDifferentJSONLD
C-Loftus Nov 25, 2024
b2f9b99
test for deleting files in s3
C-Loftus Nov 25, 2024
1426dc5
update docs and fix test for updated jsonld
C-Loftus Nov 25, 2024
aa571ec
fix one test issue
C-Loftus Nov 25, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
27 changes: 15 additions & 12 deletions .github/ISSUE_TEMPLATE/bug_report.md
Original file line number Diff line number Diff line change
@@ -1,29 +1,32 @@
---
name: Bug report
about: Create a report to help us improve
title: ''
title: ""
labels: bug
assignees: ''

assignees: ""
---


**Description**
A clear and concise description of what the bug is.

<!-- A clear and concise description of what the bug is. -->

**Steps to Reproduce**
Steps to reproduce the behavior:

<!-- Steps to reproduce the behavior: -->

**Expected behavior**
A clear and concise description of what you expected to happen.

<!-- A clear and concise description of what you expected to happen. -->

**Screenshots/Tracebacks**
If applicable, add screenshots to help explain your problem.

<!-- If applicable, add screenshots to help explain your problem. -->

**Environment**
- OS:
- Python version:
- Python env/venv:

- OS:
- Gleaner version:

**Additional context**
Add any other context about the problem here.

<!-- Add any other context about the problem here. -->
10 changes: 5 additions & 5 deletions .github/ISSUE_TEMPLATE/documentation_issue.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,16 +8,16 @@ assignees: ''
---

**Description**
A clear and concise description of the documentation issue or improvement
<!-- A clear and concise description of the documentation issue or improvement -->

**Current Documentation**
Links or description of the current documentation.
<!-- Links or description of the current documentation. -->

**Suggested Improvement**
Describe the suggested improvement or fix.
<!-- Describe the suggested improvement or fix. -->

**Steps to Reproduce (if applicable)**
If relevant, steps to reproduce the issue with the current documentation.
<!-- If relevant, steps to reproduce the issue with the current documentation. -->

**Additional Context**
Any additional context, screenshots, or code examples.
<!-- Any additional context, screenshots, or code examples. -->
8 changes: 4 additions & 4 deletions .github/ISSUE_TEMPLATE/feature_request.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,13 +9,13 @@ assignees: ''


**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
<!-- A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] -->

**Describe the solution you'd like**
A clear and concise description of what you want to happen.
<!-- A clear and concise description of what you want to happen. -->

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.
<!-- A clear and concise description of any alternative solutions or features you've considered. -->

**Additional context**
Add any other context or screenshots about the feature request here.
<!-- Add any other context or screenshots about the feature request here. -->
165 changes: 0 additions & 165 deletions .github/workflows/docker_glcon.yml

This file was deleted.

4 changes: 2 additions & 2 deletions .github/workflows/docker_gleaner.yml
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ jobs:
- id: binary
name: build
run: |
go build -o gleaner -ldflags "-s -w -X main.VERSION=${{ env.VERSION }} -X github.com/gleanerio/gleaner/pkg.VERSION=${{ env.VERSION }}"
go build -o gleaner -ldflags "-s -w -X main.VERSION=${{ env.VERSION }} -X gleaner/pkg.VERSION=${{ env.VERSION }}"
working-directory: ./cmd/gleaner
env:
GOOS: ${{matrix.goss}}
Expand All @@ -77,7 +77,7 @@ jobs:
# goarch: ${{ matrix.goarch }}
# project_path: "./cmd/gleaner"
# binary_name: "gleaner"
# ldflags: "-s -w -X main.VERSION=${{ env.VERSION }} -X github.com/gleanerio/gleaner/pkg.VERSION=${{ env.VERSION }}"
# ldflags: "-s -w -X main.VERSION=${{ env.VERSION }} -X gleaner/pkg.VERSION=${{ env.VERSION }}"

# deploy:
# needs: build
Expand Down
13 changes: 13 additions & 0 deletions .github/workflows/go_lint.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
name: Go Lint

on: [push]

jobs:
golangci:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Run Golangci-lint
uses: golangci/[email protected]

50 changes: 50 additions & 0 deletions .github/workflows/go_tests.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
name: Go Test

on: [push]

jobs:
UnitTests:

runs-on: ubuntu-latest
strategy:
matrix:
go-version: [ '1.23' ]

steps:
- uses: actions/checkout@v4
- name: Setup Go ${{ matrix.go-version }}
uses: actions/setup-go@v5
with:
go-version: ${{ matrix.go-version }}
# You can test your matrix by printing the current Go version
- name: Display Go version
run: go version
- name: Install dependencies
run: go get .
- name: Build
run: go build -v ./...
- name: Test with the Go CLI
run: go test ./...

E2ETests:

runs-on: ubuntu-latest
strategy:
matrix:
go-version: [ '1.23' ]

steps:
- uses: actions/checkout@v4
- name: Setup Go ${{ matrix.go-version }}
uses: actions/setup-go@v5
with:
go-version: ${{ matrix.go-version }}
# You can test your matrix by printing the current Go version
- name: Display Go version
run: go version
- name: Install dependencies
run: go get .
- name: Build
run: go build -v ./...
- name: Test with the Go CLI
run: go test -tags 'e2e' ./...
2 changes: 1 addition & 1 deletion .github/workflows/release_gleaner.yml
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,6 @@ jobs:
goarch: ${{ matrix.goarch }}
project_path: "./cmd/gleaner"
binary_name: "gleaner"
ldflags: "-s -w -X main.VERSION=${{ env.VERSION }} -X github.com/gleanerio/gleaner/pkg.VERSION=${{ env.VERSION }}"
ldflags: "-s -w -X main.VERSION=${{ env.VERSION }} -X gleaner/pkg.VERSION=${{ env.VERSION }}"
extra_files: README.md docs/ configs/ scripts/ assets/
# extra_files: LICENSE README.md
2 changes: 1 addition & 1 deletion .github/workflows/release_gleaner_v3.yml
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,6 @@ jobs:
goarch: ${{ matrix.goarch }}
project_path: "./cmd/glcon"
binary_name: "glcon"
ldflags: "-s -w -X main.VERSION=${{ env.VERSION }} -X github.com/gleanerio/gleaner/pkg.VERSION=${{ env.VERSION }}"
ldflags: "-s -w -X main.VERSION=${{ env.VERSION }} -X gleaner/pkg.VERSION=${{ env.VERSION }}"
extra_files: README.md docs/ configs/ scripts/ assets/
# extra_files: LICENSE README.md
Loading