Skip to content

Commit 720e3f8

Browse files
author
Jeroen Baten
committed
Created 2 new scripts:
FindCommitForIssue to find issues that can be closed because there is a commit referencing/mentioning them.
1 parent a605371 commit 720e3f8

5 files changed

+291
-38
lines changed

‎AddIssuesToProject.md

+12
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
#AddIssuesToProject
2+
3+
## Introduction
4+
After building the bugzilla2github script I needed to add all
5+
issues to a project called "Cleanup" for further processing.
6+
7+
After I learned about GitHub's pagination of output I deciced
8+
that it would not the greatest code ever. I just needed it to run once,
9+
so I just looped it 5 times and it worked. \0/
10+
11+
That's it.
12+

‎FindCommitForIssue.md

+10
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
#FindCommitForIssue
2+
3+
##Introduction
4+
While cleaning up old open issues I find issues
5+
that have been fixed but are still open.
6+
7+
So I want to list all cards in my Cleanup project that
8+
reference issues issues, then find the original Bugzilla
9+
ID number, and search git commit logs if that number is mentioned.
10+

‎FindCommitForIssue.py

+223
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,223 @@
1+
#!/usr/bin/env python
2+
# -*- coding: utf-8 -*
3+
#
4+
# Program to List Issues from a Project
5+
# - retrieve first comment
6+
# - retrieve original Bugzilla ID
7+
# - check if Bugzilla ID is mentioned in a git commit log.
8+
#
9+
# How to use the script:
10+
# 0. Create a virtual environment containing modules mentioned in freeze.txt
11+
# 1. Generate a GitHub access token:
12+
# - on GitHub select "Settings"
13+
# - select "Personal access tokens"
14+
# - click "Generate new token"
15+
# - type a token description, i.e. "bugzilla2github"
16+
# - select "public_repo" to access just public repositories
17+
# - save the generated token into the migration script
18+
# 3. Copy bugzilla2github.conf.sample to FindCommitsForIssue.conf
19+
# - Well, no, I just used the AddIssuesToProject.conf again.
20+
# - Change all settings to fit your setup
21+
# 4. Run the script. Good luck....
22+
23+
import json, requests, sys, os
24+
import ConfigParser
25+
from pprint import pprint, pformat
26+
from dateutil.tz import tzlocal
27+
import subprocess
28+
29+
reload(sys)
30+
sys.setdefaultencoding('utf-8')
31+
32+
destination_project_name = "Cleanup"
33+
#destination_column_name = "To do"
34+
destination_column_name = "In progress"
35+
page_size=100
36+
page_size=30 # 30 seems to be the GitHub maximum
37+
my_page=2
38+
39+
# read config file
40+
configFile = "AddIssuesToProject.conf"
41+
config = ConfigParser.RawConfigParser()
42+
try:
43+
config.read(configFile)
44+
# Read GitHub vars
45+
GITHUB_TOKEN = config.get('settings', 'github_token')
46+
GITHUB_URL = config.get('settings', 'github_url')
47+
GITHUB_OWNER = config.get('settings', 'github_owner')
48+
GITHUB_REPO = config.get('settings', 'github_repo')
49+
50+
51+
except:
52+
print "Error reading configfile '" + configFile + "'."
53+
print "Program aborted."
54+
sys.exit(1)
55+
56+
57+
def get_projects_from_github(repo, token):
58+
# GET /repos/:owner/:repo/projects
59+
urlparts = (str(GITHUB_URL), "repos", str(GITHUB_OWNER), str(GITHUB_REPO), "projects")
60+
url = "/".join(urlparts)
61+
62+
# pprint(url)
63+
# d=issue
64+
headers = {"Authorization": "token " + GITHUB_TOKEN,
65+
"Accept": "application/vnd.github.inertia-preview+json"}
66+
# "Accept": "application/vnd.github.golden-comet-preview+json" }
67+
# result=requests.get(url, headers=headers, data = json.dumps(d))
68+
result = requests.get(url, headers=headers)
69+
return result
70+
71+
72+
def get_columns_from_github(repo, token, project_id):
73+
# GET /projects/:project_id/columns
74+
urlparts = (str(GITHUB_URL), "projects", str(project_id), "columns")
75+
url = "/".join(urlparts)
76+
77+
#pprint(url)
78+
# d=issue
79+
headers = {"Authorization": "token " + GITHUB_TOKEN,
80+
"Accept": "application/vnd.github.inertia-preview+json"}
81+
# "Accept": "application/vnd.github.golden-comet-preview+json" }
82+
# result=requests.get(url, headers=headers, data = json.dumps(d))
83+
result = requests.get(url, headers=headers)
84+
return result
85+
86+
87+
def get_open_issues_from_github(repo, token, project_id, page=1):
88+
# GET /repos/:owner/:repo/issues
89+
urlparts = (str(GITHUB_URL), "repos", str(GITHUB_OWNER), str(GITHUB_REPO), "issues")
90+
url = "/".join(urlparts)
91+
url += "?per_page="+str(page_size)+"&page=" + str(page)
92+
#pprint(url)
93+
94+
headers = {"Authorization": "token " + GITHUB_TOKEN,
95+
"Accept": "application/vnd.github.inertia-preview+json"}
96+
# "Accept": "application/vnd.github.symmetra-preview+json"}
97+
# result=requests.get(url, headers=headers, data = json.dumps(d))
98+
result = requests.get(url, headers=headers)
99+
return result
100+
101+
def get_issue_from_content_url(issue_url):
102+
headers = {"Authorization": "token " + GITHUB_TOKEN,
103+
"Accept": "application/vnd.github.inertia-preview+json"}
104+
# "Accept": "application/vnd.github.symmetra-preview+json"}
105+
# result=requests.get(url, headers=headers, data = json.dumps(d))
106+
result = requests.get(issue_url, headers=headers)
107+
return result
108+
109+
def get_project_cards_from_github(repo, token, column_id,page=1):
110+
# GET /projects/columns/:column_id/cards
111+
urlparts = (str(GITHUB_URL), "projects", "columns", str(column_id), "cards")
112+
url = "/".join(urlparts)
113+
url += "?per_page="+str(page_size)+"&page=" + str(page)
114+
#pprint(url)
115+
116+
headers = {"Authorization": "token " + GITHUB_TOKEN,
117+
"Accept": "application/vnd.github.inertia-preview+json"}
118+
# "Accept": "application/vnd.github.symmetra-preview+json"}
119+
# result=requests.get(url, headers=headers, data = json.dumps(d))
120+
result = requests.get(url, headers=headers)
121+
return result
122+
123+
124+
125+
print "Start"
126+
print "====="
127+
# Find all defined projects
128+
projects = get_projects_from_github(GITHUB_REPO, GITHUB_TOKEN)
129+
# pprint(projects)
130+
if projects.status_code == 200:
131+
# pprint(projects.json())
132+
for project in projects.json():
133+
print "Project " + project["name"] + " has id: " + str(project["id"])
134+
135+
# Find all defined columns
136+
# my_project_id=1327353
137+
# print my_project_id
138+
my_project_id = [project["id"] for project in projects.json() if project["name"] == destination_project_name][0]
139+
print "My project id is " + str(my_project_id)
140+
# print my_project_id
141+
142+
columns = get_columns_from_github(GITHUB_REPO, GITHUB_TOKEN, my_project_id)
143+
# pprint(columns)
144+
# pprint(columns.json())
145+
146+
my_column_id = [column["id"] for column in columns.json() if column["name"] == destination_column_name][0]
147+
print "My column id is " + str(my_column_id)
148+
149+
cards=get_project_cards_from_github(GITHUB_REPO,GITHUB_TOKEN,my_column_id,my_page)
150+
#pprint(cards.json())
151+
152+
for card in cards.json():
153+
if card.get("content_url") is not None:
154+
github_issue_url=card.get("content_url")
155+
#print "card references issue "+issue_url
156+
github_issue_id=github_issue_url.split("/")[-1]
157+
#print "GitHub (not Bugzilla!) issue we are looking for is "+str(issue)
158+
# Now get first message from url and get original Bugzilla id
159+
github_issue=get_issue_from_content_url(github_issue_url)
160+
#pprint(issue.json())
161+
body=github_issue.json().get("body")
162+
#print body
163+
bugzillastring=body.split('\n', 1)[0]
164+
# Let's skip issues originally not form bugzilla
165+
if "Original Bugzilla Bug ID" in bugzillastring:
166+
start = bugzillastring.index("[") + 1
167+
end = bugzillastring.index("]", start)
168+
bugzilla_id=bugzillastring[start:end]
169+
#print "Looking for bugzilla ID: " + str(bugzilla_id)
170+
print ".",
171+
172+
#print "Is there a commit message referencing issue "+str(bugzilla_id)+"?"
173+
#print "issue url= https://github.com/LibrePlan/libreplan/issues/"+str(issue)
174+
#print "issue url= " + issue_url
175+
mydir="/home/jeroen/libreplan"
176+
cmd="git log --all --grep='#"+str(bugzilla_id)+"' "
177+
#cmd = "git log --all --grep='#" + str(bugzilla_id) + "' | grep '^commit' "
178+
179+
#cmd = "git log --all --grep='#" + str(issue) + "'"
180+
#print cmd
181+
os.chdir("/home/jeroen/libreplan")
182+
p=subprocess.Popen(cmd,
183+
cwd=mydir,
184+
stdout=subprocess.PIPE,
185+
shell=True)
186+
out, err = p.communicate()
187+
#output=subprocess.check_output([cmd,])
188+
# if "commit" in output:
189+
#pprint(out)
190+
if "commit" in out:
191+
print "\n"
192+
print "=" * 80
193+
outs=out.split("\n")
194+
for commitstr in outs:
195+
print commitstr
196+
print "-" * 80
197+
print "issue url= https://github.com/LibrePlan/libreplan/issues/"+str(github_issue_id)
198+
#print "issue url= " + github_issue_url
199+
cmd = "git log --all --grep='#" + str(bugzilla_id) + "' | grep '^commit' "
200+
#print cmd
201+
os.chdir("/home/jeroen/libreplan")
202+
p=subprocess.Popen(cmd,
203+
cwd=mydir,
204+
stdout=subprocess.PIPE,
205+
shell=True)
206+
out, err = p.communicate()
207+
#output=subprocess.check_output([cmd,])
208+
# if "commit" in output:
209+
#pprint(out)
210+
outs=out.split("\n")
211+
for commitstr in outs:
212+
print commitstr
213+
214+
215+
#os.system(cmd)
216+
#sys.exit(1)
217+
else:
218+
print "Not an original Bugzilla issue."
219+
220+
print "\n"
221+
222+
223+
print "Done checking " + str(len(cards.json())) + " cards of page " + str(my_page)+ "."

‎README.md

+7-38
Original file line numberDiff line numberDiff line change
@@ -1,42 +1,11 @@
1-
# bugzilla2github
1+
# Scripts
2+
23
Code to migrate Bugzilla (version 4.0.2) to GitHub issues
4+
and other stuff that happened after that.
35

46
# Introduction
5-
For years all bugs have been managed using Bugzilla on bugs.libreplan.org
6-
For several reasons this however is coming to an end.
7-
8-
The succesor is going to be GitHub Issues.
9-
10-
This means all current issues need to be transfered to GitHub Issues.
11-
12-
For this GitHub has a special REST API
13-
14-
# How to go from Bugzilla to GitHub Issues
15-
There are several steps to consider
16-
- How to retrieve from Bugzilla: we chose to use a direct database connection
17-
- How to upload issues to GitHub: we chose to use a special GitHub API
18-
- How to handle assignees: Chosen to reset to empty
19-
- How to handle attachments: Since the GitHub API does not support adding attachments there are several options:
20-
- It is possible to manually upload attachments to issues. At over 500 attachments this was not a really an option.
21-
- There is an unmaintained ruby script to upload files to GitHub. Decided not to use it.
22-
- Since the migration is only done once we can dump all attachments to a subdirectory, upload that to a repo and generate a link to the file in the issue comments. So that is what we will do.
23-
- Yes, we could do fancy "if already exist..." etc, but this is a one off, so we simply add issues.
24-
25-
# Sources of inspiration
26-
GitHub description of API: https://gist.github.com/jonmagic/5282384165e0f86ef105
27-
28-
Bugzilla database model: https://www.bugzilla.org/docs/2.16/html/dbschema.html
29-
And a little help: https://www.ravenbrook.com/tool/bugzilla-schema/
30-
31-
Inspired by the work of Andriy Berestovskyy
32-
who wrote Bugzilla XML File to GitHub Issues Converter
33-
https://github.com/semihalf-berestovskyy-andriy/tools/blob/master/bugzilla2github
34-
Elaborate description
35-
https://www.theozimmermann.net/2017/10/bugzilla-to-github
36-
37-
38-
example: http://bugs.libreplan.org/attachment.cgi?id=212
39-
belongs to ticket 635
40-
41-
7+
So first I build bugzilla2github.py.
8+
Soon after I build AddIssuesToProject.py
9+
Next I build FindCommitForIssue.py
4210

11+
All script shave their own md file with some explanation.

‎bugzilla2github.md

+39
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
# bugzilla2github
2+
Code to migrate Bugzilla (version 4.0.2) to GitHub issues
3+
4+
## Introduction
5+
For years all bugs have been managed using Bugzilla on bugs.libreplan.org
6+
For several reasons this however is coming to an end.
7+
8+
The succesor is going to be GitHub Issues.
9+
10+
This means all current issues need to be transfered to GitHub Issues.
11+
12+
For this GitHub has a special REST API
13+
14+
## How to go from Bugzilla to GitHub Issues
15+
There are several steps to consider
16+
- How to retrieve from Bugzilla: we chose to use a direct database connection
17+
- How to upload issues to GitHub: we chose to use a special GitHub API
18+
- How to handle assignees: Chosen to reset to empty
19+
- How to handle attachments: Since the GitHub API does not support adding attachments there are several options:
20+
- It is possible to manually upload attachments to issues. At over 500 attachments this was not a really an option.
21+
- There is an unmaintained ruby script to upload files to GitHub. Decided not to use it.
22+
- Since the migration is only done once we can dump all attachments to a subdirectory, upload that to a repo and generate a link to the file in the issue comments. So that is what we will do.
23+
- Yes, we could do fancy "if already exist..." etc, but this is a one off, so we simply add issues.
24+
25+
## Sources of inspiration
26+
GitHub description of API: https://gist.github.com/jonmagic/5282384165e0f86ef105
27+
28+
Bugzilla database model: https://www.bugzilla.org/docs/2.16/html/dbschema.html
29+
And a little help: https://www.ravenbrook.com/tool/bugzilla-schema/
30+
31+
Inspired by the work of Andriy Berestovskyy
32+
who wrote Bugzilla XML File to GitHub Issues Converter
33+
https://github.com/semihalf-berestovskyy-andriy/tools/blob/master/bugzilla2github
34+
Elaborate description
35+
https://www.theozimmermann.net/2017/10/bugzilla-to-github
36+
37+
38+
example: http://bugs.libreplan.org/attachment.cgi?id=212
39+
belongs to ticket 635

0 commit comments

Comments
 (0)