Migrate to a non-Persona login system

[kltm]

Originally from geneontology/go-ontology#12795

"Persona is no longer actively developed by Mozilla. Mozilla has committed to operational and security support of the persona.org services until November 30th, 2016. On November 30th, 2016, Mozilla will shut down the persona.org services. Persona.org and related domains will be taken offline. If you run a website that relies on Persona, you need to implement an alternative login solution for your users before this date. For more information, see this guide to migrating your site away from Persona: https://wiki.mozilla.org/Identity/Persona_Shutdown_Guidelines_for_Reliers"

Minutes from call, http://wiki.geneontology.org/index.php/Ontology_meeting_2016-10-27#persona.org

Chris: you’ll see this reflected first in Noctua; will be able to log in via GH or Google; that’ll happen before the GOC mtg. For TG, we’ll probably use the same mechanism we use for Noctua. TG may have to be down for a week or so during that change.

[kltm]

It would be good to make sure we maintain the use of the same meta-files across the ecosystem.

[mcourtot]

Hi @kltm - we were planning to discuss this on the editors call tomorrow - do you or @cmungall already have a plan for action?

[kltm]

@nathandunn will soon do a quick evaluation of switching out the authentication system. After that, we will have a more clear idea of the next steps.
My understanding from @cmungall is that the goal is to have something functioning by the end of the month, but if that goal is not met, TG will be temporarily out of commission.

[cmungall]

Place to start:

• https://github.com/geneontology/termgenie/blob/master/TermGenie/WebApplications/TermGenieJQuery/src/main/webapp/js/termgenie-login.js
• https://github.com/geneontology/termgenie/blob/master/TermGenie/WebApplications/TermGenieJQuery/src/main/java/org/bbop/termgenie/services/authenticate/AuthenticationModule.java

[mcourtot]

Thanks for the update @kltm! Please let us know if you need testers or any other help.

[nathandunn]

@kltm Short answer is that I think it can be done. You might get lucky and be able to write a different impl class and redo the binding here:

https://github.com/geneontology/termgenie/blob/master/TermGenie/WebApplications/TermGenieJQuery/src/main/java/org/bbop/termgenie/services/authenticate/AuthenticationModule.java#L49

and created a GithubBindingImpl.java

You could use Shiro, but I would just use the github api itself since @hdietze put all of this nice work into it already:

http://github-api.kohsuke.org/

maven install here:

https://mvnrepository.com/artifact/org.kohsuke/github-api

Pretty

[kltm]

@nathandunn I'd assume that using the raw GitHub OAuth2 API here would be more work than just piggy-backing on a framework that already has all the exchanges figured out already. Looking at Kohsuke's API, it seems like it already assume a token anyways. Or am I missing something here?

[nathandunn]

			String json = postRequest(browserIdVerificationUrl, assertion, audienceValue);
			
			BrowserIdVerificationResponse details = gson.fromJson(json, BrowserIdVerificationResponse.class);
			if (details.status != null) {
				if ("okay".equals(details.status)) {
					if (details.email != null) {
						// TODO verify 'details.issuer' and
						// 'details.validUntil'
						UserData userData  = userDataProvider.getUserDataPerEMail(details.email);
						sessionHandler.setAuthenticated(userData, httpSession);
						return new JsonUserData(userData);
					}
				}
				else {
					return new JsonUserData("BrowserID could not be verified status: " + details.status + " reason: " + details.reason);
				}
			}

[nathandunn]

https://github.com/zhangkaitao/shiro-example/blob/master/shiro-example-chapter17-client/src/main/java/com/github/zhangkaitao/shiro/chapter18/oauth2/OAuth2Realm.java

[nathandunn]

Using Shiro directly (as in the above example code), we should be able to handle the web application flow directly?.

https://developer.github.com/v3/oauth/#web-application-flow

[kltm]

I'd think there would be some modification for the "github" flavor.

[nathandunn]

Also, this as well . . . https://developers.google.com/api-client-library/java/google-oauth-java-client/reference/1.20.0/com/google/api/client/auth/oauth2/package-summary

Trying to use the web-flow example will probably be the easiest, and removing the popup.

https://developer.github.com/v3/oauth/#web-application-flow

The problem is that no-one has their client_id when you call this. So, we have to register ourselves under oauth applications for TermGenie. Using the client_id from the site, it triggers authentication and the proper callback (yeah!). However, we need to make sure that the token is set properly.

• demonstrate

  • request proper client and redirect (triggering github login and "authorization") and callback - https://developer.github.com/v3/oauth/#parameters
  • put token in server - https://developer.github.com/v3/oauth/#2-github-redirects-back-to-your-site - via a post (can probably use the java api)
  • get the authenticated user info (username / email) to store on the session: https://developer.github.com/v3/oauth/#3-use-the-access-token-to-access-the-api

• integrate

  • rewrite "Login" to go to the new URL https://github.com/login/oauth/authorize?client_id=
  • wire so that we should be able to add a config_file on the classpath
  • create the config file termgenie.properties with client_id, client_secret, which will hopefully be built properly
  • callback should parse json to get the Access token (2)
  • access token should request the user and parse the json (3)
  • the session handler should be set

[kltm]

If I'm following you, nobody should have a client id except the calling app. The registration and matching of URLs between the client and the provider is just the price that has to be paid for credential-less login. IIRC, the pieces of "secret" information that need to be on the client side (and I'm calling TG the "client" in the oauth2 process here) are:

• clientID
• clientSecret
• callbackURL, that matches what is registered with github.

[nathandunn]

Yes. That is my understanding, which I hope is correct.

[nathandunn]

Getting "not founds" when attempting to post to get the access_token in step 2. Looking at this, it seems like I'm doing everything correctly, but will

https://gist.github.com/technoweenie/419219/a10301414e3c3d2aa63cde9018920d6457741a55

Looks like the solution might be to send as FormData instead of JSON:

https://gist.github.com/technoweenie/419219#gistcomment-1920651

[kltm]

From the gist:

This is still experimental and could change at any moment. This Gist will serve as a living document until it becomes finalized at Develop.GitHub.com.

[nathandunn]

@kltm I think the gist is experimental (and was helpful in getting going), but the Oath2 / developer stuff I don't think is.

[nathandunn]

java.lang.NoSuchMethodError: org.slf4j.spi.LocationAwareLogger.log(Lorg/slf4j/Marker;Ljava/lang/String;ILjava/lang/String;[Ljava/lang/Object;Ljava/lang/Throwable;)V
	at org.apache.commons.logging.impl.SLF4JLocationAwareLog.debug(SLF4JLocationAwareLog.java:133)
	at org.apache.http.client.protocol.RequestAuthCache.process(RequestAuthCache.java:78)
	at org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:109)
	at org.apache.http.protocol.HttpRequestExecutor.preProcess(HttpRequestExecutor.java:176)

Have to use:

mvn dependency:tree to solve this. Getting something like this right now:

- com.google.code.findbugs:jsr305:jar:2.0.0:compile
[INFO] +- log4j:log4j:jar:1.2.17:compile
[INFO] +- org.bushe:eventbus:jar:1.4:compile
[INFO] +- junit:junit:jar:4.11:test
[INFO] |  \- org.hamcrest:hamcrest-core:jar:1.3:test
[INFO] +- org.slf4j:slf4j-log4j12:jar:1.7.10:compile
[INFO] |  \- org.slf4j:slf4j-api:jar:1.7.10:compile
[INFO] \- org.slf4j:jcl-over-slf4j:jar:1.7.10:compile

Looks good. . . very weird. Going up further I get:

- org.json.rpc:jsonrpc:jar:1.1:compile
[INFO] |  +- com.google.code.gson:gson:jar:1.7.1:compile
[INFO] |  \- org.slf4j:slf4j-api:jar:1.5.8:compile

[nathandunn]

Fixed. Now generating this error:

posting '${urlParameters}'
[GO WARN] 2016-11-28 17:50:14 (GoYamlUserDataProvider:124) Could not retrieve user data for email: [email protected]
[GO INFO] 2016-11-28 17:50:28 (SessionHandlerImpl:63) Re-using session with id: m50spam0tvag1eu2qr9cwc7t8 from ip: 70.210.133.92
posting '${urlParameters}'
2016-11-28 17:50:32.325:WARN:oejs.ServletHandler:qtp1802598046-46: /go-termgenie/gh-access
java.lang.RuntimeException: Failed to authenticate
        at org.bbop.termgenie.servlets.GHAuthenticationAccessServlet.doGet(GHAuthenticationAccessServlet.java:98)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:622)

[nathandunn]

Basically, email is not returned:

		GHUserResponse ghUserResponse = gson.fromJson(result.toString(), GHUserResponse.class);

		boolean isAuthenticated = ghUserResponse.email!=null;

		if(isAuthenticated){
			UserData userData  = userDataProvider.getUserDataPerEMail(ghUserResponse.email);
			HttpSession httpSession = req.getSession();
			sessionHandler.setAuthenticated(userData, httpSession);
		}
		else{
			throw new RuntimeException("Failed to authenticate");
		}

but then somehow it is.

• handle error properly
• handle success properly

[kltm]

The GitHub ID should be returned.
If you are interested in testing, @mcourtot, @cmungall, and @ukemi may be able to help.

[nathandunn]

Full JSON was being returned, but should have been mapping login instead of username, which is why I might be getting this error:

GHAuthenticationAccessServlet  json response: {"email":"[email protected]"}
[GO WARN] 2016-11-28 19:45:21 (GoYamlUserDataProvider:124) Could not retrieve user data for email: [email protected]

[kltm]

I'm not sure I'm understanding why your email is involved here. This needs to be against github ids.

[kltm]

For GO, we'll be moving away (as in Wednesday) from any kind of email md5 verification and just have a direct lookup against the user's kltm username.

Basically, there needs to be a way to get the current contents of users.yaml into TG. And get it updateable remotely. The current contents of termgenie-mp-user-permissions.json was initially used to seed users.yaml (along with two other files), but has since diverged a bit. Heiko was going to write something at some point to switch to the users.yaml directly instead, but never got around to it--he just did manual updates, IIRC.

Anyways, the choices are to either write a conversion script to change users.yaml to the expected json format, or to write something for TG that reads users.yaml directly. The latter would be preferable as it would make making updates more transparent (and could be down with a git pull cron job on oven before TG's nightly restart), but may involve more fiddling than desired at this point.

As well, there are still TG users who do not have a listed GitHub account yet, so somebody (not you @nathandunn) would have to collect that and add it to the users.yaml file.

[nathandunn]

Now I'm getting this:

GHAuthenticationAccessServlet  json response: {"email":"[email protected]","login":"nathandunn"}
[GO WARN] 2016-11-28 21:18:50 (GoYamlUserDataProvider:123) Could not retrieve user data for github login: nathandunn

Somehow I'm not parsing or checking it right, or it is looking in another spot. More debugging.

[nathandunn]

Getting a String Value looks like this:

key [accounts]
value [{github=nathandunn, orcid=0000-0002-4862-3181, email-md5=[7da61ebeb6459e809d61aebf28f2a315]}]

[nathandunn]

@mcourtot, @cmungall, and @ukemi . . this should be working. Can you test http://go.termgenie.org/ and report success / failure?

Thanks!

[mcourtot]

Hi @nathandunn - testing on TG works for logging at http://go.termgenie.org/

But TG commit review doesn't seem to have the correct authorisation level (http://go.termgenie.org/TermGenieReview.html): I am logged in but "The current user (Melanie Courtot) is not allowed to use this review feature."
I don't know if the latter is already meant to be working or not, but as it's the same base URL I thought it was worth checking/mentioning.

[nathandunn]

@mcourtot Thanks. I'll take a look.

[nathandunn]

@mcourtot Are there other logins / pages that also have this error so that I can check them all at once (though its probably all the same error)?

[mcourtot]

Not as far as i know, those are the only 2 TG pages I use.

[nathandunn]

Thanks. That will make things easier.

[cmungall]

TermGenieManager.html

…

On Tue, Nov 29, 2016 at 10:46 Nathan Dunn ***@***.***> wrote: Thanks. That will make things easier. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#98 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADGOeg8POF3eNdM1WtrgUb4UBoxvSN0ks5rDEjXgaJpZM4Ky7ba> .

[nathandunn]

Ah thanks. Same issue here:

• http://go.termgenie.org/TermGenieManagement.html
• http://go.termgenie.org/TermGenieReview.html

[nathandunn]

@mcourtot , @cmungall, @ukemi I think that it is all working now. I was able to get into both interfaces with a login.

[mcourtot]

Sorry to be the bearer of bug reports... the login works fine, but in the TG review itself, the commit message reads
"TermGenie commit for user: pg reviewed by null"
this should (in my case) read "TermGenie commit for user: pg reviewed by mec"

[ukemi]

I am able to get into both interfaces, but when I am in the gatekeeper it says reviewed by null.

[nathandunn]

@mcourtot @ukemi Thanks for the reports. I'll take a look. Thanks for being patient (had not worked on this stack or GH integration before last week). Local testing is a bit difficult with this stack.

If there is anything else to test that you have access to, let me know so I can minimize the number of bounces on the server.

Thanks.

• fix "reviewed by" comments

[mcourtot]

It'd be good to check what happens when creating a new term in TG, as it should also pick up the user name (=creator of the request) then. For example in the above message "TermGenie commit for user: pg reviewed by mec", "pg" (aka Pascale Gaudet) create the term. Unfortunately atm TG is down due to unseat classes - geneontology/go-ontology#12834 - so not sure how to test (but presumably the user name gets passed the same way whether you create or review the query)

[nathandunn]

Thanks. Hopefully this will fix both of those.

[nathandunn]

Try it again . . may need to create a new commit. I updated the one that was there.

[mcourtot]

I still get 'reviewed by null' (and it's still the same term requested by Pascale, so not sure whether there is another issue as you indicated you committed this one?) Note we can't create a new commit as long as the ontology is unsatisfiable.

[nathandunn]

I'll wait until the ontology is fixed to see if anything further is wrong. Editing of the message must not be persisted.

[tberardini]

I was able to commit the term. I edited the commit message to be 'TermGenie commit for user: pg reviewed by tb' instead of 'TermGenie commit for user: pg reviewed by null' before committing. Didn't look for this thread before I did that, sorry. I'd been waiting for Pascale to get back to me on geneontology/go-ontology#12809 and she finally did.

Status for commit #88703: Success

ID: GO:1990986 Label: DNA recombinase disassembly

[mcourtot]

User reported commit issue at geneontology/go-ontology#12789 (comment)

I just added the GH account to the users.yaml file, but I'm not sure why he would be able to log in and not commit. (Or how he could login without having a 'known' GH account)

Edit: Bob clarified that he logged in using username and password, which would indicate Persona login. Probably the lack of GH ID means the system reverted to using Persona? Persona would then be used to log in, but the commits require having a GH ID, so he was prevented from actually creating a commit? Seems like a plausible explanation, but leaving it to the experts to confirm :)