Deadlock D vs DebugifyingClassLoader

[Erhannis]

Found a deadlock. Here's what happens, as near as I can make out:
D is invoked to change e.g. an instance variable V on owner O. It locks on itself, and starts to make a Shadow of O. It calls getFields() on O's class. This has not yet happened, so the fields are not cached, and the Class delegates to DebugifyingClassLoader, attempting to lock it.
In the meantime, though, another thread is loading classes. DebugifyingClassLoader is locked to load the classes, and tries to call Debugify.debugifyClass(). This attempts to lock D, which results in deadlock.

Here's a trace of the deadlock, slightly anonymized:

	Thread [main] (Suspended)	
		owns: Object  (id=93)	
		owns: DebugifyingClassLoader  (id=84)	
			waited by: Thread [qtp1199720022-30-acceptor-0@771ed183-ServerConnector@1f9bf19f{SSL,[ssl, http/1.1]}{0.0.0.0:4439}] (Suspended)	
		waiting for: Class<T> (com.lambda.Debugger.D) (id=85)	
			owned by: Thread [qtp1199720022-30-acceptor-0@771ed183-ServerConnector@1f9bf19f{SSL,[ssl, http/1.1]}{0.0.0.0:4439}] (Suspended)	
		Debugify.debugifyClass(JavaClass, String) line: 293	
		DebugifyingClassLoader.findClass(String, boolean) line: 183	
		DebugifyingClassLoader.loadClass(String, boolean) line: 112	
		DebugifyingClassLoader(ClassLoader).loadClass(String) line: 357	
		Class<T>.getDeclaredFields0(boolean) line: not available [native method]	
		Class<T>.privateGetDeclaredFields(boolean) line: 2583	
		Class<T>.getDeclaredFields() line: 1916	
		ClassReflectionHelperImpl$8.run() line: 166	
		ClassReflectionHelperImpl$8.run() line: 162	
		AccessController.doPrivileged(PrivilegedAction<T>) line: not available [native method]	
		ClassReflectionHelperImpl.secureGetDeclaredFields(Class<?>) line: 162	
		ClassReflectionHelperImpl.getDeclaredFieldWrappers(Class<?>) line: 198	
		ClassReflectionHelperImpl.getAllFieldWrappers(Class<?>) line: 215	
		ClassReflectionHelperImpl$4.compute(Class<?>) line: 109	
		ClassReflectionHelperImpl$4.compute(Object) line: 105	
		LRUHybridCache$OriginThreadAwareFuture$1.call() line: 115	
		LRUHybridCache$OriginThreadAwareFuture$1.call() line: 111	
		FutureTask<V>.run() line: 266	
		LRUHybridCache$OriginThreadAwareFuture.run() line: 173	
		LRUHybridCache<K,V>.compute(K) line: 292	
		ClassReflectionHelperImpl.getAllFields(Class<?>) line: 333	
		Utilities.findInitializerFields(Class<?>, ServiceLocatorImpl, Collector) line: 1410	
		DefaultClassAnalyzer.getFields(Class<T>) line: 113	
		JerseyClassAnalyzer.getFields(Class<T>) line: 244	
		Utilities.getInitFields(Class<?>, ClassAnalyzer, Collector) line: 251	
		ClazzCreator<T>.initialize(ActiveDescriptor<?>, String, Collector) line: 157	
		ClazzCreator<T>.initialize(ActiveDescriptor<?>, Collector) line: 182	
		SystemDescriptor<T>.internalReify(Class<?>, Collector) line: 723	
		SystemDescriptor<T>.reify(Class<?>, Collector) line: 678	
		ServiceLocatorImpl.reifyDescriptor(Descriptor, Injectee) line: 416	
		Utilities.createService(ActiveDescriptor<T>, Injectee, ServiceLocatorImpl, ServiceHandle<T>, Class<?>) line: 2029	
		ServiceHandleImpl<T>.getService(ServiceHandle<T>) line: 105	
		ServiceHandleImpl<T>.getService() line: 87	
		FactoryCreator<T>.create(ServiceHandle<?>, SystemDescriptor<?>) line: 117	
		SystemDescriptor<T>.create(ServiceHandle<?>) line: 471	
		SingletonContext$1.compute(ContextualInput<Object>) line: 82	
		SingletonContext$1.compute(Object) line: 70	
		Cache$OriginThreadAwareFuture$1.call() line: 97	
		FutureTask<V>.run() line: 266	
		Cache$OriginThreadAwareFuture.run() line: 154	
		Cache<K,V>.compute(K) line: 199	
		SingletonContext.findOrCreate(ActiveDescriptor<T>, ServiceHandle<?>) line: 121	
		Utilities.createService(ActiveDescriptor<T>, Injectee, ServiceLocatorImpl, ServiceHandle<T>, Class<?>) line: 2064	
		ServiceLocatorImpl.internalGetService(Type, String, Unqualified, Annotation...) line: 711	
		ServiceLocatorImpl.getService(Class<T>, Annotation...) line: 653	
                ... main methods ...
		NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]	
		NativeMethodAccessorImpl.invoke(Object, Object[]) line: 62	
		DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43	
		Method.invoke(Object, Object...) line: 498	
		Debugger.runTarget(Class, Object[]) line: 1141	
		Debugger$1.run() line: 1199	
		Thread.run() line: 748	
	Thread [qtp1199720022-30-acceptor-0@771ed183-ServerConnector@1f9bf19f{SSL,[ssl, http/1.1]}{0.0.0.0:4439}] (Suspended)	
		owns: Class<T> (com.lambda.Debugger.D) (id=85)	
		waiting for: DebugifyingClassLoader  (id=84)	
		Class<T>.getDeclaredFields0(boolean) line: not available [native method]	
		Class<T>.privateGetDeclaredFields(boolean) line: 2583	
		Class<T>.privateGetPublicFields(Set<Class<?>>) line: 2614	
		Class<T>.getFields() line: 1557	
		Shadow.createShadowInternal(Object, boolean) line: 1160	
		Shadow.createShadow(Object, boolean) line: 1025	
		Shadow.get(Object, boolean) line: 1907	
		Shadow.get(Object) line: 1901	
		D.changeIVA(Object, Object, int, String, TraceLine) line: 389	
		ManagedSelector$Accept.<init>(ManagedSelector, SelectableChannel, Object) line: 556	
		ServerConnector$ServerConnectorManager(SelectorManager).accept(SelectableChannel, Object) line: 200	
		ServerConnector$ServerConnectorManager(SelectorManager).accept(SelectableChannel) line: 184	
		ServerConnector.accepted(SocketChannel) line: 362	
		ServerConnector.accept(int) line: 353	
		AbstractConnector$Acceptor.run() line: 603	
		QueuedThreadPool.runJob(Runnable) line: 672	
		QueuedThreadPool$2.run() line: 590	
		Thread.run() line: 748	

I'm using EclipseLink and Jetty and a few other heavy-loading things.

Now, I think I've maybe fixed the problem - it locked consistently before, and no longer locks. However, it's pretty ugly, I'm not sure it's covering all cases, it could theoretically cause some other kind of problem, and I may have applied it in more places than is necessary. Basically, I've swapped out most of D's synchronization with first synchronizing on the DebugifyingClassLoader, and THEN on D. This ensures same-order locking (in the case I saw), preventing deadlock. Still, note the caveats.

I'm making a pull request with this and a few other fixes; please look it over and see if it's acceptable.