Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

serialized parsers #1815

Merged
merged 16 commits into from
Jun 22, 2023
Merged

serialized parsers #1815

Show file tree
Hide file tree
Changes from 12 commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
93faa5b
removed public from all functions in ParseTree
jurgenvinju Jun 19, 2023
fb319d7
added interface for saving and loading parsers from disk without expo…
jurgenvinju Jun 20, 2023
5759ad8
saving now works
jurgenvinju Jun 20, 2023
7ab7be7
added loading of the parser class, but it's not enough because I forg…
jurgenvinju Jun 20, 2023
6227783
now collecting all necessary sub-classes in a jar as well and storing…
jurgenvinju Jun 21, 2023
1011cde
now we store and retrieve a parser from a jar file because the genera…
jurgenvinju Jun 21, 2023
d7abd51
avoid to load a parser generator only for getting the parser method name
jurgenvinju Jun 21, 2023
addd58a
removed debug prints
jurgenvinju Jun 21, 2023
f64a18e
added tests and documentation for the store/load parsing features
jurgenvinju Jun 21, 2023
6dadd63
removed examples because they crash the tutor due to bootstrapping re…
jurgenvinju Jun 21, 2023
6024915
set errorsAsWarnings to true otherwise the tutor chokes on uses of ne…
jurgenvinju Jun 21, 2023
800c6c3
Revert "removed examples because they crash the tutor due to bootstra…
jurgenvinju Jun 21, 2023
b643d7a
Update Relation.rsc
jurgenvinju Jun 22, 2023
6ba6a97
Update Relation.rsc
jurgenvinju Jun 22, 2023
db7f82f
Update ParseTree.rsc
jurgenvinju Jun 22, 2023
01cbace
Update ParseTree.rsc
jurgenvinju Jun 22, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,7 +125,7 @@
<artifactId>rascal-maven-plugin</artifactId>
<version>${rascal-maven.version}</version>
<configuration>
<errorsAsWarnings>false</errorsAsWarnings>
<errorsAsWarnings>true</errorsAsWarnings>
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is necessary due to double presence of rascal.jar during the tutor run. For some reason the old builtins are linked against the new code.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

have to fix this in the tutor maven-plugin somehow.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and errors as warnings fixes this? interesting

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

well yes, the tutor fails then on the ParseTree module, but the build isn't stopped by this because those errors are reported as warnings only. When I do a full bootstrap, the error goes away again, but I'd rather not have te tutor's own run-time interfere with the run-time of the examples it is running.. very difficult in the case of rascal.jar. I believe it will require JVM module layers to fix this.

<bin>${project.build.outputDirectory}</bin>
<srcs>
<src>${project.basedir}/src/org/rascalmpl/library</src>
Expand Down
33 changes: 33 additions & 0 deletions src/org/rascalmpl/interpreter/utils/JavaBridge.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@
*******************************************************************************/
package org.rascalmpl.interpreter.utils;

import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.io.PrintWriter;
Expand All @@ -30,6 +31,7 @@
import javax.tools.JavaFileObject;
import javax.tools.ToolProvider;

import org.checkerframework.checker.units.qual.t;
import org.rascalmpl.ast.Expression;
import org.rascalmpl.ast.FunctionDeclaration;
import org.rascalmpl.ast.KeywordFormal;
Expand Down Expand Up @@ -107,6 +109,10 @@ public JavaBridge(List<ClassLoader> classLoaders, IValueFactory valueFactory, Co
public <T> Class<T> compileJava(ISourceLocation loc, String className, String source) {
return compileJava(loc, className, getClass(), source);
}

public void compileJava(ISourceLocation loc, String className, String source, OutputStream classBytes) {
compileJava(loc, className, getClass(), source, classBytes);
}

public <T> Class<T> compileJava(ISourceLocation loc, String className, Class<?> parent, String source) {
try {
Expand All @@ -130,6 +136,33 @@ public <T> Class<T> compileJava(ISourceLocation loc, String className, Class<?>
}
}

public Class<?> loadClass(InputStream in) throws IOException, ClassNotFoundException {
List<String> commandline = Arrays.asList(new String[] {"-proc:none", "-cp", config.getRascalJavaClassPathProperty()});
JavaCompiler<?> javaCompiler = new JavaCompiler<Object>(getClass().getClassLoader(), null, commandline);
return javaCompiler.load(in);
}

public <T> void compileJava(ISourceLocation loc, String className, Class<?> parent, String source, OutputStream classBytes) {
try {
// watch out, if you start sharing this compiler, classes will not be able to reload
List<String> commandline = Arrays.asList(new String[] {"-proc:none", "-cp", config.getRascalJavaClassPathProperty()});
JavaCompiler<T> javaCompiler = new JavaCompiler<T>(parent.getClassLoader(), null, commandline);
javaCompiler.compile(classBytes, className, source, null);
}
catch (ClassCastException e) {
throw new JavaCompilation(e.getMessage(), e);
}
catch (JavaCompilerException e) {
if (!e.getDiagnostics().getDiagnostics().isEmpty()) {
Diagnostic<? extends JavaFileObject> msg = e.getDiagnostics().getDiagnostics().iterator().next();
throw new JavaCompilation(msg.getMessage(null) + " at " + msg.getLineNumber() + ", " + msg.getColumnNumber() + " with classpath [" + config.getRascalJavaClassPathProperty() + "]", e);
}
else {
throw new JavaCompilation(e.getMessage(), e);
}
}
}

private String getClassName(FunctionDeclaration declaration) {
Tags tags = declaration.getTags();

Expand Down
84 changes: 82 additions & 2 deletions src/org/rascalmpl/interpreter/utils/JavaCompiler.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@

import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
Expand All @@ -28,6 +29,11 @@
import java.util.List;
import java.util.Map;
import java.util.Map.Entry;
import java.util.jar.Attributes;
import java.util.jar.JarEntry;
import java.util.jar.JarInputStream;
import java.util.jar.JarOutputStream;
import java.util.jar.Manifest;
import java.util.Set;

import javax.tools.DiagnosticCollector;
Expand All @@ -41,7 +47,9 @@
import javax.tools.StandardLocation;
import javax.tools.ToolProvider;

import org.rascalmpl.library.Prelude;
import org.rascalmpl.uri.URIUtil;
import org.rascalmpl.uri.jar.JarInputStreamFileTree;

/**
* Compile a String or other {@link CharSequence}, returning a Java
Expand Down Expand Up @@ -162,10 +170,13 @@ public synchronized Class<T> compile(final String qualifiedClassName,
final DiagnosticCollector<JavaFileObject> diagnosticsList,
final Class<?>... types) throws JavaCompilerException,
ClassCastException {
if (diagnosticsList != null)
if (diagnosticsList != null) {
diagnostics = diagnosticsList;
else
}
else {
diagnostics = new DiagnosticCollector<JavaFileObject>();
}

Map<String, CharSequence> classes = new HashMap<String, CharSequence>(1);
classes.put(qualifiedClassName, javaSource);
Map<String, Class<T>> compiled = compile(classes, diagnostics);
Expand Down Expand Up @@ -245,6 +256,28 @@ public synchronized Map<String, Class<T>> compile(
}
}

public void compile(OutputStream classBytes, String qualifiedClassName, CharSequence classSource, final DiagnosticCollector<JavaFileObject> diagnostics) throws JavaCompilerException {
Map<String, CharSequence> fileMap = new HashMap<>();

try {
fileMap.put(qualifiedClassName, classSource);

// ignoring return class here
compile(fileMap, diagnostics);

// side-effect alert:
// now the local classloader contains the .class file
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this a new side-effect? or was this always a side-effect?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it was always there, and necessary for the generated sub-classes bytecode to become available when instantiating the parser class object. However, normally we'd not ignore the return class and we would not be reading ourselves directly from the virtual file system later. Definitely not purely functional methods here.

classLoader.outputClassesToJar(qualifiedClassName, classBytes);
}
catch (IOException e) {
throw new JavaCompilerException(fileMap.keySet(), e, diagnostics);
}
}

public Class<?> load(InputStream file) throws IOException, ClassNotFoundException {
return classLoader.inputClassesFromJar(file);
}

/**
* Load a class that was generated by this instance or accessible from its
* parent class loader. Use this method if you need access to additional
Expand Down Expand Up @@ -310,6 +343,8 @@ public ClassLoader getClassLoader() {
public JavaFileManager getFileManager() {
return javaFileManager;
}


}

/**
Expand Down Expand Up @@ -561,6 +596,51 @@ final class ClassLoaderImpl extends ClassLoader {
super(parentClassLoader);
}

public void outputClassesToJar(String qualifiedClassName, OutputStream output) throws IOException {
Manifest manifest = new Manifest();
manifest.getMainAttributes().put(Attributes.Name.MANIFEST_VERSION, "1.0");
manifest.getMainAttributes().put(Attributes.Name.MAIN_CLASS, qualifiedClassName);

try (JarOutputStream target = new JarOutputStream(output, manifest)) {
for (Entry<String, JavaFileObject> entry : classes.entrySet()) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so this saves all previously compiled classes to the file?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, but the compiler is organized to be instantiated per run. So you get all the class files generated from compiling one Java source file loaded into one virtual file system that is wrapped by one classloader.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the case of parser generation, this entails one main file and as many sub-classes as there are non-terminals (including lists etc.) in the grammar.

Copy link
Member Author

@jurgenvinju jurgenvinju Jun 22, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you need to compile several collaborating source files, you can use 1 instance and still save the class files into one comprehensive jar file that will work if you use the load facility here to get the files back.

Note that I have chosen to not use .class file extensions and not change . to / to create a JVM folder structure. The reason is that this encoding is to remain opaque to our users, and maximally fast to deserialize. If we switch parsing technology, we may not even store our parsers as jars or class files.

String className = entry.getKey();
JavaFileObjectImpl file = (JavaFileObjectImpl) entry.getValue();
JarEntry jarEntry = new JarEntry(className);
jarEntry.setTime(file.getLastModified());
target.putNextEntry(jarEntry);
target.write(file.getByteCode());
target.closeEntry();
}
}
}

public Class<?> inputClassesFromJar(InputStream in) throws IOException, ClassNotFoundException {
try (JarInputStream jarIn = new JarInputStream(in)) {
Manifest mf = jarIn.getManifest();
String mainClass = (String) mf.getMainAttributes().get(Attributes.Name.MAIN_CLASS);
JarEntry jarEntry;

if (mainClass == null) {
throw new IOException("missing Main-Class in jar manifest");
}

while ((jarEntry = jarIn.getNextJarEntry()) != null) {
if (!jarEntry.isDirectory()) {
var className = jarEntry.getName();
var file = new JavaFileObjectImpl(className, JavaFileObject.Kind.CLASS);

try (var fo = file.openOutputStream()) {
fo.write(Prelude.consumeInputStream(jarIn));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we could use a regular copy loop, to avoid loading it all into memory first?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, but so far this code has proven to be blazingly fast so I thought this is nice and short as it is. We average around 4 ms for storing or reading a medium sized grammar (so that includes reading and writing all byte arrays for the JVM bytecode of all sub-classes and the main parser class.

}

add(className, file);
}
}

return loadClass(mainClass);
}
}

/**
* @return An collection of JavaFileObject instances for the classes in the
* class loader.
Expand Down
Loading