Packaging with Jarjar? (Feature request?)

[tomwscott]

Is it possible / recommended / planned for jcompilo to package a project with a specified list of runtime dependencies à la uberjar or jarjar?

Has this been done elsewhere that can be provided as an example?

[danielbodart]

Yes this is planned, I've already done the hard part:

https://github.com/bodar/jcompilo/blob/master/src/com/googlecode/jcompilo/bytecode/ConstantPoolMapper.java

This rewrites constant pool entries in class files so you can "map" or regex any string. It's about 30% faster than ASM and 2-3x faster than JarJar (because it does it as you compile not after)

I was thinking that the easiest way to use this would be by convention, so you just add a 'internal.dependencies' file and it would then auto remap them to 'your.root.package.internal.XXXX' where XXXX equals the library package structure. Here is a complete example:

build/internal.dependencies contains

mvn:commons-codec:commons-codec:jar:1.10

Then lets say we where doing this in utterlyidle you would end up with the following package structure in the final jar:

com.googlecode.utterlyidle.internal.org.apache.commons.codec

Last question I have, which would you prefer?

1. Only putting classes that are referenced in, smaller jar but higher risk of missing a class if referenced by reflection or other method)
2. Just put everything from the Jar in, should work with reflection and all resources

I'm thinking just play it safe and shove everything in

[tomwscott]

I like the sledgehammer approach so I would vote for option 2 too.

One interesting point to make about internal dependencies is that it can be a bit confusing for external consumers of the package's API if the namespaces are changed and classes are prefixed with a $. JCompilo's build.java is a great example of this:

If I want to override one of the methods say: test so that it uses all files that end with Wibble instead, I need to create something like the following:

import com.googlecode.jcompilo.Environment;
import com.googlecode.jcompilo.convention.AutoBuild;
import com.googlecode.jcompilo.internal.totallylazy.$Sequence; /* Different Type */
import com.googlecode.jcompilo.tests.Tests;

import java.io.File;

import static com.googlecode.jcompilo.Compiler.compiler;
import static com.googlecode.jcompilo.internal.totallylazy.$Sequences.cons; /* Different Type */
import static com.googlecode.jcompilo.internal.totallylazy.$Strings.endsWith; /* Different Type */ 
import static com.googlecode.jcompilo.tests.Tests.tests;

public class build extends AutoBuild {
    public build(Environment environment) {
        super(environment);
    }

    @Override
    public boolean test() throws Exception {
        stage("test");
        $Sequence<File> productionJars = cons(mainJar(), dependencies());
        Tests tests = tests(env, productionJars, testThreads(), reportsDir(), endsWith("Wibble.java"), debug());
        return compiler(env, productionJars, compileOptions()).
                add(tests).compile(testDir(), testJar()) &&
                tests.execute(testJar());
    }
} 

I want to re-use a lot of the underlying functionality but the public API requires me to use the jcompilo.internal versions - this is something that caught me out initially.

I understand the need to do this for a 'library' package to avoid version conflicts but it can be confusing when consuming that library and the bundled libraries leak out into the public API. For packaging a standalone application this is less of a concern but then it also doesn't need namespace rewrites because it's not going to be consumed by another application.

I suppose having the ability to choose whether I want to rewrite namespaces or not would be useful so that I can make the most appropriate decision for the project that I'm creating - be it a library to distribute or a standalone application.

[danielbodart]

No you are quite right, it's really a mistake that that they have leaked
into the public api. Ideally they should all just be iterators/iterables.

On Thu, 9 Jul 2015 2:36 pm Tom Scott [email protected] wrote:

I like the sledgehammer approach so I would vote for option 2 too.

One interesting point to make about internal dependencies is that it can
be a bit confusing for external consumers of the package's API if the
namespaces are changed and classes are prefixed with a $. JCompilo's
build.java is a great example of this:

If I want to override one of the methods say: test so that it uses all
files that end with Wibble instead, I need to create something like the
following:

import com.googlecode.jcompilo.Environment;import com.googlecode.jcompilo.convention.AutoBuild;import com.googlecode.jcompilo.internal.totallylazy.$Sequence; /* Different Type /import com.googlecode.jcompilo.tests.Tests;
import java.io.File;
import static com.googlecode.jcompilo.Compiler.compiler;import static com.googlecode.jcompilo.internal.totallylazy.$Sequences.cons; / Different Type /import static com.googlecode.jcompilo.internal.totallylazy.$Strings.endsWith; / Different Type */ import static com.googlecode.jcompilo.tests.Tests.tests;
public class build extends AutoBuild {
public build(Environment environment) {
super(environment);
}

@Override
public boolean test() throws Exception {
    stage("test");
    $Sequence<File> productionJars = cons(mainJar(), dependencies());
    Tests tests = tests(env, productionJars, testThreads(), reportsDir(), endsWith("Wibble.java"), debug());
    return compiler(env, productionJars, compileOptions()).
            add(tests).compile(testDir(), testJar()) &&
            tests.execute(testJar());
}

}

I want to re-use a lot of the underlying functionality but the public API
requires me to use the jcompilo.internal versions - this is something
that caught me out initially.

I understand the need to do this for a 'library' package to avoid version
conflicts but it can be confusing when consuming that library and the
bundled libraries leak out into the public API. For packaging a standalone
application this is less of a concern but then it also doesn't need
namespace rewrites because it's not going to be consumed by another
application.

I suppose having the ability to choose whether I want to rewrite
namespaces or not would be useful so that I can make the most appropriate
decision for the project that I'm creating - be it a library to distribute
or a standalone application.

—
Reply to this email directly or view it on GitHub
#5 (comment).

[danielbodart]

Sorry for lack in progress, anyway I think I might have had a eureka moment.

How about if we pre-process the internal dependencies, so when you use JCompilo to download the dependency it transforms the java bytecode (and source) at download time. Then we compile as normal but our source code will reference the internal class names and then we just copy the files into the jar.

Pros:

• Faster - only transform when downloading dependency (instead of each time you compile)
• Easier to debug source code as the internal dependencies now match the source code you publish
• Any problems with the transformation should be debuggable and testable in your IDE as it just works with the post processed jars.
• Very easy to implement and parallelisable.

Cons:

• No support for publishing 2 libs, one with embedded deps and one without