- Clone the project code(https://github.com/bytedance/ByteX) to your computer and checkout a new branch for developing.
- ByteX project is composed by each module,it means that one plugin corresponds to one module.So. just create a new java library module.
- And then,configare in the module's build.gradle
//Use common dependencies and publish configuration
apply from: rootProject.file('gradle/plugin.gradle')
//extra dependencies
dependencies {
compile project(':TransformEngine')
implementation "br.usp.each.saeg:asm-defuse:0.0.5"
}The plugin module is ready now!!!
- Crate a new plugin based on ByteX, you need to create at least two classes
- One is Extension:It is much more like java bean which is used to configure the plugin
- Another is Plugin,It must implement the Plugin interface. Of course you can implement this Plugin in a simpler way:inherit from the abstract class AbsMainProcessPlugin or CommonPlugin directly. A simple example is shown as follow:
//@PluginConfig("bytex.sourcefile")
public class SourceFileKillerPlugin extends CommonPlugin<SourceFileExtension, SourceFileContext> {
@Override
protected SourceFileContext getContext(Project project, AppExtension android, SourceFileExtension extension) {
return new SourceFileContext(project, android, extension);
}
@Override
public boolean transform(@Nonnull String relativePath, @Nonnull ClassVisitorChain chain) {
//We are going to modify the bytecodes, so we need to register a ClassVisitor
chain.connect(new SourceFileClassVisitor(extension));
return super.transform(relativePath, chain);
}
@Nonnull
@Override
public TransformConfiguration transformConfiguration() {
return new TransformConfiguration() {
@Override
public boolean isIncremental() {
//The plugin is incremental by default.It should return false if incremental is not supported by the plugin
return true;
}
};
}
}- After creating the plugin, you need to make gradle recognize our plugin. We have two configurate methods shown as follow：
- Use Annotation(recommanded)
@PluginConfig("bytex.sourcefile")
public class SourceFileKillerPlugin extends CommonPlugin<SourceFileExtension, SourceFileContext> {...}
- write properties file
We need to create a properties file in the resource directory. The file name of the properties file corresponds to the id of the plugin. The id of the plugin is decided by yourself but it must be unique in the project. As is shown in the figure below, the properties file name is bytex.sourcefile.properties, you can configure it like this:
You need to configure the full class name (package name + class name) of our Plugin class in the properties file, for example:
implementation-class=com.ss.android.ugc.bytex.sourcefilekiller.SourceFileKillerPluginThen you need to configure this plugin to use this plugin in your app project:
apply plugin: 'bytex.sourcefile'Now, our new plugin has begun to take shape, to make our plugin process class files, you need to define the corresponding ClassVisitor or directly operate the ClassNode.
- publish to local
Execute the publish script in the root directory of the project.
./publish.sh
Or double-click uploadArchives to publish the plugin to the gralde_plugins directory of the local project.
You need publish the pulgin again if you want to take effect the changed code.
- publish to maven
If the plugin has been developed and passed the test, we need to publish the plugin to the maven to integrate it into the actual project. First,create a new local.properties configuration file in the root directory of ByteX and add the following configuration:
UPLOAD_MAVEN_URL=xxx
UPLOAD_MAVEN_URL_SNAPSHOT=xxx
USERNAME=xxx
PASSWORD=xxx
USERNAME_SNAPSHOT=xxx
PASSWORD_SNAPSHOT=xxx
Then, upgrade the upload_version of ext.gradle.
Similarly, execute the script or double-click uploadArchives to publish the plugin to online maven.
./publish.sh -m
- publish to snapshot
version=$current_version-${user.name}--SNAPSHOT
./publish.sh -m -t
Create a new 'run configuration' in AndroidStudio.
After publishing the plugin locally and connecting it to the app project, append parameters at the end of the build-command before executing the build-command.For example:
./gradlew clean :example:assembleDouyinCnRelease -Dorg.gradle.debug=true --no-daemon
Then switch to the Configuration that you created just now, and click the debug button.
SourceFileKiller is a custom plugin with less code and can be used as a demo. It does very simple things：delete SourceFile and line number attributes in bytecodes
If you need to develop a plugin based on ByteX in external project , you need to configure dependencies like below:
compile gradleApi()
compileOnly "com.android.tools.build:gradle:$gradle_version"
compile "com.bytedance.android.byteX:common:${bytex_version}"If you want to register the plugin with annotations, you can introduce the following dependencies (optional):
compile "com.bytedance.android.byteX:PluginConfigProcessor:${bytex_version}"
kapt "com.bytedance.android.byteX:PluginConfigProcessor:${bytex_version}"ByteX is based on ASM, we can read and write class files by registering ClassVisitor or operating ClassNode directly when processing class files. (If you need to receive the bytecode of a file as input, you can refer to the Advanced API below).
By default, the Transform formed by ByteX has at least one regular process (MainProcess) for Class files which includes the following steps:
- traverse callback：Iterate through all the build products (class files and jar class) in the project once, do analysis only and without modifying the input files;
- traverseAndroidJar callback：Iterate through all the class files in android.jar (The version of android.jar is determined by the target api in the project). It is designed for building a complete class diagram.
- transform callback：Iterate through all the build products in the project again, process the class file and output it (It may be directly written to the file as transform outputs or be used as input for the next process).
So, one process will traverse through all the classes in the project twice. We call the transform process as one TransformFlow. Developers can design their own TransformFlow( it can contain multiple traverses, or only contain transform classes but no traverse, etc.), please refer to the Advanced API.
Let's back to the SourceFileKiller Plugin we talked before, the plugin inheriteds from CommonPlugin,if we need to process the class file during the transform phase (the third step), the Plugin class needs to override one of the following two methods:
/**
* transform all the classes in the project
*
* @param relativePath relativePath of the class
* @param chain object for ClassVisitor registration
* @return if return true, this class will be outputed ；if return false, this class will be deleted.
*/
@Override
public boolean transform(String relativePath, extension chain) {
chain.connect(new SourceFileClassVisitor(context));
return true;
}
/**
* transform all the classes in the project
*
* @param relativePath relativePath of the class
* @param node classNode which contains all class infos.
* @return if return true, this class will be outputed ；if return false, this class will be deleted.
*/
@Override
public boolean transform(String relativePath, ClassNode node) {
// do something with ClassNode
return true;
}We can see that the only difference between these two overloaded methods is their input parameters.The former uses ASM's ClassVisitor, and the latter uses ASM's Tree API, which can directly handle ClassNode.
Similarly, if we need to analyze the class file during the traverse phase, the Plugin class can override the following methods:
/**
* traverse all the classes in the project
*
* @param relativePath relativePath of the class
* @param chain object for ClassVisitor registration
*/
void traverse(@Nonnull String relativePath, @Nonnull ClassVisitorChain chain);
/**
* traverse all the classes in the project
*
* @param relativePath relativePath of the class
* @param node classNode which contains all class infos.
*/
void traverse(@Nonnull String relativePath, @Nonnull ClassNode node);We recommend developers record all the modifications the plugin did while transforming the classes into logs. We can find out what has been modified by our plugins and find bugs through these logs. Each plugin has its own logger and log file which provided by ByteX. Developers can record all changes by ByteX logger as easily as common Logger When you need to record logs, you could obtain the Logger object from the context and call the corresponding log method.
The logs will be recorded in the file whose path will locate at the app/build/ByteX/$ {variantName}/${extension_name}/${logFile}. If the logFile is not configured in gradle, the file name will use $(extension_name)_log.txt by default.
At the same time, a visual html log file will be generated as transform ends. The data of this html page comes from plugins of the transform, developers don`t need to care about it, it is generated automatically. The file locates at app/build/ByteX/ByteX_report_{transformName}.html.
Tips: If the plugin has needs to generate extra files, we recommend developers use context.buildDir() to get a directory to place the files, This directory locates at the app/build/ByteX/$ {extension_name}/
In order to provide more flexibility for ByteX-based plugins, we introduce the concept of TransformFlow.
The process of processing all the build products (usually class files) is defined as a TransformFlow. A plugin can run in an independent TransformFlow , or you can take a ride of the global MainTransformFlow（Traverse, traverseAndroidJar and transform form a MainTransformFlow）.
You need to override provideTransformFlow which is a method belongs to IPlugin if you want use a customized TransformFlow for the plugin.
// Hitchhiking to the global MainTransformFlow, the way of most plugins use
@Override
protected TransformFlow provideTransformFlow(@Nonnull MainTransformFlow mainFlow, @Nonnull TransformContext transformContext) {
return mainFlow.appendHandler(this);
}
// create a MainTransformFlow which is independent of the global MainTransformFlow
@Override
protected TransformFlow provideTransformFlow(@Nonnull MainTransformFlow mainFlow, @Nonnull TransformContext transformContext) {
return new MainTransformFlow(transformer, new BaseContext(project, android, extension));
}
// create a customized TransformFlow which is independent of the global MainTransformFlow
@Override
protected TransformFlow provideTransformFlow(@Nonnull MainTransformFlow mainFlow, @Nonnull TransformContext transformContext) {
return new AbsTransformFlow(transformer, new BaseContext(project, android, extension)) {
@Override
protected AbsTransformFlow beforeTransform(Transformer transformer) {
return this;
}
@Override
protected AbsTransformFlow afterTransform(Transformer transformer) {
return this;
}
@Override
public void run() throws IOException, InterruptedException {
// do something in flow.
}
};
}
Normally，there should be a Class Diagram in each TransformFlow which constains all class relationship between classes of project、classes of jar and classes of Android.jar，it depends on your implementation of TransformFlow.
When your plugin run in MainTransformFlow (the default is this TransformFlow), the plugin will generate the class diagram of this TransformFlow automatically after traverse (including traverseArtifactOnly and traverseAndroidJarOnly), and the diagram will be placed in the corresponding context object. Class graph object can be obtained by call context.getClassGraph().
public class BaseContext<E extends BaseExtension> {
protected final Project project;
protected final AppExtension android;
public final E extension;
private ILogger logger;
private Graph classGraph;//class diagram
...
public Graph getClassGraph() {
return classGraph;
}
}Notes:
- In cases when TransformFlow does not complete the traverse (to be precise, beforeTransform of CommonPlugin), the class diagram does not exist, and the object that obtains the class diagram will be null.
- Every plugin that inherits from CommonPlugin must call the corresponding super method if it overrides the beforeTransform method, otherwise the class diagram object will not be passed to the Context object of the current plugin.
- The two TransformFlow class diagrams are isolated. Generally, each TransformFlow will modify the classes, the class diagrams generated by the two TransformFlows are generally different.
ByteX has basic reading and output capabilities for INPUTS.If you need to obtain more information such as transform inputs, project inputs, and input aars, you can use the capabilities provided by the Engine layer to obtain the corresponding file inputs.
You can get all the inputs of the transform by following way:
context.getTransformContext().allFiles()
You can get all the merged resources of the project by following way:
context.getTransformContext().getArtifact(Artifact.MERGED_RES)
You can get the location of the file by scope by following way:
context.getTransformContext().getLocator().findLocation("${className}.class",SearchScope.ORIGIN)
MainProcessHandler is bound to the MainTransformFlow processor, and each class will be processed in each step by calling the corresponding method of MainProcessHandler for processing. Generally, our plugins have already implemented this interface, and developers can override the corresponding methods to get the corresponding callbacks.
-
There are series of methods like init、 traverse, and transform all process class files through ASM in the MainProcessHandler. In order to provide greater flexibility, you can register your own FileProcessor by overriding the
List<FileProcessor> process (Process process)method . -
There a method named flagForClassReader in the MainProcessHandler which could customize the flag passed in when ClassReader calls the accept method to read the class file.The default value is
ClassReader.SKIP_DEBUG
If developers don`t want to use the upper-layer interface encapsulated by ASM to process class files, ByteX also provides a low-level API-FileProcessor.
FileProcessor is similar to the interceptor design of OkHttp. Each class file will be processed through a series of FileProcessors. The advantages of using this interface is that it is more flexible! As an interceptor, you can make the subsequent FileProcessor finish processing before processing, or you can even process it without passing it to the following FileProcessor.
public interface FileProcessor {
Output process(Chain chain) throws IOException;
interface Chain {
Input input();
Output proceed(Input input) throws IOException;
}
}
public class CustomFileProcessor implements FileProcessor {
@Override
public Output process(Chain chain) throws IOException {
Input input = chain.input();
FileData fileData = input.getFileData();
// do something with fileData
return chain.proceed(input);
}
}To register a custom FileProcessor, we also provide a more convenient way:Register the FileProcessor with the annotation @Processor on the custom Plugin.
@Processor(implement = CustomFileProcessor.class)
@Processor(implement = CustomFileProcessor.class, process = Process.TRAVERSE)
public class CustomPlugin extends CommonPlugin<Extension, Context> {...}FileHandler is an interface that is further encapsulated by FileProcessor.The input parameter is a FileData, and the FileData contains the bytecode of the file.
public interface FileHandler {
void handle(FileData fileData);
}
public class CustomFileHandler implements FileHandler {
@Override
public void handle(FileData fileData) {
// do something with fileData
}
}To register a custom FileHandler, just like FileProcessor, we also provide a more convenient way:Register the FileHandler with the annotation @Handler on the custom Plugin.
@Handler(implement = CustomFileHandler.class)
@Handler(implement = CustomFileHandler.class, process = Process.TRAVERSE)
public class FlavorCodeOptPlugin extends CommonPlugin<Extension, Context> {...} The methods in this interface correspond to the methods in the Transform interface in the Transform API.
Each plugin can customize some configurations by overriding the corresponding interface method belongs to transformConfiguration.
For example:
@Override
public TransformConfiguration transformConfiguration() {
return new TransformConfiguration() {
@Override
public Set<QualifiedContent.ContentType> getInputTypes() {
return TransformManager.CONTENT_JARS;
}
};
} Using transform api by calling registerTransform could only run your plugin before proguard or dexBuilder.If you have a plugin which must run after proguard execution,or run before（after) any task execution，you could apply the HookTransform/Task feature provided by ByteX.ByteX provide two methods(all named as hookTask) to complete the feature.(The hookTransformName method design in previous versions is still supported but marked as Deprecated，and ByteX give priority to this solution，the new solution will work on if the plugin hooks a non-transform task or runs after task execution)
For example, if you want your plugin to be executed after proguard (before dexBuilder), you can do this:
public class DoAfterProguardPlugin extends CommonPlugin<Extension, Context> {
/**
* eg：dexBuilder
* use {@link #hookTask()} and {@link #hookTask(Task)} instead
*/
@Deprecated
@Nullable
String hookTransformName() {
return "dexBuilder";
}
/**
* use hook mode.false by default
*/
boolean hookTask() {
return true;
}
/**
* How to hook this task
*
* @param task
* @return {@link HookType#Befor:plugin will run before task execution
* {@link HookType#After} :plugin will run after task execution
* {@link HookType#None} do not hook the task
*/
@Nonnull
HookType hookTask(@Nonnull Task task) {
if(task.getName().contains("dexBuilder")){
return HookType.Before;
}
return HookType.None;
}
}Some plugins may run after proguard execution，and these plugins usually need doing something like obfuscated or de-obfuscated.ByteX provides a set of class-tools for parsing proguard-mapping:
//obtain mapping file
File mappingFile = context.getTransformContext().getProguardMappingFile();
//read and parse mapping file
MappingReader mappingReader = MappingReader(mappingFile);
MappingProcessor mappingProcessor = new FullMappingProcessor();
//The class name in the mapping file is separated by '.', but internal name or desc is commonly used in asm, so we provide a MappingProcessor adapter
mappingProcessor = new InternalNameMappingProcessor(mappingProcessor);
mappingReader.pump(mappingProcessor);
//At the same time, we provide a simple retrace for asm users
FullInternalNameRetrace retrace = new FullInternalNameRetrace(mappingFile);
If you need to output some additional files during the transform, you can override the beforeTransform method and call the addFile method provided by TransformEngine.
@Override
public void beforeTransform(@Nonnull @NotNull TransformEngine engine) {
super.beforeTransform(engine);
engine.addFile("affinity",new FileData("addFile test".getBytes(),"com/ss/android/ugc/bytex/test.txt"));
}
The first parameter of addFile, affinity, can be set at will. If the affinity of the two addFile calls is the same, then the two files will be in the same output directory.
Plugin base on bytex configuared incremental by default.During incremental build, the ByteX framework will call the plugin‘s traverseIncremental method (two overloaded methods.Note: this method is called before the beforeTraverse), and will pass all files except the NotChanged.It will be decompressed if it is a Jar file and pass each entry as a single file (abstract into a FileData) in. The two methods are defined below:
/**
* traverse all incremental file which status is ADD,REMOVE or CHANGED
* file will be uncompressed which is jar input.
* only valid while during incremental build
*
* @param fileData incremental file
* @param chain If it is a class, the corresponding ClassVisitorChain will be passed to add a custom ClassVisitor, or null if there is not a class
*/
default void traverseIncremental(@Nonnull FileData fileData, @Nullable ClassVisitorChain chain) {
}
/**
* traverse all incremental file which status is ADD,REMOVE or CHANGED
* file will be uncompressed which is jar input.
* only valid while during incremental build
*
* @param fileData Incremental file, and the file must be a class file.
* @param node Tree Node
*/
default void traverseIncremental(@Nonnull FileData fileData, @Nonnull ClassNode node) {
}If your plugin supports incremental mode, but finds that incremental processing cannot be continued in the incremental mode, you can request full compilation from bytex in the beforeTraverse method or before:
context.getTransformContext().requestNotIncremental();If your plugin can't provide incremental compilation,please configuare as follows:
public class SourceFileKillerPlugin extends CommonPlugin<SourceFileExtension, SourceFileContext> {
@Nonnull
@Override
public TransformConfiguration transformConfiguration() {
return new TransformConfiguration() {
@Override
public boolean isIncremental() {
//The plugin is incremental by default.It should return false if incremental is not supported by the plugin
return true;
}
};
}
...
} ByteX adds the corresponding hook code at the beginning and end of the key lifecycle events for developers monitoring to do some operations when ByteX executes at certain timings.You can set your custom listeners with ByteXBuildListenerManager,for example:
ByteXBuildListenerManager.INSTANCE.registerByteXBuildListener(yourByteXBuildListener)
ByteXBuildListenerManager.INSTANCE.registerMainProcessHandlerListener(yourMainProcessHandlerListener) More details could be found in ByteXBuildListener and MainProcessHandlerListener
By default, ByteX has a built-in default listener to record lifecycle events . Results will be recorded in the two jsons located at app/build/ByteX/build/ after compilation is completed.
Users can config each plugin with it`s Extension in DSL.In addition, there are some properties witch are used by ByteX engine and could be configured in gradle.properties.Here is the list:
- bytex.globalIgnoreClassList:A relative file path relative to the project root directory,witch contains a list of classes(supporting pattern matching). If an exception occurs when processing the classes in the lists, ByteX will process it internally instead of throwing it to terminate the compilation
- bytex.enableDuplicateClassCheck:Whether to check for class duplication，boolean,true by default
- bytex.enableHtmlLog:Whether to generate html report，boolean,true by default
- bytex.enableRAMCache:Whether to enable memory cache to store ByteX`s cache, this configuration is used to optimize incremental build, because loading files may take time, boolean, true by default. If it is non-incremental (CI), it is recommended to configure to false
- bytex.enableRAMNodesCache:Whether to cache the Nodes in the Graph to the memory, it takes effect when enableRAMCache is true，boolean,true by default
- bytex.enableRAMClassesCache:Whether to cache the ClassEntity in Graph to memory, it takes effect when enableRAMCache is true，boolean,false by default
- bytex.asyncSaveCache：Whether to save the cache asynchronously (non-incremental plugins doing nothing), boolean , true by default.
- bytex.verifyProguardConfigurationChanged:Whether to verify whether the keep rules obtained by the plugin is the same as the keep rules when Proguard is executed. boolean , true by default.
- bytex.checkIncrementalInDebug:Whether to check non-incremental plugins running with its enableInDebug is true，boolean , false by default.
- bytex.enableSeparateProcessingNotIncremental:Whether to executing non-incremental plugins in single TransformTask automatically. If there is a plugin which is incremental, all ByteX plugins will run in non-incremental, which will greatly reduce the speed of incremental build. After the switch is turned on, the plugins that support increment will be executed together. Incremental plugins will run independently in a transform. oolean ,false by default.
- bytex.${extension.getName()}.alone:Whether to run the plugin independently，boolean , false by default.
- bytex.useFixedTimestamp:Whether to use the fixed timestamp (0) of the entity in the output jar, this has a great benefit for incremental compilation because the entity is same and the timestamp is unchanged,and tasks after bytex could hit the cache (such as DexBuilder).boolean , true by default.
- bytex.forbidUseLenientMutationDuringGetArtifact: Whether resolving configuration with lenient off when calling GradleEnv.getArtifact .This can fix the deadlock while calling FileCollection.getFiles() in some projects.boolean , false by default.
- bytex.ASM_API:The API value passed to ASM library, currently it can be ASM4, ASM5, ASM6, ASM7, ASM8, ASM9. ASM6 by default.
- bytex.enable_gradle_daemon_ignore_classloader_singleton:Whether to handle the memory leak of ByteX Graph singleton Class caused by the change of Gradle ClassLoader.true by default.
For new features, in principle, it can only be pulled from the develop branch.When it needs to be merged into the master branch, it needs to be merged into the develop branch first, and unified into the master branch later.
For simple bugfixes, you can pull the modification from the master branch directly and submit a mr to the master branch. After mr is merged, you need to submit another mr from master-> develop branch to sync the changes.
- Name Of Module
Every word must be lowercase, use "-" (underscore) to separate multiple words. Please end the module name with "-plugin" If it is a plugin module. If the feature contains both a plugin module and a runtime library ("java Or library "), please put all modules related into the one sub-directory in the ByteX root directory to avoid a feature scattered in the byteX root directory. - Package
Package name must start with "com.ss.android.ugc.bytex", and dd a subpackage name named after the feature to distinguish other modules - Comments
It is recommended to write comments for complex or core code.It is recommended to use English for comments. - Documentation
It is recommended to write the documentation after the plugin is stable.The document contains two copies in Chinese and English, both placed under the module of the corresponding plugin.