Merge pull request #433 from sofastack/yuan_dev

Support dubbo 2.7
sofastack · Jan 22, 2024 · 74873e2 · 74873e2
2 parents 7e5c6f2 + 7c1ffc9
commit 74873e2
Show file tree

Hide file tree

Showing 33 changed files with 2,247 additions and 11 deletions.
diff --git a/docs/content/zh-cn/docs/contribution-guidelines/runtime/dubbo2.7.md b/docs/content/zh-cn/docs/contribution-guidelines/runtime/dubbo2.7.md
@@ -0,0 +1,194 @@
+---
+title: dubbo2.7 的多模块化适配
+date: 2024-1-19T19:55:35+08:00
+weight: 1
+---
+
+## 为什么需要做适配
+原生 dubbo2.7 在多模块场景下，无法支持模块发布自己的dubbo服务，调用时存在序列化、类加载异常等一系列问题。
+
+## 多模块适配方案
+
+dubbo2.7多模块适配SDK
+```xml
+<dependency>
+   <groupId>com.alipay.sofa.serverless</groupId>
+   <artifactId>sofa-serverless-adapter-dubbo2.7</artifactId>
+   <version>0.5.7-SNAPSHOT</version>
+</dependency>
+```
+
+主要从类加载、服务发布、服务卸载、服务隔离、模块维度服务管理、配置管理、序列化等方面进行适配。
+
+### 1. AnnotatedBeanDefinitionRegistryUtils使用基座classloader无法加载模块类
+com.alibaba.spring.util.AnnotatedBeanDefinitionRegistryUtils#isPresentBean
+
+```java
+public static boolean isPresentBean(BeanDefinitionRegistry registry, Class<?> annotatedClass) {
+    ...
+
+    //        ClassLoader classLoader = annotatedClass.getClassLoader(); // 原生逻辑
+    ClassLoader classLoader = Thread.currentThread().getContextClassLoader();   // 改为使用tccl加载类
+
+    for (String beanName : beanNames) {
+        BeanDefinition beanDefinition = registry.getBeanDefinition(beanName);
+        if (beanDefinition instanceof AnnotatedBeanDefinition) {
+            ...
+            String className = annotationMetadata.getClassName();
+            Class<?> targetClass = resolveClassName(className, classLoader);
+            ...
+        }
+    }
+
+    return present;
+}
+```
+
+### 2. 模块维度的服务、配置资源管理
+1. com.alipay.sofa.serverless.support.dubbo.ServerlessServiceRepository 替代原生 org.apache.dubbo.rpc.model.ServiceRepository
+
+原生service采用interfaceName作为缓存，在基座、模块发布同样interface，不同group服务时，无法区分，替代原生service缓存模型，采用Interface Class类型作为key，同时采用包含有group的path作为key，支持基座、模块发布同interface不同group的场景
+```java
+private static ConcurrentMap<Class<?>, ServiceDescriptor> globalClassServices = new ConcurrentHashMap<>();
+
+private static ConcurrentMap<String, ServiceDescriptor>   globalPathServices  = new ConcurrentHashMap<>();
+```
+
+2. com.alipay.sofa.serverless.support.dubbo.ServerlessConfigManager 替代原生 org.apache.dubbo.config.context.ConfigManager 
+
+    为原生config添加classloader维度的key，不同模块根据classloader隔离不同的配置
+
+```java
+final Map<ClassLoader, Map<String, Map<String, AbstractConfig>>> globalConfigsCache = new HashMap<>();
+
+public void addConfig(AbstractConfig config, boolean unique) {
+    ...
+    write(() -> {
+        Map<String, AbstractConfig> configsMap = getCurrentConfigsCache().computeIfAbsent(getTagName(config.getClass()), type -> newMap());
+        addIfAbsent(config, configsMap, unique);
+    });
+}
+private Map<String, Map<String, AbstractConfig>> getCurrentConfigsCache() {
+    ClassLoader contextClassLoader = Thread.currentThread().getContextClassLoader();   // 根据当前线程classloader隔离不同配置缓存
+    globalConfigsCache.computeIfAbsent(contextClassLoader, k -> new HashMap<>());
+    return globalConfigsCache.get(contextClassLoader);
+}
+```
+
+ServerlessServiceRepository 和 ServerlessConfigManager 都依赖 dubbo ExtensionLoader 的扩展机制，从而替代原生逻辑，具体原理可参考 org.apache.dubbo.common.extension.ExtensionLoader.createExtension
+
+### 3. 模块维度服务发布、服务卸载
+override DubboBootstrapApplicationListener 禁止原生dubbo模块启动、卸载时发布、卸载服务
+
+- com.alipay.sofa.serverless.support.dubbo.BizDubboBootstrapListener
+
+原生dubbo2.7只在基座启动完成后发布dubbo服务，在多模块时，无法支持模块的服务发布，Ark采用监听器监听模块启动事件，并手动调用dubbo进行模块维度的服务发布
+
+```java
+private void onContextRefreshedEvent(ContextRefreshedEvent event) {
+  try {
+      ReflectionUtils.getMethod(DubboBootstrap.class, "exportServices")
+          .invoke(dubboBootstrap);
+      ReflectionUtils.getMethod(DubboBootstrap.class, "referServices").invoke(dubboBootstrap);
+  } catch (Exception e) {
+
+  }
+}
+```
+
+原生dubbo2.7在模块卸载时会调用DubboShutdownHook，将JVM中所有dubbo service unexport，导致模块卸载后基座、其余模块服务均被卸载，Ark采用监听器监听模块spring上下文关闭事件，手动卸载当前模块的dubbo服务，保留基座、其余模块的dubbo服务
+
+```java
+private void onContextClosedEvent(ContextClosedEvent event) {
+        // DubboBootstrap.unexportServices 会 unexport 所有服务，只需要 unexport 当前 biz 的服务即可
+        Map<String, ServiceConfigBase<?>> exportedServices = ReflectionUtils.getField(dubboBootstrap, DubboBootstrap.class, "exportedServices");
+
+        Set<String> bizUnexportServices = new HashSet<>();
+        for (Map.Entry<String, ServiceConfigBase<?>> entry : exportedServices.entrySet()) {
+            String serviceKey = entry.getKey();
+            ServiceConfigBase<?> sc = entry.getValue();
+            if (sc.getRef().getClass().getClassLoader() == Thread.currentThread().getContextClassLoader()) {   // 根据ref服务实现的类加载器区分模块服务
+                bizUnexportServices.add(serviceKey);
+                configManager.removeConfig(sc);   // 从configManager配置管理中移除服务配置
+                sc.unexport();   // 进行服务unexport
+                serviceRepository.unregisterService(sc.getUniqueServiceName());   // 从serviceRepository服务管理中移除配置
+            }
+        }
+        for (String service : bizUnexportServices) {
+            exportedServices.remove(service);    // 从DubboBootstrap中移除该service
+        }
+    }
+```
+
+### 4. 服务路由
+- com.alipay.sofa.serverless.support.dubbo.ConsumerRedefinePathFilter
+
+dubbo服务调用时通过path从ServiceRepository中获取正确的服务端服务模型（包括interface、param、return类型等）进行服务调用、参数、返回值的序列化，原生dubbo2.7采用interfaceName作为path查找service model，无法支持多模块下基座模块发布同interface的场景，Ark自定义consumer端filter添加group信息到path中，以便provider端进行正确的服务路由
+
+```java
+public Result invoke(Invoker<?> invoker, Invocation invocation) throws RpcException {
+  if (invocation instanceof RpcInvocation) {
+      RpcInvocation rpcInvocation = (RpcInvocation) invocation;
+      // 原生path为interfaceName，如com.alipay.sofa.rpc.dubbo27.model.DemoService
+      // 修改后path为serviceUniqueName，如masterBiz/com.alipay.sofa.rpc.dubbo27.model.DemoService
+      rpcInvocation.setAttachment("interface", rpcInvocation.getTargetServiceUniqueName());   // 原生path为interfaceName，如
+  }
+  return invoker.invoke(invocation);
+}
+```
+
+### 5. 序列化
+- org.apache.dubbo.common.serialize.java.JavaSerialization
+- org.apache.dubbo.common.serialize.java.ClassLoaderJavaObjectInput
+- org.apache.dubbo.common.serialize.java.ClassLoaderObjectInputStream
+
+在获取序列化工具JavaSerialization时，使用ClassLoaderJavaObjectInput替代原生JavaObjectInput，传递provider端service classloader信息
+
+```java
+// org.apache.dubbo.common.serialize.java.JavaSerialization
+public ObjectInput deserialize(URL url, InputStream is) throws IOException {
+    return new ClassLoaderJavaObjectInput(new ClassLoaderObjectInputStream(null, is));   // 使用ClassLoaderJavaObjectInput替代原生JavaObjectInput，传递provider端service classloader信息
+}
+
+// org.apache.dubbo.common.serialize.java.ClassLoaderObjectInputStream
+private ClassLoader classLoader;
+
+public ClassLoaderObjectInputStream(final ClassLoader classLoader, final InputStream inputStream) {
+  super(inputStream);
+  this.classLoader = classLoader;
+}
+```
+
+- org.apache.dubbo.rpc.protocol.dubbo.DecodeableRpcInvocation 服务端反序列化参数
+
+```java
+// patch begin
+if (in instanceof ClassLoaderJavaObjectInput) {
+   InputStream is = ((ClassLoaderJavaObjectInput) in).getInputStream();
+   if (is instanceof ClassLoaderObjectInputStream) {
+      ClassLoader cl = serviceDescriptor.getServiceInterfaceClass().getClassLoader();  // 设置provider端service classloader信息到ClassLoaderObjectInputStream中
+      ((ClassLoaderObjectInputStream) is).setClassLoader(cl);
+   }
+}
+// patch end
+```
+- org.apache.dubbo.rpc.protocol.dubbo.DecodeableRpcResult 客户端反序列化返回值
+
+```java
+// patch begin
+if (in instanceof ClassLoaderJavaObjectInput) {
+   InputStream is = ((ClassLoaderJavaObjectInput) in).getInputStream();
+   if (is instanceof ClassLoaderObjectInputStream) {
+       ClassLoader cl = invocation.getInvoker().getInterface().getClassLoader(); // 设置consumer端service classloader信息到ClassLoaderObjectInputStream中
+       ((ClassLoaderObjectInputStream) is).setClassLoader(cl);
+   }
+}
+// patch end
+```
+
+## 多模块 dubbo2.7 使用样例
+
+[多模块 dubbo2.7 使用样例](https://github.com/sofastack/sofa-serverless/tree/master/samples/dubbo-samples/rpc/dubbo27/README.md)
+
+[dubbo2.7多模块适配sdk源码](https://github.com/sofastack/sofa-serverless/tree/master/sofa-serverless-runtime/sofa-serverless-adapter-ext/sofa-serverless-adapter-dubbo2.7)
+
diff --git a/docs/content/zh-cn/docs/contribution-guidelines/runtime/logback.md b/docs/content/zh-cn/docs/contribution-guidelines/runtime/logback.md
@@ -0,0 +1,114 @@
+---
+title: logback 的多模块化适配
+date: 2024-1-18T15:32:35+08:00
+weight: 1
+---
+
+## 为什么需要做适配
+原生 logback 只有默认日志上下文，各个模块间日志配置无法隔离，无法支持独立的模块日志配置，最终导致在合并部署多模块场景下，模块只能使用基座的日志配置，对模块日志打印带来不便。
+
+## 多模块适配方案
+Logback 支持原生扩展 ch.qos.logback.classic.selector.ContextSelector，该接口支持自定义上下文选择器，Ark 默认实现了 ContextSelector 对多个模块的 LoggerContext 进行隔离 （参考 com.alipay.sofa.ark.common.adapter.ArkLogbackContextSelector），不同模块使用各自独立的 LoggerContext，确保日志配置隔离
+
+启动期，经由 spring 日志系统 LogbackLoggingSystem 对模块日志配置以及日志上下文进行初始化
+
+指定上下文选择器为 com.alipay.sofa.ark.common.adapter.ArkLogbackContextSelector，添加JVM启动参数
+> -Dlogback.ContextSelector=com.alipay.sofa.ark.common.adapter.ArkLogbackContextSelector
+
+当使用 slf4j 作为日志门面，logback 作为日志实现框架时，在基座启动时，首次进行 slf4j 静态绑定时，将初始化具体的 ContextSelector，当没有自定义上下文选择器时，将使用 DefaultContextSelector, 当我们指定上下文选择器时，将会初始化 ArkLogbackContextSelector 作为上下文选择器
+
+ch.qos.logback.classic.util.ContextSelectorStaticBinder.init
+
+```java
+public void init(LoggerContext defaultLoggerContext, Object key) {
+  ...
+
+  String contextSelectorStr = OptionHelper.getSystemProperty(ClassicConstants.LOGBACK_CONTEXT_SELECTOR);
+  if (contextSelectorStr == null) {
+  contextSelector = new DefaultContextSelector(defaultLoggerContext);
+  } else if (contextSelectorStr.equals("JNDI")) {
+  // if jndi is specified, let's use the appropriate class
+  contextSelector = new ContextJNDISelector(defaultLoggerContext);
+  } else {
+  contextSelector = dynamicalContextSelector(defaultLoggerContext, contextSelectorStr);
+  }
+}
+
+static ContextSelector dynamicalContextSelector(LoggerContext defaultLoggerContext, String contextSelectorStr) {
+  Class<?> contextSelectorClass = Loader.loadClass(contextSelectorStr);
+  Constructor cons = contextSelectorClass.getConstructor(new Class[] { LoggerContext.class });
+  return (ContextSelector) cons.newInstance(defaultLoggerContext);
+}
+```
+
+在 ArkLogbackContextSelector 中，我们使用 ClassLoader 区分不同模块，将模块 LoggerContext 根据 ClassLoader 缓存
+
+根据 classloader 获取不同的 LoggerContext，在 Spring 环境启动时，根据 spring 日志系统初始化日志上下文，通过 org.springframework.boot.logging.logback.LogbackLoggingSystem.getLoggerContext 获取日志上下文，此时将会使用 Ark 实现的自定义上下文选择器 com.alipay.sofa.ark.common.adapter.ArkLogbackContextSelector.getLoggerContext() 返回不同模块各自的 LoggerContext
+
+```java
+public LoggerContext getLoggerContext() {
+  ClassLoader classLoader = this.findClassLoader();
+  if (classLoader == null) {
+      return defaultLoggerContext;
+  }
+  return getContext(classLoader);
+}
+```
+
+获取 classloader 时，首先获取线程上下文 classloader，当发现是模块的classloader时，直接返回，若tccl不是模块classloader，则从ClassContext中获取调用Class堆栈，遍历堆栈，当发现模块classloader时直接返回，这样做的目的是为了兼容tccl没有保证为模块classloader时的场景，
+比如在模块代码中使用logger打印日志时，当前类由模块classloader自己加载，通过ClassContext遍历可以最终获得当前类，获取到模块classloader，以便确保使用模块对应的 LoggerContext
+
+```java
+private ClassLoader findClassLoader() {
+  ClassLoader classLoader = Thread.currentThread().getContextClassLoader();
+  if (classLoader != null && CONTAINER_CLASS_LOADER.equals(classLoader.getClass().getName())) {
+      return null;
+  }
+  if (classLoader != null && BIZ_CLASS_LOADER.equals(classLoader.getClass().getName())) {
+      return classLoader;
+  }
+
+  Class<?>[] context = new SecurityManager() {
+      @Override
+      public Class<?>[] getClassContext() {
+          return super.getClassContext();
+      }
+  }.getClassContext();
+  if (context == null || context.length == 0) {
+      return null;
+  }
+  for (Class<?> cls : context) {
+      if (cls.getClassLoader() != null
+          && BIZ_CLASS_LOADER.equals(cls.getClassLoader().getClass().getName())) {
+          return cls.getClassLoader();
+      }
+  }
+
+  return null;
+}
+```
+
+获取到合适 classloader 后，为不同 classloader选择不同的 LoggerContext 实例，所有模块上下文缓存在 com.alipay.sofa.ark.common.adapter.ArkLogbackContextSelector.CLASS_LOADER_LOGGER_CONTEXT 中，以 classloader 为 key
+
+```java
+private LoggerContext getContext(ClassLoader cls) {
+  LoggerContext loggerContext = CLASS_LOADER_LOGGER_CONTEXT.get(cls);
+  if (null == loggerContext) {
+      synchronized (ArkLogbackContextSelector.class) {
+          loggerContext = CLASS_LOADER_LOGGER_CONTEXT.get(cls);
+          if (null == loggerContext) {
+              loggerContext = new LoggerContext();
+              loggerContext.setName(Integer.toHexString(System.identityHashCode(cls)));
+              CLASS_LOADER_LOGGER_CONTEXT.put(cls, loggerContext);
+          }
+      }
+  }
+  return loggerContext;
+}
+```
+
+## 多模块 logback 使用样例
+[多模块 logback 使用样例](https://github.com/sofastack/sofa-serverless/tree/master/samples/springboot-samples/logging/logback/README.md)
+
+[详细查看ArkLogbackContextSelector源码](https://github.com/sofastack/sofa-ark/blob/master/sofa-ark-parent/core/common/src/main/java/com/alipay/sofa/ark/common/adapter/ArkLogbackContextSelector.java)
+
diff --git a/...26/dubbo26base/src/main/java/com/alipay/sofa/rpc/dubbo26/base/Dubbo26BaseApplication.java b/...26/dubbo26base/src/main/java/com/alipay/sofa/rpc/dubbo26/base/Dubbo26BaseApplication.java
@@ -52,7 +52,7 @@ public void run(String... args) throws Exception {
     }
 
     protected void installBiz(String bizDir) throws Throwable {
-        String pathRoot = "rpc/dubbo26/";
+        String pathRoot = "samples/dubbo-samples/rpc/dubbo26/";
         File bizFile = new File(pathRoot + bizDir);
         if (bizFile.exists()) {
             File tmpFile = new File(pathRoot + "target/" + bizFile.getName());

diff --git a/samples/dubbo-samples/rpc/dubbo26/dubbo26biz/src/main/resources/application.properties b/samples/dubbo-samples/rpc/dubbo26/dubbo26biz/src/main/resources/application.properties
@@ -6,6 +6,8 @@ logging.level.com.alipay.sofa.arklet=INFO
 #logging.config=classpath:log4j2-spring.xml
 spring.main.allow-bean-definition-overriding=true
 
+spring.main.allow-bean-definition-overriding=true
+
 dubbo.application.logger=slf4j
 dubbo.application.name=biz
 dubbo.consumer.serialization=java

diff --git a/samples/dubbo-samples/rpc/dubbo26/dubbo26biz2/src/main/resources/application.properties b/samples/dubbo-samples/rpc/dubbo26/dubbo26biz2/src/main/resources/application.properties
@@ -6,6 +6,8 @@ logging.level.com.alipay.sofa.arklet=INFO
 #logging.config=classpath:log4j2-spring.xml
 spring.main.allow-bean-definition-overriding=true
 
+spring.main.allow-bean-definition-overriding=true
+
 dubbo.application.logger=slf4j
 dubbo.application.name=biz2
 dubbo.consumer.serialization=java