upload files

[zhangtt2024]

Your work is excellent and very helpful to me. Can you upload the MOE and Average files in the ensembling method?

[yfzhang114]

For the average method, you can simply aggregate the results from different experts and compute the average.

Regarding the MOE (Mixture of Experts), the implementation is akin to gating. The main difference lies in the calculation of the gating weights. For gating, it's calculated as follows: $( h = \mathbf{W}_{\text{Concat}}([\tilde{y}_1, \tilde{y}_2]) + \mathbf{b} ) with ( w_1, w_2 = \text{softmax}(h) ).$ Whereas for MOE, it's computed as $( h = \mathbf{W} \mathbf{x} + \mathbf{b} ) with ( w_1, w_2 = \text{softmax}(h) ).$