read_me.txt

#A Hello, this is a preview version.

#B We have submitted the preview of our model
1. Glo_MLM.py.  # The model in the paper.
2. globle_coocurrence.py # Script for compute counts.


#C Implementation pipeline.
1. Compute n-co-occurrence matrix. See #C
2. Retrive token co-occurrence of the input sequence from n-co-occurrence matrix. See generalization_of_masking/initialization.py/def glo_fn().
     e.g., we input [x1,x2,x3,x4,x4], and the n-co-occurrence matrix is X. We need the token co-occurrence of the input sequence:
     tc = [ [X11,X12,X13,X14,X15],
         [X21,X22,X23,X24,X25],
         [X31,X32,X33,X34,X35],
         [X41,X42,X43,X44,X45],
         [X51,X52,X53,X54,X55] ]
3. Howerver, we only take into account n-neighbors or n-gram, as discussed in Eq. 3. So, in Glo_MLM.py line 112-118.
    we computer n-gram masking to mask above matrix.
    e.g., we compute a 5-gram masking:
     masking = [ [0,1,1,0,0],
                 [1,0,1,1,0],
                 [1,1,0,1,1],
                 [0,1,1,0,1],
                 [0,0,1,1,1] ]
     Note that we do not count token-self co-occurrence.
     Then, for the globle coocurrence modeling, the label is:
        gc = tc * masking = [ [0,X12,X13,0,0],
                              [X11,0,X13,X14,0],
                              [X31,X32,0,X34,X35],
                              [0,X42,X43,0,X45],
                              [0,0,X53,X54,0] ]

4. We organize the input for the globle coocurrence modeling in Glo_MLM.py line 124.
    e.g., suppose the hide state of the token is h,
          We need to predict x1 and x3. So, we have MLM_output h= [h1,0,0,0,0,   and  output_O o= [0,h2,h3,0,0,
                                                                  0,0,0,0,0,                       0,0,0,0,0,
                                                                  0,0,h3,0,0,                      h1,h2,0,h4,h5,
                                                                  0,0,0,0,0,                       0,0,0,0,0,
                                                                  0,0,0,0,0, ]                      0,0,0,0,0,]

          Then, we do in Glo_MLM.py line 133:
          a.    output_matirx = h * oT =[ [0,h1h2,h1h3,0,0],
                                                 [0,0,0,0,0],
                                                 [h3h1,h3h2,0,h3h4,h3h5],
                                                 [0,0,0,0,0],
                                                 [0,0,0,0,0]]
         b. we minimize between output_matirx and gc considering non-zero values in output_matirx.




#