Official Pytorch Version code of CIKM 2022 Paper "IntTower: the Next Generation of Two-Tower Model for Pre-ranking System"
Accept by CIKM 2022 and DLP-KDD 2022 workshop best paper
pip install -r requirements.txt
python train_movielens_IntTower.py 
The proposed model, IntTower (short for Interaction enhanced Two-Tower), consists of Light-SE, FE-Block and CIR modules. Specifically, lightweight Light-SE module is used to identify the importance of different features and obtain refined feature representations in each tower. FE-Block module performs fine-grained and early feature interactions to capture the interactive signals between user and item towers explicitly and CIR module leverages a contrastive interaction regularization to further enhance the interactions implicitly.
Paper: Xiangyang Li*, Bo Chen*, Huifeng Guo, Jingjie Li, Chenxu Zhu, Xiang Long, Yichao Wang, Wei Guo, Longxia Mao, Jinxing Liu, Zhenhua Dong, Ruiming Tang. IntTower: the Next Generation of Two-Tower Model for Pre-Ranking System
-
Light-SE Module
class LightSE(nn.Module): """LightSELayer used in IntTower. Input shape - A list of 3D tensor with shape: ``(batch_size,filed_size,embedding_size)``. Output shape - A list of 3D tensor with shape: ``(batch_size,filed_size,embedding_size)``. Arguments - **filed_size** : Positive integer, number of feature groups. """ def __init__(self, field_size, embedding_size=32): super(LightSE, self).__init__() self.softmax = nn.Softmax(dim=1) self.field_size = field_size self.embedding_size = embedding_size self.excitation = nn.Sequential( nn.Linear(self.field_size, self.field_size) ) def forward(self, inputs): if len(inputs.shape) != 3: raise ValueError( "Unexpected inputs dimensions %d, expect to be 3 dimensions" % (len(inputs.shape))) Z = torch.mean(inputs, dim=-1, out=None) A = self.excitation(Z) #(batch,reduction_size) A = self.softmax(A) #(batch,reduction_size) out = inputs * torch.unsqueeze(A, dim=2) return inputs + out
-
FE Score
def fe_score(user_rep, item_rep, user_fea_col, item_fea_col, user_embedding_dim, item_embedding_dim): score = [] for i in range(len(user_embedding_dim)): user_temp = torch.reshape(user_rep[i], (-1, user_fea_col, user_embedding_dim[i])) item_temp = torch.reshape(item_rep[-1], (-1, item_fea_col, item_embedding_dim[i])) score.append((user_temp @ item_temp.permute(0, 2, 1)).max(2).values.sum(1)) score = torch.stack(score).transpose(1, 0) return torch.sum(score, 1)
-
- Hardware（CPU/GPU）
- Prepare hardware environment with CPU or GPU processor.
- Framework
- Requirements
- deepctr==0.9.0
- deepctr_torch==0.2.7
- deepmatch==0.2.0
- keras==2.8.0
- matplotlib==3.5.2
- numpy==1.21.4
- pandas==1.4.2
- pytorch_lightning==1.6.3
- scikit_learn==1.1.1
- tensorflow==2.8.0
- torch==1.10.0
- torchkeras==3.0.2
- torchsummary==1.5.1
- torchvision==0.12.0
- tqdm==4.51.0
- xgboost==1.6.1
-
running on CPU or GPU
# run training and evaluation example pip install -r requirements.txt python train_movielens_IntTower.py
-
Compare with other model
We supply 8 models to compare inttower, you can change your model such as:
from model.dcn import DCN model = DCN(linear_feature_columns, dnn_feature_columns, task='binary', dnn_dropout=dropout, device=device)
-
Accelerate
For online inference , you can only use the last layer FE-Module to compute MaxSim Score.
Just change preprocessing/utils.py fe_score function
def fe_score(user_rep, item_rep, user_fea_col, item_fea_col, user_embedding_dim, item_embedding_dim): user_embedding, item_embedding = user_rep[0],item_rep[0] user_rep = torch.reshape(user_embedding, (-1, user_fea_col, user_embedding_dim[0])) item_rep = torch.reshape(item_embedding, (-1, item_fea_col, item_embedding_dim[0])) return (user_rep @ item_rep.permute(0, 2, 1)).max(2).values.sum(1)
@inproceedings{10.1145/3511808.3557072,
author = {Li, Xiangyang and Chen, Bo and Guo, Huifeng and Li, Jingjie and Zhu, Chenxu and Long, Xiang and Li, Sujian and Wang, Yichao and Guo, Wei and Mao, Longxia and Liu, Jinxing and Dong, Zhenhua and Tang, Ruiming},
title = {IntTower: The Next Generation of Two-Tower Model for Pre-Ranking System},
year = {2022},
isbn = {9781450392365},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3511808.3557072},
doi = {10.1145/3511808.3557072},
abstract = {Scoring a large number of candidates precisely in several milliseconds is vital for industrial pre-ranking systems. Existing pre-ranking systems primarily adopt the two-tower model since the "user-item decoupling architecture" paradigm is able to balance the efficiency and effectiveness. However, the cost of high efficiency is the neglect of the potential information interaction between user and item towers, hindering the prediction accuracy critically. In this paper, we show it is possible to design a two-tower model that emphasizes both information interactions and inference efficiency. The proposed model, IntTower (short for Interaction enhanced Two-Tower), consists of Light-SE, FE-Block and CIR modules. Specifically, lightweight Light-SE module is used to identify the importance of different features and obtain refined feature representations in each tower. FE-Block module performs fine-grained and early feature interactions to capture the interactive signals between user and item towers explicitly and CIR module leverages a contrastive interaction regularization to further enhance the interactions implicitly. Experimental results on three public datasets show that IntTower outperforms the SOTA pre-ranking models significantly and even achieves comparable performance in comparison with the ranking models. Moreover, we further verify the effectiveness of IntTower on a large-scale advertisement pre-ranking system. The code of IntTower is publicly available https://gitee.com/mindspore/models/tree/master/research/recommend/IntTower.},
booktitle = {Proceedings of the 31st ACM International Conference on Information & Knowledge Management},
pages = {3292–3301},
numpages = {10},
keywords = {pre-ranking system, recommender systems, neural networks},
location = {Atlanta, GA, USA},
series = {CIKM '22}
}