Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

模型结构问题请教 #20

Open
yhl41001 opened this issue Feb 12, 2025 · 1 comment
Open

模型结构问题请教 #20

yhl41001 opened this issue Feb 12, 2025 · 1 comment

Comments

@yhl41001
Copy link

yhl41001 commented Feb 12, 2025 •
edited
Loading

你好，有个模型结构的问题请假一下：
从论文的研究看，视觉特征对不同的LLM层重要性不同，那模型的结构应该是在不同的LLM层对视觉特征的保留数量递减，但论文的设计是在视觉特征送入LLM前对视觉特征和文本特征做了相关性压缩，而送入LLM的特征在每层的重要性仍然是一样的，所以模型的设计和论文前期对视觉特征在不同LLM层的重要性不同这个结论并无太大相关性，是这样吗？
@MiloQ
Copy link

MiloQ commented Feb 13, 2025

个人理解，相当于把prefusion模块当成 LLM的前几层，LLM本身当成LLM的后面的层，所以 LLM在后面本身接收到的信息少了也没事

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@yhl41001 @MiloQ