Add Nystromformer blog post #414
Conversation
There was a problem hiding this comment.
Thank you so much for writing this!! I left a few comments, but this is way out of my area of expertise so unfortunately I don't think I can provide any useful feedback on the actual content. I think @NielsRogge will be more helpful than me here 😄
nystromformer.md
Outdated
| @@ -0,0 +1,151 @@ | |||
| --- | |||
| title: "Nyströmformer: Approximating self-attention in linear time and memory via the Nyström method" | |||
| thumbnail: /blog/assets/86_nystromformer/paper_figure.png | |||
There was a problem hiding this comment.
Let us know if you can provide a thumbnail for the blog post (with a short version of the title), otherwise I think we can arrange to have one put together.
There was a problem hiding this comment.
I am not entirely sure what tools I can use to create a good thumbnail. If it isn't too much to ask, can you please provide one?
There was a problem hiding this comment.
Totally! Figma is a good choice, or you could use Canva instead.
Here's a ZIP file with a bunch of SVGs that you could choose from for a background, and a HF logo: https://drive.google.com/file/d/1d-uLYKWUV1K10MRmnxXfW6MAUrTd4qaT/view?usp=sharing
And the font we use is "Source Sans Pro", bold.
There was a problem hiding this comment.
Thanks for sharing this. I've pushed the thumbnail I made to this branch. Please let me know what you think.
|
|
||
| ### Can we approximate self-attention with the Nyström method? | ||
|
|
||
| Our goal is to ultimately approximate the softmax matrix in standard self attention: $$S = softmax(\frac{QK^T}{\sqrt{d}})$$ |
There was a problem hiding this comment.
This latex statement doesn't seem to render properly in the blog.
You can fix it by putting the statement on a new line. There are some other latex statements that are also broken, but I don't think that there's a way for you to preview the blog to fix it since our front-end code is closed-source. When everything else is all done with this post, I can take on fixing it before we merge!
Co-authored-by: Nima Boscarino <[email protected]>
|
Thanks a lot for your review @NimaBoscarino. I've applied the suggested changes. Looking forward to suggestions from other reviewers. |
Co-authored-by: NielsRogge <[email protected]>
Co-authored-by: NielsRogge <[email protected]>
Co-authored-by: NielsRogge <[email protected]>
Co-authored-by: NielsRogge <[email protected]>
_blog.yml
Outdated
| author: novice03 | ||
| thumbnail: /blog/assets/86_nystromformer/thumbnail.png | ||
| date: July 11, 2022 |
There was a problem hiding this comment.
To update before merge
|
Thank you so much for your review @osanseviero. I added the author card and I shall update the date right before this PR is ready to be merged. I also marked this PR as ready for review. |
|
Sounds good to me! Just to confirm - the date would be August 2nd, right? |
|
That's right! 🥳 |
|
Let's ship this 🔥 🚀 |
|
This is up now! https://huggingface.co/blog |
|
Thank you so much! There seem to be some latex issues, especially in the "How can we adapt the Nyström method to approximate self-attention?" section. How can we fix this? |
|
Looking into this :) |
|
Opened #449 |
|
great blogpost @novice03 thanks for contributing! |
Hello, this is a blog post on Nystromformer. Through this post, I wanted to:
Reviewers: @NimaBoscarino