Super Embed MVP

[matthew-carroll]

Super Embed is a package that knows how to scrape metadata from popular web services and then display a card, or other visual, that represents the given media. For example, a YouTube link could turn into a video player for the associated video. A Medium link could turn into a card that shows the title and header image from the Medium post.

This milestone is to create the initial minimum viable product for Super Embed.

What sets Super Embed apart

The pub.dev ecosystem includes a few link preview packages, e.g., any_link_preview and link_previewer. Super Embed will be different in a few significant ways.

Drop-in widgets: Super Embed will collect a growing number of fully spec'd, designed, and implemented drop-in widgets for various services, like YouTube and Twitter. The other pub.dev packages focus on a high level of configuration for a single link preview widget. Super Embed takes the opposite approach, providing widgets that are specialized for the type of content that they represent.

Responsive by default: Super Embed widgets will adjust their display for varying sizes, as appropriate. Some widgets, like a video player, may not alter their appearance in response to available space, but something like a social media card might change its layout from a row to a column as the available width decreases.

A toolkit: Super Embed will make all of its non-visual tools available in the API surface, too. Page scrapers for each service will be publicly available so that developers can utilize the same scraping behavior, even if they don't want the associated widget. Similarly, the widgets for each service will be usable even without executing a web scrape operation. This way, developers can cache scrape results locally, or even obtain scraped data from their own backend system and still utilize the polished Super Embed widgets.

Scope

Initially supported web services:

• YouTube (Superlist, Clearful)
• Loom (Superlist)
• Twitter (Clearful)
• Generic social data (SuperDeclarative)

Initially supported widget formats:

• Video player (Superlist, Clearful)
• Twitter card (Clearful)
• Generic social card (SuperDeclarative)

Funders:

• Superlist
• Clearful
• SuperDeclarative

[bk-one]

I see two ways forward. Let me know which one you prefer:

1. We start with a very generic support of whatever link we get and convert it using something like https://pub.dev/packages/metadata_fetch to just display the metadata of the link in a nice way. We could then extract the domain of all of these links and gather data on what kind of domains are being used and then start working on them one by one.
2. We start with the most commonly used ones (e.g. YouTube, Google Maps, Twitter) and establish a way to return specific widgets to embed right away.

Both work for us. I will start to add some basic UI flow to also have an idea how the transformation from Link to widget could look like (and how the user can revert that). I would suggest to have a example (or default) implementation from link to widget, that can be customized of course.

[matthew-carroll]

I like the idea of starting with a handful of specific URLs, designing their desired UI, and then implementing them. We'll probably learn a few things along the way.

[bk-one]

Okay, then I would like to start with video links, let's say YouTube and Loom specifically.

[matthew-carroll]

@Jethro87 - If you'd like to join this milestone, feel free to list any additional web services that you need to scrape, and card visuals that you need.

[Jethro87]

@matthew-carroll Other than YouTube, the service we'd like to start with is Twitter. This would be a text-based card visual with an optional image.

[matthew-carroll]

Sounds good. I'll add Twitter.

[matthew-carroll]

Based on a recommendation, I checked out a service called Iframely. Iframely fetches data from any given URL and parses relevant data based on supported domains. The parsed data can then be retrieved as embeddable HTML, or structured data.

@bk-one @Jethro87 @miguelcmedeiros - the question is whether the existence of this service should change the direction of this project?

I would say that what we're attempting with this project is the recreation of Iframely, specifically for Flutter, and with a shared/open definition for all parsers. Iframely, on the other hand, specifically targets the web stack for UI, and the structured data parsing happens on their servers, resulting in service subscriptions and fees.

I don't think that this project, or any of our clients can make use of Iframely's UI tools, because those tools would only work on web.

This project, or our clients, might make use of Iframely's massive domain parsing library. However, that use would require an Iframely account, and subscription fees. We could, in theory, provide some kind of hook from Super Embed's rendering system to the Iframely backend, but we can't automatically provide access to the Iframely API because that API requires an account and fees.

What are your thoughts about Iframely?

[bk-one]

I think it's valuable to either return structured data or an embeddable widget as options from our approach. But other than that, I don't see a reason why your research would change our approach. I would like to keep this open-source and free of charge.

So I would suggest to keep it separately. If there's a hook that someone wants to add to this project, we can encourage people to put up a PR. Superlist would not want to have that hook as of now.

[matthew-carroll]

@miguelcmedeiros @Jethro87 - I drafted an architecture document to guide the work breakdown for the MVP. Please take a look at the document and comment with any questions or concerns.

Architecture Proposal

Once you've reviewed the architecture, I'll breakdown tickets for other Bounty Hunters to work on.