RFC for embedded static resource support #350

wuzzeb · 2016-01-24T00:04:11Z

Servant already uses wai-app-static for the very simple serveDirectory which just produces a Server Raw. This isn't ideal because the routes for individual files are not represented in the type. Also, wai-app-static has the more sophisticated ability to embed resources into the executable at compile time (full disclosure, I was the original author of this code in wai-app-static although since then several people have contributed to wai-app-static).

This pull request is my RFC for an approach to extend serveDirectory to be able to specify individual resources in the API type for better exposure of the static resources that are available. Along the way, I also add support for embedding the static content at compile time.

type MyAPI = "static" :> "js" :> "bootstrap.js" :> EmbeddedContent "application/javascript"
        :<|> "static" :> "css" :> "bootstrap.css" :> EmbeddedContent "text/css"
        :<|> "static" :> "css" :> "mysite.css" :> EmbeddedContent "text/css"

At the moment I added all this into a new project, but that was just for ease of testing. If you like this approach, some of the types should be moved into Servant.API, other code merged into servant-server, and perhaps the more exotic generators such as compiling with lesscss and ghcjs moved into a new project. I can morph this code around once I get some feedback.

I went through about 4 iterations on the design before I found one I was happy with. My first iteration used wai-app-static directly but it had to do A LOT with template haskell and as always using as little template haskell as possible is good. I then realized that we only need a few internal functions from wai-app-static, so the later evolutions of the design don't depend on wai-app-static anymore. I then went through several iterations on how the combinators look. I have started using it in my own servant-based application and the combinators are working well, but I'm still open to improvements.

I wrote haddock documentation, so for more details you should see the documentation, especially the haddock comment in Servant.Server.Embedded.hs

This code allows to embed static content such as javascript and CSS into the executable at compile time so that it does not need to be distributed along with the server. In addition, this module supports processing of these resources before they are embedded, such as javascript or CSS minification. Finally, there is a development mode which will recompute each resource on every request, so allow a simple browser refresh to reload potentially changed javascript or CSS. Documentation is in the haddock comment in Servant.Server.Embedded.hs

3noch · 2016-01-24T01:16:12Z

👍 Really cool feature! Having totally self-contained web-apps is a very rare thing indeed!

jkarni · 2016-01-24T11:11:06Z

Wow, great work!

An issue with having it as a separate package is that, if it's to have instances for all the core classes, it'll need to depend on all the core packages, making e.g. wai a transitive dependency of client code that uses it.

One thing we've been talking about is starting a servant-contrib set of packages. @haskell-servant/maintainers would we want it there or in the core packages? I personally think this will be useful enough to have in the core ones. On the other hand, it's perhaps good practice to start the -contrib packages and have new combinators go there, and only then move them, so we have (a) some time to be more aggressive about release cycles and breaking API changes as we try things out, and (b) give people who are writing other interpreter classes (e.g. servant-swagger) some time to know what's in the pipeline.

utdemir · 2016-01-24T12:30:22Z

+1 for me for self-contained web apps, and compile-time file embedding would certainly be very useful in servant-server. But I'm not sure if it belongs to the API definition, because it looks like an implementation detail of server (I don't think clients or documentation generators care about if it's generated on-the-fly or on compile time.).

But, I think the part minifiers, compressors, CSS transpilers etc. belong to a different package, as you said.

And maybe it's worth to talk about the general idea of "serving static files" here. For example, on our API's, we were serving the JS client of our server on some endpoint. But in order to set Content-Type header for them, we had to define the type as Raw:

myApi :: "client.js" :> Raw
serve :: Application
serve req resp = resp $ responseLBS status200 [("Content-type", "text/javascript")]
   (BL.fromStrict . T.encodeUtf8 $ jsForApi myApi)

In order to simplify this, I wrote these instances (simplified):

data ContentType (typeName :: Symbol) (subtypeName :: Symbol)

instance Accept (ContentType typeName subtypeName) 
instance MimeRender (ContentType t s) ByteString 
instance MimeRender (ContentType t s) Text 
...

And then I was able to define API's like this

myApi :: "client.js" :> Get '[ContentType "application" "javascript"] Text
serve :: EitherT ServantError IO Text
serve = return $ jsForAPI myApi

I told you about this, because I think the TH file embedding also fits here as a separate type, something like:

instance MimeRender (ContentType t s) EmbeddedContent where
myApi :: "somejs" :> Get '[ContentType "application" "javascript"] EmbeddedContent
serve = embedFile "./somejs.js"

Maybe this RFC can cover serving static content (but not from an external file) with a specific content-type too.

soenkehahn · 2016-01-24T14:52:10Z

@wuzzeb: I very much like the idea of embedding static files into the executable. I've been using that feature from wai-app-static and it's awesome.

I do wonder however: what is the advantage of having all the static files represented in the type of the API? What serveDirectory was aimed at is use-cases that follow this pattern:

type MyJsonRestAPI =
  ... -- lots of JSON REST routes

type MyServerAPI =
  MyJsonRestApi :<|>
  Raw -- to serve some html, css and javascript that uses the API

I don't see how this would benefit from having all the static files explicitly represented in the type? I.e. I probably don't need to be able to generate type safe client functions to fetch all the static assets individually. Also I don't want all the css files to show up in generated documentation, be it by servant-docs or servant-swagger. Am I overlooking an important use-case?

These static file combinators would also force you to manually keep them in sync with the files on your disk. What I like very much about serveDirectory is that you can just throw a new file in the directory and it'll be automatically served.

Generally I'm wondering if it wouldn't be good to solve these things purely in wai. serveDirectory does not depend on anything in servant at all. (The implementation is just based on wai-app-static and although the type is given as FilePath -> Server Raw that's just fancy talk for FilePath -> Application.) Maybe it should be moved to wai-app-static (and re-exported in servant?). And maybe we do want something like serveDirectoryEmbedded. That way not only servant users, but all wai users would benefit.

soenkehahn · 2016-01-24T15:07:53Z

What I could imagine is a combinator like this:

type MyApi =
  MyJsonRestApi :<|>
  "static" :> StaticAssets

instance HasServer StaticAssets where
  type Server StaticAssets = StaticAssetsConfig
  route = ...

data StaticAssetsConfig
  = FromDisk FilePath
  | Embedded Embedded -- from wai-app-static

If I think about the different interpretations, that feels like the right amount of information that should be in the API specification. So generated documentation can say something like:

/static (and subpaths) - static assets like html, css, javascript, images, etc.

And we could have:

instance HasClient StaticAssets where
  type Client StaticAssets = FilePath -> ExcepT ServantError IO ByteString

So that you can easily generate a client function that allows to fetch assets like this: getStaticAssets "index.html".

jkarni · 2016-01-24T15:11:10Z

One obvious advantage is type-safe links to static assets.

Maybe the solution is explicit routes, but with TH to generate them from a directory?

soenkehahn · 2016-01-24T15:46:20Z

But where would you want to have type-safe links to static assets? I guess if you're generating html files within haskell... Personally I would prefer to have this functionality in a templating library or an html generation library and de-coupled from servant. Maybe that also already exists.

codedmart · 2016-01-24T15:50:03Z

I tend to agree with @soenkehahn on this. Awesome work from @wuzzeb, but I don't see the advantages of this yet to be in core. Seems more like a specified use case that would be great as a contrib lib that someone can pull in when needed.

wuzzeb · 2016-01-29T00:47:04Z

Sorry, was busy the past few days.

My first approach was much closer to serveDirectory. I didn't call it serveDirectoryEmbedded but that was essentially what it was. I still have the code around in my git history (not on github, for github I did a rebase) so we could go to it if you like. The main reason I changed to include it all in the route is ETags and the HasLink instance.

For static resources, the recommendation is to tell the browser to cache them and then change the URL used to access. For example, see here for example. This means that when the server refers to a static resource, it must change the URL each time the content changes. wai-app-static does that via including the ETag as a query-param. What I mean is the server must generate a link such as

<link rel="stylesheet" href="/static/bootstrap.css?etag=12345678"/>

and the etag must change each time the content changes. So in order for the server to generate such links, it needs to know the computed etag. With my first approach which used serveDirectoryEmbedded, the template haskell had to create a variable containing a string with the etag, but then that string had to be combined with the full path. Bringing each resource out into the actual route allows HasLink instances which easily allow the server to generate the correct link with the correct etag.

I agree, if you didn't care about etags and proper caching, a solution which hid all the resources behind a single type in the servant type would be the best approach.

jkarni · 2016-02-09T00:48:48Z

Changing the url on file changes for better caching would be amazing - I've been impressed by that feature of yesod/wai-app-static, and would be pretty happy to see it in servant too.

soenkehahn · 2016-08-11T14:48:48Z

@wuzzeb: Btw, I would love to see something like serveDirectoryEmbedded as a contribution to wai-app-static.

soenkehahn · 2016-08-15T15:54:58Z

It seems to me that perfect caching (with etag query params, like @wuzzeb describes) is ideally done in a separate library on top of wai. I think the fact that HasLink does not provide perfect caching out of the box is not worth the complexity at the type-level.

I'm in favor of not merging (and closing) this PR.

If however HasLink doesn't play nice with an external library that supports perfect caching, then we should fix that, of course.

rashadg1030 · 2021-09-16T23:57:44Z

Will this ever be merged into master? Seems like something I could use on a project I'm working on.

tchoutri

Very impressive PR, very fan of the work you've been doing.

I asked for some changes, and one optional quality-of-life suggestion at the end. If you're still good to work on the Pull Request, we can have that for the next major release of Servant, which is being planned.

tchoutri · 2021-11-17T14:17:57Z

servant-static/Setup.hs

+import Distribution.Simple
+main = defaultMain


Unneeded, this is the default

tchoutri · 2021-11-17T14:18:49Z

servant-static/src/Servant/Server/Embedded.hs

+-- After creating the 'EmbeddedEntry', 'embed' will create a haskell variable to hold the
+-- 'EmbeddedEntry'.  The name of the haskell variable is the 'EntryVarName' passed to the function
+-- which creates the generator.
+embed :: Bool -- ^ development mode?


Could we avoid Boolean blindness on the first parameter? :)

tchoutri · 2021-11-17T14:20:20Z

servant-static/src/Servant/Server/Embedded/CSS.hs

@@ -0,0 +1,76 @@
+-- | This module contains 'Generators' for processing and embedding CSS.


Very interesting idea, would love automated tests to make that we keep up with the nodejs ecosystem.

tchoutri · 2021-11-17T14:21:50Z

servant-static/src/Servant/Server/Embedded/Files.hs

+-- | Use <https://github.com/mishoo/UglifyJS2 UglifyJS2> to compress javascript.
+-- Assumes @node_modules\/uglifyjs\/bin\/uglifyjs@ exists so just run @npm install uglifyjs@.  It
+-- uses options @[\"-m\", \"-c\"]@ to both mangle and compress.
+uglifyJs :: BL.ByteString -> IO BL.ByteString
+uglifyJs = compressTool "./node_modules/uglifyjs/bin/uglifyjs" ["-m", "-c"]
+
+-- | Use <http://yui.github.io/yuicompressor/ YUI Compressor> to compress javascript.
+-- Assumes a script @yuicompressor@ is located in the path.  If not, you can still
+-- use something like
+--
+-- > compressTool "java" ["-jar", "/path/to/yuicompressor.jar", "--type", "js"]
+yuiJavascript :: BL.ByteString -> IO BL.ByteString
+yuiJavascript = compressTool "yuicompressor" ["--type", "js"]
+
+-- | Use <http://yui.github.io/yuicompressor/ YUI Compressor> to compress CSS.
+-- Assumes a script @yuicompressor@ is located in the path.
+yuiCSS :: BL.ByteString -> IO BL.ByteString
+yuiCSS = compressTool "yuicompressor" ["--type", "css"]
+
+-- | Use <https://developers.google.com/closure/compiler/ Closure> to compress
+-- javascript using the default options.  Assumes a script @closure@ is located in
+-- the path. If not, you can still run using
+--
+-- > compressTool "java" ["-jar", "/path/to/compiler.jar"]
+closureJs :: BL.ByteString -> IO BL.ByteString
+closureJs = compressTool "closure" []


Again, good ideas, any chance we could have some kind of pre-flight checks to make sure that if the required JS tools are unavailable, either we crash the app, or do nothing (with the adequate warning)

tchoutri · 2021-11-17T14:22:34Z

servant-static/src/Servant/Server/Embedded/Ghcjs.hs

@@ -0,0 +1,47 @@
+module Servant.Server.Embedded.Ghcjs (


I would advise you to remove this part for now. As things stand today, GHCJS cannot be a tier-1 target for the Servant ecosystem.

tchoutri · 2021-11-17T14:23:49Z

servant-static/src/Servant/Server/Embedded/TH.hs

+-- | For each entry, the template haskell code will produce a variable of type @'EmbeddedEntry'
+-- mime@.  The variable name is specified by a value of type 'EntryVarName', so the string must
+-- be a valid haskell identifier (start with a lower case letter, no spaces, etc.).
+type EntryVarName = String


Would the entirety of Shakespear's work in Mandarin be a valid EntryVarName? If not, I would advise that you use a more appropriate newtype. If you need a good and lightweight wrapper for ASCII text, I can advise that you use this library: hackage.haskell.org/package/text-ascii

tchoutri · 2021-11-17T14:26:16Z

servant-static/src/Servant/Server/Embedded/Types.hs

+
+-- | An etag is used to return 304 not modified responses and cache control headers.
+-- If the content changes, the etag must change as well.
+newtype Etag = Etag B.ByteString


Could you please deriving instances for the usual suspects Eq, Ord, Show (when applicable ofc).

tchoutri · 2021-11-17T14:26:31Z

servant-static/src/Servant/Server/Embedded/Types.hs

+data EmbeddedEntry (mime :: Symbol) = EmbeddedEntry {
+    eeEtag :: Maybe Etag
+  , eeApp :: Application
+}


Could you please deriving instances for the usual suspects Eq, Ord, Show (when applicable ofc).

tchoutri · 2021-11-17T14:26:35Z

servant-static/src/Servant/Server/Embedded/Types.hs

+import qualified Data.Text.Encoding as T
+
+-- | Endpoint for embedded content.
+data EmbeddedContent (mime :: Symbol) = EmbeddedContent


Could you please deriving instances for the usual suspects Eq, Ord, Show (when applicable ofc).

tchoutri · 2021-11-17T14:27:37Z

servant-static/src/Servant/Server/Embedded/Types.hs

+instance HasServer (EmbeddedContent mime) config where
+    type ServerT (EmbeddedContent mime) m = EmbeddedEntry mime
+    route Proxy _ entry = LeafRouter $ \request respond -> do
+        r <- runDelayed entry
+        case r of
+            Route e -> (eeApp e) request (respond . Route)
+            Fail a -> respond $ Fail a
+            FailFatal e -> respond $ FailFatal e
+
+instance HasLink (EmbeddedContent mime) where
+    type MkLink (EmbeddedContent mime) = Maybe Etag -> URI
+    toLink Proxy lnk metag = uri { uriQuery = q }
+        where
+            uri = linkURI lnk
+            q = case (uriQuery uri, metag) of
+                    ("", Just (Etag etag)) -> "?etag=" ++ T.unpack (T.decodeUtf8 etag)
+                    (query, Just (Etag etag)) -> query ++ "&etag=" ++ T.unpack (T.decodeUtf8 etag)
+                    (query, _) -> query


That's optional, but do you see potential forbidden instances that we could hijack with a TypeError type family in order to provide better development experience?

jhrcek

Just few suggestions from reading the docs..

jhrcek · 2021-11-17T15:16:45Z

servant-static/src/Servant/Server/Embedded.hs

+-- >        :<|> "static" :> "css" :> "mysite.css" :> EmbeddedContent "text/css"
+--
+-- Then, decide on a generator for each 'EmbeddedContent'.  There are several generators which embed
+-- files directly, minifiy files, and use 3rd party tools like less and postcss.  You can also


Suggested change

-- files directly, minifiy files, and use 3rd party tools like less and postcss. You can also

-- files directly, minify files, and use 3rd party tools like less and postcss. You can also

jhrcek · 2021-11-17T15:17:18Z

servant-static/src/Servant/Server/Embedded.hs

+-- so that it does not need to be distributed along with the server.  In addition, this module
+-- supports processing of these resources before they are embedded, such as javascript or CSS
+-- minification.  Finally, there is a development mode which will recompute each resource on every
+-- request, so allow a simple browser refresh to reload potentially changed javascript or CSS.


Suggested change

-- request, so allow a simple browser refresh to reload potentially changed javascript or CSS.

-- request, to allow a simple browser refresh to reload potentially changed javascript or CSS.

jhrcek · 2021-11-17T15:19:47Z

servant-static/src/Servant/Server/Embedded.hs

+-- disk or postcss will be re-executed on each request.  Thus when the DEVELOPMENT flag is true, a
+-- browser refresh will reload and recompute the resources from disk.
+--
+-- When the DEVELOPMENT define is false, instead at compile time the resource will be loaded, the


Suggested change

-- When the DEVELOPMENT define is false, instead at compile time the resource will be loaded, the

-- When the DEVELOPMENT define is false, instead the resource will be loaded at compile time, the

3noch mentioned this pull request Apr 21, 2016

Less specific 404 takes precendence over more specific 405 when serveDirectory is used #408

Closed

jkarni mentioned this pull request May 11, 2016

Don't use Raw for serving static directories. #507

Open

tchoutri added the 🚧 needs rebase 🚧 label Nov 17, 2021

tchoutri requested changes Nov 17, 2021

View reviewed changes

jhrcek reviewed Nov 17, 2021

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RFC for embedded static resource support #350

RFC for embedded static resource support #350

wuzzeb commented Jan 24, 2016

3noch commented Jan 24, 2016

jkarni commented Jan 24, 2016

utdemir commented Jan 24, 2016

soenkehahn commented Jan 24, 2016

soenkehahn commented Jan 24, 2016

jkarni commented Jan 24, 2016

soenkehahn commented Jan 24, 2016

codedmart commented Jan 24, 2016

wuzzeb commented Jan 29, 2016

jkarni commented Feb 9, 2016

soenkehahn commented Aug 11, 2016

soenkehahn commented Aug 15, 2016

rashadg1030 commented Sep 16, 2021

tchoutri left a comment

tchoutri Nov 17, 2021

tchoutri Nov 17, 2021

tchoutri Nov 17, 2021

tchoutri Nov 17, 2021

tchoutri Nov 17, 2021

tchoutri Nov 17, 2021

tchoutri Nov 17, 2021

tchoutri Nov 17, 2021

tchoutri Nov 17, 2021

tchoutri Nov 17, 2021

jhrcek left a comment

jhrcek Nov 17, 2021

jhrcek Nov 17, 2021

jhrcek Nov 17, 2021

		@@ -0,0 +1,76 @@
		-- \| This module contains 'Generators' for processing and embedding CSS.

	-- files directly, minifiy files, and use 3rd party tools like less and postcss. You can also
	-- files directly, minify files, and use 3rd party tools like less and postcss. You can also

	-- request, so allow a simple browser refresh to reload potentially changed javascript or CSS.
	-- request, to allow a simple browser refresh to reload potentially changed javascript or CSS.

	-- When the DEVELOPMENT define is false, instead at compile time the resource will be loaded, the
	-- When the DEVELOPMENT define is false, instead the resource will be loaded at compile time, the

RFC for embedded static resource support #350

Are you sure you want to change the base?

RFC for embedded static resource support #350

Conversation

wuzzeb commented Jan 24, 2016

3noch commented Jan 24, 2016

jkarni commented Jan 24, 2016

utdemir commented Jan 24, 2016

soenkehahn commented Jan 24, 2016

soenkehahn commented Jan 24, 2016

jkarni commented Jan 24, 2016

soenkehahn commented Jan 24, 2016

codedmart commented Jan 24, 2016

wuzzeb commented Jan 29, 2016

jkarni commented Feb 9, 2016

soenkehahn commented Aug 11, 2016

soenkehahn commented Aug 15, 2016

rashadg1030 commented Sep 16, 2021

tchoutri left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jhrcek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment