A simple asynchronous HTTP client for Scala

TGV is an experimental HTTP client for Scala that offers:

• configurable retrying in case of errors;
• strict throttling of requests, guaranteeing that the server receiving the requests will not see more than n concurrent requests per t seconds; and
• custom transformations.

TGV is based on Ning's Async HTTP Client (currently version 1.8-SNAPSHOT).

TGV is experimental code, a playground to explore non-blocking request/response IO in Scala. Currently, TGV reads the whole response from a HTTP request into memory before returning it. To make this more (resource-) efficient, there are plans to introduce a streaming API. For this, I am currently looking into Iteratees and similar techniques, like Pipes, Machines, or Conduits.

Even though TGV does come with unit tests, not all edge cases have been tested well enough yet. Use TGV in production with care.

Example, please!

Here is a short sample applications that downloads the body of some Wikipedia pages using throttling:

object SampleApp extends App {

  // Create the underlying HTTP client
  // (Note: currently, only Ning's AsyncHttpClient is supported.)
  val httpClient = new AsyncHttpClient(new Builder().
    setAllowPoolingConnection(true).
    setFollowRedirects(true).
    setCompressionEnabled(true).
    build)

  // Create threads to handle the request processing; all processing will be done
  // asynchronously, so the number of requests we can handle is not limited by the number
  // of threads configured here
  val numberOfThreads = Runtime.getRuntime().availableProcessors() * 2
  val executionService = new ExecutionContextService(numberOfThreads).start()
  implicit val executionContext = executionService.context()

  // Create a basic client (no retrying, no throttling, no transforms)
  val transport = AsyncHttpTransport(new StreamingAsyncHttpClient(httpClient))

  // Add throttling on top
  val throttled = transport.withThrottling(3 per (1 second))

  // Add retrying on top
  // Note: any requests on `retrying` will be throttled and, if needed, retried
  val retrying = throttled.withRetryStrategy(backoffStrategy = exponentialBackoffStrategy(maxRetries = 10))

  // Execute requests in parallel
  val words = List("Zurich", "Basel", "Paris", "London", "Washington", "Rome", "Palermo", "Nice", "Barcelona", "Vienna", "Berlin")
  val futures = words.map(w => retrying.body(retrying.getBuilder("http://en.wikipedia.org/wiki/" + w).build))

  // Wait until all responses have come in
  result(sequence(futures))(30 seconds)
  println("Done.")

  executionService.stop()
  httpClient.close()
  println("Shut down.")
}

Deploy to your own Maven repository

If you want sbt publish to deploy the library jar, its sources and API documentation to deploy to your in-house Maven repository, follow these steps:

1. Create, inside the project directory (and thus next to this README.md file) a file called local.sbt with content:

   publishTo := Some("My Repository" at "http://repo.my.com/some/path/")
   
   credentials += Credentials(Path.userHome / ".ivy2" / ".my-credentials")
   

   (Adapt the URL and file name .my-credentials.)

2. Create a file ~/.ivy2/.my-credentials and store the login and password to your repository in there. For a Sonatype Nexus repository, this should look something like this:

   realm=Sonatype Nexus Repository Manager
   host=repo.my.com
   user=myname
   password=mypassword
   

After this, you can do a sbt publish to deploy the library.