2.2 Example: Asynchronous Web Requests

You can't increase the performance by just adding async and await keywords to your code. In this example, we will see a bad and a good usage of coroutines.

The problem is fetching data from a url. The url https://www.canbula.com/prime/{n} returns a dictionary including the prime numbers below an integer n. Let's start with the required modules.

import asyncio
import aiohttp
import time

The second step is writing the coroutine, which fetches the data from the remote url. We get the session as an input argument because this coroutine will use an existing session instead of creating a new one for each await.

async def get_primes_below(number: int, session: aiohttp.ClientSession) -> list:
    print(f"Getting primes below {number}")
    headers = {
        "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 12_6) "
        "AppleWebKit/537.36 (KHTML, like Gecko) "
        "Chrome/106.0.0.0 Safari/537.36"
    }
    url = f"https://www.canbula.com/prime/{number}"
    async with session.get(url, headers=headers, ssl=None) as response:
        primes = (await response.json())["primes"]
    print(f"Got primes below {number}: {primes}")
    return primes

Most of the developers just think that awaiting a coroutine does not stop the main thread, but it actually stops if you don't do it correctly. Let's create a wrong scheduling:

async def wrong_scheduling():
    print(f"Starting wrong async calls")
    start = time.time()
    session = aiohttp.ClientSession()
    objects = [get_primes_below(n, session) for n in [50, 10, 30, 40, 20]]
    for object in objects:
        await object
    end = time.time()
    await session.close()
    print(f"Ending wrong async calls")
    print(f"Time taken: {end - start:.2f} seconds")

We intentionally use the list of integers in a shuffled order as [50, 10, 30, 40, 20] to get the responses in a disordered fashion. Let's see the output:

Starting wrong async calls
Getting primes below 50
Got primes below 50: [2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47]
Getting primes below 10
Got primes below 10: [2, 3, 5, 7]
Getting primes below 30
Got primes below 30: [2, 3, 5, 7, 11, 13, 17, 19, 23, 29]
Getting primes below 40
Got primes below 40: [2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37]
Getting primes below 20
Got primes below 20: [2, 3, 5, 7, 11, 13, 17, 19]
Ending wrong async calls
Time taken: 49.34 seconds

That set of requests took almost 50 seconds. But if you have noticed that the requests are sent one by one, so nothing has changed here whether we use a coroutine instead of traditional functions. They just run like the old fashion sequential functions. Let's see another scheduling:

async def correct_scheduling():
    print(f"Starting correct async calls")
    start = time.time()
    session = aiohttp.ClientSession()
    objects = [get_primes_below(n, session) for n in [50, 10, 30, 40, 20]]
    await asyncio.gather(*objects)
    await session.close()
    end = time.time()
    print(f"Ending correct async calls")
    print(f"Time taken: {end - start:.2f} seconds")

We have just changed the await object part with asyncio.gather. Let's see the output:

Starting correct async calls
Getting primes below 50
Getting primes below 10
Getting primes below 30
Getting primes below 40
Getting primes below 20
Got primes below 10: [2, 3, 5, 7]
Got primes below 20: [2, 3, 5, 7, 11, 13, 17, 19]
Got primes below 30: [2, 3, 5, 7, 11, 13, 17, 19, 23, 29]
Got primes below 40: [2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37]
Got primes below 50: [2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47]
Ending correct async calls
Time taken: 15.08 seconds

Now it took only 15 seconds. So turning your functions into coroutines and call them by using await keyword is not enough to get better performance.

You can apply this knowledge to some other problems such as reading files, fetching data from database or API requests, which will be your everyday job if you will do something as a backend developer.