Skip to content

smol tips

Firing a million async requests with Python

There is no reason you'd ever need to fire a million requests to a single server so don't do this. like totally. don't. do. this.

import asyncio
import aiohttp

async def get_stuph(url, session, i):
    print(f"Firing request #{i}. brrr.")
    await session.get(url)

async def main():
    url = "http://some_url.nope"
    async with aiohttp.ClientSession() as session:
        _ = await asyncio.gather(
            *[
                get_stuph(url, session, i)
                for i in range(1_000_000)
            ]
        )

asyncio.run(main())

Check file size before downloading it with Python

If you're yolo-ing on the web and downloading a lot of content, especially arbitrary media files using a crawler, it might be useful to first check the mimetype & filesize before downloading it.

To do this with Python's requests module, you'll have to set stream=True and examine the headers for size & mime type. Following that, you can retrieve the content.

Set a retry strategy for Python requests

If you're overly enthuiastic with your requests to a server, it can get passive aggressive and give you the silent treatment or get overwhelmed and ask for a vacation. To mitigate that, give it some space with a retry strategy. Here's one such strategy with Python's requests package.