> Plain-text rendering of a visual post, for machine consumers. > Charts, diagrams and images are omitted; their captions are kept inline. > Titles on collapsible asides are dropped, their contents kept. > The complete version is at https://abhay.fyi/blog/concurrent-downloads-with-python-using-asyncio-or-thread-pools/ # Concurrent downloads with Python using asyncio or thread pools > This shows how to download multiple files in parallel with Python using asyncio or thread pools. - Author: Abhay Kashyap (https://abhay.fyi/#person) - Published: 2022-10-06 - Canonical: https://abhay.fyi/blog/concurrent-downloads-with-python-using-asyncio-or-thread-pools/ --- When downloading a large number of files with Python, you are I/O bound. A vanilla implementation with `requests` like the one below would yield sequential, blocking calls with files downloaded one at a time. ```python import time import requests start = time.perf_counter() urls = [ "https://i.imgur.com/AD3MbBi.jpeg", "https://i.imgur.com/zYhkOrM.jpeg", "https://i.imgur.com/LRoLTlK.jpeg", "https://i.imgur.com/gtWsPu9.jpeg", "https://i.imgur.com/jDimNTZ.jpeg", ] for url in urls: print(f"Downloading {url}") resp = requests.get(url) with open(url.split("/")[-1], 'wb') as f: f.write(resp.content) print(f"Done downloading {url}") print(f"Total time: {time.perf_counter() - start}") ``` In the snippet above, when `requests.get(url, stream=True)` is called, the CPU enters an idle state and waits for a response from the server. Once response is received, it proceeds to save the file & process the next url as evident from the output below. ```bash Downloading https://i.imgur.com/AD3MbBi.jpeg Done downloading https://i.imgur.com/AD3MbBi.jpeg Downloading https://i.imgur.com/zYhkOrM.jpeg Done downloading https://i.imgur.com/zYhkOrM.jpeg Downloading https://i.imgur.com/LRoLTlK.jpeg Done downloading https://i.imgur.com/LRoLTlK.jpeg Downloading https://i.imgur.com/gtWsPu9.jpeg Done downloading https://i.imgur.com/gtWsPu9.jpeg Downloading https://i.imgur.com/jDimNTZ.jpeg Done downloading https://i.imgur.com/jDimNTZ.jpeg Total time: 1.2269745559970033 ``` The core idea to speed this up is to fire requests without waiting for a response, essentially pushing data throughput closer to server or channel capacity. In Python, this can be done in a few ways and I'll cover two popular ones - process/thread pools and asyncio. Also, while the examples here reference downloads, it can be applied to any I/O bound task. --- ## 1. Process/Thread pools One familiar approach here would be to create multiple processes/threads and fire requests in parallel. While you can do this with [`multiprocessing`](https://docs.python.org/3/library/multiprocessing.html) or [`threading`](https://docs.python.org/3/library/threading.html) modules, you should probably use [`concurrent.futures`](https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.ThreadPoolExecutor) module instead since it provides a nicer interface. Here's a simple example using `ThreadPoolExecutor`: ```python import time from concurrent.futures import ThreadPoolExecutor import requests start = time.perf_counter() urls = [ "https://i.imgur.com/AD3MbBi.jpeg", "https://i.imgur.com/zYhkOrM.jpeg", "https://i.imgur.com/LRoLTlK.jpeg", "https://i.imgur.com/gtWsPu9.jpeg", "https://i.imgur.com/jDimNTZ.jpeg", ] def download_image(url): print(f"Downloading {url}") resp = requests.get(url) with open(url.split("/")[-1], 'wb') as f: f.write(resp.content) print(f"Done downloading {url}") with ThreadPoolExecutor(max_workers=5) as executor: executor.map(download_image, urls) print(f"Total time: {time.perf_counter() - start}") ``` Here, a maximum of 5 threads are created and each thread is assigned a url to download. You can see that the requests are fired in parallel and the total time taken is much less than the sequential version. Process pools work in a similar way, only they are heavier but offer more isolation with no GIL constraints. ```bash Downloading https://i.imgur.com/AD3MbBi.jpeg Downloading https://i.imgur.com/zYhkOrM.jpeg Downloading https://i.imgur.com/LRoLTlK.jpeg Downloading https://i.imgur.com/gtWsPu9.jpeg Downloading https://i.imgur.com/jDimNTZ.jpeg Done downloading https://i.imgur.com/AD3MbBi.jpeg Done downloading https://i.imgur.com/zYhkOrM.jpeg Done downloading https://i.imgur.com/gtWsPu9.jpeg Done downloading https://i.imgur.com/jDimNTZ.jpeg Done downloading https://i.imgur.com/LRoLTlK.jpeg Total time: 0.4605532810019213 ``` When to use it: - You don't want to deal with async code and prefer a more familiar interface. - If you want to process files after downloading them and have multiple core (using a process pool would be faster than only asyncio) Gotchas: - You might be limited by the number of processes/threads you can create on your system. - If you're using some sort of shared state, this would require some manual synchronization. - If you're using a large number of processes/threads, you might run into connection limits & memory issues. --- ## 2. Asyncio Another way to achieve this is to use [`asyncio`](https://docs.python.org/3/library/asyncio.html), a library to write concurrent code that is native to Python (3.4+). This will let you side-step threads if you don't want to deal with them. Note that you can use `asyncio` within processes/threads as well which might be useful in some cases. Also, Python's `requests` library is not async by default, so you'll need to use [`aiohttp`](https://docs.aiohttp.org/en/stable/) instead if you want pure async or [`httpx`](https://www.python-httpx.org/) if you want a sync/async hybrid. Here's a simple example using `asyncio`: ```python import time import asyncio import aiohttp start = time.perf_counter() urls = [ "https://i.imgur.com/AD3MbBi.jpeg", "https://i.imgur.com/zYhkOrM.jpeg", "https://i.imgur.com/LRoLTlK.jpeg", "https://i.imgur.com/gtWsPu9.jpeg", "https://i.imgur.com/jDimNTZ.jpeg", ] async def download_image(url): print(f"Downloading {url}") async with aiohttp.ClientSession() as session: async with session.get(url) as resp: with open(url.split("/")[-1], 'wb') as f: f.write(await resp.read()) print(f"Done downloading {url}") async def main(): await asyncio.gather(*[download_image(url) for url in urls]) asyncio.run(main()) print(f"Total time: {time.perf_counter() - start}") ``` The above snippet follows a similar behavioral pattern as the thread pool example. ```bash Downloading https://i.imgur.com/AD3MbBi.jpeg Downloading https://i.imgur.com/zYhkOrM.jpeg Downloading https://i.imgur.com/LRoLTlK.jpeg Downloading https://i.imgur.com/gtWsPu9.jpeg Downloading https://i.imgur.com/jDimNTZ.jpeg Done downloading https://i.imgur.com/AD3MbBi.jpeg Done downloading https://i.imgur.com/jDimNTZ.jpeg Done downloading https://i.imgur.com/zYhkOrM.jpeg Done downloading https://i.imgur.com/gtWsPu9.jpeg Done downloading https://i.imgur.com/LRoLTlK.jpeg Total time: 0.4095943249994889 ``` Pros: - No real limit on the number concurrent requests you can make since you're not limited by the number of processes/threads you can create. - You can always use `asyncio` within processes/threads if you need to. Gotchas: - Connection limits & memory issues are still a concern. - Not very intuitive if you're not familiar with async code. - You'll need to switch to `aiohttp` or `httpx` if you want to use async requests. - If you're using Jupyter notebooks, you'll need to use [`nest_asyncio`](https://github.com/erdewit/nest_asyncio) to make it work or [run it within an event loop](https://nocomplexity.com/documents/jupyterlab/tip-asyncio.html). --- ### references - [SuperFastPython](https://superfastpython.com/learning-paths/), an excellent source to get a deep understanding of concurrency in Python.