Import modulesįor this example, we only need the requests and multiprocessing Python modules to download files in parallel. The code in this tutorial uses only modules available from the Python standard library, so no installations are required. The tutorial demonstrates how to develop a generic file download function in Python and apply it to download multiple files with serial and parallel approaches. With a parallel file download routine, you can download multiple files simultaneously and save a considerable amount of time.

This serial approach can work well with a few small files, but if you are downloading many files or large files, you’ll want to use a parallel approach to maximize your computational resources. The easiest way to download files is using a simple Python loop to iterate through a list of URLs to download. There are several ways for automating file downloads in Python. Results = ThreadPool(5).Automating file downloads can save a lot of time. Each call will take the next element in urls list import requestsįrom multiprocessing.pool import ThreadPool However, this puts substantial load on the server and you need to be sure that the server can handle such concurrent loads. Also note that we are running 5 threads concurrently in the script below and you may want to increase it if you have a large number of files to download. Without the iteration of the results list, the program will terminate even before the threads are started. Note the use of results list which forces python to continue execution until all the threads are complete.

The following python program shows how to download multiple files concurrently by using multiprocessing library which has support for thread pools. The download program above can be substantially speeded up by running them in parallel. # if url is abc/xyz/file.txt, the file name will be file.txt However the download may take sometime since it is executed sequentially. The following python 3 program downloads a list of urls to a list of local files. # the file name at the end is used as the local file nameĪfter running the above program, you will find a file named 'posts' in the same folder where you have the script saved. # if the url is, the file name will be file.txt # assumes that the last segment after the / represents the file name The following example assumes that the url contains the name of the file at the end and uses it as the name for the locally saved file.

The following python 3 program downloads a given url to a local file. You may need to prefix the above command with sudo if you get permission error in your linux system.

Python download a file from url