Search⌘ K
AI Features

Writing a Threaded Downloader

Explore how to build a threaded file downloader in Python by creating a Thread class that downloads files concurrently. Understand how to use threading and queues to efficiently manage multiple downloads and keep your applications responsive.

We'll cover the following...

The previous example wasn’t very useful other than as a tool to explain how threads work. So in this example, we will create a Thread class that can download files from the internet. The U.S. Internal Revenue Service has lots of PDF forms that it has its citizens use for taxes. We will use this free resource for our demo. Here’s the code:

Python
# Python 2 version
import os
import urllib2
from threading import Thread
class DownloadThread(Thread):
"""
A threading example that can download a file
"""
def __init__(self, url, name):
"""Initialize the thread"""
Thread.__init__(self)
self.name = name
self.url = url
def run(self):
"""Run the thread"""
handle = urllib2.urlopen(self.url)
fname = os.path.basename(self.url)
with open(fname, "wb") as f_handler:
while True:
chunk = handle.read(1024)
if not chunk:
break
f_handler.write(chunk)
msg = "%s has finished downloading %s!" % (self.name,
self.url)
print(msg)
def main(urls):
"""
Run the program
"""
for item, url in enumerate(urls):
name = "Thread %s" % (item+1)
thread = DownloadThread(url, name)
thread.start()
if __name__ == "__main__":
urls = ["http://www.irs.gov/pub/irs-pdf/f1040.pdf",
"http://www.irs.gov/pub/irs-pdf/f1040a.pdf",
"http://www.irs.gov/pub/irs-pdf/f1040ez.pdf",
"http://www.irs.gov/pub/irs-pdf/f1040es.pdf",
"http://www.irs.gov/pub/irs-pdf/f1040sb.pdf"]
main(urls)

This is basically a complete rewrite of the first script. In this one we import the os and urllib2 modules as well as the threading module. We will be using urllib2 to do the actual downloading inside the thread class. The os module is used to extract the name of the file we’re downloading so we can use it to create a file with the same name on our machine. In the DownloadThread class, we set ...