Writing a Threaded Downloader

We'll write a downloader in python using multiple threads to learn more about the threads

We'll cover the following...

The previous example wasn’t very useful other than as a tool to explain how threads work. So in this example, we will create a Thread class that can download files from the internet. The U.S. Internal Revenue Service has lots of PDF forms that it has its citizens use for taxes. We will use this free resource for our demo. Here’s the code:

Press + to interact
# Python 2 version
import os
import urllib2
from threading import Thread
class DownloadThread(Thread):
"""
A threading example that can download a file
"""
def __init__(self, url, name):
"""Initialize the thread"""
Thread.__init__(self)
self.name = name
self.url = url
def run(self):
"""Run the thread"""
handle = urllib2.urlopen(self.url)
fname = os.path.basename(self.url)
with open(fname, "wb") as f_handler:
while True:
chunk = handle.read(1024)
if not chunk:
break
f_handler.write(chunk)
msg = "%s has finished downloading %s!" % (self.name,
self.url)
print(msg)
def main(urls):
"""
Run the program
"""
for item, url in enumerate(urls):
name = "Thread %s" % (item+1)
thread = DownloadThread(url, name)
thread.start()
if __name__ == "__main__":
urls = ["http://www.irs.gov/pub/irs-pdf/f1040.pdf",
"http://www.irs.gov/pub/irs-pdf/f1040a.pdf",
"http://www.irs.gov/pub/irs-pdf/f1040ez.pdf",
"http://www.irs.gov/pub/irs-pdf/f1040es.pdf",
"http://www.irs.gov/pub/irs-pdf/f1040sb.pdf"]
main(urls)

This is basically a complete rewrite of the first script. In this one we import the os and urllib2 modules as well as the threading module. We will be using urllib2 to do the actual downloading inside the thread class. The os module is used to extract the name of ...

Access this course and 1400+ top-rated courses and projects.