Easy Tutorial
❮ Scrapy Detail Verilog Loop ❯

Efficiency Comparison Experiment of Single-threading, Multi-threading, and Multi-processing in Python

Category Programming Technology

>

Python is a language that runs in an interpreter. According to research, there is a Global Interpreter Lock (GIL) in Python. When using multi-threading (Thread), it cannot take advantage of multi-core capabilities. However, using multi-processing (Multiprocess) can leverage the advantages of multi-core to truly improve efficiency.

Comparative Experiment

Research indicates that if the multi-threaded process is CPU-intensive, then multi-threading does not offer much efficiency improvement. On the contrary, it may lead to a decrease in efficiency due to frequent thread switching, and multi-processing is recommended. If it is IO-intensive, multi-threaded processes can use the idle time when IO is blocked and waiting to execute other threads, thus improving efficiency. Therefore, we compare the efficiency of different scenarios based on the experiment.

Operating System CPU Memory Hard Drive
Windows 10 Dual-core 8GB Mechanical Hard Drive

(1) Import the required modules

import requests
import time
from threading import Thread
from multiprocessing import Process

(2) Define a CPU-intensive computation function

def count(x, y):
    # Make the program complete 500,000 calculations
    c = 0
    while c < 500000:
        c += 1
        x += x
        y += y

(3) Define an IO-intensive file reading and writing function

def write():
    f = open("test.txt", "w")
    for x in range(5000000):
        f.write("testwrite\n")
    f.close()
def read():
    f = open("test.txt", "r")
    lines = f.readlines()
    f.close()

(4) Define a network request function

_head = {
            'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36'}
url = "http://www.tieba.com"
def http_request():
    try:
        webPage = requests.get(url, headers=_head)
        html = webPage.text
        return {"context": html}
    except Exception as e:
        return {"error": e}

(5) Test the time required for linear execution of IO-intensive operations, CPU-intensive operations, and network request-intensive operations

# CPU-intensive operations
t = time.time()
for x in range(10):
    count(1, 1)
print("Linear CPU", time.time() - t)
# IO-intensive operations
t = time.time()
for x in range(10):
    write()
    read()
print("Linear IO", time.time() - t)
# Network request-intensive operations
t = time.time()
for x in range(10):
    http_request()
print("Linear Http Request", time.time() - t)

>

Output

-

CPU-intensive: 95.6059999466, 91.57099986076355, 92.52800011634827, 99.96799993515015

-

IO-intensive: 24.25, 21.76699995994568, 21.769999980926514, 22.060999870300293

-

Network request-intensive: 4.519999980926514, 8.563999891281128, 4.371000051498413, 4.522000074386597, 14.671000003814697

(6) Test the time required for multi-threaded concurrent execution of CPU-intensive operations

counts = []
t = time.time()
for x in range(10):
    thread = Thread(target=count, args=(1,1))
    counts.append(thread)
    thread.start()
e = counts.__len__()
while True:
    for th in counts:
        if not th.is_alive():
            e -= 1
    if e <= 0:
        break
print(time.time() - t)

>

Output: 25.69700002670288, 24.02400016784668

(7) Test the time required for multi-threaded concurrent execution of IO-intensive operations

``` def io(): write() read()

t = time.time() ios = [] t = time.time() for x in range(10): thread = Thread(target

❮ Scrapy Detail Verilog Loop ❯