Python requests Module
Python comes with the requests module, which is primarily used for sending HTTP requests. The requests module is more concise than the urllib module.
Example
# Import the requests package
import requests
# Send a request
x = requests.get('https://www.tutorialpro.org/')
# Return the webpage content
print(x.text)
Each call to a requests method returns a response object, which contains the specific response information.
The response information is as follows:
| Attribute or Method | Description |
|---|---|
| apparent_encoding | Encoding method |
| close() | Close the connection to the server |
| content | Return the response content in bytes |
| cookies | Return a CookieJar object containing cookies sent back from the server |
| elapsed | Return a timedelta object containing the time elapsed from sending the request to the response arrival, useful for testing response speed, e.g., r.elapsed.microseconds indicates the response time in microseconds |
| encoding | Encoding used to decode r.text |
| headers | Return the response headers in dictionary format |
| history | Return a list of response objects containing the request history (URLs) |
| is_permanent_redirect | Return True if the response is a permanently redirected URL, otherwise False |
| is_redirect | Return True if the response is redirected, otherwise False |
| iter_content() | Iterate over the response |
| iter_lines() | Iterate over the lines of the response |
| json() | Return the JSON object of the result (the result must be in JSON format, otherwise an error is raised) |
| links | Return the parsed header links of the response |
| next | Return the PreparedRequest object for the next request in the redirect chain |
| ok | Check the "status_code" value, return True if less than 400, otherwise False |
| raise_for_status() | Return an HTTPError object if an error occurs |
| reason | Description of the response status, such as "Not Found" or "OK" |
| request | Return the request object that led to this response |
| status_code | Return the HTTP status code, such as 404 and 200 (200 is OK, 404 is Not Found) |
| text | Return the response content in unicode |
| url | Return the URL of the response |
Example
# Import the requests package
import requests
# Send a request
x = requests.get('https://www.tutorialpro.org/')
# Return the HTTP status code
print(x.status_code)
# Description of the response status
print(x.reason)
# Return the encoding
print(x.apparent_encoding)
Output:
200
OK
utf-8
Request a JSON data file and return the JSON content:
Example
# Import the requests package
import requests
# Send a request
x = requests.get('https://httpbin.org/get')
# Return JSON data
print(x.json())
Output:
{'args': {},
'headers': {'Accept': '*/*',
'Accept-Encoding': 'gzip, deflate',
'Host': 'httpbin.org',
'User-Agent': 'python-requests/2.25.1',
'X-Amzn-Trace-Id': 'Root=1-63aa454d-7bdf6f2d340541ee7e969687'},
'origin': '124.78.42.136',
'url': 'https://httpbin.org/get'}
requests Methods
The requests methods are as follows:
| Method | Description |
|---|---|
| delete(url, args) | Send a DELETE request to the specified URL |
| get(url, params, args) | Send a GET request to the specified URL |
| head(url, args) | Send HEAD request to the specified url |
| patch(url, data, args) | Send PATCH request to the specified url |
| post(url, data, json, args) | Send POST request to the specified url |
| put(url, data, args) | Send PUT request to the specified url |
| request(method, url, args) | Send the specified request method to the specified url |
Using requests.request() to send a GET request:
## Example
Import the requests package
import requests
Send request
x = requests.request('get', 'https://www.tutorialpro.org/')
Return webpage content
print(x.status_code)
Output result as follows:
200
Setting request headers:
## Example
Import the requests package
import requests
kw = {'s':'python tutorial'}
Set request headers
headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36"}
params accepts a dictionary or string of query parameters, dictionary type is automatically converted to url encoding, no need for urlencode()
response = requests.get("https://www.tutorialpro.org/", params = kw, headers = headers)
View response status code
print (response.status_code)
View response header character encoding
print (response.encoding)
View full url address
print (response.url)
View response content, response.text returns data in Unicode format
print(response.text)
Output result as follows:
200 UTF-8 https://www.tutorialpro.org/?s=python+tutorial
... other content ...
The post() method can send a POST request to the specified url, generally in the following format:
requests.post(url, data={key: value}, json={key: value}, args)
-
**url** The request url.
-
**data** Parameter is a dictionary, list of tuples, bytes, or file object to be sent to the specified url.
-
**json** Parameter is a JSON object to be sent to the specified url.
-
**args** Other parameters such as cookies, headers, verify, etc.
## Example
Import the requests package
import requests
Send request
x = requests.post('https://www.tutorialpro.org/try/ajax/demo_post.php')
Return webpage content
print(x.text)
Output result as follows:
<p style='color:red;'>This content is requested using the POST method.</p><p style='color:red;'>Request time: 2022-05-26 17:30:47</p>
POST request with parameters:
## Example
Import the requests package
import requests
Form parameters, parameter names are fname and lname
myobj = {'fname': 'tutorialpro','lname': 'Boy'}
Send request
x = requests.post('https://www.tutorialpro.org/try/ajax/demo_post2.php', data = myobj)
Return webpage content
print(x.text)
Output result as follows:
<p style='color:red;'>Hello, tutorialpro Boy, how are you today?</p>