kiwix/zimit
1
0
Fork 0
mirror of https://github.com/openzim/zimit.git synced 2025-09-24 04:30:11 -04:00
Code Issues Packages Projects Releases Wiki Activity

increased check_url timeouts

Browse Source
This commit is contained in:
renaud gaudin 2022-07-25 08:41:08 +00:00
parent 857e044c84
commit ce68493087
2 changed files with 3 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
CHANGELOG.md
View File

@ -17,6 +17,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Using warc2zim version xxx ⚠️ use released warc2zim before releasing - Using warc2zim version xxx ⚠️ use released warc2zim before releasing
- Using browsertrix-crawler 0.7 ⚠️ use non-beya before releasing - Using browsertrix-crawler 0.7 ⚠️ use non-beya before releasing
- Fixed `--allowHashUrls` being a boolean param - Fixed `--allowHashUrls` being a boolean param
- Increased `check_url` timeout (12s to connect, 27s to read) instead of 10s
## [1.2.0] - 2022-06-21 ## [1.2.0] - 2022-06-21

4
zimit.py
View File

@ -396,10 +396,10 @@ def check_url(url, scope=None):
url = urllib.parse.urlparse(url) url = urllib.parse.urlparse(url)
try: try:
resp = requests.head( resp = requests.head(
url.geturl(), stream=True, allow_redirects=True, timeout=10 url.geturl(), stream=True, allow_redirects=True, timeout=(12.2, 27)
) )
except requests.exceptions.RequestException as exc: except requests.exceptions.RequestException as exc:
print(f"failed to connect to {url}: {exc}", flush=True) print(f"failed to connect to {url.geturl()}: {exc}", flush=True)
raise SystemExit(1) raise SystemExit(1)
actual_url = urllib.parse.urlparse(resp.url) actual_url = urllib.parse.urlparse(resp.url)