Fail on all HTTP error codes in check_url

This commit is contained in:
benoit74 2023-10-23 10:47:24 +02:00
parent 00051453e1
commit d8f6cef7f3
No known key found for this signature in database
GPG Key ID: B89606434FC7B530
2 changed files with 7 additions and 0 deletions

View File

@ -5,6 +5,12 @@ All notable changes to this project are documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html) (as of version 1.2.0).
## Unreleased
### Changed
- Scraper fails for all HTTP error codes returned when checking URL at startup (#223)
## [1.5.3] - 2023-10-02
### Changed

View File

@ -451,6 +451,7 @@ def check_url(url, scope=None):
resp = requests.head(
url.geturl(), stream=True, allow_redirects=True, timeout=(12.2, 27)
)
resp.raise_for_status()
except requests.exceptions.RequestException as exc:
print(f"failed to connect to {url.geturl()}: {exc}", flush=True)
raise SystemExit(1)