kiwix/zimit
1
0
Fork 0
mirror of https://github.com/openzim/zimit.git synced 2025-09-27 06:43:38 -04:00
Code Issues Packages Projects Releases Wiki Activity
165 Commits 8 Branches 44 Tags
Commit Graph

30 Commits

Author SHA1 Message Date
renaud gaudin
4b7e504d99 Updated test and stats to new crawl.json format 2023-01-31 11:12:36 +00:00
renaud gaudin
ce68493087 increased check_url timeouts 2022-07-25 08:41:08 +00:00
renaud gaudin
857e044c84 Fixed --allowHashUrls incorrectly requiring a value 2022-07-18 10:23:16 +00:00
renaud gaudin
8c6d2bfb45 using browsertrix-crawler 0.7 beta 2022-07-04 15:08:49 +00:00
renaud gaudin
142970bc0a Fixed #137: normalizes homepage redirects to standart ports 2022-06-22 09:57:01 +00:00
renaud gaudin
e32aac3ec0 code styling 2022-06-21 17:05:08 +00:00
renaud gaudin
932f97c999 updated tests for crawler and warc2zim 2022-06-21 16:55:32 +00:00
renaud gaudin
1f490ace8f Updated to browsertrix-crawler 0.6 and warc2zim 1.4 2022-06-21 12:04:56 +00:00
Ilya Kreymer
acf0aaf552 update to latest browsertrix-crawler 
test with dev build of warc2zim 1.4.0 release
 2022-06-14 14:58:33 +00:00
renaud gaudin
2e9c129523 new crawler folder structure 2021-05-12 17:03:48 +00:00
renaud gaudin
f746f7b020 use same waitUntil defaults as current crawler 2021-03-04 10:40:12 +00:00
renaud gaudin
cfa4b0e7f8 capture and incorporates limit info from crawl 2021-02-15 17:20:43 +00:00
renaud gaudin
76c92bdb4c Fixed #76: more flexible url redirects acceptance 
- accepts redirects to same first-level domain
- accepts redirects matching scope
 2021-01-15 12:50:53 +00:00
renaud gaudin
871f7ab58d Add custom-css option support (warc2zim) 2021-01-14 18:11:22 +00:00
renaud gaudin
3ffa34d46e Enhanced --statsFilename support 
- `--statsFilename` to now represent overall zimit progress and not just crawling
- Exposing a simpler (`done`, `total`) json format for progress
- Live converting individual step's progres into this file
- using warc2zim 1.3.3 for its `--progress-file` support
- Currently arbitrarily assigning 90% to crawl and 10% to warc2zim
 2020-12-10 10:44:39 +00:00
Ilya Kreymer
5084c54af6 stats: add support for stats output after every page crawled, fixes #39 
tests: integration test checks for stats.json
 2020-12-02 16:28:25 +00:00
Ilya Kreymer
c0bb0503b8 add support for --useSitemap <url> flag to load additional URLs, potentially fixing #34! 
reformat
 2020-11-14 22:01:36 +00:00
Ilya Kreymer
a930542af8 mobile + user agent support: 
- add support for custom user agent suffix +Zimit with email address specifyable via --adminEmail cmd arg #38
- add ability to crawl as mobile device with --mobileDevice flag (default to iPhone X)
add integration tests runnable in docker via github actions
logging: print temp dir, flush print statements for immediate logging
 2020-11-14 20:10:16 +00:00
renaud gaudin
0082d313ae Code formatting 
- Added requests as a dependency (although currently brought in by warc2zim)
- removed unused imports
- black code formatting and some cleanup
- revamped actual_url fetching
 2020-11-10 09:12:34 +00:00
Ilya Kreymer
5b640f2f8b main page redirect check: check if specified URL is a redirect, and use final URL if it is. Reject if redirect goes to a different domain, as suggested in #42 2020-11-10 06:07:27 +00:00
Ilya Kreymer
c228c8300c split zimit from core browsertrix-crawler, which has been moved to https://github.com/webrecorder/browsertrix-crawler 
use versioned browsertrix-crawler:0.1.0 image
part of #45
 2020-11-03 17:21:54 +00:00
Ilya Kreymer
ae9aba7a00 set default newContext to page as before 2020-10-28 18:19:27 +00:00
Ilya Kreymer
a425cd6956 - add 'newContext' command line option to specify the context for each new url: new page, new session, or new browser 
- convert the scope option to be a regex instead of just prefix
- remove custom wabac.js, now using released version in warc2zim
 2020-10-27 18:00:44 +00:00
Ilya Kreymer
91fe76c56e work on automated capture of vidoe (#9) 
- add autoplay behavior to reload known video sites to autoplay
- for video/audio on page, queue directly for loading if video.src or audio.src set to valid url, otherwise load through play in browser (may be slower)
- add extra wait if reloading for autoplay
- timeouts: set timeout for puppeteer-cluster double to timeout of page to avoid hitting that timeout during regular operation
- use browser from oldwebtoday/chrome:84 and puppeteer-core instead of puppeteer browser for consistent results
- temp testing: use custom wabac.js sw for testing (will use default from warc2zim), using warc2zim fuzzy-match branch for now
 2020-10-21 06:09:10 +00:00
Ilya Kreymer
ab4e2e1a14 cleanup: fix typo in print msg 2020-10-20 15:55:25 +00:00
Ilya Kreymer
904c95963c update to warc2zim 1.2.0, fixes from code review: 
- pass warc directory to warc2zim, supported in 1.2.0
- use Path for temp_root_dir
- use seconds instead of millis for page timeout, update help text
- fix help text for --scope
- restrict waitUntil to valid choices
 2020-10-19 19:44:01 +00:00
renaud gaudin
fb2232d8b1 not using partial paths 2020-10-19 15:14:46 +00:00
Ilya Kreymer
2e2db2f352 simplification: remove zimit user, su, and run chrome as root with --no-sandbox 
log exclusion regex
 2020-10-16 21:04:10 +00:00
Ilya Kreymer
2c1b401e93 fix codefactor complaints 2020-10-16 19:11:31 +00:00
Ilya Kreymer
c26fe5d4cd replace run.sh with python runner zimit.py, as suggested in #28 
should fix arg parsing issues in #28,#18
warc2zim now called directly from zimit.py, both for arg check and for actual zim creation
crawler renamed to crawler.js, no longer handles zim creation, only crawling
add signal handling to both zimit and crawler.js for smooth shutdown, should fix #25
pywb: update to latest dev version with dedup support, add redis for deduplication
 2020-10-16 18:54:04 +00:00