NOT READY: warcio test #66

wumpus · 2019-01-26T00:26:25Z

An opinionated WARC standards-conformance tool.

Ready for review - I have yet to work on test coverage.

$ warcio test test/data/*.warc.gz test/data/*.warc
test/data/example-bad-non-chunked.warc.gz
  saw exception 
    ERROR: non-chunked gzip file detected, gzip block continues
    beyond single record.

    This file is probably not a multi-member gzip but a single gzip file.

    To allow seek, a gzipped WARC must have each record compressed into
    a single gzip member and concatenated together.

    This file is likely still valid and can be fixed by running:

    warcio recompress <path/to/file> <path/to/new_file>
  skipping rest of file
test/data/example-resource.warc.gz
  WARC-Record-ID <urn:uuid:6e7f60da-2c7b-11e7-aaf7-0242ac120007>
    WARC-Type resource
    digest pass
    comment: unknown field, no validation performed Warc-Referer https://webrecorder.io/temp-GRWZVUTV/temp/test/record/http://example.com/
    comment: unknown field, no validation performed Warc-User-Agent Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.81 Safari/537.36
test/data/example.warc.gz
  WARC-Record-ID <urn:uuid:a9c5c23a-0221-11e7-8fe3-0242ac120007>
    WARC-Type request
    digest not present
    error: WARC-IP-Address should be used for http and https requests
  WARC-Record-ID <urn:uuid:e6e395ca-0221-11e7-a18d-0242ac120005>
    WARC-Type revisit
    digest present but not checked
    recommendation: missing recommended header WARC-Refers-To
    comment: field was introduced after this warc version WARC-Refers-To-Target-URI http://example.com/ 1.0
    comment: field was introduced after this warc version WARC-Refers-To-Date 2017-03-06T04:02:06Z 1.0
  WARC-Record-ID <urn:uuid:e6e41fea-0221-11e7-8fe3-0242ac120007>
    WARC-Type request
    digest not present
    error: WARC-IP-Address should be used for http and https requests
test/data/example-wget-bad-target-uri.warc.gz
  WARC-Record-ID <urn:uuid:CEF11DC9-8D86-4F4B-9B8C-2235515B4537>
    WARC-Type request
    digest pass
    error: uri must not be within <> warc-target-uri <http://example.com/>
    error: invalid uri scheme, bad character warc-target-uri <http://example.com/>
  WARC-Record-ID <urn:uuid:FD8A6D04-AF8A-4A36-A889-8094487CDF2D>
    WARC-Type response
    payload digest failed sha1:B2LTWWPUOYAH7UIPQ7ZUPQ4VMBSVC36A
    error: uri must not be within <> warc-target-uri <http://example.com/>
    error: invalid uri scheme, bad character warc-target-uri <http://example.com/>
  WARC-Record-ID <urn:uuid:E5AC383F-F107-47BC-99B7-176FD8DE6E94>
    WARC-Type metadata
    digest pass
    error: uri must not be within <> warc-target-uri <metadata://gnu.org/software/wget/warc/MANIFEST.txt>
    error: invalid uri scheme, bad character warc-target-uri <metadata://gnu.org/software/wget/warc/MANIFEST.txt>
  WARC-Record-ID <urn:uuid:543BCA4F-A305-4383-B511-0BCF23F7AD8D>
    WARC-Type resource
    digest pass
    error: uri must not be within <> warc-target-uri <metadata://gnu.org/software/wget/warc/wget_arguments.txt>
    error: invalid uri scheme, bad character warc-target-uri <metadata://gnu.org/software/wget/warc/wget_arguments.txt>
  WARC-Record-ID <urn:uuid:CCD67DB5-13FA-447B-BF05-BF1BDC8ED3A0>
    WARC-Type resource
    digest pass
    error: uri must not be within <> warc-target-uri <metadata://gnu.org/software/wget/warc/wget.log>
    error: invalid uri scheme, bad character warc-target-uri <metadata://gnu.org/software/wget/warc/wget.log>
test/data/example-wrong-chunks.warc.gz
  saw exception Invalid WARC record, first line: <!doctype html>
  skipping rest of file
test/data/post-test.warc.gz
  WARC-Record-ID <urn:uuid:59a6b068-cbc2-4767-9525-33043d2709c7>
    WARC-Type request
    digest pass
    error: WARC-IP-Address should be used for http and https requests
  WARC-Record-ID <urn:uuid:5eb8ee92-cda1-4503-a7a3-c63f1ab6515b>
    WARC-Type request
    digest pass
    error: WARC-IP-Address should be used for http and https requests
  WARC-Record-ID <urn:uuid:c79a62e3-5a4b-450d-a093-3a7fefa09664>
    WARC-Type request
    digest pass
    error: WARC-IP-Address should be used for http and https requests
test/data/example-digest-bad.warc
  WARC-Record-ID <urn:uuid:a9c5c23a-0221-11e7-8fe3-0242ac120007>
    WARC-Type request
    payload digest failed: sha1:1112H3S6NNFQ2MSVX7XZKYAYSCX5QBYJ
    error: WARC-IP-Address should be used for http and https requests
  WARC-Record-ID <urn:uuid:a9c5c23a-0221-11e7-8fe3-0242ac120007>
    WARC-Type request
    digest pass
    error: WARC-IP-Address should be used for http and https requests
  WARC-Record-ID <urn:uuid:a9c5c23a-0221-11e7-8fe3-0242ac120007>
    WARC-Type request
    digest pass
    error: WARC-IP-Address should be used for http and https requests
  WARC-Record-ID <urn:uuid:a9c5c23a-0221-11e7-8fe3-0242ac120007>
    WARC-Type request
    digest pass
    error: WARC-IP-Address should be used for http and https requests
test/data/example-iana.org-chunked.warc
  WARC-Record-ID <urn:uuid:c46fbf5f-0876-4652-a348-e9b6c322eabb>
    WARC-Type request
    digest pass
    error: WARC-IP-Address should be used for http and https requests
test/data/example-trunc.warc
  WARC-Record-ID <urn:uuid:a9c51e3e-0221-11e7-bf66-0242ac120005>
    WARC-Type response
    block digest failed: sha1:DR5MBP7OD3OPA7RFKWJUD4CTNUQUGFC5
    payload digest failed sha1:G7HRM7BGOKSKMSXZAHMUQTTV53QOFSMK
    WARNING: Record not followed by newline, perhaps Content-Length is invalid
    Offset: 2560
    Remainder: b'\x00\x00\r\n'
  WARC-Record-ID <urn:uuid:a9c5c23a-0221-11e7-8fe3-0242ac120007>
    WARC-Type request
    digest not present
    error: WARC-IP-Address should be used for http and https requests
test/data/example.warc
  WARC-Record-ID <urn:uuid:a9c5c23a-0221-11e7-8fe3-0242ac120007>
    WARC-Type request
    digest not present
    error: WARC-IP-Address should be used for http and https requests
  WARC-Record-ID <urn:uuid:e6e395ca-0221-11e7-a18d-0242ac120005>
    WARC-Type revisit
    digest present but not checked
    recommendation: missing recommended header WARC-Refers-To
    comment: field was introduced after this warc version WARC-Refers-To-Target-URI http://example.com/ 1.0
    comment: field was introduced after this warc version WARC-Refers-To-Date 2017-03-06T04:02:06Z 1.0
  WARC-Record-ID <urn:uuid:e6e41fea-0221-11e7-8fe3-0242ac120007>
    WARC-Type request
    digest not present
    error: WARC-IP-Address should be used for http and https requests

codecov · 2019-01-26T01:14:20Z

Codecov Report

❗ No coverage uploaded for pull request base (develop@59198eb). Click here to learn what that means.
The diff coverage is 86.44%.

@@            Coverage Diff             @@
##             develop      #66   +/-   ##
==========================================
  Coverage           ?   96.19%           
==========================================
  Files              ?       19           
  Lines              ?     2078           
  Branches           ?      390           
==========================================
  Hits               ?     1999           
  Misses             ?       36           
  Partials           ?       43

Impacted Files	Coverage Δ
warcio/archiveiterator.py	`100% <ø> (ø)`
warcio/tester.py	`88.96% <100%> (ø)`
warcio/recordloader.py	`98.69% <100%> (ø)`
warcio/bufferedreaders.py	`94.81% <57.89%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 59198eb...fc19c7d. Read the comment docs.

wumpus · 2019-01-26T02:01:08Z

@N0taN3rd traditionally you've been my best reviewer :-)

test/test_tests.py

N0taN3rd · 2019-01-29T22:33:15Z

warcio/recordloader.py

@@ -55,7 +55,7 @@ class ArcWarcRecordLoader(object):
    NON_HTTP_SCHEMES = ('dns:', 'whois:', 'ntp:')
    HTTP_SCHEMES = ('http:', 'https:')

-    def __init__(self, verify_http=True, arc2warc=True):
+    def __init__(self, verify_http=True, arc2warc=True, fixup_bugs=True):


In the spirit of how digest checking is handled by ArchiveIterator, fixup_bugs=True should probably default to False

warcio/warcio/archiveiterator.py

Line 46 in 759ab07

check_digests=False):

The existing code did the bug fixup unconditionally, I'm preserving that default.

digest checking defaults off because it's expensive and @ikreymer prefers people remain in the dark as to how many invalid warcs there are :-)

warcio/tester.py

N0taN3rd · 2019-01-29T22:36:03Z

warcio/archiveiterator.py

@@ -43,12 +43,13 @@ class ArchiveIterator(six.Iterator):
    def __init__(self, fileobj, no_record_parse=False,
                 verify_http=False, arc2warc=False,
                 ensure_http_headers=False, block_size=BUFF_SIZE,
-                 check_digests=False):
+                 check_digests=False, fixup_bugs=True):


fixup_bugs=True -> fixup_bugs=False in the spirit of check_digests=False

wumpus · 2019-01-30T01:54:08Z

@N0taN3rd has done a preliminary review, the main addition since then is some global checks.

At this point I think the code is feature-complete, well, for the things I'm planning for the first pass, and the main work remaining is coverage.

wumpus added 2 commits January 25, 2019 16:05

warcio test

ff1f543

documentation

ebb721f

wumpus mentioned this pull request Jan 26, 2019

Warc tester #59

Open

wumpus added 2 commits January 25, 2019 16:42

tests

7aa060d

tests

24f3000

wumpus added 18 commits January 26, 2019 08:39

coverage

40f9fc6

python 2.7 test fix

c70e68e

python 2.7 fixes

1847633

coverage

2c676db

py2 testing

97ee457

py2 windows testing

df50151

coverage

858a752

branch coverage

5bfffea

py2 branch coverage

bb31f14

py2 testing

cc54259

add record ids to test

2b8d596

preserve capitalization in messages

c704fe9

capitals and colons

3839fa1

use valid record ids

8b9032d

warc-segment-number cleaner recommendation

2a10b23

segment origin id

81c9f0a

timestamp checking

c78343a

buglet

efe0fda

N0taN3rd suggested changes Jan 29, 2019

View reviewed changes

global checks

7a26644

wumpus added 2 commits January 31, 2019 12:03

check -v; capitalize most commentary

1d6fd9d

...

5b716b7

wumpus added 29 commits November 4, 2019 22:30

python 2.7 fixes

68938bd

coverage

234468a

py2 testing

e7f88e7

py2 windows testing

8662073

coverage

291460e

branch coverage

69080d5

py2 branch coverage

2e1d820

py2 testing

bbdb57b

add record ids to test

fc2d7b4

preserve capitalization in messages

d1fe18e

capitals and colons

484da9c

use valid record ids

4687497

warc-segment-number cleaner recommendation

bcfe672

segment origin id

7f715c0

timestamp checking

2583f19

buglet

8eb87e8

global checks

3a8747e

check -v; capitalize most commentary

f7cd1db

...

b570b6c

revisits and global detection with just one file

921e748

show errors for decompression and unchunking failures

4265b62

make this function reentrant

08e6bd9

narrow exception; fix bug not reading to the end of a chunked buffer

d1f48ed

...

6e44a44

put tester output in external files

59198eb

wip

b61878e

tests pass

2d2b7d5

merge

f4bc076

comments

fc19c7d

acidus99 mentioned this pull request Sep 12, 2023

warcio accepts a bare LF everywhere a CRLF is required by the spec #157

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

NOT READY: warcio test #66

NOT READY: warcio test #66

wumpus commented Jan 26, 2019

codecov bot commented Jan 26, 2019 •

edited

wumpus commented Jan 26, 2019

N0taN3rd Jan 29, 2019

wumpus Jan 30, 2019

N0taN3rd Jan 29, 2019

wumpus commented Jan 30, 2019 •

edited

NOT READY: warcio test #66

Are you sure you want to change the base?

NOT READY: warcio test #66

Conversation

wumpus commented Jan 26, 2019

codecov bot commented Jan 26, 2019 • edited

Codecov Report

wumpus commented Jan 26, 2019

N0taN3rd Jan 29, 2019

Choose a reason for hiding this comment

wumpus Jan 30, 2019

Choose a reason for hiding this comment

N0taN3rd Jan 29, 2019

Choose a reason for hiding this comment

wumpus commented Jan 30, 2019 • edited

codecov bot commented Jan 26, 2019 •

edited

wumpus commented Jan 30, 2019 •

edited