{"payload":{"feedbackUrl":"https://github.com/orgs/community/discussions/53140","repo":{"id":679466349,"defaultBranch":"prod","name":"csv2db","ownerLogin":"Kevin-Prichard","currentUserCanPush":false,"isFork":false,"isEmpty":false,"createdAt":"2023-08-16T23:08:24.000Z","ownerAvatar":"https://avatars.githubusercontent.com/u/3037843?v=4","public":true,"private":false,"isOrgOwned":false},"refInfo":{"name":"","listCacheKey":"v0:1692227304.0","currentOid":""},"activityList":{"items":[{"before":"f2d5394a3a633c15728c4afae142884a5a9cd02b","after":"e767ee846b09cea0fca8778eafd881f19a687738","ref":"refs/heads/prod","pushedAt":"2023-10-22T21:42:05.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"Kevin-Prichard","name":"Kevin Prichard","path":"/Kevin-Prichard","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/3037843?s=80&v=4"},"commit":{"message":"- experiment_with_pipes: sandbox for understanding how subprocess pipes really work","shortMessageHtmlLink":"- experiment_with_pipes: sandbox for understanding how subprocess pip…"}},{"before":"b5914d6451869acb286b8cf0e8bf5c7438d1552a","after":"f2d5394a3a633c15728c4afae142884a5a9cd02b","ref":"refs/heads/prod","pushedAt":"2023-10-22T21:40:45.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"Kevin-Prichard","name":"Kevin Prichard","path":"/Kevin-Prichard","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/3037843?s=80&v=4"},"commit":{"message":"minor cleanup","shortMessageHtmlLink":"minor cleanup"}},{"before":"a42f72b1e5cdd5ea212ec0ffea6c2e5e0ba0c1e9","after":"b5914d6451869acb286b8cf0e8bf5c7438d1552a","ref":"refs/heads/prod","pushedAt":"2023-08-22T06:23:50.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"Kevin-Prichard","name":"Kevin Prichard","path":"/Kevin-Prichard","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/3037843?s=80&v=4"},"commit":{"message":"Update README.md and csv2db.py for current usage instructions","shortMessageHtmlLink":"Update README.md and csv2db.py for current usage instructions"}},{"before":"715f9cf352d1b9814fbb309166333df77c51a5c4","after":"a42f72b1e5cdd5ea212ec0ffea6c2e5e0ba0c1e9","ref":"refs/heads/prod","pushedAt":"2023-08-22T06:23:07.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"Kevin-Prichard","name":"Kevin Prichard","path":"/Kevin-Prichard","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/3037843?s=80&v=4"},"commit":{"message":"- count_rows: returns the rowcount per table","shortMessageHtmlLink":"- count_rows: returns the rowcount per table"}},{"before":"03f598af485dee8a28ecd995fc354a60679dc671","after":"715f9cf352d1b9814fbb309166333df77c51a5c4","ref":"refs/heads/prod","pushedAt":"2023-08-21T20:56:16.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"Kevin-Prichard","name":"Kevin Prichard","path":"/Kevin-Prichard","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/3037843?s=80&v=4"},"commit":{"message":"- csv2db: add cli options --create-only, --show-struct, and --save-struct; add docstring to create_import_sqlite()\n- csv_scanner: add CSVScanner.stats property; critically-\n  - convert spaces to underscores in scanned fieldnames\n  - repair regex as decimal was utterly broken, with DOUBLEs turning up as INTEGERs\n  - re-add --max-rows implementation\n- choose_key_fields: began work on a foreign key mapper, but it's complicated by inconsistently named parent and child fields, so doing by hand","shortMessageHtmlLink":"- csv2db: add cli options --create-only, --show-struct, and --save-st…"}},{"before":"75f12f7ea38ea1323023126d0ed045e1cf1ffa11","after":"03f598af485dee8a28ecd995fc354a60679dc671","ref":"refs/heads/prod","pushedAt":"2023-08-20T20:56:29.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"Kevin-Prichard","name":"Kevin Prichard","path":"/Kevin-Prichard","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/3037843?s=80&v=4"},"commit":{"message":"- csv_scanner: remove all traces of progress reporting\n- csv2db: add progress reporting cli params, pass thru progress params to *IOProgressWrapper classes, set most logger statements to .debug(), use BytesIOProgressWrapper for sql inserts\n- bytes_io_stats_wrapper: adds BytesIOStatsWrapper, which is like TextIOStatsWrapper except it \"has-a\" BufferedIOBase\n- bytes_io_progress_wrapper: adds BytesIOProgressWrapper, which extends BytesIOStatsWrapper, like TextIOProgressWrapper extends TextIOStatsWrapper\n- no tests yet for the two BytesIO*Wrapper classes... soon come\n- the pairs of classes [BytesIO, TextIO] * [StatsWrapper, ProgressWrapper] might be consolidated, their functionality is nearly identical\n  - the handling of newline counting is different (make it a class const)\n  - TextIOStatsWrapper would need to \"has-a\" TextIOBase instead of inheriting TextIOWrapper which doesn't add much","shortMessageHtmlLink":"- csv_scanner: remove all traces of progress reporting"}},{"before":"d2934870c5f211cfabf22e407a643f22ef6fb1a0","after":"75f12f7ea38ea1323023126d0ed045e1cf1ffa11","ref":"refs/heads/prod","pushedAt":"2023-08-20T20:41:33.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"Kevin-Prichard","name":"Kevin Prichard","path":"/Kevin-Prichard","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/3037843?s=80&v=4"},"commit":{"message":"- test_progress_wrapper: renamed to test_text_progress_wrapper\n- test_text_progress_wrapper: expanded coverage of TextIOProgressWrapper\n- text_io_progress_wrapper: improved & corrected progress parameter handling incl. raises when incorrect","shortMessageHtmlLink":"- test_progress_wrapper: renamed to test_text_progress_wrapper"}},{"before":"b498b9bf3279c3cb99cb2e0837ad2dbf5873fc45","after":"d2934870c5f211cfabf22e407a643f22ef6fb1a0","ref":"refs/heads/prod","pushedAt":"2023-08-20T20:38:04.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"Kevin-Prichard","name":"Kevin Prichard","path":"/Kevin-Prichard","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/3037843?s=80&v=4"},"commit":{"message":"- test_equivalencies: move test data & pathname declarations to separate file so they may be shared by other tests\n- test_data: file where test_equivalencies test data refs were moved","shortMessageHtmlLink":"- test_equivalencies: move test data & pathname declarations to separ…"}},{"before":"b556d9b0ee1cff7d765c7ea09f42f0dcad547fa8","after":"b498b9bf3279c3cb99cb2e0837ad2dbf5873fc45","ref":"refs/heads/prod","pushedAt":"2023-08-19T09:56:27.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"Kevin-Prichard","name":"Kevin Prichard","path":"/Kevin-Prichard","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/3037843?s=80&v=4"},"commit":{"message":"- text_io_progress_wrapper: TextIOProgressWrapper, a subclass of TextIOStatsWrapper for progress updates while reading a TextIO\n- text_io_stats_wrapper: add self._line_num\n- csv2db: now using TextIOProgressWrapper to report table scan progress; eliminated get_table_lengths() -replaced by rows_est in TextIOProgressWrapper\n- csv_scanner: removed progress-reporting 1-off code, replaced by TextIOProgressWrapper in csv2db\n- test_stats_wrapper: tests for TextIOStatsWrapper\n- test_progress_wrapper: tests for TextIOProgressWrapper","shortMessageHtmlLink":"- text_io_progress_wrapper: TextIOProgressWrapper, a subclass of Text…"}},{"before":"4b26ca4355478b09ac289563c887cadfa7150b36","after":"b556d9b0ee1cff7d765c7ea09f42f0dcad547fa8","ref":"refs/heads/prod","pushedAt":"2023-08-18T21:31:01.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"Kevin-Prichard","name":"Kevin Prichard","path":"/Kevin-Prichard","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/3037843?s=80&v=4"},"commit":{"message":"- text_io_stats_wrapper: remove commented-out code\n- all else: pep8 cleanup & compliance","shortMessageHtmlLink":"- text_io_stats_wrapper: remove commented-out code"}},{"before":"4eb503bcac8b67a62819118ec76f83e745e96b5e","after":"4b26ca4355478b09ac289563c887cadfa7150b36","ref":"refs/heads/prod","pushedAt":"2023-08-18T21:25:01.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"Kevin-Prichard","name":"Kevin Prichard","path":"/Kevin-Prichard","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/3037843?s=80&v=4"},"commit":{"message":"- text_io_stats_wrapper.py: move class TextIOStatsWrapper to own file\n- csv2db: use only file name as table_name, not the path leading to the file name\n- csv_scanner: move out TextIOStatsWrapper\n- test_equivalencies: add test 'test_that_zipped_csvs_have_expected_structure'","shortMessageHtmlLink":"- text_io_stats_wrapper.py: move class TextIOStatsWrapper to own file"}},{"before":"30c305dfa486f5a46e62c27285eebd5d1daae475","after":"4eb503bcac8b67a62819118ec76f83e745e96b5e","ref":"refs/heads/prod","pushedAt":"2023-08-18T20:47:09.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"Kevin-Prichard","name":"Kevin Prichard","path":"/Kevin-Prichard","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/3037843?s=80&v=4"},"commit":{"message":"- requirements.txt\n- tests/data/*: one zip, plus the three files it contains\n- tests/test_equivalencies.py: initial tests for determining that this module works correctly and will remain working correctly\n- csv2db.py:\n  - zip_walker: using named parameters, better var and param names, sharing file length with CSVScanner\n- csv_scanner.py:\n  - CSVScanner: in anticipation of externalization of progress reporting, I parameterized CSVScanner for row-wise and pct-wise reporting, and corrected and cleaned up the logic to work properly\n  - TextIOStatsWrapper: this class provides current `.char_num` (character position) and  `.line_num` (line number) of the underlying TextIOBase, since of neither IOBase nor TextIOBase offer them","shortMessageHtmlLink":"- requirements.txt"}},{"before":"df6caf13a01ffbf562221d63a368e2af0929c458","after":"30c305dfa486f5a46e62c27285eebd5d1daae475","ref":"refs/heads/prod","pushedAt":"2023-08-17T20:37:42.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"Kevin-Prichard","name":"Kevin Prichard","path":"/Kevin-Prichard","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/3037843?s=80&v=4"},"commit":{"message":"- csv2db.py:\n  - zip_walker: implement name_filter regex handling.  re.IGNORE_CASE is always assumed. The entire expression is up to caller, including ^ and $ if needed","shortMessageHtmlLink":"- csv2db.py:"}},{"before":"2fd86f29c3f76c988effad0bc55b0c9072f0fec0","after":"df6caf13a01ffbf562221d63a368e2af0929c458","ref":"refs/heads/prod","pushedAt":"2023-08-17T20:05:17.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"Kevin-Prichard","name":"Kevin Prichard","path":"/Kevin-Prichard","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/3037843?s=80&v=4"},"commit":{"message":"- csv2db.py: rename --file/-f param to --zip/-z\n  - zip_walker:\n    - output_dir: removed\n    - name_filter_rx: added (zip member filtering regex), for skipping archive member names other than spec'd\n    - max_files: removed\n    - max_rows: default None\n    - output_fn: default None\n- TODO.md: moved to Issues\n- .gitignore: ignore .ai dir from a plugin","shortMessageHtmlLink":"- csv2db.py: rename --file/-f param to --zip/-z"}},{"before":"07cc95df49e0b5d4e0563ab872ec4673c0f26ffe","after":"2fd86f29c3f76c988effad0bc55b0c9072f0fec0","ref":"refs/heads/prod","pushedAt":"2023-08-17T02:13:48.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"Kevin-Prichard","name":"Kevin Prichard","path":"/Kevin-Prichard","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/3037843?s=80&v=4"},"commit":{"message":"- README.md: updated with instructions\n- csv2db.py: command line interface that runs CSVScanner and imports CSV data to a SQLite db\n- csv_scanner.py: home of class CSVScanner; given an open file handle, it scans and conkulates the most appropriate column types for each column\n- .gitignore: skip .idea directory\n- TODO.md: notes about future fixes","shortMessageHtmlLink":"- README.md: updated with instructions"}},{"before":"55f3a7d7d18bf0e7a1dda4f5467ef1aae02e0184","after":"07cc95df49e0b5d4e0563ab872ec4673c0f26ffe","ref":"refs/heads/prod","pushedAt":"2023-08-16T23:12:52.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"Kevin-Prichard","name":"Kevin Prichard","path":"/Kevin-Prichard","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/3037843?s=80&v=4"},"commit":{"message":"Update README.md","shortMessageHtmlLink":"Update README.md"}},{"before":null,"after":"55f3a7d7d18bf0e7a1dda4f5467ef1aae02e0184","ref":"refs/heads/prod","pushedAt":"2023-08-16T23:08:24.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"Kevin-Prichard","name":"Kevin Prichard","path":"/Kevin-Prichard","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/3037843?s=80&v=4"},"commit":{"message":"Initial commit","shortMessageHtmlLink":"Initial commit"}}],"hasNextPage":false,"hasPreviousPage":false,"activityType":"all","actor":null,"timePeriod":"all","sort":"DESC","perPage":30,"cursor":"djE6ks8AAAADnRY6CQA","startCursor":null,"endCursor":null}},"title":"Activity · Kevin-Prichard/csv2db"}