1.0.0

What's Changed

• Add FileBasedRestartableBatchDataProcessor. by @PhilipMay in #154
• Add option to pass OpenAI key with os env. by @PhilipMay in #156
• Change data format to pkl / joblib for FileBasedRestartableBatchDataProcessor. by @PhilipMay in #157
• Fix unknown file issue in FileBasedRestartableBatchDataProcessor. by @PhilipMay in #158
• Avoid duplicate results in FileBasedRestartableBatchDataProcessor. by @PhilipMay in #159
• Add openai async function. by @PhilipMay in #160
• Add len to FileBasedRestartableBatchDataProcessor by @PhilipMay in #161
• Handle orphaned lock files for FileBasedRestartableBatchDataProcessor. by @PhilipMay in #162
• Add ignore error in FileBasedRestartableBatchDataProcessor data loading. by @PhilipMay in #163

Full Changelog: 0.12.3...1.0.0

0.12.3

What's Changed

• Add clean function clean_all_invisible_chars_and_strip. by @PhilipMay in #153

Full Changelog: 0.12.2...0.12.3

0.12.2

What's Changed

• Add SoMaJo token extractor. by @PhilipMay in #152

Full Changelog: 0.12.0...0.12.2

0.12.0

What's Changed

• Fix Somajo md link extraction. by @PhilipMay in #150

Full Changelog: 0.11.1...0.12.0

0.11.1

What's Changed

• Add pytest-random-order plugin and configure options. by @PhilipMay in #147
• Add default language for detection. by @PhilipMay in #149

Full Changelog: 0.11.0...0.11.1

0.11.0

What's Changed

• Add ArangoDB upload tool. by @PhilipMay in #141
• Add hypothesis tests for OpenAiTokenCounter class. by @PhilipMay in #143
• Add support for new OpenAI API. by @PhilipMay in #144

Full Changelog: 0.10.0...0.11.0

0.10.1rc1

What's Changed

• Add ArangoDB upload tool. by @PhilipMay in #141

Full Changelog: 0.10.0...0.10.1rc1

0.10.0

What's Changed

• Add fit and distance flags to TextDistance class for call order checks. by @PhilipMay in #122
• Add ArangoDB utility module. by @PhilipMay in #123
• Add ArangoDB collection backup functionality. by @PhilipMay in #124
• Add Beautiful Soup specific tools. by @PhilipMay in #126
• Update ruff config with "ALL" default. by @PhilipMay in #134
• Add pytest-rerunfailures dependency and mark data tests as flaky. by @PhilipMay in #135
• Add hypothesis tests. by @PhilipMay in #136
• Add XML tag detection function. by @PhilipMay in #139

Full Changelog: 0.9.0...0.10.0

0.9.1rc1

What's Changed

• Add fit and distance flags to TextDistance class for call order checks. by @PhilipMay in #122
• Add ArangoDB utility module. by @PhilipMay in #123
• Add ArangoDB collection backup functionality. by @PhilipMay in #124

Full Changelog: 0.9.0...0.9.1rc1

0.9.0

What's Changed

• Compare data loaders with original implementation. by @PhilipMay in #111
• Add contrib guide. by @PhilipMay in #98
• Fix target-version typo in pyproject.toml. by @PhilipMay in #114
• Update typing in somajo.py to use Literal. by @PhilipMay in #115
• Improve module documentation. by @PhilipMay in #116
• Add TextDistance class for calculating cosine distance between texts. by @PhilipMay in #113
• Fix data loading. by @PhilipMay in #119

Full Changelog: 0.8.1...0.9.0