Skip to content

Latest commit

 

History

History
7 lines (3 loc) · 192 Bytes

ARCHITECTURE.md

File metadata and controls

7 lines (3 loc) · 192 Bytes
Raw

Architecture

This document explains the what and why behind the full tokenization process. You can also check example.py for a comprehensive implementation of the process.

Overview