Korean Sentence Segmentation

AIHub Korean Dataset
JSON Parsing
Data Cleaning
Regular Expression
TXT Merge
Multiprocessing
Exploratory Data Analysis
- Length of Source List
- The Number of Character
- Capacity
- Preprocessing Runtime Calculator
- Preprocessing Memory & Process & Thread
Future Work
- Dataframe(Pandas or Polars), Dictionary Optimization
- The Searcher of Source JSON
  To compare JSON and TXT extracted from JSON
  File Naming & Storage System (Before & After File Name Match in Excel)
  What's the Source TXT File Name to Each Line in Proprocessed TXT File? (Dataframe)
- Remove Warning kss 3.7.3 Message: "[Korean Sentence Splitter]: Too long text! turn off quotes calibration!"
- Cython Multithreading

Library

kss regex pandas

Name		Name	Last commit message	Last commit date
Latest commit History 108 Commits
first-try		first-try
report		report
LICENSE		LICENSE
README.md		README.md
aihub_json_post_proprecssing.ipynb		aihub_json_post_proprecssing.ipynb
aihub_json_preprocessing_automatic_patent_classification_data.ipynb		aihub_json_preprocessing_automatic_patent_classification_data.ipynb
aihub_json_preprocessing_british_korean_data_on_patents_in_major_countries_linked_to_industrial_information.ipynb		aihub_json_preprocessing_british_korean_data_on_patents_in_major_countries_linked_to_industrial_information.ipynb
aihub_json_preprocessing_document_summary_text.ipynb		aihub_json_preprocessing_document_summary_text.ipynb
aihub_json_preprocessing_general_common_sense.ipynb		aihub_json_preprocessing_general_common_sense.ipynb
aihub_json_preprocessing_korean_corpus_data_based_on_large_scale_purchase_books.ipynb		aihub_json_preprocessing_korean_corpus_data_based_on_large_scale_purchase_books.ipynb
aihub_json_preprocessing_legal_regulations_(such_as_terms_and_conditions_of_judgment)_text_analysis_data.ipynb		aihub_json_preprocessing_legal_regulations_(such_as_terms_and_conditions_of_judgment)_text_analysis_data.ipynb
aihub_json_preprocessing_machine_reading.ipynb		aihub_json_preprocessing_machine_reading.ipynb
aihub_json_preprocessing_machine_reading_data_for_administrative_documents.ipynb		aihub_json_preprocessing_machine_reading_data_for_administrative_documents.ipynb
aihub_json_preprocessing_news_article_machine_reading_data.ipynb		aihub_json_preprocessing_news_article_machine_reading_data.ipynb
aihub_json_preprocessing_professional_corpus.ipynb		aihub_json_preprocessing_professional_corpus.ipynb
aihub_json_preprocessing_reading_books_by_machine.ipynb		aihub_json_preprocessing_reading_books_by_machine.ipynb
aihub_json_preprocessing_summary_and_report_generation_data.ipynb		aihub_json_preprocessing_summary_and_report_generation_data.ipynb
aihub_json_preprocessing_summary_of_book_materials.ipynb		aihub_json_preprocessing_summary_of_book_materials.ipynb
aihub_json_preprocessing_summary_of_thesis_materials.ipynb		aihub_json_preprocessing_summary_of_thesis_materials.ipynb
aihub_json_preprocessing_web_data_based_korean_corpus_data.ipynb		aihub_json_preprocessing_web_data_based_korean_corpus_data.ipynb
aihub_txt_delete_duplicate_sentence.ipynb		aihub_txt_delete_duplicate_sentence.ipynb
aihub_txt_merge.ipynb		aihub_txt_merge.ipynb
data_preprocessing.py		data_preprocessing.py
extract_source_text.py		extract_source_text.py
reading_data.py		reading_data.py
sentence_segmentation.py		sentence_segmentation.py
txt_preprocessing.py		txt_preprocessing.py

License

minseok0809/korean-sentence-segementation

Folders and files

Latest commit

History

Repository files navigation

Korean Sentence Segmentation

Library

About

Topics

Resources

License

Stars

Watchers

Forks

Languages