PyBuilder Example - Producer/Consumer for Kinesis Data Streams

Description

kinesis data streams 관련 CLI(Command-line Instruction) code snippets

Kinesis Streams Consumer

kinesis data streams 에서 실시간으로 데이터를 읽어오는 CLI

$ python src/main/python/consumer/kinesis_consumer.py --help
usage: kinesis_consumer.py [-h] [--stream-name STREAM_NAME]
                           [--iter-type {TRIM_HORIZON,LATEST}]
                           [--region-name REGION_NAME]

optional arguments:
  -h, --help            show this help message and exit
  --stream-name STREAM_NAME
                        kinesis stream-name
  --iter-type {TRIM_HORIZON,LATEST}
                        kinesis stream shard iterator type: [TRIM_HORIZON,
                        LATEST]
  --region-name REGION_NAME
                        aws region name (default: us-east-1)

Kinesis Streams Producers

kinesis data streams 에서 실시간으로 데이터를 넣는 CLI

Name	Description
sentences	임의의 문장 생성
words	임의의 단어와 발생 횟수(1) 튜플 생성 (예: ("hello", 1)
stocks	임의의 주식 정보 생성

예를 들어, 임의의 문장을 생성해서 Kinesis Data Streams에 넣으려면, 다음과 같이 AWS_REGION_NAME 과 KINESIS_STREAM_NAME 환경 변수를 설정하고, sentences.py 스크립트를 실행한다.

$ export AWS_REGION_NAME='us-east-1'
$ export KINESIS_STREAM_NAME='sentences-streams'
$ python sentences.py

Build & Deploy

PyBuilder를 이용해서 전체 코드 조각들을 mono repository로 관리

Useful PyBuilder commands

pyb -t: 전체 작업 목록을 표시한다.
pyb checkstyle: pylint를 이용해서 code style을 검사한다.
pyb package: 배포한 소스 코드나 resources, script 파일들을 배포 준비 디렉터리(target)에 패키징한다.
pyb publish: 패키징된 소스 코드와 resources, script 파일들을 배포한다.
pyb clean: 배포 준비 디렉터리(target)의 파일들을 삭제한다.

git clone 명령어를 실행해서 소스 코드를 다운로드 한 후에, 아래와 같이 python virtualenv 환경을 생성하고, 필요한 python 패키지를 설치한다.

$ python3 -m venv kinesis-streams-samples
$ source bin/activate
$ pip install - requirements.txt

전체 작업(Task)을 pyb -t 또는 pyb --list-tasks 를 이용해서 확인 할 수 있다.

$ pyb -t
Tasks found for project "kinesis-streams-samples":
                         analyze - <no description available>
                      checkstyle - check code style using pylint
                                   depends on tasks: prepare
                           clean - Cleans the generated output.
                 compile_sources - Compiles source files that need compilation.
                                   depends on tasks: prepare
                         install - Installs the published project.
                                   depends on tasks: package publish(optional)
      install_build_dependencies - Installs all build dependencies specified in the build descriptor
                                   depends on tasks: prepare
            install_dependencies - Installs all (both runtime and build) dependencies specified in the build descriptor
                                   depends on tasks: prepare
    install_runtime_dependencies - Installs all runtime dependencies specified in the build descriptor
                                   depends on tasks: prepare
                kinesis_consumer - kinesis data streams consumer
               kinesis_producers - kinesis data streams producers
         kinesis_stocks_producer - kinesis data streams producer: stocks
               list_dependencies - Displays all dependencies the project requires
                         package - Packages the application. Package a python application.
                                   depends on tasks: compile_sources run_unit_tests(optional)
                         prepare - Prepares the project for building. Creates target VEnvs
               print_module_path - Print the module path.
              print_scripts_path - Print the script path.
                         publish - Publishes the project.
                                   depends on tasks: package verify(optional)
           run_integration_tests - Runs integration tests on the packaged application.
                                   depends on tasks: package
                  run_unit_tests - Runs all unit tests.
                                   depends on tasks: compile_sources
                          verify - Verifies the project and possibly integration tests.
                                   depends on tasks: run_integration_tests(optional)

pyb checkstyle로 pylint를 이용해서 소스 코드의 coding style을 검사할 수 있다.

$ pyb kinesis_consumer checkstyle
PyBuilder version 0.12.7
Build started at 2020-08-13 01:45:14
------------------------------------------------------------
[INFO]  Building kinesis-streams-samples version 1.0.dev0
[INFO]  Executing build in kinesis-streams-samples
[INFO]  Going to execute tasks: kinesis_consumer, checkstyle
[INFO]  Creating target 'build' VEnv in 'kinesis-streams-samples/target/venv/build/cpython-3.6.9.final.0'
[INFO]  Processing dependency packages 'pylint~=2.5.0' to be installed with {'upgrade': True}
[INFO]  Creating target 'test' VEnv in 'kinesis-streams-samples/target/venv/test/cpython-3.6.9.final.0'
[INFO]  Executing pylint on project sources.
[WARN]
--------------------------------------------------------------------
Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00)


[INFO]  Pass (score=10.00, cutoff=8.50)
------------------------------------------------------------
BUILD SUCCESSFUL
------------------------------------------------------------
Build Summary
             Project: kinesis-streams-samples
             Version: 1.0.dev0
      Base directory: kinesis-streams-samples
        Environments:
               Tasks: kinesis_consumer [0 ms] prepare [5026 ms] checkstyle [856 ms]
Build finished at 2020-08-13 01:45:23
Build took 8 seconds (8941 ms)

배포할 소스 코드 및 Resources, Script 등을 pyb package 를 이용해서 배포 준비 디렉터리(target)에 패키징 할 수 있다.

$ pyb kinesis_consumer package
PyBuilder version 0.12.7
Build started at 2020-08-13 01:25:19
------------------------------------------------------------
[INFO]  Building kinesis-streams-samples version 1.0.dev0
[INFO]  Executing build in kinesis-streams-samples
[INFO]  Going to execute tasks: kinesis_consumer, package
[INFO]  Creating target 'build' VEnv in 'kinesis-streams-samples/target/venv/build/cpython-3.6.9.final.0'
[INFO]  Processing dependency packages 'pylint~=2.5.0' to be installed with {'upgrade': True}
[INFO]  Creating target 'test' VEnv in 'kinesis-streams-samples/target/venv/test/cpython-3.6.9.final.0'
[INFO]  Building distribution in kinesis-streams-samples/target/dist/kinesis-streams-consumer
[INFO]  Copying scripts to kinesis-streams-samples/target/dist/kinesis-streams-consumer
------------------------------------------------------------
BUILD SUCCESSFUL
------------------------------------------------------------
Build Summary
             Project: kinesis-streams-samples
             Version: 1.0.dev0
      Base directory: kinesis-streams-samples
        Environments:
               Tasks: kinesis_consumer [0 ms] prepare [5026 ms] compile_sources [0 ms] run_unit_tests [0 ms] package [0 ms]
Build finished at 2020-08-13 01:25:27
Build took 8 seconds (8084 ms)

배포할 소스 코드 및 Resources, Script 등을 패키징해서 pyb publish로 배포할 수 있다.

$ pyb kinesis_consumer publish
PyBuilder version 0.12.7
Build started at 2020-08-13 01:33:55
------------------------------------------------------------
[INFO]  Building kinesis-streams-samples version 1.0.dev0
[INFO]  Executing build in kinesis-streams-samples
[INFO]  Going to execute tasks: kinesis_consumer, publish
[INFO]  Creating target 'build' VEnv in 'kinesis-streams-samples/target/venv/build/cpython-3.6.9.final.0'
[INFO]  Processing dependency packages 'pylint~=2.5.0' to be installed with {'upgrade': True}
[INFO]  Creating target 'test' VEnv in 'kinesis-streams-samples/target/venv/test/cpython-3.6.9.final.0'
[INFO]  Building distribution in kinesis-streams-samples/target/dist/kinesis-streams-consumer
[INFO]  Copying scripts to kinesis-streams-samples/target/dist/kinesis-streams-consumer
------------------------------------------------------------
BUILD SUCCESSFUL
------------------------------------------------------------
Build Summary
             Project: kinesis-streams-samples
             Version: 1.0.dev0
      Base directory: kinesis-streams-samples
        Environments:
               Tasks: kinesis_consumer [0 ms] prepare [5863 ms] compile_sources [0 ms] run_unit_tests [0 ms] package [0 ms] run_integration_tests [0 ms] verify [0 ms] publish [0 ms]
Build finished at 2020-08-13 01:34:04
Build took 8 seconds (8921 ms)

모든 배포 관련 작업이 완료되면, pyb clean 을 이용해서 배포 준비 디렉터리(target)의 내용을 삭제할 수 있다.

$ pyb clean
PyBuilder version 0.12.7
Build started at 2020-08-13 01:27:36
------------------------------------------------------------
[INFO]  Building kinesis-streams-samples version 1.0.dev0
[INFO]  Executing build in kinesis-streams-samples
[INFO]  Going to execute task clean
[INFO]  Removing target directory kinesis-streams-samples/target
------------------------------------------------------------
BUILD SUCCESSFUL
------------------------------------------------------------
Build Summary
             Project: kinesis-streams-samples
             Version: 1.0.dev0
      Base directory: kinesis-streams-samples
        Environments:
               Tasks: clean [49 ms]
Build finished at 2020-08-13 01:27:39
Build took 3 seconds (3139 ms)

Name		Name	Last commit message	Last commit date
Latest commit History 12 Commits
src/main/python		src/main/python
.gitignore		.gitignore
.pylintrc		.pylintrc
README.md		README.md
build.py		build.py
pyproject.toml		pyproject.toml
requirements.txt		requirements.txt

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

src/main/python

src/main/python

.gitignore

.gitignore

.pylintrc

.pylintrc

README.md

README.md

build.py

build.py

pyproject.toml

pyproject.toml

requirements.txt

requirements.txt

Repository files navigation

PyBuilder Example - Producer/Consumer for Kinesis Data Streams

Description

Kinesis Streams Consumer

Kinesis Streams Producers

Build & Deploy

Useful PyBuilder commands

About

Releases

Packages

Languages

ksmin23/kinesis-streams-samples

Folders and files

Latest commit

History

Repository files navigation

PyBuilder Example - Producer/Consumer for Kinesis Data Streams

Description

Kinesis Streams Consumer

Kinesis Streams Producers

Build & Deploy

Useful PyBuilder commands

About

Topics

Resources

Stars

Watchers

Forks

Languages