move source code into localstack-core #10800

thrau · 2024-05-09T14:32:41Z

Motivation

This is the first transitional step into a multi-distribution + namespace package localstack setup. Before we introduce multiple distributions and a namespace package, we can move the localstack source code into localstack-core, and use the src layout to keep everything backwards compatible, while at the same time committing one of the biggest involved changes, which is mkdir localstack-core && mv localstack localstack-core.

Once we introduce multiple distribution, each distribution will have a flat layout again, but that requires restructuring of the entire project setup.

Find all relevant information on how to adopt the change in #10860

Changes

the localstack source code (localstack package) now lives in ./localstack-core rather than ./. In the container, it now moved from /opt/code/localstack/localstack to /opt/code/localstack/localstack-core/localstack

Merging the PR

I would like to manually merge this PR because i'd like the file move and config updates separate from each other. this will make it easier to understand in the history, potentially make rebasing easier, and also allow us to clearly ignore the renames (although they should be by default by git).

turns out this was not really needed so i just squash merged

TODO

What's left to do:

github-actions · 2024-05-15T11:43:37Z

S3 Image Test Results (AMD64 / ARM64)

2 files ±0 2 suites ±0 3m 10s ⏱️ -10s
400 tests ±0 348 ✅ ±0 52 💤 ±0 0 ❌ ±0
800 runs ±0 696 ✅ ±0 104 💤 ±0 0 ❌ ±0

Results for commit d5a3fea. ± Comparison against base commit b8290ff.

♻️ This comment has been updated with latest results.

github-actions · 2024-05-15T12:48:18Z

LocalStack Community integration with Pro

2 files ±0 2 suites ±0 1h 41m 19s ⏱️ -54s
2 979 tests ±0 2 674 ✅ ±0 305 💤 ±0 0 ❌ ±0
2 981 runs ±0 2 674 ✅ ±0 307 💤 ±0 0 ❌ ±0

Results for commit 52fbeec. ± Comparison against base commit 6eeb7cc.

♻️ This comment has been updated with latest results.

dominikschubert · 2024-05-16T05:39:42Z

.circleci/config.yml

@@ -134,7 +134,7 @@ jobs:
                    circleci-agent step halt
                  else
                    source .venv/bin/activate
-                    python -m localstack.testing.testselection.scripts.generate_test_selection_from_pr /tmp/workspace/repo $CI_PULL_REQUEST target/testselection/test-selection.txt
+                    PYTHONPATH=localstack-core python -m localstack.testing.testselection.scripts.generate_test_selection_from_pr /tmp/workspace/repo $CI_PULL_REQUEST target/testselection/test-selection.txt


This should be removed after #10830 has been merged, right?

no this has nothing to do with the plugin loading, but just the src layout.

without installing localstack with pip install -e . into the activated venv, the python binary cannot find the source code when a src layout is used.
another option is to cd into localstack-core, or move test selection scripts out of ls all together

dominikschubert · 2024-05-16T05:43:21Z

@thrau

collecting a few of my findings below, will update with more when I encounter them

ASF generation is broken `FileNotFoundError: [Errno 2] No such file or directory: 'localstack/aws/api/stepfunctions'
CloudFormation resource provider generation script is broken and needs to be adjusted since it assumes a few things around the localstack/localstack-ext directory setup
- this is only used manually, so I can just fix that after the merge

alexrashed

Awesome! This is going to be a great step towards a proper structure and code separation between the different areas / hierarchies in the codebase!
The changes themselves are looking great, and the separation in the two commits is super-helpful (so +1 for the manual merge of the PR to avoid a squash commit).
The changes right now are nearly covering all cases, I just found a few small things that still need to be adjusted and some thins we could run to find potential integration issues early:

The S3 workflow trigger needs to be adjusted (hasn't been changed, so I cannot comment on it in the PR diff view):

localstack/.github/workflows/tests-s3-image.yml

Lines 7 to 15 in e69d60f

    
           - localstack/aws/*.py 
        
           - localstack/aws/handlers/*¨ 
        
           - localstack/aws/protocol/** 
        
           - localstack/aws/serving/** 
        
           - localstack/aws/api/s3/** 
        
           - localstack/http/** 
        
           - localstack/runtime/** 
        
           - localstack/services/s3/** 
        
           - localstack/*.py

The a path in a step in the ASF update action needs to be updated (see comment).
The test runs are currently testing the integration using the "companion branch detection". Maybe it would be useful to just create a dev release from this commit before actually merging it? With this published dev release we could trigger the different pipelines in our downstream projects, testing the integration. If we find issues we can fix them without any interruptions (we can just directly yank or delete the dev release on PyPi).

Let me know if I can help anywhere along the way! 🚀

alexrashed · 2024-05-20T12:52:34Z

localstack-core/localstack/aws/scaffold.py

@@ -525,7 +527,7 @@ def create_code_directory(service_name: str, code: str, base_path: str):
 @click.option("--doc/--no-doc", default=False, help="whether or not to generate docstrings")
 @click.option(
    "--path",
-    default="./localstack/aws/api",
+    default="./localstack-core/localstack/aws/api",


Unfortunately, this change isn't enough to fix the ASF update action...
I did a test-run of the pipeline on this branch: https://github.com/localstack/localstack/actions/runs/9158663716/job/25177480610
The run is looking good, but the change detection is broken:

fatal: ambiguous argument 'localstack/aws/api/': unknown revision or path not in the working tree.

This should be easily fixed by adjusting the paths in this job step:

localstack/.github/workflows/asf-updates.yml

Lines 54 to 67 in fa4ab5a

- name: Check for changes

id: check-for-changes

run: |

# Check if there are changed files and store the result in target/diff-check.log

# Check against the PR branch if it exists, otherwise against the master

# Store the result in target/diff-check.log and store the diff count in the GitHub Action output "diff-count"

mkdir -p target

(git diff --name-only origin/asf-auto-updates localstack/aws/api/ 2>/dev/null || git diff --name-only origin/master localstack/aws/api/ 2>/dev/null) | tee target/diff-check.log

echo "diff-count=$(cat target/diff-check.log | wc -l)" >> $GITHUB_OUTPUT

# Store a (multiline-sanitized) list of changed services (compared to the master) in the GitHub Action output "changed-services"

echo "changed-services<<EOF" >> $GITHUB_OUTPUT

echo "$(git diff --name-only origin/master localstack/aws/api/ | sed 's#localstack/aws/api/#- #g' | sed 's#/__init__.py##g' | sed 's/_/-/g')" >> $GITHUB_OUTPUT

echo "EOF" >> $GITHUB_OUTPUT

thrau changed the title ~~Move source code~~ move source code into localstack-core May 9, 2024

thrau force-pushed the move-source-code branch from 3747957 to 0e55845 Compare May 9, 2024 17:31

alexrashed mentioned this pull request May 14, 2024

add compatibility for editable wheels and src layout localstack/plux#20

Merged

thrau force-pushed the move-source-code branch 2 times, most recently from 56f75ac to 25f59c4 Compare May 14, 2024 23:15

thrau added the semver: minor Non-breaking changes which can be included in minor releases, but not in patch releases label May 15, 2024

thrau mentioned this pull request May 15, 2024

update plux to 1.10.0 #10830

Merged

dominikschubert reviewed May 16, 2024

View reviewed changes

thrau force-pushed the move-source-code branch from 4bd2138 to f833a91 Compare May 16, 2024 20:20

thrau added a commit that referenced this pull request May 16, 2024

move localstack source code to localstack-core module (#10800)

b5633dc

thrau added a commit that referenced this pull request May 16, 2024

configure project to work with src layout (#10800)

22d4d0f

thrau force-pushed the move-source-code branch from f833a91 to 22d4d0f Compare May 16, 2024 23:30

thrau added a commit that referenced this pull request May 18, 2024

move localstack source code to localstack-core module (#10800)

730b2ad

thrau added a commit that referenced this pull request May 18, 2024

configure project to work with src layout (#10800)

a7dea96

thrau force-pushed the move-source-code branch from 22d4d0f to a7dea96 Compare May 18, 2024 15:41

thrau added a commit that referenced this pull request May 18, 2024

move localstack source code to localstack-core module (#10800)

ce88415

thrau added a commit that referenced this pull request May 18, 2024

configure project to work with src layout (#10800)

990e8c2

thrau force-pushed the move-source-code branch from a7dea96 to 990e8c2 Compare May 18, 2024 15:49

thrau added a commit that referenced this pull request May 18, 2024

move localstack source code to localstack-core module (#10800)

a02eb69

thrau added a commit that referenced this pull request May 18, 2024

configure project to work with src layout (#10800)

776b5a7

thrau force-pushed the move-source-code branch from 990e8c2 to 776b5a7 Compare May 18, 2024 15:53

thrau added a commit that referenced this pull request May 19, 2024

configure project to work with src layout (#10800)

64d41ac

thrau force-pushed the move-source-code branch from 776b5a7 to 64d41ac Compare May 19, 2024 12:13

thrau added a commit that referenced this pull request May 19, 2024

configure project to work with src layout (#10800)

02c26d9

thrau force-pushed the move-source-code branch from 64d41ac to 02c26d9 Compare May 19, 2024 13:21

thrau added a commit that referenced this pull request May 19, 2024

configure project to work with src layout (#10800)

52fbeec

thrau force-pushed the move-source-code branch from 02c26d9 to 52fbeec Compare May 19, 2024 14:12

thrau requested review from Morijarti and cloutierMat as code owners May 19, 2024 22:33

thrau requested review from dominikschubert and removed request for simonrw, Morijarti, viren-nadkarni, steffyP, ackdav, giograno, joe4dev, silv-io, macnev2013, dfangl, pinzon, sannya-singal, bentsku, MEPalma, cloutierMat and baermat May 19, 2024 22:34

alexrashed reviewed May 20, 2024

View reviewed changes

dominikschubert mentioned this pull request May 21, 2024

Internal Refactoring: Towards a Multi-Distribution Setup #10860

Open

alexrashed mentioned this pull request May 23, 2024

extract docker build / push make targets to helper script #10872

Open

4 tasks

move localstack source code to localstack-core module (#10800)

e92e255

thrau added a commit that referenced this pull request May 24, 2024

configure project to work with src layout (#10800)

9659aa1

thrau force-pushed the move-source-code branch from 52fbeec to 9659aa1 Compare May 24, 2024 14:31

configure project to work with src layout (#10800)

d5a3fea

thrau force-pushed the move-source-code branch from 9659aa1 to d5a3fea Compare May 24, 2024 15:03

thrau merged commit 958890a into master May 24, 2024
29 of 35 checks passed

thrau deleted the move-source-code branch May 24, 2024 15:16

thrau mentioned this pull request May 24, 2024

fix healthcheck in dockerfile #10894

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

move source code into localstack-core #10800

move source code into localstack-core #10800

thrau commented May 9, 2024 •

edited

github-actions bot commented May 15, 2024 •

edited

github-actions bot commented May 15, 2024 •

edited

dominikschubert May 16, 2024

thrau May 16, 2024

dominikschubert commented May 16, 2024 •

edited

alexrashed left a comment

alexrashed May 20, 2024

	- localstack/aws/*.py
	- localstack/aws/handlers/*¨
	- localstack/aws/protocol/**
	- localstack/aws/serving/**
	- localstack/aws/api/s3/**
	- localstack/http/**
	- localstack/runtime/**
	- localstack/services/s3/**
	- localstack/*.py

	- name: Check for changes
	id: check-for-changes
	run: \|
	# Check if there are changed files and store the result in target/diff-check.log
	# Check against the PR branch if it exists, otherwise against the master
	# Store the result in target/diff-check.log and store the diff count in the GitHub Action output "diff-count"
	mkdir -p target
	(git diff --name-only origin/asf-auto-updates localstack/aws/api/ 2>/dev/null \|\| git diff --name-only origin/master localstack/aws/api/ 2>/dev/null) \| tee target/diff-check.log
	echo "diff-count=$(cat target/diff-check.log \| wc -l)" >> $GITHUB_OUTPUT

	# Store a (multiline-sanitized) list of changed services (compared to the master) in the GitHub Action output "changed-services"
	echo "changed-services<<EOF" >> $GITHUB_OUTPUT
	echo "$(git diff --name-only origin/master localstack/aws/api/ \| sed 's#localstack/aws/api/#- #g' \| sed 's#/__init__.py##g' \| sed 's/_/-/g')" >> $GITHUB_OUTPUT
	echo "EOF" >> $GITHUB_OUTPUT

move source code into localstack-core #10800

move source code into localstack-core #10800

Conversation

thrau commented May 9, 2024 • edited

Motivation

Changes

Merging the PR

TODO

github-actions bot commented May 15, 2024 • edited

S3 Image Test Results (AMD64 / ARM64)

github-actions bot commented May 15, 2024 • edited

LocalStack Community integration with Pro

dominikschubert May 16, 2024

Choose a reason for hiding this comment

thrau May 16, 2024

Choose a reason for hiding this comment

dominikschubert commented May 16, 2024 • edited

alexrashed left a comment

Choose a reason for hiding this comment

alexrashed May 20, 2024

Choose a reason for hiding this comment

thrau commented May 9, 2024 •

edited

github-actions bot commented May 15, 2024 •

edited

github-actions bot commented May 15, 2024 •

edited

dominikschubert commented May 16, 2024 •

edited