aetest fails to start dev_appserver.py due to favoring now-unsupported python2.7

[derat]

I'm having trouble with the google.golang.org/appengine/v2/aetest package failing to start dev_appserver.py due to apparent logic to prefer Python 2.7, which seems to be unsupported by dev_appserver.py now.

---

Minimal repro:

% dpkg -l google-cloud-sdk-app-engine-python python2 python3 python-is-python3
...
ii  google-cloud-sdk-app-engine-python 428.0.0-0    all          Python runtime for Google App Engine
ii  python-is-python3                  3.9.2-1      all          symlinks /usr/bin/python to python3
ii  python2                            2.7.18-3     amd64        interactive high-level object-oriented language (Python2 version)
ii  python3                            3.9.2-3      amd64        interactive high-level object-oriented language (default python3 version)

aetest_bug.go:

package main

import (
    "log"
    
    "google.golang.org/appengine/v2/aetest"
)

func main() { 
    inst, err := aetest.NewInstance(&aetest.Options{})
    if err != nil {
        log.Fatal("Failed starting dev_appserver: ", err)
    }
    inst.Close()
}

go.mod (module path is bogus):

module codeberg.org/derat/aetest

go 1.19
    
require google.golang.org/appengine/v2 v2.0.3
    
require github.com/golang/protobuf v1.3.1 // indirect

% go run aetest_bug.go
ERROR: Python 2 is not compatible with the Google Cloud SDK. Please use Python version 3.5 and up.

If you have a compatible Python interpreter installed, you can use it by setting
the CLOUDSDK_PYTHON environment variable to point to it.

2023/05/01 19:23:49 Failed starting dev_appserver: unable to find admin server URL
exit status 1

---

In the aetest package, instance.go seems to contain code that deliberately favors python2.7 over python (which is symlinked to python3 on my system, which is then symlinked to python3.9):

appengine/v2/aetest/instance.go

Lines 169 to 177 in 1bdfe3f

	func findPython() (path string, err error) {
	for _, name := range []string{"python2.7", "python"} {
	path, err = exec.LookPath(name)
	if err == nil {
	return
	}
	}
	return
	}

Should findPython() be updated to only look for Python 3.5 or later, or is the issue that aetest needs to maintain compatibility for old versions of dev_appserver.py that may only work under Python 2.7?

[ludoch]

Sorry about this.
For now, the devappserver is only supported with Python2 and not Python3 yet, but the entire Cloud SDK is moving to Python3 only.

See updated Cloud SDK documentation regarding forcing py27 for the devappserver:
https://cloud.google.com/appengine/docs/legacy/standard/python/tools/using-local-server
Maybe forcing CLOUDSDK_DEVAPPSERVER_PYTHON env variable to your python2 executable will do the trick?

[derat]

I think I may have the opposite problem from what you're describing. My /usr/bin/dev_appserver.py (from google-cloud-sdk-app-engine-python 428.0.0-0, which seems to be the latest version in the Apt repo) seems to require Python 3, not Python 2:

% python2.7 /usr/bin/dev_appserver.py
ERROR: Python 2 is not compatible with the Google Cloud SDK. Please use Python version 3.5 and up.
                                                                              
If you have a compatible Python interpreter installed, you can use it by setting
the CLOUDSDK_PYTHON environment variable to point to it.   
                                                                              
% python3.9 /usr/bin/dev_appserver.py
usage: dev_appserver.py [-h] [-A APP_ID] [--host HOST] [--port PORT]
...

dev_appserver.py works fine on my system when I run it directly (since I'm using Python 3.9 then), but it doesn't work when it's started by aetest, apparently due to the findPython() function that I quoted choosing Python 2.7 instead.

Setting CLOUDSDK_DEVAPPSERVER_PYTHON or CLOUDSDK_PYTHON to contain /usr/bin/python3.9 doesn't seem have any effect in overriding aetest's behavior.

Please let me know if I'm misunderstanding the situation, though.

[derat]

Here's a disgusting hack that works around this by tricking aetest into using Python 3:

package main

import (
    "io/ioutil"
    "log"
    "os"
    "path/filepath"

    "google.golang.org/appengine/v2/aetest"
)

func main() {
    dir, err := ioutil.TempDir("", "aetest.")
    if err != nil {
        log.Fatal(err)
    }
    defer os.RemoveAll(dir)

    if err := os.Symlink("/usr/bin/python3", filepath.Join(dir, "python2.7")); err != nil {
        log.Fatal(err)
    }
    os.Setenv("PATH", dir+":"+os.Getenv("PATH"))

    inst, err := aetest.NewInstance(&aetest.Options{})
    if err != nil {
        log.Fatal("Failed starting dev_appserver: ", err)
    }
    inst.Close()
}

% go run aetest_bug.go
INFO     2023-05-02 00:16:56,880 devappserver2.py:321] Skipping SDK update check.
WARNING  2023-05-02 00:16:56,880 devappserver2.py:337] DEFAULT_VERSION_HOSTNAME will not be set correctly with --port=0
WARNING  2023-05-02 00:16:57,068 simple_search_stub.py:1196] Could not read search indexes from /tmp/appengine.testapp.redacted/search_indexes
INFO     2023-05-02 00:16:57,070 <string>:398] Starting API server at: http://localhost:33449
INFO     2023-05-02 00:16:57,075 dispatcher.py:276] Starting module "default" running at: http://localhost:37253
INFO     2023-05-02 00:16:57,076 admin_server.py:70] Starting admin server at: http://localhost:46745
INFO     2023-05-02 00:16:58,079 stub_util.py:360] Applying all pending transactions and saving the datastore
INFO     2023-05-02 00:16:58,081 stub_util.py:363] Saving search indexes
...

[ludoch]

Sorry, maybe I was not clear enough.
Yes /usr/bin/dev_appserver.py which is a thin wrapper for the cloud SDK needs Python3 as all binaries from cloud SDK added in /usr/bin.
But this wrapper is later executing another "devappserver" process (the one from original standalone GAE SDK python) which stills needs python2
So my point is that CLOUDSDK_DEVAPPSERVER_PYTHON should point to a python2 environment.

Glad you have it working for you now.
Will try to see how to change appengine/v2/aetest/instance.go to avoid the issue...

[derat]

Is there any reason why aetest needs to explicitly choose a Python interpreter instead of just running /usr/bin/dev_appserver.py directly? That seems like it would do the right thing, at least on my system (although maybe the shebang in /usr/bin/dev_appserver.py should also be using python3 now rather than just python).

[derat]

For other users who run into this, an easier workaround to avoid the broken code is to just set APPENGINE_DEV_APPSERVER_BINARY beforehand:

os.Setenv("APPENGINE_DEV_APPSERVER_BINARY", "dev_appserver.py")
inst, err := aetest.NewInstance(&aetest.Options{})

[derat]

I think that this is no longer an issue due to dev_appserver.py having been updated to use Python 3 (at 468, I believe).