[abctl] not exposing the Airbyte API (on port 8001)

[dschiavu]

What happened?

Hi,

I'm trying out Airbyte OSS via abctl by following these instructions and I have noticed this version of Airbyte OSS does not expose the Airbyte API port 8001 to the Docker host - see https://github.com/airbytehq/abctl/blob/19f1e0567c82b046643db87388048e6c270d5bd0/internal/local/k8s/cluster.go#L118

I this by design, or?

Also, I've found this piece of documentation confusing (scroll down to Kubernetes) as it's unclear whether it refers to an actual local Kubernetes server, or abctl's custom kind-based K8S distribution (which runs Kubernetes within a single Docker container).

I see the only alternative, if one wants to use the API locally, is to run Airbyte OSS via Docker Compose.

Thanks,
Dan

What did you expect to happen?

Use Airbyte OSS's API locally

Abctl Version

$ abctl version
version: v0.3.2
modified: false

Docker Version

$ docker version
Client:
 Cloud integration: v1.0.35+desktop.13
 Version:           26.1.1
 API version:       1.45
 Go version:        go1.21.9
 Git commit:        4cf5afa
 Built:             Tue Apr 30 11:44:56 2024
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.30.0 (149282)
 Engine:
  Version:          26.1.1
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.9
  Git commit:       ac2de55
  Built:            Tue Apr 30 11:48:04 2024
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.31
  GitCommit:        e377cd56a71523140ca6ae87e30244719194a521
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

OS Version

$ uname -a
Darwin dmac395 23.4.0 Darwin Kernel Version 23.4.0: Fri Mar 15 00:12:49 PDT 2024; root:xnu-10063.101.17~1/RELEASE_ARM64_T6020 arm64

[dschiavu]

Just to add this, I can see the airbyte-server-api pod is running inside the kind K8S cluster, so there shouldn't be any reason not to expose the Airbyte API port to the host, unless I'm missing something. Thanks.

$ docker exec -it airbyte-abctl-control-plane crictl images
IMAGE                                                TAG                            IMAGE ID            SIZE
docker.io/airbyte/airbyte-api-server                 0.59.0                         eeaddaf01b715       416MB
docker.io/airbyte/bootloader                         0.59.0                         4aacb1a88ae79       478MB
docker.io/airbyte/connector-builder-server           0.59.0                         049690bcdce1a       1.15GB
docker.io/airbyte/cron                               0.59.0                         772b79d844ec6       497MB
docker.io/airbyte/db                                 0.59.0                         9ab17b3b84b17       91.5MB
docker.io/airbyte/mc                                 latest                         257a084c4a61b       13.3MB
docker.io/airbyte/server                             0.59.0                         011c0fd08770f       501MB
docker.io/airbyte/webapp                             0.59.0                         ac95237c5d712       27.7MB
docker.io/airbyte/worker                             0.59.0                         d197c0ea1d0f3       630MB
docker.io/bitnami/kubectl                            1.28.9                         cc7e898349f4b       104MB
docker.io/kindest/kindnetd                           v20240202-8f1494ea             4740c1948d3fc       25.3MB
docker.io/kindest/local-path-helper                  v20230510-486859a6             d022557af8b63       2.92MB
docker.io/kindest/local-path-provisioner             v20240202-8f1494ea             6c806bad1e899       17.3MB
docker.io/minio/minio                                RELEASE.2023-11-20T22-40-07Z   a007889fa308a       48.7MB
docker.io/temporalio/auto-setup                      1.20.1                         bdbc6025e069e       136MB
registry.k8s.io/coredns/coredns                      v1.11.1                        2437cf7621777       16.5MB
registry.k8s.io/kube-apiserver-arm64                 v1.29.1                        16389be12cae2       83.3MB
registry.k8s.io/kube-apiserver                       v1.29.1                        16389be12cae2       83.3MB
registry.k8s.io/kube-controller-manager-arm64        v1.29.1                        22c006523d934       77.7MB
registry.k8s.io/kube-controller-manager              v1.29.1                        22c006523d934       77.7MB
registry.k8s.io/kube-scheduler-arm64                 v1.29.1                        a999c7253375f       59.1MB
registry.k8s.io/kube-scheduler                       v1.29.1                        a999c7253375f       59.1MB
registry.k8s.io/ingress-nginx/controller             <none>                         adf677c1adaa5       99.9MB
registry.k8s.io/ingress-nginx/kube-webhook-certgen   <none>                         296b5f799fcd8       22.3MB
registry.k8s.io/etcd                                 3.5.10-0                       79f8d13ae8b88       65.2MB
registry.k8s.io/kube-proxy-arm64                     v1.29.1                        289db618fddf6       86.6MB
registry.k8s.io/kube-proxy                           v1.29.1                        289db618fddf6       86.6MB
registry.k8s.io/pause                                3.7                            e5a475a038057       268kB

[marcosmarxm]

Did you try to expose the API using the Kubernetes instructions?

[dschiavu]

These instructions?

Kubernetes

If you are running an instance of Airbyte locally using kubernetes, you can access the Airbyte API of the local instance by:

Setting up a port forward to the airbyte-server kube svc by running kubectl port-forward svc/airbyte-server-svc 8001:80 &
Making a call to http://localhost:8001/api/public/v1/ or the health endpoint at http://localhost:8001/api/public/health
OAuth endpoints and workspace updates are not supported in OSS currently.

Yes, I did, and the kubectl commands were erroring out on my machine.

I don't have the error messages at hand now since I've moved on to launching Airbyte via Docker Compose, but they were something along the svc/airbyte-server-svc service not existing in Kubernetes, so it couldn't create the port-forward. Indeed, when listing the K8S services, the service wasn't there.

[marcosmarxm]

I wasn't able to reproduce the issue @dschiavu and it look you can't now as moved to Docker compose. I'll close the issue, if you have the same please open a new one.