docs: Add Python SDK docs

This commit adds Python SDK documentation and a new set of nodes to the
navigation tree, It adds the Python SDK introduction, installation steps,
get started guide and additional guide. It adds the code for the guides
as individual code snippets. The additional guide provides a multi-build
example similar to the one used for the Go SDK.

Signed-off-by: Vikram Vaswani <vikram@dagger.io>

docs/current/162770-faq.md

@@ -6,7 +6,7 @@ slug: /162770/faq
 
 ### What language SDKs are available for Dagger?
 
-We currently offer a technical preview of the [Go SDK](/sdk/go) as well as a [CUE SDK](/sdk/cue). Waiting for your favorite language to be supported? [Let us know which one](https://blocklayer.typeform.com/to/a6m5gKSS), and we'll notify you when it's ready.
+We currently offer technical previews of a [Go SDK](/sdk/go) and a [Python SDK](/sdk/python) as well as a [CUE SDK](/sdk/cue). Waiting for your favorite language to be supported? [Let us know which one](https://blocklayer.typeform.com/to/a6m5gKSS), and we'll notify you when it's ready.
 
 ### I am stuck. How can I get help?
 

docs/current/543069-index.md

@@ -26,10 +26,10 @@ Dagger executes your pipelines entirely as [standard OCI containers](https://op
 
 Dagger may be a good fit if you are...
 
-* A developer wishing your CI pipelines were code instead of YAML
-* Your team's "designated devops person", hoping to replace a pile of artisanal scripts with something more powerful
-* A platform engineer writing custom tooling, with the goal of unifying continuous delivery across organizational silos
-* A cloud-native developer advocate or solutions engineer, looking to demonstrate a complex integration on short notice
+* A developer wishing your CI pipelines were code instead of YAML.
+* Your team's "designated devops person", hoping to replace a pile of artisanal scripts with something more powerful.
+* A platform engineer writing custom tooling, with the goal of unifying continuous delivery across organizational silos.
+* A cloud-native developer advocate or solutions engineer, looking to demonstrate a complex integration on short notice.
 
 ## How does it work?
 
@@ -72,7 +72,9 @@ To get started with Dagger, simply choose a SDK, then follow that SDK's getting
 | If you are... | then you should... |
 | -- | -- |
 | a Go developer | Use the [Go SDK](sdk/go) |
+| a Python developer | Use the [Python SDK](sdk/python) |
 | looking for an excuse to learn Go | Use the [Go SDK](sdk/go) |
+| looking for an excuse to learn Python | Use the [Python SDK](sdk/python) |
 | waiting for your favorite language to be supported | [Let us know which one](https://blocklayer.typeform.com/to/a6m5gKSS), and we'll notify you when it is ready
 | a fan of the [CUE](https://cuelang.org) language | Use the [Dagger CUE SDK](sdk/cue) |
 | enjoying the declarative nature of YAML, but wish it were more powerful | Take a look at the [CUE language](https://cuelang.org). If you like what you see, the [CUE SDK](sdk/cue) may be a good fit. |

docs/current/sdk/python/234291-guides.md

@@ -0,0 +1,7 @@
+---
+slug: /sdk/python/234291/guides
+---
+
+# Guides
+
+- [Create a Multi-Build CI Pipeline](./guides/648384-multi-builds.md)

docs/current/sdk/python/459708-index.md

@@ -0,0 +1,68 @@
+---
+slug: /sdk/python
+---
+
+# Dagger Python SDK
+
+<div class="status-badge">Technical Preview</div>
+
+## What is the Dagger Python SDK?
+
+The Dagger Python SDK contains everything you need to develop CI/CD pipelines in Python, and run them on any OCI-compatible container runtime.
+
+Currently the Python SDK consists of:
+
+* A Python package
+* This documentation
+
+## Who is it for?
+
+The Dagger Python SDK may be a good fit if you are...
+
+* A Python developer wishing your CI pipelines were Python code instead of YAML.
+* A developer who needs CI/CD, and is looking for an excuse to learn Python.
+* Your team's "designated devops person", hoping to replace a pile of artisanal scripts with something more powerful.
+* A platform engineer writing custom Python tooling, with the goal of unifying continuous delivery across organizational silos.
+* A data engineer looking to better integrate with your organization's CI/CD or MLOps pipelines.
+
+The Dagger Python SDK may *not* be a good fit if you are...
+
+* A developer who doesn't know Python, and is not interested in learning it.
+* Someone who loves writing YAML all day, thank you very much.
+* A container skeptic: the less containers are involved, the happier you are.
+
+## How does it work?
+
+```mermaid
+graph LR;
+
+subgraph program["Your Python program"]
+  lib["Python package"]
+end
+
+engine["Dagger Engine"]
+
+oci["OCI container runtime"]
+subgraph A["your build pipeline"]
+  A1[" "] -.-> A2[" "] -.-> A3[" "]
+end
+subgraph B["your test pipeline"]
+  B1[" "] -.-> B2[" "] -.-> B3[" "] -.-> B4[" "]
+end
+subgraph C["your deployment pipeline"]
+  C1[" "] -.-> C2[" "] -.-> C3[" "] -.-> C4[" "]
+end
+
+lib -..-> engine -..-> oci -..-> A1 & B1 & C1
+```
+
+1. Your Python program imports the Dagger Python package.
+2. Using the Python package, your program opens a new session to a Dagger Engine: either by connecting to an existing engine, or by provisioning one on-the-fly.
+3. Using the Python package, your program prepares API requests describing pipelines to run, then sends them to the engine. The wire protocol used to communicate with the engine is private and not yet documented, but this will change in the future. For now, the Python package is the only documented API available to your program.
+4. When the engine receives an API request, it computes a [Directed Acyclic Graph (DAG)](https://en.wikipedia.org/wiki/Directed_acyclic_graph) of low-level operations required to compute the result, and starts processing operations concurrently.
+5. When all operations in the pipeline have been resolved, the engine sends the pipeline result back to your program.
+6. Your program may use the pipeline's result as input to new pipelines.
+
+## Get started
+
+To learn more, [install the Python SDK](./866944-install.md) and [start using it](./628797-get-started.md).

docs/current/sdk/python/628797-get-started.md

@@ -0,0 +1,160 @@
+---
+slug: /sdk/python/628797/get-started
+---
+
+# Get Started with the Dagger Python SDK
+
+## Introduction
+
+This tutorial teaches you the basics of using Dagger in Python. You will learn how to:
+
+- Install the Python SDK
+- Create a Python CI tool to test an application
+- Improve the Python CI tool to test the application against multiple Python versions
+
+## Requirements
+
+This tutorial assumes that:
+
+- You have a basic understanding of the Python programming language. If not, [read the Python tutorial](https://www.python.org/about/gettingstarted/).
+- You have a Python development environment with Python 3.10 or later. If not, install [Python](https://www.python.org/downloads/).
+- You have Docker installed and running on the host system. If not, [install Docker](https://docs.docker.com/engine/install/).
+- You have a Python application with tests defined and in a [virtual environment](https://packaging.python.org/en/latest/tutorials/installing-packages/#creating-virtual-environments).
+
+:::note
+This tutorial creates a CI tool to test your Python application against multiple Python versions. If you don't have a Python application already, clone an existing Python project with a well-defined test suite before proceeding. A good example is the [FastAPI](https://github.com/tiangolo/fastapi) library, which you can clone as below:
+
+```shell
+git clone https://github.com/tiangolo/fastapi
+```
+
+The code samples in this tutorial are based on the above FastAPI project. If using a different project, adjust the code samples accordingly.
+:::
+
+## Step 1: Install the Dagger Python SDK
+
+:::note
+The Dagger Python SDK requires [Python 3.10 or later](https://docs.python.org/3/using/index.html). Using a [virtual environment](https://packaging.python.org/en/latest/tutorials/installing-packages/#creating-virtual-environments) is recommended.
+:::
+
+Install the Dagger Python SDK in your project's virtual environment using `pip`:
+
+```shell
+pip install dagger-io
+```
+
+## Step 2: Create a Dagger client in Python
+
+Create a new file named `test.py` and add the following code to it.
+
+```python file=snippets/get-started/step1/test.py
+```
+
+This Python stub imports the Dagger SDK and defines an asynchronous function named `test()`. This `test()` function performs the following operations:
+
+- It creates a Dagger client with `dagger.Connection()`. This client provides an interface for executing commands against the Dagger engine.
+- It uses the client's `container().from_()` method to initialize a new container from a base image. In this example, the base image is the `python:3.10-slim-buster` image. This method returns a `Container` representing an OCI-compatible container image.
+- It uses the `Container.exec()` method to define the command to be executed in the container - in this case, the command `python -V`, which returns the Python version string. The `exec()` method returns a revised `Container` with the results of command execution.
+- It retrieves the output stream of the last executed command with the `Container.stdout()` method and prints its contents.
+
+Run the Python CI tool by executing the command below from the project directory:
+
+```shell
+python test.py
+```
+
+The tool outputs a string similar to the one below.
+
+```shell
+Hello from Dagger and Python 3.10.8
+```
+
+## Step 3: Test against a single Python version
+
+Now that the basic structure of the CI tool is defined and functional, the next step is to flesh out its `test()` function to actually test the Python application.
+
+Replace the `test.py` file from the previous step with the version below (highlighted lines indicate changes):
+
+```python file=snippets/get-started/step3/test.py
+```
+
+The revised `test()` function now does the following:
+
+- It creates a Dagger client with `dagger.Connection()`, passing the additional `dagger.Config(log_output=sys.stderr)` configuration to display the output from the Dagger engine.
+- It uses the client's `host().workdir().id()` method to obtain a reference to the current directory on the host. This reference is stored in the `src_id` variable.
+- It uses the client's `container().from_()` method to initialize a new container from a base image. This base image is the Python version to be tested against - the `python:3.10-slim-buster` image. This method returns a new `Container` class with the results.
+- It uses the `Container.with_mounted_directory()` method to mount the host directory into the container at the `/src` mount point.
+- It uses the `Container.with_workdir()` method to set the working directory in the container.
+- It chains `Container.exec()` methods to install test dependencies and run tests in the container.
+- It uses the `Container.exit_code()` method to obtain the exit code of the last executed command. An exit code of `0` implies successful execution.
+
+:::tip
+The `from_()`, `with_mounted_directory()`, `with_workdir()` and `exec()` methods all return a `Container`, making it easy to chain method calls together and create a pipeline that is easy and intuitive to understand.
+:::
+
+Run the Python CI tool by executing the command below:
+
+```shell
+python test.py
+```
+
+The tool tests the application, logging its operations to the console as it works. If all tests pass, it displays the final output below:
+
+```shell
+Tests succeeded!
+```
+
+## Step 4: Test against multiple Python versions
+
+Now that the Python CI tool can test the application against a single Python version, the next step is to extend it for multiple Python versions.
+
+Replace the `test.py` file from the previous step with the version below (highlighted lines indicate changes):
+
+```python file=snippets/get-started/step4a/test.py
+```
+
+This revision of the CI tool does much the same as before, except that it now supports multiple Python versions.
+
+- It defines the test matrix, consisting of Python versions `3.7` to `3.11`.
+- It iterates over this matrix, downloading a Python container image for each specified version and testing the source application in that version.
+
+Run the CI tool by executing the command below:
+
+```shell
+python test.py
+```
+
+The tool tests the application against each version in sequence and displays the following final output:
+
+```shell
+Starting tests for Python 3.7
+Tests for Python 3.7 succeeded!
+Starting tests for Python 3.8
+Tests for Python 3.8 succeeded!
+Starting tests for Python 3.9
+Tests for Python 3.9 succeeded!
+Starting tests for Python 3.10
+Tests for Python 3.10 succeeded!
+Starting tests for Python 3.11
+Tests for Python 3.11 succeeded!
+All tasks have finished
+```
+
+One further improvement is to speed things up by having the tests run concurrently. Here's a revised `test.py` which demonstrates how to do this (highlighted lines indicate changes):
+
+```python file=snippets/get-started/step4b/test.py
+```
+
+Run the tool again by executing the command below:
+
+```shell
+python test.py
+```
+
+Now, the tool performs tests concurrently, with a noticeable difference in the total time required.
+
+## Conclusion
+
+This tutorial introduced you to the Dagger Python SDK. It explained how to install the SDK and use it with a Python package. It also provided a working example of a Python CI tool powered by the SDK, demonstrating how to test an application against multiple Python versions in parallel.
+
+Use the SDK Reference to learn more about the Dagger Python SDK.

docs/current/sdk/python/866944-install.md

@@ -0,0 +1,15 @@
+---
+slug: /sdk/python/866944/install
+---
+
+# Installation
+
+:::note
+The Dagger Python SDK requires [Python 3.10 or later](https://docs.python.org/3/using/index.html). Using a [virtual environment](https://packaging.python.org/en/latest/tutorials/installing-packages/#creating-virtual-environments) is recommended.
+:::
+
+Install the Dagger Python SDK in your project's virtual environment using `pip`:
+
+```shell
+pip install dagger-io
+```

docs/current/sdk/python/guides/648384-multi-builds.md

@@ -0,0 +1,41 @@
+---
+slug: /sdk/python/648384/multi-builds
+displayed_sidebar: "current"
+---
+
+# Create a Multi-Build CI Pipeline
+
+## Introduction
+
+The Dagger Python SDK makes it easy to build an application for multiple OS and architecture combinations. This guide provides a working example of a Python CI tool that performs this task.
+
+## Requirements
+
+This guide assumes that:
+
+- You have a Python development environment with Python 3.10 or later. If not, install [Python](https://www.python.org/downloads/).
+- You are familiar with the basics of the Python SDK and have it installed. If not, read the [Python SDK guide](../628797-get-started.md) and the [Python SDK installation instructions](../866944-install.md).
+- You have Docker installed and running on the host system. If not, [install Docker](https://docs.docker.com/engine/install/).
+- You have an application that you wish to build. This guide assumes a Go application, but you can use an application of your choice.
+
+:::tip
+Dagger pipelines are executed as standard OCI containers. This portability enables you to do very powerful things. For example, if you're a Python developer, you can use the Python SDK to create a pipeline (written in Python) that builds an application written in a different language (Go) without needing to learn that language.
+:::
+
+## Example
+
+Assume that the Go application to be built is stored in the current directory on the host. The following code listing demonstrates how to build this Go application for multiple OS and architecture combinations using the Python SDK.
+
+```python file=../snippets/multi-builds/build.py
+```
+
+The `build()` function does the following:
+
+- It defines the build matrix, consisting of two OSs (`darwin` and `linux`) and two architectures (`amd64` and `arm64`).
+- It creates a Dagger client with `dagger.Connection()`.
+- It uses the client's `host().workdir().id()` method to obtain a reference to the current directory on the host. This reference is stored in the `src_id` variable.
+- It uses the client's `container().from_()` method to initialize a new container from a base image. This base image contains all the tooling needed to build the application - in this case, the `golang:latest` image. This `from_()` method returns a new `Container` class with the results.
+- It uses the `Container.with_mounted_directory()` method to mount the host directory into the container at the `/src` mount point.
+- It uses the `Container.with_workdir()` method to set the working directory in the container.
+- It iterates over the build matrix, creating a directory in the container for each OS/architecture combination and building the Go application for each such combination. The Go build process is instructed via the `GOOS` and `GOARCH` build variables, which are reset for each case via the `Container.with_env_variable()` method.
+- It obtains a reference to the build output directory in the container with the `with_directory()` method, and then uses the `Directory.export()` method to write the build directory from the container to the host.

docs/current/sdk/python/snippets/get-started/step1/test.py

@@ -0,0 +1,27 @@
+"""
+Execute a command
+"""
+
+import anyio
+import dagger
+
+async def test():
+    async with dagger.Connection() as client:
+
+        python = (
+            client.container()
+
+            # pull container
+            .from_("python:3.10-slim-buster")
+
+            # get Python version
+            .exec(["python", "-V"])
+        )
+
+        # execute
+        version = await python.stdout().contents()
+
+        print(f"Hello from Dagger and {version}")
+
+if __name__ == "__main__":
+    anyio.run(test)

docs/current/sdk/python/snippets/get-started/step3/test.py

@@ -0,0 +1,40 @@
+"""
+Run tests for a single Python version.
+"""
+
+import sys
+import anyio
+import dagger
+
+async def test():
+    async with dagger.Connection(dagger.Config(log_output=sys.stderr)) as client:
+
+        # highlight-start
+        # get reference to the local project
+        src_id = await client.host().workdir().id()
+
+        python = (
+            client.container()
+            .from_("python:3.10-slim-buster")
+
+            # mount cloned repository into image
+            .with_mounted_directory("/src", src_id)
+
+            # set current working directory for next commands
+            .with_workdir("/src")
+
+            # install test dependencies
+            .exec(["pip", "install", "-e", ".[test]"])
+
+            # run tests
+            .exec(["pytest", "tests"])
+        )
+
+        # execute
+        await python.exit_code()
+
+        print("Tests succeeded!")
+        # highlight-end
+
+if __name__ == "__main__":
+    anyio.run(test)