Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python: Remove dependencies to .env file #1779

Closed
sbaidachni opened this issue Jun 29, 2023 · 5 comments · Fixed by #6193
Closed

Python: Remove dependencies to .env file #1779

sbaidachni opened this issue Jun 29, 2023 · 5 comments · Fixed by #6193
Assignees
Labels
ai connector Anything related to AI connectors kernel Issues or pull requests impacting the core kernel python Pull requests for the Python Semantic Kernel

Comments

@sbaidachni
Copy link

https://github.com/microsoft/semantic-kernel/blob/16d52c155a2c8193fef7207a8816ac8c4a8eed2c/python/semantic_kernel/utils/settings.py#L16C7-L16C7

The current implementation of openai_settings_from_dot_env() and azure_openai_settings_from_dot_env() uses dotenv_values(".env") to initialize configuration parameters from the .env file. At the same time, there are many ways to initialize environment variables not using the .env file. For example, in Azure DevOps we can use variable groups and nobody host keys in .env file in a repository. Therefore, we would like to see an implementation that is not stick to .env directly. Here is an example, how it can be implemented:

load_dotenv()
deployment = os.environ.get("AZURE_OPENAI_DEPLOYMENT_NAME")
api_key = os.environ.get("AZURE_OPENAI_API_KEY")
endpoint = os.environ.get("AZURE_OPENAI_ENDPOINT")

In this implementation I can use .env locally to host variables or I can setup environment variables in any other way (like in DevOps or GitHub) and it will work.

@nacharya1 nacharya1 added the python Pull requests for the Python Semantic Kernel label Jul 3, 2023
@matthewbolanos matthewbolanos added kernel.core kernel Issues or pull requests impacting the core kernel and removed kernel.core labels Jan 2, 2024
@moonbox3 moonbox3 added the needs help These are issues that are good candidates for the community to work on label Feb 26, 2024
@moonbox3
Copy link
Contributor

moonbox3 commented Mar 8, 2024

Note for internal SK: keeping this older work item open as it is pertinent and on our backlog.

@eavanvalkenburg
Copy link
Member

Let's investigate using pydantic-settings: https://lewoudar.medium.com/my-favorite-way-to-configure-a-python-project-5a222c579d4c

@MaxiPigna
Copy link

Same issue. I am deploying an app on an Azure App Service with Environment Variables configured, but it fails since it cannot find the .env file.
image

@eavanvalkenburg
Copy link
Member

Thanks for reporting @MaxiPigna this is indeed one of the issues with the current approach, unfortunately for you, you will have to directly use env without using the settings helpers. But we will fix it!

@moonbox3
Copy link
Contributor

moonbox3 commented May 8, 2024

@moonbox3 moonbox3 self-assigned this May 9, 2024
@moonbox3 moonbox3 added ai connector Anything related to AI connectors and removed needs help These are issues that are good candidates for the community to work on labels May 9, 2024
github-merge-queue bot pushed a commit that referenced this issue May 16, 2024
### Motivation and Context

SK Python is tightly coupled to the use of a `.env` file to read all
secrets, keys, endpoints, and more. This doesn't scale well for users
who wish to be able to use environment variables with their SK
Applications. By introducing Pydantic Settings, it is possible to use
both environment variables as well as have a fall-back to a `.env` file
(via a `env_file_path` parameter), if desired.

By introducing Pydantic Settings, we are removing the requirement to
have to create Text/Embedding/Chat completion objects with an `api_key`
or other previously required information (in the case of
AzureChatCompletion that means an `endpoint`, an `api_key`, a
`deployment_name`, and an `api_version`). When the AI connector is
created, the Pydantic settings are loaded either via env vars or the
fall-back `.env` file, and that means the user can create a chat
completion object like:

```python
chat_completion = OpenAIChatCompletion(service_id="test")
```

or, to optionally override the `ai_model_id` env var:

```python
chat_completion = OpenAIChatCompletion(service_id="test", ai_model_id="gpt-4-1106")
```
Note: we have left the ability to specific an `api_key`/`org_id` for
`OpenAIChatCompletion` or a `deployment_name`, `endpoint`, `base_url`,
and `api_version` for `AzureChatCompletion` as before, but if your
settings are configured to use env vars/.env file then there is no need
to pass this information.

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->

### Description

The PR introduces the use of Pydantic settings and removes the use of
the python-dotenv library.
- Closes #1779 
- Updates notebooks, samples, code and tests to remove the explicit
config of api_key or other previous .env files values.
- Adds new unit test config using monkeypatch to simulate env variables
for testing
- All unit and integration tests passing

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [X] The code builds clean without any errors or warnings
- [X] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [X] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄
bochris pushed a commit to bochris/semantic-kernel that referenced this issue May 16, 2024
SK Python is tightly coupled to the use of a `.env` file to read all
secrets, keys, endpoints, and more. This doesn't scale well for users
who wish to be able to use environment variables with their SK
Applications. By introducing Pydantic Settings, it is possible to use
both environment variables as well as have a fall-back to a `.env` file
(via a `env_file_path` parameter), if desired.

By introducing Pydantic Settings, we are removing the requirement to
have to create Text/Embedding/Chat completion objects with an `api_key`
or other previously required information (in the case of
AzureChatCompletion that means an `endpoint`, an `api_key`, a
`deployment_name`, and an `api_version`). When the AI connector is
created, the Pydantic settings are loaded either via env vars or the
fall-back `.env` file, and that means the user can create a chat
completion object like:

```python
chat_completion = OpenAIChatCompletion(service_id="test")
```

or, to optionally override the `ai_model_id` env var:

```python
chat_completion = OpenAIChatCompletion(service_id="test", ai_model_id="gpt-4-1106")
```
Note: we have left the ability to specific an `api_key`/`org_id` for
`OpenAIChatCompletion` or a `deployment_name`, `endpoint`, `base_url`,
and `api_version` for `AzureChatCompletion` as before, but if your
settings are configured to use env vars/.env file then there is no need
to pass this information.

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->

The PR introduces the use of Pydantic settings and removes the use of
the python-dotenv library.
- Closes microsoft#1779
- Updates notebooks, samples, code and tests to remove the explicit
config of api_key or other previous .env files values.
- Adds new unit test config using monkeypatch to simulate env variables
for testing
- All unit and integration tests passing

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->

<!-- Before submitting this PR, please make sure: -->

- [X] The code builds clean without any errors or warnings
- [X] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [X] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ai connector Anything related to AI connectors kernel Issues or pull requests impacting the core kernel python Pull requests for the Python Semantic Kernel
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

7 participants