-
Notifications
You must be signed in to change notification settings - Fork 2.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Python: Remove dependencies to .env file #1779
Comments
Note for internal SK: keeping this older work item open as it is pertinent and on our backlog. |
Let's investigate using pydantic-settings: https://lewoudar.medium.com/my-favorite-way-to-configure-a-python-project-5a222c579d4c |
Thanks for reporting @MaxiPigna this is indeed one of the issues with the current approach, unfortunately for you, you will have to directly use |
### Motivation and Context SK Python is tightly coupled to the use of a `.env` file to read all secrets, keys, endpoints, and more. This doesn't scale well for users who wish to be able to use environment variables with their SK Applications. By introducing Pydantic Settings, it is possible to use both environment variables as well as have a fall-back to a `.env` file (via a `env_file_path` parameter), if desired. By introducing Pydantic Settings, we are removing the requirement to have to create Text/Embedding/Chat completion objects with an `api_key` or other previously required information (in the case of AzureChatCompletion that means an `endpoint`, an `api_key`, a `deployment_name`, and an `api_version`). When the AI connector is created, the Pydantic settings are loaded either via env vars or the fall-back `.env` file, and that means the user can create a chat completion object like: ```python chat_completion = OpenAIChatCompletion(service_id="test") ``` or, to optionally override the `ai_model_id` env var: ```python chat_completion = OpenAIChatCompletion(service_id="test", ai_model_id="gpt-4-1106") ``` Note: we have left the ability to specific an `api_key`/`org_id` for `OpenAIChatCompletion` or a `deployment_name`, `endpoint`, `base_url`, and `api_version` for `AzureChatCompletion` as before, but if your settings are configured to use env vars/.env file then there is no need to pass this information. <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> ### Description The PR introduces the use of Pydantic settings and removes the use of the python-dotenv library. - Closes #1779 - Updates notebooks, samples, code and tests to remove the explicit config of api_key or other previous .env files values. - Adds new unit test config using monkeypatch to simulate env variables for testing - All unit and integration tests passing <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [X] The code builds clean without any errors or warnings - [X] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [X] All unit tests pass, and I have added new tests where possible - [ ] I didn't break anyone 😄
SK Python is tightly coupled to the use of a `.env` file to read all secrets, keys, endpoints, and more. This doesn't scale well for users who wish to be able to use environment variables with their SK Applications. By introducing Pydantic Settings, it is possible to use both environment variables as well as have a fall-back to a `.env` file (via a `env_file_path` parameter), if desired. By introducing Pydantic Settings, we are removing the requirement to have to create Text/Embedding/Chat completion objects with an `api_key` or other previously required information (in the case of AzureChatCompletion that means an `endpoint`, an `api_key`, a `deployment_name`, and an `api_version`). When the AI connector is created, the Pydantic settings are loaded either via env vars or the fall-back `.env` file, and that means the user can create a chat completion object like: ```python chat_completion = OpenAIChatCompletion(service_id="test") ``` or, to optionally override the `ai_model_id` env var: ```python chat_completion = OpenAIChatCompletion(service_id="test", ai_model_id="gpt-4-1106") ``` Note: we have left the ability to specific an `api_key`/`org_id` for `OpenAIChatCompletion` or a `deployment_name`, `endpoint`, `base_url`, and `api_version` for `AzureChatCompletion` as before, but if your settings are configured to use env vars/.env file then there is no need to pass this information. <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> The PR introduces the use of Pydantic settings and removes the use of the python-dotenv library. - Closes microsoft#1779 - Updates notebooks, samples, code and tests to remove the explicit config of api_key or other previous .env files values. - Adds new unit test config using monkeypatch to simulate env variables for testing - All unit and integration tests passing <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> <!-- Before submitting this PR, please make sure: --> - [X] The code builds clean without any errors or warnings - [X] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [X] All unit tests pass, and I have added new tests where possible - [ ] I didn't break anyone 😄
https://github.com/microsoft/semantic-kernel/blob/16d52c155a2c8193fef7207a8816ac8c4a8eed2c/python/semantic_kernel/utils/settings.py#L16C7-L16C7
The current implementation of openai_settings_from_dot_env() and azure_openai_settings_from_dot_env() uses dotenv_values(".env") to initialize configuration parameters from the .env file. At the same time, there are many ways to initialize environment variables not using the .env file. For example, in Azure DevOps we can use variable groups and nobody host keys in .env file in a repository. Therefore, we would like to see an implementation that is not stick to .env directly. Here is an example, how it can be implemented:
load_dotenv()
deployment = os.environ.get("AZURE_OPENAI_DEPLOYMENT_NAME")
api_key = os.environ.get("AZURE_OPENAI_API_KEY")
endpoint = os.environ.get("AZURE_OPENAI_ENDPOINT")
In this implementation I can use .env locally to host variables or I can setup environment variables in any other way (like in DevOps or GitHub) and it will work.
The text was updated successfully, but these errors were encountered: