to_snake alias generator bug -- great first issue

[jevins09]

Initial Checks

• I confirm that I'm using Pydantic V2

Description

Issue Summary:

When using the to_snake alias generator from pydantic.alias_generators in Pydantic v2, it appears that CamelCase strings are converted to lowercase without the inclusion of underscores between words. This behavior is unexpected and differs from the conventional understanding of snake_case formatting.

Steps to Reproduce:

Import to_snake from pydantic.alias_generators.
Apply to_snake to a CamelCase string, such as "HTTPResponse".

Expected Result:

The expected output for to_snake("HTTPResponse") should be "http_response", maintaining the standard snake_case convention where words are separated by underscores and converted to lowercase.

Actual Result:

The actual output is "httpresponse", which is simply the input string converted to lowercase without any underscores separating the words.

Additional Notes

I also have the same issue when using to_camel alias generator when populating by name.

from pydantic import BaseModel, ConfigDict
from pydantic.alias_generators import to_camel


class CamelAliasModel(BaseModel):
    model_config = ConfigDict(
        alias_generator=to_camel,
        populate_by_name=True,
    )

class Foo(CamelAliasModel):
    http_response_code: int
    

f = Foo.model_validate({"HTTPResponseCode": "200"}) # Throws error

### pydantic_core._pydantic_core.ValidationError: 1 validation error for Foo
### httpResponseCode
### Field required [type=missing, input_value={'HTTPResponseCode': '200'}, input_type=dict]
    

Example Code

# `to_snake`
from pydantic.alias_generators import to_snake

result = to_snake("HTTPResponse")
print(result)  # Output: 'httpresponse'

# `to_camel`

from pydantic import BaseModel, ConfigDict
from pydantic.alias_generators import to_camel


class CamelAliasModel(BaseModel):
    model_config = ConfigDict(
        alias_generator=to_camel,
        populate_by_name=True,
    )

class Foo(CamelAliasModel):
    http_response_code: int
    

f = Foo.model_validate({"HTTPResponseCode": "200"}) # Throws Error

Python, Pydantic & OS Version

pydantic version: 2.4.2
 pydantic-core version: 2.10.1
 pydantic-core build: profile=release pgo=false
 python version: 3.11.3 (main, Jul  6 2023, 12:41:34) [Clang 14.0.0 (clang-1400.0.29.202)]
 platform: macOS-14.1-arm64-arm-64bit

[sydney-runkle]

@jevins09,

Thanks for reporting this. I'd agree, we have a bug in our to_snake logic. This should be a one-liner fix in the definition of to_snake, so it's an awesome first issue if you're interested.

In the second case, I don't think that the camel case alias generator is doing anything wrong. You wouldn't want mashed_potatoes to be turned into MASHEDPotatoes, which is effectively what you're expecting in your inquiry re to_camel. In my simple example, we'd expect mashedPotatoes 🥔.

[jevins09]

@sydney-runkle Thanks for taking a look!

I made my very first PR to pydantic regarding to_snake logic: #8316.

I'd love to get some eyes on it.

In my second case, I was actually expecting it to work both ways with:

f = Foo.model_validate({"HTTPResponseCode": "200"}) 
f = Foo.model_validate({"httpResponseCode": "200"})

since the camel_case equivalent of both HTTPResponseCode and httpResponseCode would be http_response_code.

I can't really validate that this is the expected behavior though.

[sydney-runkle]

@jevins09,

PR looks great - I've approved and merged 🚀.

The to_camel converter works by removing underscores and uppercasing the first letter of each word other than the first. Thus, there's no way for it to assume that it should capitalize all of HTTPR from http_response_code.

[manuel-garnier]

Note that number management has been changed with this issue resolution.

def to_snake_old(camel: str) -> str:
    snake = re.sub(r'([a-zA-Z])([0-9])', lambda m: f'{m.group(1)}_{m.group(2)}', camel)
    snake = re.sub(r'([a-z0-9])([A-Z])', lambda m: f'{m.group(1)}_{m.group(2)}', snake)

    return snake.lower()

def to_snake_new(camel: str) -> str:
    # Handle the sequence of uppercase letters followed by a lowercase letter
    snake = re.sub(r'([A-Z]+)([A-Z][a-z])', lambda m: f'{m.group(1)}_{m.group(2)}', camel)
    # Insert an underscore between a lowercase letter and an uppercase letter
    snake = re.sub(r'([a-z])([A-Z])', lambda m: f'{m.group(1)}_{m.group(2)}', snake)
    # Insert an underscore between a digit and an uppercase letter
    snake = re.sub(r'([0-9])([A-Z])', lambda m: f'{m.group(1)}_{m.group(2)}', snake)
    # Insert an underscore between a lowercase letter and a digit
    snake = re.sub(r'([a-z])([0-9])', lambda m: f'{m.group(1)}_{m.group(2)}', snake)

    return snake.lower()

print(to_snake_old('ZONE1_F')) # 'zone_1_f'
print(to_snake_new('ZONE1_F')) # 'zone1_f'

I'm not sure if this was intended and which one should be preferred though as literature about snake case does not talk much about numbers. An option could be to add more alias_generators with different policies.

@sydney-runkle: any feedback on this?