With DRF 3.3.x, validated_data have boolean to False by default

[GerardPaligot]

Hi,

I have a very simple test case in my open source project where I checked that when I make a PUT request on a profile without any parameter, all fields keep their values.

Unfortunately, I upgraded my dependency of DRF from the version 3.1.1 and with the latest version, validated_data is no more empty when we don't specify any parameter so we update fields of our profile.

We should keep nothing in validated_data

Test case:

def test_update_member_details_without_any_change(self):
        response = self.client_authenticated.put(reverse('api-member-detail', args=[self.profile.user.id]))

        self.assertEqual(response.status_code, status.HTTP_200_OK)
        # Some asserts on the response and fields of self.profile.

Serializer:

class ProfileValidatorSerializer(serializers.ModelSerializer, ProfileUsernameValidator, ProfileEmailValidator):
    class Meta:
        model = Profile
        fields = ('id', 'username', 'email', 'is_active', 'date_joined',
                  'site', 'avatar_url', 'biography', 'sign', 'show_email',
                  'show_sign', 'hover_or_click', 'email_for_answer', 'last_visit')
        read_only_fields = ('is_active', 'date_joined', 'last_visit',)

    def update(self, instance, validated_data):
        # Validated_data not empty.
        pass

validated_data value with DRF 3.3.x:

{
  u'show_email': False, 
  u'hover_or_click': False, 
  u'show_sign': False, 
  u'user': {
    u'is_active': False
  }, 
  u'email_for_answer': False
}

Note: It seems take all boolean fields and specify False as value.

validated_data value with DRF 3.1.1:

{}

[xordoquy]

It's a bit strange that you seem to have errors inside the validated_data.
Does your view call the serializer's is_valid ?

[GerardPaligot]

Yes, I let DRF invokes it. This is my view:

class MemberDetailAPI(RetrieveUpdateAPIView):
    queryset = Profile.objects.all()
    lookup_field = 'user__id'
    obj_key_func = DetailKeyConstructor()

    @etag(obj_key_func)
    @cache_response(key_func=obj_key_func)
    def get(self, request, *args, **kwargs):
        # Code ...

    @etag(obj_key_func, rebuild_after_method_evaluation=True)
    def put(self, request, *args, **kwargs):
        self.permission_classes = (IsAuthenticatedOrReadOnly, IsOwnerOrReadOnly)
        return self.update(request, *args, **kwargs)

    def get_serializer_class(self):
        if self.request.method == 'GET':
            return ProfileDetailSerializer
        elif self.request.method == 'PUT':
            return ProfileValidatorSerializer

[xordoquy]

This seems to be linked to #2776. Do you force the content type to json in you tests ?

[GerardPaligot]

This seems to be linked to #2776.

Maybe a side effect?

Do you force the content type to json in you tests ?

Nope, the complete test case is:

    def setUp(self):
        self.client = APIClient()
        self.profile = ProfileFactory()

        client_oauth2 = create_oauth2_client(self.profile.user)
        self.client_authenticated = APIClient()
        authenticate_client(self.client_authenticated, client_oauth2, self.profile.user.username, 'hostel77')

        caches[extensions_api_settings.DEFAULT_USE_CACHE].clear()

    def test_update_member_details_without_any_change(self):
        """
        Updates a member but without any changes.
        """
        response = self.client_authenticated.put(reverse('api-member-detail', args=[self.profile.user.id]))

        self.assertEqual(response.status_code, status.HTTP_200_OK)
        self.assertEqual(self.profile.user.id, response.data.get('id'))
        self.assertEqual(self.profile.user.username, response.data.get('username'))
        self.assertEqual(self.profile.user.email, response.data.get('email'))
        self.assertEqual(self.profile.user.is_active, response.data.get('is_active'))
        self.assertIsNotNone(response.data.get('date_joined'))
        self.assertEqual(self.profile.site, response.data.get('site'))
        self.assertEqual(self.profile.avatar_url, response.data.get('avatar_url'))
        self.assertEqual(self.profile.biography, response.data.get('biography'))
        self.assertEqual(self.profile.sign, response.data.get('sign'))
        self.assertFalse(response.data.get('show_email'))
        self.assertEqual(self.profile.show_sign, response.data.get('show_sign'))
        self.assertEqual(self.profile.hover_or_click, response.data.get('hover_or_click'))
        self.assertEqual(self.profile.email_for_answer, response.data.get('email_for_answer'))

The content type doesn't seem forced.

[xordoquy]

So since you're not enforcing JSON, HTML is used.
Unfortunately there's no way we can detect with HTML whether an empty boolean is empty because it wasn't provided or because it's false (basically what #2776 is about). Therefore we have to assume empty boolean fields have in fact false value.

This limitation also applies to Django's BooleanField which is the reason NullBooleanField also exist.

Your options:

• enforce the test content type to JSON but be warned that posting empty HTML data may alter your instance.
• Use NullBooleanField. I'm not 100% sure but those should be set to None by default or not set at all.

[GerardPaligot]

Maybe you have closed a little bit too fast my issue. I checked global settings and I force well JSON:

REST_FRAMEWORK = {
    'DEFAULT_PAGINATION_CLASS': 'zds.api.pagination.DefaultPagination',
    # Active OAuth2 authentication.
    'DEFAULT_AUTHENTICATION_CLASSES': (
        'oauth2_provider.ext.rest_framework.OAuth2Authentication',
        'rest_framework.authentication.SessionAuthentication',
    ),
    'DEFAULT_PARSER_CLASSES': (
        'rest_framework.parsers.JSONParser',
        #'rest_framework.parsers.XMLParser',
        'rest_framework_xml.parsers.XMLParser',
        'rest_framework.parsers.FormParser',
        'rest_framework.parsers.MultiPartParser',
    ),
    'DEFAULT_RENDERER_CLASSES': (
        'rest_framework.renderers.JSONRenderer',
        #'rest_framework.renderers.XMLRenderer',
        'rest_framework_xml.renderers.XMLRenderer',
        'rest_framework.renderers.BrowsableAPIRenderer',
    ),
    'DEFAULT_THROTTLE_CLASSES': (
        'rest_framework.throttling.AnonRateThrottle',
        'rest_framework.throttling.UserRateThrottle'
    ),
    'DEFAULT_THROTTLE_RATES': {
        'anon': '60/hour',
        'user': '2000/hour'
    },
    'TEST_REQUEST_DEFAULT_FORMAT': 'json'
}

[xordoquy]

Indeed.

[GerardPaligot]

I don't know if it is useful but I inspected a bit and when I'm at this line with a BooleanField, html.is_html_input(dictionary) returns True because an empty QueryDict have the attribute getlist:

def is_html_input(dictionary):
    # MultiDict type datastructures are used to represent HTML form input,
    # which may have more than one value for each key.
    return hasattr(dictionary, 'getlist')

Edit: is_html_input returns True for all fields.

[nazarewk]

Looks like a sideeffect of this issue's fix #3314

[tomchristie]

Does not reproduce on master. Either has been resolved, or else the test case was erroneously not sending JSON requests as expected (eg. different settings were actually in use). The later seems likely as this does describe the HTML form behaviour.

Happy to reconsider if anyone can raise a reproducible case.

[hitanshsingla]

Hi, I am facing this strange issue continuously:
When I PUT some boolean fields, it works as expected.
But when I PUT a file, it resets all fields to False.

Screenshot 1:
I set draft and external to true

Screenshot 2:
When I upload a file, all boolean fields set themselves to false

Models.py:

is_draft = models.BooleanField(default=True, db_index=True)
is_external = models.BooleanField(default=False, db_index=True)

views.py

            serializer = PlaylistSerializer(pl, data=request.data)
            if serializer.is_valid():
                serializer.save()
                return Response(serializer.data, status=status.HTTP_200_OK)
            else:
                return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST)

serializers.py

class PlaylistSerializer(ModelSerializer):
    class Meta:
        model = Playlist
        fields = '__all__'
        extra_kwargs = {'access_key': {'write_only': True}}

It would be great if you could check this.
Thank you!
Hitansh

[xordoquy]

There's nothing to check, that's how the booleans in HTML form work in the HTML specs: no value means False value.
Nothing we can do in DRF about that.

[192dot]

This is similar to this issue: #1101