decodeFromBufferedSource throws JsonDecodingException on big JSON

[Artman12]

Describe the bug
Trying to use new Okio streams, introduced recently and encountering JsonDecodingException when trying to decode big JSON via decodeFromBufferedSource().

decodeFromString() works fine.

To Reproduce

@Serializable
data class Node(
    val children: List<Node>?
)

fun test() {
        fun getNodes(count: Int, depth: Int) : List<Node> {
            val ret = mutableListOf<Node>()
            if (depth == 0) return ret
            for (i in 0 until count) {
                ret.add(Node(getNodes(1, depth - 1)))
            }
            return ret
        }

        val rootNode = Node(getNodes(1000, 10))

        val json = Serializer.json.encodeToString(rootNode)

        val buffer = okio.Buffer().apply {
            write(json.toByteArray())
        }

//        val decodedNode = Serializer.json.decodeFromString<Node>(json) // works fine
        val decodedNode = Serializer.json.decodeFromBufferedSource<Node>(buffer) // throws JsonDecodingException 
    }

Exception error:

kotlinx.serialization.json.internal.JsonDecodingException: Expected colon ':', but had 'EOF' instead at path: $.children[107].children[0].children[108]
    JSON input: .....hildren":[{"children":[{"children":[{"children":[{"children"

I've created github repository with reproducible example: https://github.com/Artman12/OkioTest

Expected behavior
decodeFromBufferedSource decodes JSON correctly.

Environment

• Kotlin version: 1.7.10
• Library version: 1.4.0-RC
• Kotlin platforms: JVM, Native

[ihostage]

Hm 🤔 GH didn't close this automatically when #2007 was merged.

[qwwdfsad]

It closes automatically only when it is merged to master. The PR was merged to dev branch, master contains only sources for the latest release

[zacharee]

I think this came back or #2007 wasn't a full fix. With a JSON file that's about 800KB on JVM, I get the unexpected EOF error at seemingly random indices.

• Kotlin 1.9.21.
• Serialization 1.6.2.
• Okio 3.6.0.

Example trace:

Exception in thread "AWT-EventQueue-0" kotlinx.serialization.json.internal.JsonDecodingException: Unexpected EOF at path: $[389].timeMillis
JSON input: .....6406963009","status":true}}},{"timeMillis":1701922485310,"ce
	at kotlinx.serialization.json.internal.JsonExceptionsKt.JsonDecodingException(JsonExceptions.kt:24)
	at kotlinx.serialization.json.internal.JsonExceptionsKt.JsonDecodingException(JsonExceptions.kt:32)
	at kotlinx.serialization.json.internal.AbstractJsonLexer.fail(AbstractJsonLexer.kt:598)
	at kotlinx.serialization.json.internal.AbstractJsonLexer.fail$default(AbstractJsonLexer.kt:596)
	at kotlinx.serialization.json.internal.AbstractJsonLexer.consumeString(AbstractJsonLexer.kt:405)
	at kotlinx.serialization.json.internal.ReaderJsonLexer.consumeKeyString(ReaderJsonLexer.kt:149)
	at kotlinx.serialization.json.internal.AbstractJsonLexer.consumeString(AbstractJsonLexer.kt:383)
	at kotlinx.serialization.json.internal.AbstractJsonLexer.consumeStringLenient(AbstractJsonLexer.kt:463)
	at kotlinx.serialization.json.internal.AbstractJsonLexer.consumeStringLenientNotNull(AbstractJsonLexer.kt:439)
	at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeStringKey(StreamingJsonDecoder.kt:329)
	at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeObjectIndex(StreamingJsonDecoder.kt:227)
	at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeElementIndex(StreamingJsonDecoder.kt:178)
	at dev.zwander.common.data.HistoricalSnapshot$$serializer.deserialize(HistoricalSnapshot.kt:9)
	at dev.zwander.common.data.HistoricalSnapshot$$serializer.deserialize(HistoricalSnapshot.kt:9)
	at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeSerializableValue(StreamingJsonDecoder.kt:69)
	at kotlinx.serialization.encoding.AbstractDecoder.decodeSerializableValue(AbstractDecoder.kt:43)
	at kotlinx.serialization.encoding.AbstractDecoder.decodeSerializableElement(AbstractDecoder.kt:70)
	at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeSerializableElement(StreamingJsonDecoder.kt:168)
	at kotlinx.serialization.encoding.CompositeDecoder$DefaultImpls.decodeSerializableElement$default(Decoding.kt:538)
	at kotlinx.serialization.internal.CollectionLikeSerializer.readElement(CollectionSerializers.kt:80)
	at kotlinx.serialization.internal.AbstractCollectionSerializer.readElement$default(CollectionSerializers.kt:51)
	at kotlinx.serialization.internal.AbstractCollectionSerializer.merge(CollectionSerializers.kt:36)
	at kotlinx.serialization.internal.AbstractCollectionSerializer.deserialize(CollectionSerializers.kt:43)
	at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeSerializableValue(StreamingJsonDecoder.kt:69)
	at kotlinx.serialization.json.internal.JsonStreamsKt.decodeByReader(JsonStreams.kt:43)
	at kotlinx.serialization.json.okio.OkioStreamsKt.decodeFromBufferedSource(OkioStreams.kt:63)

[sandwwraith]

@zacharee Do you have a stable reproducer?

[zacharee]

I did some more testing and it looks like it isn't the JSON library, but some quirk in Okio's file writing/reading not properly syncing when using FileSystem.source() and FileSystem.sink(). Wrapping those operations in a Mutex fixes my issue.