Test incremental UTF-8 decoding for iOS on a cloud Mac

If the same response reads correctly locally but occasionally produces garbled text in a cloud Mac pipeline, check the decoding boundaries first. Networks deliver chunks of bytes, not complete characters; the three bytes of “中” may arrive in three separate chunks. Converting each chunk to a string independently can turn valid text into replacement characters, even if no call crashes. The standalone Swift script below defines and verifies this contract without relying on how a real network happens to split the data.

Define the byte-stream contract first

Input must be UTF-8, and output must contain only fully decoded Unicode scalar values. Empty chunks leave the state unchanged, incomplete sequences carry over to the next chunk, and any remaining bytes at the end of the stream cause a failure. Do not interpret “no text output right now” as a closed connection.

This implementation strictly rejects malformed input: isolated continuation bytes, overlong encodings, surrogate encodings, and sequences beyond the Unicode limit must all fail. Replacing malformed text may be appropriate for some display scenarios, but it must not happen silently in protocol fields, configuration, or persisted records.

Network chunk boundaries, character boundaries, and application message boundaries are three different things. Successful UTF-8 decoding does not mean a complete line or message has arrived.

If the protocol declares another encoding, reject it at the entry point or pass it to an explicit conversion layer. Do not guess by repeatedly trying different encodings. This implementation also leaves byte order marks intact; the upper layer decides whether to allow them.

Implement bounded incremental state

Save the following code as utf8-contract.swift. It retains only the current incomplete character, carrying at most three bytes between calls; the amount of output from each call still scales with the input chunk size. After an error, discard the instance rather than feeding it more data.

import Foundation

enum DecodeFailure: Error {
    case invalidByte
    case truncated
}

struct StrictUTF8 {
    private var pending: [UInt8] = []
    private var expected = 0

    mutating func push(_ bytes: [UInt8]) throws -> String {
        var output: [UInt8] = []

        for byte in bytes {
            if pending.isEmpty {
                switch byte {
                case 0x00...0x7F:
                    output.append(byte)
                    continue
                case 0xC2...0xDF:
                    expected = 2
                case 0xE0...0xEF:
                    expected = 3
                case 0xF0...0xF4:
                    expected = 4
                default:
                    throw DecodeFailure.invalidByte
                }
                pending.append(byte)
                continue
            }

            var allowed: ClosedRange<UInt8> = 0x80...0xBF
            if pending.count == 1 {
                switch pending[0] {
                case 0xE0: allowed = 0xA0...0xBF
                case 0xED: allowed = 0x80...0x9F
                case 0xF0: allowed = 0x90...0xBF
                case 0xF4: allowed = 0x80...0x8F
                default: break
                }
            }

            guard allowed.contains(byte) else {
                throw DecodeFailure.invalidByte
            }
            pending.append(byte)

            if pending.count == expected {
                output.append(contentsOf: pending)
                pending.removeAll(keepingCapacity: true)
                expected = 0
            }
        }

        guard let text = String(bytes: output, encoding: .utf8) else {
            throw DecodeFailure.invalidByte
        }
        return text
    }

    func finish() throws {
        guard pending.isEmpty else {
            throw DecodeFailure.truncated
        }
    }
}

The leading byte determines the sequence length, the second byte narrows the permitted range, and subsequent bytes are checked against the continuation-byte range. This catches surrogate and out-of-range encodings before string conversion. The implementation has no automatic recovery logic and does not treat ASCII following an invalid byte as a trustworthy new message.

Enumerate chunk partitions instead of repeating requests

Append the following tests. For this short sample, enumerating every “split or do not split” combination between adjacent bytes gives more complete coverage than checking a single two-chunk split. Empty chunks are also inserted to verify that they do not clear pending state.

let sample = "A中🙂e\u{301}"
let bytes = Array(sample.utf8)

for mask in 0..<(1 << (bytes.count - 1)) {
    var decoder = StrictUTF8()
    var chunk: [UInt8] = []
    var result = ""

    for index in bytes.indices {
        chunk.append(bytes[index])
        let cut = index == bytes.count - 1
            || (mask & (1 << index)) != 0
        if cut {
            result += try decoder.push(chunk)
            result += try decoder.push([])
            chunk.removeAll(keepingCapacity: true)
        }
    }

    try decoder.finish()
    precondition(Array(result.utf8) == bytes)
}

The comparison uses UTF-8 bytes so that canonically equivalent string comparisons cannot hide changes to the content. The combining accent also illustrates that a complete scalar is not necessarily a complete user-perceived character. The decoder may emit the letter first and the combining mark later; if the interface needs to operate on graphemes, handle that at a higher layer.

The number of partitions grows exponentially with sample length, so use exhaustive enumeration only for short, fixed test vectors. For longer text, generate partitions with a fixed random seed and save the sequence of chunk lengths whenever a test fails.

Turn rejection paths into assertions too

Accepting valid text proves only half the contract. Append these vectors to cover, respectively, an isolated continuation byte, an overlong encoding, a surrogate, a value beyond the Unicode limit, an incorrect continuation byte, and a truncated ending. Feed each input one byte at a time to ensure chunking cannot bypass error detection.

let invalid: [[UInt8]] = [
    [0x80],
    [0xC0, 0xAF],
    [0xED, 0xA0, 0x80],
    [0xF4, 0x90, 0x80, 0x80],
    [0xE4, 0x41],
    [0xF0, 0x9F, 0x99]
]

for input in invalid {
    var decoder = StrictUTF8()
    var rejected = false

    do {
        for byte in input {
            _ = try decoder.push([byte])
        }
        try decoder.finish()
    } catch {
        rejected = true
    }

    precondition(rejected)
}

Also add valid range boundaries, such as the four-byte sequence for the highest valid scalar value. Do not use the presence of a replacement character in the string as your only criterion: the input may legitimately contain that character. Strict decoding must make its decisions from the original bytes.

Integrate with the pipeline and preserve error boundaries

Run the script in the selected Xcode command-line tools environment:

xcrun swift utf8-contract.swift

Normal completion means the checks passed; any thrown error or failed assertion must block subsequent tasks. For production integration, move the decoder into shared source code and the vectors into a test target. The standalone script is useful for validating the algorithm first, but it does not replace compiling and running the actual iOS target.

Give each connection its own instance and call it serially in receive order. Create a new instance on reconnect; never prepend half a character from the old connection to a new response. Cancellation should also discard the state without taking the successful-completion path.

Finally, check the consumer: output from each chunk must go only into a temporary buffer. Commit to persistent storage or trigger actions only after a complete application message has passed validation. If the first half has already been written to the database when the second half reveals an invalid encoding, strict decoding cannot automatically undo those side effects. Error logs need only retain the byte offset, chunk length, and error category; avoid uploading the full response content by default. The acceptance criterion is then more than “no garbled text”: changing the chunk boundaries must not change the result, and errors must never count as successful delivery.

Frequently asked questions