Commit 2f236ff0 authored by Kourser's avatar Kourser
Browse files

fix: trois défauts que seule une vraie requête à un modèle fait sortir

LM Studio sur le Mac, google/gemma-4-e4b, interrogé depuis l'app dans le
simulateur sur « Profileuse ». La génération a échoué sur « La requête a
expiré. » — une phrase du système, pas de nous.

Le délai était de cinq minutes pour tout le monde.
`timeoutIntervalForRequest` compte le silence, et une complétion non
streamée n'envoie rien tant qu'elle n'est pas finie : toute la lecture du
bloc plus toute l'écriture doivent tenir dedans. Mesuré ici, un bloc de
19 608 jetons d'entrée et 1 999 de sortie prend 1 227 s. Un modèle du
réseau local a maintenant 2 400 s, le cloud garde les 300 s qui lui
suffisent.

L'`URLError` traversait le client sans être enveloppée et arrivait telle
quelle à l'écran, sans nommer le modèle, le point d'accès ni le délai
écoulé — les trois seules choses à regarder. Même leçon que le bug
Calibre : une erreur qui ne nomme rien ne sert à rien.

Une réponse atteignant le plafond de sortie faisait échouer toute la
génération. Sur ce modèle, chaque requête de notes tape ce plafond, donc
aucun livre long n'aurait abouti même avec un délai généreux. La coupure
est signalée sur la réponse plutôt que levée : des notes coupées restent
des notes, seul un script coupé est refusé — on ne s'en apercevrait
qu'en voiture.

Enfin, les notes sont écrites sur disque dès qu'elles reviennent,
indexées par le contenu du bloc et non par sa position. Un échec au
dernier bloc jetait l'heure déjà passée ; relancer reprend maintenant là
où ça s'était arrêté, et l'écran dit combien de parties sont déjà lues.

Aucun résumé n'est allé au bout pour autant : à 1 227 s le bloc, les
quatre blocs du livre plus le repli et l'écriture dépassent l'heure et
demie. Le code tient, le modèle non. Noté comme tel dans le README.

Co-Authored-By: default avatarClaude Opus 5 <noreply@anthropic.com>
parent 2838732a
Loading
Loading
Loading
Loading
+22 −0
Changes for CHANGELOG.md: 22 added lines, 0 removed lines.
Original line number Diff line number Diff line
@@ -7,7 +7,29 @@ adopte le [versionnage sémantique](https://semver.org/lang/fr/).

## [Non publié]

### Ajouté
- Les notes de lecture prises bloc par bloc sont gardées sur disque pendant la
  génération d'un résumé, indexées par le contenu du bloc. Un échec en cours de
  route ne jette plus l'heure déjà passée : relancer reprend là où ça s'était
  arrêté, et l'écran dit combien de parties sont déjà lues. Les notes sont
  effacées une fois le script écrit.

### Corrigé
- Le délai d'attente d'un modèle était de cinq minutes pour tout le monde. Or
  `timeoutIntervalForRequest` compte le silence, et une complétion non streamée
  n'envoie rien tant qu'elle n'est pas finie : toute la lecture du bloc plus
  toute l'écriture doivent tenir dedans. Mesuré sur un modèle 4B servi par LM
  Studio, un bloc de 19 600 jetons met 1 227 s. Un modèle du réseau local a
  maintenant 2 400 s, le cloud garde 300 s.
- Une erreur réseau arrivait à l'écran sous sa forme système — « La requête a
  expiré. » — sans nommer le modèle, le point d'accès ni le délai écoulé, c'est
  à dire exactement ce qu'il faut regarder. Les `URLError` sont désormais
  enveloppées et le message les nomme.
- Une réponse atteignant le plafond de sortie faisait échouer toute la
  génération. Sur le même modèle 4B, chaque requête de notes tapait ce plafond,
  donc aucun livre long n'était résumable. La coupure est maintenant signalée
  plutôt que levée : des notes coupées restent des notes, seul un script coupé
  est refusé.
- La dernière ligne de la page était tranchée en deux. Le lecteur EPUB
  ignorait la marge de sécurité basse, donc la vue web passait sous le pied de
  page et sous la barre de narration au lieu de se repaginer. Visible sur un
+61 −21
Changes for Packages/Core/Sources/PodbookUI/Digest/DigestGenerator.swift: 61 added lines, 21 removed lines.
Original line number Diff line number Diff line
@@ -18,10 +18,17 @@ public struct DigestGenerator: Sendable {
    private let client: any LLMClient
    /// Tokens the prompt may occupy. Leaves room for the answer.
    private let inputBudget: Int
    /// Notes survive a failed attempt, so a retry only does what is left.
    private let notesCache: DigestNotesCache?

    public init(client: any LLMClient, inputBudget: Int = 24_000) {
    public init(
        client: any LLMClient,
        inputBudget: Int = 24_000,
        notesCache: DigestNotesCache? = nil
    ) {
        self.client = client
        self.inputBudget = inputBudget
        self.notesCache = notesCache
    }

    public func generate(
@@ -58,6 +65,9 @@ public struct DigestGenerator: Sendable {
            ))
            inputTokens += response.inputTokens
            outputTokens += response.outputTokens
            // A script that stops two sentences from the end is worse than no
            // script: you only find out in the car.
            if response.wasTruncated { throw LLMError.truncated }
            scriptText = response.text
        } else {
            let chunks = TokenEstimate.chunks(of: trimmed, maxTokens: available)
@@ -65,24 +75,21 @@ public struct DigestGenerator: Sendable {
            for (index, chunk) in chunks.enumerated() {
                try Task.checkCancellation()
                progress?(.summarising(index: index, total: chunks.count))
                let response = try await client.complete(LLMRequest(
                    system: DigestPrompts.notesSystem,
                    user: DigestPrompts.notesUser(
                        book: book, chunk: chunk, index: index, total: chunks.count
                    ),
                    maxOutputTokens: 2000
                notes.append(try await note(
                    on: chunk, book: book, bookID: bookID,
                    index: index, total: chunks.count,
                    inputTokens: &inputTokens, outputTokens: &outputTokens
                ))
                inputTokens += response.inputTokens
                outputTokens += response.outputTokens
                notes.append(response.text)
            }

            try Task.checkCancellation()
            progress?(.writing)
            // The notes themselves can overflow on a very long book; one more
            // fold keeps the final request inside the window.
            let folded = try await fold(notes: notes, book: book, available: available,
                                       inputTokens: &inputTokens, outputTokens: &outputTokens)
            let folded = try await fold(
                notes: notes, book: book, bookID: bookID, available: available,
                inputTokens: &inputTokens, outputTokens: &outputTokens
            )
            let response = try await client.complete(LLMRequest(
                system: system,
                user: DigestPrompts.reduceUser(for: plan, book: book, notes: folded),
@@ -90,7 +97,10 @@ public struct DigestGenerator: Sendable {
            ))
            inputTokens += response.inputTokens
            outputTokens += response.outputTokens
            if response.wasTruncated { throw LLMError.truncated }
            scriptText = response.text
            // The script exists now; the notes have done their job.
            await notesCache?.clear(bookID: bookID)
        }

        return DigestScript(
@@ -104,9 +114,43 @@ public struct DigestGenerator: Sendable {
        )
    }

    /// One chunk's notes, taken from the cache when they are already there.
    ///
    /// Keyed by the chunk's own text, so this is safe across attempts: a note
    /// can only come back for exactly the passage it was written about.
    private func note(
        on chunk: String,
        book: DigestPrompts.Book,
        bookID: Int,
        index: Int,
        total: Int,
        inputTokens: inout Int,
        outputTokens: inout Int
    ) async throws -> String {
        if let cached = await notesCache?.note(bookID: bookID, for: chunk) {
            return cached
        }
        let response = try await client.complete(LLMRequest(
            system: DigestPrompts.notesSystem,
            user: DigestPrompts.notesUser(
                book: book, chunk: chunk, index: index, total: total
            ),
            maxOutputTokens: 2000
        ))
        inputTokens += response.inputTokens
        outputTokens += response.outputTokens
        // Deliberately kept even when the model ran into the cap: notes are a
        // means, not the product, and on a 4B model served from the Mac every
        // notes request hit it. Refusing them made a whole book impossible to
        // summarise at all.
        await notesCache?.save(response.text, bookID: bookID, for: chunk)
        return response.text
    }

    private func fold(
        notes: [String],
        book: DigestPrompts.Book,
        bookID: Int,
        available: Int,
        inputTokens: inout Int,
        outputTokens: inout Int
@@ -119,17 +163,13 @@ public struct DigestGenerator: Sendable {
            // Pairs rather than everything at once: halving each round
            // terminates, and each request stays comfortably inside the window.
            for pair in stride(from: 0, to: notes.count, by: 2) {
                try Task.checkCancellation()
                let joined = notes[pair..<min(pair + 2, notes.count)].joined(separator: "\n\n")
                let response = try await client.complete(LLMRequest(
                    system: DigestPrompts.notesSystem,
                    user: DigestPrompts.notesUser(
                        book: book, chunk: joined, index: pair, total: notes.count
                    ),
                    maxOutputTokens: 2000
                folded.append(try await note(
                    on: joined, book: book, bookID: bookID,
                    index: pair, total: notes.count,
                    inputTokens: &inputTokens, outputTokens: &outputTokens
                ))
                inputTokens += response.inputTokens
                outputTokens += response.outputTokens
                folded.append(response.text)
            }
            notes = folded
        }
+79 −0
Changes for Packages/Core/Sources/PodbookUI/Digest/DigestNotesCache.swift: 79 added lines, 0 removed lines.
Original line number Diff line number Diff line
import CryptoKit
import Foundation

/// Notes already taken on a book's chunks, kept between attempts.
///
/// Reading one chunk of a book costs one request. On a model served from the
/// Mac that request runs for minutes — measured at well over ten on a 4B model
/// with a chunk of some 23 000 tokens — and a whole book is several of them in
/// a row. A failure on the last chunk used to throw away the whole hour and
/// start the retry from nothing.
///
/// So each note is written down the moment it comes back, keyed by the content
/// of the chunk it describes rather than by its position. Change the book, the
/// chunk size or the text extraction, and the keys simply stop matching: stale
/// notes are never reused, they are just ignored.
public actor DigestNotesCache {
    private let root: URL
    /// Loaded once per book and kept, so a run of twenty chunks does not read
    /// the same file twenty times.
    private var loaded: [Int: [String: String]] = [:]

    public init(root: URL? = nil) {
        if let root {
            self.root = root
        } else {
            let base = FileManager.default.urls(for: .applicationSupportDirectory, in: .userDomainMask).first
                ?? FileManager.default.temporaryDirectory
            self.root = base.appendingPathComponent("Podbook/DigestNotes", isDirectory: true)
        }
        try? FileManager.default.createDirectory(at: self.root, withIntermediateDirectories: true)
    }

    public func note(bookID: Int, for chunk: String) -> String? {
        notes(bookID: bookID)[Self.key(chunk)]
    }

    public func save(_ note: String, bookID: Int, for chunk: String) {
        var current = notes(bookID: bookID)
        current[Self.key(chunk)] = note
        loaded[bookID] = current
        persist(bookID: bookID)
    }

    /// Called once a script has been written: the notes have served their
    /// purpose, and a book's worth of them is not worth keeping around.
    public func clear(bookID: Int) {
        loaded[bookID] = nil
        try? FileManager.default.removeItem(at: file(bookID))
    }

    public func count(bookID: Int) -> Int { notes(bookID: bookID).count }

    private func notes(bookID: Int) -> [String: String] {
        if let loaded = loaded[bookID] { return loaded }
        let onDisk = (try? Data(contentsOf: file(bookID)))
            .flatMap { try? JSONDecoder().decode([String: String].self, from: $0) }
            ?? [:]
        loaded[bookID] = onDisk
        return onDisk
    }

    private func persist(bookID: Int) {
        guard let data = try? JSONEncoder().encode(loaded[bookID] ?? [:]) else { return }
        try? data.write(to: file(bookID), options: .atomic)
    }

    private func file(_ bookID: Int) -> URL {
        root.appendingPathComponent("\(bookID).json")
    }

    /// The chunk's own content is the key. Hashed rather than stored, because
    /// a chunk is tens of thousands of characters and the key sits in a
    /// dictionary that is written to disk after every note.
    static func key(_ chunk: String) -> String {
        SHA256.hash(data: Data(chunk.utf8))
            .map { String(format: "%02x", $0) }
            .joined()
    }
}
+20 −1
Changes for Packages/Core/Sources/PodbookUI/Digest/DigestView.swift: 20 added lines, 1 removed line.
Original line number Diff line number Diff line
@@ -21,6 +21,11 @@ public struct DigestView: View {
    @State private var failure: String?
    @State private var listening: DigestScript?
    @State private var player: NarrationPlayer?
    /// Shared across attempts on purpose: that is the whole point of it.
    @State private var notesCache = DigestNotesCache()
    /// Notes already on disk for this book, shown so a retry does not look
    /// like starting over.
    @State private var keptNotes = 0

    public init(
        bookID: Int,
@@ -77,6 +82,19 @@ public struct DigestView: View {
                    if let failure {
                        Label(failure, systemImage: "exclamationmark.triangle.fill")
                            .foregroundStyle(.orange)
                        // Without this, a retry after a failure looks like
                        // starting the whole book again — and on a model
                        // served from the Mac that is an hour the person has
                        // no reason to believe they will not pay twice.
                        if keptNotes > 0 {
                            Text(
                                keptNotes == 1
                                    ? "Une partie du livre est déjà lue. Relancer reprendra là."
                                    : "\(keptNotes) parties du livre sont déjà lues. Relancer reprendra là."
                            )
                            .font(.footnote)
                            .foregroundStyle(.secondary)
                        }
                    }
                }

@@ -195,7 +213,7 @@ public struct DigestView: View {
            do {
                try meter.authorise(client.descriptor)
                let source = await text(plan.scope)
                let generator = DigestGenerator(client: client)
                let generator = DigestGenerator(client: client, notesCache: notesCache)
                let script = try await generator.generate(
                    plan: plan, bookID: bookID, title: title, author: author,
                    text: source,
@@ -209,6 +227,7 @@ public struct DigestView: View {
            } catch {
                failure = (error as? LocalizedError)?.errorDescription
                    ?? error.localizedDescription
                keptNotes = await notesCache.count(bookID: bookID)
            }
        }
    }
+91 −14
Changes for Packages/Core/Sources/PodbookUI/Digest/LLMClient.swift: 91 added lines, 14 removed lines.
Original line number Diff line number Diff line
@@ -70,11 +70,25 @@ public struct LLMResponse: Sendable, Hashable {
    public var text: String
    public var inputTokens: Int
    public var outputTokens: Int
    /// The model ran into `maxOutputTokens` instead of finishing.
    ///
    /// Reported rather than thrown, because whether it matters depends on what
    /// was being asked for. A script cut off two sentences from the end is
    /// useless; reading notes cut off at the same place are still reading
    /// notes. Measured on a 4B model served from the Mac, every single notes
    /// request hit the cap — throwing here made the whole book unsummarisable.
    public var wasTruncated: Bool

    public init(text: String, inputTokens: Int = 0, outputTokens: Int = 0) {
    public init(
        text: String,
        inputTokens: Int = 0,
        outputTokens: Int = 0,
        wasTruncated: Bool = false
    ) {
        self.text = text
        self.inputTokens = inputTokens
        self.outputTokens = outputTokens
        self.wasTruncated = wasTruncated
    }
}

@@ -90,6 +104,11 @@ public enum LLMError: Error, Sendable, Hashable, LocalizedError {
    case http(status: Int, body: String)
    case decoding(String)
    case transport(String)
    /// The request outlived its patience. Carries what to look at: a model
    /// served from the Mac is slow in a way a cloud model never is, and the
    /// bare system sentence — « La requête a expiré. » — names neither the
    /// endpoint nor the model.
    case timedOut(endpoint: String, model: String, seconds: Int)
    case budgetExceeded(spent: Double, cap: Double)

    public var errorDescription: String? {
@@ -104,6 +123,11 @@ public enum LLMError: Error, Sendable, Hashable, LocalizedError {
        case .http(let status, let body): "HTTP \(status) : \(body)"
        case .decoding(let detail): "Réponse illisible : \(detail)"
        case .transport(let detail): "Erreur réseau : \(detail)"
        case .timedOut(let endpoint, let model, let seconds):
            """
            Pas de réponse de \(model) sur \(endpoint) au bout de \(seconds) s. \
            Vérifie que le modèle est bien chargé et que le serveur tourne toujours.
            """
        case .budgetExceeded(let spent, let cap):
            String(format: "Plafond mensuel atteint : %.2f € dépensés sur %.2f €.", spent, cap)
        }
@@ -119,17 +143,32 @@ public struct OpenAICompatibleClient: LLMClient {
    private let baseURL: URL
    private let apiKey: String?
    private let transport: any HTTPTransport
    private let timeout: TimeInterval

    /// `timeoutIntervalForRequest` counts **silence**, not total time, and a
    /// non-streaming completion sends nothing at all until the answer is
    /// finished. So the whole job — reading the prompt, then writing the
    /// answer — has to fit inside it.
    ///
    /// Measured here: one chunk of about 23 000 tokens on a 4B model served by
    /// LM Studio on this Mac runs well past ten minutes. Five minutes, the old
    /// value, could not have worked, and the screen said only « La requête a
    /// expiré. » A cloud model answers in seconds and keeps the shorter leash.
    public static let localTimeout: TimeInterval = 2400
    public static let cloudTimeout: TimeInterval = 300

    public init(
        baseURL: URL,
        model: String,
        apiKey: String? = nil,
        descriptor: LLMDescriptor? = nil,
        transport: any HTTPTransport = URLSessionTransport(timeout: 300)
        transport: (any HTTPTransport)? = nil
    ) {
        self.baseURL = baseURL
        self.apiKey = apiKey
        self.transport = transport
        let timeout = Self.isLocal(baseURL) ? Self.localTimeout : Self.cloudTimeout
        self.timeout = timeout
        self.transport = transport ?? URLSessionTransport(timeout: timeout)
        self.descriptor = descriptor ?? LLMDescriptor(
            id: "openai-compatible",
            displayName: baseURL.host ?? "Compatible OpenAI",
@@ -176,12 +215,17 @@ public struct OpenAICompatibleClient: LLMClient {
            headers["Authorization"] = "Bearer \(apiKey)"
        }

        let response = try await transport.send(HTTPRequest(
        let response = try await Self.send(
            HTTPRequest(
                method: "POST",
                url: baseURL.appendingPathComponent("chat/completions"),
                headers: headers,
                body: body
        ))
            ),
            through: transport,
            model: descriptor.model,
            timeout: timeout
        )
        try Self.check(response)

        struct Answer: Decodable {
@@ -207,13 +251,41 @@ public struct OpenAICompatibleClient: LLMClient {
        guard let choice = answer.choices.first, let text = choice.message?.content else {
            throw LLMError.decoding("aucun contenu dans la réponse")
        }
        if choice.finish_reason == "length" { throw LLMError.truncated }

        return LLMResponse(
            text: text,
            inputTokens: answer.usage?.prompt_tokens ?? 0,
            outputTokens: answer.usage?.completion_tokens ?? 0
            outputTokens: answer.usage?.completion_tokens ?? 0,
            wasTruncated: choice.finish_reason == "length"
        )
    }

    /// Without this, a `URLError` walks straight out of the client and reaches
    /// the screen as the system's own sentence, which names neither the model
    /// nor the endpoint — the two things to look at when a model on the Mac
    /// stops answering.
    static func send(
        _ request: HTTPRequest,
        through transport: any HTTPTransport,
        model: String,
        timeout: TimeInterval
    ) async throws -> HTTPResponse {
        do {
            return try await transport.send(request)
        } catch let error as URLError {
            let endpoint = request.url.absoluteString
            switch error.code {
            case .timedOut:
                throw LLMError.timedOut(
                    endpoint: endpoint, model: model, seconds: Int(timeout)
                )
            case .cannotConnectToHost, .cannotFindHost, .networkConnectionLost:
                throw LLMError.transport(
                    "\(endpoint) injoignable — \(error.localizedDescription)"
                )
            default:
                throw LLMError.transport("\(endpoint) — \(error.localizedDescription)")
            }
        }
    }

    static func check(_ response: HTTPResponse) throws {
@@ -250,12 +322,13 @@ public struct AnthropicClient: LLMClient {
        model: String = AnthropicClient.defaultModel,
        effort: String = "high",
        baseURL: URL = URL(string: "https://api.anthropic.com/v1")!,
        transport: any HTTPTransport = URLSessionTransport(timeout: 300)
        transport: (any HTTPTransport)? = nil
    ) {
        self.apiKey = apiKey
        self.effort = effort
        self.baseURL = baseURL
        self.transport = transport
            ?? URLSessionTransport(timeout: OpenAICompatibleClient.cloudTimeout)
        self.descriptor = LLMDescriptor(
            id: "anthropic",
            displayName: "Anthropic",
@@ -295,7 +368,8 @@ public struct AnthropicClient: LLMClient {
            output_config: OutputConfig(effort: effort)
        ))

        let response = try await transport.send(HTTPRequest(
        let response = try await OpenAICompatibleClient.send(
            HTTPRequest(
                method: "POST",
                url: baseURL.appendingPathComponent("messages"),
                headers: [
@@ -304,7 +378,11 @@ public struct AnthropicClient: LLMClient {
                    "anthropic-version": "2023-06-01",
                ],
                body: body
        ))
            ),
            through: transport,
            model: descriptor.model,
            timeout: OpenAICompatibleClient.cloudTimeout
        )
        try OpenAICompatibleClient.check(response)

        struct Answer: Decodable {
@@ -341,8 +419,6 @@ public struct AnthropicClient: LLMClient {
                ?? "sans précision"
            throw LLMError.refused(reason)
        }
        if answer.stop_reason == "max_tokens" { throw LLMError.truncated }

        let text = answer.content
            .filter { $0.type == "text" }
            .compactMap(\.text)
@@ -354,7 +430,8 @@ public struct AnthropicClient: LLMClient {
        return LLMResponse(
            text: text,
            inputTokens: answer.usage?.input_tokens ?? 0,
            outputTokens: answer.usage?.output_tokens ?? 0
            outputTokens: answer.usage?.output_tokens ?? 0,
            wasTruncated: answer.stop_reason == "max_tokens"
        )
    }
}
Loading