Downloads and the shelf
book.json last, so an interrupted download is simply resumed.Books live on servers: Komga for comics and manga, Mealie for recipes, Calibre-Web for EPUBs and CBZs. A book is read on the reader, often far from Wi-Fi, so it is downloaded first, onto the ShelfShelf: The downloaded books on the reader, one folder each, with book.json written last.. Two constraints shape how. The reader is slow: preparing a page for the panel takes real CPU time, so it is done once, at download time, not at every page turn. And the reader is unreliable: Wi-Fi drops, the battery dies, the reader falls asleep. A download must survive all of that without ever leaving a broken book behind.
From a server to the reader
The app asks every kind of server for books through one Go interface, servers.Server
(Three servers, one interface). What matters here is the book's Format: a
book of PageImages (Komga's, and Mealie's recipes) is fetched page by page through
OpenBook; a CBZ or an EPUB is fetched whole through OpenFile.
The download pipeline
Tapping a book that is not on the reader calls downloadqueue.EnqueueLocked. Books download one
at a time, in the order asked: each is ready as soon as it can be, instead of all of them late.
If the queue was idle, the main thread starts the downloader with state.Go, and the
downloader calls shelf.Store.Download for each book in turn.
For a book of page images (Komga's), Download runs two groups of workers joined by a small
queue:
- Fetchers (
fetchWorkers = 3) take the page numbers in order and ask the server for each page. Only a few at once: more only slows the server down. - A queue of fetched pages (
fetchedQueue = 2): enough to keep the fetchers busy, few enough not to hold many pages in memory. - Preparers, one per core (
prepareWorkers, at least 2, at most 8, as each holds a decoded page of 20 to 40 MB), turn each page into a .pg4.pg4: Inkshelf’s page image file: a small header, then two pixels per byte (4 bits each), ready to draw. file: decode, fit to the screen, apply the Tone curveTone curve: A table that maps each input gray to an output gray: here it stretches the contrast and darkens the mid-tones for e-ink., ditherDithering: Hiding the lack of grays by spreading each pixel’s rounding error onto its neighbours, so the eye sees the right average. to 16 grays16 grays: The InkPad 3 shows 16 levels from black to white, 17 apart on the 0–255 scale: 0, 17, 34, … 255., pack two pixels a byte.
The progress the app shows is not "pages done" but "pages 1 to n all on disk": a page that
finishes early waits until every page before it is there (finishPage). That is the number
of pages you could read right now.
A CBZ goes through the same workers, but its file is fetched whole first and its pages taken out
of it; the archive is deleted once the pages are prepared. A ReflowableReflowable: A book whose text is laid out again for the screen and font size, like a web page, as opposed to fixed layout. EPUB is
fetched whole and kept as book.epub: it is read through once to build its index.json, its
cover becomes cover.pg4, and its pages are laid out later, while reading.
Never half a file
A reader can die at any instant. Two rules make that harmless.
Every file is written under a temporary name, then renamed. writeFileAtomic writes
0007.pg4.tmp, then renames it 0007.pg4. A rename is an
Atomic renameAtomic rename: Write a file under a temporary name, then rename it: other code sees the old file or the whole new one, never half.: the file under its real name is either absent or whole. A
.tmp cut short by a crash has a name no code reads. A CBZ or an EPUB is streamed to
book.cbz.part or book.epub.part the same way, and renamed once every byte is in.
book.json is written last. It holds the book's titles, page count, format and progress.
Store.Book reads it to decide whether a book is downloaded. So a folder without it is,
by definition, an interrupted download. The next Download of that book picks it up: the
fetchers check pageOnDisk and skip every page already there.
The reader died in the middle of this download: whole pages, a .tmp cut short, no book.json. Tap to download again, or start a new download and pull the plug yourself.
The EPUB tab shows the difference between the two failures. Pulling the plug leaves
book.epub.part; the next download creates it again, empty, and starts over (a partial EPUB is
never resumed, only a whole book.epub is kept). Wi-Fi dropping is different: the request
would wait forever, so a watchdogWatchdog: A timer that gives up on a download when no data has arrived for 30 seconds. gives up after 30 s without a byte
(stallTimeout), and
the download fails with "the download stalled", removing the .part.
How Inkshelf does it
downloadqueue.downloadercallsStore.Downloadwith a progress callback. The callback is anUpdatewhose change,noteProgressLocked, asks for a redraw only on the screens that show it, for the first page, then at most every 2 s (progressRepaint): each redraw takes the CPU from the download.Downloadpasses progress through areporter: a goroutine of its own and a channel that keeps only the latest value, so a slow callback never holds up the workers.- While downloading, the reader is kept awake:
pocketbook.PreventSleepFor(awakeFor), one minute, renewed every 30 s. - After a book, the downloader saves its cover for offline use and, inside one
Update, records the result and takes the next book from the queue.
Going deeper
- The first error stops all.
finishPagerecords the first error and cancels the context, which stops the other workers; the preparers drain the queue so that no fetcher stays blocked. - Old books.
pageOnDiskalso accepts0001.png: books downloaded by older versions keep working. - Fixed-layout EPUBs, whose pages are drawn like pictures, are drawn page by page into
.pg4files at download time and kept as a book of pages. - The HTTP client.
httpclient.Newmakes the one client every request goes through, with a one-minute timeout; Calibre-Web downloads files through a copy without that timeout, which is why files need the watchdog. - Cancelling. Tapping the book being downloaded cancels its context; the list then says "Cancelled". Whatever was written stays, ready to resume.
Where it lives in the code
internal/app/downloadqueue/downloadqueue.go · EnqueueLocked, downloaderThe queue, one book at a time, and the goroutine that downloads it.internal/shelf/download.go · Store.Download, fetchLoop, prepareLoopFetchers and preparers joined by a queue of two.internal/shelf/download_file.go · fetchFile, stallTimeoutA whole file to .part, renamed once complete; the 30 s watchdog.internal/shelf/write_atomic.go · writeFileAtomicWrite to .tmp, then rename.internal/shelf/store.go · Store.Book, Store.Savebook.json: read to know a book is there, written last.
The full story: Files on the reader.
Next: once a book is on the reader and you read it, your progress has to go back to the server: Syncing reading progress.