What is an EPUB?

An EPUB is a small website zipped into one file: Inkshelf follows container.xml to the package document, and the package document's spine gives it the chapters, in reading order.

A comic is easy to show: it is a stack of pictures, one per page. A novel is not. Its text has no pages until someone lays it out for a screen and a font size. The file format that carries such books is EPUBEPUB: A zip of web pages (XHTML), stylesheets, images and fonts, plus a package file that says in what order to read them., and before Inkshelf can draw a single word it has to find the words: which files are chapters, in what order, styled how.

This chapter opens an EPUB. The next one turns a chapter into something the reader can lay out.

📦

Think of a website saved for offline reading, zipped, with a note on top saying "the index is in this folder". The pages are the chapters, the stylesheets are the book's design, and the note is there so that any program can find its way in.

A website in a zip

Inside an EPUB you find web files: chapters written in XHTML (strict HTML), CSS stylesheets, images and sometimes fonts. Three more files are there for programs, not for readers:

• mimetype, first in the zip and stored without compression. It holds the 20 characters application/epub+zip, so a program can recognize an EPUB from its first bytes without unzipping anything.
• META-INF/container.xml, the only file whose path is fixed. It says where the package document is: books put it in OEBPS/, EPUB/ or at the root, as they like.
• The package document (the .opf file), the book's description: what files it has and in what order to read them.

The book below is the one Inkshelf's own tests open (in epub_test.go): a cover, two chapters, a page of notes, a stylesheet, a navigation document.

InteractiveUnzipping a book

EPUB

book.epub

one file, a zip in disguise

Its first bytes, read as text:

PK 03 04bytes 0–3: a zip entry… 26 bytes …4–29: sizes, datesmimetype30–37: its nameapplication/epub+zip38–57: stored, not compressedPK 03 0458: next entryMETA-INF/container.xmlits namexœ³±È…compressed

To your computer this is one file. But rename it book.zip and any zip tool opens it: an EPUB is a folder of web files, zipped.

Try this: Unzip it, then play the steps: each one opens the file that container.Open reads next. Click any file to see what is inside; the arrows show which files it points to.

Inkshelf never unzips the book to disk. Go's archive/zip reads the files straight from the EPUB, one at a time, when they are needed. And it does not check mimetype: it downloaded the file as a book, it knows what it is.

The package document: manifest, spine and navigation

The Package document (OPF)Package document (OPF): The EPUB’s table of contents for machines: metadata, the manifest of every file, and the spine. has three parts that matter:

• The metadata: the title, the language, the authors, the identifiers, and whether the book is reflowable or fixed layout (more on that below).
• The ManifestManifest: The list of every file in the EPUB, each with an id and a media type.: every file of the book, each with an id, a path and a media type. The manifest is a list of what exists, not of what to read.
• The SpineSpine: The reading order: a list of manifest ids. Items marked linear="no" are reached only by links (notes, for example).: the reading order, as a list of itemrefs, each naming a manifest id.

An itemref with linear="no" is part of the book but not of the reading flow: notes, answers to exercises. Inkshelf keeps every spine item (Book.Spine), but only the linear ones become its chapters (Book.Chapters), the ones "Next page" walks through. A non-linear item is shown only when a link leads to it, in a popup over the page.

A manifest item may also name a fallback: "if you cannot show me, show this one instead". A book can offer a PDF for readers that read PDF, with an XHTML version behind it. firstSupported follows that chain until it finds an item it can show (XHTML or an image), and stops at the first item it has already seen, because books do contain loops. An item marked scripted that has a fallback goes to its fallback too: Inkshelf runs no scripts.

InteractiveSpine, manifest, chapters

notes is linear="no"The last itemref points to:

chapter 1/3

Chapter 0 (counted from 0, 3 in all): itemref coverpage, shows text/cover.xhtml.

Try this: Press play to turn pages through the chapters. Untick linear="no" and play again: the notes join the flow. Then point the last itemref at a PDF with fallbacks, or at a loop.

The table of contents a person sees is a separate file again: the Navigation documentNavigation document: The human table of contents of an EPUB 3, an XHTML page with a <nav> list (or the older NCX file)., an XHTML page whose <nav epub:type="toc"> holds a nested list of links. EPUB 2 books have an older XML file instead, the NCX; Book.Navigation reads the nav, and the NCX only if the nav gives no table of contents.

Locked books

A zip can hold encrypted files, listed in META-INF/encryption.xml. Books sold with DRM encrypt their chapters, and Inkshelf cannot read those: container.Open refuses them with ErrDRM, "the book is protected by DRM".

But many books that are not locked at all have an encryption.xml too. Publishers obfuscate the fonts they embed, so that nobody copies them out of the book: the first 1040 bytes (IDPF's method) or 1024 bytes (Adobe's) are mixed with a key made from the book's identifier. That is not a lock, since the key is in the book. So checkEncryption accepts these two algorithms and refuses any other.

InteractiveWhat checkEncryption accepts

• ✓ OEBPS/fonts/serif.otf is encrypted with http://www.idpf.org/2008/embedding: only a font obfuscation, fine. Its path is noted, to undo it when the font loads.

Open goes on to META-INF/container.xml.

Try this: Switch between the three cases. Only an algorithm other than the two font obfuscations stops the book.

Reflowable or fixed layout

Most books are ReflowableReflowable: A book whose text is laid out again for the screen and font size, like a web page, as opposed to fixed layout.: the text has no pages of its own. The reader lays it out for its screen and its text size, and lays it out again when you change either. That is what the next chapters are about.

Some books are Fixed layoutFixed layout: A book whose pages have a set size and position for everything, like a PDF; Inkshelf draws each page whole.: comics, children's books, cookbooks. Each spine item is one page designed at a fixed size, with text and pictures placed exactly. Reflowing them would wreck the design, so they are only scaled to fit the screen. The package's metadata says which: rendition:layout is pre-paginated for fixed layout, and an itemref may override it for its own page.

InteractiveThe same page, reflowable and fixed

Text size44 px

Reflowable

Chapter 1

The night was dark, and the rain had not stopped since noon. Nobody in the village went out.

Only the old ferryman sat by the river, waiting for a passenger who had promised to come.

At midnight a lantern appeared on the far bank. It swung slowly, as if the hand that held it were tired.

page 1 of 1

Laid out again for this screen and this size: the lines break elsewhere, the pages are counted again.

Fixed layout (pre-paginated)

Designed at 600×800, scaled ×2.34 to fit. The text size does nothing: the page's text is part of its picture.

Not to scale: both screens are drawn smaller, and the reflowable text a little larger than it would be.

Try this: Change the text size and watch the reflowable page count; the fixed page does not move. Then switch screens: the fixed page keeps its shape and leaves bands.

How Inkshelf does it

container.Open runs once per book, in this order:

1. checkEncryption: no encryption.xml, or only obfuscated fonts, else ErrDRM.
2. packageDocumentPath reads META-INF/container.xml. If it names several package documents (other renditions of the book), the first one is the book.
3. readMetadata, readManifest (which also finds the nav, the NCX and the cover image) and readSpine, which follows fallbacks and keeps the linear items as chapters. A book with no chapter fails to open, and so does one whose spine names an id missing from the manifest, or a chapter missing from the zip.

Package epub then decides what each chapter becomes (documentOf in chapter.go):

• a pre-paginated item becomes one image block, its page drawn whole;
• an item nothing can show becomes one line: "This part of the book cannot be shown.";
• an image becomes an image block;
• XHTML is built into a document: the subject of the next chapter.

A book whose chapters are all pre-paginated (Book.FixedLayout) takes another road at download time: each page is drawn by Book.RenderPage, prepared like a comic page, and the EPUB is deleted. From then on it is read exactly like a comic.

Going deeper

• Paths. Every path in container is a path inside the zip. A link is relative to the file it is in, and is percent-decoded: the test book's images/cover%20page.png is the file OEBPS/images/cover page.png. A .. at the root stays at the root.
• Media types books get wrong. A spine item with no usable fallback but a media type that mentions xml or html (some books write text/xml), or none at all, is read as XHTML anyway.
• Mixed books. In a book that is partly reflowable, a pre-paginated chapter's document is a single image block whose path starts with rendered-page:. When the layout asks for that image, Book.Resource draws the page at 1404×1872, the InkPad 3's screen.
• Fonts. For an obfuscated font, Inkshelf tries each of the book's identifiers as the key (Adobe's key is a UUID that books often give as a second identifier), then recovers Adobe's key from the font's own header.
• Encodings. The test book's second chapter is written in Latin-1, not UTF-8. Inkshelf decodes it: "Café crème." comes out right.

Where it lives in the code

• internal/ebook/epub/container/container.go · OpenOpens the zip and loads the book: encryption check, container.xml, package document.
• internal/ebook/epub/container/encryption.go · checkEncryptionRefuses DRM with ErrDRM; accepts the two font obfuscations.
• internal/ebook/epub/container/package_document.go · readSpineThe reading order, fallbacks followed, linear items kept as chapters.
• internal/ebook/epub/container/package_document.go · firstSupportedWalks a manifest fallback chain, stopping at a loop.
• internal/ebook/epub/container/navigation.go · NavigationThe table of contents, page list and landmarks, from the nav or the NCX.
• internal/ebook/epub/chapter.go · documentOfWhat a spine item becomes: a built document, an image, a drawn page, or a line of apology.
• internal/ebook/epub/prepaginated.go · FixedLayoutWhether every chapter is pre-paginated: a book of pages.
• internal/shelf/prepare_fixed_layout.go · downloadFixedLayoutTurns a fixed-layout EPUB into comic pages at download time.

The full story: From an EPUB to pixels, and what Inkshelf passes of the W3C's tests in EPUB 3.3 conformance.

Next: a chapter is XHTML and CSS. From XHTML and CSS to a document turns it into something the layout code can use.

Check yourself

Which file of an EPUB has a path that never changes?

The test book’s spine has four itemrefs, one of them linear="no". How many chapters does Inkshelf count?

A book’s encryption.xml lists its fonts with the IDPF obfuscation algorithm. What happens?

You make the text bigger. What happens to a pre-paginated page?