If you just graduated with a computer science degree and you have any interest whatsoever in being involved in any kind of startup, or being one of a few people in a small company with a lot of responsibility, now is the time to do it - when you’re young.

If you don’t have any interest in that, don’t do it - don’t do it just because you think it’s what people do. But if you’re like, “Oh, I always wanted to be one of five programmers working on a project” or something, “I always wanted to be involved in a startup” or whatever, now is the time to do it.

It will only get harder to do that later. Having a job like that early in your career, where you were one of a small number of people, will force you to learn how to do a whole bunch of stuff, and that will make you a much more valuable employee when you get tired of the startup world or when you want to go to a company that’s not going to go under. Later, you will have so much more real-world experience and knowledge than people who went to work for IBM right out of school - to throw IBM under the bus - or went to work for some big company.

When you go to a big company, you do learn things on the job, but it’s a much more stable environment in terms of what’s expected of you. You don’t go in there and suddenly have seventeen jobs and have to learn them all now. It’s going to be more sustainably paced, let’s say, but you will learn less; it will take you longer, and your skills will be more narrow because there are 75,000 other people who do their own specialized jobs.

That’s not to say, “Do startups when you’re young because that’s the time to burn you out”. You shouldn’t have burnout even when you’re young. What I am saying is that if you are in a company with a small number of people, it can still be a healthy work environment, and you will still be required to learn how to do way more jobs simply because there just aren’t enough people. Someone’s got to figure out how to administer this Linux machine - congratulations, you’re the sysadmin now. Someone’s got to learn this new API or this new language - congratulations, it’s you and one other person.

You will learn so much and you will be battle-tested. And when your company inevitably goes under - because that’s what happens to most startups - when you apply for that job at Apple or Google or whatever, you should look a lot better than the other candidates because you will literally know how to do more stuff.

- John Siracusa on ATP episode 647 ▸ 1:34:07

Great discussion from the ATP folks, highly recommend listening to the entire segment.

Pagefind is a fully static search library that aims to perform well on large sites, while using as little of your users’ bandwidth as possible, and without hosting any infrastructure.

Delightful project I accidentally stumbled upon while building this very blog. It pre-computes all the search indexes at build time and then packages into a gloriously simple frontend. Who knows if I’ll ever post enough that it’s worth using. For the time being, it lives at /search.

It spits out a directory structure like this:

dist/pagefind
├── fragment
│   ├── en_4733ec7.pf_fragment
│   ├── en_5c9a98e.pf_fragment
│   ├── en_7ca223a.pf_fragment
│   └── en_953c689.pf_fragment
├── index
│   └── en_4d96258.pf_index
├── pagefind-entry.json
├── pagefind-highlight.js
├── pagefind-modular-ui.css
├── pagefind-modular-ui.js
├── pagefind-ui.css
├── pagefind-ui.js
├── pagefind.en_f57a1155c8.pf_meta
├── pagefind.js
├── wasm.en.pagefind
└── wasm.unknown.pagefind

The .js, .css, and even the wasm stuff all made sense, but I was curious about the binary blobs in the .pf_fragment, .pf_index, and .pf_meta files.

Weirdly (and somewhat ironically), I could not find any documentation on the actual binary format the indexes were being stored as. I poked around a bit before deciding to dig into the source code.

With the help of Claude, I’ve figured out that they’re using Concise Binary Object Representation via the minicbor Rust lib and sort of pieced together the root data structures. I have linked to them below.

.pf_fragment

#[derive(Serialize, Debug, Clone)]
pub struct PageFragmentData {
    pub url: String,
    pub content: String,
    pub word_count: usize,
    pub filters: BTreeMap<String, Vec<String>>,
    pub meta: BTreeMap<String, String>,
    pub anchors: Vec<PageAnchorData>,
}

#[derive(Serialize, Debug, Clone)]
pub struct PageAnchorData {
    pub element: String,
    pub id: String,
    pub text: String,
    pub location: u32,
}

Code

.pf_index

/// A single word index chunk: `pagefind/index/*.pf_index`
#[derive(Encode)]
pub struct WordIndex {
    #[n(0)]
    pub words: Vec<PackedWord>,
}

/// A single word as an inverse index of all locations on the site
#[derive(Encode, Clone, Debug)]
pub struct PackedWord {
    #[n(0)]
    pub word: String,
    #[n(1)]
    pub pages: Vec<PackedPage>,
}

/// A set of locations on a given page
#[derive(Encode, Clone, Debug)]
pub struct PackedPage {
    #[n(0)]
    pub page_number: usize, // Won't exceed u32 but saves us some into()s
    #[n(1)]
    pub locs: Vec<i32>,
}

Code

.pf_meta

/// All metadata we need to glue together search queries & results
#[derive(Encode, Debug)]
pub struct MetaIndex {
    #[n(0)]
    pub version: String,
    #[n(1)]
    pub pages: Vec<MetaPage>,
    #[n(2)]
    pub index_chunks: Vec<MetaChunk>,
    #[n(3)]
    pub filters: Vec<MetaFilter>,
    #[n(4)]
    pub sorts: Vec<MetaSort>,
}

/// Communicates the pagefind/index/*.pf_index file we need to load
/// when searching for a word that sorts between `from` and `to`
#[derive(Encode, PartialEq, Debug)]
pub struct MetaChunk {
    #[n(0)]
    pub from: String,
    #[n(1)]
    pub to: String,
    #[n(2)]
    pub hash: String,
}

#[derive(Encode, Debug)]
pub struct MetaPage {
    #[n(0)]
    pub hash: String,
    #[n(1)]
    pub word_count: u32,
}

Code