Software estimation that doesn't suck

2025-12-04T00:00:00+00:00

You've been asked to give an estimate about when you're gonna be able to ship some code. Fine fine, except that there's something riding on this… a trade show, seasonal deadline, etc. Point is: there's another part of the business depending on you doing a good job here.

But software estimates are notoriously inaccurate, to the degree that people frequently double or halve them depending on what they think of the team. Think about that for a sec: a change of 50% in either direction based solely on personal factors. Absolutely wild. 

Part of that is that software engineers have a weird aversion to figuring out how to do better here. As a result, a lot of people never get better than off-the-cuff "I can knock that out in a week" style estimates, and develop a reputation for being inaccurate. These skills are worth learning, though, if only because giving useful estimates means that you have the time you need to do the rest of your job correctly! You can also build credibility as someone who understands how to navigate complex systems, and end up with more of a say in the processes you're a part of.

I'm going to share a process that I've personally used to prepare and update estimates for all sorts of software projects. It's based based off the advice in Software Estimation: Demystifying the Black Art</a> by Steve McConnell. I'm not following this book to the letter, but the process ends up being enough to communicate with to some degree of accuracy. That's fine, since that's typically what you're really after. However, it's possible that I've fatally compromised some important statistical property. If you read this and you notice, I'd appreciate a heads-up!

This is going to be a longer post, but here's the overall process in five steps:

Break down the work into high-level outcome-focused areas.</li>
Break each item down into tasks small enough to feel familiar.</li>
Go to your team and ask them to give a "about this long" guess as well as a best case and a worst-case for each item.</li>
Do a little math to get a range based on percentage confidence.</li>
Keep your estimates up to date as the project progresses.</li> </ol>
This has worked well for me in companies which lean more towards big-plan-up-front as well as companies that try to stick with a more "agile" approach.
However, sometimes people read this and think "oh no! It's waterfall!" and bounce off it without getting any benefit. So: there's nothing about this process that prevents you from breaking the work into smaller independently-releasable components, or shrinking the project scope, or putting things behind feature flags for beta users, or whatever release strategy you're comfortable with. If you're in an organization that's all-in on agile, you could even use this process to just make sure you're not overcommitting in a single sprint.
It's also usually true that scope or project management style are things you're inheriting instead of something you have direct influence over. Even when you do, working up a proper estimate builds long-term credibility that lets you have more influence over the system.1</a>
High-Level Breakdown</h2>
To get our top level breakdown, we need to address two problems:

People usually communicate estimates as single points, but ranges are more realistic. There are always a bunch of possible outcomes, and we need to be able to cover them. So instead of saying "3 weeks", you might say "we're 90% confident that we'll land it in 1–5 weeks." That is, given all our possible outcomes, 90% of them end up shipping in that time. If you must give a single point, give the upper end of the range.</li>
People are really bad at giving percentage-confident estimates to work up a range. That gets worse as the range expands: you could easily give me a 90% confident estimate for the weight of a glass of water, but if I ask you for the weight of all the water in Lake Michigan you're likely to get it wrong by a couple orders of magnitude. (If you want to try, make a guess and then check the answer in the footnotes.2</a>)</li> </ul>
We'll have to overcome both issues to succeed. To start, we're going to break down work into high level areas. Say we're adding emoji reactions to posts on some businessware app. We might think of these as the high-level components:

Clicking the reaction button and selecting an emoji adds it to the post.</li>
Clicking an existing reaction adds +1 to it along with your name. If you had already done that, it removes the +1 instead.</li> </ul>
From here, we'd break this down into some finer-grained work items:

Add reaction button to posts</li>
Handle request to add reaction to post</li>
Send reactions+attribution to the frontend along with post payload</li>
Show people who reacted on hover</li>
Add +1 to the reaction on click

Uses the same backend route as adding a new reaction, just a slightly different UX</li> </ul> </li>
Remove your own reaction if clicked when present</li>
Handle request to remove reaction from post</li> </ul>
You might get these by talking through the functionality with a designer or PM. Starting out high-level and user-focused usually helps to ensure you don't miss anything.
In addition to breaking down the work, this is a good place to make sure that we're not missing any implicit work that we need to do. Missing items are the biggest reason why my estimates have been inaccurate in the past, so making sure to consider all the work in the project—even the indirect or wrap-up stuff—helps a lot!
For example, does this work require a refactor? What's new/unusual about this? What happens in failure modes? I ask questions about the project using something I call "the feature-proofing checklist</a>." As I said, missing work has been the biggest source of inaccuracy when preparing estimates, so don't skip it!
Task-Level Breakdown</h2>
Next we need to break these tasks down further to prepare them for estimation. You'll know something is small enough if you could imagine it being in a single PR. If your team regularly uses huge PRs, break it down to items that would take less than two days instead.
You'd do this for all of the tasks, but for the sake of keeping things brief, I'm only going to do it for a couple:

Add reaction button to posts

Upgrade post component to our new standard pattern</li>
Add reaction button, styles, event handlers with DOM tests</li>
Test for keyboard and screen reader accessibility</li> </ul> </li>
Handle request to add reaction to post

Add API endpoint to handle POST</li>
Add database table to store many-to-many relationship between posts and users with reaction data</li> </ul> </li> </ul>
I'm spitballing here, since this isn't a real app, but let's pretend that the existing post component is pretty messy. Maybe it's old code—one of the first things added in this project—and it hasn't had any love in a while. This is the perfect time to upgrade it, since doing so will remove incidental complexity and make future updates faster.
The important thing here is to go from "weight of Lake Michigan" to "weight of a glass of water": you want to get these into small enough chunks that the work gets familiar enough that people can imagine actually doing it. This is something you were probably going to have to do anyway to make the work legible to stakeholders; you might as well get the bonus of an accurate estimate.
How Big is Each Task?</h2>
Next, we'll prepare time-based estimates. We need three numbers for each task in your breakdown. Specifically:

The normal-case estimate.

What's your guess about how long this item will take?</li>
Aim for 50% confidence on these tasks. Half the time you do this task, it should take longer. The other half, it should take shorter.</li> </ul> </li>
The worst-case estimate.

If everything went wrong—broken builds, libraries not having needed features, unexpectedly-complex code—how long would it take? I like to use the guideline of "how long would this have to take before we would choose to take a different approach?"</li>
This should almost always be quite a bit longer than the normal-case estimate.</li>
Aim for 90% confidence here. (If you did this task 10 times, 9 of them would take less time than the worst-case estimate.)</li> </ul> </li>
The best-case estimate.

If someone got into some amazing flow state and everything magically fell into place, how long would it take?</li>
This should always be shorter than the normal-case estimate, although for very short tasks it may not be much shorter.</li>
Aim for 90% confidence again here. (If you did this task 10 times, 9 of them would take more time than the best-case estimate.)</li> </ul> </li> </ul>
It will probably help to put these numbers in a spreadsheet!
Even though it's a bit more work, this is the best place to get your team involved. You'll find out areas that need to be broken down further, as well as areas that you thought would be difficult but maybe won't be. These are some of my favorite discussions when planning, since it gets assumptions into the open where we can deal with them, and reveals differences in opinion between different team members.
This might seem like a lot, but it usually goes quickly. Don't skip all three numbers, either: if you ask people for just one number you'll usually get something that's closer to the best-case estimate than the normal case. As we'll see in a minute, that'll mean an overly-optimistic final estimate!
For our example, let's say we talked with our team and got these estimates:
Item</th> Worst Case</th> Normal Case</th> Best Case</th></tr></thead>

Upgrade post component to new standard pattern</td> 3 days</td> 1 day</td> 4 hours</td></tr>
Add reaction button, styles, event handlers with DOM tests</td> 4 days</td> 2 days</td> 1 day</td></tr>
Test for keyboard and screen reader accessibility</td> 1.5 days</td> 4 hours</td> 2 hours</td></tr>
Add API endpoint to handle reaction POST</td> 2 days</td> 1 day</td> 4 hours</td></tr>
Add database table to store many-to-many relationship between posts+users with reaction data.</td> 2 days</td> 2 hours</td> 1 hour</td></tr> </tbody></table>
Building an Estimate</h2>
Finally we're getting to the point where we can produce something other people would recognize as an estimate!
First, we'll need to calculate the "expected case." In this style of estimation, that's pretty simple: just sum up the normal case times. If you've kept to a standard of 50% confidence for these, you're doing well! For the part of this project we're using as an example, we get 4 days 6 hours (which I'm choosing to call 4.75 days for maths's sake.)
Item</th> Worst Case</th> Normal Case</th> Best Case</th></tr></thead>

Upgrade post component to new standard pattern</td> 3 days</td> 1 day</td> 4 hours</td></tr>
Add reaction button, styles, event handlers with DOM tests</td> 4 days</td> 2 days</td> 1 day</td></tr>
Test for keyboard and screen reader accessibility</td> 1.5 days</td> 4 hours</td> 2 hours</td></tr>
Add API endpoint to handle reaction POST</td> 2 days</td> 1 day</td> 4 hours</td></tr>
Add database table to store many-to-many relationship between posts+users with reaction data.</td> 2 days</td> 2 hours</td> 1 hour</td></tr>
Total</td> 12.5 days</td> 4.75 days</td> 2.375 days</td></tr> </tbody></table>
Our next step will be to turn this into a range by getting the standard deviation and then using it to create a percent-confidence-based estimate.
If you have fewer than 10 tasks^{3</a>, this math is pretty easy: stddev = (sum(worstCase) - sum(bestCase)) / 3.3</code>. (You're just going to have to trust me on the magic constant for now!)}
In our case, that's (12.5 - 2.375) / 3.3</code>, or 3.07.
However, if you have 10 tasks or more it's better to calculate (worstCase - bestCase) / 3.3</code> for each individual task, then square it to get the variance. To get the final standard deviation, we'll take the square root of the sum of the variances. Like this:
Item</th> Worst Case</th> Normal Case</th> Best Case</th> Stddev</th> Variance</th></tr></thead>

Upgrade post component to new standard pattern</td> 3 days</td> 1 day</td> 4 hours</td> 0.756</td> 0.574</td></tr>
Add reaction button, styles, event handlers with DOM tests</td> 4 days</td> 2 days</td> 1 day</td> 0.909</td> 0.826</td></tr>
Test for keyboard and screen reader accessibility</td> 1.5 days</td> 4 hours</td> 2 hours</td> 0.379</td> 0.143</td></tr>
Add API endpoint to handle reaction POST</td> 2 days</td> 1 day</td> 4 hours</td> 0.455</td> 0.207</td></tr>
Add database table to store many-to-many relationship between posts+users with reaction data.</td> 2 days</td> 2 hours</td> 1 hour</td> 0.568</td> 0.323</td></tr> </tbody></table>
That's a total variance of 2.073, for a final standard deviation of 1.440.
(That's pretty different than the group formula's answer of 3.07; that's because with few data points we can't take advantages of the law of large numbers and have to assume that we're 90% confident in the entire data set instead of in each individual point.)
Finally, you take your expected case and compute a range based on what percentage confidence you want to communicate:
Percentage Confident</th> Calculation</th></tr></thead>

2%</td> expected - 2 * standard deviation</td></tr>
10%</td> expected - 1.28 * standard deviation</td></tr>
16%</td> expected - standard deviation</td></tr>
20%</td> expected - 0.84 * standard deviation</td></tr>
25%</td> expected - 0.67 * standard deviation</td></tr>
30%</td> expected - 0.52 * standard deviation</td></tr>
40%</td> expected - 0.25 * standard deviation</td></tr>
50%</td> expected</td></tr>
60%</td> expected + 0.25 * standard deviation</td></tr>
70%</td> expected + 0.52 * standard deviation</td></tr>
75%</td> expected + 0.67 * standard deviation</td></tr>
80%</td> expected + 0.84 * standard deviation</td></tr>
84%</td> expected + standard deviation</td></tr>
90%</td> expected + 1.28 * standard deviation</td></tr>
98%</td> expected + 2 * standard deviation</td></tr> </tbody></table>
So for our project, using the individual standard deviation calculation, we'd get a 90% confident estimate by calculating 4.75 + 1.28 * 1.44</code>, or 6.59 days. Filling out the table, we get:
Percentage Confident</th> Estimate</th></tr></thead>

2%</td> 1.87</td></tr>
10%</td> 2.91</td></tr>
16%</td> 3.31</td></tr>
20%</td> 3.54</td></tr>
25%</td> 3.79</td></tr>
30%</td> 4.00</td></tr>
40%</td> 4.38</td></tr>
50%</td> 4.75</td></tr>
60%</td> 5.11</td></tr>
70%</td> 5.50</td></tr>
80%</td> 5.96</td></tr>
84%</td> 6.19</td></tr>
90%</td> 6.59</td></tr>
98%</td> 7.63</td></tr> </tbody></table>
Communicating the Estimate</h2>
We've all heard Spock or a robot or whatever say things like "there is a 95.83462% chance of failure!" There's even a TV Tropes page on it.</a> But when you're communicating estimates? Don't do that!
We can calculate our estimates to whatever precision we want. Since our units are days, calculating to the hundredths place means we're claiming a precision of 14 minutes and 24 seconds.
This seems reasonable until you remember that our inputs are… well, if you're doing this process for the first time, let's call them "fuzzy." Even when you're trying to get to 90% accuracy, you might get lucky and hit 70% the first time.
But at this point, you do need to give someone a number—boss, PM, whoever. My usual way to do it: take the 50th percentile and the 98th percent confidence numbers 4</a> and put them into human language.
I want to justify each step here, so:

"We expect this to take between 4.75 and 7.63 days." Precise but no good, since whoever you're giving this too will hear accuracy when you only intended to communicate precision. So…</li>
"We expect this to take between 5 and 8 days." Better, and if I were communicating internally (e.g. with a PM assigned to the team) I might stop here. Note that I rounded up; rounding down to 4 days would take us from 50% confident to 30%. Overall, however, it's still a fairly narrow range. I might improve it to…</li>
"We expect this to take 1–2 weeks." We're 90% confident this work will take at least 4.75 days, so rounding up to a workweek is not unreasonable. And 8 workdays definitely represents the better part of two workweeks.</li> </ol>
This is different than the kinds of random padding people do with off-the-cuff estimates ("engineers always pad by 20%; I'll cut it by that much") since we're working from reasonable estimates for individual items. However, we know we're not going to be 100% accurate, or even accurate to within a tenth of a day in many cases. It's reasonable to represent the fuzziness in our inputs with fuzziness in our outputs.
Adjusting for Inaccuracy</h2>
As you proceed through the project, you're going to immediately get useful data about how accurate your estimates were. The evidence shows up immediately in your bug tracker: how long did it take between starting and finishing the task? I can tell you that you're not gonna hit 100% of your estimates! Fortunately, we can account for that.
Let's talk about where we got the magic divisor 3.3 from. That's actually the amount of standard deviations that you think the range spans, and it depends on the percentage confidence you are trying to hit during the estimation process. Since I told you to aim for 90%, we use 3.3. Here are some more:
Actual Outcomes within Range</th> Divisor for standard deviations</th></tr></thead>

10%</td> 0.25</td></tr>
20%</td> 0.51</td></tr>
30%</td> 0.77</td></tr>
40%</td> 1.0</td></tr>
50%</td> 1.4</td></tr>
60%</td> 1.7</td></tr>
70%</td> 2.1</td></tr>
80%</td> 2.6</td></tr>
90%</td> 3.3</td></tr>
99.7%</td> 6.0</td></tr> </tbody></table>
This works in reverse, too: if we find that we've been inaccurate in our estimates as the project progresses, we can recalculate the estimate based on our real accuracy.
So let's pretend that we're done with the part of the project we've been estimating so far and have hit a milestone. Hooray and congrats! Remembering that this is just the first part of the project, let's update the estimate for the rest.
Say we've got 15 tasks and our actual working time fell within the bounds of the estimate for 12 of them. That ends up being 80% coverage, not 90%. Redoing the calculations, we get a standard deviation of 1.83, up from 1.44. That's not a huge jump, but it's enough to know that we need to widen our range. If we were to recalculate our sample tasks, the most precise version of the range would go from 4.75–7.63 days to 6.58–8.41 days.
This may be an unpleasant surprise to your stakeholders, but it's better to share this kind of news early instead of waiting until the very end of the project to drop that everything is going to take 25% longer than you originally estimated. Point is, you now have the tools to adjust! In the second project you estimate with this method, you'll account for more of the things that you missed the first time.
Because you know things will shift around as you get more data, you should agree with your stakeholders that you'll be providing updates on the estimate as you move through the project. If they're expecting to hear from you, delivering that news gets way easier (and it's not that much additional work if you've kept the spreadsheet from earlier around.)
You can also begin in a better place once you have some historical data. If you know that your team almost always ends up at 70%, you can use that to calculate your initial estimate. (You should still work on identifying gaps and building skill, though, so that you can get to 90%!)
Basically, as soon as you can ground your estimate in reality, do that!
That's It!</h2>
Just to get everything in one place, here are the steps we've just been over:

Break down work into high-level areas.</li>
Break those areas down until each task feels like something you'd put in a PR.</li>
Work with your team to estimate the best-case, normal-case, and worst-case scenarios for each task.</li>
Figure out the standard deviation represented by the range between best and worst case, then use it to calculate a time range that you can communicate. (50% and 98% is nice.)</li>
Re-estimate at least at agreed-upon milestones. If you have to share bad news, share it quickly so that your team can get ahead of it and build trust and credibility.</li> </ol>
If you've enjoyed this, or want to learn more, I'd really recommend picking up Software Estimation: Demystifying the Black Art. In addition to this technique (which is based off of several chapters, but most notably chapter 10) there's a lot about ways to build up an estimate based off analogies to historical data, as well as useful tactics for communicating with stakeholders.
Give this a try, though! I'd love to know how it turns out for you!
However, before I leave you…
Reasonable Objections</h2>
People tend to be skeptical of this process. Seems like a lot? For what benefit? So I want to address some of that:
Time Cost</h3>

This seems like a lot of effort? Is the improvement here really worth it? </blockquote>
I think so, yes! (Or I wouldn't be writing about it, would I?)
Basically: as a team leader, this is work you'd be doing anyway. Even if you aren't doing it directly, someone else (maybe your PM) is breaking down projects into smaller components that can be worked as tasks.
After that, it's pretty common for a stakeholder to count the tasks/stories/points in the project and divide by the team's velocity to get a simple number. This process provides benefits above and beyond that because:

We're breaking down work down past where an external stakeholder or PM would necessarily have expertise. Doing this requires you to build a good idea of the feature/project/requirements, and puts you in a position to be a good partner.</li>
It gives your team a chance to come up to speed on the same, as well as creating opportunities for them to point out issues.</li>
It's generally more accurate than a simple velocity calculation.</li> </ol>
I talked a lot about the statistical machinery in this post since it's important to use that correctly. However, a lot of the benefit of this process comes from the discussions you have to have to run it.
Anchoring Bias</h3>

How do you run the estimation sessions? Don't people anchor on the first number they hear? Doesn't this process also bias towards the loudest or most senior person's view? </blockquote>
This is sort of orthogonal to the estimation process, but it's important!
First, I've played a game with people when running these meetings. Instead of just opening the floor and saying "what do you think is the worst-case estimate here?" I'll ask people to show a number of fingers on the count of three. Then we have a discussion about why there's a broad range, or simply accept the lowest/highest answer for a narrow one (depending on if we're doing worst-case or best-case.)
All that goes out the window, though, if a team has unhealthy group dynamics. If you have a single person or group dominating the conversation, you're going to get a worse result. Fixing this is outside the scope of this post.
What about fixing 50% confidence?</h3>

How do you calibration 50% confidence for the normal case estimates? Seems like a tricky skill to learn. Most teams will be guessing. </blockquote>
Yep, most teams will be guessing at first. However, we'll get data pretty quickly as to how well we guessed. Unfortunately, I don't have a convenient statistical trick to adjust the expected case if you're consistently ending up on one side of the line. However, that's data that you can take into the next projects and learn from.
McConnell also has a number of strategies in the book that help increase accuracy here.
Identifying Gaps</h3>

You mentioned that the biggest inaccuracies have been due to missing work. How do you get better at that skill? </blockquote>
A combination of best practice and battle scars. After each gap, I've been collecting questions to ask to avoid the issue in the future. Again, my list is available at feature-proofing checklist</a> and I encourage you to steal it and make it your own! (I've given this to new team leaders as a handout for years, and it's usually been pretty helpful!)
Some notes on mine:

It starts with a bunch of "why" questions. If you can't answer those for your team, you're not in a good place. This also helps push back on junk work that people haven't thought through.</li>
It's a lot of work, but this is another case where you should be doing most of this anyway.</li>
You don't need to answer all the questions up front or by yourself. The first section is work with your PM or stakeholders, the second is for gap coverage, and the third has a few questions for nearer the tail end of the project.</li> </ul>
What if your estimate is unacceptably long?</h3>

What if someone expects a feature to take 3 weeks and you come back with a 3-month estimate? </blockquote>
Yeah, this happens. It's painful, but a good opportunity to make implicit assumptions explicit. While it can be annoying to have to negotiate/cut scope, that's part of the job sometimes too.
In my experience, this only has to happen once or twice before people start asking your opinion much earlier in the process. So there's a virtuous cycle here as well, in terms of doing a good job on engineering leadership!
^1
Of course, it's possible that you're in a Cassandra</a> situation and credibility and accuracy aren't the problems. If that's the case, I've been there and this process isn't the way out. Sorry! </div>
^{2
about 6 trillion US (short) tons, or 5 trillion metric tons.
</div>}
^{3
If you have fewer than 10 items in your list, you may want to consider breaking them down more. At 10 items and above you start to benefit from the law of large numbers, where errors in your estimates will start cancelling out.
</div>}
^{4
Why 50% and 98%? Two reasons. First, people are generally happy with "whether we finish early is basically a coin flip, but you can make plans around hitting the end date." Second, I can remember the calculations off the top of my head: 50% is expected + stddev</code> and 98% is expected + 2 * stddev</code>.
</div>}
Should we design for iffy internet? 2025-06-16T00:00:00+00:00 I keep hearing claims like this:Not everyone in the US has access to stable, reliable internet, even in 2025.</li> Web developers should stop assuming people have fast internet connections and slim their payloads accordingly.</li> </ul> This seems intuitively true to me—programmers are gonna have better connectivity because that takes money, and programmers are well-paid. But what's the actual scope of the problem? I dug around, and here's some data. My goal here is not to beat anyone over the head with "THOU SHALT NOT ASSUME GOOD INTERNET" but to give an idea about the scope of broadband rollout in the US in a way that can help inform choices we make when designing software. If you don't feel like reading this whole thing, here's the bottom line up front: you can probably assume internet access in somewhere around 97% of US households, but you should not assume that it's better than around 25Mbps down and 3Mbps up, and latency may be significantly worse than you previously assumed. This is likely worse for B2C software than B2B. I'm going to pull from two US government agencies here: the Federal Communications Commission (FCC)</a>and the National Center for Education Statistics (NCES)</a> (part of the Department of Education.) All the data I'm referencing was published well before the current administration started gutting the bureaucracy, so I think it's fairly reliable. Assumptions</h2> Before we begin, there are a bunch of ways to define "stable" or "reliable" internet connections. For the purposes of this post, I'm defining that as a terrestrial link with at least 25Mbps down and 3Mbps up. Terrestrial because—well, have you ever tried to use a satellite connection for anything real? Latency is awful, and the systems tend to go down in bad weather. They also have had fairly low data caps historically so you had to use your connection judiciously, although this may be changing.1</a>2</a></li> 25/3Mpbs because that's a fairly common cable package speed, and also around the minimum I’ve been able to use modern SaaS apps on without it getting ridiculously frustrating. Zoom says</a> you need 1–4 Mbps in both directions to do a video call.</li> </ul> The FCC's broadband map lets you set both these criteria. Let's have a look at it. The FCC Broadband Map</h2> The FCC tracks access to broadband internet in the US. As a result, they publish a map that's more accurate than the maps you see from ISPs. This isn't completely error-free, but individuals and companies can both submit data on coverage at a street address level and challenge inaccuracies in the existing data. For our purposes, I think it's good enough! I'm going to grab some screenshots of the map to fix it at the current data for discussion. That way we can get an idea about regional coverage. Strangely, they don't let you zoom out enough to grab a screenshot of the whole country so I'm going to look at the west. That'll get both urban and rural coverage, as well as several famously internet-y locations (San Francisco Bay Area, Seattle.) If you want to follow along or compare how the data has changed since this post was published, you can get the map yourself at broadbandmap.fcc.gov</a>. First, here's a hex map of coverage using the criteria we set above. The darker each hex is, the higher the percent coverage within it. Dark blue is 100%, white is 0%, grays or faded blues are in between. This shows pretty much what I'd expect: coverage is fine in and around cities and less great in rural areas.3</a> (The Dakotas are an interesting exception; there's a co-op up there</a> that connected a ton of folks with gigabit fiber. Pretty cool!) Although, remember that these are minimum criteria. When I'm building software, I'm mostly doing it on my home internet. A test just now says I get 367/71Mpbs. What does the country look like if I were to expect all my users to have similar connections? The map lets us filter for 250/25; let's look: Cities are pretty much unchanged, but rural coverage gets much worse. (Except again for North Dakota, the reigning champion of rural fiber.) So what does this tell us about how we should design software? One big takeaway: if you design for the availability of fast internet connections, you'll exclude many people in rural areas. This may or may not be OK for your market—"good internet" tends to be in population centers, and population centers tend to contain more businesses and consumers. You have to make that call! However, it's also worth keeping in mind that this is a map of commercial availability, not market penetration. Hypothetically, you could get the average speed of a US residential internet connection, but the FCC doesn't make such a statistic available.4</a> However, we have another source of data! Student Internet Access</h2> The US Department of Education occasionally tracks student internet access. Since classwork and homework have moved more online (especially during the early days of COVID) it's useful data for policy-making. The most recent data I could find is from 2021</a> but it gives us both a baseline for internet availability and some demographic data. Unfortunately, they don't track bandwidth, just raw availability. Still useful, though. Here are the top-line statistics for all students in the US across several versions of the report: Category</th> % Available 2019</th> % Available 2021</th> Difference</th></tr></thead> Any internet access</td> 94.6%</td> 97.1%</td> +2.5</td></tr> Smartphone-only</td> 6.5%</td> 4.5%</td> -2.0</td></tr> No internet Access</td> 5.4%</td> 2.9%</td> -2.5</td></tr> </tbody></table> In 2021, they say that these statistics cover 66,108,000 students, which means 2.97 million students only had mobile access as of 2021. Access gets worse, of course, with lower household income. Here's how those stats look for the lowest quartile: Category</th> % Available 2019</th> % Available 2021</th> Difference</th></tr></thead> Any internet access</td> 88.6%</td> 94.4%</td> +5.8</td></tr> Smartphone-only</td> 14.1%</td> 9.8%</td> -4.3</td></tr> No internet access</td> 11.4%</td> 5.6%</td> -5.8</td></tr> </tbody></table> Despite the fact that these numbers are going down, that's still a huge number of people in absolute terms (1.87 million students in this category with only mobile access.) What Do We Do With This Information?</h2> So that's a lot of words to say this: despite gains in the last couple of years, it's still not safe to assume that every user of your software either has access to stable internet, or is willing and able to pay to get high speeds. That said, I'm deliberately not making any moral judgments here. If you think you're in a situation where you can ignore this data, I'm not going to come after you. But if you dismiss it out of hand, you're likely going to be putting your users (and business) in a tough spot. I think it's worth considering a couple of scenarios in the parts of your software that someone interacts with regularly: What if that person is on a slow link? If you've never had bad internet access, maybe think of this as plane wifi. Rural satellite connections behave very similarly: high latency, speed and (sometimes) low data caps.</li> What if that person is on a mobile/metered/capped network? Remember that 4G is like 5/1Mbps, and 3G is even worse. Big downloads are probably not a great idea.</li> </ol> This is also a very US-centric view, plus it doesn't consider latency from distance between your data center and your user's device. Still, though, I think this shows that this problem is real and we should take it into account when designing software. ^{1 Despite these drawbacks, satellite has high availability because you just need to have access to power instead of laying cable or building towers. It's still the best option for a lot of the rural US. </div> ^{2 I grew up in a semi-rural area (mountains) and this was the only real option. As a kid, I really hated it! The latency was too bad to play games with my friends online, and I also regularly hit our household's download cap and got us all throttled to basically-dialup for the rest of the month. </div> ^{3 I'm not going to embed a screenshot, but mobile coverage looks pretty much the same if you set the same bandwidth requirements. According to the FCC, that's 5G-NR</a> at 35/3Mbps. 4G has far better coverage in rural areas, but only gets 5/1Mbps. </div> ^{4 The ones published by various speed testing companies vary so wildly that I don't think they're worth paying attention to, even for broad decision-making. </div>} Local-First Foundations 2025-01-22T00:00:00+00:00 I'm writing a book about local-first software tentatively called Local-First From Scratch</a>. This is a snippet from that book—an intro to why you should care about local-first software and what it gets you. I thought it ended up being a pretty good summary of what the whole project is about, so I wanted to share it as a blog post as well! If you picked up this book, you might have some idea about what "local-first" means1</a>, but let's get on the same page: other than "works offline", what are we aiming for? In short, local-first software moves ownership of data from "somewhere in the cloud" to your local devices. This has a bunch of consequences. We like some of them: Updates are practically instant because the network does not have to be involved. No loading times or spinners! You can also work completely offline.</li> You are in control of your data and control who gets access to it.</li> A company going through an "incredible journey" acquihire doesn't mean you lose access to your work.</li> </ol> Of course, implementing these ideas is a little harder than just storing data in a local SQLite database. There's a cost paid in complexity and figuring out new ways of doing things. For example, we'd typically enable sharing by having a central server hold all the state. But we can't do that if the user is offline and can't reach the server!2</a> So local-first also raises some questions we need to answer: You probably have more than one device (phone, computer, etc.) Which one is the "real" copy?</li> If you want to share your work with someone else, how?</li> When you share—either with other devices or other people—how do you avoid conflicting writes?</li> </ol> We want satisfying answers to these without giving up local control. The typical approach—and the one we'll implement in this book—is CRDTs. ^1If you want to learn more about local-first principles, Ink & Switch's essay Local-first software: You own your data, in spite of the cloud</a> is a great place to keep going. You may hear folks refer to the "local-first manifesto"—this is the thing they're talking about. </div> ^{2 This blows up a lot of SaaS business models that depend on a subscription to a central server. We're not going to address the business ramifications of local-first software in this book. If you're interested in that, as of early 2025 the Local-first Software Discord hosts frequent online meetups where people share their experiences. </div> That's it for the preview. Keep an eye out for the book hopefully later in 2025!} platform engineers work on the meta-product 2024-09-23T00:00:00+00:00 In what is platform engineering?</a>, I gave this definition: Platform engineering, as a discipline, takes a coherent approach to improving an engineering organization's output in ways that the rest of the business can see and understand. </blockquote> I've had some thoughts and discussions since then, though, and I think that definition needs revising. Specifically, I tried to be very clear in the last post that I didn't think platform engineering could/should be gatekept. Problem is, that previous definition has some scoping issues—mostly around vagueness—that could work against that goal. For one thing, it gives a bad-faith interpretation some gotchas: you could be doing the work of platform engineering, but have someone argue that your approach is not "coherent" or that the business cannot see and understand your work. I gave examples in the last post, but that feels like a bad substitute for the definition itself being clearer. For another, the language here is too broad: there are differences between SRE, devops, and platform engineering, but this definition does not give you the tools to distinguish between them. To be fair, I think this is minor; this definition gives enough information to get started, or to explain what you're doing to someone outside your immediate work. But I also think definitions should allow for clear distinctions, and this one doesn't. Fortunately, I think I see a way around all these problems in two steps. First: who is a platform engineer? A platform engineer is someone who does the work of platform engineering. </blockquote> If you make art, you're an artist. If you make music, you're a musician. If you do the work of platform engineering, you're a platform engineer. That definition resists gatekeeping, which I like. But what is the work? Platform engineers work on the internal product that other teams use to make the customer-facing product. </blockquote> Or more succinctly: Platform engineers work on the meta-product. </blockquote> I like a couple things about this. First, this clearly includes the things we care about: CI/CD, developer experience, tooling, moving observability tools to where developers can see and use them, etc. If it’s part of the internal product, then it's in scope. Second, it distinguishes between the work of platform engineering and SREs or devops. For example, when I've worked in devops or SRE roles, I've cared a lot about what applications are running where, what kind of traffic loads we're expecting, or how much we're spending to deliver the product. These are all customer-facing concerns, and might not be in scope for platform engineering.1</a> The meta-product here would be working on things like adopting Terraform or Kubernetes, which automates a lot of that work and allows teams to ship the product more efficiently. Now, someone might say "ah, but SREs care a lot about automation! That's the meta-product and actually a big part of the gig!" Yes, good, that's correct! But check out the new definition: are they working on the meta-product? Yes? Then they're doing platform engineering in addition to SREing. That's fine; I acknowledge the overlap here. Part of the utility of this definition is that it allows us to recognize this overlap, while still making a distinction. People hired into platform engineering roles may (and probably will) work with SREs on cloud automation projects, but that's not their only thing. Let's examine this from the dev side as well: at a small company, you probably don't have someone specifically doing platform engineering as their whole role (for, say, more than 50% of their time.) But you still need a platform: gotta set up CI and a deployment target, at minimum. Congratulations, you're doing platform engineering! Nobody can tell you you’re not. Finally, in the platform engineering role itself, this serves as a good charter for an individual or team. Where there’s overlap, you can share the load. Where there’s not, it’s clearly your responsibility. So there you have it: a better definition. This still isn’t perfect, but I think it’s a big improvement over the last one. ^{1 I know about (and have worked on) teams named "platform engineering" who spend most of their time taking care of cloud infrastructure. I think that's more a titling issue than anything, and it depends more on the age of the organization than their level of seriousness. Terms vary, people use language differently, it's fine. </div>} what is platform engineering? 2024-09-09T00:00:00+00:00 Today I'm starting my third1</a> platform engineering job. During the interview process, I had a few people ask me what exactly a platform engineer does, and I realized that… I've had my own short answer—"a platform engineer finds and reduces friction in the engineering organization"—but that's more an example than a proper definition, plus it doesn't cover everything.</li> There are a bunch of confusing/contradictory answers floating around the internet if you just search it. This is made worse by vendors who seem to want to gatekeep the term to sell products. (Can you really even be a platform engineer if you're not running an IDP???2</a> Contact our sales team!)</li> </ol> Anyway, I though it'd be useful to try and sum up the things that platform engineering is/does in a succinct definition. Here's what I came up with. I'm not saying it's perfect, but it's helped me understand my role better: Platform engineering, as a discipline, takes a coherent approach to improving an engineering organization's output in ways that the rest of the business can see and understand. </blockquote> Breaking some of that down: an coherent approach: The thing that sets platform engineering apart is that it focuses on making an integrated whole out of processes that are often otherwise approached in a piecemeal (or incoherent) way. (You could also say "holistic" or "integrative" or "product-minded approach.")</li> improving an engineering organization's output: By focusing on output, platform engineering takes a fairly wide area of responsibility. Improvement can mean a bunch of different things, as can output. Some examples: improving p99 response times, increasing test coverage, decreasing CI pipeline runtime, coordinating penetration tests, consolidating docs, finding new project management tools, auditing and classifying runtime exceptions, and training teams in new tools and methodologies. (As a matter of fact, I've done all of those at some point or another as a platform engineer.)</li> As a counterexample, platform engineers typically do not work on the "people" side of the engineering organization. For example, we don't train managers or consult with teams on the best ways to work together. But we do conduct ourselves with the human side in mind.</li> </ul> </li> in ways the rest of the business can see and understand: to be effective, platform engineers need to be able to talk about our work in ways that the rest of the business can buy into. For example, this might mean being able to talk about both the system's ability to handle increases in transaction volume and what that means for scaling runway. Usually this means quantifying achievements, but it also may mean being able to talk about, say, the impacts of developer experience on voluntary attrition.</li> </ul> To put it another way, platform engineers work on the engineering organization instead of just in it, at least to some degree. They can't spend 100% of their time here though: platform engineering needs to have a solid idea of the day-to-day challenges and pains the organization experiences in order to do an effective job. This also isn't work that can be gate-kept, no matter how much a vendor might want to. You need to do at least some of this work to have a functioning software business in the first place. For example, how does your code make it to production?3</a> How are builds and tests run? Engineers need to do this work regardless of if they have someone specifically focused on platform engineering, but it might be at the cost of not focusing on the things that they excel at. You sometimes hear people talking about this with the terms "inner loop" vs "outer loop." Time spent on the primary process of improving the product (coding, writing tests, etc) is the inner loop, and time spent on stuff like deployments or requirement definition is the outer loop. The value in hiring someone to do platform engineering for your business is that they can work on minimizing the time everybody else has to spend in the outer loop. There are a couple other interesting things that fall out of this definition too. For example, it explains why platform engineering tends to be the interface between product engineering and security or QA (because they're responsible for output and coherence) and why platform engineers may be later hires at companies (because smaller companies may have more visibility into cross-departmental processes.) At any rate, I found this definition helpful and I hope you do too! ^{1 Third-ish, anyway. Maybe fourth, as the eldest of those was a devops-ish role focused entirely on building a deployable PaaS on top of Apache Mesos. At any rate, at least two of the four jobs predate the term "platform engineering" as we know it now by a couple of years. The principles here have been relevant in all those positions, though! </div> ^{2 Integrated developer platform, basically an integrated portal to an organization's docs, best practices, deployment patterns, service catalog, etc. </div> ^{3 This hints at the fact that platform engineers, devops, and SREs have very closely related responsibilities. I'd say the defining difference is what work each would consider "the inner loop." DevOps and SREs I've worked with have been primarily concerned with safely delivering and running software while platform engineers I've worked with tend to focus more on developer productivity and "shift left" work. This has not been globally true, though. </div>}}} Kratky in the basement 2024-04-01T00:00:00+00:00 Hello hello! Today is April Cools</a>, which is like April Fools but for posting stuff outside what you normally write about instead of unfunny jokes. Hope you enjoy! I want to eat more salads, but: Viral and bacterial outbreaks come up pretty regularly in the US food system. I don't feel super paranoid about produce where you can wash or avoid eating the skin, but I feel worse in cases where usually you eat the whole plant.</li> Large-scale farming techniques sometimes seem pretty dubious, even in products labeled as organic.</li> Because of how our food system is set up, produce is regularly transported hundreds of miles before being consumed.</li> </ol> Luckily for me, doing hydroponic gardening at home in my basement is a way around this: I won't get listeria cross-contamination from pig farms or whatever—there are no pigs in my basement!</li> Growing indoors means I have precise control over the environment the plants are growing in. There is no need for pesticide or herbicide. (And I have lots of choice in what nutrients I use.)</li> I can walk downstairs to harvest lettuce which avoids all the waste and carbon emissions from transportation, plus the produce could literally not be fresher.</li> </ol> Plus I think hydroponics are cool,1</a> which is another helpful motivation! Problem is, I've always overcomplicated things… for example, I've drawn up plans to turn the entire house into a hydroponic garden, feed the whole neighborhood, start a local plastic recycling and hydroponic produce market, etc. You get the idea. So earlier this year, I decided to commit to doing something as small as I could: get at least one salad on a plate before the end of March. And I did it! Here's how. Hydroponics Crash Course</h2> I'm still a beginner and I'll get some of this wrong for sure, but before I can explain how my setup works I need to share a little theory. Plants need like 4 things2</a> to thrive: Water</li> Bioavailable nutrients</li> Light</li> Air</li> </ol> First up, water and nutrients are taken together in hydroponics. Mixing the right hydroponic nutrient solution could be a long post all by itself, so I'll just sum it up by sharing some details that I was confused by at first: Nutrient formulations are marked with three-number value like 10-10-10 or 8-15-36. Those values are nitrogen-phosphorous-potassium, or NPK. As I understand it, they're percentages: a 10-10-10 mix has 10% of each. That's a good starter mix for lettuce, but I've read that 8-15-36 will also work.</li> You measure pH and electrical conductivity in the nutrient solution. You need to care about pH because if it's too high or low then plants can't actually absorb the nutrients from the water.</li> You need to care about electrical conductivity because it's a good proxy for how much total nutrient is in the solution (since saltier water is more electrically conductive.) Too much and you'll get "nutrient burn" where the edges of leaves turn brown. Too little and the plant just won't grow correctly.</li> </ul> </li> </ul> Light is usually a cost/value tradeoff. Good grow lights are expensive, and smaller ones usually have a price premium due to home marijuana growers in the US market.3</a> I bought mine from spider-farmer.com</a> (got a pair of SF300s for $99 on sale, $135 normal price.) There is not a lot to think about if you're willing to accept this as the biggest cost of the initial setup, but the most important thing is that you have to have a full-spectrum light: plants grow best in reds and blues, basically (which makes sense if you think about it; leaves are green because they're reflecting those wavelengths instead of absorbing them for photosynthesis.) That leaves air, which different systems adapt to in different ways. The basic problem is that plants can drown if you submerge their roots in water that isn't properly aerated. This is a problem in soil gardening too; it's why you need to ensure good drainage in your planting beds. In most systems I was looking at, there was some level of circulation or aeration which required a bunch of pumps and air stones. But it turns out that you don't need all that at a small scale: you can just use the Kratky method. The Kratky Method</h3> So that all leads to Kratky: You basically take a bucket, make sure the plants can get their roots wet, and just let them grow. Plants will grow regular roots down into the reservoir, and send off air roots (they're hairier and shorter than the ones that absorb water) sideways above the waterline as the water evaporates. The big benefit is that you don't need pumps, air stones, or any powered components aside from a light (and if you do it outside, you don't even need that.) That kind of simplicity is what I was after! The big drawback is that the nutrient concentration will go up over time as the water evaporates, so you need to start lower. For lettuce, I've read that starting at half-strength nutrient concentration is fine. You also will probably not be able to cycle new plants into the system, as their roots will not be able to reach the lower water levels. My Setup</h2> So here's how I put all that theory into practice. As a reminder, I wanted to get a single salad out of a system that was as simple as I could possibly make it. Here's how things looked at the start of the grow. I only needed one light for six plants (which I calculated based on how much room a typical lettuce plant needs to grow.) Here's what it looked like near the middle of the grow below the waterline. You can see the air roots coming off the sides of the longer roots! Then finally, here's how the plants looked near the end of the grow about 45 days in: They stayed at this stage for about another month while we used all the plants in salads. We did that slowly at first, because we weren't 100% sure if they were actually safe to eat, but we sped up pretty quickly as we didn't see any ill effects. I wish we had eaten them faster, though: the nutrient concentration went up in the reservoir and most of the plants ended up with nutrient burn that I had to trim off before eating. The easiest way I found to harvest the plants was to cut them off at the stem with a sharp knife, then chop them the same as I would a grocery store lettuce. I plucked all the leaves off one as an experiment, though: it took me a lot longer, but then the plant was able to regrow most of its leaves for a second harvest. If it was easier to change the water in this system, I might do that more. Anyway, here's the shopping list if you wanted to replicate something like this: A reservoir. I used a 27 gallon (~100 liter) tote with holes drilled in the lid.</li> Net cups and grow media. I used some standard net cups off of amazon with some rockwool that I got from a local hydroponics shop.</li> Seeds.</li> Lights. I used Spider Farmer SF300s, which I plan to use for future grows as well. Link above.</li> Nutrient solution. I don't have a strong recommendation here—I bought some vegetative growth nutrient mix from the hydroponics shop. It's clearly designed for the vegetative phase of marijuana, but it has about the right NPK and it worked fine. I monitored the pH and conductivity for the first 30 days or so, but stopped once it stabilized.</li> </ol> I want to make some improvements already, though: I'm going to switch to 5 gallon (18 liter) buckets for future grows. 25 gallons of water weighs about 200 pounds (~90kg), and the tote isn't really set up for that much weight. Using small buckets would let me run different experiments with pH and nutrient composition as well.</li> I did the grow in my basement. That meant a pretty constant temperature of about 60°F (15.5°C.) It might be worth getting some kind of grow tent with a small heater to raise the temperature.</li> My hydroponics setups have a fan blowing on the plants to stimulate them in this kind of environment. I think I ought to try that!</li> I'm not sold on rockwool as a grow medium—I found out while writing this post that I have not been sufficiently careful about handling it! I have coco pellets too, but I'm not sure how to start seeds in that.</li> </ul> So that's where I'm at on hydroponics! I'm really excited to improve on this, but I'm already pretty happy with where things are right now. ^{1 If you don't think hydroponics are cool, that's fine. But if the other stuff in this list resonates with you, consider looking up local farmers markets or produce co-ops, which address some of these problems in similar ways. </div> ^{2 Ok, so there's actually a bunch more here. Plants need the right temperature, humidity, protection from insects or animals, a place to put down roots, etc. But you need to solve for these four to even get started. </div> ^{3 I call it the "weed tax", although I can't recall if I read that or came up with it. </div>}}} sourcing secrets from 1Password 2024-02-27T00:00:00+00:00 I've been setting up a personal Kubernetes cluster</a> recently. The whole thing is set up with Argo CD</a>, which basically means that I make changes to the cluster by committing to a git repo and pushing. Really handy, especially when I make a mistake and need to clean something up or revert. One problem with this, though, is that it's probably not a great idea to commit secrets to the repository. I don't want things like passwords and API keys sitting there in clear text on my laptop or on the remote. If I were setting up a cluster that other people were going to operate, I'd probably set up something like Vault</a>, but since it's just me I can get away with creating the secrets by hand. That said, I still don't want to leave them on disk unencrypted! Enter the opw</code> CLI for 1Password</a>. It can output secret data as JSON, and I can use jq</code></a> to transform that into Kubernetes secret objects, which I can pipe into kubectl</code> to apply in the cluster. Let's take a simple example: an image pull secret. I have the secret stored in 1Password under a specific vault. If I run op item get --vault k8s "image pull secret"</code>, I'll get something like the following: ID: y63hsk7qy5osug5n2qupwhtz3e Title: image pull secret Vault: k8s (2qaqjigknr3htch273i2mho5vi) Created: 2 weeks ago Updated: 2 weeks ago by Brian Hicks Favorite: false Version: 1 Category: LOGIN Fields: password: password-goes-here username: username-goes-here</code></pre> I can also ask for it in JSON format by adding --format json</code> to the end: { "id": "y63hsk7qy5osug5n2qupwhtz3e", "title": "image pull secreet", "version": 1, "vault": { "id": "2qaqjigknr3htch273i2mho5vi", "name": "k8s" }, "category": "LOGIN", "last_edited_by": "B76OR7I6DNENPAQOQFFO6FKQOM", "created_at": "2024-02-12T18:37:38Z", "updated_at": "2024-02-12T18:37:38Z", "additional_information": "brian@brianthicks.com", "fields": [ { "id": "password", "type": "CONCEALED", "purpose": "PASSWORD", "label": "password", "value": "password-goes-here", "reference": "op://k8s/image pull secret/password", "password_details": { "strength": "FANTASTIC" } }, { "id": "username", "type": "STRING", "purpose": "USERNAME", "label": "username", "value": "username-goes-here", "reference": "op://k8s/image pull secret/username" }, { "id": "notesPlain", "type": "STRING", "purpose": "NOTES", "label": "notesPlain", "reference": "op://k8s/image pull secret/notesPlain" } ] }</code></pre> We can then pipe that into the following jq program to get an object indexed by the field names: .fields | map({ key: .label, value: .value }) | from_entries</code></pre> That produces this: { "password": "password-goes-here", "username": "username-goes-here", "notesPlain": null }</code></pre> And we can add onto that jq program to produce some JSON in the shape of a Kubernetes secret: .fields | map({ key: .label, value: .value }) | from_entries | { apiVersion: "v1", kind: "Secret", type: "kubernetes.io/dockerconfigjson", metadata: { name: "ghcr-image-pull-secret", }, data: { ".dockerconfigjson": { auths: { "image-host.io": { auth: "\(.username):\(.password)" | @base64 } } } | @base64 } }</code></pre> Assuming that's in image-pull-secret.jq</code>, we can now load the secret into Kubernetes like so: op item get --vault k8s "image pull secret" --format json | \ jq -f image-pull-secret.jq | \ kubectl apply -n whatever-namespace -f -</code></pre> Like everything, this has some tradeoffs: the secrets are stored securely and never hit the disk, which is great! But in exchange, I have to run a command manually if I want to update the secrets. This means that if I'm creating a new app, it'll probably fail at least once. Same if I have to move my cluster (which I already did once</a>) and have to apply all the manifests from scratch. So in the end, would I recommend this? Yes, if you're in my situation! But if you're running a cluster where other people have to operate it too, this is probably not the best approach. It's good to know it's possible, though! modeling CRDTs in Alloy - counters 2023-11-27T00:00:00+00:00 Hey, welcome back! Last time, we introduced conflict-free replicated datatypes (CRDTs) and modeled them in Alloy by using boolean OR</code> as our merge function</a>. As a quick recap, CRDTs give you eventual consistency, no matter how out of sync the data originally was. This makes them great for local-first or networked multiplayer applications. If you can write a merge</code> function for a data structure that is commutative, associative, and idempotent, you might be able to build a CRDT on top of it. Further, we're modeling all these semantics in Alloy</a>, a lightweight formal methods tool good for modeling and analyzing data structures. We'll see some cool stuff it can do in a second. Counting Birds</h2> Last time our document was a single boolean. That's useful, but we can't build an application on top of it, so let's keep moving. To get much further, we're going to have to figure out how to sync numbers back and forth. Let's use the the Great Backyard Bird Count</a> as an example. Say we're counting red-tailed hawks</a>: Everyone starts at zero.</li> Person A sees a hawk! Ka-kaw! They increment the counter by one.</li> Person B sees another hawk! Screeee! They increment the counter by one, too.</li> Person C sees two hawks. Wow! Of course, they increment the counter by two.</li> Day's over; good job everyone! Sync up your apps and lets see how many hawks we saw total.</li> </ol> At first glance, we might want to use +</code> as the merge function. But remember, that's not idempotent. So this might happen among three people: The group has seen four hawks total. Person A and person B have seen 1 each, person C has seen 2. Nobody has talked to anyone else.</li> Person A and person B sync and set both their totals to 1 + 1 = 2</code> (one from each, the initial count)</li> Person A and person C sync and set both their totals to 2 + 2 = 4</code> (two from A because of the previous sync)</li> Person B and person C sync and set both their totals to 2 + 4 = 6</code></li> Further syncs happen, and the result increases every time.</li> </ol> In other words, we can return with a huge number of hawks when really we only saw four. That's not what we wanted! This fails because +</code> is not idempotent: adding any meaningful number will always change the result. What if we used max</code> instead? It's commutative, associative, and idempotent, but the semantics don't work out here: if we used max</code> to merge, we'd only get to see the max of hawks that any one person had seen. However, max</code> would work if we synced one counter for each person and then added them together locally to get the result! Let's look at an example: Just like before, the group has seen four hawks total. Person A and person B have seen 1 each, person C has seen 2. Nobody has talked to anyone else.</li> Person A and person B sync their totals, going from person A's {a: 1}</code> and person B's {b: 1}</code> to {a: 1, b: 1}</code>.</li> Person A and person C sync their totals, going from person A's {a: 1, b: 1}</code> and person C's {c: 2}</code> to {a: 1, b: 2, c: 2}</code>.</li> Person B and person C sync their totals, finishing the sync. Now everyone has {a: 1, b: 1, c: 2}</code> and anyone can compute that we've seen four hawks total.</li> </ol> When you have a conflicting value (say B saw another hawk later) you take the max</code> of that person's count when syncing.1</a> This also has the nice property of moving information through the system without requiring everyone to talk to everyone else. Consider what would happen if we did A+B, A+C, A+B instead of ending with B+C. We'd end up with the same value, since A+C would have given A the count to pass along to B, except that B and C never have to talk directly. It's eventually consistent! Moving to Alloy</h2> Let's model this scheme in Alloy. We'll model an ID to represent each individual birdwatcher: sig ID {}</code></pre> We'll also model a Document</code> to allow each person to keep their count and the count of each of their peers: sig Document { id: one ID, var counts: ID -> lone Natural, }</code></pre> A -> lone B</code> means "each A maps to at most one B." In other words, it makes ID</code> unique in counts</code> for each Document</code> (the way you'd expect a dictionary to work in an implementation.) We have an error already, though. When I run this, Alloy gives me and edge case where we use the same ID for multiple documents:2</a> If this happened we'd lose data since we couldn't distinguish between values we needed to add and those we needed to max</code>. Better disallow that in the model: fact "no two documents can have the same ID" { // "disj" means d1 and d2 can't be the same document no disj d1, d2: Document | d1.id = d2.id }</code></pre> As always, adding a fact</code> weakens our spec by making assumptions. So let's think: can we actually get rid of ID collisions when we implement this spec? Well, maybe… in real life, we'd use some ID scheme that we were reasonably sure would never collide. In production CRDTs, it seems like this means using UUIDs</a> a lot of the time. Good enough for me; let's move on. With this change, we can get some more interesting instances of the data structure. For example, here's what it could look like when these counts</code> maps are filled out: (You don't need to know how to read this example, but if you want to: the keys in the map are represented by the labels on the arrows. For example, "Document 0 has a count for ID 1 of 0".) Merging</h2> We don't have a way to sync yet, so let's start fixing that. Let's define merge</code>, as well as adding checks that it satisfies our three laws: fun merge[d1, d2: ID -> lone Natural]: (ID -> lone Natural) { // intent: set each value in the document to the maximum value // of that key in all the documents we're currently merging. let allCounts = d1+d2 | { id: ID, count: Natural | id->count in allCounts and count = nat/max[allCounts[id]] } } check MergeIsCommutative { all d1, d2: Document | merge[d1.counts, d2.counts] = merge[d2.counts, d1.counts] } check MergeIsAssociative { all d1, d2, d3: Document | merge[merge[d1.counts, d2.counts], d3.counts] = merge[d1.counts, merge[d2.counts, d3.counts]] } check MergeIsIdempotent { all d1, d2: Document | merge[d1.counts, d2.counts] = merge[merge[d1.counts, d2.counts], d2.counts] }</code></pre> Alloy says it can't find a counterexample for these three checks, so it looks like we might be in business. Actions</h2> Next let's set up some things our documents can do. First, we should be able to increment a number in a document by removing the old count for this document and ID and adding the new one: pred increment[d: Document] { let current = d.counts[d.id] { counts' = counts - d->d.id->current + d->d.id->bounded_inc[current] } }</code></pre> About all the arrows: a->b->c</code> creates a three-valued relation (you can think of it like a 3-tuple if it helps: (a, b, c)</code>.) Since counts</code> is actually a mapping of document-to-id-to-natural, we need this level of precision. However, this is the only place we'll this since writing d.counts</code> will pop the Document</code> off the left-hand side of the relation, leaving us with ID -> Natural</code> (as in our sig</code>.) I had to write a bounded_inc</code> helper function to write increment</code>. Here it is: fun bounded_inc[n: Natural]: Natural { n = nat/last implies nat/last else nat/inc[n] }</code></pre> This was necessary because Alloy's numbers are actually just regular sig</code>s with an ordering.3</a> Since we only deal with a bounded number of instances for each sig, addition and counting has a gotcha. Say you only knew the numbers 1 and 2: counting works fine up to a certain point, but if someone asked you "ok, and what's after two?" you wouldn't know. This is exactly what's going on with Alloy, which will return a blank set if you run out of numbers. Specs can define semantics specific to their applications, though, and in ours I've decided that incrementing past the max is simply not allowed—a number will just stop growing a certain point. This should not really affect the soundness of our spec, but choosing limits like this requires careful thought. Alloy is not designed for numerical computing, and forcing it means dealing with weird edge cases like this. Fortunately, that's the only place in the spec we should have to deal with this. Now We Can Sync!</h2> Let's move on to sync</code>, which should merge two documents: pred sync[d1, d2: Document] { let merged = merge[d1.counts, d2.counts] | counts' = counts ++ (d1->merged + d2->merged) }</code></pre> Next the little bit of boilerplate that we need to treat these as actions: pred init { // intent: set every document to 0 for its own ID counts = { d: Document, i: ID, count: Zero | d.id = id } } fact traces { init always { (some d: Document | increment[d]) or (some d1, d2: Document | sync[d1, d2]) } }</code></pre> Finally, let's check idempotence like last time</a>: check SyncIsIdempotent { always all d1, d2: Document | (sync[d1, d2]; sync[d1, d2]) implies counts' = counts'' }</code></pre> Alloy can't find a counterexample for this, so I think we're done! These documents should work as CRDTs! In fact, our counters here model something called a "GCounter" (a "grow-only" counter.) There are plenty of implementations out there if you want search for that term and find some real code to read! But… GCounters can only count up. That's a problem, right? You might think "ah, we can get around that—each node could set its own count to whatever it wanted!" But we'd hit problems there on the first sync: since our merge</code> function uses max</code>, any time a node's counter decremented it'd go back up to the highest-seen value on the next sync. There's a nice way out of this predicament, though: use two counters per document, positive</code> and negative</code>. Both counters always go up (so max</code> doesn't give us trouble) and when you need to get the value out of the document, you sum up all the values of both counters, then get the final value from positive - negative</code>. Ta-da! The final can now go up and down, and you have something called a "PNCounter" (a "positive/negative" counter.) You might also look at this and think "yikes, that looks like a lot of memory for a single number… that would just be a single integer if this wasn't distributed." Yep! There's always going to be some overhead for syncing. You can minimize it, though, and people who make production-class CRDTs (like Ink and Switch</a>) clearly think about data compression a lot! In fact, Jake Lazaroff has written an excellent blog post about doing this in an image-based CRDT</a>. Go read that next! Thanks to Alcino Cunha on the Alloy forums for helping me understand how set comprehensions could be used for merge</code></a>, as well as pointing out that it should be possible to specify our checks with only events. Thanks also to Jake Lazaroff for reviewing an early draft and suggesting better ways to structure this. ^{1 This only works if you can trust the people doing the counts, though. If person B says "wow, I heard person A saw FIFTY HAWKS" then that information will spread without any recourse for A to say "er, no, I only saw one." These kind of security and sharing situations seems to be open questions in the field. I guess you could sign values, but that'd add a lot of overhead in any decently-sized document. Tricky! </div> ^{2 Images throughout this post are generated by Alloy—that's one of the things I like best about it. Visualizing edge cases like this makes it really easy to talk about potential drawbacks to modeling with other engineers or people making product decisions. </div> ^{3 That is, the numbers you work with in sigs. You can get the cardinality of a set (e.g. #Document</code>) without these restrictions. However, you can't store the cardinality in a sig</code>, so we still have to work around this restriction. </div> ^{4 Briefly: you can make a GSet (grow-only set) whose merge</code> is the union of two sets. That gives you the set equivalent of our GCounter—it can only add new items. To remove items, you do the same trick as PNCounter and keep two sets, then get the final value by computing union(add) - union(remove)</code>. This means that you can never re-add an item that you've removed, though. We'll touch on how to get around that next time. </div>}}}} sticky table headers with bottom borders 2023-11-13T00:00:00+00:00 Say you have a table like this: <table> <thead> <tr> <th>Column 1</th> <th>Column 2</th> <th>Column 3</th> </tr> </thead> <tbody>  </tbody> </table></code></pre> You might try to separate the header from the body like this: table.fancy > thead > tr { border-bottom: 1px solid #ccc; }</code></pre> That works, but what if you want the <thead></code> element to have position: sticky</code></a> too? If you used a border, the border will keep scrolling while the header sticks. Oh no! I got around this by using box-shadow</code> instead. It works, despite feeling a little hacky: table.fancy > thead > tr { box-shadow: inset 0px -1px #ccc; }</code></pre> Except that Safari won't display the shadow below the header. More hacks required! Apply the shadow to all the cells inside it: table.fancy > thead td { box-shadow: inset 0 -1px #ccc; }</code></pre> This feels even hackier than the original fix, but it works fine in all the browsers that I tested! modeling CRDTs in Alloy - introduction and the importance of idempotence 2023-10-09T00:00:00+00:00 I've been interested in local-first software</a> for a long time, and recently attended an event about it with a bunch of luminaries from various research groups. I learned a lot, and it rekindled my interest in syncable data structures. I've tried to sync data over the years with varying degrees of success. For example, I've known about operational transformations (OT)</a> for a while (via Quill</a>, but I have never been able to get the syncing operations working to my satisfaction. Message ordering is crucially important in OT, and it's hard to get right. I get the impression that the engineers who worked on Google Wave (which also used OT) also struggled with this. One of them (Joseph Gentle) was quoted as saying "Unfortunately, implementing OT sucks. There's a million algorithms with different tradeoffs, mostly trapped in academic papers. […] Wave took 2 years to write and if we rewrote it today, it would take almost as long to write a second time." (Worth noting that he later said it might not take as long today</a>, but still… pain!) In contrast to OT, conflict-free replicated data types (CRDTs) seem really promising! Wikipedia has a good summary</a>, which I'll quote: In distributed computing, a conflict-free replicated data type (CRDT) is a data structure that is replicated across multiple computers in a network, with the following features: The application can update any replica independently, concurrently and without coordinating with other replicas.</li> An algorithm (itself part of the data type) automatically resolves any inconsistencies that might occur.</li> Although replicas may have different state at any particular point in time, they are guaranteed to eventually converge.</li> </ol> </blockquote> These properties appeal to me! If CRDTs can automatically and independently converge, no matter how far apart they drift, then you could store them wherever and however you like. Offline support should be simple, and adding live collaboration features should not mean spending years figuring out edge cases. Sign me up! What are CRDTs made from, though? It looks like typically it's some internal stuff, a function to resolve the internal stuff to a value</code>, and a function to merge</code> two of the data structure together. The merge function is the number-one-most-important thing about this whole arrangement, and has to satisfy1</a> these three laws: commutative (so merge(a, b)</code> must be the same as merge(b, a)</code>)</li> associative (so merge(merge(a, b), c)</code> must be the same as merge(a, merge(b, c))</code>)</li> idempotent (so merge(a, b)</code> must be the same as merge(merge(a, b), b)</code>.2</a>)</li> </ul> If all these are true, we can always merge two states, regardless of how many changes have been made since we last merged, and get the same result on both sides of the sync. Cool! These seem like the kinds of things that Alloy</a> would be good at modeling, so let's try it! Over the next couple of posts, I'm going to build a handful of data structures that follow these three rules and compose them into data structures we could build an application on top of. Let's get a taste for this with OR(bool, bool)</code>. First off, does it satisfy our three properties? commutative: OR(True, False)</code> is the same as OR(False, True)</code></li> associative: OR(OR(True, False), True)</code> is the same as OR(True, OR(False, True))</code></li> idempotent: calling OR(x, True)</code> gives the same result as OR(OR(x, True), True)</code>.</li> </ul> So it looks like OR</code> might be fine, but let's be sure. We can check this in Alloy by first defining3</a> a boolean: enum Bool { True, False }</code></pre> Then a merge function, which we check is correct: fun merge[a, b: Bool]: Bool { // OR a = True implies a else b } check MergeIsCorrect { // read as: given `a` and `b`, which are `Bool`, the // following condition must hold. all a, b: Bool | (a = True or b = True) implies merge[a, b] = True }</code></pre> And finally, our three properties: check MergeIsCommutative { all a, b: Bool | merge[a, b] = merge[b, a] } check MergeIsAssociative { all a, b, c: Bool | merge[merge[a, b], c] = merge[a, merge[b, c]] } check MergeIsIdempotent { all a, b: Bool | merge[a, b] = merge[merge[a, b], b] }</code></pre> Great! Alloy can't find any errors with OR</code>.4</a> We expected that, so let's break it in interesting ways. What if the merge function was XOR(bool, bool)</code>5</a> instead? fun merge[a, b: Bool]: Bool { a = b implies False else True }</code></pre> Now Alloy finds that we've broken idempotency: merge[False, True]</code> is True</code>, but merge[merge[False, True], True]</code> is False</code>. This would cause big problems if we used XOR</code> as a merge function: the act of syncing could change our values! We can imagine this sequence of events: Node A sets the value to True</code></li> Node B sets the value to False</code></li> The nodes sync, both resolving to True</code> (since XOR(True, False)</code> is True</code>)</li> The nodes sync again, both resolving to False</code> (since XOR(True, True)</code> is False</code>)</li> Further syncs would stay at False</code> until someone changed the value to True</code>, at which point we'd repeat from step 3.</li> </ol> But in a fun twist of fate, we don't have to imagine this; we can ask Alloy to generate these traces for us. First we'll make a document that contains a changeable value: sig Document { var value: one Bool, }</code></pre> Then we'll define a not</code> for our boolean and a couple of operations (flipping an arbitrary value and syncing two documents): fun bool_not[b: Bool]: Bool { b = True implies False else True } // intent: when `flip` is true, the provided document will go // from True to False or False to True in the next time step. // This serves as our proxy for someone clicking a button in // a UI or whatever. pred flip[d: Document] { // implementation: `value` is actually a set of tuples from // documents to booleans, and `value'` is the same but in the // next time step. We set the entire value explicitly // (instead of just saying like `d'.value = False`) because // otherwise Alloy could say "and what if something changes // that you weren't expecting?" // // Syntax: `->` creates a tuple, and `++` replaces all // existing tuples referencing `d` in the set. value' = value ++ d->bool_not[d.value] } // intent: merge two documents together in a way that all the // `merge` rules hold. Maybe imagine we have `d1` locally and // are getting `d2` over the network (but remember that order // should not matter!) pred sync[d1, d2: Document] { let merged = merge[d1.value, d2.value] { value' = value ++ (d1->merged + d2->merged) } }</code></pre> Now wire them up so these events can happen over time:6</a> pred init { // `Document` is a set of documents, and `->` will apply pairwise // for the sets in its arguments, so this means this says "all // documents start at `False`" value = Document -> False } fact traces { // this says "start with `init`, then pick one of these things to // happen at each time step." Putting it in a `fact` means that // any assertions we make from now on assume that we want these // instructions. init always { (one d: Document | flip[d]) or (some d1, d2: Document | sync[d1, d2]) } }</code></pre> We can now check that our sync is idempotent:7</a> check SyncIsIdempotent { // at all times, if sync happens twice in a row (the `;` syntax) then // the next value (`value'`) and the one after that (`value''`) should // be the same. always all d1, d2: Document | (sync[d1, d2]; sync[d1, d2]) implies value' = value'' }</code></pre> If we make OR</code> our merge</code> function, this works fine and Alloy can't find any counterexamples. If we use XOR</code> for merge</code>, though, Alloy finds the trace I mentioned could show up earlier. Here's what it shows us. Step</th> Description</th> Image</th></tr></thead> 1.</td> We start off with all documents at False</code>.</td> </td></tr> 2.</td> One of the documents flips to True</code>.</td> </td></tr> 3.</td> The documents sync, converging to True</code>.</td> </td></tr> 4.</td> The documents sync again, converging to False</code> because XOR(True, True)</code> (the previously synced state) is False</code>.</td> </td></tr> 5.</td> Stay here forever or return to 2.</td> </td></tr> </tbody></table> In case you're encountering them for the first time, these diagrams are generated by Alloy—they're pretty fantastic for showing other people exactly what can happen in a system. (That's also why Document 0</code> is d1</code> and so on. The documents are defined separately, and the d1</code> and d2</code> labels come from our check</code> so we can see what's doing what.) Alloy can do more than this, too. If we modify our condition to have 3 distinct documents like this: check SyncIsIdempotent { always all disj d1, d2, d3: Document | (sync[d1, d2]; sync[d2, d3]) implies value' = value'' }</code></pre> Then Alloy shows the situation getting even sillier. You only need one of the nodes to be False</code>, at which point it's possible that the documents as a whole could never converge again and just flap back and forth randomly forever. It all depends on which documents sync with which other documents. Turns out idempotency is pretty important! That's it for today. Next time we'll go from booleans-of-dubious-usefulness to counters! Big thanks to Jake Lazaroff</a> for his help reviewing this post. If you're interested in this stuff, he also recently published an interactive intro to CRDTs</a> that you may enjoy! ^{1 This is generally true but it's not always true when syncing. There's a further distinction here between operation- and state-based CRDTs. I'm not going to get into it right now, but if I understand correctly, this distinction is about reducing the data you have to transfer by relaxing one of the three requirements while adding strict causal message ordering. For now, we're going to model state-based CRDTs, where this won't come up. </div>}^{2 I think argument position doesn't matter for idempotency, since commutativity and associativity should mean we can rearrange the arguments however we like, but I don't have a proof of this. </div>}^{3 Alloy has a boolean type I could just import, but it's useful to define exactly the operations you want to understand the system better, so I've done that instead. </div>}^{4 Note that I'm not saying "Alloy proved that this is correct." That's intentional! Alloy doesn't do proofs; it only exhaustively checks models up to a certain bound (four instances of each sig</code> by default.) I have only had a handful of times where I've had to increase the bound, though. In practice, I've been more than happy to trade exhaustiveness for speed. </div>}^5Exclusive OR. In other words, "both of these can't be True</code>." Wikipedia has more details, as usual</a>. </div> ^6I'm skipping over a lot of detail here. If this is your first time seeing this, there are more details and explanation on the setup here at modeling Git internals in Alloy</a>. </div> ^7I'm pretty sure this is the right way to check this, but I had some trouble working it out and had to ask on the Alloy forum</a>. If you try to do something similar and run into difficult, you might try asking there too! </div> Modeling in-flight requests in Alloy 2023-10-02T00:00:00+00:00 In Alloy</a>, sometimes I want to model clients and servers (or any relationship where two entities are talking.) With network stuff, it's always possible for a message to not be received, or to be received twice, or out of order, et cetera. I'd like to model that! Because I just want to learn the basics here, I'll try to keep this as simple as possible. So instead of requests and responses or any other protocol, we'll stick to just messages: var sig Message {} var sig Delivered in Message {}</code></pre> In English, "We have a set of Message</code>s, some of which have been delivered and some of which haven't. Both sets can change over time." In addition to the things we define in a model like this, we have to think about the world they live in. Here I'm imagining that these messages form a stream going from some node A to another node B—no return messages, though, only one way! Next we'll take care of a little boilerplate: how the system starts (with no messages) and a do-nothing message so our traces can run forever. pred init { no Message } pred do_nothing { Message' = Message Delivered' = Delivered }</code></pre> (Message'</code> means "Message</code> in the next step." If this is unfamiliar to you, here's a previous post where I went into more detail about how time works in Alloy</a>.) Next we want to send a message: pred send_message { // action: we want exactly `one` new message // without making any other changes to `Messages` one m: Message' - Message { Message' = Message + m } // frame (holding everything else still) Delivered' = Delivered }</code></pre> We can do something similar to model the request being received: pred receive_message { // action: mark exactly one message as delivered. one m: Message - Delivered { Delivered' = Delivered + m } // frame Message' = Message }</code></pre> Note that this can't be true (that is, can't happen) if we don't have at least one message in Message - Delivered</code>, which can read as "The set of Message</code> minus the set of Delivered</code>." Finally, we set up a state machine telling Alloy when each predicate can be true over time: fact traces { init always { do_nothing or send_message or receive_message } }</code></pre> That works pretty well! If you step through the traces here, you can get instances where messages are delivered out of order, never delivered, etc. We can't get duplicate messages in this model, but that wouldn't be the worst thing in the world to add! Since this is demonstrating the lack of safety on the network, we can't add a check like "all messages are eventually delivered", but we can make assertions that the individual actions are doing what they say they will. For example, we know that if send_message</code> is true then we will always have exactly one new message and no changes to Delivered</code>: check SendMessageNextStepHasExactlyOneNewMessage { always send_message implies one Message' - Message } check SendMessageDoesNotChangeDelivered { always send_message implies Delivered' = Delivered }</code></pre> I debated including Delivered' = Delivered</code> above, though… it's a property I care about, but it's also a frame condition. If you squint and think of this like a unit test, it's just testing that the implementation is what I expect. But on the other hand, if Alloy allows express my intent this way, where's the problem? We can make similar assertions about receive_message</code>: check ReceiveMessageOnlyMarksMessagesAsDelivered { always receive_message implies one Delivered' - Delivered } check ReceiveMessageDoesNotChangeMessages { always receive_message implies Message' = Message }</code></pre> (If you're typing this in by hand and want to go back to just exploring instances, don't forget you can add a run {}</code> block at the end to generate something more explorable!) So with that, we have a fairly nice skeleton to add more network operations onto. For example, imagine that these messages mutated some state. We can now make sure that the state changes are correct in the face of out-of-order or dropped messages. Give it a try and see if you can extend the model to do that! tsort 2023-09-25T00:00:00+00:00 On the command line, you can use tsort</code> to sort a graph into a list (a topological ordering.) Say you have this chart showing what you need to get dressed: You can write all those arrows down in a file as dependencies (in order of "X blocks Y," for example "Shirt blocks Tie," "Socks block Shoes.") Shirt Tie Undershirt Shirt Shirt Belt Pants Belt Underwear Pants Socks Shoes Pants Shoes</code></pre> If you pass that to tsort</code> you get this, one order in which you could put on clothes to get dressed: Socks Undershirt Underwear Shirt Pants Tie Shoes Belt</code></pre> It kinda suggests lanes of parallelism, too. For example, you could hypothetically put socks, undershirt, and underwear at the same time. Granted, you'd have to have some machine like in Wallace and Gromit in The Wrong Trousers, but that turned out so well for them! 😆 On a more serious note, I've used this to order tasks in projects in Linear</a>. Combined with blocker relationships, it makes it clear to everyone which tasks should go first. Trying and Failing to Implement Artificial Ignorance 2023-09-18T00:00:00+00:00 I like the ideas behind artificial ignorance</a>. Summed up: You can't know all the "bad" events that can happen in a system. Modern attackers have basically unlimited kinds of attacks.</li> You can know all the unexceptional, boring events that could happen. That list might go on for a long time, but it'll end at some point.</li> </ol> System logs tend to contain all the events, so removing the unexceptional events should leave you with just the "interesting" ones. Hypothetically you could make yourself a big regex and then use grep</code> and other shell tools to look over the logs and email yourself interesting events. This won't catch just security events, either: you'll get warnings about bad configurations, disk space, etc. You'll refine this over time, too—new unexceptional events will happen over time and you’ll add them to the list. Doing this means you’ll only ever see novel things you need to deal with. I find this idea compelling. It seems analogous to running a linter on code: you get things you would not have caught otherwise that might have caused problems later. Saves time and stress, right? Except, when I tried to apply this I ran into problems. Basically, the sticking point shows up when creating the list: it grows and grows and ends up too long to manage. Having to create and maintain a list of all the things I don't care about makes for too much work, even after turning down log levels wherever I could. Over the past couple of months, I've put probably 5 or 6 hours into sitting down with journalctl</code> dumps and writing regex. Too much! I give up! I can't seem to shake the idea, though… I just like the payoff too much! That time cost has to come down, though… maybe approaching the task a different way? Maybe it would help to take this service-by-service or day-by-day instead of looking at a whole system for a whole month? Or maybe I just need to accept that some services will always produce an unreasonable amount of logs (looking at you, Gitea) and give up on them totally? Surely I can find a way forward! crunch_str 2023-09-11T00:00:00+00:00 When writing montage</a> I wanted to have a little view of my current task up in the task bar with xbar</a>. The GraphQL API made quick work of that—I just asked for the current session and printed it to stdout. Xbar can pick that up and display it, no problem. It looks like this right now as I write this: That's all well and good, but sometimes my sessions have longer names like "scrub active projects and promote inactive ones" or even longer for tickets at work. Then it takes up way too much room on the screen and pushes other stuff I want in the status bar off the screen. That's annoying! I want it to use, like… 40–50 characters, max. At this point a reasonable person would just truncate the session name to the first 40 characters or whatever and add an ellipsis. But I don't want to be reasonable! I want this to be offbeat and fun! I got to thinking about more fun ways to shorten these strings. It's safe to assume I kinda know what the task says, so could I make the task shorter by removing letters from the string that weren't essential to understanding? What if… if I know a shorter version of a word, I just substitute it. For example, I can use "&" instead of "and" or "7" instead of "seven." Since this is for me, text speak is also fine: "u" for "you", "y" for "why", etc.</li> if a word has double letters, remove 'em! "Occurrence" becomes "ocurence."</li> drop vowels from the middle of words. That transforms "ocurence" into "ocrnc"</li> drop consonants from the middle of words. "ocrnc" becomes "ornc", "onc", "oc" over time</li> if a word has two letters left, make it acronym-y. "oc" is now "O"</li> </ol> This does make the words unintelligible pretty fast, though, so I shorten longer words first (reasoning that a longer word has more chance to remain legible if shortened.) This works pretty well! There's the odd session that gets crunched into nonsense, but clicking the status in my task bar shows the full version of the session so it's fine. It's delightful, too! For example, I had a short session today to fix something in my neovim setup. The task "fix alternative test thing for our particular test layout" crunchified down to "fix arntv test thing 4 r ptclr test lout." Useful as a reminder, but inscrutable if you don't know what the session was about in the first place. Just the level of weirdness I was looking for! All that said, I implemented this whole thing as a Rust crate in the Montage workspace. I can't imagine other uses for it, but it's reusable if you want to shorten strings in a silly way yourself! You can get it at git.bytes.zone</a>. Like the rest of montage</a>, it's licensed BSD 3-Clause. Have fun! once p is true, it's always true 2023-08-28T00:00:00+00:00 In Alloy</a>, to say "once this is true, it's always true", you say: always (p implies always p)</code></pre> This is true because always</code> means "in all states starting from this one." So, the outer "always" means that the condition is always true starting from the initial time. Once p</code> is true, the implies</code> says that it will always remain true. If you did just p implies always p</code> without the wrapper, that would be saying "if p</code> is true at the initial time then p</code> will always be true." Source: Alcino Cunha's answer to my question about this on the Alloy discourse instance</a> montage 2023-08-21T00:00:00+00:00 I've liked the concept of the Pomodoro technique</a> (basically 25 minute work and 5 minutes rest on a loop through the day) for a long time. I get a lot out of it! Problem is, I'd like to use an app for this but nothing I've tried has been "sticky" enough for me to use long-term… they all have some drawback or other I don't want to live with. So like any good software person, I made my own! Long story short, you can get it at git.bytes.zone/brian/montage</a>. Here's what makes it different: Other apps I've used assume they can take over for your todo list or daily log. Montage only asks that I label my sessions for status and reporting.</li> Other apps are "polite" with easily-ignorable chimes. Montage gets in my face by yelling at me with the system text-to-speech commands every two seconds when the timer is up. I showed this to a colleague, who described it as "dystopian", but it's the dystopia I chose!</li> Most other apps I've used don't expose a nice API for session control or reporting, but Montage is built around a GraphQL API</a> and uses that for everything—there's nothing the CLI does that you can't get out of it as JSON.</li> </ul> By exposing a proper API, I was also able to build an OmniFocus</a> extension so that I can just click "start task" in the app to start something. I was also able to build an xbar</a> extension that shows my current task (or break) in the status bar. I get reports with a purpose-made command in the CLI that produces Markdown for me to copy into Obsidian</a>. But again, it's just talking to the API so I have the flexibility to produce information about my workdays in any format I want. I'm pretty happy with Montage so far, and I also have plenty of ideas for more things it should do! For example, it doesn't have any opinion on how long you should work before being reminded to take a break. I'm planning to build that in as a configurable thing in "vex" (the subcommand that talks to me with TTS.) I'd also like Raycast</a> and Obsidian extensions. At some point it may also make sense to build a GUI or an app, but for now I'm not too bothered by occasionally having to open up the SQLite database for some manual changes. Anyway, like it says in the README</a>, go ahead and give this a shot if it seems interesting to you and you're OK with being called "Brian" by a computer every 25 minutes! moving genrules to library rules 2023-08-14T00:00:00+00:00 In buck2 basics</a>, we made two rules: genrule( name = "some-target", out = "target-file", cmd = "echo 'Hello, World!' > $OUT", ) genrule( name = "yelling-target", out = "target-file", cmd = "tr '[:lower:]' '[:upper:]' < $(location :some-target) > $OUT", )</code></pre> Let's lift these to named rules so that we have a nicer API and don't mix the commands with the parameters. Let's make a couple rules to say</code> and yell</code> (to implement some-target</code> and yelling-target</code>, respectively.) Open a new file talking.bzl</code> and put this in it: def _say_impl(ctx: "context"): return [DefaultInfo()] say = rule( impl = _say_impl, attrs = { "message": attrs.string(), "out": attrs.string(default = "out"), }, )</code></pre> This won't do anything yet, but it's about the minimum possible rule that lets us do this back in BUCK</code>. I've rendered this as a diff so you can see the before and after: +load(":talking.bzl", "say") -genrule( - name = "some-target", - out = "target-file", - cmd = "echo 'Hello, World!' > $OUT", -) +say( + name = "some-target", + message = "Hello, World!", +) genrule( name = "yelling-target", out = "target-file", cmd = "tr '[:lower:]' '[:upper:]' < $(location :some-target) > $OUT", )</code></pre> Now we have :some-target</code> redefined as a call to say</code>, so let's implement it: def _say_impl(ctx: "context"): + out = ctx.actions.declare_output(ctx.attrs.out) + + ctx.actions.write(out, ctx.attrs.message) + - return [DefaultInfo()] + return [DefaultInfo(default_output = out)] say = rule( impl = _say_impl, attrs = { "message": attrs.string(), "out": attrs.string(default = "out"), }, )</code></pre> OK! Here's what's happening there: we're declaring an output file named whatever we passed in in ctx.attrs.out</code>. We didn't specify that in BUCK</code> so it's the default value, which we called "out". Importantly, whenever we declare an output we have to "bind" it by using it in an action before the end of the implementation.</li> So, next, we write the message to that out file. That binds the output.</li> Then finally, we say that that output is our default output.</li> </ul> At this point, if you run buck2 build //:yelling-target</code> it'll produce the same output. We changed how we're building the file, but the build graph as a whole produces the same thing. I really like this property; it means I can improve parts of the build graph in isolation without worrying too much about upstream or downstream dependencies. Anyway, let's do the same thing to :yelling-target</code> now by making a yell</code> rule: def _yell_impl(ctx: "context"): out = ctx.actions.declare_output(ctx.attrs.out) ctx.actions.run( [ "bash", "-c", "tr '[:lower:]' '[:upper:]' < $1 > $2", "--", ctx.attrs.src, out.as_output(), ], category = "yell", ) return [DefaultInfo(default_output = out)] yell = rule( impl = _yell_impl, attrs = { "src": attrs.source(), "out": attrs.string(default = "out"), }, )</code></pre> We have some more stuff happening here: We're calling a bash</code> script to get the shell redirection working correctly. I write a lot of little bash scripts like this when making Buck rules. There are probably better ways to do this, but it works fine for what we're doing here.</li> When we pass out</code> to the args of our script, we call .as_output()</code> on it. That lets Buck know that we're binding the output in that command.</li> We're requiring attrs.source()</code> in src</code>, which tells Buck that we're expecting some already-existing input file.</li> </ul> Now in BUCK</code>, we can modify our rules like so: -load(":talking.bzl", "say") +load(":talking.bzl", "say", "yell") say( name = "some-target", message = "Hello, World!", ) -genrule( - name = "yelling-target", - out = "target-file", - cmd = "tr '[:lower:]' '[:upper:]' < $(location :some-target) > $OUT", -) +yell( + name = "yelling-target", + src = ":some-target", +)</code></pre> Since src</code> is a source</code>, we provide the target pattern</a> of the input file we want. That means our build graph is unchanged once again! And we're done! Now we can call buck2 build //:yelling-target</code>, same as before. Tada! Anyway, that's all I have for Buck stuff—like I said in buck2 basics</a>, I'll be watching the project to see when I can use it again! buck2 basics 2023-08-07T00:00:00+00:00 I just finished an experiment with Buck 2</a> at work. We didn't end up using it, but it was more for organizational reasons than technical ones. I'll be watching the project to see if it would make sense to use in the future! One thing that held me back during the early part of the experiment was not having enough learning material around the basics. That makes sense: it's a pretty new project (at least in this iteration) and there are a lot of concepts to learn. So instead of teaching things from first principles, I want to run head-first into getting something simple set up. We won't have any more functionality than we could get with some simple make</code> rules, but since the person who does the work does the learning</a> I hope this will be helpful! That said, keep in mind that I'm no expert—just some person who learned enough to get stuff done! I'm going to assume you have buck2 available on your PATH</code>. If you don't, download it from the latest</code> tag on the facebook/buck2 repo</a>. Once you do that, you can run buck2 init --git</code> to get a basic project set up for the rest of this post. </aside> The Simplest Possible Rule</h2> We'll start off with the simplest possible thing: a file that says "Hello, World!" We'll use a genrule</code> for this, which lets you say what shell commands to run to produce some file. In the root BUCK</code> file, write this: genrule( name = "some-target", out = "target-file", cmd = "echo 'Hello, World!' > $OUT", )</code></pre> That configures a target named some-target</code> that you can build with buck2 build //:some-target</code>. If you add --show-output</code> to that, it'll tell you where the file is, and if you cat buck-out/v2/gen/root/213ed1b7ab869379/__some-target__/out/target-file</code> (or whatever path Buck gives you) it out it'll say "Hello, World!" First build done. Yay! So what do all the parts in that rule mean? genrule(…)</code> is the rule you're calling. I think "gen" here is short for "generic" (or maybe "generate"?) Anyway, it's the simplest thing that the build system can possibly do.</li> name</code> defines how you'll refer to this target from other rules or from the CLI. See the target pattern docs</a>, which also explain why you refer to this as //:some-target</code>.</li> out</code> is the file that will be produced by the rule</li> cmd</code> is the command that will be run to produce the file. $OUT</code> looks like an environment variable reference, but as far as I can tell Buck is actually interpolating it into the string before calling the command. The docs for genrule</code> have some examples of other variables you can use</a>.</li> </ul> More!</h2> Next, let's chain two rules together. Here's another rule in the same file: genrule( name = "yelling-target", out = "target-file", cmd = "tr '[:lower:]' '[:upper:]' < $(location :some-target) > $OUT", )</code></pre> The new thing here is $(location ...)</code>, which interpolates the location of some the output of the named target into the command line. I had a hard time finding out if there was anything else like location</code> that I could use. It turns out that the call there is actually a lookup for the "location" attribute on the target inside Buck's build graph. That means different rules will produce different things! But generally speaking, location</code> will work for files in general and executable</code> will work for files that you need to run. </aside> Now if you build everything and go look at the output (using --show-output</code>), you will get a new file that says "HELLO, WORLD!". If you change the original rule, the new one will get built too. So with these two rules (plus the thing in toolchain</code>), you can: Run buck2 build //...</code> to build both targets</li> Run buck2 build //:some-target</code> to build only the "Hello, World!" file.</li> Run buck2 build //:yelling-target</code> to build the "HELLO, WORLD!" file, which will build :some-target</code> as well, if necessary.</li> </ul> Try changing things around (for example by changing the message) and seeing what happens. You can also run buck clean</code> to remove the targets to get a fresh build. Next time, we'll talk about moving these from genrule</code>s into a library that we can import and call. Modeling Git Internals in Alloy, Part 3: Operations on Blobs and Trees 2023-04-24T00:00:00+00:00 In the last two posts, we've modeled Git's internal data structures in Alloy</a>. First, we handled blobs and trees</a>, then commits and tags</a>. During all that, we introduced some fact</code>s—things that Alloy will accept as true, regardless of whether those assumptions hold in real life. Today, we'll model some Git operations to see if those facts actually hold! In particular, we'll look at: hash-object</code> to get a Blob</code></li> update-index</code> and write-tree</code> to get a Tree</code></li> </ul> To do that, we're going to have to introduce changes over time to our model. In Alloy, this means two things: Marking the parts of our model that can change.</li> Defining transitions between one state and the next.</li> </ol> Once we do that, we can make assertions about time, for example that something is always or never true, or becomes true eventually given some other condition. Let's begin by redefining Blob</code> to be able to change over time. To do that, we just prepend var</code> to its definition: var sig Blob {}</code></pre> If we run this, we get new goodies in the instance viewer: On the left, we have time step 0—the initial state of the instance. On the right, we have the next step. Alloy will always show at least two time steps, and in this case Alloy also shows us that the last step repeats forever (by showing a looping arrow over the second step.) Where we had “New” before to get a new instance, we now click “New Init”—that'll get us another starting state for the model. We can also scrub back and forth on the timeline by clicking the left and right arrow buttons. If you click “New Fork”, Alloy will make something new happen starting from the second visible step (the one on the right.) I'd encourage you to load up Alloy and play around with this some—it'll look totally random right now, but that's just because we're not controlling how Blob</code> changes at all right now, so Alloy allows it to change in arbitrary ways. Adding Processes</h2> This model isn't super useful yet, since we don't actually say how blobs can change over time. Let's add some structure so we can define that! To start with, we need to define an initial state. Seems fine to start with the same thing Git does: an empty repo with no objects. pred init { no Blob }</code></pre> We'd normally use a predicate to check if a condition holds, and that's the same here except with the addition of time. Said another way, at any time that init</code> is true, there will be no Blob</code>s. We'll place this init</code> at a precise moment in time (namely, the first time step) very soon here. From there, one reasonable thing to do is to do nothing. We'll need a step like this for most models, since Alloy needs to be able to say “… and then nothing ever happens again for the rest of time.” This is traditionally called a stutter step: pred stutter { Blob' = Blob }</code></pre> We get a new bit of syntax here: saying '</code> after something means you're referring to that thing in the next time step. So Blob' = Blob</code> (which I pronounce “blob prime equals blob”) means that Blob</code> will stay exactly the same whenever the stutter</code> predicate is true. Now let's combine these! Right now, we can keep the system the same across time by saying “init is true at the beginning, and then stutter is true forever.” We'd do that with another predicate that traces those events through time: pred traces { init always stutter }</code></pre> This introduces another new piece of syntax: always</code> means “from this point going forward, this condition is always true.” In other words, we're saying “at the current time step, init</code> is true. Going forward, stutter</code> is always true.” Effectively, that means that we start with no blobs and never add any. Finally, we tell Alloy that it should assume that the traces</code> predicate is always true: fact { traces }</code></pre> Now if we evaluate the model, we see that it has exactly zero blobs forever: If we click “New Init” now, we get the usual “there are no more instances” message: Hashing Objects</h2> Now we have a blank canvas and can start adding operations. Let's start as we did in part 1</a> with git hash-object</code>. As a refresher, this command takes some content, hashes it, and stores it according to the hash. You can invoke it like this: $ echo 'Hello, Alloy!' | git hash-object -w --stdin 39528abd81b13b2731d47f86206351a61f1e6484</code></pre> In Alloy, that looks like adding a new predicate: pred hashObject { one Blob' - Blob }</code></pre> Since we only care about the presence of the blobs and not their content, it's sufficient to say, “there is one (and only one) blob in the next state that doesn't appear in the previous state.” Next, we'll tell traces</code> that hashObject</code> is one of the things that can be true at any time step: pred traces { init always { stutter or hashObject } }</code></pre> Just to reiterate what this means, we're now saying “init</code> is true at the start. Forever after, either stutter</code> or hashObject</code> is true.” If it helps, you can kind of think of this as saying “init</code> is true at the start; afterward, either stutter</code> or hashObject</code> can happen.” However, this is not strictly accurate: Alloy has no concept of events happening, only conditions which happen to be true at any given time. If you execute this spec, you can now see blobs coming into existence between time steps (you may need to click “New Fork” to get this to happen.) Adding to the Index</h2> To finish up, we're going to create trees. As a reminder, this is two operations: Adding a blob to the index at a certain path</li> Creating the tree from the index</li> </ol> On the command line, this looks like this: $ git update-index --add --cacheinfo 100644 \ 39528abd81b13b2731d47f86206351a61f1e6484 hello-alloy.txt $ git write-tree 3ee29075f260c5eebd8b9480b6464a7612668dde</code></pre> To model this in Alloy, we'll need to model the index as well as trees. First, the index: one sig Repo { var index: set Blob + Tree, }</code></pre> (This is one sig</code> because there can only be one repo in our model. We've assumed there was one implicitly—we wouldn't have been able to add blobs otherwise—it's just explicit now.) We'll also need to add trees: var sig Tree { var children: set Blob + Tree, }</code></pre> Like Blob</code>, Tree</code> is var</code>: we can create new trees from an index. But unlike Blob</code>, Tree</code> has children. These are also marked var</code>, despite an individual tree's contents never changing, because we need to be able to change the mapping of trees to blobs over time. Before we can add any events on these, we'll need to define what they look like initially, and say that they never change in the stutter step: pred init { no Blob + no Tree + no children } pred stutter { Blob' = Blob + Tree' = Tree + children' = children + index' = index }</code></pre> We could have omitted no children</code> in init</code> because there really can't be any trees or blobs present in children</code> if there are no trees or blobs in the whole system, but I find it's better to be explicit. We'll also need to add similar lines as a frame condition in hashObject</code>: pred hashObject { one Blob' - Blob + + // frame + Tree' = Tree + children' = children + index' = index }</code></pre> We need to do this because of the behavior we observed at first: if we don't control the next state of every varying thing in our model, Alloy allows it to change in arbitrary ways. This can be annoying, but so far I've found it's something I can get over in order to get the useful stuff Alloy can do. But now that we have that done, we can add blobs and trees to the index: pred updateIndex[bt: Blob + Tree] { index' = index + Repo->bt // frame Blob' = Blob Tree' = Tree children' = children }</code></pre> Our action here is saying “the next version of stage</code> is the old version plus the addition of bt</code>.” We haven't seen -></code> before in this post; it means “all of Repo</code> maps to all of bt</code>.” It so happens that we have only one thing in each of those sets, so this is effectively saying “add the mapping of repo-to-bt to the index.” If you want a firmer technical term, -></code> gives you the Cartesian product of the two sides. That means {a, b}->{1, 2}</code> would be {a->1, a->2, b->1, b->2}</code> </aside> Now that we have an index, let's take care of write-tree</code>: pred writeTree { one t: Tree' - Tree { Tree' = Tree + t children' = children + t->Repo.index } // frame Blob' = Blob index' = index }</code></pre> A couple of details here: We're dancing around a little by saying “I want one tree in Tree'</code> that doesn't exist in Tree</code>, and then that Tree'</code> is only changed by that new tree.” I feel this is a little awkward but it's the best way I can find to get exactly one new thing. If you're an Alloy expert reading this and know a better way to do this, I'd really appreciate it if you could let me know!</li> Since -></code> maps all of the left- and right-hand sides, this time we're saying “our new tree has all the things currently in the index.”</li> </ul> With that taken care of, we need to unstage the items afterward. One final action: pred resetIndex { no index' // frame Blob' = Blob Tree' = Tree children' = children }</code></pre> Now, finally, we can add these predicates to the list of conditions that Alloy will consider at every time step: pred traces { init always { stutter or hashObject + or (one bt: Blob + Tree | updateIndex[bt]) + or writeTree + or resetIndex } }</code></pre> At this point, I'd usually show some instances from the visualizer and ask if we thought they were reasonable, but that gets harder to do as you add more ways for the system to change. Instead, we're going to jump right into… Making Assertions</h2> Remember how we had this fact</code> back in the static version of the model? fact "trees cannot refer to themselves" { no t: Tree | t in t.^children }</code></pre> We added this because it didn't look like it was in the design of Git for trees to refer to themselves (that is, appear in their own children, grandchildren, etc.) Well, now that we're modeling operations instead of the data structures alone, we can ask Alloy to think about all the state transitions in the system and tell us whether that's allowed: check TreesCannotReferToThemselves { always (no t: Tree | t in t.^children) }</code></pre> Remembering that always</code> means “true at this time and forever after”, we're asking it to check if there's any series of time steps where a tree can appear in its own children</code>. Alloy can't find a counterexample for this—yay! Here's what that looks like: Executing "Check TreesCannotReferToThemselves" Solver=minisatprover(jni) Steps=1..10 Bitwidth=4 MaxSeq=4 SkolemDepth=1 Symmetry=20 Mode=batch 1..10 steps. 27665 vars. 1790 primary vars. 67627 clauses. 275ms. No counterexample found. Assertion may be valid. 45ms. . contains 9 top-level formulas. 10ms.</code></pre> Let's also check that the content of trees cannot change with any of our operations. This is as much a unit test of our modeling as it is a property of Git: check TreesAreImmutable { always (all t: Tree | always t.children = t.children') }</code></pre> And again, no counterexample found. Finally, let's do something a little trickier. We said it should be OK for files to have the same children—we assume that they'll have different filenames. We can ask Alloy to generate this as a counterexample, just to make sure it's possible. To do this, we make the opposite assertion (that two distinct trees can't have the same children): check NoDuplicateTrees { always (no disjoint t1, t2: Tree | t1.children = t2.children) }</code></pre> We get a counterexample, but not the one we wanted: it turns out that in our model you can call write-tree</code> over and over and get a new tree each time. If you do that in a real repo you'll just the same tree over and over. We could deal with this in a few ways: We could add a guard condition to writeTree</code> so that it cannot be true if more than one empty tree would be produced. BUT in real life, Git allows you to call write-tree</code> at any time.</li> We could add a precondition to traces</code> to do the same, BUT that models someone just never does the wrong thing, so I'd rather assume that anything can be called at any time.</li> We could add a fact</code> about empty trees, but every fact we have introduces assumptions to our model.</li> </ul> In this case, I think adding a fact</code> is worth it, since we don't represent the behavior of empty trees accurately now and introducing a fact helps with accuracy. Here's how we do it: fact "there can only be one empty tree" { always lone t: Tree | no t.children }</code></pre> This hinges on lone</code> enforcing that there will be zero or one Tree</code>s matching this description. We could also say no disjoint t1, t2: Tree | no t1.children and no t2.children</code>, but that's a bit of a mouthful. With the addition of the fact, though, I only see the kinds of traces in the instance viewer that I was expecting. For example: Start off with nothing: </li> hash an object: </li> stage the object: </li> write a tree: </li> without resetting, write another tree: </li> </ol> I think we're good at this point! Just to recap, today we've modeled: blobs and hash-object</code></li> the plumbing we need to make trees—a repo, the index, staging, and resetting</li> trees and write-tree</code></li> </ol> Next time we'll go a bit further by adding commits. Stay tuned! Before I leave you, though, here's the complete model from this post. It's quite a bit longer than other ones we've worked with, but it's also doing way more: var sig Blob {} pred hashObject { one Blob' - Blob // frame Tree' = Tree children' = children index' = index } var sig Tree { var children: set Blob + Tree, } fact "there can only be one empty tree" { always lone t: Tree | no t.children } one sig Repo { var index: set Blob + Tree, } pred updateIndex[bt: Blob + Tree] { index' = index + Repo->bt // frame Blob' = Blob Tree' = Tree children' = children } pred writeTree { one t: Tree' - Tree { Tree' = Tree + t children' = children + t->Repo.index } // frame Blob' = Blob index' = index } pred resetIndex { no index' // frame Blob' = Blob Tree' = Tree children' = children } pred init { no Blob no Tree no children } pred stutter { Blob' = Blob Tree' = Tree children' = children index' = index } pred traces { init always { stutter or hashObject or (one bt: Blob + Tree | updateIndex[bt]) or writeTree or resetIndex } } fact { traces } check TreesCannotReferToThemselves { always (no t: Tree | t in t.^children) } check TreesAreImmutable { always (all t: Tree | always t.children = t.children') } check NoDuplicateTrees { always (no disjoint t1, t2: Tree | t1.children = t2.children) }</code></pre> Modeling Git Internals in Alloy, Part 2: Commits and Tags 2023-04-10T00:00:00+00:00 Last week</a>, we started modeling Git's internals in Alloy</a>. We added blobs (to store content) and trees (to organize it into a filesystem.) We ended up with this model: abstract sig Object {} sig Blob extends Object {} sig Tree extends Object { children: set Object } fact "trees cannot refer to themselves" { no t: Tree | t in t.^children }</code></pre> … which produces instances that look like this: Today, we're going to add commits and tags to this model! Commits</h2> Going back to the Git Internals - Git Objects chapter of the Git book</a>, we can take a tree hash we produced in the last post and make a commit with git commit-tree</code>: $ git commit-tree 3ee29075 -m 'Commit message' 8cc0d4f4ddfde6efa9a8fced667d4d51574a36ec</code></pre> We can add more commits (and history) by repeating this command, but specifying a parent ID for each subsequent commit. $ git commit-tree 3ee29075 -m 'Second commit' -p 8cc0d4 bc8d9d27a206d0e933be3e445c82cbef09da54d1 $ git commit-tree 3ee29075 -m 'Third commit' -p bc8d9d 844bcca25118c27b0322aacd49edb73d8fac827f</code></pre> Then we can view the lineage of the most recent commit with git log</code>: $ git log 844bcc commit 844bcca25118c27b0322aacd49edb73d8fac827f Author: Brian Hicks <brian@brianthicks.com> Date: Fri Mar 3 12:36:14 2023 -0600 Third commit commit bc8d9d27a206d0e933be3e445c82cbef09da54d1 Author: Brian Hicks <brian@brianthicks.com> Date: Fri Mar 3 12:35:49 2023 -0600 Second commit commit 8cc0d4f4ddfde6efa9a8fced667d4d51574a36ec Author: Brian Hicks <brian@brianthicks.com> Date: Fri Mar 3 12:32:37 2023 -0600 Commit message</code></pre> But… would it work to commit a blob hash, or does it only work with tree? The book doesn't say, so I tried, and it looks like the hash you pass in as the first argument to git commit-tree must be a tree. If you try to make a commit based on a blob, git won't let you: $ git cat-file -p 3ee29075 100644 blob 39528abd81b13b2731d47f86206351a61f1e6484 hello-alloy.txt 100644 blob 9b4b40c2bca67e781930105fa190b9b90235cfe5 hello-blob.txt $ git cat-file -p 39528a Hello, Alloy! $ git commit-tree 39528a -m 'Can you commit a blob?' fatal: 39528abd81b13b2731d47f86206351a61f1e6484 is not a valid 'tree' object</code></pre> So it looks like a commit has to have a tree, a message, and zero or more parents (you can have more than one; this is how merge commits work.) All this is confirmed by man git-commit-tree</code>! We'll leave messages out of our model because they don't matter for any properties we might care about, but otherwise we'll add this to our model: sig Commit extends Object { parent: set Commit, tree: one Tree, }</code></pre>Finding mismatches between Git's model and ours</h3> Let's look at the instances Alloy produces and see if we think any of that feels off. To start, we get relatively normal-looking instances, such as two commits with the same tree: But we also get some wilder instances. For example, it looks like our model allows trees to have commits as children: I'm not sure whether that'd be allowed, but it's easy to verify by asking Git to add a commit to the staging area: $ git update-index --add --cacheinfo 100644 \ 8cc0d4 commits-are-stageable.txt fatal: git update-index: --cacheinfo cannot add 8cc0d4</code></pre> Nope, doesn't work. That's fine. We'll just update our definition of Tree</code> to say that they can't have commits as children. Since we're dealing with sets here, we can write “all objects besides commits” as Object - Commit</code>, which makes the new definition of Tree</code> look like this: sig Tree extends Object { children: set Object - Commit }</code></pre> That's not all the weirdness taken care of, though: we also get commits which are their own parents, or cycles of commits who are each other's parents: Like last time, this is technically possible: if you can find just the right content for the commit messages and trees, you could conceivably get a commit to refer to itself. Like before, though, this is likely to break git in some awful ways (segfaults!) If we were modeling Git to try to find bugs or security vulnerabilities, I'd say we should allow this. But, as before, we're trying to learn how this is supposed to work, so let's disallow it in the same way we disallowed trees being their own parent: fact "commits can't be their own parent" { no c: Commit | c in c.^parent }</code></pre>Tags</h2> With commits done, we have only one more object type to model: the tag. Tags are like commits, but instead of pointing to a tree and parent they point to a commit, and you can move them later (as opposed to everything else we've seen so far, which is immutable.) Here's how we'd model that: sig Tag extends Object { commit: one Commit, }</code></pre> Running the model like this shows that we've implicitly allowed trees to contain tags (because now Object - Commit</code> includes Tag</code>) which we didn't mean. We could say Object - Commit - Tag</code>, but at this point I think it'd be better to rephrase Tree.children</code> to contain only what we want: sig Tree extends Object { children: set Blob + Tree, }</code></pre> Now we can get tags on commits. Yay! We've now reached the end of the first part of our Git-modeling journey: we have all the objects! (There are also refs, though, which work like tags but aren't stored with the git objects. You can read more about those in the Git Internals - Git References chapter of the Git book</a>.) Here's the model we're finishing with: abstract sig Object {} sig Blob extends Object {} sig Tree extends Object { children: set Blob + Tree, } fact "trees cannot refer to themselves" { no t: Tree | t in t.^children } sig Commit extends Object { parent: set Commit, tree: one Tree, } fact "commits can't be their own parent" { no c: Commit | c in c.^parent } sig Tag extends Object { commit: one Commit, }</code></pre> From here, our next step is to model the operations we can take on this model to check if the properties we wrote earlier actually hold when we use Git's commands. Stay tuned! Modeling Git Internals in Alloy, Part 1: Blobs and Trees 2023-04-03T00:00:00+00:00 Today we're going to learn some Alloy</a> by modeling Git</a> objects. What's Alloy?</h2> I wrote this post for people who are meeting Alloy for the first time. If that's you, here's a quick intro: Alloy is a piece of software (with its own domain-specific programming language) used for making and checking models of other software. You use it to do "lightweight formal methods</a>", focusing on approachable language and practical applications over full specification and formal proofs. Alloy helps you think through problems by showing you both what's possible and implied in the models you give it. It can model whatever you care to express in its language, although some things lend themselves to modeling better than others. Some things I've had good luck with: data structures, database schemas, and UI states. Alloy can exhaustively check conditions in models under a certain size. This gives us a tradeoff: you don't get a full proof that your condition holds (although you can get close enough for most purposes) but it can execute your specs much faster than other tools in the formal methods space. This is almost always fine. Just be cautious about saying things like "I proved this using Alloy"—it's not that kind of tool. You can download Alloy at alloytools.org</a>. If you're interested in learning more after reading this, I've also written modeling database tables in Alloy</a> as well as a few</a> other</a> posts</a>. Git's Internals</h2> Now, to Git. Git stores all the code and commits in your repo in a content-addressable store. That means that if you know the hash of something, you can retrieve it from the store. This allows Git to do cool things like syncing and deduplication, but it's also the source of some of the weirder parts of Git's behavior from a beginner's perspective. Once I learned about the internals, I found it a lot easier to reason about what it was doing. If you're learning about them for the first time now, I hope you have a similar experience! I based these models off of the Git Internals - Git Objects chapter of the Git book</a>. I won't go into nearly as much detail about how Git works as they do there; this post will be more about modeling. If learning both the modeling tool and the domain at once is too much, go read that chapter and come back. Blobs</h2> Let's look at blobs (used for basic content storage) first. We can tell git</code> to create one like so: $ echo 'Hello, blob!' | git hash-object -w --stdin 9b4b40c2bca67e781930105fa190b9b90235cfe5</code></pre> Git gives us a hash of the content on stdout. To retrieve it again, we just ask: $ git cat-file -p 9b4b40c2bca67e781930105fa190b9b90235cfe5 Hello, blob!</code></pre> Like I mentioned above, these blob objects are content-addressed. In practice, that means they can't be duplicated. If you run the command to store Hello, blob!</code> twice, Git will give you the hash again but not change anything on disk (since the blob is already stored.) We can get a new blob, along with a new hash, by changing the content: $ echo 'Hello, Alloy!' | git hash-object -w --stdin 39528abd81b13b2731d47f86206351a61f1e6484</code></pre> (Git produces these hashes consistently across different installations, by the way. Try running them in some repo on your computer. You should get the same hashes!) Since this is basically the same behavior as sets, we can model the whole thing in Alloy like so: sig Blob {}</code></pre> This introduces a sig</code>, our basic unit of modeling. I find it easiest to think of sig</code>s as sets: we can have zero Blob</code>s, or one, or many, but never more than one of the same Blob</code> in the set. We can ask Alloy to execute this model, giving us examples of Blob</code>. If we do so, then click "Show", it gives us a window like this: You can click "New" a bunch of times here. Alloy will start off with no blobs, then show one, then two, three, and four. At four, if you click "New" again Alloy will tell you that "There are no more satisfying instances." This means that we've seen all the possible ways for there to be blobs under the current configuration. Alloy limits us to four of each sig</code> by default. We can raise this if we need, but in my experience Alloy can find most interesting things under this limit. At any rate, no matter how many blobs we have, we still haven't modeled enough to show useful things about Git. Next up in the Git book: trees! Trees</h2> Trees work basically like directories in a filesystem hierarchy. Their hash is based on the hash of all the content they contain, plus the filenames of that content. We can create one by associating blobs with filenames in the index (kinda like git add foo.txt</code> but with the object hash instead of reading from the project source.) It looks like this: $ git update-index --add --cacheinfo 100644 \ 9b4b40c2bca67e781930105fa190b9b90235cfe5 hello-blob.txt $ git update-index --add --cacheinfo 100644 \ 39528abd81b13b2731d47f86206351a61f1e6484 hello-alloy.txt $ git write-tree 3ee29075f260c5eebd8b9480b6464a7612668dde</code></pre> This gives us a tree object containing the hashes we wrote before. We might try and model trees like this in Alloy: sig Tree { children: set Blob, }</code></pre> But that won't work: trees can also contain trees! We can demonstrate using git read-tree</code> (it works like update-index --add</code>, but puts the content of the tree under the given prefix.) $ git reset --hard # to clear out the staged files $ git read-tree --prefix=greetings 3ee29075f260c5eebd8b9480b6464a7612668dde $ git write-tree a0776403e71047444987c67405757a1dbeb0f263 $ git cat-file -p a0776403e71047444987c67405757a1dbeb0f263 040000 tree 3ee29075f260c5eebd8b9480b6464a7612668dde greetings</code></pre> We need to change our modeling to allow for this. Luckily for us, it turns out that both blobs and trees belong to the category of "Git objects." Alloy makes this easy to model: abstract sig Object {} sig Blob extends Object {} sig Tree extends Object { children: set Object }</code></pre> A sig</code> being abstract</code> means that there aren't ever going to be any things that are only Object</code>, but that Object</code> can serve as a superset for anything extending it (you can think of it like an abstract class in an object-oriented language.) Blob</code> and Tree</code> are now defined as extending Object</code>. This means that they are non-overlapping subsets of Object</code>. In other words, nothing is both a Blob</code> and a Tree</code>. Finally, we give Tree</code> some children</code>. Defining it with set</code> means that we could have zero, one, or many children (which is possible in git</code>; I checked on an empty repo.) If we want one or more, we could say some</code>. There's also one</code> (which is what it sounds like) and lone</code> (which is zero or one exactly.) These are all called "multiplicity operators." You don't have to specify one, although I always do because I forget what the default is and it's nicer to be explicit anyway. Let's execute this model and have a look at some of the instances. For example, we can get totally normal-looking trees which contain other trees and blobs: However, we've accidentally allowed some other stuff like… Recursion! (dun dun dunnnnn)</h2> Our current model allows the data structures we want, but it also allows stuff that doesn't make sense in Git. For example, trees can contain themselves, and contain the trees that contain them. Here's an example with all that happening at once: This shouldn't be possible, theoretically. Remember that this is a content-addressed storage system, so you need to know something's hash to refer to it. That means for something to refer to itself, you'd have to know the hash in advance, and the hash of the thing you create would change depending on that hash. Since hash algorithms aren't perfect, you can get two values that produce the same hash (a collision) but you'd have to get really lucky (or unlucky, depending on your perspective.) I was curious about whether this was actually even possible or if Git would reject it, so I did some digging. Someone on Reddit</a> had the same question and changed the source code to be dramatically more likely to have a collision (by changing the code to only use the first byte of the SHA-1 hash.) They got a segfault, which tells me that this isn't something that the Git authors designed for. Sidebar: what are we modeling, again? We hit a fork in the road here in practical modeling terms: we can either allow or disallow this case in our model. This happens to me a lot when using Alloy; it's good at bringing up weird cases I hadn't previously considered. In this case, I think you can make a good argument for either direction: If we allow recursion in trees, we're keeping our eyes open to the fact that SHA-1 is not completely collision-free, or that some day it may be broken in a way that makes it trivial to find collisions.</li> If we disallow recursion in trees, we simplify our conceptual model significantly. We're not nearly done with the data model (we still have commits, refs, tags, etc to add) and allowing every single little edge case will make it harder to understand the system as a whole. In this case we're modeling to learn about Git's internals, so I think disallowing this case makes more sense. Git clearly relies on the fact that it's very hard to cause a cycle here. If I was modeling Git objects to (for example) find bugs or potential security issues, I think I'd make the opposite decision.</li> </ul> </aside> We'll use a fact</code> to disallow this case. When we declare something as fact</code>, Alloy will not allow any instances which would make the fact false. We should use these carefully—as in other kinds of reasoning, more axioms mean making a weaker argument (even though it's hard to not use any at all.) fact "trees cannot refer to themselves" { no t: Tree | t in t.^children }</code></pre> The ^</code> operator is new to us—it follows the "children" relationship one or more times. That means that we're saying "there is no Tree</code> that is it's own child, or that child's child, or that child's child's child, and so on." Once we do this, Alloy cannot find any more instances where trees refer to themselves, but it can still find plenty where trees contain other trees or blobs. Success! We can, however, still get two trees pointing to the same blob. If we assume that these two trees are using the same filename for the blob, this shouldn't be possible, because they'd have the same hash. So we find ourselves at another fork in the road: we could either model naming files within trees or not. Like last time, you can make a convincing argument either way: If we add names, we'll be sticking closer to the way the actual system works. Trees exist as the equivalent of a directory in a filesystem, so it's a little weird to ignore names totally.</li> If we don't add names, we'll keep the model simpler. However, we can assume that any time we see two trees with the same contents, at least child has a different name. Since a directory entry is a string, and there are effectively an infinite amount of strings, this seems like a safe assumption.</li> </ul> Again, I have to ask what we're modeling. If, right now, we were modeling the way that git checkout</code> turns a tree into files and directories in the filesystem, we might want to add names. But since we're trying to understand Git's internal structure, it probably makes sense to leave them out (it makes our fact</code> above much nicer, for one.) I usually add a comment documenting this, though, and I will in our final model. That's all… for now</h2> Today we've learned: how to make a sig</code>, how to connect two sig</code>s together, and how to force Alloy to disallow certain cases. We've also (maybe) learned a little bit about blobs and trees. The full, final model is below: abstract sig Object {} sig Blob extends Object {} sig Tree extends Object { // We're not modeling the mapping from name to object // for simplicity's sake. If you see two trees with the // same children, assume it's fine because the children // have different file names. children: set Object } fact "trees cannot refer to themselves" { no t: Tree | t in t.^children }</code></pre> Over the next couple of posts, we'll build on this by: adding more Git object! Commits, references, and tags.</li> modeling how commands like git hash-object -w</code>, git update-index</code>, and git write-tree</code> modify the object database to verify that statements like "trees can't refer to themselves" are actually true given the operations we can run.</li> modeling how everyday Git commands (the "porcelain"; things like git add</code> and git commit</code>) map onto these lower-level operations.</li> and, if we're very lucky, we can model how git</code> syncs this all around when you run git push</code> or git pull</code>.</li> </ul> Stay tuned! If you'd like to be notified when I publish any of those posts, you can subscribe below. aligning Markdown tables in Helix 2023-03-13T00:00:00+00:00 One of the things I like most about using Helix</a> (and Kakoune</a>, when I used it before) is the multiple cursor support. I use them whenever I can! Today, I want to share how Helix solves a common task: formatting a Markdown table. When I first write Markdown tables, they tend to look like this (with some data from this LEGO Database on Kaggle</a>.) Ragged-right formatting and just enough syntax for a parser to tell the headers from the rows: | ID | Name | Category ID | |-|-|-| | 11092 | Gorilla Fist | 27 | | 11212 | Plate 3 x 3 | 14 | | 11209 | Tyre 21 x 9.9 | 29 | | 11640pr0003 | ELECTRIC GUITAR SHAFT Ø3.2 NO. 3 | 27 |</code></pre> This looks fine when rendered, but it's somewhat annoying in the source. In other editors I've used, you either need to do the alignment by hand (tedious!) or use some plugin. In Helix and Kakoune, doing this is fairly simple: you just need to combine some editing primitives. Bottom-line-up-front, the key sequence is mips\|<ret>&</code>. But that looks just like line noise, so let's break it down: First, I hit mip</code> to select the whole paragraph, then select pipes (s\|<ret></code>.) That gives me one selection per pipe character: Then I hit &</code>, the “align selections” operator, which gets me the final alignment: | ID | Name | Category ID | |- |- |- | | 11092 | Gorilla Fist | 27 | | 11212 | Plate 3 x 3 | 14 | | 11209 | Tyre 21 x 9.9 | 29 | | 11640pr0003 | ELECTRIC GUITAR SHAFT Ø3.2 NO. 3 | 27 |</code></pre> This still renders just fine, but it's a little nicer reading experience to have the line separating the headers from the body rows full of hyphens. That one's easy too: after dismissing the selections with ,</code>, you can navigate to the row, hit x</code> to select the whole line, then s</code> to select characters, enter space as the section, and r-</code> to replace all the selections with hyphens. Now it's a perfectly nice Markdown table that's very pleasant to read in the source: | ID | Name | Category ID | |-------------|----------------------------------|-------------| | 11092 | Gorilla Fist | 27 | | 11212 | Plate 3 x 3 | 14 | | 11209 | Tyre 21 x 9.9 | 29 | | 11640pr0003 | ELECTRIC GUITAR SHAFT Ø3.2 NO. 3 | 27 |</code></pre> how to add weak fairness on actions 2023-03-06T00:00:00+00:00 In Alloy</a>, you can add weak fairness to actions (that is, ensure that they'll always eventually fire if they're able) by saying: (eventually always p) implies (always eventually q)</code></pre> In English, this means: "if p</code> is eventually true, then q</code> will eventually be true at all points afterwards." In other words, p</code> is the condition under which q</code> can happen. Note that you should be careful about existential (some</code>) vs universal (all</code>) quantifiers with this kind of statement. For example: all t: Thing { (eventually always enabled[t]) implies (always eventually do[t]) }</code></pre> Saying all</code> here means that every time the condition enabled</code> is true the action do</code> is eventually taken. If this said some t: Thing</code> instead, that would mean that this condition would only be true some of the time. Source: Alcino Cunha's answer to my question about this on the Alloy discourse instance</a> give aliases when inverting relations 2023-02-27T00:00:00+00:00 Say you're using Alloy</a> to model some tree structure. For example, a file system with directories and files: abstract sig Node { parent: lone Node, } sig File, Directory extends Node {} fact "a node cannot be its own parent" { no n: Node | n in n.^parent } fact "there is only one root node" { one n: Node | no n.parent }</code></pre> This gives us nice file system objects that always terminate in a single node and without any recursive shenanigans, like this one: However, the model currently implicitly allows this that we don't want to include based on our mental model of the filesystem. For example, files that have other files as their parent: That doesn't work: a directory can have children, but a file needs to have contents instead (e.g., images, text, etc.) So let's disallow that! We'll do that by adding a new fact like “files shouldn't have any children.” We could do that by saying this: fact "files shouldn't have any children" { no File.~parent }</code></pre> (In case you haven't encountered the ~</code> operator before, it just flips the relation. That makes it child-to-parent instead of parent-to-child. More about this in fields as sets</a>) Alloy can work fine with this, but in my opinion, using ~</code> like this makes the intent of the spec less clear. I always have to think about what the ~</code> is changing, even if I wrote the code not ten minutes before! Fortunately, we can clarify this by making an alias with a fun</code> and then define our fact</code> in those terms instead: fun children: Node -> Node { ~parent } fact "files shouldn't have any children" { no File.children }</code></pre> This seems much clearer to me; the assertion we're making reads almost exactly how we'd write it in English. I like that a lot! It makes communicating about the spec easier, both for me and for anyone else who ends up reading it. In general, I'm finding that I really like to use fun</code>s to clarify my intent, or to put a durable label on some form of node. Doing this requires a little mindset shift from writing pred</code>s or fact</code>s, since fun</code>s have to yield a subset of data instead of a boolean, but in my experience it results in higher-quality communication! what is the randomart image for? 2023-02-20T00:00:00+00:00 When you generate an SSH key (like I did when looking at signing commits with SSH keys</a>), you get a “randomart image” from ssh-keygen</code>. These things: +--[ED25519 256]--+ |..o | |.+ | | .E | |.. . +o o | | .+. S+=+ | | .... o.*+o* | | oo*.o+oo. | | o o=+.oo+ | | .o.oo==B++ | +----[SHA256]-----+</code></pre> That looks nice, and I appreciate art in my terminal, but what is it for? So, let's see… first, I looked in man ssh-keygen</code>, because that's the context in which I see these, but it doesn't have any hits for randomart</code> or art</code>. But on a hunch I tried man ssh</code>, and that page has some info! Here's what it says: Because of the difficulty of comparing host keys just by looking at fingerprint strings, there is also support to compare host keys visually, using random art. By setting the VisualHostKey option to “yes”, a small ASCII graphic gets displayed on every login to a server, no matter if the session itself is interactive or not. By learning the pattern a known server produces, a user can easily find out that the host key has changed when a completely different pattern is displayed. Because these patterns are not unambiguous however, a pattern that looks similar to the pattern remembered only gives a good probability that the host key is the same, not guaranteed proof. To get a listing of the fingerprints along with their random art for all known hosts, the following command line can be used: $ ssh-keygen -lv -f ~/.ssh/known_hosts</code></pre></blockquote> Ok, that makes a lot of sense! There have been plenty of times when I've logged on to a new server, and it asks me if I want to trust key such-and-such. I mostly just say “yeah, that's fine” without paying attention, so I see how the art could help me remember what's right! If I run the command given in the manual page, I get this for GitHub: 256 SHA256:+DiY3wvvV6TuJJhbpZisF/zLDA0zPMSvHdkr4UvCOqU github.com (ED25519) +--[ED25519 256]--+ | | | . | | o | | o o o . | | .B S oo | | =+^ =... | | oo#o@.o. | | E+.&.=o | | ooo.X=. | +----[SHA256]-----+ 2048 SHA256:nThbg6kXUpJWGl7E1IGOCspRomTxdCARLviKw6E5SY8 github.com (RSA) +---[RSA 2048]----+ | =+o...+=o.. | |o++... *o . | |*.o. *o. | |oo. ..o.= . | |.+o. .. S = | |*=+ . o = . | |OE . . o | | o . | | | +----[SHA256]-----+ 256 SHA256:p2QAMXNIC1TJYWeIOttrVc98/R1BUFWu3/LiyKgUfQM github.com (ECDSA) +---[ECDSA 256]---+ | .o=X*+ .o.=| | .o=O o | | . . . E . .| |o .. . . o | | + . +S o.o . .| |. . . o++.... o.| | o o. ...+| | o . o .oo.| | . ... o....| +----[SHA256]-----+</code></pre> The manual also mentioned that you can set VisualHostKey</code> to yes</code> in your SSH client config to get that information every time. If I do that, I can see that my SSH client is using the ED25519 key: $ ssh git@github.com Host key fingerprint is SHA256:+DiY3wvvV6TuJJhbpZisF/zLDA0zPMSvHdkr4UvCOqU +--[ED25519 256]--+ | | | . | | o | | o o o . | | .B S oo | | =+^ =... | | oo#o@.o. | | E+.&.=o | | ooo.X=. | +----[SHA256]-----+ PTY allocation request failed on channel 0 Hi BrianHicks! You've successfully authenticated, but GitHub does not provide shell access. Connection to github.com closed.</code></pre> It shows a different image when I git push</code> to my dotfiles repo</a>, which also makes sense—they're different servers, so they have different keys. I left this on for a while and see if I started to be able to recognize the randomart images in the way the manual implies I'd be able to, and it turns out it worked! For example, I've started seeing the GitHub fingerprint (above) as something like the Statue of Liberty if it were a cat (don't ask why; that's just how my brain sees it), and the signature for git.bytes.zone</code> (below) as the Vagrant logo. How interesting! +---[RSA 2048]----+ |.+=+. **o| |...+ . *.o+| |o + o . = =. | | + . + . E . + | | o .+ S = + .| | o .o+ . o o o | | ..o.+ o . | | o... . | | oo | +----[SHA256]-----+</code></pre> All in all, I think I'd recommend doing this. Even if the keys never change, it's pleasing to see the art show up in your terminal. And if they do, I now have one more tool to know if something's wrong. One final note: I mentioned this in the Recurse Center</a> Zulip and someone mentioned that they had recently come across the paper that describes the algorithm for making these</a>. Now we know! fields as sets 2023-02-13T00:00:00+00:00 Today, let's look at how relationships between sig</code>s in Alloy</a> work. Say, for example, you're modeling database tables (as I have previously</a>) and want to represent the relationship between people and their favorite flavor of ice cream: sig Person { favoriteIceCream: lone Flavor, } sig Flavor {}</code></pre> You can ask Alloy for examples of this, and it will show you a bunch of different ways Person</code> and Flavor</code> can be related. For example, here's an instance where there are three distinct people who each like a different flavor: The favoriteIceCream</code> relationship here can be used in a bunch of interesting ways. For example, if we want to assert (contrary to our use of lone</code>) that everybody has a favorite ice cream flavor, we'd do it like this: check NobodyDoesntLikeIceCream { all p: Person | some p.favoriteIceCream }</code></pre> Alloy finds a counterexample for this (since we used lone</code>, which means some people don't have a favorite flavor of ice cream), but it demonstrates the first usage of the relation: looking up fields as if they were methods or attribute accesses in an object-oriented language. x.fieldName</code> will always work basically like you might have already intuited. Relations can do more than this, though, since they're actually secretly sets of tuples. If you load up the instance above in Alloy and examine it in the table view, you'll see that it's defined something like this: this/Person</code></th> favoriteIceCream</code></th></tr></thead> Person$0</code></td> Flavor$2</code></td></tr> Person$1</code></td> Flavor$1</code></td></tr> Person$2</code></td> Flavor$2</code></td></tr> </tbody></table> This looks suspiciously like a table in a database, right? Well, good news: it basically works that way too! What .</code> does under the hood is essentially equivalent to a SQL join: it selects rows matching a pattern on the left and gives you access to the equivalent rows on the right. The cool thing about .</code> is that it can do this with a set in either position. Even though it looks like you're using a single value with p.favoriteIceCream</code>, p</code> is actually a subset of people—it just happens to be one with only one member. So what other sets can you do this to? Well, if we add a children</code> field like this: sig Person { children: set Person, favoriteIceCream: lone Flavor, }</code></pre> We could look at some parent p</code> and get the ice cream flavors they'd need to buy for their children with p.children.favoriteIceCream</code>, or their grandchildren with p.children.children.favoriteIceCream</code>. Chaining works in the way you'd expect, except with set semantics instead of having to do loops or list comprehensions. We could also get the set of reachable ice cream flavors by using the whole set of Person</code> on the left: Person.favoriteIceCream</code>. We could also get the sad, lonely flavors that nobody loves with Flavor - Person.favoriteIceCream</code>. That's not all you can do, though. The ~</code> operator flips a relation around: if favoriteIceCream</code> is a mapping from Person</code> to Flavor</code>, ~favoriteIceCream</code> is one from Flavor</code> to Person</code>. We can then do the same tricks as above. For example, we can find out who doesn't like ice cream at all by doing Person - Flavor.~favoriteIceCream</code> (since that will be all the people, minus the people who have a favorite ice cream set.) For me, realizing that basically everything is a set, and that ~</code> and .</code> worked set-wise unlocked a lot of uses for Alloy that I hadn't considered before. I hope reading this helped you, too! By the way, we haven't even gone over all the interesting things you can do: Hillel Wayne's documentation site</a> has a big section on sets and relations</a> that's worth a read if you'd like to learn more. signing commits with SSH keys 2023-02-06T00:00:00+00:00 I sign all my git commits. “Why” is not super important here (but basically: I participate in some open source organizations for whom supply chain security is important.) Right now, I want to focus on nice ways to make the signatures. To start with, here are some GitHub docs</a> on signing commits with different kinds of keys. I currently use GPG keys, but they're kind of annoying to manage (at least for me.) A better option for me might be to sign my commits with SSH keys. But, what are the consequences of doing that? I'm gonna run some experiments to figure that out. To start with, I use two git forges: GitHub and Gitea</a> (although I plan to migrate to Forgejo</a> once it's in a stable version of nixpkgs.) I want to make sure that I can get verified commits on both. I'm most interested in what happens when I revoke keys. Generally speaking, I generate one SSH key per machine I use for development. That way, no key ever has to leave the machine it was generated on. So what happens if I retire a machine, along with its key? Will all my commits turn into unsigned commits? GitHub's documentation implies that it might be fine, but I don't know. Testing Stuff Out</h2> Well, then: let's try! I'm gonna create a new repo and key: $ git init test-ssh-key-signing Initialized empty Git repository in test-ssh-key-signing/.git/ $ cd test-ssh-key-signing $ ssh-keygen -t ed25519 -f test-key.ed25519 Generating public/private ed25519 key pair. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in test-key.ed25519 Your public key has been saved in test-key.ed25519.pub The key fingerprint is: SHA256:mevdSg3mn8+unpYYaB7+Sw2D6k4/VmViamKCEHHma0M brianhicks@sequoia.local The key's randomart image is: +--[ED25519 256]--+ |..o | |.+ | | .E | |.. . +o o | | .+. S+=+ | | .... o.*+o* | | oo*.o+oo. | | o o=+.oo+ | | .o.oo==B++ | +----[SHA256]-----+</code></pre> First, I'll make a commit that's signed with my normal GPG key (which I have set globally): $ git commit --allow-empty -S -m "this commit has been signed with my regular GPG key"</code></pre> Then I'll reconfigure the repo to sign using the local key and make another commit: $ git config gpg.format ssh $ git config user.signingKey "$(pwd)/test-key.ed25519" $ git commit --allow-empty -S -m "this commit has been signed with the test SSH key"</code></pre> Now I can check the signatures on the commits. Unfortunately, it looks like git</code> needs some additional configuration to verify the signature with the SSH key: $ git log --show-signature error: gpg.ssh.allowedSignersFile needs to be configured and exist for ssh signature verification commit 60c4afaae4bb3a2013648a26a99b994e868ee4ae (HEAD -> main) No signature Author: Brian Hicks <brian@brianthicks.com> Date: Thu Jan 19 11:02:37 2023 -0600 this commit has been signed with the test SSH key commit 04627098cc096584dac81c43ad70b40851880c18 gpg: Signature made Thu Jan 19 11:02:12 2023 CST gpg: using RSA key 66BAD9732604D23EF4A55B75C4F324B9CAAB0D50 gpg: issuer "brian@brianthicks.com" gpg: Good signature from "Brian Hicks <brian@brianthicks.com>" [ultimate] Author: Brian Hicks <brian@brianthicks.com> Date: Thu Jan 19 11:02:12 2023 -0600 this commit has been signed with my regular GPG key</code></pre> Very well, let's set an allowed signers file up. I found some instructions on GitLab to set up the allowed signers file</a>: $ echo "$(git config --get user.email) namespaces=\"git\" $(cat test-key.ed25519.pub)" > allowed_signers $ git config gpg.ssh.allowedSignersFile "$(pwd)/allowed_signers"</code></pre> Now I can get both signatures verified: $ git log --show-signature commit 60c4afaae4bb3a2013648a26a99b994e868ee4ae (HEAD -> main) Good "git" signature for brian@brianthicks.com with ED25519 key SHA256:mevdSg3mn8+unpYYaB7+Sw2D6k4/VmViamKCEHHma0M Author: Brian Hicks <brian@brianthicks.com> Date: Thu Jan 19 11:02:37 2023 -0600 this commit has been signed with the test SSH key commit 04627098cc096584dac81c43ad70b40851880c18 gpg: Signature made Thu Jan 19 11:02:12 2023 CST gpg: using RSA key 66BAD9732604D23EF4A55B75C4F324B9CAAB0D50 gpg: issuer "brian@brianthicks.com" gpg: Good signature from "Brian Hicks <brian@brianthicks.com>" [ultimate] Author: Brian Hicks <brian@brianthicks.com> Date: Thu Jan 19 11:02:12 2023 -0600 this commit has been signed with my regular GPG key</code></pre> Cool. Now let's test how forges deal with rotating these keys. GitHub</h2> GitHub says that SSH key signing is less involved but lacks features, namely that an SSH signing key can't be revoked. Fair enough; there's no mechanism for that. They don't say what happens when you roll keys, though, so let's try it. First, I'll push my test repo to GitHub: $ gh repo create --private --source . --push ✓ Created repository BrianHicks/test-ssh-key-signing on GitHub ✓ Added remote https://github.com/BrianHicks/test-ssh-key-signing.git Enumerating objects: 3, done. Counting objects: 100% (3/3), done. Delta compression using up to 10 threads Compressing objects: 100% (2/2), done. Writing objects: 100% (3/3), 1.25 KiB | 1.25 MiB/s, done. Total 3 (delta 0), reused 0 (delta 0), pack-reused 0 To https://github.com/BrianHicks/test-ssh-key-signing.git * [new branch] HEAD -> main branch 'main' set up to track 'origin/main'. ✓ Pushed commits to https://github.com/BrianHicks/test-ssh-key-signing.git</code></pre> I haven't uploaded the new key, but I have enabled vigilant mode</a>, so their web UI shows the commit as unverified: However, if I go to my settings and add the SSH key as a signing key, it shows as verified! If I remove the signing key, the commit status goes back to unverified, which makes sense. Since I uploaded this key as a signing key instead of an authentication key, it does not let me log in: $ ssh -o "IdentitiesOnly=yes" -i test-key.ed25519 git@github.com git@github.com: Permission denied (publickey).</code></pre> Good! I'd be uncomfortable allowing authentication from machines with decommissioned keys, but I'm at least OK-ish with saying “the commits that I made with this are still valid forever”, as long as I can be reasonably confident that the key material is completely destroyed, so no new commits can be made. If I wanted to say "any commits with signatures after this date should not be considered verified", I could use a GPG key (but you can maybe still mess around with that, since you can change the date of a commit?) Anyway, signing commits in this way seems like it satisfies my security requirements, so it might be possible. I still need to look at Gitea, though! Gitea/Forgejo</h2> For Gitea/Forgejo, there's some documentation by Gitea</a>. It only mentions GPG keys. I wonder if it'd “just work” for SSH-based signing, though? Let's find out! I don't have the CLI installed for Gitea, so I created a private repo in my account and pushed to it. Afterward, the commit shows up as unverified, as expected: I added the key to my account as a regular authentication key, but that didn't change anything. However, I noticed a new “Verify” button in the web UI. Once I verified my SSH key (by signing a string with some instructions provided on the page) the commit showed up as verified: This creates a problem, though: it doesn't look like I can accept the key for signing but not authentication. For example, if I try to authenticate to my git host with the key it lets me: $ ssh -o "IdentitiesOnly=yes" -i test-key.ed25519 git@git.bytes.zone PTY allocation request failed on channel 0 Hi there, brian! You've successfully authenticated with the key named brianhicks@..., but Gitea does not provide shell access. If this is unexpected, please log in with password and setup Gitea under another user. Connection to git.bytes.zone closed.</code></pre> This is maybe worth asking either the Gitea or Forgejo development teams about. Of course, I could just be thinking incorrectly about the security model here: it could be the case that they don't want to allow this because you can't revoke SSH keys. Either way, it seems like if I rotate my keys, I either have to allow authentication access to old keys forever, which I really don't want to do. Looks like I'll be sticking with my GPG key for signing commits for now! What does expect 1 mean in Alloy? 2023-01-30T00:00:00+00:00 When you open up Alloy</a>, the default run</code> in the menu is “Run Default for 4 but 4 int, 4 seq expect 1.” I know what “for 4” means (up to 4 of every sig</code>) and but 4 int</code> means (4 bits worth of integers) but I don't know what “expect 1” means! Let's look in the spec</a>… nothing! It shows the “run X” and “for Y” and “but Z SigName” but not “expect.” 🤔 OK, on to the Alloy forums, then</a>. Daniel Jackson says it allows you to say how many results you expect. You used to be able to expect an arbitrary number, but that didn't turn out to be useful, and now you can just say “1” (for some results) or “0” (for no results.) It's useful for testing regressions when developing Alloy itself. I was able to use expect 0</code> to produce a spec that fails, as expected: fact { some u: univ | u not in univ } run {} for 4 expect 0</code></pre> (In English: “I want you to check that the set of all sets does not contain itself.” This is not possible in Alloy's set semantics.) When evaluating this, Alloy says, “No instance found. Predicate may be inconsistent, as expected.” Note the “inconsistent”, meaning “you've created a conflict in your expectations.” If I switch it to expect 1</code>, it says “Predicate may be inconsistent, contrary to expectation.” advice you might as well take 2023-01-22T00:00:00+00:00 I've read some nice articles recently which I can sum up as “advice you might as well take.” This is stuff that's good to consider at the beginning of a project, or when you're about to add a feature to some existing software. These articles run counter to YAGNI (You Ain't Gonna Need It), the software design principle that says you should only ever add things you'll be using right away. Many even call this out in their titles! One even addresses this by coining an alternative term, PAGNI (Probably Are Gonna Need It), so let's start with that one: PAGNIs: Probably Are Gonna Need Its</h2> PAGNIs - Probably Are Gonna Need Its</a> by Simon Wilson. Makes a good rule for when to allow complexity in advance: When should you over-ride YAGNI? When the cost of adding something later is so dramatically expensive compared with the cost of adding it early on that it’s worth taking the risk. On when you know from experience that an initial investment will pay off many times over. </blockquote> His PAGNIs: make kill-switches for mobile apps to force upgrades or contain security issues, have automated deployments and continuous integration, build pagination, and keep detailed API logs, including POST bodies. Preemptive Pluralization is Probably Not Evil</h2> Preemptive Pluralization is Probably Not Evil</a> by swyx. Summed up by this quote: Before you write any code — ask if you could ever possibly want multiple kinds of the thing you are coding. If yes, just do it. Now, not later. </blockquote> Basically, if: it's a lot of effort to go from being able to handle zero things to being able to handle one thing</li> AND, it's a similar amount of effort to go from one to many</li> </ul> Then it probably makes sense to go directly from zero to many if you think that'll ever be needed. There's also some interesting discussion on the C2 wiki</a>, including some times when it's good to avoid this. Out of these pieces of advice, this is the one I've used the most. However, I'd recommend modeling out your system with something like Alloy</a> first and seeing what that gets you! You Might as Well Timestamp It</h2> You might as well timestamp it</a> by Jerod Santo. Summary: if you store boolean fields as nullable timestamps, then you have basically the same semantics but get the ability to say when the value went to “true” for free. You do use a little more disk space, but not much, so you'll have to decide whether that's worth it. (At least in PostgreSQL, booleans are 1 byte while a timestamp is 8.) However, I would imagine that's fine for most uses. Probably Are Gonna Need It: Application Security Edition</h2> Probably Are Gonna Need It - Application Security Edition</a> by Jacob Kaplan-Moss. Inspired by Simon Wilson's PAGNI post, but focused on security. There's too much good stuff in here to summarize, but some highlights for me were to always consider the “abusive ex” persona when you're designing your app and his thoughts on security admin interfaces. Summing Up</h2> These articles show a ton of cases where it's worth breaking YAGNI. I appreciate this because, at a higher level, I think that oversimplifying our mental model of the software we're building can risk failure just as much as overcomplexity can. Even worse, by oversimplifying we can harm the people who use our software by failing to consider their safety as we're building (e.g. the “abusive ex” persona.) I want to close with a quote I really love from Hillel Wayne's Reject Simplicity, Embrace Complexity</a>: Simplicity is good. We should write simple code. But complexity is unavoidable. We do a disservice to ourselves by pretending that any software can be simple if we just try hard enough. Instead, we should study the factors that lead to complex software. That way we can learn how to recognize, predict, and manage complexity in our systems. And then we can seek simplicity within that context. It won’t give us simple software, but it will help us write simpler software. Nuance is better than mantras. </blockquote> pagerank for my Obsidian notes 2023-01-17T00:00:00+00:00 I recently learned a little more about how the Pagerank</a> algorithm works. I thought it was cool, and it made me wonder what the ranks for my Obsidian</a> vault looked like, so I built a little tool. It's fairly simple: it just parses [[links]]</code> out of the vault with a regex and then uses the simple-pagerank</a> crate for Rust to calculate the ranking. For my vault, the most centrally connected node is my employer. It's the top node by quite a lot (with a score of 4.9), followed by the town where I live (2.0), then members of my family (around 2) and notes on projects I'm doing and tools I use (around 1.) This makes a lot of sense to me; I primarily use Obsidian as a way to keep a work log and write about things related to that, as I wrote about previously</a>. If you'd like to try this for yourself, you can get the code at bytes.zone/brian/obsidian-pagerank</a>, although you'll need to have either Nix</a> or Rust</a> installed in advance. If you decide to try it, let me know if you find anything interesting! The Value of a Model is More Making than Having 2023-01-09T00:00:00+00:00 After making a bunch of models in Alloy</a>, I'm curious if formal modeling is more useful as a thing to do (a process) or a thing to create (an artifact.) Making / Process</h2> On the modeling-as-a-process side, I've had good luck taking the time to carefully model a system and asking domain experts how the model is wrong. In those cases, I tend to get one of two responses: “Oh, we take care of that case with X” (which means I need to change my model)</li> “Uh-oh, we hadn't thought of that” (which means we might need to fix bugs or make other changes to the system in question)</li> </ul> For example, I recently modeled an upcoming feature my team would be working on and found several problems (in this case, actions we hadn't disallowed that would violate our assumptions about what data would be shown on the page.) I brought those to the folks in charge of the spec, and it turned out they had considered several—but not all—of them and that they had already decided that the consequences were acceptable. I'm still glad I did it, though, since it gave me a much better sense of what we were building and the consequences of our approach. I've also found it useful to sit down with other people and have a conversation using formal modeling; the model acts as a conversation partner to keep us from assuming different things about the world and point out problems in our thinking. This has even been useful when one or more of the humans involved hasn't known about formal modeling in advance because I can explain stuff on the fly. For example, as I wrote previously</a>: For example, at work we were building a feature with inline commenting (like Google Docs.) We started by modeling Google Docs' comments and it turned out that each comment had something like 32 unique states. We realized this was going to be way too much for our use case, and simplified it down to 4. We probably could have had this realization without using Alloy, but it made the conversation much simpler—the designer and I sat down and looked at the visualizations, then kept saying "but for us, states X and Y would be the same…" until we arrived at a simpler model. It was really nice, and only took about an hour and a half of modeling to come up with a solution that satisfied all our requirements! </blockquote> This is basically “lightweight formal methods</a>”—we're not out to solve the Byzantine generals problem or whatever, but just to make sure there aren't any big issues we hadn't thought of or flaws we can't live with. Having / Artifact</h2> On the models-as-artifacts side, I've had some success, but less than modeling-as-a-process. To start, I've tried to turn Alloy and TLA+ models into documentation. Alloy's ability to generate examples and its literate markdown support is amazing, but both still require the reader to have some basic familiarity with Alloy's syntax and how to interpret the visualizations. This can work really well for tutorials and explorations, though: modeling database tables in Alloy</a> was originally written as a literate Alloy file (and still lives in my notes in a form that Alloy could pick up and run.) At work, we now have a few documentation files explaining different features and referencing a common tutorial I wrote for background. It works alright, although I'm not sure how many people have read them (but I have sent links to folks who wanted a refresher on the features, which seems to have worked.) Beyond documentation of fairly static features, the model and the actual system almost always diverge. People add features and fix bugs in ways that violate the assumptions or assertions of the model, at which point the model is making strong claims that don't match up with real life. The weird thing, though, is that this is something you want. It's not useful to specify your model all the way down to the level of your code. It can be distracting to have random states changing that don't matter for the thing you're trying to assert about the model. The map is not the territory</a>, after all! For example, your real-life HTTP service always has to deal with network problems, but your model might not need to. It depends on what you're trying to figure out about the system, and that might change over time as you make models of different parts of the system! So which is it?</h2> Based on my experiences with formal modeling tools, I think the process of creating a model is more valuable than the thing you produce. I've seen fewer drawbacks to approaching modeling as a process or as a communication tool without worrying about preserving the model for the long term. However, my opinion here may change as I practice more. I've already produced a few models-as-documentation for stabler and longer-lived parts of systems at work, and those have been helpful to introduce people to the concepts involved in an isolated way. Maybe trying to find more ways that models have long-term value will change my mind here! Thank you to Ezzeri Esa and Miccah Castorina at the Recurse Center</a> for reviewing a draft of this post! Modeling Database Tables in Alloy 2023-01-02T00:00:00+00:00 Formal methods tools like Alloy are not just for proving correctness properties in distributed systems (or whatever) but can be a useful tool for mere mortals like me! As I mentioned previously</a>, I've been using Alloy to help me avoid data modeling mistakes before they're encoded in hard-to-change places, like database schemas. In this post, I'll show my thought process as I use Alloy to model the schema for a project for my makerspace, including some of the mistakes that Alloy helped me avoid. I joined the Inventor Forge Makerspace</a> recently to get access to some better tools for home recycling</a>. There are some cool tools there (Lasers! CNC machines! Plasma cutters!) that need both special training and have high demand among the folks who use the space. Right now, we use a shared Google calendar to schedule time on different machines. Unfortunately, I don't have a personal Google account, so it's harder to schedule than I'd like! There are a couple of other people in this situation too, so I'm in the process of building a tool to track training and scheduling. We'll start by modeling Member</code> (someone who can join the makerspace) and a Tool</code> (something that we need training and scheduling for.) We'll use Alloy's basic building block of sig</code>s for this. If you haven't been exposed to Alloy before, you can think of sig</code>s as sets that can contain values (like rows in a database table.) So you can think of Member</code> as the set of all members of the makerspace, which can contain values like Member0</code> and Member1</code>. sig Member {} sig Tool {}</code></pre> We can ask Alloy to generate examples for this small model. They won't be super useful at this point—we haven't modeled any interactions between Member</code> and Tool</code>—but they let us get a sense of what Alloy can provide us. For example, here's an instance with two members and two tools: We can ask Alloy for any number of instances, and we'll be looking at these frequently to find the kinds of edge cases we're interested in. Connecting Members and Tools</h2> To get authorization to use a tool at my makerspace, you need to contact a "tool champion" who is responsible for doing safety training. (They're not just "trainers" because they also maintain, fix, and upgrade the tools.) In our model, we'll first connect Member</code> and Tool</code> by adding a new sig Champion</code>, which has a relationship with both. sig Champion { tool: one Tool, member: one Member, }</code></pre> When we're turning this model into a real database, Champion</code> will become a champions</code> table, with foreign keys to tools</code> (tool_id</code>) and members</code> (member_id</code>.) Can you be the champion for more than one tool at once?</h3> Now that we have some relationships, we can poke around in the instances Alloy generates to see if we can find anything that feels off. For example, Alloy shows that we're allowing duplicate champions for the same member and tool: This could cause us problems later. Imagine what could happen if we also had a boolean field named active</code> and a champion resigned. We could end up with one row that says they're active and another that says they're inactive. We'd have to define some conflict semantics in the application, making sure to only change the one we care most about, or all of them, or whatever. But we can get around all that by disallowing these duplicate rows! Fortunately, this is fairly simple to achieve in a database schema by adding a unique index on champions.tool_id</code> and champions.member_id</code>. We can do something similar in our Alloy model by introducing a fact</code> (a logical statement that Alloy will assume is always correct.) fact "a unique index on champions prevents duplicate member+tool combinations" { all t: Tool, m: Member | lone c: Champion | c.tool = t and c.member = m }</code></pre> In English, this is saying "for every combination of Tool</code> and Member</code>, there's at most one Champion</code> that connects them." (The mnemonic for lone</code> is "less than or equal to one". In this case, you can think of it as "if there's one, there's only one.") Just to emphasize, Alloy will take our word that this fact is always true in our system instead of checking it for us! Because of this, I like to include some justification in the description for how this is true in the system I'm modeling. In this case, that means explaining the mechanism for how we're going to make sure this property holds: the unique index. To make sure we've got this completely covered, we'll also assert that we solved the problem. In this case, we'll assert that the case we found earlier can't come up because of our new fact: check NoDuplicateChampions { no disjoint c1, c2: Champion | c1.tool = c2.tool and c1.member = c2.member }</code></pre> We need disjoint</code> here because, otherwise, Alloy could choose the same Champion</code> as both c1</code> and c2</code>, in which case the assertion would be trivially disprovable—two references to the same champion would naturally have the same tool and member. Alloy does not find any counterexamples to this check, so we're safe to continue. Trainings</h2> Now that we have champions members to join the space, tools for them to use, and champions to train them… how do we get access to the tools? Trainings! Someone can have access to a tool that they've trained for, so we can represent it as another sig</code>/table: sig Training { trainer: one Champion, trainee: one Member, }</code></pre> We'll add another validation, as well: that the trainer and trainee can't be the same person. We may be able to do some index trickery to get this to work, but I think it's more likely that we'll have to validate this at the app level. fact "the application code validates that trainer != trainee" { no t: Training | t.trainer.member = t.trainee }</code></pre> Including this justification makes me wonder what the application should do if someone inserts a row that violates this fact into trainings</code> directly, but I'm going to leave that for the moment. Examining More Instances</h3> Let's look at the model instances to see if there's anything worth fixing now. Fortunately, there's not a lot: we've already ruled out quite a few problems with our modeling so far. That means we'll get mostly "normal" instances. For example, here's what it looks like when a champion has trained someone on a tool: A champion can also train two members on the same tool. Also fine: But Alloy also gives us this, in which the same member is trained multiple times: Thinking through our domain, this might be ok. Because the training is for safety, it's not uncommon for someone to request to have a refresher on a tool if it's been a while since they've used it. This is especially common with the woodworking tools, where there are a lot of sharp things near your hands. However, is it OK in our database schema, or would we rather disallow duplicates and (for example) update the date of the training if someone re-trains? I'll have to talk to the person in charge of our current record-keeping system to find out! Reservations</h2> We've already found a couple of interesting questions in our modeling; let's see if we can find more by modeling reservations on machines (finally, we get to the reason I'm doing this in the first place!) sig Reservation { start: one Int, end: one Int, auth: one Training, } { -- a CHECK enforces ordering in the database start < end }</code></pre> We'll use Int</code>s to specify the start and end times of the reservation and specify that they have to be ordered. One weird corner I'm noting about this design is that since I've used Training</code> as the authorization (just to say that you can't get time on a tool you haven't been trained on) you have to access the tool in a roundabout way: given a Reservation</code> named r</code>, it's r.auth.trainer.tool</code>. Could be worse, but that's gonna be a lot of joins in the database. There are a couple of alternatives here: we could reference Tool</code> directly from Reservation</code>, but then we could then construct a Reservation</code> for a tool that a member hasn't been trained on using a Training</code> for another tool. We'd have to enforce that in the app, and I'd prefer not to—the database should take care of data integrity, where possible.</li> we could embed a Tool</code> in Training</code> as part of the foreign key to Champion</code>, then do the same in Reservation</code>. This would be a bunch of coordination work, but might be nicer: it'd mean that we couldn't change Champion.tool</code> without figuring out how to update Training</code> and Reservation</code> at the same time. It'd take more coordination, but it'd give our data model some nice additional robustness.</li> </ol> I'm partial to adding the foreign keys from option 2, but for the sake of completing our model (and this post) let's move on. Preventing Conflicting Reservations</h3> To finish the reservations</code> table, we should rule out overlapping reservations for the same tool. To start with, we can assert (incorrectly) that reservations already can't overlap to get Alloy to give us a counterexample: fun reservedTimes[r: Reservation]: Int { r.start.*next & r.end.^prev } check NoConflictingReservations { all disjoint r1, r2: Reservation | r1.auth.trainer.tool = r2.auth.trainer.tool implies no reservedTimes[r1] & reservedTimes[r2] }</code></pre> The way to read this is "given two different Reservation</code>s with the same tool, there should be no overlap in their scheduling." The implies</code> there works like a conditional statement in an imperative language: in this case, you can read it as "if X then Y" instead of "X implies Y," so "if the tools are the same, the times can't overlap." The next</code> and prev</code> relations on start</code> and end</code> are new as well—they come from Alloy's ordered set support and implicitly exist on ordered sets like Int</code>. They let you do things like calling 1.next</code> to get 2</code>. You can use the recursion operators here (another new concept to us,) where *</code> will repeat a lookup zero or more times, and ^</code> will repeat a lookup one or more times. That means that calling r.start.*next</code> gives us r.start</code> plus all the times after it, and r.end.^prev</code> gives us all times before the end, not including r.end</code> (since one reservation should be able to start exactly when another ends.) If we find the set intersection of those two lookups (the middle of a Venn diagram), we get a range, so saying r.start.*next & r.end.^prev</code> gets us the range of reserved times for r</code>. We can use this (in reservedTimes</code>) to make assertions that there's no overlap between the two reservations. As expected, Alloy finds a violation of this check more-or-less instantly: In this case, the same member is making both reservations. I'm not convinced that that's OK, but we'll leave it alone for now and focus on the fact that r2</code> starts at time 0</code> before r1</code> ends at 4</code>. It looks like we can use a GiST index in PostgreSQL to exclude overlapping ranges</a>. In fact, the last example on that page shows how you can avoid conflicts for a meeting room, which is similar to our example! So, for now, let's introduce a new fact with the constraint that would be introduced by adding such an index: fact "a GiST index prevents overlapping ranges" { all disjoint r1, r2: Reservation | (r1.auth.trainer.tool = r2.auth.trainer.tool and r1.start <= r2.start) implies r1.end <= r2.start }</code></pre> We only have to make one assertion here (that r1</code> ends before r2</code> starts) because Alloy will check every possible combination of Reservation</code> (that's what all</code> does.) That means that every reservation will eventually be checked as both r1</code> and r2</code>. Anyway, Alloy now says that it can't find any counterexamples, which is what we want. Browsing Instances Again</h2> I think we're done now, but I'm just going to browse a couple more examples to make sure. (I do this a lot.) First, we get a completely normal reservation: No conflicts and the structure looks OK to me! Next, we have one reservation starting as soon as the previous one ends: This one's a little funny since someone is using two different training authorizations. This is the kind of example I'll have to bring up when I talk with the person who keeps the records now. Having a diagram like this can help explain the case I'm thinking about to someone else, so I'm glad I found it. I'm also going to ask whether it's OK for one member to have two consecutive reservations—I don't know! I also notice we're not getting any instances where the champion can schedule a reservation, but that's something that should be able to happen. We can get Alloy to show us if it's possible or not by asserting it can't happen and then looking at the counterexample: check ChampionCantSchedule { no c: Champion, r: Reservation | r.auth.trainee = c.member }</code></pre> Alloy comes up with this. A champion can be trained by another champion, at which point they can make reservations. That seems reasonable, but how do you bootstrap it? Maybe Champion</code> actually needs to be a subset of Training</code> (that is, a boolean field on the training</code> table?) Or maybe we should relax the requirement that a champion can't train themselves? This is an edge case that I might not have thought of otherwise! But let's leave this here for now. With only a page or so of Alloy code, we've chased out a bunch of potential bugs in our model and found some interesting jumping-off points for questions while I continue to develop the model into an actual application. This has been pretty typical of my experience using Alloy in this way: in all but the simplest cases, it finds questions that I need to answer with very little effort on my part. Hopefully, at this point, you're interested and want to learn more about Alloy. Here are some good links: The Alloy project site</a> (where you can get the software and some older learning material)</li> Hillel Wayne's introduction to Alloy 6</a>, and his reference docs mostly covering Alloy 5</a></li> A draft book about designing software in Alloy 6</a></li> </ul> In general, most learning material does not cover Alloy 6, which came out about a year ago as of this writing. The links above are the only things I've found that do, although there may be more by the time you're reading this! Thanks to Hazem at the Recurse Center</a> and Tessa Kelly at NoRedInk</a> for reviewing drafts of this post! Digital Gardening in Obsidian 2022-12-26T00:00:00+00:00 I have been using Obsidian</a> for all my notes for something like 6 months now. That's really everything, from temporary measurements to journaling to serious documentation work. I've tried Obsidian a couple of times before and bounced off of it, so I decided to try to do things differently this time. Instead of starting out by figuring out a system for where everything lives in advance, I've just thrown almost everything in the default folder and sorted it out with linking and search. Some structure has emerged over time, which I'm fairly happy with, but it's more to do with practices than things that would make it easier for someone else to read my vault. These practices basically boil down to "digital gardening"—they allow me to keep things neat and tidy and observe growth without being overwhelmed by it. Here's some of what I do! Rollup Journaling</h2> I've been keeping a log in my daily notes. These logs can contain answers to journal prompts like "what have I been avoiding?" but most of the time it's just a roughly accurate timestamped log of what I've been up to in a given day. Every Monday I summarize these daily notes into a weekly digest. These have more of a journaling flavor to them, since one of the goals is to help me discover patterns, but the template is still pretty basic: a header for highlights, one for lowlights, and one for things I want to change in the next week. If the particular Monday I'm doing this on also starts a new month, I'll also do a summary of the weeks in the previous month following the same basic pattern. Same for the last 3 months of monthly notes, if it happens to be the first Monday of April, July, October, or January. I haven't been doing this long enough to do a yearly rollup yet, but I may do a 2022 yearly rollup note in January. I've been really happy with this! The big benefit so far I've seen is being able to see back in time with more clarity. There have been times this year where I've felt stuck—like I'm not making progress and haven't for a long time—but looking back in my notes I can see that I actually have done some pretty major things in the past year. This has had a nice side benefit that my end-of-year review at work was really easy to write: I just looked back at my quarterly reviews, as well as a few monthly ones, and pulled out some good stuff. It doesn't completely get rid of recency bias, but it does mean that I'm not only focusing on stuff that happened in the last month or two. I don't know if "rollup journaling" is the best name for this, but it's one that's stuck with me for some reason. I got the idea for doing this a couple years (and tries at Obsidian) ago based on Britney Braxton's talk at JuneteenthConf 2020, Journaling as a Dev</a>. Spaced Repetition</h2> I've been using the community spaced repetition plugin</a> to periodically review notes. It's a bit of a hack—this plugin is clearly not meant to be used like this—but it works. I add a #review</code> tag to every note that I want to periodically come back to and improve or tweak (this is actually most of my notes.) I then ask the spaced repetition plugin if there's anything for me to look at. If I make any changes, I'll mark the review as "hard" so the plugin will show it to me earlier. If I don't think it needs any changes, I'll mark it as "easy" to send it further into the future. I'm not terribly disciplined about doing this every day, but it has a nice benefit of improving notes that need to improve while sending ones that don't into the far future for me to get reminded of periodically. I think I originally got the idea for this from an Andy Matuschak note</a>. Pages that Don't Exist, but Should</h2> When writing notes, I link freely, even if the page I'm linking to doesn't exist yet. Then, every couple of days, I'll have a look at my "pages that don't exist but should" note, which has a dataview</a> query that shows me all those nonexistent pages, ranked by how many incoming links they would have if they did exist. For example, if I look at that page right now, I can see that I haven't made a note for Phoenix Liveview—I've been looking into it for a side project and have made notes on a couple of conference talks where I've linked to it. There's also a missing note for some features of the product I work on, since I've mentioned them in daily notes. I'm fairly happy with this approach. It's a bit easier than looking for greyed-out nodes in the graph view, and over time it's lead to some interesting practices. For example, I started linking the names of my coworkers when I paired or collaborated with them on something, which eventually lead to keeping notes on what other teams around the company are responsible for. I'm not sure I would have started that (quite useful) practice if I hadn't been paying attention to what "desire paths</a>" I was creating in my vault. In case it's useful to someone else, here's the dataview script I embed for this. There's probably a better way to get this information, but it's been working well as-is. This version filters out daily notes and doesn't consider media files, but you may want to remove those lines if being prompted to create daily notes far in the past or future sounds useful to you: let phantoms = dv .pages() .flatMap((p) => p.file.outlinks.map((link) => [link, p.file])) .groupBy((p) => p[0]) .filter((g) => g.rows.length > 1) .filter( (g) => !g.key.path.endsWith(".png") && !g.key.path.endsWith(".jpeg") && !g.key.path.endsWith(".jpg") && !g.key.path.endsWith(".mov") && !g.key.path.match(/^\d\d\d\d-Q\d$/) && !g.key.path.match(/^\d\d\d\d-\d\d-\d\d$/) && !g.key.path.match(/^\d\d\d\d-\d\d$/), ) .map((g) => { return { key: g.key, rows: g.rows.map((r) => r[1]) }; }) .filter((g) => dv.page(g.key) === undefined) .sort((g) => g.rows.length, "desc"); dv.table( ["Page to Create", "Linked From"], phantoms.map((g) => [g.key, g.rows.map((p) => p.link)]), );</code></pre>Other Dataview Shenanigans</h2> Other than the stuff mentioned above, I have a handful of useful dataview pages. For example, I'm looking into replacing an older car we own right now. The page for that project has a dataview query summing up research I've done into cars I might want to buy to replace it. I also have a view of everything tagged #possible-blog-post</code> sorted by when I think I might want to publish it. None of these are huge, but they do a good job of gathering information from around my vault that I probably want to see all at once. Anyway</h2> That's how I organize my Obsidian vault. I have a few more plugins I use (e.g. a calendar widget</a>, improved daily notes</a>, templater</a>, and a map view</a> for when I was looking at stuff to do on vacation) but that's the core of it. I hope you can take some of this and get use out of it for yourself! Scope in the TH element 2022-12-19T00:00:00+00:00 The <th></code> (table head) element in HTML accepts a scope</code> attribute. According to MDN</a>, it can be one of these: row</code> and col</code>, which means that the header relates to all the cells of the group (row or column, respectively) in which it's found</li> rowgroup</code> and colgroup</code>, which specifies a group of rows or columns. I'm not sure how to make a group like this, though!</li> </ul> <th scope="row"></code> can be really useful for accessibility. Say you have a table like this: Name</th> Text-to-speech enabled</th> Edit student</th></tr></thead> Valencia Flowers</td> yes</td> <button></code></td></tr> Romeo Harrison</td> no</td> <button></code></td></tr> </tbody></table> If the "name" cells are <th scope="row"></code>, screen readers can contextualize "yes" or "no" and the edit button when navigating between rows in a non-name column. For example, one might say something like "text-to-speech enabled, Valencia Flowers, row, yes." Projects 2022-12-14T00:00:00+00:00 Just a little note: inspired by /now pages</a>, I've put a bunch of current and past projects up on the home page at bytes.zone</a>, listed by date at bytes.zone/projects</a>. I'll update them occasionally, adding blog posts for RSS readers as relevant. If you're interested in what I've been up to lately, there it is! Alloy 2022-12-12T00:00:00+00:00 Alloy</a> is a tool for modeling software. It's best at representing things like databases or data structures, but in my experience it does fine with things like UI states or processes as well. I've found Alloy most helpful in exploring the relationship between different parts of a system. Unlike other tools in the formal methods space, Alloy has a visualizer. For me, this is a huge advantage: not only does it make it easier for me to think about my models, but it also makes communicating about specs with people who are not familiar with Alloy much nicer. Having something you can talk about or mark up enables better communication, as opposed to having to read states and relationships out of a table. To illustrate that, here's an oversimplified version of email. I've told Alloy that there are many accounts on a server, and that accounts can send emails back and forth. This should even work across servers, as long as the servers can deliver to each other. The code looks like this: sig Email { from: one Account, to: one Account, } sig Account { server: one Server, } sig Server { deliversTo: set Server, }</code></pre> To start off, Alloy can generate a "metamodel" (that is, the relationships between things it knows about.) It basically looks like you might expect given the code: With no code changes, it can also generate as many instances of the model as you care to ask it for. Here's one, where an email is being sent from one server to another: I've found it useful to set up relationships like this, then ask the tool to generate examples. If something looks off or out of the ordinary, I can go back and have a think (maybe with other people) about whether or not that thing should be allowed. For example, I notice that one email server above can't see the other, and the other can't deliver to its own accounts. Is that reasonable? Depending on the situation you're modeling, it may or may not be! You can use Alloy to make assertions about your models as well. In the example above, for example, you could assert that there couldn't be any undeliverable email (that is, any email going between servers without a deliversTo</code> relationship.) If you've allowed that possibility somehow, Alloy will tell you why and generate a counterexample. Here's how we'd do that: check NoUndeliverableEmail { all e: Email | e.to.server in e.from.server.*deliversTo }</code></pre> In English, this means "for an email to be deliverable, the server in from</code> must be able to reach the server in to</code> through the deliversTo</code> relation, 0 or more times. And, by the way, every Email is deliverable." Of course, we haven't guaranteed that assertion in any way, so Alloy comes up with a counterexample: In this, Server1</code> can deliver everywhere, Server0</code> can deliver to Server2</code>, and Server2</code> can't deliver anywhere. Unfortunately for everyone involved, Account0</code> on Server2</code> wants to send an email to their buddy on Server1</code>. Guess you're outta luck, pal! We can try to fix this, of course, by saying that all of these deliversTo</code> relations are symmetric. We'd just establish some fact that Alloy will take as given: fact "all delivery relationships are symmetric" { deliversTo = ~deliversTo }</code></pre> This works because deliversTo</code> is actually a set of tuples; the server.deliversTo</code> is doing a lookup into it! The ~</code> operator reverses all the tuples, so we're asserting that the forward version is equal to the backwards one—symmetric! But in the face of this stunning victory, Alloy replies "ah, yes, but what if the servers are on completely separate networks?" We can keep going down this rabbit hole of generating examples and making assertions until we're satisfied that the system has few enough problems to implement! Why is this Useful?</h2> I've used Alloy in a lot of situations now where I would previously have just been guessing at edge cases in the model. It's been really useful to show me exactly what problems exist in my model and helping me communicate with other people about why we should care about them. Alloy helps simplify models, as well! For example, at work we were building a feature with inline commenting (like Google Docs.) We started by modeling Google Docs' comments and it turned out that each comment had something like 32 unique states. We realized this was going to be way too much for our use case, and simplified it down to 4. We probably could have had this realization without using Alloy, but it made the conversation much simpler—the designer and I sat down and looked at the visualizations, then kept saying "but for us, states X and Y would be the same…" until we arrived at a simpler model. It was really nice, and only took about an hour and a half of modeling to come up with a solution that satisfied all our requirements! Yeah, but…</h2> So all that's super exciting, and I love using Alloy, but it'd be silly to pretend that it's perfect. Here's some examples: It's very easy to give Alloy contradicting instructions that rule out some case you care about. Sometimes this even means that Alloy can't come up with any examples at all! It can be really difficult to debug this when you're getting started. (But if you change the solver to "MiniSat with Unsat Core" you can at least get a hint at what might be conflicting.)</li> Alloy only checks a finite size of the models you generate (4 of everything by default.) This means that it won't find bugs that require there to be a huge number of something. However, almost everything I've done in Alloy gives useful feedback with the default limits—it checks all states comprehensively within the given size—and it's easy (bordering on trivial) to raise the limit if you suspect there would be a bug if you had more instances. Just know going in that you're not going to get proof for an infinitely sized model. This property does mean that Alloy is very, very fast, though!</li> As a beginner, it's hard to know how to learn. The usual learning path is to buy and work through Software Abstractions by Alloy's author, Daniel Jackson</a>. However, Alloy has evolved over time and now has important features—like temporal logic—that are not covered in the book. To make matters worse, documentation outside of the normal learning path often means reading the spec</a>. However, once you know a bit about Alloy, and even while you're learning, Hillel Wayne's Alloy reference</a> is great.</li> </ul> Another one of Alloy's weakness used to be time—that is, it was hard to model things that could change—but the release of Alloy 6</a> added temporal logic and made this situation a lot more tenable. I still haven't completely come to grips with the new features, but I see concepts in the spec that I first learned about in TLA+—although I appreciate that Alloy uses English-language keywords instead of symbol operators like []</code> (always</code> in Alloy) or <></code> (eventually</code> in Alloy.) All told, I recommend learning Alloy! I've found it tremendously helpful and I think it's too bad that it's not better-known by industry programmers. tree-grepper 2021-08-31T00:00:00+00:00 A while ago I wanted to build an import graph from all the frontend code at NoRedInk to build some developer tools. The code I wrote ended up working fine, but it was also pretty messy… tons of big regular expressions to make sure I got all the corner cases and whitespace allowed in the syntax. I thought there probably was a better way, especially since I had just learned about tree-sitter</a>. I also wanted to learn some Rust, so… well, it sounded like a fun little project! Fast forward about a year and I just released tree-grepper</code> 2.0.0</a>! It lets you search very quickly across large projects full of diverse filetypes, using tree-sitter grammars and search queries. The big benefit here is that it's easy to expand: tree-sitter is getting really popular, what with language servers and Neovim extensions</a> and everything, so it's pretty likely that someone has already built a parser that we can just add! Currently tree-grepper lets you search Elm, Haskell, JavaScript, Ruby, Rust, and TypeScript, but it's pretty easy to add more, and the README has a step-by-step guide</a>. Tree-grepper is focused only on search: it doesn't do linting (like semgrep</a>) or AST-based refactoring (like comby</a>.) You can get structured match data out of it, but any further processing is another tool's responsibility. This let me cut down on scope significantly, and optimize for searching alone. Hooray for the Unix philosophy! Extracting An Import Graph</h2> Let me show off what it can do a little bit by implementing the parsing task I set out to do originally! Tree-sitter implements an s-expression query API</a> which tree-grepper exposes directly. We'll use that to query for all the imports in NoRedInk/noredink-ui</a>: $ tree-grepper --query elm '(import_clause)' | head -n 5 ./styleguide-app/Category.elm:18:1:query:import Sort exposing (Sorter) ./styleguide-app/Main.elm:3:1:query:import Accessibility.Styled as Html exposing (Html, img, text) ./styleguide-app/Main.elm:4:1:query:import Browser exposing (Document, UrlRequest(..)) ./styleguide-app/Main.elm:5:1:query:import Browser.Dom ./styleguide-app/Main.elm:6:1:query:import Browser.Navigation exposing (Key)</code></pre> What we're doing here is asking tree-grepper</code> to give us all the import_clause</code> nodes it finds in elm</code> files (I'll tell you how to find out how this is called an import_clause</code> later.) Each match gets printed as a line of text along with the exact position in the source file. We don't want the entire match, though, just the module name (the part after import</code> but before as</code> or exposing</code>). Let's try and get just the module name: $ tree-grepper --query elm '(import_clause (upper_case_qid)@import)' | head -n 5 ./styleguide-app/Category.elm:18:7:import:Sort ./styleguide-app/Main.elm:3:8:import:Accessibility.Styled ./styleguide-app/Main.elm:4:8:import:Browser ./styleguide-app/Main.elm:5:8:import:Browser.Dom ./styleguide-app/Main.elm:6:8:import:Browser.Navigation</code></pre> So now we're asking for upper_case_qid</code> nodes inside import_clause</code>s. If we tag the nodes we care about (by naming them after an @</code>), tree-grepper</code> will only output the parts we tagged. So far, so good! But how about module Foo exposing (..)</code> at the tops of files, too? Easy: just add another --query</code>! $ tree-grepper \ --query elm '(module_declaration (upper_case_qid)@module)' \ --query elm '(import_clause (upper_case_qid)@import)' \ | head -n 5 ./styleguide-app/Category.elm:1:8:module:Category ./styleguide-app/Category.elm:19:8:import:Sort ./styleguide-app/Main.elm:1:8:module:Main ./styleguide-app/Main.elm:3:8:import:Accessibility.Styled ./styleguide-app/Main.elm:4:8:import:Browser</code></pre> This follows the same pattern: we want a named child of another named node in the file, and tree-grepper</code> manages walking the tree to give it to us. JavaScript Imports</h2> What about JavaScript import</code> clauses? We can mix different languages as easily as we mix different queries, but let's do one at a time for simplicity's sake: $ tree-grepper --query javascript '(import_statement (string (string_fragment)@import) .)' ./lib/TextArea/V4.js:1:31:import:../CustomElement</code></pre> This is the end of a import * as Foo from "Bar"</code> clause. The .</code> at the end is an anchor: it tells tree-sitter that we care that the thing we're matching is the last child of its parent node. You can also put a .</code> right after the node name to match on the first node only, or on both sides to enforce that you are matching the only node. We're also matching a string fragment out of the string we require. This lets us remove any quoting characters so we get ../CustomElement</code> out instead of '../CustomElement'</code>. We could also put anchors here to make sure we're not getting any interpolated strings, but I haven't needed to do that yet so we're not here either. JavaScript Requires</h2> Finally, let's get require</code> calls: $ tree-grepper --query javascript '(call_expression (identifier)@_fn (arguments . (string (string_fragment)@require) .) (#eq? @_fn require))' | head -n 5 ./styleguide-app/manifest.js:1:8:require:../lib/index.js ./script/percy-tests.js:1:29:require:@percy/script ./script/axe-puppeteer.js:4:27:require:puppeteer ./script/axe-puppeteer.js:5:25:require:axe-core ./script/axe-puppeteer.js:6:37:require:url</code></pre> This is quite the query! Let's break it up over multiple lines to talk about it: (call_expression (identifier)@_fn (arguments . (string (string_fragment)@require) .) (#eq? @_fn require))</code></pre> (call_expression (identifier) (arguments))</code> is a function or method call, in our case require("url")</code>. However, without anchors or specifying the arguments we're not saying anything about the contents, just that it's a call. In this case, we care that the arguments are only a single string, so we specify that as (arguments . (string (string_fragment)@require) .)</code>. Finally, we don't want just any function with a single string argument; we only want require</code> statements. Tree-sitter exposes a couple of matcher functions (#eq?</code> for string equality and #match?</code> for regular expressions) to select the things we want here. To use them, we name the node we care about (here @_fn</code> with a leading underscore to tell tree-grepper to drop it from the output,) then give the match to #eq?</code> along with a bare word (require</code>) to check for equality. Now we only match nodes that look like require('axe-core')</code>, but only pull out the inner axe-core</code> string that we care about! Putting It All Together</h2> Of course, we can do this all at once, smashing all our queries together in one giant tree-grepper</code> invocation. $ tree-grepper \ --query elm '(import_clause (upper_case_qid)@import)' \ --query elm '(module_declaration (upper_case_qid)@module)' \ --query javascript '(import_statement (string (string_fragment)@import) .)' \ --query javascript '(call_expression (identifier)@_fn (arguments . (string (string_fragment)@require) .) (#eq? @_fn require))' \ | head -n 5 ./styleguide-app/Category.elm:1:8:module:Category ./styleguide-app/Category.elm:19:8:import:Sort ./styleguide-app/Main.elm:1:8:module:Main ./styleguide-app/Main.elm:3:8:import:Accessibility.Styled ./styleguide-app/Main.elm:4:8:import:Browser</code></pre> That works for any amount of queries you'd care to throw at it! In fact, it's more efficient to run this way since it only has to walk the filesystem and parse files once! You can also get more information by specifying the format: the json</code> and pretty-json</code> formats have match end locations as well as starts, and they include the node names returned. You can use that to get an overview of all the node names in a grammar: $ echo 'console.log("Hello, World!")' > hello.js $ tree-grepper --query javascript '(_)' --format pretty-json hello.js [ { "file": "hello.js", "file_type": "javascript", "matches": [ { "kind": "program", "name": "query", "text": "console.log(\"Hello, World!\")\n", "start": { "row": 1, "column": 1 }, "end": { "row": 2, "column": 1 } }, ... snip ... ] } ]</code></pre> I've actually had to shorten that significantly because it's so long! If you'd like to see some full output, check the all_{language}.snap</code> snapshot test files in the tree-grepper repo</a>. I really enjoyed writing this tool and learning more about tree-sitter, and I hope you find it useful! You can also get the source and instructions for contributing at github.com/BrianHicks/tree-grepper</a>. Tree-grepper is packaged using Nix, so if you have that you can just install it like nix-env -if https://github.com/BrianHicks/tree-grepper/archive/refs/heads/main.tar.gz</code>. If you have Nix flakes enabled, you can also run nix shell github:BrianHicks/tree-grepper</code> to get a shell with tree-grepper</code> already available. The Four Rules of Simple Design: In Conclusion 2021-08-25T00:00:00+00:00 OK, it's been a couple of weeks since I published Rule 4: Code for Now</a>, and I've had some interesting feedback! The biggest thing first: how do you apply these? Surely you don't just go down the list one at a time, right? Well, no, I view it more like like a state machine: you enter at Rule 1: Simplify When the Program Works</a> and move from there back and forth between Rule 2: Clarify Your Intent</a> and Rule 3: Centralize Behavior</a>. Once you think you're more-or-less done, you check your work with Rule 4: Code for Now</a>. When you don't have anywhere else to go—that is, you don't see an opportunity to apply any of the rules—you call it good and move on to the next thing you need to do that day! That's not the only way to apply the rules, of course, but it's one that I've found works pretty well for me. J.B. Rainsberger applies the original rules something like this as well: see his post The Four Elements of Simple Design</a> for more, in particular his axioms of modular design</a>. Of course, it's not the only way by a long shot! One of my colleagues at NoRedInk</a> pointed out that he'd like to rearrange these rules: Code for now</li> Simplify when the program works</li> Clarify intent</li> Centralize behavior</li> </ol> That could certainly work too! It's up to you how you apply these, if you want to apply them at all. There's also some tension between the rules. For example, in Rule 3: Centralize Behavior</a> I talked a lot about making a Point2d</code> class. But… is that structure really necessary? Couldn't it violate Rule 4: Code for Now</a>? How do you decide which to apply? In the end, you've either got to choose randomly or find a tiebreaker: for example, you could use Sandi Metz' Rules For Developers</a>, which emphasize very small classes. In that context, building a Point2d</code> may or may not be OK depending on how close your class is to 100 lines. Personally, I've usually gone the other way over the time: I've randomly chosen to prioritize rules randomly based on the situation at hand. However, since part of the appeal of this whole exercise is that they allow us to reduce guessing about the future, I'm going to continue to explore situations where there seem to be holes or conflicts! And, to that end: if you read through the original sources (or mine) and are inspired to make your own remix of the four rules, I'd love to read it! Please send me your takes! Let's build on each other's work to discover how to make better software. Thanks for reading! Rule 4: Code for Now 2021-07-13T00:00:00+00:00 Last week, we talked about centralizing behavior</a> as part of the four rules of simple design series</a>. This week, the final rule: code for now. This is about focusing on building the smallest thing that can possibly work. Another way to put it: are you writing code because you're guessing that you'll need something later? Stop it! This shows up most commonly when building larger pieces of program structure. In my experience: In object-oriented languages like Python and Ruby, it can look like deep class hierarchies and composition by mixins or inheritance.</li> In Haskell, it can look like lots of custom typeclasses or adding high-complexity features like lenses or monad transformer stacks.</li> In Elm, it can look like factoring out every little piece of state into a separate model/update/view triple, or splitting things into smaller and smaller modules.</li> </ul> In summary: don't add structure to code before you need it. (And if we're being honest, this rule is not far from "You Aren't Gonna Need It" in different clothes.) What Happens If You Ignore This?</h2> Have you ever seen FizzBuzzEnterpriseEdition</a>? That. But Where Do I Put Code?</h2> Removing unnecessary structure can cause some confusion when you want to add some functionality but there's no obvious place for it. In that case: add structure, because you need it now. The point of the rule is to avoid guessing about what you'll need in the future, not to avoid all kinds of structure. This rule is also the catch-all at the end of the four rules. You shouldn't need to invoke it often. If you don't know where to put code, make sure you're centralizing behavior</a> or clarifying your intent</a>. Rule 3: Centralize Behavior 2021-07-06T00:00:00+00:00 Last week, we talked about clarifying intent</a> as part of the four rules of simple design series</a>. This week, we're on to rule 3: centralize behavior. Imagine you're working on project for a school, trying to make sure that the site addresses users appropriately: students should be addressed casually ("Hello, Adrian!") while teachers should be addressed more formally ("Your teacher, Dr. Beans".) Imagine that this has been done, but a little bit at a time and in different ways, so that these forms of address are implemented in different and inconsistent ways across the site. That makes what seems like a simple change much more difficult! In order to make this change tractable, you might create a function or method that takes a User</code> or Role</code> domain concept and returns the right string form, then change each call site across the whole project to use that. A big project, sure, but a pretty reasonable approach! This demonstrates our third rule: centralize behavior. This is similar in spirit to the ideas behind Don't Repeat Yourself. In that context, "repetition" isn't talking about the written form of code so much as authority over defining the behaviors and ideas in the system. A similar concept comes up in Once and Only Once: define each concept in your system in exactly one place, and don't split up that authority! What Happens If You Ignore This</h2> Like in the scenario above, ignoring this rule results in hard-to-change code which needs lots of extra effort to modify successfully. Did you get all the little pieces? Better hope so! Problem is, it's often hard to see this happening—the name change I described above feels a bit contrived because it's easy to see, but template systems in web frameworks will push you away from specifying one-off logic and towards reusable functions or methods. I've seen this come up more often in serialization and validation logic: places where it's very tempting to just dig into some internal data structure instead of defining behavior where the code lives. One question to ask is "who should be responsible for this behavior?" In the long term, it's nice to answer this by writing behavior attractors. Behavior Attractors</h2> Behavior attractors (a term coined by Corey Haines in his book Understanding the 4 Rules of Simple Design</a>) are places in your code where it becomes natural to put related code, like water flowing downhill. This gives you a natural way to centralize behavior to keep your system simple and easy to understand. Let's look at an example! Understanding the 4 rules of Simple Design uses Conway's Game of Life throughout, so let's just steal the example Haines uses to explain this: how do we figure out the neighbors of a cell? When you're implementing the Game of Life, you might end up writing Cell.neighbors</code> in the course of answering "is this cell alive or dead in the next cycle?" However, this is not necessarily the best way to implement this, because the concept of a neighbor really depends on the topology of the grid. "Neighbor" means something different for 2- versus 3-dimensional grids, not to mention things like hyperbolic geometry. That suggests that "are these cells neighbors" should be defined by the coordinate system, but you might not come to this realization if you're just passing around 2-dimensional coordinates like (x, y)</code>. The behavior attractor solution is to move those coordinates to a central definition (say, 2DCoord</code>) and implement the neighbors</code> function/method there instead. That means you might end up with 2DCoord(1, 1).neighbors</code> instead of Cell.at(1, 1).neighbors</code>. You get a clearer separation of concerns: a 2DCoord</code> can clearly define neighbors in a way that a Cell</code> shouldn't be responsible for. The behavior attractor made this possible, and will make your life easier the next time you need to ask a question about the coordinate system too! If you find that you don't have a good place to put some code, especially if you find yourself just putting it close to the place you need it, consider writing a behavior attractor! Rule 2: Clarify Your Intent 2021-06-29T00:00:00+00:00 Last week, we talked about simplifying when the program works</a> as part of the four rules of simple design series</a>. This week, rule two: clarify your intent. The original four rules prioritize clarity of communication1</a>. For example, Beck's formulation says "states every intention important to the programmer." Fowler tightens that up to "reveals intent", and Rainsberger refines it further to "improve names." But what intent are we trying to communicate clearly? Here are a few things you might want to consider: What's this thing for?</li> How are these two things related?</li> Why is this code doing things this way instead of that way?</li> Please don't do this thing if you also do that thing.</li> Which part of this is core to the program, and which is just supporting structure?</li> Where does the stuff you send here end up? Or, where is this thing getting stuff from?</li> </ul> You can communicate these things in a lot of ways, but I want to talk about three: names, comments, and constructors. Names</h2> Say you have a method called process_data</code>2</a>. Both of those words are really vague! What if we made them better? What's process</code>? What's data</code>? Maybe you look at the method body and find out it downloads a CSV of accounts and selects all the ones the sales team has identified as likely to buy a premium license. So in this case, process</code> would be incorrect—it's get</code> or download</code>. And it's not just any old data</code>, it's clients</code>, specially ones on the threshold of upgrading. So, while there are more ways you could improve this (e.g. separating the downloading from the data processing), renaming it to get_threshold_clients</code> already makes it way easier to understand your code: imagine seeing process_data</code> vs get_threshold_clients</code> at a call site. Which would you prefer when reading later? There's a lot to think about when choosing a good name, so instead of saying more, I'm just going to link some things I've found helpful: Naming Things in Elm by Ally McKnight</a>, Picking better names for variables, functions, and projects by Tom MacWright</a>, What's in a Name? Anti-Patterns to Hard Problem by Katrina Owen</a>, and "Naming Things" is a Poor Name for Naming Things by Hillel Wayne</a>. Comments</h2> Another thing you could do to clarify your intent is to explicitly write it down in a comment. There's the constant debate on "what" comments vs "why" comments, of course—here's Hillel Wayne again on why you need both</a>—but in either form, the best comments give the reader insight into what was going on and what you needed when you wrote your code. I like to write long documentation comments framed like "Hello, future us! I hope you're having great day. Here's what's up." I've found those to be helpful, both when my coworkers review the code and when we revisit it months later while trying to do something else. Having little hints of intention scattered throughout the code helps remember things we need to keep in mind. Plus, like trying to find a good name, documenting why something exists can often help us realize some other refactor that could help us do what we're doing better! Constructors</h2> In addition to names and comments, you can often clarify your intent in by making certain circumstances impossible by construction. As a trivial example, you might know that a value can sometimes be null, so you represent it as an optional type. Or, if you know you'll always have at least one item, you can use a non-empty list to hold the data. I've found this to be a lot easier in ML-family languages like Haskell or Elm, but you can do it in object-oriented languages as well. There are tons of talks and examples of this online if you search for things like "make illegal states unrepresentable". I like Richard Feldman's 2016 talk Make Impossible States Impossible</a>. What Happens If You Ignore This?</h2> If you don't clarify your intent when coding, you end up in situations where you have to re-establish context. In names, that might mean having to jump to the definition to figure out what's actually happening. In the case of comments, that might mean having to spend a long time looking through the code in order to find the shape of the program's core algorithm. In constructors, it means the same but trying to figure out if something should be possible or not. If you don't clarify intent, you'll also miss out on great refactoring opportunities: in the get_threshold_clients</code> example above, we might see that we want to separate getting the raw data and applying our precise filtering rules. That makes the code more modular (and more testable!) but we might not have seen it if we didn't stop to clarify intent. Conclusion</h2> In summary, remember that code is read many more times than it's written. Keeping that in mind can go a long way to making things better for your coworkers (or just you, 6 months from now.) There are lot of ways to do this—way more than we can go over here—but you can get a long way by focusing on names, comments, and constructors. ^{1 In fact, I think the entirety of the four rules is secretly about communication: write tests to communicate, remove duplication to make sure you're not saying the same thing twice, remove any unnecessary distractions, and here where we're being pretty explicit about intention! </div>}^2Example inspired by Corey Haines' Understanding the Four Rules of Simple Design</a> </div> Rule 1: Simplify When the Program Works 2021-06-22T00:00:00+00:00 Last week, I introduced My Take on the Four Rules of Simple Design</a>. To sum up, I've found Kent Beck's four rules of simple design really interesting to think about, and I wanted to come up with my own formulation! This week, we meet the first rule: simplify when the program works. It's similar to the original first rule ("runs all the tests"), but with a broader reach. So in addition to passing tests we might include things like: Is it possible to get the program into an illegal or "impossible" state by using it?</li> Do the user-facing parts of the program look the way that the designer intended?</li> Is the program accessible to all kinds of users? For example, can the program be operated using only a keyboard? Only a mouse? A screen reader? (Broadly, Is the program Perceivable, Operable, Understandable, and Robust</a>?</li> Has QA tested this? What did they think?</li> </ul> Some of those might not apply to your project, and some of them have really long iteration times, but that's actually part of the point: if you find that your program is broken in some way, it probably makes sense to fix it before trying to make it faster or organize the code better. Unit tests are just one way of looking at this, although certainly an attractive one since they can be automated. What Happens If You Ignore This?</h2> What happens if you start simplifying the code before the program works? Well, when I've ignored this advice, I notice I tend to start ripping out useful code or changing things in ways that don't make sense. I end up totally muddling my code with unrelated changes, and despite all the effort the program still doesn't work right. Eventually, I've maybe made something kinda-sorta like the change I tried to make, but it's way harder for my team to review. In my experience, I need to be in one of two modes: making changes to make the program work, or making changes to make the code simpler. "Simplify when the program works" has been helpful for me in recognizing when I need to switch! Avoiding Perfectionism</h2> But what happens if the vagueness in this rule works against you? For example, you can pile so many things on to "works" that it becomes "perfect." Setting an unattainable standard means never shipping, and we'd like to at least ship some day, so let's try to avoid perfectionism, right? One way around this is to change your relationship to your standards, at least temporarily. Something doesn't have to be perfect to be working. For example, maybe messy code is fine for now, or maybe it's OK for some tests to fail—just for a bit—while you work on accessibility in the UI. Said differently, we can take on tech debt to avoid perfectionism. We usually want to avoid that, and you might have some "yuck!" reaction to reading it, but there are places where it makes sense. I think Yvonne Lam</a> really nailed it with her concept of tech debt as housework</a>, specifically: The thing with tech debt is that in order to have a useful discussion, you need to be able to talk about *who* is affected and how -- who is going to be woken up, be pulled off other work, be unable to perform some critical function, etc. You can't hide risk and have that talk. — Yvonne Lam on Twitter</a> </blockquote> Let's apply that: who is going to be affected by changing your standards, and how? This will be different in different settings, so asking these questions with your team makes a lot of sense. As an example, I care a lot about accessibility. I don't want to ship software that prevents people from using assistive technology like screen readers. In my company's case, our users are kids learning to write better. I really care about not messing that up! My team shares that value, so when we found out in a recent project that we'd have to choose between having more complicated view code versus shipping a program that was inaccessible to screen readers, we chose the complex code. We know we're going to have to clean up the code sometime, but for now we're OK with maintaining some slightly complicated code to make sure we're accommodating everyone's needs. Because we'd already had the conversation about what values we wanted our code to embody, we were able to make a choice that lined up with those values. Even though the software wasn't perfect, I think we made the right tradeoff. Point is: talk with your team about this! Every project, team, and person has a different set of values, so trying to nail down what "works" means can highlight differences in how we think about the things we're building. Make The Change Easy</h2> Another way to apply this rule is to "make the change easy, then make the easy change." (Coincidentally, another Kent Beck-ism</a>.) In other words, when you go to your code to make a change, consider refactoring towards simpler code first instead of waiting until after you're done. It might seem like this is incompatible with the rule—after all, if you're planning to make a change you already know the program doesn't work according to the specification—but we often have to do some cleanup in order to make the change possible! For the purposes of this rule, "works" can also mean "is amenable to change!" Conclusion</h2> So, to sum up: "Simplify when the program works" depends a lot on what you mean by "works." (It also depends on what you mean by "simplify", but that's what the other rules are for—coming soon!)</li> This rule can be helpful as a signal to switch from writing-code-to-make-the-program-work mode to clarifying-code-for-future-us mode.</li> Set a high bar, but no so high that you can't ever ship.</li> </ul> See you next week for rule 2: clarify your intent. My Take on the Four Rules of Simple Design 2021-06-15T00:00:00+00:00 I've been thinking about how to write better tests recently, and I happened on Kent Beck's four rules for simple design. In a nutshell, these rules were meant to give objective criteria for the question "is this code simple?" The original formulation goes like this: a piece of code can be said to be simple if it... Runs all the tests.</li> Has no duplicated logic. Be wary of hidden duplication like parallel class hierarchies</li> States every intention important to the programmer.</li> Has the fewest possible classes and methods.</li> </ol> — Kent Beck, Extreme Programming Explained: Embrace Change</a> </blockquote> In practice, I really like that while the intention may have been to make an objective metric for simplicity, people tend to treat these rules as imperfect but useful. That is, they give us a way to judge if a piece of code is better or worse than another one. This turns out to be a helpful way of judging internal quality of a piece of software—and improving that is why we write tests and refactor code in the first place! I think this usefulness has made these rules surprisingly uncontroversial (at least to me, based on how much I know we programmers like bikeshedding!) The most controversial topic, in fact, seems to be which order to apply rules 2 and 3 in. In reading about this, I collected a bunch of interpretations of the four rules from various authors: Rule</th> Kent Beck</a></th> Martin Fowler</a></th> J.B. Rainsberger</a></th> Corey Haines</a></th></tr></thead> 1.</td> Runs all the tests.</td> Passes the tests</td> Passes its tests</td> Tests Pass</td></tr> 2.</td> Has no duplicated logic. […]</td> No duplication</td> Minimizes Duplication</td> No Duplication (DRY)</td></tr> 3.</td> States every intention important to the programmer.</td> Reveals intention</td> Maximizes clarity</td> Expresses Intent</td></tr> 4.</td> Has the fewest possible classes and methods.</td> Fewest elements</td> Has fewer elements</td> Small</td></tr> </tbody></table> I have enjoyed applying these rules to my own code, but in the spirit of "what I cannot create, I do not understand," I thought it'd be fun to try and make my own formulation. In doing this, I'd like to avoid simply paraphrasing the four rules—I want to be able to explain what makes code simple to someone who has never heard of this concept. That means being able to make a reasonable argument for each point, as well as talking about the consequences of leaving them off. I originally thought that'd be one post, but then I did a bunch of research to make sure I wasn't just making stuff up and it turned out super long. So here's what we're going to do: over the next month, I'll be publishing one post per week about a new rule. When I do that, I'll come back to this post and link them up. If you're here after July 2021, welcome! Get started by clicking the links below. Otherwise, you can sign up to be notified about new posts by putting your email in the box at the bottom of the post. Simplify When the Program Works</a> (June 22)</li> Clarify Your Intent</a> (June 29)</li> Centralize Behavior</a> (July 6)</li> Code for Now</a> (July 13)</li> </ol> Afterwards, I'll probably publish a conclusion including reader feedback in late July. I'd love to hear from you as you're reading these things; you can email me at brian@brianthicks.com</a>. Trying (and Failing) to Speed Up String.startsWith 2021-03-01T00:00:00+00:00 When I was optimizing elm-csv</a>, I noticed that Elm's String.startsWith</code> is implemented using JavaScript's indexOf</code> method. It looks something like this: function startsWith(haystack, needle) { return haystack.indexOf(needle) === 0; }</code></pre> This strikes me as inefficient! Doesn't that mean that if haystack</code> doesn't begin with needle</code>, it'll still look through the whole string? That seems to waste a lot of cycles on work after we already know the result. After thinking about it for a little bit, I imagined a faster way: function startsWith(haystack, needle) { return haystack.slice(0, needle.length) === needle; }</code></pre> But, I didn't know for sure if it would be faster, so I started measuring things. After a few benchmarks things were looking promising! I found that String.prototype.slice</code> and String.prototype.length</code> operate in more-or-less constant time, and that ===</code>, while not constant-time, is heavily optimized. Hypothetically, I thought, this means that a slice</code>-based implementation of startsWith</code> could run much quicker than scanning through the whole string with indexOf</code>. Chrome Uses Actual Magic in Their Strings</h2> But just to make sure my intuition about indexOf</code> being slow was correct, I decided to benchmark that as well. Specifically, I decided to benchmark how long browsers took to find a single character in a string that doesn't contain the character: "abc".repeat(size).indexOf("d");</code></pre> I expected that every time I multiplied size</code> by 10, I should see roughly a 10x slowdown in operations per second. But then… Browser</th> 1 "abc"</th> 10 "abc"s</th> 100 "abc"s</th> 1,000 "abc"s</th></tr></thead> Chrome 88.0.4324.182</td> 866,026,834</td> 869,256,882</td> 858,563,862</td> 865,370,315</td></tr> Firefox 85.0</td> 43,557,275</td> 44,947,174</td> 41,059,919 (-8.65%)</td> 16,831,928 (-59.01%)</td></tr> Safari 14.0.1</td> 49,240,581</td> 34,661,639 (-29.61%)</td> 7,162,615 (-79.01%)</td> 877,449 (-87.75%)</td></tr> </tbody></table> Chrome has… NO REAL SLOWDOWN? WHAT? This data suggests that Chrome can find a substring in any length string in constant time. How is that even possible? At this point, I felt pretty out of my depth, and decided I needed to ask my friend Luke Westby, Real-Life Computer Genius™</a>, what was going on. He knew right where to look and figured out that Chrome has a special optimization for single-character search strings in indexOf</code></a>. In other words, people have hand-tuned V8 over the years to go ridiculously fast in all sorts of situations, and this is one of them. Well, OK, let's try finding some multi-character string instead. I used "nope", which doesn't trigger this optimization, but still fails to match "abc". Does that give us a result that matches my intuition? Browser</th> 1 "abc"</th> 10 "abc"s</th> 100 "abc"s</th> 1,000 "abc"s</th></tr></thead> Chrome 88.0.4324.182</td> 885,211,831</td> 863,845,523 (-2.41%)</td> 849,821,335 (-1.62%)</td> 837,391,727 (-1.46%)</td></tr> Firefox 85.0</td> 58,648,716</td> 44,107,472 (-24.79%)</td> 40,529,513 (-8.11%)</td> 16,768,019 (-58.63%)</td></tr> Safari 14.0.1</td> 102,071,445</td> 26,242,114 (-74.29%)</td> 4,212,339 (-83.95%)</td> 467,329 (-88.91%)</td></tr> </tbody></table> That's kinda like what I expected, but Chrome is still incredibly fast at this! Well, OK, that's fine. Good for them. But at this point I got curious (and a little devious): how can I defeat Chrome's optimizations? What is the worst-case scenario that they can't possibly have handled already? I tried to imagine the algorithm that you'd have to write to implement indexOf</code> and came up with something like this: Look at the first character in haystack</code>.</li> If it matches the first character in needle</code>, look at the next pairs of characters in the two strings one by one.</li> If you get to the end of needle</code> and they all match, you've found the index.</li> But if one pair doesn't match, you've got to start over at the next character in haystack</code>.</li> </ol> With that in mind, what's combination of needle</code> and haystack</code> would perform the worst? I think it's any pair of strings where haystack</code> is all but the last character of needle</code> repeated over and over. So, maybe a benchmark like this? "abc".repeat(size).indexOf("abcd");</code></pre> That can never succeed, but it'll make the browser do a lot of extra work reading the initially-matching "abc" over and over. Let's see the results: Browser</th> 1 "abc"</th> 10 "abc"s</th> 100 "abc"s</th> 1,000 "abc"s</th></tr></thead> Chrome 88.0.4324.182</td> 877,221,230</td> 863,929,216 (-1.52%)</td> 688,398,676 (-20.32%)</td> 125,023 (-99.98%)</td></tr> Firefox 85.0</td> 61,384,762</td> 12,874,505 (-79.03%)</td> 1,366,202 (-89.39%)</td> 137,245 (-89.95%)</td></tr> Safari 14.0.1</td> 100,546,438</td> 25,948,095 (-74.19%)</td> 4,170,375 (-83.93%)</td> 467,753 (-88.78%)</td></tr> </tbody></table> Finally, Chrome has similar performance to other browsers. I don't think this is representative of real workloads, but even so Chrome performs admirably. Hats off to the V8 team for this implementation, but honestly, it's good to know they're still mortal! Comparing the Implementations</h2> So now that we know more about the component pieces involved in this optimization (for example, knowing to avoid single-character test strings), we can check if slice</code> actually gives us a real-world speedup. In setting this up, I decided to go for the middle of the road to try and benchmark the browsers under something that you can kind of squint at and think "yeah, that will probably happen reasonably often in real life." In this case, that meant setup looked like this: haystack = "abc".repeat(100); needle = "abc";</code></pre> Then the indexOf</code> and slice</code> benchmarks looked like the code at the top of this post: haystack.indexOf(needle) === 0;</code></pre>haystack.slice(0, needle.length) === needle;</code></pre> That gave me these results: Browser</th> indexOf</th> slice</th> % Diff</th></tr></thead> Chrome 88.0.4324.192</td> 871,312,454</td> 870,810,291</td> -0.06%</td></tr> Firefox 86.0</td> 32,428,394</td> 46,845,953</td> +44.46%</td></tr> Safari 14.0.1</td> 430,854,455</td> 1,352,674,261</td> +213.95%</td></tr> </tbody></table> Note that Chrome and Firefox had released new versions between me testing indexOf</code> and this test. I kept the old data here because it doesn't seem to have changed the performance, but I just want to be up front that the versions are different! Anyway, that said: looks like a result within the margin of error for Chrome (around 0.4%), but a reasonable speedup for Firefox and a big win for Safari. What does the failure case look like? (Same haystack</code>, but needle = "nope"</code>): Browser</th> indexOf</th> slice</th> % Diff</th></tr></thead> Chrome 88.0.4324.192</td> 874,951,212</td> 854,116,367</td> -2.44%</td></tr> Firefox 85.0</td> 31,026,729</td> 22,180,114</td> -28.51%</td></tr> Safari 14.0.1</td> 4,189,768</td> 1,508,770,435</td> +35,910.83%</td></tr> </tbody></table> This time, Chrome has a slowdown greater than the measurement error, which surprised me! Same with Firefox, but to a larger degree. But the real story here is Safari, which can now do this operation over 1.5 billion times per second. Billion! What even! Is It Worth It?</h2> So, with all that measurement done, is this optimization worth it? I'm actually not really sure. This is basically nothing in Chrome but means making some real performance tradeoffs in Firefox. And, even though it looks like a huge win in Safari, my claims can only be as strong as the underlying assumptions I'm making in choosing test data. So, just to be explicit about that: These benchmarks assume that haystack</code> is a long-ish string (300 characters.) Many of the gains we see here are erased when the strings are short. Safari in particular, drops to only about a 100% gain in the failure case with a shorter string (I tested at 3, 30, 150 characters, all with similar results.)</li> The needle</code> strings are neither a single character nor a worst-case scenario for indexOf</code> performance.</li> Success and failure cases are equally likely. We can't make any assumption other than that at the runtime level, although we could make a good argument to switch this for individual applications. For example, we could choose to use slice</code> in an application where we expected most checks on startsWith</code> would succeed, since that's basically a wash in Chrome and much faster in Firefox and Safari.</li> We care about browsers based on global total share instead of splitting out into desktop or mobile. This is another area where you could make different decisions at the application level: if the users of your app skew more towards using Chrome or more towards Safari, this payoff for this optimization gets better or worse.</li> </ul> Of course, if you can measure that your application would benefit from this optimization, go ahead and do it yourself! Calling String.slice 0 (String.length haystack) === needle</code> has some overhead in Elm (due to equality having different semantics) but you may still able to get the majority of the benefits if your users are mostly using Safari or you're running iOS webkit views or something like that. Just make sure to actually measure it! Optimizations at this level can be helpful, but they're more often dwarfed by DOM stuff in the runtime. As for me: at this point, I'm not going to make a PR to elm/core</code> with this improvement, even though I intended to when I set out. However, I found it interesting to look into these things, and I learned a lot about how strings are optimized in various browsers, so I'm gonna call it a win overall. nix-script 2021-02-23T00:00:00+00:00 I like writing quick little scripts to avoid having to remember how to do things. Most of the time I start in bash, thinking the task won't be too complicated... but then before I know it I'm having to reread the bash man pages to figure out how arrays work for the thousandth time. So really, I'd rather write scripts in a language that offers some more safety and better data structures. We're trying to learn Haskell at work, so maybe that? Let's see a hello world: #!/usr/bin/env runghc main :: IO () main = putStrLn "Hello, World!"</code></pre> That seems pretty reasonable, and only takes like 300ms to compile and run. Not awful, but it's not as fast as a bash script, and it means I have to use Haskell's standard prelude instead of something safer like relude</a> or nri-prelude</a>. It's reasonable to solve this, though: I can just use nix-shell</code> to get a ghc</code> with packages. #!/usr/bin/env nix-shell #!nix-shell -p "(pkgs.haskellPackages.ghcWithPackages (ps: [ ps.text ]))" #!nix-shell -i runghc {-# LANGUAGE OverloadedStrings #-} import Data.Text.IO main :: IO () main = Data.Text.IO.putStrLn "Hello, World!"</code></pre> Well, that works, but I've traded speed for flexibility: run time has grown to over 2 seconds! Eep! I don't think I should have to make this trade, so I wrote nix-script</code></a>. It transparently manages a compilation cache for these kinds of scripts, and lets you specify dependencies and build commands inside your source file! That means the nix-shell</code> example above can be rewritten like so: #!/usr/bin/env nix-script #!buildInputs (pkgs.haskellPackages.ghcWithPackages (ps: [ ps.text ])) #!build ghc -O -o $OUT_FILE $SCRIPT_FILE {-# LANGUAGE OverloadedStrings #-} import Data.Text.IO main :: IO () main = Data.Text.IO.putStrLn "Hello, World!"</code></pre> The first time you run that, it'll compile the script to a binary, then run it. That takes about two seconds on my machine. The second time, and going forward until you change the script, it detects that it already compiled to a binary, so it just runs the compiled version. That takes 30ms or so for me! Big improvement! But it's really not that fun to have to figure out that #!build</code> line every time, and I always forget how to call pkgs.haskellPackages.ghcWithPackages</code> correctly... so there's also a wrapper script called nix-script-haskell</code> that makes this nicer: #!/usr/bin/env nix-script-haskell #!haskellPackages text {-# LANGUAGE OverloadedStrings #-} import Data.Text.IO main :: IO () main = Data.Text.IO.putStrLn "Hello, World!"</code></pre> And, in addition to the speed boost, we can depend on any package in the Nix ecosystem! For example, here's how you'd add and call jq</code>: #!/usr/bin/env nix-script-haskell #!runtimeInputs jq import System.Process main :: IO () main = do formatted <- readProcess "jq" ["--color-output", "."] "{\"name\": \"Atlas\", \"species\": \"kitty cat\"}" putStr formatted</code></pre> It's also pretty easy to create more wrapping interpreters, so we also ship one for bash. Even though it's not a compiled language, we can cache the nix environment with the exact dependencies the script needs! You can get this at github.com/BrianHicks/nix-script</a>. There are installation instructions in the README, both for standalone use and use within a larger Nix project. Enjoy, and let me know if—and how—you use this! elm-csv, package and talk 2021-02-15T00:00:00+00:00 In the beginning of February, I spent a lot of time working on a new library for parsing CSV data in Elm. (Basically, the existing libraries did not have andThen</code> and I needed it!) The resulting library turned out quite nice, I think, and in addition it's very fast! I gave talk to the Elm London user group about it, which you can see at elm-csv: be as boring—and as fast—as possible</a> or below. If you'd just like to see the code, it lives at github.com/BrianHicks/elm-csv</a>. </iframe> </div> </figure> Should Elm files be long or short? 2021-01-11T00:00:00+00:00 the Elm Guide page on app Structure</a> says: [Don't] Prefer shorter files. In JavaScript, the longer your file is, the more likely you have some sneaky mutation that will cause a really difficult bug. But in Elm, that is not possible! Your file can be 2000 lines long and that still cannot happen. </blockquote> Someone asked about this on the Elm Discourse</a> recently, wondering if this advice meant they should prefer longer Elm files over shorter ones. I thought this was a really interesting question! I answered on Discourse, but I also want to clean up my answer to share a little wider. Here goes! So, personally, I don't really agree with that part of the docs. I've worked in codebases with short files and longer files, and it didn't seem like either style was more or less prone to mutation bugs. That makes me think that file length is not the problem, mutation is. And mutation is not a problem in Elm—it's just not allowed in the language at all, regardless of file length! What does that mean for how we write Elm? Should Elm files be longer or shorter than JavaScript files? Well, neither really! If we accept that file length is not strongly correlated with this kind of bug, it doesn't matter! Elm files may be longer as a consequence of the things I'm gonna say below, but that's a consequence only—not a goal! Now, with that out of the way, we can get to the more interesting question: what is the right file length? I think to answer that we need to figure out what a module is actually good for. In Elm, I use modules for two reasons: encapsulation: hiding or exposing bits of the implementation to make a cohesive API.</li> namespacing: defining things like toString</code> under different name spaces.</li> </ol> The second is kiiiinda a consequence of the first, so I think we can squint and ignore it. So for our purposes, I'm asserting that files are for encapsulation. That means the right file length is the one where your encapsulation works to prevent bugs, either by preventing encapsulation-violating behavior or making your code easy enough to understand that you can verify it's doing the right thing. That doesn't mean long files or short files, but correct-for-the-situation files. I realize that this basically boils down to "it depends", which I know is a bit unsatisfying. Sorry! Maybe it will help if I share some assorted things I use to know if a module needs to be broken up (or not): Yes, split if: there's a function that should not be able to access internals of a data structure, but which can because it lives in the same module. Of course, the opposite is true too... if there's a function that needs to be able to access internals but can't the module boundary may be in the wrong place!</li> Yes, split if: there are two independent data structures in the same module. One smell here is if I have to write fooToString</code> and barToString</code> in the same module, and foo</code> and bar</code> are independently valuable, it may be time to give at least one a new home. The opposite ("join if…") looks the same as the previous item: if you split a single data structure into two modules, you'll frequently need to share too many internals and so should move everything into one file.</li> No, don't split if: there is just "too much code" in a file. I've split apps into Model.elm</code>, Update.elm</code>, View.elm</code> and almost always regretted it. Even if it's a little tricky to navigate a long file, it's way better than having to perform module gymnastics to prevent import loops.</li> </ul> All that said: like the rest of that page in the guide, I would encourage people to write long files and then break them up instead of making tiny files before it's actually necessary. I've been using Elm for something like half a decade now and the only reliable way I've found to put module boundaries in the right places is to wait and see what the right place is. I suppose that means I am, in effect, advocating for long files over short ones but again: that's a consequence, not a goal! Learning Requires Effort 2021-01-05T00:00:00+00:00 I recently finished How to Take Smart Notes</a>. The core idea I took away from the book is that learning requires effort. Two Kinds of Effort</h2> I think people generally recognize that learning requires effort, but have an easier time applying that to physical activities than mental ones. You don't really go around assuming that Olympic athletes are just people who jumped out of bed the morning of the games and decided to win gold in ice dancing. Instead, we recognize that athletes train hard for many years in order to reach the top of the field. But when it comes to learning in more abstract ways, it's somehow easier to forget that effort is necessary! For example, I've made the mistake many times of assuming I learned something just because I reached the last page of a book. With that in mind, I can think of at least two ways of putting in the effort to learn. There are definitely more, though! How to Take Smart Notes recommends using the Zettelkasten (slip-box) system covered</a> many</a> places</a> this past summer. In this method, "effort" looks like making notes by rephrasing and making connections to things you already know. Educationally speaking, this is called elaboration. Outside of note-taking, I see this in how programmers learn new tools and techniques. When someone says they want to learn something, a common response goes "well, make a small but useful project and see." I've told people that many times myself without recognizing that I'm telling people to learn through effort. These examples make "learning requires effort" seem pretty general to me, since it applies both in places I expected and places I didn't. That has the feel of a useful principle which I can use to judge if I'm learning or not! Am I putting in effort? If not, I'm probably not learning! Someone Else Can't Learn For You</h2> "Learning requires effort" also means that someone else can't learn for you. Think about it: if learning requires effort, and someone else is putting in the effort, then it's not you doing the learning. That's another mistake I've made many times over the years! It's natural for me to try and find the perfect resource when what I actually need to just get into the work of effortful learning as quickly as I can. I've seen this show up in several ways. The first is conflating building a collection with learning. Where's the learning effort in just collecting information from other people, whether papers, conference talks, or blog posts? If you're not putting in the effort to connect new stuff to what you already know, you're just playing gotta-catch-em-all with PDFs! The second is similar, but a little more specific: using other people's flashcards. I've seen quite a few places where you can download flashcard decks for various computer science concepts. At first glace, that seems fine, but where's the learning there? It's certainly a lot of effort to memorize a flashcard deck by rote, but it's the kind of effort that makes you learn the deck, not the subject. Making flashcards, on the other hand, can actually create connections as you think about how best to structure the subject in flashcard form. Put It All Together</h2> So, if learning requires effort, how can we learn? Simple: just put in the effort. But, oops, it's actually not so simple: what effort? What's right for you and for what you're learning? If you just thrash and flail, by trying things that are way beyond your current skill or level of understanding, you won't actually learn much. To be effective, the things you're trying to learn have to be just beyond what you can do already (in the zone of proximal development</a>.) You also have to make sure you're learning in a way that you can actually apply your new knowledge. That means working towards some level or standard of mastery, ideally one against which you can measure your progress. (For example, being able to clear the bar at a certain level in the high jump.) That's what a good teacher, mentor, or coach is for: someone who has helped other people learn the thing you're trying to learn can give you good advice on the best places to apply your effort as well as a standard of mastery to grow towards. Tracking and Lasting 2020-12-29T00:00:00+00:00 Just a short post to note that I've added a colophon</a> to this site. It goes over how to get the source of this site (git), what I track (not much, and no third-party stuff), and how the page is designed to last. I admit it's pretty self-indulgent, but I wanted to be public about some of the things I've been thinking about while building this site. Please feel free to email me</a> any questions or comments about it! I'd love to make it as clear as possible. To repeat the link, that's the colophon page</a>, and linked to from the footer. Check it out! time-to-horse 2020-12-22T00:00:00+00:00 Some colleagues of mine at NoRedInk coined a new term: time-to-horse, the time it takes you to get on your horse and ride. They used this in the context of their team getting familiar with a new area of our code, but I think it's also useful for talking about how long it takes someone to make meaningful contributions when joining a team. I think you probably want your time-to-horse to be as low as possible in either case. There are a bunch of things that affect this (like documentation, pairing and collaboration practices, and continuous integration pipeline quality) but I think they all boil down to "do you have fast feedback loops?" Phantom ID Types 2020-12-14T00:00:00+00:00 I previously wrote about tradeoffs of custom ID types in Elm</a>, and promised to explore a few points in the space. This is one! Recently I built a little game for my son to learn his letters and decided to see if I could find a nicer way to solve id-as-string problem. My goal was to find something in between the wild wild west of using String</code>s directly and the repetition of defining ID types for every resource. Anyway, I found something new to me. Maybe it'll be useful to you, too! The gist: you can make a reusable ID type by using phantom types (that is, types with a variable that appears in the definition but not any of the constructors.) Let's define a module for our IDs: module Id exposing (Id(..), decoder, sorter) import Json.Decode as Decode exposing (Decoder) import Sort exposing (Sorter) type Id thing = Id String decoder : Decoder (Id thing) decoder = Decode.map Id Decode.string sorter : Sorter (Id thing) sorter = Sort.by (\(Id id) -> id) Sort.alphabetical</code></pre> You use it in external definitions like this: module Shelter exposing (Shelter) import Cat exposing (Cat) import Id exposing (Id) import Json.Decode exposing (Decoder) type alias Shelter = { name : String , adoptableCats : List (Id Cat) } decoder : Decoder Shelter decoder = Decode.map2 Shelter (Decode.field "name" Decode.string) (Decode.field "adoptableCats" (Decode.list Id.decoder))</code></pre> Of course, this improvement still doesn't come for free. Id thing</code> is still not comparable</code> so you need a workaround if you want to use it in a Dict</code> or Set</code> (sorter</code> above.) You also lose some of the assurance that you're not constructing or matching on Id</code> in places where you shouldn't be. To put it another way, you can make a bad ID pretty easily: just call Id "a hot dog is a salad"</code>. Because the type is not used in the constructor, that's a valid whatever... Id Cat</code>, Id Dog</code>, it's all the same to the constructor. You also can't embed the ID of a record in the record itself (doing so would be a recursive definition.) I managed to get around needing this in my alphabet game (records do not need to refer to themselves because whenever they're displayed there's another piece of data containing the ID.) If you need it, you just do a little type trick: module Cat exposing (Id, Cat, decoder) import Id import Json.Decode exposing (Decoder) {-| An module-internal identifier. Has to be a custom type instead of an alias so the compiler can detect if it's being used improperly. -} type CatId = CatId type alias Id = Id.Id CatId type alias Cat = { id : Id , name : String , purriness : Int } decoder : Decoder Cat decoder = Decode.map3 Cat (Decode.field "id" Id.decoder) (Decode.field "name" Decode.string) (Decode.field "purriness" Decode.int)</code></pre> All the instances of CatId</code> should be erased during compilation. That is, they shouldn't end up adding any weight to your compiled JavaScript! And there you have it! To sum up: with this technique, you get the normal stuff you get by using custom types as IDs, like: nice type-checking and good errors (with really obvious fixes, like "you have a Id Dog</code> but you need an Id Cat</code>.")</li> reasonable levels of code reuse</li> </ul> But with these tradeoffs: You have to deal with comparable</code> not being extendable by custom types (which is true without a phantom type, too.)</li> You have to be disciplined about not deconstructing/matching against IDs in places where it wouldn't make sense to take responsibility for constructing them.</li> To get IDs in the records themselves (as opposed to just in Dict</code>s or whatever) you have to do a type trick.</li> </ul> So would I do this again? I think I would avoid using this in a project where I didn't have a high confidence that my fellow programmers knew the intent of the module. That means that I wouldn't to publish a package using this, or contribute code to a high-traffic open source repo using this pattern. But on my work team in private code, or in small apps that I build for myself, I'll definitely be coming back to this! Bang Shortcuts 2020-12-08T00:00:00+00:00 Another you-get-to-join-the-lucky-ten-thousand</a> post here! There are a bunch of shortcuts involving exclamation marks that shells will automatically expand. There are so many things you can do with these! (See the "History Expansion" section of the Bash manual page</a>) I just want to highlight a few of the ones I use every day: !!</code> expands to the same command you just typed. For example, entering !!</code> will just re-run your last command. You can use this to fix goof-ups. For example, sudo !!</code> will re-run the last command with sudo</code> prepended. Perfect for when you apt-get install</code> something as your login user instead of root</code>. </li> !$</code> expands to the last argument of the last command I mostly use this to perform several operations on the same file. For example, I mv config/database.yml.sample config/database.yml</code>, then kak !$</code> to edit it to customize. I also use this frequently to preview something that I've just generated. For example, make dist/index.json</code> and then less !$</code> (or jq . !$</code> to pretty-print JSON!) </li> !*</code> expands to all the arguments of the last command I don't use this one as much, but it's helpful when I misspell a command. For example, I use a tool called aide</code> at work. I often misspell it (e.g. aid build monolith</code>), say "ah shoot, again?" and then correct with aide !*</code>. </li> </ul> Some shells (ZSH at least) will automatically expand these for you if you hit tab after typing them so you can see what you're in for before you commit. the hyper key 2020-12-02T00:00:00+00:00 Have you ever noticed that no app binds a default keyboard shortcut to shift+control+option+command+whatever? That means you have a potential global keybinding prefix that will never have any conflicts! Brett Terpstra calls this the hyper key</a>. I like that! Since this is hard to hit (which, well, that's why nobody uses it by default) I have this bound to a special key on my Kinesis</a>, but I find it acceptable to use on my normal laptop QWERTY keyboard too. Here are some goodies I get: hyper + arrow: move a window to the top/right/bottom/left half of my screen with Divvy</a></li> hyper + space: open Things</a>' quick input</li> hyper + semicolon: open Things' quick input with autofill</a></li> hyper + j: open the link for my next meeting with Next Meeting</a></li> hyper + enter: jump to Dash</a></li> hyper + ': jump to iTerm2</a></li> </ul> There are a few more I have bound (fullscreen, middle-of-screen shortcuts) but those are the main ones. I've found them really handy over the years. Let me know if you have any really good ones for yourself! elo-anything 2020-11-24T00:00:00+00:00 Part of my job as a team lead is dealing with a big list of work that my team is responsible for, but which we're not going to do immediately. Hypothetically, this is work we should be doing all the time, but we're a small team with a lot of responsibility, so our list tends to grow. We should work on the most impactful tasks first... but in a list of 40 or 50 little things, it's hard to know what's most important. This makes the job much harder! After a recent quarter where we didn't accomplish any ownership work at all (screaming face emoji), something had to change. So I built a tool to help! Before we continue, I'm assuming it's possible to get time allocated to maintenance/ownership work. I've heard of (and been on) engineering teams that can't get this time, but that's not a problem my current team has. I have a good enough relationship with our stakeholders that they trust me if I say something needs to be worked on. Plus, we've baked it into our schedules: at minimum, the team spends 15% of our time on maintenance/ownership work in any given week. (I save my Fridays for this. It's nice to end the week by hammering out 4 or 5 quick bug fixes!) But even with the time allocated, we can't make progress if we don't know what to work on, so I set out to figure out how to select tasks from our big list. So what do we work on?</h2> To start, whatever selection method needs to give me a rough order pretty quickly. I can't choose the most important thing among 50 items; there are just too many things to keep in my head and I'll have to guess. But it is possible if I'm picking from 10, or better yet 5, so getting even a rough order helps a lot! Items get added to the list over time, too, so the selection method needs to handle that gracefully. I don't want to make a bajillion comparisons for every new item to figure out exactly where it goes. And again, a rough order is acceptable: this doesn't have to be extremely precise, just good enough to choose which things to work on next. I'd also like the order to be able to change over time without having to re-sort the entire list. If we learn something new or change our minds about where we want to focus, it should be easy to get a new order. The Elo rating system</h2> Fortunately for us, there's something that solves for these constraints well already: the Elo rating system</a>! It defines a way to get a rough ranking of a big list as well as a way to adjust the ranks quickly when an item is over- or underrated. It's been used to rank chess players forever, too, so I'm not starting from scratch on a novel algorithm! To order a list of items (let's call them players to match the original use), you start with everyone at a given rating and start playing matches. When a player wins, they take a portion of the losing player's rating proportional to the difference between the two ratings. Then to get rankings, you simply sort players from the highest rating to lowest. The algorithm used to determine the rating change says: When two players are evenly matched, the winner's rating will go up by a little. There's not a lot of useful information in match outcomes when players are evenly ranked: a win could indicate a skill difference, but it could also be a fluke.</li> If an underrated player wins against someone more highly-ranked, the underrated player's rating will go up a lot. A win here has more information: the winner could be underrated or the loser could be overrated. A fluke could still be possible, but adjustments like these will stabilize rankings over time.</li> On the other hand, if a strong player wins against someone much lower-ranked, their score doesn't rise as much. It shouldn't be worth a lot of points to win against players you're expected to beat every time.</li> </ul> There's a little more to the math to adjust for scenarios where there are few matches played versus many, but that's the basic idea. These rules mean that if you win consistently you'll go up in ranking (or if you lose, you'll go down.) Given enough matches, you tend to get a stable ordering of players according to their level! So does the Elo rating system fit my criteria? A new or under-rated player can rise in the ranks quickly by challenging random higher-ranked players.</li> The more comparisons we make in the list, the closer to the true order items will get.</li> If we change our minds about something's importance, the rating can change relatively quickly.</li> </ul> Sounds like a plan! Ok! Let's use it!</h2> To make a long story short, I built an Elm app to create and maintain these rankings for me, and you can use it right now at elo.bytes.zone</a> (or get the source at git.bytes.zone/brian/elo-anything</a>.) Some features: You can rank however many items you need, and add and remove them at any time.</li> There's an undo button (which is helpful for second-guessing!)</li> New items are treated specially. Ideally, they'll get to the right ranking in 5 matches or less, and the system favors selecting new items so you can finish those play-in matches quickly.</li> You can load and save rankings to persist items over time.</li> </ul> I've been super happy to use this to sort our task list, and now our ownership work is consistently being accomplished! A nice benefit that I hadn't anticipated: when I go through the list with my team's PM, we can have a productive conversation about the relative importance of this versus that, which has revealed new information about differences in our priorities! Using this rating system has lead to a couple of good conversations about what the team should be working on, which I've been really happy about. Anyway, that's a wrap. Let me know if you end up using elo-anything. It's fairly polished at this point but I'm sure there are still weird edge cases we could find and fix! Tradeoffs of Custom ID Types in Elm 2020-10-26T00:00:00+00:00 In Elm, we store data in records. Here's a model to define a Cat</code>. Mew! module Cat exposing (Cat) type alias Cat = { id : String , name : String , purriness : Int }</code></pre> This particular implementation has a problem: you can do whatever with id</code> since it's a String</code>. Wanna concatenate it with another one? Go right ahead. Regex match? Why not! Using a String</code> here means that the compiler can't tell you if you're using the id</code> in the way it was intended to be used. One name for this is primitive obsession</a>, meaning code uses a language primitive (String</code> here) instead of a domain-specific object. This can make it harder to reason about and refactor the code, not to mention the fact that it doesn't guard against the problems above. A Custom Type</h2> We can fix this in Elm by making a custom ID type: module Cat exposing (Cat, CatId) type CatId = CatId String type alias Cat = { id : CatId , name : String , purriness : Int }</code></pre> With this, you get pretty much all the benefits of wrapping a primitive in a class in other languages: we can distinguish between a CatId</code> and a (hypothetical) DogId</code>. Before defining a type, these would have both been String</code>s and totally indistinguishable.</li> We've disallowed String</code> operations on the type. There will be no more concatenation, regex matching, et cetera.</li> If we don't export the CatId</code> constructor, nobody outside the Cat</code> module can construct a value of CatId</code>. With this setup, we can be pretty safe trusting values of CatId</code> have been constructed in safe ways (for example, that they've been deserialized from JSON provided by the server.)</li> </ul> The benefits have been well-documented elsewhere and are (in my opinion) pretty unambiguous, so I'm keeping this brief. I want to spend more time talking about the drawbacks of doing it in this particular way and how to get around them! Testing is Harder</h2> First of all, we can no longer construct values of CatId</code> in tests. This means we cannot construct any records or structures that depend on CatId</code> either. No Cat</code>, no Model</code> containing Cat</code>s, etc. I've seen some test code provide hand-rolled JSON values to a Decoder Cat</code> to get around this, but I'm not sure that's such a good idea. It raises complexity in tests and ties them to the decoder</code> instead of only the implementation we're trying to verify. Instead, I usually end up writing some constructor function like catIdForTestOnly : String -> CatId</code>. It has the same effect as exposing the constructor, but labels your intent clearly. I'd call this pragmatic, but I know there are reasonable people who'd call it a code smell. That's fine! We can disagree! But, regardless of your approach, you'll have to deal with the tradeoff of the hidden constructor here. No comparable</code> Implementation</h2> Second, you can't use CatId</code> where the compiler expects something matching comparable</code>. This shows up pretty frequently because elm/core</code>'s Dict</code> needs keys to be comparable</code> in order to provide fast lookups. Fortunately for us, the Elm community know about this (hopefully temporary) problem and has published a bunch of different packages to help get around it. I like rtfeldman/elm-sorter-experiment</code>, in which we just need to define a custom sorter function and pass it to dictionary and set constructors. For CatId</code>, that looks like this: idSorter : Sorter CatId idSorter = Sort.by (\(CatId id) -> id) Sort.alphabetical</code></pre> Now we can construct dictionaries and sets containing CatId</code> for fast lookups and still get the benefits we want. Wrapping Up</h2> So, that's one way to make sure you're doing the right things with IDs (or other small data) in your Elm code. Should you do it? I think that despite the tradeoffs, it's a yes most of the time! I've had a lot of success with doing this, and I'd recommend it to others. That said, there are some more ways to solve this problem, and I'm planning on exploring them in future posts. Renaming Files with Braces 2020-10-19T00:00:00+00:00 This has been well-documented elsewhere, but in the interests of helping someone else join the lucky ten thousand</a> today: Bash, ZSH, and other shells will expand comma-sepated arguments surrounded by braces into positional arguments. As an altogether contrived example, imagine you want to echo "cat car can" in the terminal. You could write echo cat car can</code>, right? But then you'd be writing ca</code> twice. That won't do! We must be maximally efficient! We can use brace expansion to only write ca</code> once: echo ca{t,r,n}</code>. The shell will expand that to your original "cat car can." Ta-da! Next, say you want to add "bat bar ban" to the output. Good news: the shell will expand more than one set of braces in the same word! echo {c,b}a{t,r,n}</code> will get you "cat car can bat bar ban". Sweet! So, back to real life... when is this actually useful? Well, I use it all the time for renaming or moving files! mv someFile.txt{,.bak}</code> expands to mv someFile.txt someFile.txt.bak</code>, adding the .bak</code> extension</li> conversely, mv someFile.txt{.bak,}</code> removes the .bak</code></li> mv config/database.yml{.sample,}</code> expands to mv config/database.yml.sample config/database.yml</code></li> cp {src,dist}/index.js</code> works great in Makefile</code>s too!</li> </ul> Some shells (ZSH at least) will even expand that if you hit tab</code> so you can see what you're going to execute before you hit commit. Handy! state-transition tables 2020-10-13T00:00:00+00:00 I was reading Wikipedia the other day (as you do) and found out about state-transition tables</a>. Basically, state transition tables show how a state machine transitions between different states. It's an alternative to drawing a state diagram</a> that helps you find holes in your logic. Wikipedia shows some pretty abstract tables, so I'm going to model a vending machine instead. To simplify things, we'll serve a single drink for a single quarter. The idealized version of the interaction with this machine (the "happy path") is: Put a quarter in</li> Press the button for the drink you want</li> Get the drink</li> </ol> To implement this, we have to manage two independent pieces of state: whether you've put money in the machine and whether it has at least one drink left to vend. Let's model the interaction above with a one-dimensional state-transition table. Using only one dimension keeps the modeling as simple as possible while still capturing enough detail to be useful: we have a column each for input, current state, next state, and side effects. To find out what happens after an event you just find the input and current state rows you care about and look at the matching next state and side effect. For our vending machine, it might look like this: Input</th> Current State</th> Next State</th> Side Effect</th></tr></thead> Insert Quarter</td> No Money, Some Drinks</td> Some Money, Some Drinks</td> -</td></tr> Hit Button</td> Some Money, Some Drinks</td> No Money, Some Drinks</td> Vend Drink</td></tr> </tbody></table> But, of course, we have to model what happens when we do things that are not on the happy path. Unfortunately, the one-dimensional version of the table doesn't give us a great view of that! To figure out where we have holes, we need to add more dimensions. Let's reorganize our states along the vertical axis and inputs along the horizontal axis to get a two-dimensional state-transition table. To read this table, match the current state along the vertical axis with the input along the horizontal. Our next state and side effects live in the intersections (I've separated them with a /</code>): ↓ Current State / Input →</th> Insert Quarter</th> Hit Button</th></tr></thead> No Money, Some Drinks</td> Some Money, Some Drinks / Nothing</td> </td></tr> Some Money, Some Drinks</td> </td> No Money, Some Drinks / Vend Drink</td></tr> No Money, No Drinks</td> </td> </td></tr> Some Money, No Drinks</td> </td> </td></tr> </tbody></table> And we see, uh... problems. When we look at things this way, it's clear that we've only defined two of the possible 8 outcomes! Writing things down in an orderly way revealed that we haven't specified all of the possibilities implied by our modeling. Let's fill the rest out. To make things easier, when the state stays the same or there's no side effect I've marked -</code>: ↓ Current State / Input →</th> Insert Quarter</th> Hit Button</th></tr></thead> No Money, Some Drinks</td> Some Money, Some Drinks / -</td> - / Beep</td></tr> Some Money, Some Drinks</td> - / Refund Quarter</td> No Money, Some Drinks / Vend Drink</td></tr> No Money, No Drinks</td> - / Refund Quarter</td> - / Beep</td></tr> Some Money, No Drinks</td> ???</td> ???</td></tr> </tbody></table> But when we fill things out, we can see that we have a potentially weird situation: what if we somehow have some money, but no drinks? The state machine should prevent that, since there's no new state field that could create this situation. But it's feasible to get there either via programming (for example, by modeling the state as two independent fields) or hardware issues (for example, someone prying open the machine to leave quarters in an atypical act of vandalism.) Our modeling has revealed this undefined behavior way before we got to the code parts of our application, and the hardest part was making a table and looking for empty cells. Now I can take this same table to a stakeholder or domain expert and have a productive conversation about what they think should happen. I'd call that a win for just a little time spent modeling! (oh, and bonus: if you're using Elm, the one-dimensional form here is probably pretty familiar. "Input, Current State, Next State, Output" does the same job as update : msg -> model -> ( model, Cmd msg )</code>!) Thanks to Charlie Koster and Richard Feldman for reviewing drafts of this post. clown computing 2020-10-12T00:00:00+00:00 I keep using the term "clown computing" in conversations but I've never defined it. Clown computing is a riff on "cloud computing", but instead of provisioning your software according to high availability or other actual reasonable architectural principles, you just stuff as much software as you can into a single cloud VM. Like a clown car. Get it? Fuzzy Finding with Levenstein Distance 2020-10-06T00:00:00+00:00 When I'm navigating around a codebase I know, I prefer to use a fuzzy file finder instead of browsing a directory tree. Lowering the barrier to navigating between files is really helpful for me to move around a codebase quickly and get work done. Ideally, I type a few characters and end up in the file that I was thinking of, quick as that. But in late 2019, I got fed up trying to fuzzy-find files in my main repo at work. It has something like 8,000 checked-in files and 170,000 untracked files (bundled gems, node_modules</code>, etc.) so it took a long time to load all the files. To add to that, fzf</code>'s default configuration favors short matches near the beginning of the string. When I'm deep in the project hierarchy, I want long matches with my term near the end first! In short: my fuzzy matches became both slow and irrelevant. Not a good combination for a tool meant to save me time! Both of these things are fairly easy fixes on their own (I could sub in git ls-files</code> for find</code> to get matches and configure fzf to sort by long/end instead of short/beginning.) But I decided that I wanted to improve my experience: in addition to end-of-filename matching, I want to favor matches for equivalent test files. If I'm in a file and I type "spec" or "test" (depending on the language conventions) I should ideally get to the relevant test code. Given that most test files have some name symmetry to the files they test (e.g. app/models/user.rb</code>, spec/models/user_spec.rb</code>), what I want is files that are named similarly to the file I'm currently editing. Enter Levenshtein distance, an algorithm which calculates the amount of edits you'd have to change to change a string into another string. For example: a</code> to ab</code> has an edit distance of 1, because you add b</code>.</li> ab</code> to a</code> also has an edit distance of 1, because you remove b</code>.</li> a</code> to b</code>, on the other hand, has an edit distance of... 1! The first time I looked at this, I thought it was 2 (add 1, remove 1) but replacements also count as a single operation!</li> ok, more complex: grammar</code> to programmer</code> is 4. (add pro</code>, replace a</code> with e</code>)</li> </ul> You can do this with any two strings, although it requires more calculations as the strings get longer. Let's apply this to filenames! Starting from app/models/user.rb</code>, spec/models/user_spec.rb</code> has an edit distance of 8, but a less similarly-named file, app/controllers/admin_controller.rb</code>, has an edit distance of 22. If we sort our candidate files by their edit distance from the source string, we can tell fzf</code> that the input files are already in preference order (--tiebreak=index</code>) and get exactly the editing experience I was after! So, does this work? I'm happy to report that it totally does! I've been using similar-sort for about a year and a half now with no modifications, and I anticipate being able to use it as long as I need! And now you can grab it as well at git.bytes.zone/brian/similar-sort</a>. That repo includes instructions for building and installing, as well as integration with Kakoune and Vim. It's licensed CC BY-SA 4.0 (since that's the license for the implementation of the Levenshtein distance implementation I grabbed from Wikipedia.) Enjoy, and let me know if you use it! callCabal2nix 2020-09-22T00:00:00+00:00 When you're setting up a Haskell project in Nix, most advice says "call cabal init</code> and then cabal2nix . > default.nix</code> to get a buildable Haskell project." But it turns out you don't have to run cabal2nix</code> by hand to keep it up to date! pkgs.haskellPackages.callCabal2nix</code></a> does the same thing as cabal2nix</code>, but without writing a file. The nixpkgs manual doesn't mention it since it uses import-from-derivation, which can cause some problems with Hydra</a> but I've found it super helpful so I'm writing it up! If you need more detail than what I've written here, diving into the source is probably your best bet</a>. The basic usage: call pkgs.haskellPackages.callCabal2nix</code> (note the lowercase n</code> in nix</code>) with... the project name</li> the path to the project source</li> the set of options you'd normally provide to a call to the stuff cabal2nix</code> generates</li> </ol> If you want something similar to the default.nix</code> produced by the cabal2nix</code> instructions in the nixpkgs manual</a>, put this in default.nix</code>: { pkgs ? import <nixpkgs> { }, ... }: pkgs.haskellPackages.callCabal2nix "name-of-your-haskell-project" ./. { }</code></pre> callCabal2nix</code> is always a sibling of ghcWithPackages</code>, so you can pin your compiler version in the same way: { pkgs ? import <nixpkgs> { }, ... }: pkgs.haskell.packages.ghc865.callCabal2nix "name-of-your-haskell-project" ./. { }</code></pre> If you rely on getting a build environment with cabal2nix</code>, the output callCabal2nix</code> has a .env</code> which you can provide to a mkShell</code>'s inputsFrom</code> stanza: { pkgs ? import <nixpkgs> { }, ... }: pkgs.mkShell { inputsFrom = [ (pkgs.haskellPackages.callCabal2nix "project" ./. { }).env ]; }</code></pre> If you were previously calling cabal2nix</code> with command-line options, you can instead use callCabal2nixWithOptions</code></a> (again, note the lowercase n</code> in nix</code>.) This form adds a string before the final argument which is eventually used literally in a call to cabal2nix</code></a>. That's it! Enjoy not having to write make</code> rules to call cabal2nix</code>! Oh and one final note: all the code links in this post are links to specific lines the repo as of the time of writing. If you are trying to find more information in the future, you may want to switch to the master</code> branch of nixpkgs and sniff around for callCabal2nix</code> to get the most up-to-date view. git root 2020-09-10T00:00:00+00:00 I sometimes find myself in a situation where I've cd</code>'d several levels into a project and don't remember exactly where I am, but I want to get back to the project root for my next command. This doesn't come up for me alllll the time, but the last time it did I decided to do something about it. To cut this short: if you run git rev-parse --show-toplevel</code>, git</code> will print out the location you cloned the repo. In other words, if you run git clone https://git-host.com/user/project.git ~/code/project</code>, that command will return ~/code/project</code>. So I've just added that to my aliases: root = "rev-parse --show-toplevel"</code>. Now I can do git root</code> or cd $(git root)</code>. That makes this problem much less annoying for me! Someone pointed out to me that I can probably also add an alias in my shell like alias cdroot='cd $(git root)'</code> to make this even easier, but I haven't felt the need for that as urgently yet. tmux-session 2020-04-16T00:00:00+00:00 I wrote a little script to create and fast-switch between different tmux</a> sessions, and thought I'd share it real quick so others can use it too. It's got two composable components you can use in your own stuff: The Script Itself</h2> If I'm not running the right tmux session, I run tmux-session</code>. If I'm not in a tmux client (that is, just in a plain shell) it creates a session with a nice name and opens it. If such a session already exists, it switches to it instead. This works inside a client too! The point is for it to always do the right thing so I don't have to think about the four cases: outside and inside a client, with the target session existing or not. The session names are based on git checkouts, when possible. If you invoke the script from within a git checkout, it'll find the root of the project (the directory containing .git</code>) and name the session after it. If you're outside a git checkout, it'll name the session after the current directory. Here's the commented shell script that you can stick somewhere in your $PATH</code>: #!/usr/bin/env bash set -euo pipefail # find the project root directory by examining each parent starting from the # current directory. If you use a different VCS, you can change `$ROOT/.git` # below to something else: for example, Mercurial would use `$ROOT/.hg`. ROOT="${1:-$(pwd)}" while ! test -d "$ROOT/.git" && test "$ROOT" != "/"; do ROOT=$(dirname $ROOT) done # if we walked all the way up and hit the root directory, just name the # session after the current directory. Handy for creating sessions when # you're working in directories like ~/Downloads that won't be tracked # with git. if test "$ROOT" = "/"; then ROOT=$(pwd) fi # tmux doesn't allow dots in session names, so we replace them with # dashes. This shows up surprisingly often in my life! For example, my dotfiles # repo is named `dotfiles.nix`, which gets normalized to `dotfiles-nix`. This # is close enough to the actual name that I know what I'm selecting when I # switch to it. SESSION=$(basename $ROOT | sed 's/\./-/g') # are we in a tmux client already? If we are, `$TMUX` will be set. if test -z "${TMUX:-}"; then # ah, we're not in a client? Create a session and enter it! Some quirk in # my installation make the TMUX_TMPDIR and -u2 necessary when starting # the server; you may not need it. exec env TMUX_TMPDIR=/tmp tmux -u2 new-session -As "$SESSION" -c "$ROOT" else # we're already in a client? Neat. Let's make a new session with the # target name in the background... if ! tmux has-session -t "$SESSION" > /dev/null; then tmux new-session -ds "$SESSION" -c "$ROOT" fi # ... and then switch our current client to it! exec tmux switch-client -t "$SESSION" fi</code></pre>The Quick Jumper</h2> I like to be able to jump between project directories quickly by just typing a couple letters of their name. fzf</code></a> helps a lot here! Here's the commented ZSH source (but I think this would work in bash as well!) Note that this assumes my own checkout pattern (I check out all repos like ~/code/owner/project</code> and have some git aliases to manage that for me.) You could just as easily adapt it to your own checkout patterns, or just a commonly used projects list that you keep as a file (just pipe that into fzf!) tmux_jump() { # starting in ~/code... BASE="$HOME/code" # ... look for directories exactly two levels deep (`~/code/owner/project`) # and match them with fzf. In this case we break ties by favoring matches # on the project name instead of the owner name (implementation means # favoring matches closer to the end of the string.) This is simplified a # little bit with the `--select-1 --query="$1"` line: if there's only one # match for the argument passed in as the first argument to this function, # we select immediately instead of asking for an interactive selection. SELECTED=$(find "$BASE" -mindepth 2 -maxdepth 2 -type d | sed "s|$BASE/||g" | fzf --tiebreak=end --select-1 --query="$1") # fzf will exit with a non-zero code if you ctrl-c or ctrl-g out of # it. We use this as a signal that we don't want to jump after all. if [[ "$?" != 0 ]]; then echo "cancelling!"; return 1; fi # call tmux-session on the *full* path to the matched project! tmux-session "$BASE/$SELECTED" } # and alias this so I can just do `t bytes.zone` instead of having to type # tmux_jump every time. alias t=tmux_jump</code></pre> And... that's it! Hope you enjoy it! Reducing Asset Size With Subsetting 2020-03-03T00:00:00+00:00 Note: as of December 2020, this site no longer uses custom fonts. This script is still useful, but you're not loading the output in your browser right now. When I was building this site, I noticed that my web fonts were kind of big: 52K fonts/OpenSans.woff2 48K fonts/Exo2-BoldItalic.woff2 24K fonts/Jetbrains-Mono.woff2 48K fonts/OpenSans-Italic.woff2 48K fonts/Exo2-Regular.woff2 48K fonts/Exo2-Bold.woff2 48K fonts/OpenSans-BoldItalic.woff2 52K fonts/OpenSans-Bold.woff2 368K total</code></pre> If you happened to load all the fonts with a cold cache, you'd be downloading a bit over a third of a megabyte. These were by far the heaviest part of the site, especially compared to the HTML files. Those are something like 12kb each, a quarter of the size of even one font. I don't like that difference, so let's see if we can make them smaller! But what's in those files anyway? Why are they so big? Consider Exo 2</a>, my heading font: in addition to the normal ASCII characters, it has a ton of Greek, Cyrillic, and Vietnamese characters. This is great! Tons of people speaking all kinds of languages can use this font to communicate… but since this site doesn't use those characters, serving them doesn't do anyone any good. Fortunately, this is a known problem, and we can solve it with something called subsetting</a>. Subsetting, in general, is where you retrieve only the parts you need from a large data set. In fonts, this means removing all the glyphs you don't need from a font before you serve it. Sounds like a plan! But how? Breaking it Down</h2> Well, since this is a static site, all the content is known in advance. Hypothetically, that means that I can examine all the markdown files and figure out what characters I need to render it. But when I thought about this more, I realized it wouldn't work: what about the navigation and footer? I also render headers and code samples in different fonts than I do the body copy. Well, again, static site! I can surely just inspect the output, right? But another challenge there: the exact font—including things like bold and italic—is set with CSS. Since normal, bold, and italic live in separate files, I need to calculate the style rules the way a browser would to be able to make the smallest possible subsets. But that means… oh… I should probably just drive a headless browser, huh? Sigh. Accepting the Inevitable</h3> I really didn't want to set up a headless browser for this. I had some bad experiences in the past trying to automate things with Selenium, not to mention all the pain I feel writing Capybara tests at work… I felt like it would be slow, error-prone, and be hard to set up in Netlify</a>. Well, good news if you're in the same boat: the Puppeteer</a> project makes this way easier than it used to be! I evaluated several options (including Servo and jsdom) before settling on it, and I was happily surprised! Get the Data, Already!</h2> The basic strategy here looks like: start a headless Chrome instance with puppeteer</li> load an HTML page</li> find all the visible text nodes</li> remember their content and computed style</li> </ol> That's it; let's go! First we start the browser and load a page: (async () => { const puppeteer = require("puppeteer"); const browser = await puppeteer.launch(); const page = await browser.newPage(); for (file of process.argv.slice(2)) { await page.goto("file://" + file); // the rest of our script (next code block) } })();</code></pre> Aside: I don't know if these should be const</code> or let</code>... I've heard both but the puppeteer docs use const</code> so I'm going to, too Then let's walk the DOM to find all the text nodes: const fileOut = await page.evaluate(function () { // we have to define everything we need inline here, since `page.evaluate` // runs everything in the browser context. function uniqChars(text) { let out = []; for (let char of text) { if (!out.includes(char)) { out.push(char); } } out.sort(); return out.join(""); } // we're going to accumulate information about each text node in this object. // see the next code snippet for how we actually use it. let out = {}; // todo is a stack of nodes we have left to visit. We start with the top node // of the document and work our way down from there. let todo = [window.document]; while (todo.length !== 0) { let node = todo.shift(); // `childNodes` is not an array, but you can still access its children by // index. We are descending into the node so we just push all the children // onto the stack. It doesn't matter much for this application whether we // use a depth-first or a breadth-first search—we want to get all the // nodees either way. for (var i = 0; i < node.childNodes.length; i++) { todo.push(node.childNodes[i]); } if (node.nodeName === "#text") { // the rest of our script (next code block) } } return out; });</code></pre> Finally we can accumulate our calculated styles. I do this in an object with the style information as the key (out</code> above) so we can add characters to it easily: let styles = window.getComputedStyle(node.parentElement); // we don't want to include text in invisible nodes (things like the guts of // <script> or <style> tags.) Just skip 'em! if (styles.display === "none") { continue; } for (let face of styles.fontFamily.split(", ")) { // fonts with spaces in the name get quoted, so we need to remove those. face = face.replace(/"/g, "", -1); let info = { face: face, weight: styles.fontWeight, style: styles.fontStyle }; let key = JSON.stringify(info); if (out[key]) { out[key].chars = uniqChars(out[key].chars + node.textContent); } else { out[key] = { font: info, chars: uniqChars(node.textContent) }; } }</code></pre> Finally, after analyzing each file, we combine the objects into a list. (I haven't shown this but you can see it in the source as of this writing</a>) When the script returns, we end up with a JSON blob like this. This is just the homepage</a> and I've removed all the fallback fonts. If I had passed more files into the script (for example, the output of find</code>) I would see much more information in this array: [ { "font": { "face": "Exo 2", "weight": "700", "style": "normal" }, "chars": " .HTbehnorstyz👋" }, { "font": { "face": "Exo 2", "weight": "400", "style": "normal" }, "chars": "acdeklopst" }, { "font": { "face": "Open Sans", "weight": "400", "style": "normal" }, "chars": " !',-.04:ABCDEHIJLMNORSTYabcdefghiklmnoprstuvwyz❤️" } ]</code></pre> There are some interesting things happening here. First, I mentioned that I only was using ASCII… well, that's a bit of a lie; I'm also using a few emoji. The script found and included these. Second, I'm not even using all the ASCII characters; q</code> and x</code> are absent from Open Sans (my body font.) True, this is just for one page, but it holds across the rest of the content: each font ends up only needing to include well under 100 glyphs to be able to render everything on the whole site. Subsetting, for Real</h2> After all that analysis, how do we actually subset the fonts? I'm using a tool called pyftsubset</code>, available in the fonttools</a> Python package. Put simply, to subset a font, you need to call pyftsubset {input font} --unicodes={code points}</code>. There are a few more flags in the actual script I use, but that's basically it. The script itself</a> is not terribly exciting, so I'm not going to show much of it here. It basically slurps down the output of the face-finding script, calls pyftsubset</code> on the result, and prints some statistics about the filesize difference: Finding subsets of 8 fonts on 10 pages Subset ./dist/fonts/Exo2-Bold.woff2 from 46656 to 5608 bytes (12.02% of original size, 61 glyphs) Subset ./dist/fonts/OpenSans.woff2 from 50116 to 8240 bytes (16.44% of original size, 81 glyphs) Subset ./dist/fonts/Exo2-Regular.woff2 from 46300 to 3440 bytes (7.43% of original size, 35 glyphs) Subset ./dist/fonts/Jetbrains-Mono.woff2 from 22368 to 10564 bytes (47.23% of original size, 87 glyphs) Subset ./dist/fonts/OpenSans-Italic.woff2 from 48148 to 3668 bytes (7.62% of original size, 18 glyphs) Subset ./dist/fonts/OpenSans-Bold.woff2 from 51932 to 4548 bytes (8.76% of original size, 27 glyphs)</code></pre> And looking at the final font sizes, I feel much better aabout shipping these to people's browsers. With this result, I can load all the glyphs in all the fonts I need in less size than just one of the original font files: 12K dist/fonts/OpenSans.woff2 12K dist/fonts/Jetbrains-Mono.woff2 4.0K dist/fonts/OpenSans-Italic.woff2 4.0K dist/fonts/Exo2-Regular.woff2 8.0K dist/fonts/Exo2-Bold.woff2 8.0K dist/fonts/OpenSans-Bold.woff2 48K total</code></pre> Aside: I also have some uncompressed-but-unused files in the output of my site, but they aren't ever referenced, so a browser shouldn't ever load them. I'm not counting those towards the filesize total here. Success!</h2> In the end, I reduced the size of my font files by about 87%, going from 386K to 48K. The final subsets have about 300 total characters to render my site. While this will grow over time as I add more content, it should level out pretty quickly. In short: it worked! So, in writing about this, would I recommend you do it too? Solid maybe! I think it makes sense to do this on a site where: You're using custom fonts. If you're using generic font names (like sans-serif</code> or monospace</code>) or your font stack is a mix of built-ins and system fonts (e.g. Helvetica, Arial, sans-serif</code>), there's no need to save on bandwidth here; your readers already have the whole font!</li> All your content is known in advance. If you have anything that's dynamically loaded, you'll get really weird results where some—but not all—of your characters will be replaced by the fallback font.</li> Fonts are actually the largest asset class. If you're serving a 10mb JPEG as a background image, compressing that is a lot easier and will be a much better experience for your readers than figuring out a subsetting pipeline for your site. The framework I use, elm-pages</a>, takes care of compressing images for me. If yours doesn't, searching around for image optimization tools will probably net you some easy wins.</li> You don't care about cache misses (in the short term.) Your custom subset is all but guaranteed not to be in a visitor's cache. For a site like mine, a really high percent of visitors are likely to be new. That makes me feel more OK about serving fresh font files every time. But a big note here: on a site with more content, the subsets will tend to stabilize to the characters you actually use. In the long run, the fonts won't be quite this small but they will be much stabler.</li> </ol> If your site meets all those conditions, give it a try! I've linked to the source of my scripts in the page above, and you can check out the full latest source for this site at git.bytes.zone/bytes.zone/bytes.zone</a>. (is it just me or is it echoing in here?) Have fun!}}}

bytes.zone - Posts

Software estimation that doesn't suck

Should we design for iffy internet?

Local-First Foundations

platform engineers work on the meta-product

what is platform engineering?

Kratky in the basement

sourcing secrets from 1Password

modeling CRDTs in Alloy - counters

Actions</h2> Next let's set up some things our documents can do. First, we should be able to increment a number in a document by removing the old count for this document and ID and adding the new one:</p>

sticky table headers with bottom borders

modeling CRDTs in Alloy - introduction and the importance of idempotence

Modeling in-flight requests in Alloy

tsort

Trying and Failing to Implement Artificial Ignorance

crunch_str

once p is true, it's always true

montage

moving genrules to library rules

buck2 basics

Modeling Git Internals in Alloy, Part 3: Operations on Blobs and Trees

Modeling Git Internals in Alloy, Part 2: Commits and Tags

Modeling Git Internals in Alloy, Part 1: Blobs and Trees

aligning Markdown tables in Helix

how to add weak fairness on actions

give aliases when inverting relations

what is the randomart image for?

fields as sets

signing commits with SSH keys

What does expect 1 mean in Alloy?

advice you might as well take

pagerank for my Obsidian notes

The Value of a Model is More Making than Having

Modeling Database Tables in Alloy

Digital Gardening in Obsidian

Scope in the TH element

Projects

Alloy

Yeah, but…</h2> So all that's super exciting, and I love using Alloy, but it'd be silly to pretend that it's perfect. Here's some examples:</p>

tree-grepper

The Four Rules of Simple Design: In Conclusion

Rule 4: Code for Now

Rule 3: Centralize Behavior

Rule 2: Clarify Your Intent

Rule 1: Simplify When the Program Works

My Take on the Four Rules of Simple Design

Trying (and Failing) to Speed Up String.startsWith

nix-script

elm-csv, package and talk

Should Elm files be long or short?

Learning Requires Effort

Tracking and Lasting

time-to-horse

Phantom ID Types

Bang Shortcuts

the hyper key

elo-anything

Tradeoffs of Custom ID Types in Elm