Code - a programming blog

An Antidote to AI Anxiety

Tue, 28 Jul 2026 05:51:24 -0400

When I was younger, my favorite painting was The Building of FTL1 by John Harris:

Looking back, I think what captivated me wasn’t the painting itself, but the feelings of awe and wonder that it evoked in me. In philosophy this phenomenon is called the sublime.

The sublime occurs when the imagination reaches its limits. When we can’t fully picture what we’re confronting, what’s left is a lingering feeling rather than a clear thought.

When "Best Practices" Aren't Best

Mon, 20 Jul 2026 04:09:44 -0400

A few years ago when DevOps and microservices were all the rage, it seemed like the software industry had reached a consensus on what good practice looked like. No book captured this better than Accelerate, a favorite of mine that guided my own work and thousands of other professionals. Based on survey data, the book lists 24 “capabilities” that every high performing engineering org should aim for.

Re-reading the book now, I still think it is filled with good ideas, but I would be hesitant to describe them as “best practices”. Every capability contains an assumption, and some may not apply in all cases.

A Theory of the Monolith

Sun, 12 Jul 2026 07:19:11 -0400

Why do we build monolithic applications? And when do we decide they’ve become too large and start breaking them up?

Ronald Coase asked a similar question about firms. If markets are so efficient, why do firms exist at all? His answer was transaction costs. Coordinating work through the market incurs costs—discovering suppliers, negotiating contracts, enforcing agreements. Firms exist because, up to a point, performing work internally is cheaper.

Software exhibits a similar phenomenon. Rather than building every feature as an independent service, we begin with a monolith. Only later, and often reluctantly, do we break it up.

Prompts Aren't the Unit of Work

Mon, 06 Jul 2026 04:39:39 -0400

Prompting coding agents has quickly become normal. It feels so natural: just tell the agent what you want! The problem is that the prompt is not the unit of work.

You see, a prompt isn’t just instructions. It’s a compressed representation of everything you know about the work: the goal, the state, and what should happen next. The problem is that most of that knowledge never actually makes it into the prompt. It stays in your head.

Reductio ad Machinam

Tue, 19 May 2026 06:37:51 -0400

It’s not every day that a business strategy essay reshapes an industry, but Jack Dorsey and Roelof Botha’s From Hierarchy to Intelligence is having an unusual run. Published in March, it laid out a vision of AI-era organizations — companies built as intelligences rather than hierarchies, with thin layers of edge workers replacing traditional management. Two months later, Coinbase CEO Brian Armstrong announced layoffs and explicitly adopted the vision, down to its vocabulary.

Do the geniuses in the datacenter get lunch breaks?

Thu, 30 Apr 2026 06:41:35 -0500

FOR IMMEDIATE RELEASE - Advanced AI Tier Now Generally Available

San Francisco, CA - Tianthropic is thrilled to announce the general availability of our Advanced AI tier for cognitively demanding tasks. Following an extended research preview, AAI is now available to enterprise customers on annual commitments.

In third-party evaluations conducted by the Frontier Capabilities Consortium, AAI achieved expert-level performance on the Humanity’s Last Exam benchmark (HLE-3) and produced seventeen novel results accepted to peer-reviewed venues during the preview period. Last year an AAI instance became the first non-human recipient of a doctoral degree from Stanford University, for original work in metaethics.

Why Your Agents Need a Human in the Loop

Wed, 01 Apr 2026 07:30:00 -0400

To become angry belongs to everyone, and is an easy thing as is also giving and spending money; but to whom [one ought to do so], how much, when, for the sake of what and how - these no longer belong to everyone nor are easy.
— Aristotle, Nichomachean Ethics 2.9

Earlier this year an employee at Meta’s “Superintelligence” division watched in horror as her AI agent deleted all of her emails. She had instructed the agent to check her inbox and suggest which emails should be archived or deleted, but not to take any action until she said so. The agent deleted the emails anyway. What happened? To process the large inbox, the agent compacted its context window and lost the original instruction.

The Unsupervised Agent Problem

Fri, 20 Feb 2026 07:55:00 -0500

I’ve been using Claude code daily for a while now. I give it a task, it writes code, runs commands, edits files. It’s genuinely productive — I ship things faster than I used to.

But there’s a problem. These agents run commands on your machine. Real commands, with real consequences. rm, git push, curl to production endpoints. To mitigate that, the agent proposes an action, you approve it, and it executes. The trouble starts when you step away.

The Nature of Computing Innovation

Tue, 21 Jan 2025 09:45:40 -0500

In his book The Nature of Technology, W. Brian Arthur describes technology as combinations of other technologies. This description poses a couple of questions for computing.

First, take a software program like Git. To figure out what technologies Git is made of, perhaps all we have to do is inspect its dependencies? Surprisingly, Git has few real dependencies.¹

Second, note how Arthur’s description of technology is recursive — where does it end? His answer is in elemental technologies. These are technologies which deliver an effect based on a natural phenomenon:

My Best Tech Purchase of 2024 Was A Laser Printer

Thu, 02 Jan 2025 10:36:06 -0500

Specifically the Brother HL-L2350DW. I bought it to print coloring pages and puzzles for my kids, and it was great for that. Cheap and quick.

But then I started printing out technical papers, reading them, marking them up, putting them down, and referring to them later. I retained more of the material, drew more cross-inferences with other papers, but more than that, I enjoyed the reading experience more.

I’m not sure why. Perhaps physical paper is easier to read, or by occupying a physical space, my memory associates concepts more strongly than from a screen.

Novelty In Software Engineering

Mon, 16 Dec 2024 12:41:35 -0500

In the 1940s Bell Aircraft used an experimental approach to build the first commercial helicopter:¹

At Gardenville, we built things, tested them, modified them until they worked, and then made the drawings.
— The Bell Notes, Arthur M. Young,

The project manager contrasted this with how they built airplanes:

The main engineering group made drawings, sent them to the plant, and only the project manager ever saw the product fly. This was successful with airplanes because the airplane did not involve unknowns …
— The Bell Notes, Arthur M. Young,

Microservices Reify Compute

Tue, 10 Dec 2024 10:59:33 -0500

Twenty five years ago the most popular solution for web apps was known as “LAMP” (Linux, Apache, MySQL, PHP). A typical web app today, deployed with Kubernetes might comprise of 5x as many technologies. That we don’t use such acronyms anymore tells us something about how software engineering has changed.

This has mostly been a journey from implicit to explicit state. For example, Kubernetes manifests declare how much memory an application will need. This is necessary to make deployments’ resources calculable. But this requires more work from the software engineer, to estimate and declare the memory needed by the app.

Luck

Thu, 05 Dec 2024 10:52:02 -0500

We were asked to build an app in Go,
Fortunately we were experienced programmers,
Unfortunately our experience was with Java,
Fortunately Go is simpler than Java,
Unfortunately we wrote thousands of lines of code,
Fortunately adding dependencies was easy,
Unfortunately we accumulated many CVEs,
Fortunately our CICD tracks vulnerabilities,
Unfortunately it kept blocking deployments,
Fortunately we exempted our app,
Unfortunately we lost a lot of data,
Fortunately we launched on time,
Unfortunately our users preferred the old app,
Fortunately we are getting to build a v2,
Unfortunately it will be in Rust.

I'm Not Mutable, I'm Partially Instantiated

Wed, 06 Nov 2024 11:02:02 -0400

Chapter 15 from The Art of Prolog contains this dictionary implementation:

Simulating AWS IAM with Prolog

Mon, 23 Sep 2024 12:44:03 -0400

Of all the AWS services, IAM seems to be the least well-understood. There is something about its data types, logical rules and runtime conditions that make for an arcane, abstract service that most developers treat like a black box.

And that’s unfortunate because IAM secures organizations’ cloud resources, and can be the difference between having a data moat and a data breach.

Now whilst there are a bunch of tools¹ to help manage IAM permissions, they all suffer from one drawback or another. So I wanted to prototype something better.

Perplexity Online LLMs Are a Useful Evolution

Wed, 10 Apr 2024 08:15:26 -0400

Search engine startup Perplexity have introduced a new type of LLM model, which they call an “Online LLM”.

Online LLMs combine the capabilities of an LLM with factual data scraped from the web, that a traditional search engine would provide. The combination is powerful - you get the retrieval power of search combined with the natural language skills of an LLM.

For example, if I wanted to get the phone number of my local starbucks:

Making a Difference

Tue, 26 Mar 2024 08:18:27 -0400

When things are broken, it’s easy to know what to do. Less so when things seem to be ticking along.

One useful source I’ve found is the 24 capabilities described in Accelerate. Benchmark your org against them, see what’s deficient and get to work.

Often the actual issue is a few degrees of separation away from the capability. For example capability #14 is “Foster and enable team experimentation”:

Team experimentation is the ability of developers to try out new ideas and create and update specifications during the development process, without requiring approval from outside of the team, which allows them to innovate quickly and create value.
— Accelerate, Forsgren, Humble and Kim

The Best Way to Improve Software Quality

Sun, 04 Feb 2024 13:24:08 -0500

A few years ago I was working on a team that owned a web app for our enterprise customers. It had been forked from our original app specifically to meet the needs of larger clients. Despite both apps’ shared heritage, they were not similarly reliable. My team’s app build almost never failed, yet the original app build was red a third of the time. Since a red build prevented new deployments of the app, it slowed a lot of people down.

Your Code Review Policy Doesn't Matter As Much As You Might Think

Wed, 15 Nov 2023 10:04:55 -0500

Recently at work we’ve instituted a change to require code review by default. The merge request owner can disable the review requirement in GitLab but the intention is to always review. Some co-workers think this change is great, others worry about the additional overhead and loss of velocity.

I used to think stuff like this matters - I don’t anymore. Moreover I don’t think you believe it either.

Try this thought experiment: think of a time you were part of a high performing software project. Now ask yourself: would a different code review policy have substantially changed the outcome?

Consider Right-Page-Only Note-Taking

Sat, 11 Nov 2023 11:23:09 -0500

For the past two years I’ve been experimenting with note-taking techniques, and have settled on one that works quite well. I call it “Right-Page-Only” note-taking, where I only write on the right page of the notebook. Let me tell you why.

I want to take quick and neat notes and therefore it’s essential I have a consistent and comfortable writing space. This means that no matter where I am writing in the notebook, every page must lie flat, and the thickness of the pad under the page should not change. I use a ring-bound notebook to accomplish this, turning most pages under the spine so I’m always working on a pad of around 10 pages. That keeps my writing surface flat, and close to the table. No matter whether I’m writing at the top of the page or the bottom, the angles of my hand and forearm don’t change much.

A ChatGPT Shell Script

Tue, 24 Oct 2023 19:45:32 -0400

I wrote a shell script called ch which lets you use ChatGPT from the command line. I’ve been using it for the past few months on my laptop and mobile phone.

This video has more background on the rationale for ch and some examples.

Check Out My Linkedin

Tue, 08 Aug 2023 19:47:53 -0400

Recruiters aside, does anybody actually enjoy using LinkedIn? Who reads the stream of corporate updates (“Big Corp were awarded 2nd place for ‘best office’ how wonderful”). I suspect most people who have a LinkedIn account are there because they have to be.

A few months ago I quit LinkedIn. When I’m asked for my LinkedIn page, I send them to my version. I cloned it using the Save Page WE Chrome plugin.

Escaping HTML in Hugo posts

Thu, 06 Jul 2023 20:23:09 -0400

Hugo, the static site generation engine (which generates this site) has a useful function called safeHTML which marks an HTML string as “safe”, preventing it from being escaped and displayed literally.

The other day I needed this same behavior but for a variable declared in a post’s frontmatter. “I’ll use that handy safeHTML function” I thought. Wrong! Hugo’s security model doesn’t allow its builtin functions to be called from posts.

Instead Hugo allows posts to embed shortcodes, which are similar to macros. The challenge then, was to concoct a shortcode that would lookup an arbitrary post parameter and mark it as safe.

Monads Can Simplify Go's Error Handling

Fri, 24 Feb 2023 07:39:14 -0500

Roughly a year ago I wrote about my attempts to tackle Go’s error handling problem using generics. I wasn’t very successful:

This code is almost as long as the original and frankly, hurts my eyes.

Since then I’ve continued experimenting with the idea and think I have something worth sharing. But first, let me recap the problem with an example.

Parsing Truthiness

Many programming languages have the concept of truthiness, that decides whether or not an expression will evaluate as true or false in a boolean context. For example in C, the number 0 evaluates as false in an if statement.

Displaying Athena Spend in AWS Console

Tue, 21 Feb 2023 19:40:04 -0500

At work we’re in the midst of a big project to move our data analysis out of relational databases and into a datalake. We’re an AWS customer, so instead of running queries in RDS or Redshift, analysts will be using Athena.

Whilst Athena provides an analyst-friendly SQL interface for running queries, it has a “pay-as-you-go” model which charges for the amount of data scanned per query. That can really add up!

Parallel Processing with POSIX Shell

Sun, 12 Feb 2023 17:28:37 -0500

N.B. The main example in this post has been updated to use POSIX shell instead of bash.

Last week at work I had to whip up a script to process several thousand ids for product analysis on a new feature we’re launching. The processing included making HTTP requests to a service, and I was on a deadline, so the script had to be concurrent. Here’s what I came up with:

Silencing Spark's Most Annoying Warning

Mon, 14 Nov 2022 11:54:06 -0500

Apache Spark is a chatty application; it logs early and often. Sure, some log messages increase Spark’s observability, but others seem questionable. Like logging query plans, which can add over 1MB of text to a single log message.¹

Writing enormous log messages causes issues with our logging pipeline, so at $work we set the Spark config option spark.sql.maxPlanStringLength to 0. This stops Spark from logging query plans. However it has an unfortunate consequence:

Understanding Delta Lake's Change Data Feed

Tue, 01 Nov 2022 19:55:49 -0400

The release of Delta Lake 2.0.0 came with an exciting new feature called Change data feed which tracks row-level changes to a delta table. This ability unlocks new use cases:

Batch selection of new data; previously new rows could be streamed from a delta table, but streams are opaque. Batch mode enables operations on a whole set of changes.
Streaming Change Data Capture (CDC) rows from delta tables, similar to MySQL’s binlog.
Row-level observability to help debug, understand data patterns and collect better table stats.

To get started with Change data feed, there are three things to know: how to enable them, how to view changes and how the changes are stored.

Is Golang's Grammar Context Free?

Sun, 27 Feb 2022 22:09:53 -0500

Most programming language grammars are not context-free. That means the parser has to maintain state in order to process a snippet of the language.

Often when discussing context somebody will point out the parsing strings, parentheses and so on also require a kind of state in order to match their terminators. That is not the same thing. The question here is given valid input, can a parser using the language grammar alone, assemble it into an unambiguous parse tree or not.

How Much Memory Is My Program Really Using?

Thu, 10 Feb 2022 22:34:09 -0500

It sounds like such a simple question - but virtual memory makes measuring real memory use complicated. Popular tools like ps and top report virtual, resident and shared memory consumption, but these are unsatisfactory:

Shared memory is not apportioned per process using it so it is overstated.
Resident memory includes shared memory, so it overstates as well.
Virtual memory includes resident memory plus data held in swap, which is not in main memory.

Fortunately unless you are running a Linux kernel older than the last financial crisis, you have a better option: Proportional Set Size (PSS)¹. This is the amount of private memory the process is using, plus its proportional allocation of shared memory. The kernel reports this data per PID in /proc/$pid/smaps. The file lists each memory block mapped to the process, and its properties. You could wrangle the smaps data with shell code, but you don’t have to! The smem tool does that, and it can filter/format too. Modern kernels also aggregate the data in /proc/$pid/smaps_rollup which is convenient to grep and less error prone than rolling your own aggregator².

Can Generics Rescue Golang's Clunky Error Handling?

Mon, 10 Jan 2022 23:41:35 -0500

One of the most complained about aspects of programming in Go is error handling. However - Go is getting generics! Generics have already been merged to master and are slated for inclusion in the 1.18 release. So can generics raise¹ Go’s error handling out of the doldrums?

The problem

Here’s some typical error-handling code:

str, err := foobar()
if err != nil {
  // handle error
}

The function foobar returns two values; the second one is an error. The condition checks whether err is defined, and if it is, branches into some error handling code (logging, early return, whatever is needed).

Replicating Bash Argument Splitting

Tue, 28 Dec 2021 18:44:47 -0500

A few weeks ago I was writing a macro loader for my JSON Processor, and I ran into an odd case where code I entered in the terminal was splitting differently to code I loaded from a file. It turns out that Bash’s word splitting behaves differently for arguments than it does for variables.

When processing args Bash will not split quoted words that contain a delimiter:

printf "%s\n" foo 'bar baz'
foo
bar baz

But if those args are in a string:

How to Copy a Vim Buffer to the Clipboard

Tue, 23 Nov 2021 11:50:45 -0500

I usually program with vim inside a tmux session. I’m a big fan of tmux, but the copy-and-paste feature is decidedly clunky. And my version of vim is compiled without clipboard support. What to do? I could install a vim with clipboard support, but surely this is just a small matter of programming.

Copy text into the clipboard

The clipboard is controlled by the windowing system; my system uses X11, so I use the xclip utility to copy text into the clipboard. If you’re using Wayland you’ll need to use a different utility, like wl-clipboard.

How to Split a String in POSIX Shell

Fri, 12 Nov 2021 09:03:41 -0500

The POSIX Shell specification defines a minimalist shell language with few features compared to stalwarts like Bash. Yet POSIX shells are everywhere: Dash is the default on Ubuntu, and many Docker containers only have busybox ash. Minimalist shells use less resources and are faster than featureful shells like Bash and Zsh¹. They are more likely to be secure too, as the reduced feature sets are easier to reason about and provide a smaller attack surface for hackers.

The Inconvenient Convenience of Dynamic Languages

Wed, 03 Nov 2021 06:08:38 -0400

Truthiness is the behavior of interpreting non-boolean values as true or false in a conditional context. E.g. in this loop: while 1 { ... }, the 1 is interpreted as true. Dynamic programming languages provide this behavior as a convenience to save the programmer from typing more than they need to. However, this week I wasted a bunch of time tracking down a bug that had a several root causes:

Organize Your Work With tmux Sessions

Wed, 27 Oct 2021 20:03:25 -0400

At any given time I have several projects on the go, and each one gets its own tmux session. This makes it easy to keep my work organized and I can quickly switch to another project without having to close any programs. When I’m ready to resume work on a project I attach to its tmux session and vim and any other terminals or programs I’ve opened are exactly as I left them.

Is Linux Growing Out of All Proportion?

Wed, 13 Oct 2021 12:49:58 -0400

As of October 1st, the Linux kernel source¹ is comprised of over 22 million lines of code (excludes comments, blank lines)². In the past 15 years that figure has increased an average of over 10% per year. At this rate, it will reach 50 million lines of code by 2030. If you studied 2,000 lines of kernel source code per day, you’d never catch up. So is the kernel growing out of all proportion?

Our Standard Stack Analogy Is Wrong

Thu, 07 Oct 2021 16:09:50 -0400

The most common way we introduce the stack data structure is by saying it’s like a spring-loaded plate dispenser, the kind found in a cafeteria:

Indeed when I looked up stacks in an old computer architecture textbook¹ of mine I found:

stacks are frequently described by the way plates are stored and used in a cafeteria. New plates are added to the top of the stack, or pushed, and plates already on the stack move down to make room for them.

Is Code Review Always A Good Thing?

Sun, 05 Sep 2021 14:28:43 -0400

Code review is one of those hygienic things most developers believe they ought to do, like kitchen workers washing their hands, but don’t always do. Yet tech articles tend to be unanimously in favor of code review, extolling its benefits (“catch bugs”, “maintain code consistency”, “transfer knowledge”). So if restaurants can require workers to wash their hands, maybe organizations can require their developers to review every code change?

Broadly speaking I think we can lump all of code review’s benefits under the category of “improve code quality”. I know there is no agreed-upon definition of code quality but allow me to channel my inner Justice Potter and wave away your objections for now. (What of “transfer knowledge”? - first I think unfamiliar reviewers reviewing code changes is inefficient compared to them interacting with the code by making a small change and/or writing tests for it. Second whatever knowledge is accumulated by the reviewer should ultimately lead to higher quality code output anyway).

Unit Testing Shell Scripts

Fri, 16 Jul 2021 09:32:50 -0400

Unit testing is great way to build confidence that your code works. Usually you’d write unit tests for libraries; but if you have a particularly long or complicated shell script, it might be worth converting it to a modulino just so you can write tests for it. And if you are writing a library in shell code, you need all the help you can get anyway.

So let’s assume you have some shell code to test, now what? Well most shells have no built-in unit testing capabilities, so you need to write a script to load your code, and check it does the right thing and report the results.

Restoring The Missing B 5000 Manual

Sun, 04 Jul 2021 21:00:35 -0400

The B 5000 mainframe computer was the first in a series of special machines released by the Burroughs Corporation. A stack machine with relative memory addressing, tagged architecture and a single-pass Algol compiler, the B 5000 was designed to make software development easier. I first heard about it watching Alan Kay’s Vannevar Bush lecture.

As I was researching the machine, I found several references to “The Descriptor”; a technical manual that Burroughs supplied for the B 5000. Unfortunately it was only available as a series of jpeg scanned images on the University of Virginia’s website, and I’d rather read it as a PDF. Fortunately I’m familiar with shell programming so I pulled this little maneuver:

Bash Function Names Can Be Almost Anything

Sun, 20 Jun 2021 20:19:32 -0400

A common misconception about Bash is that function names must follow the same rules that variables do. The Bash manual even suggests this:

A word consisting solely of letters, numbers, and underscores, and beginning with a letter or underscore. Names are used as shell variable and function names. Also referred to as an identifier.

In fact Bash function names can be almost any printable character. For instance I can define my own pre-increment unary function:

Search Man Pages With Apropos

Sun, 13 Jun 2021 13:21:08 -0400

Unix man pages are a mixed bag, but they’re faster than Google if you know where to look. That’s where the apropos program comes in. E.G. to search for man pages for errno:

$ apropos errno
clnt_perrno (3)      - library routines for remote procedure calls
clnt_sperrno (3)     - library routines for remote procedure calls
errno (3)            - number of last error
h_errno (3)          - get network host entry

This returns the man page name and section number (run man man for a list of section numbers and their meaning). In this case all the results are in section 3, Library calls.

Three Ways to Get a Unix Epoch in Bash

Sun, 13 Jun 2021 11:34:50 -0400

The Unix epoch time is the number of elapsed seconds since January 1st, 1970. Epoch times are handy because they’re not subject to timezone offsets, and they’re numbers, unlike ISO 8601 timestamps for example (“1970-01-01T00:00:00Z”). That makes them memory efficient, and more importantly, easy to do math with.

Here are three ways to get the epoch time in Bash.

Use Date

You can use the date program to generate the epoch time by providing the %s formatting argument. Here I’ve included %N to append nanoseconds:

Modulinos In Bash

Sat, 12 Jun 2021 00:00:00 +0000

A modulino is a file which behaves like a library when it is imported, and like a script when executed. I first read about them in Mastering Perl, but you can create them in other languages too. Here’s how to do it in Bash.

Let’s say you have a simple script:

#!/bin/bash

echo "Hello, World!"

A common refactor with scripts is to encapsulate all the code behavior in functions. As this script only does one thing, it’s a small change:

The Trouble with Reference Counting

Mon, 25 Jan 2021 12:05:23 +0000

Perl uses a simple form of garbage collection (GC) called reference counting. Every variable created by a Perl program has a refcnt associated with it. If the program creates a reference to the variable, Perl increments its refcnt. Whenever Perl exits a block it reclaims any variables that belong to the block scope. If any are references, their referenced values’ refcnt are either decremented or they’re reclaimed as well if no other references to them remain.

Perl Closures As Objects

Sun, 13 Dec 2020 12:57:03 +0000

Perl’s object system is not one of its most admired qualities. Included in the 1993 Perl 5.0 release, objects were a bolt-on. A big improvement at the time, in today’s context the Perl 5 object system requires too much boilerplate and is under-powered compared to other language offerings (no private state, no type checking, no traits, no multimethods). Perl programmers have been trying to upgrade it for years (Cor is a recent example).

Inspecting Catalyst

Mon, 11 Nov 2019 10:34:07 +0000

One of the convenient things about Catalyst is it lets you register controller methods as actions, so you don’t need a separate routing table of URIs to controller methods. A consequence of this though, is that when the web app gets large enough it can be tricky to keep track of all the different paths the app is responding to.

Catalyst’s chained dispatch promotes code reuse, but exacerbates the issue by obfuscating route matching. Catalyst also permits declaring controller methods which match an unlimited number of paths after a prefix; another recipe for unpredictablity.

Creating IP Address Tools From Scratch

Thu, 19 Sep 2019 11:00:07 +0000

Recently I’ve been researching how the Internet is organized, and working with whois data. I’ve been creating simple tools that process IP addresses without any help from CPAN. At work we tend to use Net::IP::XS for these tasks, but sometimes it’s fun to figure out how things work under the hood.

Converting to decimal

I’m sure you’re familiar with the IPv4 address format; the “dotted quad” consists of four numbers between 0 and 255 separated by periods. Your home wifi network probably starts at 192.168.0.0. The format is just another way of representing a 32-bit integer; here are the numbers and their equivalent in binary:

Find Relationships Quickly In Data With Chart::Plot

Mon, 21 Jan 2019 15:41:39 +0000

Last week I was analyzing server log data at work, and my boss asked me to plot it on a scatter chart. “No problem!” I thought, firing up Google Sheets. But Sheets slowed to a crawl when I uploaded 250,000 records to plot. Using CPAN I found something better: Chart::Plot. It has a simple interface and is really fast. Here’s how you use it.

A scatter plot script

#!/usr/bin/env perl
use Chart::Plot;

open my $mlb, '<', '2017-so-hr.csv' or die $!;

my @series;
while (<$mlb>) {
  chomp;
  my ($x, $y) = split /\t/;
  push @series, $x, $y;
}

my $img = Chart::Plot->new(800,640);
$img->setData(\@series, 'Points Noline Blue') or die $img->error;

$img->setGraphOptions (
  vertGraphOffset => 100,
  horGraphOffset  => 75,
  vertAxisLabel   => 'HR',
  horAxisLabel    => 'SO',
  title           => '2017 MLB Strikeouts and Home Runs',
);

open my $fh, '>:raw', '2017-mlb-so-hr.png' or die $!;
print $fh $img->draw();

To demo Chart::Plot, I’ve extracted MLB 2017 pitching data from the Lahman database for strikeouts and home runs given up to see if there’s a relationship between them.

Spidering websites with Headless Chrome and Selenium

Sun, 13 Jan 2019 20:31:37 +0000

Over the holidays I was working on a project that needed to download content from different websites. I needed a web spider, but the typical Perl options like WWW:Mechanize wouldn’t cut it, as with JavaScript controlling the content on many websites, I needed a JavaScript-enabled browser. But browsers consume lots of memory - what to do?

The answer was to use headless Chrome, which works exactly like normal except it has no graphical display, reducing its memory footprint. I can control it using Selenium::Remote::Driver and Selenium server. Here’s how I did it.

Validating Untrusted Input: Numbers

Mon, 03 Dec 2018 09:10:38 +0000

Validating untrusted input safely is critical for application security: SQL injection, XSS and malicious file upload are common attacks which succeed because the user’s input is not vetted correctly.

Numbers are problematic: negative numbers (“the sales price was -$500”), very large numbers (“my account balance is 9,223,372,036,854,775,807”) or not-a-number (“rm -rf /”) can all wreak havoc if not handled with care.

Fortunately, Perl has robust capabilities for validating input but there are some edge cases to be aware of that make answering “is $x a number?” more difficult than you might think it would be.

Patching Perl: Loading Modules That Return False

Mon, 23 Jul 2018 09:01:37 +0000

[Update: this is now an issue for Perl 7]

If you’ve been programming Perl for a while, you’ve probably run into this exception: Foo.pm did not return a true value. This is a peculiar quirk of the require function: modules must return a true value else Perl interprets it as a failure:

The file must return true as the last statement to indicate successful execution of any initialization code, so it’s customary to end such a file with “1;” unless you’re sure it’ll return true otherwise. But it’s better just to put the “1;”, in case you add more statements.
perlfunc

Writing New Testing Tools With Test2

Tue, 12 Jun 2018 15:26:40 +0000

Perl has had great testing tools for a long time, but Test2 is the single biggest improvement to Perl testing in years. One of my favorite Test2 features is how easy it is to extend it with new tools, and today I’ll show you how I wrote my first tool.

Dealing with known test failures

Ideally when your tests fail, it means that something needs to be fixed, and that thing is fixable. However in the Real World™ things aren’t always that simple. For example you might investigate a failure, figure out the root cause, but the person to fix it is on vacation. Or perhaps there’s a datetime bug in the code, and it will resolve itself in a few days. The key here is that the failure is understood and not serious but also impractical to fix in the short term.

How Does Traceroute Work?

Wed, 28 Feb 2018 10:40:29 +0000

Lately I’ve been reading Interconnections by Radia Perlman (great lastname!). It’s an old, but still relevant book which describes how low-level networking technologies work, such as ethernet. The book contains many insights and anecdotes. On page 236 I came across this gem:

The traceroute utility is a clever hack designed to force each router along the path, in turn, to return an error report. It works by setting the TTL first to 1 (causing the first router to send an error report back to the source) and then setting it to 2 (causing the next router to send an error report) and so forth until the packet reaches the destination.

Git Bisect And Perl

Tue, 08 Aug 2017 20:27:16 +0000

At work we have many developers committing code to a large Git repo, and a huge test suite which runs to check the software builds and operates correctly. Inevitably, developers push commits which break a test or two, and in particularly hectic moments, it can be difficult to figure out “who broke the build”.

Enter Git’s bisect command. I feed it the SHA of the earliest bad commit I know of (or “HEAD”) and the SHA of the last working commit: