Lessons learned: 100 interviews about numerical JavaScript

We ran a hundred customer-discovery interviews in seven weeks earlier this year, reaching out to anyone we could think of who might have something to say about high-performance numerical and scientific computation in JavaScript (which is the thing stdlib exists to do), as well as those with knowledge about sustaining open-source ecosystems. We also searched the Journal of Open Source Software (JOSS) for work involving JavaScript—a remarkably short list—and wrote to the people behind the papers.

So, the sample is what two people's address books could reach, plus what one literature search could find. Our networks largely held people we already knew, which shaped the corpus.

Projects overestimate how well they know their users. It's in the corpus over and over—maintainers confident about what users need, users describing something adjacent but not the same when you ask them directly with no maintainer in the room. Which is not a maintainer failing, or a developer failing. Expertise and certainty aren't correlated the way we'd like them to be, and Tavris and Aronson have the bleak version of it in Mistakes Were Made (But Not by Me): "training does not increase accuracy; it increases people's confidence in their accuracy."[1] That's also how a wrong assumption persists—nothing in the loop is set up to catch it, so it just keeps being the thing everyone knows. We ran the interviews, found the pattern, and still catch ourselves in it.

What we did, and what the numbers can't say

The interviews were the "customer-discovery requirement" of the NSF I-Corps program, which stdlib went through as part of its POSE award. The count (100) and the timeline (seven weeks) came with it. Two of us doing them.

Recruiting a hundred people that fast doesn't leave much room for a sampling strategy, and two of our three channels were just us. My side of that reach was R and data visualization; Athan's was scientific Python and numerical JavaScript.

The third channel wasn't a network. To find academics actually using JavaScript in a lab, we used a mix of GitHub repo surveying and