Jay Hoffmann

Books, movies, and code


That Time the Internet Broke

RE: The Plight of NPM, etc.

I won’t pretend to be an expert on NPM, package managers or open source. But last week, something really interesting happened. And it brought into view two issues that have been swirling around in the ether: the dependency tangle that is the Node / Javascript community and the problems that arise from private vs open source projects.

Here’s what happened. Azer Koçulu announced that he was removing all of his publicly available modules from the NPM registry after his module Kik was removed, against his wishes, by the NPM team. See, NPM is a private company, owned in part by Isaac Z. Schlueter. And when lawyers from the app Kik came knocking, Isaac was quick to comply over Azer’s head.

In Azer’s words:

This situation made me realize that NPM is someone’s private land where corporate is more powerful than the people, and I do open source because,Power To The People.

So Azer goes and removes his 273 modules from NPM, including some pretty global sounding ones like “map” and “attr” and, most importantly, “left-pad”. And just like that, a giant section of the Internet just flat out breaks. Why? Because thousands and thousands of developers had one of these libraries as a dependency for their app. Or maybe one of their dependencies had one of these dependencies. Or maybe one of their dependencies’ dependencies had one of these libraries as… well, you get the point. As The Register puts it:

Unfortunately, one of those dependencies was left-pad. The code is below. It pads out the lefthand-side of strings with zeroes or spaces. And thousands of projects including Node and Babel relied on it.

In the Javascript community, we’ve taken the concept of micro-packages to an unreal extreme, to the point where our careful web of dependencies can unravel at any moment. There are no standard libraries or robust toolkits, everything’s a tiny dependency bundled alongside a hundred others. So when one thing breaks, everything breaks. We like to think that we’ve created this neat, decoupled architecture but all we’ve proven is that things rely on external libraries that much more. And it’s way too easy for things to break.

I think David Haney probably had the best perspective on this whole problem:

On what possible plane of existence is this a better solution to past problems? How are hundreds of dependencies and 28,000 files for a blank project template anything but overly complicated and insane?

Or put simply:

Every package that you use adds yet another dependency to your project. Dependencies, by their very name, are things you need in order for your code to function. The more dependencies you take on, the more points of failure you have.

Honestly, Haney’s viewpoint on this whole thing is the most practical I’ve read so I’d recommend giving it a read.

And then there’s the whole issue of open source libraries being hosted and distributed on a closed-source, private package manager. Because  rather then take a second to see why this might be a problem, the NPM team simply reacted by removing the ability to unpublish packages altogether. Which does not sound very open-source friendly to me. Legally, it’s well within NPM’s rights to do what they did, and I actually think they made the right call with the situation they had, but it points to a bigger issue. There’s really no truly public place for open source. Without this, we rely on the wills and whims of individuals to manage the needs of entire communities.

From Dave Winer:

I worry about GitHub. It plays such a central role. But eventually the VCs are going to want an exit. Then what happens?… We need a framework, legal and social, for projects that are not “owned” but are just there.

This whole thing might seem ridiculous, and it is. But it might spark some interesting conversation. For now, it’s worth keeping an eye on. If you have some time to kill, there’s an interesting conversation about it over on Hacker News as well.