Microsoft finally ?’s open source: A chat with npm’s founder

For Microsoft, 2015 was a year that’ll go down in the history books — and not because it released a new version of Windows or Office, but because the company fundamentally shifted in a big way by embracing open source.

John Shewchuk, a technical fellow at Microsoft whose job it is to bring the community closer to the company, started a video podcast called Decoded, which offers regular deep insight into developer focused topics with interesting people from the industry.

This week’s episode digs into Node Package Manager (npm), one of the most popular package management systems out there, with the company’s founder Isaac Schlueter.

I sat down and talked to both Shewchuk and Schlueter, who had a lot to say about open source, and where we’re headed.

The 💜 of EU tech

The latest rumblings from the EU tech scene, a story from our wise ol' founder Boris, and some questionable AI art. It's free, every week, in your inbox. Sign up now!

TNW: I’m sure you get asked this a lot, Isaac, but how’d you come up with the idea for npm?

Isaac: I was at Yahoo, for four years, but really started to get frustrated with two disconnected languages on the frontend and backend. Ryan Dahl showed up with Node and I was like ‘wow, this is the one to get behind’ — it felt very ‘Javascripty’ and unix-like, which really appealed.

In the early days of the Node community, people would share what they wrote and talk about ideas, best practices and more on a mailing list. I started writing a script that would do this by hand and worked with others to come up with the package.json format.

When it was ready, I kept sending pull requests to people to add it to their repos. At the time it was really easy to get people to add one more file to their repos to try it out, these days it’s a lot harder.

Eventually I quit my job to finish this thing and at the start of 2010 we had a version that kind of worked.  By the time the first Nodeconf came around in 2011, we already had 2000 packages on npm. By the time the second came around it was more like 12,000.

TNW: Can you tell us more about what a technical fellow does at Microsoft, John? 

John: It’s very different today than it was a few years ago. Satya Nadella and Steve Ballmer realized that they needed to engage with the developer community. The DPE team at the time was in sales mode, pitching Microsoft’s technology to people and just… selling.

The cloud had vastly accelerated development cycles, so we had a thesis: instead of selling, we had to go out and partner with people to get in conversations with them.

We started Decoded to tell the stories of leaders in the field directly and doing it in partnership in them. We really made the transition into open source, which is incredibly fun. Instead of just selling Azure to them, we sit down and talk about what’s cool and what’s happening in the wider technology industry.

TNW: When did you realize npm was going to get big? Was there a moment where you thought it could be used millions of times every month?

Isaac: At the end of 2013 I thought “oh crap, this is a very serious thing” when the npm infrastructure was running on top of donated infrastructure and a large part of the devops/management work was being done by myself.

The folks who donated the equipment helped us out but couldn’t keep up and said we had to pay or take npm somewhere else. We had a lot of outages and just outgrew that technical approach.

At that point we founded npm Inc., hired a team of people to do operations and raised a couple of million to build for-profit services. The first of those was private packages in the npm registry, then we launchd on-site npm registries for enterprises in late 2015. Now we’re refining and optimizing those.

TNW: What was your biggest mistake building npm?

Isaac: That’s a big question. There’s a lot of technology decisions that were wrong, but hindsight is 20/20. If you’re too risk averse and don’t make any mistakes, you probably aren’t doing anything interesting.

Personally, it was probably running the infrastructure myself while trying to get Node in the right place at the same time. Node’s in a foundation now, but at the time we hadn’t decided what was going to happen.

For npm, our biggest mistake was probably peer dependencies. At the time it seemed like a great idea, but we still regret it. There’s no way we could’ve seen it turning out this way, but it opens the door for dependency hell, leaving it to the user to solve it.

TNW: Microsoft’s changed a lot since you joined in 1993, John, particularly its attitude toward open source. How did that even come about?

John: For me, that journey wasn’t as long as it was for other people. I was used to open source and worked on Project Athena at MIT, so the environment where people shared protocols and source code was familiar.

At Microsoft, it was the opposite and it wasn’t until Satya started moving into a leadership role that the shift really kicked off — it took at least five years. When I joined, the default was that everything should be proprietary. It took a significant approval process to work on open source projects, so I went to the lawyers and business leaders to change the default.

The hardest part was probably the legal team needed to develop the expertise for open source so they could understand it. Eventually, our default way of working with a partner became checking into a open source project and proceeding from there.

TNW: What’s the most important part of your role as steward of npm?

Isaac: I don’t make any tech decisions or write code right now, but instead I have two constituencies: looking after the staff/investors and the community. In the long term that means keeping npm funded and running forever.

That could change one day, but we want to make sure we’re working together and building the right products — and create value for the company at the same time.

What’s most near to my heart is bringing open source’s way of doing things to companies, like Microsoft. Building modular code, in parallel with a package manager makes sense inside a larger company, but many don’t know how to get there.

TNW: What does the future of npm look like?

Isaac: Well, we’ve thought about desktop tooling but we’re not sure it’s the best place for us to iterate. The future of npm is bringing the good parts of open source into companies, because that’s what they’ll pay for.

We’re a small company, so we want to make sure we do right by delivering that. A term for this is ‘intrasource.’ That’s working to deliver open source software and ideas publicly, so they can be collaborated on.

Microsoft is a great validation of intrasource as a concept, where it takes off is when they approach their own development like that: loose teams, highly modular projects. Thanks to GitHub, there’s a whole generation of developers that get jobs and expect that in their workplaces.

You can check out the latest Decoded episode here right now, which features an interview between John Shewchuk and Isaac Schlueter, as well as hands on with npm and coding examples.