I am the CTO of Joyent, the father of DTrace and an OS kernel developer for 20 years. AMA!

Someone asked:

What's the most beautiful piece of code you've seen over the years. (Code you wrote your self excluded)

Unfortunately they forgot a question mark, so the comment was nuked, but here's my reply:

Oh, exciting question -- and one that I reserve the right to answer more than once! ;) One piece of code that I absolutely love is this little bit of auto-scaling in the kernel memory allocator. It's very tight and elegant -- a really clever technique that I haven't seen used nearly enough. This is written by Jeff Bonwick, who later went on to (along with Matt Ahrens) invent ZFS. Jeff was my mentor at Sun, and a personal inspiration -- and I think that that little snippet of code embodies Bonwick.

Why so many "known names" left Joyent last year? like Brendan Gregg, Jason Hoffman, Deirdré, the pony tail guy..

There are a bunch of different answers (one for each person, really), and I really don't want to speak for people (and I obviously need to respect their confidentiality). I will say that when you are as idiosyncratic as we are (our own operating system, our own orchestration software, our own storage substrate, our own runtime -- even if all open source), it takes an emotional toll, and I think it becomes tempting to find a path that feels less like taking on the universe. Of course, we at Joyent did this by implementing LX branded zones (a project that has actually been one of the most invigorating of my career), but I think for others, it was just all too much and other paths became tempting. I don't fault them (at all) for it; I feel fortunate to have worked with them, and look forward to our paths crossing in the future.

Does that question your belief that Joyent's way of creating technology is the correct one?

No, it doesn't. I believe this deep, deep in my marrow -- and that belief is reaffirmed every day. But I have grown much more understanding as to the emotional exhaustion that such deep beliefs can impart -- and it's important to me that people can feel like they can leave without feeling like they have been defeated. This is a challenge, and it's one that I think that many companies that are taking big technological swings face -- and (believe it or not) it was actually a factor for me in advocating open sourcing the stack: give people a way of taking a break without feeling like they need to abandon their work.

As a sysadmin i have to say i don't like docker, sorry. but! i might be wrong. Could you shed some light and say something about docker pros for sysadmins instead of talking only about developers?

I think that Docker is going to be as big a win for operators because it allows us to get the software that we run in production looking more and more like what the developer is running in development. We're not there yet, of course, but I think that an all-container world is one that benefits the operator in terms of efficiency, deployability, portability, etc. More thoughts on this are in my talk on the future of Docker in production.

Would you rather fight one horse-sized Sun Microsystems, or a hundred duck-sized Oracles?

I think I would rather fight one horse-sized Sun Microsystems than one duck-sized Oracle -- let alone a hundred. Never underestimate the depravity of a sociopathic duck!

Just wondering, what happens when you anthropomorphize Larry Ellison?

More seriously, what was it like working on the Bonwick team, and have you accomplished a similar environment at Joyent?

Bonwick had an enormous influence on me in that he got me to change my expectations of myself: prior to working at Sun, I viewed defects as inevitable and that software was like much else that we engineered (namely, fated to be plagued by unknowable and transient errors). But in working with Jeff -- who came to software from math, not from engineering -- I came to appreciate that the better analogue for software is a mathematical proof, and that the beauty in software is that we fashion a proof that functions as a machine. As a result, we (and and we in software alone) can create machines that are perfect. This was tremendously inspiring to me, and has had a profound effect on my career and in the pleasure I take in the work.

In terms of accomplishing the same at Joyent, I certainly aspire to inspire software engineers in the same way. I think sometimes I succeed and sometimes I fail -- but I think the successes have outnumbered the failures, and I continue to be inspired by the team that we have built here.

I took the skipping college path through sysadmin/developer land and am only now working on a CS degree (so I already directly see the value in this answer and your comments about actually getting a degree first instead of going my route). That said, my discrete math professor will love your answer.

Good for you! I think it's terrific that you're back in school. I know that that's a very hard route (especially if you have a family!), but know that you're making the right decision. Indeed, as they say, "education is wasted on the young", and your practical experience is likely leading to a bunch better educational experience for you!

Have you ever kissed a girl? (AKA: Which Cantrillism has gotten you into the most trouble at home or at work?)

Ha! Well, I can't say that I didn't see that one coming. For those who weren't reading Usenet twenty years ago, this question is a reference to an infamous response of mine to Linux kernel developer David Miller. (My response was actually an obscure Saturday Night Live reference.) Let me be clear that I (obviously?) regret it: I was young and it was stupid (and it was Usenet post-Eternal September; youth and stupidity were not in short supply). And in a lifetime of Cantrillisms, I hope that that one forever remains the one I most regret!

Do you think ARM as a platform will ever get the traction for large scale Cloud deployments?

If yes - why or if no - why?

So, I personally think it will be a very long time before we see ARM with traction in the cloud, simply because their emphasis (historically, anyway) has been on power rather than absolute performance. In the cloud, my (our) power bill isn't nearly as important as performance -- and if you give me a fire-breathing CPU, I can often make more with higher tenancy (especially with OS-based virtualization) than I can save power by running ARM. Of course, Intel has taken their eye of the ball in the past, and things could change. In particular, a major weakness of x86 is that it (still!) must relive the entire history of x86 on every reboot -- and it's (still!) hampered by a bunch of close-source firmware. If ARM has an opening, it's in making an open and simpler system -- but they cannot dismiss the importance of having world-beating performance.

What was your impression about the Itanium series? It seems to have been shoved into a dirty corner nobody wants to talk about.

Oh, I'm happy to talk about it! In fact, allow me to tell a story: as a young engineer, I attended a talk at Stanford, by John Crawford, architect of the Itanium (then known as Merced). He was presenting a bunch of benchmarks that seemed... dubious. And while Merced kicked butt on Eight Queens, it was terrible on gcc (and everything else that looked vaguely real-world). Even as brash as I might have been (ahem), I wasn't about to say anything, but then some guy in the back of the room just got on Crawford on this. Crawford tried to shake him off, but the guy was just a badger, at one point saying "Doesn't this just willfully ignore everything we've learned about commercial workloads over the last decade?" At this point, I turned around -- who the hell was this guy?! -- and saw that it was none other than John Hennessy himself. A Silicon Valley gangland slaying if ever there was one -- and told me everything I (or anyone) needed to know about Itanium's fate.

Your computing history stories are really fun. Just the other day I had the opportunity to link a bunch of people to your rm -rf / story.

EDIT: link: https://news.ycombinator.com/item?id=8896812

As long as you're playing the classic tracks, another personal favorite from that era:

https://news.ycombinator.com/item?id=4306515

As a follow up question it was rumored a couple years ago that there where builds of SmartOS for ARM but it was only waiting for 64bit hardware due to zfs... Care to confirm?

Aren't rumors much more exciting when they're neither confirmed nor denied?

I simple have to ask this question (full disclaimer: longtime functional nerd with specialty in Erlang):

On a scale from totally fine to batshit crazy, where do you place Erlang/OTP? Where do you place Node.js on the same scale? And why?

(this in the vain hope I can get Bryan on one of his famous rants :)

So, Erlang is definitely nutty (and it's certainly tough to have a language where "if" doesn't mean what you think it means!), but my biggest problem with Erlang is that it becomes entirely undebuggable when it spins out of control. We once had RabbitMQ at the core of our stack, but have ripped it out because of its adventures in what we dubbed "the rabbit hole": RabbitMQ would stop doing useful work for minutes on end while the rest of our system backed up behind it. I tried every trick in my book to figure out what the hell that thing was doing (and as you can imagine, I have a pretty thick book), but I ultimately gave up -- and we ripped it out. Is that Erlang's fault? RabbitMQ's? Or does the fault lie with the node.js AMQP client? My hunch is that the defect that sent us into the rabbit hole was actually on the client -- but I ultimately blame Erlang for its lack of debuggability.

Thank you for your answer. It is peculiar, because I normally tout Erlang debuggability as one of its strengths. It essentially has the like of DTrace built in (and DTrace support in the C parts to boot nowadays). You can query the live system while it is running in production by connecting to it. It isn't the most intuitive platform however, and sadly many of the good old tried and true UNIX-book tricks doesn't work, since the environment is so different.

As for clients, the major one I've seen is that many of them forgets to set TCP_NODELAY, which shows itself as a nice little latency hiccup in tcpdump :)

Nice you solved the problem by ripping it out and replacing it with something you know. If there is any rule to go by it would be "pick the system on which you are familiar with its peculiarities, be it Illumos, FreeBSD, Linux, ..."

I would love to agree with you, but I enabled the DTrace probes, and the VM promptly died (this is in production, so no do-overs on that one). Looking at the code, it was clear that it was a botched DTrace is-enabled probe that no one had ever bothered to turn on. This is kind of my problem with Erlang: "use this!", and it's like "okay, I did and it died; what else do you have?" (The same is true for Erlang's janky "core dumps", which I found to be useless -- and I couldn't find anyone who could make use of them.) Erlang has created for itself a legend of robustness, but to be totally blunt with you, I'm not sure that it's not just a myth...

I have written Erlang in Anger specifically to help debugging live systems, which could (hopefully) have helped with your issues if you were to see them again.

For what it's worth, the 'built-in like of DTrace' likely referred to actual tracing of Erlang function calls. See recon_trace, redbug, dbg, and so on.

Thank you for that link! I don't mean to throw Erlang under the bus (I do, after all, ultimately believe that this an AMQP client issue), I just ended my Erlang relationship on a very sour note. I don't doubt that I will encounter Erlang in production again, so I would like to understand better how to debug it -- I'll definitely check out Erlang in Anger! (Also, thank you for Learn You Some Erlang for Great Good!: I did use that in some of our darkest hours and it was helpful -- and well written!)

At the time when Solaris was open source, and in particular dtrace since it was opened a few months before Solaris, what was Sun's (or your, to be specific) expectation regarding technologies like dtrace and zfs with regard to Linux? Was the CDDL designed to prevent Sun technologies from entering Linux?

Great question, and the answer was that we didn't know -- but the expectation was that it would be ported to Linux relatively quickly. I remember vividly standing over a terminal with a bunch of people as we actually launched OpenSolaris (like, clicked carriage return on making the DTrace code live -- which was the first in the chute), and the Sun Legal guy and I were chatting. We were both wondering if DTrace was going to show up in Linux in a month or if it would take two years. But that was the range of guesses: neither of us believed that the Linux community themselves would hold up CDDL as an obstacle, and certainly if you told me that a decade later, DTrace wouldn't be in Linux because of licensing FUD, I wouldn't have believed you. Of course, in hindsight, it all seems so clear: NIH is enormously powerful, and we were fools for discounting it. After all, as Jared Diamond pointed out in "Collapse", the Norse in Greenland starved to death when their domesticated livestock died rather than eat fish like heathens...

Do you think there's a natural progression for a company to act like Oracle once it becomes big or powerful enough?

No. What Oracle does is Wrong (capital "W") in that they have no regard for the importance of a social contract; no company ever outgrows its responsibilities to society.

If Unix were invented today, with all of the knowledge and understanding we have now, and with no obligation to support any of the software currently written, would the process inheritance model stay the same? Would it make more sense for processes to be owned to the next available parent, or is the current mechanism of straight-to-the-top the superior model? and why?

Explanation

Currently, when a child of a child process becomes orphaned (its parent dies), the child is owned to the top-level process, typically init as pid 1 (though in the case of zones this is slightly different as there is a shared process space). Would it make more sense for the child to be owned to the next available parent and not just go straight up the chain?

Say there is a chain like:

p1 -> p2 -> p3 -> p4

If p3 dies, then p4 will be owned straight to p1 with no regard for p2 in the inheritance chain. It seems like it would make more sense for p4 to be owned to p2 in this event, as p2 is the next available parent in this chain, and has the closest relation to p4.

Many service management frameworks encounter this "problem" and have unique ways of solving it: Illumos has contracts, others use ptrace, and Upstart flat out says it can't track this. Note that the solutions usually require some sort of mechanism in the kernel to track processes, and cannot effectively and efficiently be implemented solely in the userland.

I think you're highlighting one (of many) challenges that we inherited from Unix. I obviously have great reverence for the system and its principles, but things get gritty at the margins. Reparenting is definitely one of those goofy areas of the system that hasn't ever made sense to me -- which is why we implemented contracts in illumos. I still believe that contracts (or equivalent) are the best way to solve this problem, and I would love to see equivalents in other systems. On the opposite end of the spectrum: ptrace is unequivocally the worst way to do it -- with apologies to Upstart...

Thanks, it's nice to know I am not the only one who questioned the reparenting logic that is taken as a given in most systems.

Contracts are definitely the best way I've seen to deal with this, but if you were to rewrite Unix today for a brand new world, would you change the inheritance/reparenting logic? or is contracts the proper method to control/follow collections of processes?

That's a good question -- I'm so used to the dirty sheet of paper that I can forget what a clean one might look like. ;) But I do think that contracts solve this problem at least well enough -- and don't carry the extra burden of a from-scratch rewrite of the system.

If ranking is organisational cancer, how do you figure out how much to pay different engineers in a flat organisation without creating a secret tumor timebomb when people do eventually figure out where they sit?

I don't think it's a time-bomb: salary is private and should be private. And remember, ranking is organizational cancer because it says that you tautologically cannot have an organization of exclusively high performers -- which should in fact be the goal of every organization.

I notice you didn't actually answer his question, which was

<context> how do you figure out how much to pay different engineers <more context>?

It's not some plug-and-chug formula, and really it's the job of technical leadership to make sure that engineers are compensated well enough that they can freely focus on what really matters: mission, team and technology. Over-compensate people, and you will attract the wrong kind of people; under-compensate them and you will lose the right kind -- but there is no formula as to how to strike the balance.

What do you see as the core competencies needed for software engineers in the next five years?

One of the concerns that I have is that stacks are becoming ever-more complicated (accelerated by open source, which has allowed us to live the dream -- and the nightmare -- of componentization) but that our education isn't keeping pace. So permit me to be the anti-Peter Thiel: budding software engineers need to complete their formal education, first and foremost. I know that this isn't necessarily a popular opinion, but formal computer science education is essential, and I have seen (and continue to see) a marked difference between those who have completed it and those who haven't. An essential part of that formal education: learning what you don't know. If you don't come out of a formal education humbled by it all (and astonished that a software system can so much as boot, let alone run, let alone run well), then you missed the point.

Follow up question then - if you are a software engineer who hasn't completed a formal CS degree, would you suggest that? I went to school for Mechanical engineering where we did a ton of programming, but didn't have classes in Big O and other formal topics. Despite that I've been a 'software engineer' for 15 years.

I hear you, and I would suggest at least doing some online stuff in CS. You already have a technical degree (which, as you know, speaks volumes to your persistence and ability to tackle hard problems), but I do think that it's worth understanding the principles of computer science. Plus, you'll likely find it really fun. ;)

Thanks for the suggestion there. I tend to fumble on those basics on interviews which hurts, but then excel when asked to solve real problems (which aren't part of a lot of interviews).

Argh -- I hate that! We don't do interviews like this, for exactly this reason. Should you ever wish to work at Joyent, we will give you homework, not an in-class exam. And that assumes that you're not already participating in one the open source communities that we participate: hands down our highest fidelity mechanism for hiring is to hire those that are already able participants in the code bases that are at the center of our business. If you are already doing good work in SmartOS or SmartDataCenter or Manta or Node.js, the interview will consist only of determining that you are, in fact, the person behind the work!

Recently Joyent announced a Docker PaaS offering. What benefits to Docker do you see vs zones (joyent branded or lx) with good config management? IOW why should I be excited to use Docker on SmartOS if I already have good configuration management and custom datasets?

First of all, don't let me talk you out of any SmartOS love. ;) As you know, you can get much of what Docker offers today with SmartOS -- but I think that the image creation and layering process of Docker (along with its developer friendliness) represents a real step forward. For ourselves (and in particular, for Manta), we would love to use Docker to deploy our SmartOS images -- and thanks to some terrific work by Josh Wilsdon on Joyent, this largely falls out of our Docker effort.

but I think that the image creation and layering process of Docker (along with its developer friendliness) represents a real step forward

in what way?

In what way is it not? We were earlier to OS-based virtualization than just about anyone, and I don't know of anyone doing container image layering the way Docker did it before Docker did it...

How do you as a leader continue to ensure an innovative, dev-centric engineering culture as your organization grows?

I think you need to encourage people to take risks, back them up when those risks fail, recognize them when those risks pay off -- and constantly connect everyone to the larger mission. I'm not sure if I'm succeeding on that front or not, but to a degree the proof is in the pudding: and we have been (and have continued to be) able to pull off things in system software that amaze and delight -- which is a pretty high bar for a layer of software that has much more ability to enrage than to please!

Regarding your scaling engineers talk: How would you run an engineering organization that has passed the 200-300 people mark and cannot have everybody reporting to you?

That's an excellent question, and I would almost flip it around: should any software engineering organization ever get that large? Or put another way: I've never seen an organization that large that doesn't in fact break down into a confederation of teams that are essentially our size -- which may well be due to Dunbar's number.

So, you are suggesting that software is the output of an coherent engineering community. When you break the coherence of the community, you break the software. Fair enough, though the problem now has moved up: How do you coordinate the different communities. In your view, is that the job of the CTO or someone else's?

Coordinating the different communities is very tough -- and yes, that must fall to the CTO and to the VP of Engineering as a team. The CTO should be a little more forward and outward looking and the VP of Engineering a little more tactical -- but they should work together as a team to assure that fiefdoms either don't arise (unlikely) or that they at least are different expressions of the same culture. To be honest, I'm not sure that there's a good model for this, and if I found myself the CTO of an entity that large, I would try to model myself on Sun's Greg Papadopolous for whom I had (and have) great admiration.

Hi Bryan, I love your talks, especially when you talk about computer history. Do you have any books to recommend about UNIX history, and computer history in general?

Great question, and one that I could spend all day answering. (And just might!) The must-read is Soul of a New Machine by Tracy Kidder. Beautifully written, and amazingly current for a story that took place 35+ years ago. I would next recommend A History of Modern Computing by Paul Ceruzzi and From Airline Reservations to Sonic the Hedgehog: A History of the Software Industry by Martin Campbell-Kelly. Finally, a personal favorite is IBM's 360 and Early 370 Systems by Emerson Pugh. Read those, and you too will find yourself wowing friends and love ones with anecdotes from early computing! (I have actually read aloud the incomparable Prologue from Soul at a dinner party -- pity my family!)

Which GNU/Linux distro do you use and why?

Doing this makes you a fan of odd things. I definitely like Ubuntu and see the value there, and I actually like all of the pretty green screen-era ASCII art of yum -- but I have also become a total fan of Busybox and (as of this morning!) Alpine. So deliciously small!

What is your stance on tech internships and what is the best internship experience you've heard of (doesn't have to be from a company that still exists)?

Full disclosure: the questioner was an intern at Joyent last summer. ;)

I remain a strong believer in internships -- mine (at QNX) was incredibly valuable, and many of the engineers that I work with today started as interns. My favorite internships are still the ones that are pretty loosely structured: bright students unleashed on problems that are considered too risky for the broader team. This isn't the way that everyone does it (and some places give interns infamously boring work), but in my experience it has the most positive potential upside. This philosophy is due to the late (great) Dan Hildebrand at QNX, one of my mentors at QNX, and someone who always believed that people should attempt insanely hard problems, not be content with easy ones.

ELI5: What is Docker ?

First of all, as the father of three kids, if you want me to actually reply to you like you're one of my kids, it will involve a poo metaphor (2 year old and 7 year old) or baseball metaphor (7 year old and 10 year old) because that seems to be the only way they retain things. Explaining Docker in terms of poo and baseball seems a bit daunting, so I might explain it this way: Docker allows you to take a bunch of disparate binaries and pickle those together as an image that can then be deployed as a unit into production. This allows the developer on the laptop to develop exactly what runs in production (or that's the idea anyway). I have been known to sum it up this way: Docker will do to apt what apt did to tar. Let me know if that still misses the mark!

So a golden image, but for containers instead of Operating Systems?

A golden image that can be layered: a key aspect here is that you can easily build on the work of others.

What was the best release of Solaris? Why?

Solaris 7, of course!

What do you think of unikernels+docker in the Linux world (https://www.linux.com/news/enterprise/cloud-computing/819993-7-unikernel-projects-to-take-on-docker-in-2015) ?

I admire their spirit.

What are your thoughts on the io.js fork of Node.js?

As I have written before, the power of open source is the power to fork. In that regard, I view forks in open source as the framers of the United States looked to states: as laboratories for experimentation that contribute to a healthy union (and see my earlier response for my perspective on how this has been essential to Linux). That said (and, to continue with the constitutional analogy, to make the federalist argument), total decentralization can approach chaos -- and projects with similar (but spuriously different and unmergeable) forks can undermine their shared community (viz.: MySQL). So while I believe in forks, I also believe that it behooves their initiators to clarify (and contrast) their principles.

So where does that leave io.js with respect to node.js? I can't speak for io.js, but I can speak for the principles that we have for node.js. We believe that node.js is and should be an enterprise-grade substrate for modern services: debuggable, observable, stable, reliable, and high-performing. We also believe that the goals for node.js transcend one company, and that the project therefore needs to be led by a foundation -- which we at Joyent are actively advancing. To the degree that our principles don't represent the principles of others (and that isn't a value judgement: not all of the world is a production system), a fork may be healthier than having endless smaller conflicts that actually represent deeper rifts.

I like that analogy, thanks for the reply, Bryan! When you're done with the AMA, might you be able to paraphrase that analogy and send it to me in the form of a pull request? https://github.com/nickdesaulniers/What-Open-Source-Means-To-Me

Yes, will do -- a bunch of interesting thoughts in there!

Hey Bryan, it's Brock, friend and former Joyeur :)

What part of the stack would you say is the most challenging to work on or has the biggest challenges and why?

Is there a system/program that you feel does an exceptionally good job at solving a particularly difficult problem? If so, which one?

edit: Looks like you already answered my second question.

Hey Brock! I think that each layer has its own challenges. The kernel can be really nasty (of course) because when the illusion maker stutters, the very fabric of reality can be torn. But I think it's a mistake to think that the kernel is the "hardest" software: distributed systems have absolutely grisly failure modes that we are even less well-equipped to debug. In terms of systems/programs that do an exceptionally good job: I am often inspired by Dave Pacheco's node.js modules and systems -- and would very strongly recommend his NodeSummit talk on industrial-grade node.js.

What's the oldest Sun hardware that you still personally own (if any)?

I am the proud owner of maffei1, UltraSPARC serial number 5 (long before first customer ship), and the first delivered to the kernel group in Mountain View. maffei1 actually predates me at Sun, but there was a horrifying moment when it was marked to be thrown out, and I saved it for the good of humanity. When I get home, I'll dig it out of the garage and post photos of it -- the proto cage practically drips with the sweat of bringup...

Is it true that DTrace was responsible for the end of sun4m support in Solaris?

Absolutely not! It may, however, be true that we waited for sun4m to be ripped out before integrating it. But my conscience is clear: I had tried to wait out sun4c for the cyclic subsystem, but due to managerial shilly-shallying, came to the conclusion that I couldn't wait any longer and implemented it (which was a giant pain in the ass). Of course, moments after I integrated it, the decision came down to rip out sun4c -- and despite my protests, they refused to even cut one more build. So if DTrace killed sun4m, it was to repay a blood debt.

In a recent talk you eschewed professional titles. But there was no recommendation or alternative that was suggestd.

So how does promotions work in Joyent? Is there a hirearchy?

What do you suggest as an alternative?

Salary varies according to experience and role and (as I mentioned elsewhere) salary remains confidential (and should).

I am an admirer of your great work (going back to the days at Sun), many thanks for this AMA.

Sorry for the offtopic gripe but when people send their resume to Joyent to apply for posted positions, would you consider it too much for the HR department to take the small amount of time needed to fire off a rejection letter? I think very highly of Joyent and am truly disappointed to see it act the same way as many others. When job seekers take the time to apply with a cover letter and CV, it would seem only fair to e.g. take like 1 minute to copy paste a canned response instead of just ignoring them.

With tremendous respect for your work and obviously this small complaint does not reflect on you personally. Be well.

Thanks for the kind words -- and I'm sorry to hear about your experience. Generally, we pride ourselves on our professionalism -- but sometimes things fall through the cracks. Can you send my e-mail at my first name (spelled correctly!) at joyent and cc: jenny at the same and we'll get it sorted out one way or the other?

Who is the greatest Fishpong player of all time?

Dave Lightman, no question.

The best way to combat piracy in the movie and music industry are services like Netflix/Spotify/Napster etc that offer their users access to all the content they desire for a monthly fee.

Do you think there will be something similar for software in the near future that combats piracy, while making profit at the same time?

Isn't that effectively SaaS?

Years ago I had a hosting account with textdrive.com and at the time (around 2004 I believe) it was probably the most developer-friendly hosting option around. It always came up in conversation as a good choice. Something weird happened later with Joyent buying them or something and it all seemed to go to pot. I miss those days.

Can you comment on what happened and who snuffed the flame?

That era pre-dates me at Joyent (I showed up in 2010). I can say that Dean Allen was the flame behind Textdrive, but I would leave it to Dean, Jason Hoffman and David Young to explain the origin story of how that turned into Joyent.

In terms of Joyent today, I can tell you that we've never been more bullish -- and it feels like being ten years too early to OS-based virtualization is finally proving worth the wait with the release of Triton. For Joyent, the "good old days" may well be now!

Would you bring back ps pipe grep podcast?

If we do, it will surely be called pgrep!

Hi Mr. Cantrill! Thank you for doing this AMA! Do you have any suggestions on what students interested in CS should do in terms of programming and learning CS topics?

Happy to help, but a little more context would be great: at what level? And what formal options are available?

High school student, and taking classes online or reading textbooks

Okay, that helps a lot. Your priority for right now is to point yourself to the best university-level curriculum you can find. There is (for the moment, anyway) a sea of difference between high school- and university-level computer science. So you want to do well in your math courses (sadly, calculus is really meaningless for CS -- but you want to do well nonetheless), take whatever computer science you can (but that's often very limited) and figure out the next step for yourself. If you can give me additional context (where in the world you are located and what options at age eighteen you might have), I can give more specific guidance...

I'm a high school senior who will probably go to Georgia Tech for CS. I know Python and Java, but I'm not too experienced in it. I've started to read essays from James Mickens, Paul Graham, and Eric Raymond to have a deeper understanding of CS. Also, I've started to look into open source projects like AOSP.

Good for you! Georgia Tech is a terrific school -- you are well on your way! Next step for you (from my perspective): learn C. No matter what level you end up being in the stack, learning C forces you to understand the machine at a deeper level than other environments -- and you will be amazed how much that understanding helps even when at much higher layers of software. I would recommend Stephen Kochan's Programming in C coupled with Peter van der Linden's Deep C Secrets. You can also get K&R, but it's one of those books that makes much more sense once you know C. The sooner you learn C, the better: even if you end up spending the rest of your career at a much higher level, the foundation you build by learning C as a younger engineer will serve you well for many years.

Back before containers were all the rage and there were things like userspace linux and openvz and the like, I'd have friends of mine who were Solaris fans swear up and down that zones were Just Better, but never explain why.

Now Linux has cgroups and namespaces for things like network stacks and process trees and things like Docker are using them to do better containers and Solaris folks still swear up and down that Zones are Just Better, but I can't find a technical explanation as to what makes them better.

As a Solaris person who swears up and down that Zones are better than anything in Linux, can you explain what about the implementation of Zones makes them superior to using cgroups, namespaces, etc.?

Thanks!

Yes, that's a great question. At the highest level, I would say the difference is that Zones -- like the FreeBSD Jails that inspired them -- are a first-class, kernel-level primitive for containment and virtualization. Linux, by contrast, lacks such an in-kernel construct; Linux "containers" are actually higher-level primitives built on the low-level mechanisms you mention: cgroups and namespaces. The problem is that these two mechanisms are necessary but not sufficient for true containment -- and they essentially represent iterative, evolutionary approaches to the problem. The catch is that OS-level containment actually necessitates revolutionary change -- something that Linux has really struggled with (witness the inability to absorb DTrace, ZFS, anything like SMF, etc.). So, for example: namespaces have only had "user namespaces" (namely, the ability to have a container be in an orthogonal UID space) since 3.8 -- long after Linux was claiming to have containers. To those of us for whom security is an absolute constraint (and to those of us for whom running root in a container is a constraint on the problem), this iterative approach is deeply troubling. Linux containers are now in a bit of a catch-22 in that they aren't really being deployed in multi-tenant, internet-facing environments due to security concerns -- but because they aren't deployed in such environments, it's hard for them to earn the trust necessary to be deployed there in the first place. (That is, it's a bit of a "you first" problem.) By contrast, the deep approach that we took in Zones (and the fact that they have been deployed in multi-tenant, internet-facing production systems for nearly a decade) gives the confidence that this Zones are a trusted, secure containment mechanism -- and explains why so many of "us" swear by them.

For details on the complexities and limitations of the Linux approach, I would recommend this excellent presentation by Jérôme Petazzoni.

Thanks for the detailed response! I'll take a look at that presentation!

My understanding is that zfs isn't in Linux because the GPL doesn't play nice with the cddl, rather than a technical limitation. But your point is well received.

Also, good luck with Triton. Looks neat.

Well, Andrew Morton infamously dismissed ZFS as a rampant layering violation, so I wouldn't be so quick to conclude that the lack of ZFS in Linux is purely a licensing concern. (Indeed, the uncontested presence of ZFS on Linux gives the lie to the licensing FUD.)

Why do you call yourself the "father of DTrace"? Is Mike Shapiro (co-creator) another "father of DTrace"? Or does Mike agree with you claiming to be the father?

Mike, Adam and I were the co-creators of DTrace, and each of us was invaluable in its creation. But it is also true that the original ideas at the core of DTrace were actually mine, and they pre-date my arrival at Sun: when I was an undergraduate, I sketched out my thinking for dynamic kernel instrumentation to my undergraduate advisor. His reaction (and, it must be said, my thinking as well) was that there must be some reason that this was impossible; if it could be done, it would have been done by then! When I went out to interview with Sun, I sketched out the same ideas for Jeff Bonwick, wondering why they were impossible to implement. What happened next was so important to me personally that I know exactly where I was at that moment: in Bart Smaalders' old blue van on Willow Road crossing the 101 coming back from a very nice lunch in Palo Alto. At that moment, Jeff said: "Yeah, that's a good idea, actually. You should come here and do it!" I did indeed come out to Sun to work with Jeff -- and the next year I went back to school to recruit Mike (who had been my classmate and who had stayed for a one-year Masters degree). Once at Sun, we agreed that we would work together on two big ideas: we would first work together on the debugger that Mike wanted to write (which became MDB and remains an invaluable tool that I use every day) and we would then work together on my dynamic tracing idea.

So Mike is the father of MDB even though I did a lot of important work on it; I am the father of DTrace even though Mike and Adam did a lot of important work on it. (With my apologies for the wordy response; it's very important to me that credit goes where it's due, and that I don't leave the impression that I was in any way solely responsible for DTrace!)

If you could recommend three books to someone who's just starting out as a unix sysadmin / engineer, what would they be?

Okay, I'm an old schooler in this regard, but I would honestly start with the man pages. Speaking for illumos/SmartOS, there's a lot of very good stuff there -- and having spent a ton of time in the Linux man pages, I can (broadly) vouch for those as well.

Do you think that Joyent opened Pandora's box by resurrecting LX?

There are a lot of potential Linux users who might jump onto LX and use full CentOS or Ubuntu containers - not just Docker.

Which is great! We actually got that stuff working first, so no problem if people want to use full CentOS or Ubuntu -- or (as I was toying around with this morning) Alpine Linux, of which I am definitely the newest fan.

Joyent has put forth a tremendous effort in the form of LX branded zones to support Docker. Is there any quid-pro-quo from the Docker community to get more images to be cross-platform?

Ha! Nope. I think that the Docker community is going to need to get cross-platform images for ARM long before SmartOS -- though we would certainly welcome SmartOS Docker images! ;)

Do you expect that Joyent will continue to make investments in maintaining the pkgsrc repository for native SmartOS? I realize that many people seem to want to work within a Linux environment, but there are a number of us out there for whom the thought of having to embrace Linux (even with an illumos kernel) is extremely distasteful.

Yes, we will absolutely continue to invest in pkgsrc. And if you're a pkgsrc user, be sure to check out pkgsrc-2014Q4!

Have there been any notable technologies or solutions for which your initial impressions or predictions for its future were completely wrong?

blog post on this: https://www.joyent.com/blog/predicteria-2015

So that blog post has a bunch -- and there are a bunch more. My miss rate is high in terms of businesses (P.S. so is everyone's), but I can remember in particular thinking that Google was a moronic idea. (Another search engine?! When we already have Alta Vista and Lycos?!)

In terms of technologies, my miss rate is slightly lower (though still high) and I've been wrong plenty there too. One notable miss for me: damn, did I ever want Parrot to be the one VM that would rule them all...

smartos-live seems to have a lot of pull requests on github. Is someone at Joyent actually looking at those?

Yeah, sorry about that. We've been very busy on Triton and we haven't given smartos-live pull requests the love that they need. Was there one in particular that is important to you?

Do you consider creating a "docker like" ecosystem for zones?

We actually thought we would do one better.

On a baseball scouting scale, how would you rate CoffeeScript? 60? 80?

Hits for average. Minus arm.

What are the most exciting aspects of where Unix is going in terms of large scale projects? And how can a newbie get involved?

Well, I'm obviously biased -- but I'm very excited about where we're going in SmartOS. In terms of getting involved, it's all open source! C'mon in -- the water's fine!

What is your greatest regret code wise? What is one piece of code you wish you hadn't unleashed on the world, or at least taken the time to improve before doing so?

Oh, great question! There was a wad that we called "Bryan's Folly" in which I tried to reimplement DTrace ioctls as DOF -- but I came to the conclusion that that was a disaster before I integrated it. Of the code that I integrated, one thing that I definitely came to regret was a Perl monstrosity that I implemented as part of the command-line interface for a storage appliance. But out of this monstrosity came something important: I rewrote it in JavaScript -- and in so doing recognized that I didn't just tolerate JavaScript, I actually liked it!

Hi Bryan!

If you could make any one unilateral amendment to US Constitution, what would it be and why?

Thanks for doing this AMA!

Fun question! Given that this is my unilateral amendment, I'm not going to cure any of society's ills (sorry, society!) and give us software engineers our equivalent of the first amendment: an amendment brimming with rights that courts will be interpreting for years. In my Software Rights Amendment, I would establish reverse engineering as a natural right (akin to the right to free speech), I would clarify that APIs are not protectable by copyright (suck it, Oracle!), I would reduce the length of patent protection (17 years is way too long) and I would require that any software-based patents require the source code of their embodiments to be publicly available. Please write your representative and encourage them to support the Software Rights Amendment! ;)

Why would I deploy docker instead of a well-tuned VM today? Keep in mind that I'll eventually need to patch and scale the application running in either.

Today? I think it's arguably a push and I think it depends on what you're comfortable with. But in the future, I think containers clearly resonate with both developers and operators -- and I can tell you from our experience that their efficiency and tenancy gains are unachievable with VMs. So while I don't know that I would say that you should be deploying on Docker today, I definitely think you should be familiarizing yourself with it...

What do you think of the unikernel movement? Projects like Mirage, HalVM, Erlang on Xen, OSv etc?

While these projects clearly sacrifice the features of storage solutions like ZFS and introspection of DTrace, they gain a lot of flexibility letting the engineer avoiding legacy cornerstones of huge OSes that often get in the way of doing things because their features/behaviours were not considered.

I think that they're interesting at some level, and I certainly admire anyone who is trying to innovate in terms of systems software. That said, I do think that there are some issues with the approach -- and indeed, I think one ends up with many of the issues of hardware virtualization compounded with the difficulties of kernel development. For example, how do you debug one of these when it goes out of its mind?! (Just to head it off: the answer "you restart it" will be awarded a failing grade -- along with a "See me".) So the unikernel movement reminds me of the exokernel movement from which it presumably drew some inspiration: interesting ideas, but ones that may serve more as a negative result (i.e., what not to do) rather than as a blueprint for future systems.

If there is one thing you could say to 12 your old you, what would it be?

I would give myself advice that my mother gave me some years later when my family life got very rocky: the strongest sailor knows the roughest seas. It can be very hard to grit things out when you're a kid, and I had a particularly mean peer group at the private school that I attended. (That aspect of life improved tremendously a few years later when I went to a magnet program at an inner city high school.)

Any thoughts on the app container spec? Any chance that we'd see Triton become a platform for containers implementing the spec?

In the abstract, an app container spec is a good thing -- and to the degree that it gets traction, it will absolutely be something that Triton supports. That said, we have not been bashful about our opinions about Rocket. While I applaud the desire for standardization, I think CoreOS focused on the wrong layer: to me and to us, the remote API is where the action is, not the container itself.

Could you say some nice things about Linux? We all know it's not that bad. Some parts of it are pretty cool!

Also, why do you think Oracle is still bothering with SPARC?

I think I give Linux credit where it's due. There are a couple of things that Linux pioneered, but first among them is the general forkaphilia: by encouraging forking instead of fearing it, Linux allowed robust packaging competition (yielding several very capable systems) while at the same time preserved a unified kernel by allowing for innovation down-stream. These are two very important -- if not nice -- aspects of Linux.

In terms of Oracle and SPARC: NFI. It's clear that Oracle views SPARC as a physical embodiment of its rapaciousness, and the unit volume has continued to decline. At some point, even lawnmowers cannot defy the laws of physics...

Hello Bryan,

Thank you for this AMA. I fairly unfamiliar with unix and this has been insightful. How did you get into OS development was that what interested you from the get go? Also what advice do you have for an aspiring developer?

I got into OS development in my undergraduate computer science degree. I was blessed to attend a school with a legendary operating systems course, and I found it deeply inspiring. To me, I love OS development because (and this may seem surprising to some), I realized at a relatively young age (~19) that I love debugging. The greatest satisfaction that I have had is nailing a nasty bug -- it's an experience that (for me, anyway) is so visceral as to be nearly primal. (Viz.: when finally reproducing a nasty bug that had been plaguing us for weeks in production, I let out a guttural yell so loud that a co-worker reported "it sounded like someone had cut their finger -- like, off.")

In terms of advice for the aspiring developer: my most important advice would be simply finish your formal education.

Do you have any recommendations/suggestions for someone wanting to dive into kernel development? I've made some decent progress working with "xv6" but that code is (intentionally) so much simpler than modern kernels.

I think that's a good way to start! Another person that I would reach out to is Ryan Zezeski. He was like you in that he wanted to dive in, and the path he took was particularly interesting -- and (I dare say) replicable for someone who is interested in getting involved in kernel development. Good luck, and don't hesitate to reach out on Twitter, IRC (#smartos) or the discussion lists (e.g. smartos-discuss).

As someone who clearly loves UNIX and its history, what was it about UNIX that hooked you in?

(FWIW I've been a fan for quite a few years now, I love your talks and I've found what you and the Joyent team do to be inspirational, as well as your former work at Sun. Thanks for all the stuff.)

Thanks for the kind words! In terms of what hooked me in, I think it's the same thing that hooks in everyone: understanding the elegance of the model. For me, that was the first assignment in my operating systems course, which was to implement a Unix shell. At first, it seems insanely daunting, but once you understand the file descriptor and process models, it's actually a surprisingly concise (and indeed, beautiful) program.

If you were to design a new software license what terms would you put into it? How would you address the free-rider problem?

I actually love the MPL 2.0 -- and when we open-sourced SDC and Manta we very deliberately chose the MPL 2.0 for our de novo software.

Alright, this isn't so much a question regarding something you or your company has done, but I figure I'll give it a shot.

I've been studying for the CCNA for a bit now, but I have some concerns, not specific to that cert, but pretty much all certs in general. I'm a long time linux user, going on about 15 years now. I've become familiar with windows as well. I lack any certification whatsoever at this point. I've decided to study for the CCNA because it seems relatively prevalent across the board. There are more/better jobs I could get with a CCNA than Network+, for instance. Buuuuut, there aren't a huge number of CCNA jobs near me. Money is an issue, I can't just move to where jobs are. There also aren't a huge number of any other IT-cert-required jobs near me, either(unless I'm just not looking in the right places.)

I also don't have the money to go to school for a traditional 4 year degree from a legit/accredited school. Probably not a 2 year program at a community college either. I doubt that any other cert would be any better than CCNA. I refuse to do A+, I'm not going to spend a million hours learning information that is probably at least 50% totally obsolete and I'd really rather not go clean dry jizz out of laptop keyboards at best buy after spending hundreds of dollars and a shitload of time to learn what a PDIP is.

What would you do in this situation? I have the ability to learn on my own, but it seems regardless of what I prepare to get a cert in, there aren't any jobs around me. I don't know if I should just say "tough shit" and go back to working in shitty pizza places for basically less than minimum wage, waste a shitload of money on a cert that isn't going to get me a job because there are none where I live, or... I don't know. I honestly don't know what other options possibly exist. Because that's pretty much it, as far as I can tell.

If you taught yourself Linux, you can learn computer science and can learn to be a software engineer. You are right to be skeptical of the certs -- I think that they're not very useful. Are you sure you don't have the money to go to community college? The community college system is one of the greatest assets of American higher education in its overwhelming accessibility. I would pursue a parallel route: go to the nearest/best community college you can afford and at the same time, start the best online free courseware you can find in computer science. (Do not pay for the intensive "coding boot camps" -- from my perspective, they are broadly crooked.) Start attending local meetups, and learn both specific technologies (node.js/JavaScript, Go, Python, Docker) and (importantly!) the fundamental computer science principles that underlie them. This is not an easy route (and will depend on your life situation: what, if anything, is your day job or means of support -- and do you have kids?), but you can definitely do it. On the one hand, the path ahead is steep and long -- but on the other, if you can keep putting one foot in front of another even when you don't want to, you absolutely will get there. And please, on behalf of your future self, let me beg you: don't go back to work at a shitty pizza place any longer than it takes to pay the freight to get to a better place.

Bringing docker to SmartOS is cool, but why focus on Linux container compatibility rather than improving the Solaris userspace so people actually want to run their apps in zones or Solaris docker containers? I've noticed SmartOS is more and more GNU-ey than traditional Solaris, so maybe that's ongoing?

I ask because I think the underlying framework of Solaris is superior to Linux in several ways (SMF, crossbow, dtrace). The biggest reason why you don't see people using it a lot is generally because they'll tell you "I don't know how to get XYZ app working on it".

Part of the appeal of what we've done with respect to Triton is that it brings Crossbow and DTrace to Linux containers -- which is pretty neat. But that still leaves SMF -- to say nothing of the other advantages of running SmartOS native. This remains very important to us, and as you noted, SmartOS is much "more GNU-ey" than traditional Solaris, because of our embrace of the open source ecosystem via pkgsrc. Speaking of which, we've made some really important advancements there recently with pkgsrc-2014Q4, as outlined by Joyent's Jonathan Perkin -- if you're interested in native SmartOS, you'll definitely want to check it out!

As the father of DTrace, how do you think the prospect of the DTrace? Do you have any new feature plan for DTrace? What do you think of Oracle's porting DTrace on Linux?

I think the prospects of DTrace are pretty good -- I certainly still use it every day. ;) The most important immediate direction for DTrace is its (true) arrival in Linux via Triton -- and making sure that esoteric features like USDT providers and (more challengingly) ustack helpers can work on Linux binaries. (We've done the former; we're working on the latter.)

In term's of Oracle and DTrace: it remains true that Oracle's port is the most complete -- and it's also the only one that is (or indeed, can be) closed source. That Oracle has left their port proprietary is classic Oracle: willful disregard for the social contracts of open source, or for the legacy of Sun. (And indeed, had they not bought Sun, this act would violate the license of DTrace.) Leave it to Oracle to pioneer new shades of mendacious behavior!

There are some useful DTrace user guides, but it seems there lacks a detailed or drill-down analysis of DTrace source code. Do you have any plans to write some articles or books about DTrace source code? I think it maybe help more people to know DTrace.

For the source code itself, I would put it up against anything for the thoroughness of its commenting; once you understand the DTrace guide, go here:

https://github.com/illumos/illumos-gate/blob/master/usr/src/uts/common/sys/dtrace_impl.h https://github.com/illumos/illumos-gate/blob/master/usr/src/uts/common/dtrace/dtrace.c

If that's too low-level but you're looking for something beyond (or in addition to) the documentation, see out 2004 USENIX paper and my 2006 ACM article. And then of course, there are Brendan Gregg's books: DTrace and Systems Performance. Point is: there's plenty of information out there -- you just need to read it.

Have you contributed the code to Linux? If not now, will you contribute it one day?

Contribute what, exactly? DTrace is all open source.

How do you feel when you are editing very sensitive function?

An old essay of mine actually covers this pretty thoroughly. Backstory on that: it was actually an e-mail sent out to the entire Solaris development organization after a new engineer had broken the system very badly in a way that I felt was reckless. Instead of admonishing him personally, I put that energy into writing the e-mail. It didn't actually change his behavior, unfortunately (he went on to break the system badly a couple of more times before it was suggested that this probably wasn't the right job for him), but it did survive the test of time, albeit now couched in dated, Solaris-, Sun- and MPK17-specific language.

What do you think the future holds for Joyent on the business front? Stay private? IPO?

Do you think Joyent has the long-term gusto to remain a competitor to the likes of Amazon, Google Compute Engine, and the like?

Edit: Thanks for doing this AMA, Brian. Never crossed paths with you at Joyent, but as an oooold school Joyeur, it's neat to see the company grow and evolve.

We're bullish: it's clear (to us, anyway) that Docker and its container model are the future -- and we believe that our Triton stack is very well positioned to be the stack of choice for Docker in production. Of course, we're still a startup and we're competing with giants the likes of which you mentioned -- but giants can fall. Indeed, they always have...

Is Illumos KVM still tracking Linux version 2.6.34?

We track it for security issues, but no more -- our implementation has diverged relatively significantly, both in terms of KVM and QEMU (where we deliberately jumped off the train when we decided that we didn't like their direction). So the primary thing we react to with respect to KVM and QEMU is our own deployment -- and it's been very, very stable for quite some time now.

Who was the female in the April Fools Day video?

Could you give me a little context here? It's entirely conceivable that I have missed a meme...

Apparently you aren't the same Joylent. http://www.joylent.eu/products/joylent-green#

Well that explains that! No, that is emphatically not us -- we are Joyent not Joylent...

Would you rather fight one horse-sized duck, or a hundred duck-sized horses?

If it has to be said, this one was much better.