I am a reddit employee - AMA

Can you tell Reddit it didn't take too long to make that page for me? I feel bad every time that screen appears. Poor Snoo :(

Yeah, that error page is a bit of misnomer unfortunately. The text is overly broad. It's on my to-do list to change.

How many lines of text is your to-do list? (estimate if needed)

161.

Do you slack on the job and go on reddit like the rest of us?

Definitely. I have to consciously tear myself away from some threads. I also have a monumental to-do list, which makes the idea of slacking off and surfing around reddit very tempting.

What is the biggest "Oh shit" moment you can think of that you had to deal with in regards to reddit?

Also, can you describe reddit in 2011 in 5 or less words?

What is the biggest "Oh shit" moment you can think of that you had to deal with in regards to reddit?

1. "The president is unable to access the site."

2. "We're requiring all employees to move to San Francisco."

Also, can you describe reddit in 2011 in 5 or less words?

Hanging on by a thread.

Ha, who told you the first one? I remember reddit dying during that.

/u/hueypriest

Whats a rough estimate for how many servers Reddit uses?

We run 300-400 EC2 instances during peak hours.

Any chance of a rundown of what they are? How many web servers, databases, reverse proxies, etc?

Also, how often does syncing occur between databases? Would you be able to explain the process that you guys use? As a web developer that's never had to sync anything, I've always wondered what is the correct way of doing so.

Just ran the numbers.

230 app servers

73 memcache servers

16 postgres servers

15 cassandra servers

11 load balancers

5 asynchronous job processing servers

~30 other random infrastructure servers

are you running memcache on EC2 or using elastic cache? followup... if you are using ec2, what was the driving force for that decision... if you are using elasticcache, how has the service layer treated you?

We're running memcache on EC2. Elasticache isn't a viable option due to how we depend upon memcache. Our memcache use is a rather core portion of the stack, and our use of it puts some unusual constraints on it. For example, forgotten writes or split brain will actively result in data corruption on reddit.

We're moving the memcache stuff to mcrouter. You can find a talk about mcrouter here. One of the reddit sysadmins, Ricky Ramirez, gives a talk about half-way through the video on our use of memcache.

When Yahoo decided "no more remote working" there was a fair amount of backlash in techie circles that this was pretty silly and backwards for a tech company.

Speaking for myself, I do "web stuff" for a living, and despite being not even 10% the standard to work at a high-profile / high-traffic site like Yahoo or Reddit (therefore not exactly in a position to dictate terms), I personally feel any company with a "no remote working ever" policy would entirely dissuade me from applying.

Firstly - I genuinely feel remote working makes me more productive for my employer as well as enabling much better "work/life balance", I can't imagine giving it up from a selfish perspective.

Secondly - I feel like that sort of stance is just a 'red flag' in a company. It implies to me likely inflexibility with employees in other matters, and a kind of 'defeatism' around online communications that is highly ill-fitting in a company based around the same - i.e., if a company does not believe they can get even a few dozen employees to positively and productively interact without being in the same place, how do they suppose their platform can support millions of users positively and productively interacting from remote locations?

What is the reddit counterargument to all this sort of rambling?

As a remote worker myself, I'm likely a bit biased on this matter. I too have found that I tend to be more productive when working remotely. However, I will say that being under one roof has benefits which cannot really be replicated with remote work (at least, given current technology). Being in the room with your coworkers to just be within earshot can provide a huge amount of benefit, in certain situations.

Firstly - I genuinely feel remote working makes me more productive for my employer as well as enabling much better "work/life balance", I can't imagine giving it up from a selfish perspective.

Interestingly, my work/life balance has gotten worse since I started working remotely. When I work from home, I tend to become exceedingly focused on work, to the point where I'll forget to eat or drink for many hours. Pulling me out of that usually requires some concerted effort. Additionally, the lack of physical separation from work and home creates a weird sort of mental state that I find difficult to coherently describe :)

Secondly - I feel like that sort of stance is just a 'red flag' in a company.

I think that's a fair concern, if you know nothing more about the company. It's just one of those things you have to counter balance against the other aspects of how a company works. While I think allowing remote work can often be a positive, I personally wouldn't turn down a company which disallowed it without weighing all of the other factors first.

In regards to flexibility, I will note that we are rather flexible on allowing folks to work from home when needed. When I did work at the San Francisco office, I tended to work from home 5-6 days a month (sometimes more).

What is the reddit counterargument to all this sort of rambling?

I think the argument which reddit, inc has made is that remote work works well for some companies, but not for others. The company has taken the stance that we can improve by moving everyone under one roof. While I don't necessarily agree with that stance (again, I'm biased), I think time will tell whether the outcome is positive or negative for the company.

But will they grant you a relocation package, covering all costs to move from Alaska to SF?

As I commented here, relocation packages are being made available to all employees who move. I'd rather not reveal further details.

How has being an employee changed your thoughts on Reddit and do you still use it the same way an average Redditor would?

This is a great question.. one that I could probably talk about for a very long, rambly time. I'll try to jot down a few snippets for the sake of brevity.

After being immersed in this world for nearly four years, I have come to learn the following things about reddit, people, and internet communities in general. Note that many of these things have quite a bit of nuance, and they don't apply universally.

• The kindness of strangers meeting on the internet will surprise you.

• Likewise, the degree to which some people actively pour effort into being unkind will surprise you.

• There are people who will literally spend 16 hours of their day on reddit, every day.

• It is incredibly easy to be a cynic, and it is incredibly difficult to counter cynicism.

• It is equally incredibly easy to be a pedant, and the internet tends to reward pedantry.

• If two events occur alongside each other, people tend to believe there is no possible way they could be unconnected, no matter how little information they have about the events.

• People tend to absorb information like they're watching a movie or reading a novel. No event in a piece of fiction is insignificant, but sometimes random shit does happen for no apparent reason in real life.

• There are companies in existence which think highly of their employees, and allow them to generally speak freely.

I'll add a few more things as I think of them. Looking at that list, I realize that most of the stuff is negative :) There are plenty of positive things I have learned, they're just harder for me to put into words.

reward pendantry

Did you mean pedantry?

Isn't it ironic? ^:)

[deleted]

I wrote a bit about that here. I wasn't awake for the decision itself, but I agreed with it (for reasons indicated in that post).

Here's a post he made when it happened that goes into great detail.

TL;DR reddit's thumbnails made it a legal liability to have links to those pictures.

I think a tl;dr doesn't do service to the nuances of the situation. I explicitly chose to not include a tl;dr on that giant post because of this. I can say that what you've described is not the entire story. Thumbnails were a part of the problem, but not the end-reason for the ban.

For those of you that haven't read it, and want to more fully understand the situation, I'd highly recommend you do. You don't have to agree with it obviously, but it will hopefully provide some fuller context.

What are the specs of your GabeN Worshipping Device?

Some actually asked me that very recently. Here is the answer (copied from another thread):

This is my baby. It's about 3 yrs old now: http://i.imgur.com/HlCg6.jpg

• Corsair 800D case.
• GTX 580 with the factory waterblock.
• Gigabyte mobo with Z68 chipset.
• Intel I7 2600K
• RAID-0 Crucial SSDs

Single loop watercooling, with the following parts:

• Bitspower compression fittings
• EK CPU waterblock
• 120mm Gentle Typhoons on the radiator
• XSPC Dual 120mm radiator
• XSPC Dual 5.25" reservoir with the MCP655 pump.

Been pretty solid.. except.. the GTX 580 does crash if I don't push it slightly over factory voltage. I RMA'd it twice and it continued to happen. So, I've been running it overvoltage for a couple yrs now :P

The MCP655 died within the first year - impeller seized. I was pretty surprised that happened since those pumps are usually solid.

Have you pulled off the plastic cover yet?!

Nope.

Do you think the $50 million is going to change reddit culture? Do people think this is the beginning of the end and that reddit will wind up like so many other online communities that fell by the wayside?

Do you think the $50 million is going to change reddit culture?

In terms of corporate culture, my opinion is that a company going from <10 employees to 60 has a much more drastic and inevitable affect on culture than fundraising does (at least, in our case). The way people work in small groups does not scale to large groups without making at least some adjustments.

The money will obviously give us resources to do more things, which also has an effect on both corporate culture and site culture. When you have an excess of resources, you act differently than when you had limited resources - this is one of the foundations of economics. I hope that the extra resources will have a positive effect on the community, namely by doing things like hiring people dedicated to finally fixing the goddamn mail system.

Do people think this is the beginning of the end and that reddit will wind up like so many other online communities that fell by the wayside?

I would sincerely hope not. Any investor in reddit worth their salt would recognize that the value of reddit lies within the community. If you leave the community by the wayside, you piss away your investment. We sought investors that recognize this, and I think the ones which were chosen "get it". I'd recommend reading the Sam Altman IAmA to get some additional insight into this.

What are some of the best benefits of working for Reddit ?

I have some amazingly talented and interesting coworkers who I'm deeply privileged to work alongside.

To point out one example, the relationship that the sysadmins have with the dev team at reddit has been amazing. I cannot stress this enough: at most companies, dev teams and sysadmin teams are usually at odds with each other. At reddit, we work as one. The devs are my peers. When shit is broken, we all work on finding the cause, and don't waste time blaming the infra side or the development side. This is an incredibly rare relationship, one which I hope will continue as long as reddit exists.

As a dev whose team is constantly at odds with our sysadmins/others in IT, can you provide maybe some insight as to how your two teams work together so well?

Like what are some of the things in particular that help promote that environment?

I think a primary reason this happened is the foundational setting of the team. We were all seen as the reddit admin team, keeping the site alive, and we all struggled together to keep shit running during some extremely unstable years. There was no organizational separation, and we all got along really well. As the company has grown, we've been lucky enough to maintain this.

I've often pondered how to take a sysadmin and dev team at odds and get them to work the way we do. Unfortunately it may really come down to the foundations and working relationships of the team members, which you can't really force. I've worked at places that had dev/sysadmin schisms where a huge amount of effort was put into having us get along and work together, but it rarely materially improved the situation - the attempts were just too forced. We'd all go back to our respective groups and resume the normal non-cohesive working relationship.

I've never managed a large group of people, so I have no idea if this would work. But, forcefully merging the two groups, so they had the same reporting structure, and worked in the same area, may have a positive result. It would be incredibly painful, and there is a non-zero chance for team disaster. If anyone has tried something like this with dev/sysadmin teams, I'd be curious to hear how it went.

I read that Jared Leto and Snoop Dogg recently invested in Reddit.

1) is this true?

2) will this change anything for you?

Edit: lego my leto

1. They were part of the recent investment round.
2. This is a very broad question :) I don't expect that their specific involvement will change much. You can read some thoughts by the leader investor, Sam Altman, in his IAmA.

I guess I don't have a question, just something I have wanted to say to you. I really appreciate the way that you handled the "explanatory" post about the fappening and the shitstorm of comments that followed. I was going to PM you but I figured you were being buried in messages. You handled that completely professionally. It was impressive. Cheers.

So I guess that does lead to a question. Is your job really so great that it outweighs the avalanche of negative bullshit you get from the people you are essentially working to please?

Edit: This was directed at /u/alienth ;if there is any confusion

Thank you, I appreciate it.

Is your job really so great that it outweighs the avalanche of negative bullshit you get from the people you are essentially working to please?

The job is incredibly challenging, and I love being challenged. The stuff I do at reddit I wouldn't get to do at any other sysadmin job anywhere else. The negative bullshit is a pain, but it has allowed me to hone some useful skills.

I regularly talk with my peers that are in standard corporate sysadmin jobs. They generally don't have to deal with stuff like talking to a community, putting up with some absolutely insane users, or getting harassed with a deluge of comments telling them to kill themselves. Still, I don't think I'd ever trade this job for a standard sysadmin job. It is a horrible thing to watch amazing talent in other people be shackled to the mediocrity of corporate life.

How was the lambeosaurus selected out of all of the other possible dinosaur candidates?

Because the lambeosaurus was the narwhal of the dinosaur age.

I will tell you the deep, dark secret behind the lambeosaurus post.

Back in January, I suddenly realized we were at the very end of the month, and we hadn't made one blog post. My initial thought was to try a post which arbitrarily declared one thing to be the best of many things. I realized that most options for the 'thing' would result in a huge amount of argument. I couldn't declare the best video game, the best food, the best musician, or the best anything where people were likely to flood in with their own deeply held opinions. Dinosaurs turned out to be something silly enough. Declaring the best dinosaur on the reddit blog would cause almost everyone to immediately realize we were just being silly.

After that point, I just started searching for some relatively unknown dinosaur that looked a bit silly and had a silly name, and stumbled upon the pinnacle of all dinosaurs, the Lambeosaurus. I asked /u/powerlanguage to create me a graph in the shape of the lambeosaurus, and things went from there.

This was all done in the space of a few hours, and stands out as one of the most enjoyable days I've had at reddit.

Will all of your moving expenses be covered by Reddit, or are you responsible for a part of (or even all of) the relocation?

Relocation packages are being made available to all employees that are affected by the required move. Each person's moving situation is different, so I'd rather not share further details.

I will say that from my perspective, the company is making a concerted and deliberate effort to keep me onboard and assist with the move.

Will you be receiving any sort of raise aside from the package to compensate for, for the lack of a better term, fucking retarded cost of living in SF?

Yeah, as I commented elsewhere, folks moving will be receiving cost-of-living adjustments. It would be impossible to move otherwise, given the insane cost of SF.

Of course, moving from some areas you can't expect to have a similar quality-of-life in SF. It is reasonable for folks living in the midwest to own a house with a yard. That is not something which is viable for most people in SF.

How do you guys feel about comments that this is a stealth lay off,knowing lots of people won't make the move or can afford to?

I feel that those commenting on the situation are assuming that is the case based on the following evidence:

1. Most shitty companies will use these type of tactics to lay people off, and shitty companies are common.
2. We recently had $50M raised for investment, and investment rounds are often tied with corporate restructuring (and layoffs).

I completely get this line of reasoning. From the outside, it is incredibly difficult to imagine how this couldn't be what is happening here. However, I can say that internally, knowing what I know, this is absolutely not the case. The company is making some very concerted, deliberate efforts into keeping folks onboard. I cannot count the number of times this has been reiterated to us.

I do think that the moving situations sucks. As I commented elsewhere, I'm not looking forward to deciding between moving my family cross country or leaving a site I've put a huge amount of effort into supporting. However, I do recognize that this is not a case of a stealth layoff.

Do you use RES in the office?

I do not.

Why not?

As others indicated, there are security concerns when using privileged accounts with third-party plugins. Additionally, as maintainers of the site, we should be subjected to all of the annoyances that come with the default experience :)

Do you have any early thoughts on the new internal policy limiting candor during admin AMA threads? (it will be drafted/distributed early next week)

Hardy har har.

What are your work hours like? I imagine it can't be 24/7 but are you always on stand by if an emergency happens?

I'm 24/7 on-call. Lately I've been working 60ish hours a week in preparation for my parental leave. I tend to try and keep a normalish business-hours schedule, but I'm awake when the servers need me. As such, I don't have hard requirements on what time I work.

Congrats!!

On the baby. Not the overtime. Hope that was clear.

Thanks :D

How did you pick your Reddit username? any cool backstory?

Sorry, it's a bit boring. When I was a youngster, I came up with the handle 'alioth', after slamming random sounds together. Sometime later I signed up for some site where that username was taken, so I derived 'alienth' from 'alioth'.

I later learned that 'alioth' is reportedly the word for the tail of a fatted sheep in some arabic language, so I guess I dodged that.

As a guy who just got a telescope, Alioth is also a highly visible star.

Ah yeah, I forgot about that. Speaking of, here is the thing about 'tail of a fat sheep': http://en.wikipedia.org/wiki/Epsilon_Ursae_Majoris#Name_and_etymology

How did you get this job? Is it hard or do you just stare at a computer all day? What about when reddit is down, is there a special team to fix it or do you all have to do some kind of coding-fu to fix it?

How did you get this job?

I saw an opening and applied. This was about ~4 years ago now, and the candidacy process was.. arduous :)

Is it hard or do you just stare at a computer all day?

I stare at a computer all day, and it is also very challenging. Such is the life of a sysadmin, doubly so for a reddit admin.

What about when reddit is down, is there a special team to fix it or do you all have to do some kind of coding-fu to fix it?

When reddit is down, either myself or one of the other two folks on the sysadmin team will be immediately alerted. We'll then spend some time figuring out what went wrong, and then hopefully expeditiously fix the issue. Sometimes this requires some adjustment to the code, sometimes it requires turning something off and then turning it back on again.

What is your opinion on the life expectancy of reddit? How many years do you think it has left? Would you call reddit a mainstream website yet?

What is your opinion on the life expectancy of reddit?

I personally think this is an unknowable. The death of reddit is something that could happen. The site exists as a structure to support the community. If the community leaves or becomes disinterested, then reddit will cease.

How many years do you think it has left?

I think this is equally unknowable. Barring some unforeseen circumstance, we're probably on pace to grow for a couple more years. Beyond that it is hard to say. I certainly hope that reddit, or something like it, is around for some time.

Neat! What's the most rewarding aspect of your job?

I get to work on a truly unique site that has an incredible variety of interesting (and at times, deeply distressing) challenges. I can say pretty certainly that I will never get the opportunity to work at another place quite like reddit.

I also get to be part of supporting a community which truly holds a unique place on the internet. Yes, bad things do happen on reddit. A lot of good, uplifting, and deeply interesting things happen, too.

Can I see your data model?

Well, the code is open source, so you can try and dig around there if you'd like.

I will try to give an extremely brief overview of what things look like:

Almost all objects on reddit are 'things'. Accounts are 'things', comments are 'things', and so on. 'Things' are stored in a postgres database, in a separate table for each type of 'thing', with a schema that basically looks like this:

thing table
  Column  |           Type             
----------+-------------------------
 thing_id | bigint                     
 ups      | integer                     
 downs    | integer                     
 deleted  | boolean          
 spam     | boolean             
 date     | timestamp

(The ups/downs even exist for things which can't be voted on; we store arbitrary counters in there for those things).

'Things' have attributes associated with them. Some examples of attributes are an account name, the contents of a comment, and the URL of a link. Attributes are stored in postgres, in a separate table for each thing, with a schema that looks like this:

data table
  Column  |  Type 
----------+--------
 thing_id | bigint 
 key      | text  
 value    | text  
 kind     | text

The other data type we have is a 'relation'. Relations indicate where two things are related. For example, when a user subscribes to a subreddit, they get a relation linking their account 'thing' to the subreddit 'thing'. The relations are stored in postgres, with a separate table for each relation type, with a schema that looks like this:

relation table
  Column   |           Type              
-----------+------------------------
 rel_id    | bigint                       
 thing1_id | bigint                      
 thing2_id | bigint                       
 name      | text                         
 date      | timestamp with time zone 

Relations also have data attributes. For example, a relation between an account and a subreddit has an attribute indicating what permissions that user has on the subreddit. Relation attributes are stored in a table identical what the 'data table' looks like from above, except instead of cross-referencing with a 'thing_id', we cross-reference with a 'rel-id'.

90% of the canonical data on reddit is stored in the above model. All of the stuff from postgres is objectified in the code when we read it, and those objects are automatically stored in memcache for fast retrieval.

Most of the rest of the data we store surrounds the denormalization of canonical data. For example, the list of links on your user page is a stored in a denormalized relation. Almost all of these type of denormalized data sets are stored in Cassandra, and the data models vary quite a bit. We have around ~10TB of data stored in Cassandra. Here are some of the column families we have in cassandra. Their names will give you an idea of what they do:

LinksByAccount
LinkVotesByAccount
MessagesByAccount
GildingsByThing

And that is a brief rundown of most of the data models in use at reddit.

So if you upvote a post, then three hours later you remove the upvote, is there any record that those actions ever occurred?

There is now, but that may be changing in the near future. When you take back an upvote it becomes a 'none vote' in the database. That portion of the database is being changed, primarily because it is no longer tenable to have 5TB of PG databases dedicated to votes :P

Would you be so kind as to paste (and describe if not obvious) whatever is in your clipboard (buffer) right now?

It is some comment I was working on in this thread :)

Slightly more interesting is the last few lines in my terminal history. (I've removed some private lines).

Line 12646 is where I counted my todo list for a question in the IAmA.

12622  2014-10-03 18:23  vi ~/notes/scratchpad
12623  2014-10-03 18:23  git diff --cached
12624  2014-10-03 18:24  git commit -m 'scratchpad update.' notes/scratchpad
12626  2014-10-03 18:24  git status
12629  2014-10-03 18:24  gpg --list-keys
12636  2014-10-03 18:25  gp origin
12637  2014-10-03 18:26  vi ~/notes/scratchpad
12638  2014-10-03 19:10  gd -a
12646  2014-10-04 16:01  cat notes/scratchpad | grep -v '^$' | wc -l
12648  2014-10-04 17:10  cheese
12650  2014-10-04 17:10  sudo apt-get install cheese
12651  2014-10-04 17:10  cheese

What is "cheese"?

It is the program I used to take the verification photos. Just a simple webcam program.

Alienth, you are one of my favorite admins and I really appreciated your follow-up post on the whole Fappening fiasco. My one question is,

Have you ever driven down the entire Alaska Highway?

Thanks!

I have gone up and down the AK highway a few times now. Twice when I was a youngster, and once with my wife about a year ago. It's a fun drive, but can be stressful. For anyone considering it, I've posted a compendium of tips over in /r/alaska.

Which coworker has the cutest dog?

I'm allergic to dogs. As such, the answer is a null set.

A null set? Which one? :)

(This phrase, instead of the usual "the null set", leapt out at the mathematician in me.)

This is my null set. Many are like it, but this one is mine.

Do you prefer the weather of Alaska to SF?

Yes, considerably. I was born and raised in Alaska. It is my home. The cold weather can be annoying at times, but I love it.

Do your fellow employees have any competitions surrounding karma? If so, /u/drunken_economist has to have a slight advantage.

I often can't remember my karma. I only tend to be interested in it when it is nearing a mildly pleasing number, like a palindrome or power of 2.

We joke about karma on occasion, but it isn't something I've ever put any amount of effort into.

Are you hiring?

Yep, see /r/redditjobs.

What are a few of your favourite subreddits?

I love /r/AskScience and /r/AskHistorians, and I think they are pretty unique places on the internet. There are so many folks in there who have dedicated their lives to various areas of study, and seeing responses from them is incredibly interesting. It's the same reason I'm a fan of the Numberphile and Sixty Symbols youtube shows.

On the humorous side, I really enjoy /r/cablefail, /r/osha, and /r/techsupportgore.

For the interesting pictures side, I'm a fan of the SFWPorn network, the Imaginary Network, and places like /r/TheWayWeWere.

How big of a problem are bots on reddit? The bad bots not the good ones.

Eh, I tend to think that which bots are 'bad' is something the community determines. If a bot is being a pain, moderators will ban it, and users will downvote and report it. If we notice a bot that is being greatly annoying and being banned or reported like crazy, we'll step in and take action.

That said, I think we do need a better way of managing bots. There has been talk internally and in the community of a sort of 'bot registration' system. I'm not sure if that's the route we'll take. It's something we have yet to figure out.

If you need beer recommendations in San Fran just let me know.

I'm not local but I know who's making the sexy suds.

Wait this is an Ama so I need to ask something...

Why Alaska? What do you like about Alaska? What do you do on your down time in Alaska? Do they have good pizza up there? Do they put salmon and stuff on pizza? That sounds gross. What do you do when it's either dark or light for a super long time?

<3 Praise be the Alaskan admins and their ice lord skills

Why Alaska? What do you like about Alaska?

I was born and raised here, so I'm sure that makes up for a significant part of the reason. It's relatively quiet, I like the weather, and I have family and friends close-by.

What do you do on your down time in Alaska?

Most people do hunting or fishing in AK. I've never done either in my life. Occasionally my wife and I will drive around, up to hatcher's pass or other places just to chill. I usually spend my downtime hanging out with my wife just chilling at home, or playing games with folks. I'm not much of an outdoors person, and I'm completely happy to not leave the house for long periods of time.

Do they have good pizza up there? Do they put salmon and stuff on pizza?

I s'pose. The number of pizza options are pretty limited. There are some places which will put salmon or reindeer meat on their pizza, but I think that's mostly for the tourists. I wouldn't waste a bunch of salmon on pizza, myself :)

What do you do when it's either dark or light for a super long time?

Eh, this doesn't affect me all that much, since I don't do outdoors stuff. I definitely prefer the night, and I work better during the long winter nights.

Any hidden easter eggs or little things that might excite the average redditor that we might not know about?

There are a couple easter eggs, but nothing extravagant. Revealing them would spoil the fun!

How tasty are our tears?

I prefer vinegar.

How many alts do you have?

There are dozens of us! Dozens!

Truth be told, I don't have the time to maintain many alts. I'll occasionally hop on one to make some poor attempt at humour.

[deleted]

I do feel the tinges of burnout at times. I've felt this a lot in my career; it is a common aspect of being a sysadmin. I've consciously worked to recognize when it I'm starting to get burned out, and when I recognize it I try to action to remedy it (when possible). This hasn't always gone perfectly, and sometimes I have worked myself into a situation where I became extremely burned out.

You said you're a remote employee (til the move... that sounds rough, hopefully it turns out well), how do you keep servers running if you're not physically near them? I imagine most of the time it's software related stuff but what about when it's physical things, like something got unplugged, overheated, etc.? Do you have people you call to check on things?

Our infrastructure exists entirely within Amazon's AWS product. As such, I never physically interact with any hardware.

If you weren't doing this, what would you want to be doing and where would you be?

I would probably be cleaning my house, and installing an infant car-seat in my Jeep.

I wear many hats

Do you play TF2?

^{P.S. We need photos of the hats}

I have, but not recently. When I did live in SF, we would play it semi-regularly. It's a fun game, but not something I'd play long-term.

What's a lesser known subreddit you've grown fond of?

I was a big fan of /r/gue back when it was active. I'd really like to see a place on the internet for formal debate.

I'm subscribed to 400 subreddits right now, so it's a bit hard to think of a small one I'm particularly fond of. I like /r/osha, but I wouldn't call that 'lesser known'.

Housing is so much more expensive in 'Frisco. Will you be given a cost of living allowance on too of your salary?

As Yishan has indicated elsewhere, employees moving will receive a cost-of-living adjustment. It would be impossible to move people otherwise, considering SF's outrageous cost of living.

. It would be impossible to move people otherwise, considering SF's outrageous cost of living.

Wouldn't it be wiser to move the group to Kansas then? The cost of living is so little, and Texas is another option as well. Moving everyone to SF makes 0 sense outside of someone just wanting to show off "We work here too!"

I'm not a fan of SF, but I'll admit that it is a sort of epicenter for finding talented tech folks. That isn't to say that there aren't talented people in many places, there is just a decent density in SF. If we needed to hire more people, it'd probably be pretty hard to attract folks to Kansas.

When you're out fishing, you try to find the place with the most fish. ^{disclaimer: i have never fished.}

As a SysAdmin what would to tell to a College Student who's looking into it right now?

I'd recommend reading the sysadmin IAmA linked in the post. There is some information regarding that question there.

Unfortunately I didn't start my career in the same path you're taking, namely because I dropped and started working at 17. As such, I'm not sure how much relevant advice I can give.

The one thing I tend to tell folks is to never leave things up to magic. If there is something you ever don't understand about a system, dig into it until you do understand it. It is a bit of a depth-first approach, and it can be brutal. Still, it is the primary way I learned, and I've found that this method has afforded me an extremely well-honed set of skills.

Will this be your last AMA as a reddit employee?

I'll try to do annual IAmAs as long as I'm an employee.

What is your favourite cereal?

Raisin Nut Bran.

Hi, Jason.

1) I remember reading in a previous thread of yours that you weren't a fan of RHEL. Is that still the case? If so, why? I usually hear it the other way around.

2) How do you make major decisions when it comes to introducing new technologies/systems to the stack, and how do you know if it will be a good fit for reddit? I'm curious to know this since reddit is unique in how it handles traffic. Since reddit is so small, is it something all of the employees are made aware of first and discussed, or do you just go for it?

I remember reading in a previous thread of yours that you weren't a fan of RHEL. Is that still the case? If so, why? I usually hear it the other way around.

Eh, I'm still not a big fan of RHEL. That isn't to say I think RHEL is doing things wrong, I just personally prefer debian-flavoured options. Back in the long-long ago, I think the debian/apt package system was a much better approach than the equivalent RPM solutions. Here is an article from 2003 pointing out some of those annoyances: http://www.linuxdevcenter.com/pub/a/linux/2003/09/11/linux_annoyances.html

The RPM based approach has caught up since then, but I still personally prefer deb/apt.

I do think that corporations running linux are probably best served by RHEL, due to the enterprise support.

How do you make major decisions when it comes to introducing new technologies/systems to the stack, and how do you know if it will be a good fit for reddit?

When evaluating a new technology or solution, it is usually a team effort involving both the dev team and the sysadmin team. Given our scale, it is really difficult to determine if a new technology is going to suffice before we actually implement it in production. For example, a couple years back we started working Apache ZooKeeper into the stack for configuration and locking. It excelled at real-time configuration, but we quickly realized that it was not up to par for locking (we do a lot of locking).

There are other times where the evaluation will be long and drawn out. For an example of such an evaluation, I'd recommend reading this summary I wrote on the project to get us on a new CDN.

How does it feel knowing that your every move is being monitored by your co-workers?

Eh, I wouldn't call it 'monitored'. We keep an eye out for each other, and help out where we can.

What are your favorite and least favorite aspects of working for Reddit?

Favourite can be found here, most frustrating can be found here.

Is reddit substantially increasing your pay as part of relocation?

Relocated employees will be receiving a cost-of-living adjustment, as indicated by Yishan.

When not working do you browse/post on reddit in your personal time? And do you use a different username/account to be anonymous to your coworkers or to disassociate yourself from being /u/alienth?

I do browse in my personal time, and I occasionally will comment and post stuff with /u/alienth. I haven't commented with an alt in a really long time, except where it was blatantly obvious that it was me.

1. What is the best part of being a reddit employee?

2. Do you comment less than before?

1. You can find an answer to that over here.
2. Truth be told, I probably do. I'm in many different projects at reddit these days, so the time I have for commenting or interacting with users has gone down quite a bit.

How do people usually react when you tell them you work for Reddit?

Gets you a lot of ladies, amiright?

Most of the people I interact with already know I work with reddit. If it is someone I don't know, it usually doesn't come up.