Salutations ladies and gents,

Today marks the 2-yr anniversary of my last IAmA, so I figured it might be time for another one.

I wear many hats at reddit, but my primary one is systems administration. I've dabbled in everything from community stuff to legal stuff at one time or another.

I'll be here throughout a good chunk of the afternoon. Ask away!

Here's a photo verifying nothing other than the fact that I am capable of holding a piece of paper.

Edit: Going to take a break to grab some food. I'll be wandering in and out to answer more throughout the next few days. Thanks for the questions all!



Comments: 4016 • Responses: 42  • Date: 

leontes1978 karma

Is it true that you guys can just give gold to anyone without having to pay for it?

alienth1952 karma

Yes. But asking for it or hinting about it definitely doesn't help your odds.

burgerforhire1700 karma

Since your job is to be on reddit all day, do your bosses ever catch you filling out spread sheets or filing earnings reports?

alienth695 karma

I'm sure /u/yishan would fire me immediately if he saw me toiling over spreadsheets.

mrbarkyoriginal1081 karma

Which sub gives you the most day to day headaches?

alienth1280 karma

I'd rather not name names. It is probably ones that you don't expect.

There are folks on the site which take joy in spending as much time and resources as possible to cause as much real-world damage as possible. It is an unending struggle to keep them at bay. I'm not going to name those folks, as doing so would only feed them.

wildmetacirclejerk868 karma

should be under /r/InternetAMA as per karmanauticals rules

sorry boss man, i love the site :)

Also, is chooter replying for you?


alienth251 karma

If /u/chooter was replying for me, my answers would likely be much less rambly and much more interesting.

ffigeman635 karma

Is there anything that counts as NSFW for you?

alienth813 karma


RuskiesInTheWarRoom507 karma

In your view, how has the employee culture and attitude changed in the wake of the very high profile negative press Reddit has received via the /u/violentacrez affair, the /r/findbostonbombers debacle, and the recent seddit kickstarter controversies (and others)? Are there any clear, overt indications or movements or pressures by Reddit (big-R) to deal with these types of things before they arise?

How do you feel these types of events have changed the communities themselves as Reddit grows and changes (from the good ol days)?

alienth456 karma

Well, those incidents certainly have been learning experiences, as stressful as some of them were.

Obviously we would like to step in and address situations before they blow up. We're very passionate about the site and when bad shit happens, we feel horrible about it. Some stuff can't be anticipated, some stuff can't be prevented, and some stuff we purposefully stay out of to try to maintain as much neutrality as we can (despite that being very painful at times).

There have been no pressures from "on high" regarding dramatic situations. We the employees are in charge of our destiny, and we make the calls for what needs to be done to address problems. Since we are in charge of our destiny, that also means that this stuff falls directly on our shoulders.

I hope that the community has learned from some of these situations so that they can be more cognizant of some of the real-world consequences of actions on the site.

snoharm104 karma

With the constant community turnover that comes with being one of the largest social media websites on the internet, the community can't really be relied on to know the history of our embarrassing moments. That leaves responsibility to keep things sane with subreddit mods and admins - when new subreddits pop up for something akin to the Boston bomber hunt, will the Admins be stepping in in the future? What did you learn from that incident and how has it effected your policy, written or unwritten?

alienth41 karma

With the constant community turnover that comes with being one of the largest social media websites on the internet, the community can't really be relied on to know the history of our embarrassing moments.

True enough. One point I'd like to bring up is that we won't grow forever. There will be scenarios where lessons are ingrained into the community mindset. Even if only 5% of users are able to recall, they can make a difference.

That leaves responsibility to keep things sane with subreddit mods and admins - when new subreddits pop up for something akin to the Boston bomber hunt, will the Admins be stepping in in the future? What did you learn from that incident and how has it effected your policy, written or unwritten?

God forbid another incident like this occurs, we will obviously use our experience from this last incident to help us make a decision. Every situation is different, so I can't begin to imagine how the lessons of the past might apply to the decisions of the future. All I can say is that we will use the knowledge this past experience combined with a heavy amount of judgement when deciding what is necessary. We obviously want to maintain our neutrality, but there is a threshold where we may be required to step in.

EvilBosom345 karma

What's your favorite piece of Reddit history? ex. Today you tomorrow me, cumbox, or any significant thread

alienth801 karma

I'm having trouble finding the thread now, but there was a post after the earthquake in Japan where someone was worried about their grandmother (maybe mother?). Someone in the thread indicated he lived nearby, and he went to check on her and brought her some food.

Can't find that thread.. someone help me out here :P

Edit: /u/hoikarnage found it here. Followup thread is here. Thanks!

smooshie340 karma

IMO, one of the major failings on Reddit is the moderator system. The top mod is, for all intents and purposes, king. He can close a subreddit visited by the President and Bill Gates on a whim, he can make the default news or political subreddits omit any news he doesn't deem worthy or relevant (or which goes against his political bias), he can do absolutely nothing while a gigantic subreddit withers away (so long as he merely logs in once every 2 months), and he can make a once-friendly subreddit a hive of hate and bigotry.

The various solutions I've seen proposed (users voting for moderators, all the moderators voting on policies, etc) have their own failings , but are there any discussions about revamping the mod hierachy/power system, particularly for default or large subreddits?

alienth362 karma

I think one of the primary things we need to do here is take some of focus off of the defaults. Right now the defaults define reddit for many people. There is a vast amount of non-default content and discussion out there, but so much focus goes into the defaults that they've become a defacto standard.

If we can make the front page a bit more dynamic and make it easier for people to quickly discover what other subreddits exist, I believe the site will be healthier overall.

Sorry for being a bit vague. We're aware that the current situation is less than ideal and we're working on improving it.

RockinOutCockOut316 karma

Theoretically, if the NSA started fucking Reddit, what safe-word would you use to let us know?

alienth656 karma

Foliage. (christ I hope i don't use that by mistake now)

jderm1308 karma

When you say "at reddit" do you work from home or reddit HQ?

alienth408 karma

Both. I primarily work at the office in San Francisco, but I also work from home somewhat often. I have a considerable commute, and I like working in solitude at times :)

whatsaphoto83 karma

How big is the reddit team? Do you collaborate with different companies often?

alienth117 karma

I think we're in the mid-20s now. There are around 10 folks that focus on the tech side.

I don't do much collaboration with companies. If I'm speaking with another company, it is typically a vendor (Amazon, Akamai, etc).

roastedbagel288 karma

Your alien looks like he was stepped on :(

What's the most nervous you've ever been during a time Reddit has gone down? What were the ccauses and circumstances, and what did you do to correct the issue?

Edit: I can't decide what to flair this post as, either science/tech, other, or nsfw just so you get a cutsie little purple alien winking.

alienth593 karma

The Obama IAmA. I actually developed a good eye twitch from that which lasted a few days.

When we were working on it, solving one bottleneck caused people to flood into a different bottleneck, tanking the site down once again. It was a marathon of wincing.

We learned a lot in the end, so I s'pose that is good.

milop01281 karma

Whats your favourite subreddit?

alienth455 karma

Lately I've really enjoyed keeping up with /r/redditdayof.

tuniltwat272 karma


alienth446 karma

How did you gt your job at reddit.

I saw a job posting in the blog subreddit, and applied. I wasn't looking for a job, but I didn't to pass up the opportunity. I was living in TX at the time and the concept of moving to San Francisco seemed completely irrational at the time. Luckily it all worked out in the end :)

What do you mainly do?

My primary focus is on the infrastructure of reddit. I take care of the servers. Have a more detailed answer to this elsewhere in the thread.

How are you guys going to improve reddit in the future?

One thing I'm really looking forward to is multireddits (currently in beta for gold users). I had 500 subscriptions and a single front-page was less than ideal for that scenario. Using the multi feature I'm able to split subreddits up and categorize them. It's the biggest change we've done in a while, and I'm very excited to see what people end up using it for.

Don't you get tired of reddit and how much time do you spend on the website every day?

The time I spend on the site varies from week to week. I do occasionally get weary for different reasons, and just focus on the behind-the-curtain stuff for a while. Other times I'll spend a considerable amount of time on the site. All depends on my work load and mood for the week.

donaldgately90 karma

You work on infrastructure, servers. What's up with the servers anyway? Seems like they are overloaded regularly.

alienth31 karma

Kinda answered here.

Hash3m268 karma

How do you procrastinate?

alienth190 karma

I tend to procrastinate on things I need to do at home by working.

For example, I took a few days off recently to do some spring cleaning, and I conveniently ended up working all three days.

Clutch70193 karma

First off, ty to the admins on the domain level ban of Quickmeme. Manipulating our community ain't cool.

How much stock is really put into mod evidence gathered when tending to matters such as this fiasco?


alienth123 karma

Whenever we get reports of something shady going on, we have to independently verify it before taking any action. There are obviously some sources which tend to be more reliable, but regardless we have to investigate it ourselves.

Dreijer_154 karma

What is your favorite website that isn't Reddit?

alienth511 karma

Google Reader :( Oh, and TeamLiquid.

window5153 karma

Is Reddit paid for some of the AMAs that it hosts?

alienth276 karma

We are not, nor do we ever want to. We do offer advice to anyone that wants it, but we do not accept money.

Tim-Sanchez53 karma

What kind of advice do you offer?

alienth121 karma

We try to give folks an idea of what they're getting into. Many of the notable people that do IAmAs are completely unfamiliar with reddit, so it is a very foreign environment.

We try to encourage people to engage in discussion instead of looking at it solely as another marketing loudspeaker. The people that engage in actual discussion with reddit users tend to have the best IAmAs.

ConcreteFox124 karma

Do you believe Reddit is currently at, or has past, its prime? If not, when do you see Reddit reaching its peak? What will cause its ultimate downfall?

alienth94 karma

I dunno, that's like asking if human society is past its prime or not, and what will cause its ultimate downfall. I really have no clue.

At the present, the site is still growing. That's all I can say for certain :)

peteflew117 karma

Do you reddit at work when you're working for reddit?

alienth158 karma


indignant_meerkat115 karma

What would you say is your greatest contribution to Reddit?

alienth219 karma

Getting through the outage-filled months of 2011. Things were in a very bad way during those times. A tonne of technical debt was piled up, and working through it was very painful. At the end of 2011, I managed to get things in a somewhat stable state (with considerable help from the dev team). Stability is still pretty far from where I'd like it to be, but we've come very far, and we're continually working on improvements.

One of the things I'm really proud of is the SOPA/PIPA examination blog post. I was really disappointed in the hyperbole flying in both directions, so I wanted to take time to pull the text apart and explain what was being said. It was far outside my usual comfort zone, which I liked. It was also exhausting, since I'm by no means proficient in that area :)

jerseyroller107 karma

What kind of infrastructure requirements does a site of this magnitude have? I can imagine the amount of rack space, servers, switches ETC are off the charts.

alienth134 karma

The site is entirely hosted on AWS. These days we're clocking in around 350-400 instances of varying sizes.

We use many different pieces of tech to keep running. To name a few:

  • Postgres
  • Cassandra
  • memcached
  • haproxy
  • nginx
  • rabbitmq
  • zookeeper
  • hadoop
  • gunicorn

hemite41 karma

What do you guys use hadoop for?

alienth33 karma

Traffic stat processing, mostly.

OMEGA_MODE95 karma


alienth215 karma

I think the site has become fairly influential in certain crowds. When the President takes notice, it is obviously an indicator that we are nearing mainstream. We're still obviously orders of magnitude away from Facebook/Twitter scale.

All of our revenue comes from ads and gold.

SomeRandomFruit85 karma

What do you do, exactly? (no offense intended)

alienth116 karma

Day to day I manage the infrastructure of reddit. This entails keeping an eye on all of the servers, putting out any fires, building out new infrastructure, and planning for the future.

belgarion8971 karma

Have you ever had to literally put out a fire?

alienth15 karma

Only the ones started by my counterpart, /u/rram. My work desk is covered in a canvas, so I have to watch out when he's waving lighters around.

Dalanzadgad74 karma

Neutrally speaking, what do you think about Reddit being primarily political supporters of liberal or progressive causes? What kind of culture on reddit do you think contributed to this overall political stance?

alienth154 karma

I think that a good chunk of this is a natural result of the demographic which visits the site. When you have a majority of young tech-inclined folks, it seems to me that the political average tends to swing to the left. Of course this then attracts other folks from the left, so it can be a bit self-feeding.

I imagine the majority of reddit will probably lean in that direction for some time, given current trends. Indeed the internet as a whole seems to lean in that direction. Time will tell if this shifts at all.

tamammothchuk67 karma

Is there a comprehensive list of subreddits somewhere?

Edit: I reddit using only my mobile....sorry if this is a dumb question.

alienth153 karma will give you a list, but you have to paginate through it so it isn't ideal.

An alternative for a full-ish list is (/u/Deimorz is now going to murder me for linking to a page which is likely quite heavy)

EagleScout91163 karma

Is systems administration fun?

alienth95 karma

I enjoy it quite a bit :) I like fixing things, and I like fixating on things. When an interesting problem pops up, it is quite interesting taking it apart and learning exactly what is going on or how to solve it.

LittleMizz59 karma

Is there a reason for you guys not expanding faster? Several times a day the "servers are slow"-message on Reddit, and I mean, it's not like Reddit is some independent site worked on by college kids with their lunch money.

alienth93 karma

We can expand the servers as fast as we want. Unfortunately that doesn't really solve the problems. If simply expanding the servers would solve all of these issues, I would kick up everything this afternoon :)

Right now our biggest struggle is that some layers of the site are no longer holding up as they used to. The app has an extensive internal-caching system which has served us mostly well for years. However at our current scale it is beginning to cause a lot of problems. What sucks about this is that we have to completely re-develop those pieces of the app to solve the problems. Just developing those solutions takes a considerable amount of time. Additionally, trying to figure out a solution which will get us through the next set of years takes an incredible amount of thought.

Overall we've come a long ways from where we were in 2011, in terms of stability. Obviously we have a long ways to go, and the past month or two has definitely been challenging. All I can assure you of is that we're spending a huge amount of time on these problems.

LittleMizz18 karma

Gotcha, thanks for the answer! Also, are there any plans on making an "official" iOS/Android-app for Reddit?

alienth70 karma

Our current strategy is to enable other app creators to flourish. We don't have plans on our own app at this time. Obviously that could change in the future.

IMO AlienBlue is by far the best iOS app.

olskeywolskey56 karma

What is your salary? (Are you having second thoughts about AMA instead of AMAA?)

alienth141 karma

You can ask me anything; doesn't mean I have to answer everything :)

idkwhattoputasmyname55 karma

Do you guys have any control over individual subs? Like can you monitor them and take them down if necessary? What's your least favorite sub?

alienth89 karma

We have the power to take down anything on the site. However we leave most of that decision making up to the moderators. The mods call the shots on what is acceptable in the subreddits which they manage. Obviously there are some cases where we must step-in, but overall it is extremely uncommon.

jollyjack48 karma

What stuff do you guys do behind the scenes to make our Reddit experience better, that we don't know about?

alienth99 karma

To quote a coworker: "Like the night janitor - you know that someone is emptying the garbage cans, but you don't really think about it".

We spend a huge amount of time going after spam, vote botting, and various other evildoers. Bad shit inevitably slips by, but the stuff that does is the tip of the iceberg.

sysvival22 karma

Fellow sysadmin here. i have a couple of questions regarding your network.

-I read somewhere you use haproxy for load balancing is this still true?

-What software do you guys use to analyze your webserver logs? If you use logfiles from your LB's, how large are the logs from a 'normal' days usage?

alienth26 karma

Yep, haproxy all the way. It has been good to us.

We have around 200G of various logs a day. It is all shipped off to S3 and then deleted after 90 days.

Bronze_Wizzard19 karma

How many cats do you own?

alienth24 karma


inrouted1515 karma

Being a systems admin and having to put out fires I would assume that you are very intimate with the inner workings of EC2.

  • Has using EC2 (or other mix of cloud architecture that I do not know about) made it possible for reddit to flourish?

alienth21 karma

reddit has been operating on a very lean staff since inception. Shortly after I joined the tech team consisted of 2 sysadmins and 1 dev. I don't think we could have survived if we also had to worry about managing the physical resources of our own infrastructure.

Obviously any platform-as-a-service product has its pros and cons. I think since we moved to it it has been positive overall, given our requirements (crazy fast growth with very small team).

dgrocksolid13 karma


alienth83 karma

I am degree-less.

PanchoLopez1013 karma

Did NSA ask Reddit for information like they did to facebook and google?

alienth24 karma

We're not involved in that program. We can be compelled to turn over stuff with a subpoena, but we fight tooth and nail if a request is overly broad or bs.

Of course, this turns into a chicken-and-egg of whether you believe me or not. I've expounded on this heavily elsewhere, so you can dig through my comments if you'd like.