We're The Backblaze Cloud Team (Managing 750+ Petabytes of Cloud Storage) - Back 7 Years Later - Asks Us Anything!

Can you send pretty server porn pictures?

How sustainable is your pricing for ‘unlimited’ backup? Are most users only storing a small amount?

Are you looking at/offering cloud compute, or just storage?

Yev here -> What 14 Petabytes of storage looks like, 180TB Pod (old school), Opened Storage Pod

Here's a few to get you started...I'll send more later ;)

Edit (above for cleanup, below for more hot server pics)

Here's some good good cables -> Cable Porn, Cabling Porn

What switches are those in the storage pods pics ? :D

Good question - no idea. That picture was from a while ago (been a minute since I was in the data center)...let me go find out.

Edit* -> Asked the data center team and they think those are Enterasys (but from a long time ago). We now use a combination of: Arista, Dell, and some older Force10s.

Hey, I can see my data from here!

Good thing it's encrypted...

Hah for sure. For what it's worth, I started my own (very small) business late last year and signed up for your service, and I think you guys do a great job.

Thank you so much :D

RMA-able, eh? You can return the goods to my home address, I'll PM you. ;)

Hah - probably way past their bye-bye time :P

Are those Storinators or something custom?

Yev here -> Great question! Those are NOT Storinators. But here's the funny story - Protocase, was our original contract manufacturer for our storage pods. Since we open sourced the design, a few years in, Protocase created a company called 45drives.com and that's where the Storinators are from! So...it's the reverse, these are our "something custom" pods that begot the Storinators!

Edit - typo

Did you ever entertain Cleversafe --> IBM COS for your peta --> exa scale object storage? What are/were your thoughts on their tech?

Yev here -> We've written all of our own code to handle that large of scale (Zettabyte-scale architecture) so switching or using another provider would be fairly expensive for us. Plus we're all about cost optimization, so a lot of existing systems are/were out of the question due to cost. One of our Operations Engineers used to work there though, so that's cool!

The moment I clicked on the first picture, all of my external drives here in my home office spun up.

they know......they know.

they know......they know.

We know.

Those are some serious cables in the Cable Porn photo. Do the cable origin and termination points have to match up, or will the system figure it out?

Yev here -> paging /u/bzelliott for his expertise!

:D

Glad you like it ;)

How sustainable is your pricing for ‘unlimited’ backup? Are most users only storing a small amount?

If you are curious, here is a "histogram" of the "Personal Backup Customers" backup sizes as of December 31, 2018:

https://i.imgur.com/iVEuwUT.jpg

You will need to zoom in to see the information. As you can see, we lose money on a few customers at the high end (we cannot store 430 TBytes of data for only $6/month), but since more customers just want to be reasonable and backup their laptops we are profitable and fully sustainable on the "average".

Holy Cow, who the heck is uploading 430TB of data? I'm guessing linus from linus media group?

Yev here -> hah, think they're using Google Gsuite for that now :P

When you were sponsoring Critical Role, did Sam ever run an ad idea by you beforehand? Was there anything you nixed?

Yev here - Bidet Critter! No, nothing was ever off the table, completely trusted Sam to do a great ad. My personal favorite was the infomercial with Marisha and Tal! Sam was amazing to work with. Crazy creative!

Are you able to share why you guys don't do ads for them anymore? I miss Yev!

Yev here -> Hi! Definitely! I posted this over on /r/criticalrole when my last episode aired -> https://www.reddit.com/r/criticalrole/comments/a4c9z3/no_spoilers_backblaze_sponsorship_ending/ebj31d6/. I had an AMAZING time working with G&S (still do w/ LA by Night) and the Critical Role team, and it was a great partnership! The TL/DR is that at some point you reach a saturation level, and have to look at different advertising/sponsorship avenues (plus they are in great hands now). It was a great ride! Hopefully I can still wiggle my way in there every now and again. That other post has more deets!

How is your Hard Drive ordering done, do you like just call up Seagate and say you want 2,000 Hard Drives or what?

And finally, how are returns of bad/broken drives still in warranty handled?

Yev here -> we asked our purchasing department for a better answer but until they write back here's what I think happens: we call the manufacturers and say, "Hey we need _X_ amount of drives, what's your lowest price?" And then we go with the one who gives us the smallest dollar amount. As for returns they're done through the warranty process, most manufacturers have an RMA portal that can be utilized using the serial numbers on the drives.

No more driving around the SF bay buying external HDDs on sale? I miss those days...

I did like exploring the bay area! This WAS my map after all -> https://www.backblaze.com/blog/wp-content/uploads/2012/10/Around_the_Bay_trip.jpg (from https://www.backblaze.com/blog/backblaze_drive_farming/).

I believe you guys wrote the story about the rash of 2 Tb drives with high failure rates . Did the vendor treat you fairly and make things right ? Or are you avoiding that vendor ? I had the same problem on my home system with the same drives .

Yev here -> Yes, those were the 3TB Seagate drives (but honestly many drives we were using around that time suffered higher failure rates) - and that vendor is great! We buy tons of Seagate drives (if you look at the hard drive stats posts you'll see them with a high percentage of our fleet) -> https://www.backblaze.com/b2/hard-drive-test-data.html.

Amazon Web Services has just announced pricing for its new Glacier Deep Archive and it seems among the lowest on the market for what i see as a "last line of defense" backup. But i've heard many good things about Backblaze, so can i ask in what way are your services and pricing structure different, and for which use cases you think you have the better value proposition ? I'm totally a noob with cloud storage BTW (but considering to get one for my Synology) so feel free to correct any misconceptions i might have.

Yev here -> Great question! We saw the news ourselves. Here's some back of envelope math we sent around the other day when this news was announced:

Assuming 14TB of storage - 14TB with Backblaze - instant ‘retrievability’ - $70 per month (vs. $322 per month for AWS S3). 14TB with AWS Glacier - minutes to 12 hours retrievability - $56 per month (fees apply). 14TB with AWS Deep Glacier - at LEAST 12 hours retrievability - $14 per month (fees apply).

Both Glacier and Deep Glacier also have a lot of retrieval fees/quirks if you want to speed up the process, but if you're willing to wait it's an OK proposition. The trouble comes if you want that data quickly. We charge $0.01/GB to download so the total(ish - assuming low transactions) cost of storage would be about 14TB/month and $140 to download all of it. And that's all you'd really pay with us.

Great explanation, thanks. Are you considering adding some lower tier of retrievability to compete in that space as well ? I ask that as someone more interested in pricing than speed of retrieval (that "last line of defense" backup idea). OTOH i feel your solutions are probably easier to use than AWS which also command a premium.

Are you considering adding some lower tier of retrievability to compete in that space as well

Not at the moment. We're hyper-focused on our offering and scaling that up to meet the needs of the many. A lot of folks want a Cloud Storage service that will be inexpensive and highly available, so that's where our energy is focused at the moment. Building out a lower-tier of storage would mean large-scale architectural changes (in a lot of those low availability services they use tape and/or DVD/s to house the data) and that's a lot of work!

How many of you are Tims?

Yev here ->

How many of you are Tim's?

At least 3...but we'll never tell who.

well if you know what a tim is, we found at least 1...

Shit

The most important question. HI is the reason I got bb

Welcome aboard!

So recently Linus from LinusTechTips made a video about backing up their 1 Petabyte storage servers. They ended up tricking Google Drive to accept that large of data. Would your service work for their needs? Or is 1 petabyte from a single user too much? Any comments about it?

​

Heres the video: https://youtu.be/y2F0wjoKEhg

Yev here ->

is 1 petabyte from a single user too much?

Definitely not. We have a lot of B2 Cloud Storage users with over 1PB of data. If they're just using it for storage/backup/archive we'd definitely work for them. The problem with tricking Google Drive to accept that amount is that's how you end up with unlimited services shuttering or raising prices (BitCasa, OneDrive Unlimited, Amazon Unlimited Storage, etc...). It makes it not sustainable, so while you technically can do that, we'd recommend using services specifically designed for that type of usage (plus can you imagine downloading or recovering 1PB from Google Suite...ooof).

​

Edit -> typo

Haha thanks. Is it even possible to recover 1PB?? I’m sure that would take months from any service.

From us? Yea. From Google Drive...no idea - that should be a follow-up video :P

Anyone else immediately upvote this because they've sponsored Critical Role?

Yev here -> Thanks! That was a fun time...AND I appreciate the upvote! :D <3

​

Hi, what are some good resources to understand about cloud implementation? Like more technical things that a student interested in pursing a career in cloud computing could understand from? Thanks in advance!

Yev here -> I can't speak to learning about cloud computing in general, but one of the most fascinating things that we've made was this explanation of how our Reed-Solomon Erasure Coding works for our vaults. We made the video with our Cloud Architect a few years ago and it was literally the only time I actually understood Matrix Algebra. Other than that our blog post on how we implemented "Vaults" is pretty interesting and might provide some guidance on different aspects of the cloud that you might find interesting: Backblaze Vaults.

What's the biggest single restore job someone has requested through Backblaze?

Yev here ->

We had a person once to 9 4TB restores to get all their data back, so that'd be about 35TB or so? Which is...quite a bit. /u/clunkclunk can give more detail!

750 petabytes.... That's it?? Heh jk

Yev here - Well that number is a month or two old, we're projecting to hit 1 Exabyte by the end of the year. ;-)

Is that useable data or does that include/raw and under managed. How much is duplicated?

The 750 is used (active) storage. We're deploying about 20-30 PB per month, and that gets filled up within the next few months. We try not to have too much "unused data" on hand as that is capital intensive and we're largely bootstrapped. We deduplicate data per client (Windows and Mac) on the backup side to avoid re-uploading data excessively from every machine.

Did you support the critical role Kickstarter?

Yev here -> Yes of course! I gotta get those letter openers!!!!

What was your guy's favorite ad bit from the man, Sam Riegel, on Critical Role?

Yev here -> I am very partial to this one -> https://www.youtube.com/watch?v=hnVAnmTNaHQ because it was friggin' hilarious. Taliesin's "All these wires, I can't take it anymore!" still kills me.

How did you guys get involved with critical role?

Yev here -> Funny story actually. I found Critical Role when they did their one-shot with Vin Diesel. I never played D&D before that and was intrigued by it. So I went back and started watching Season 1. A few weeks into binging the show, it dawned on me that they weren't taking any sponsorships. I run the online ads/sponsorships for Backblaze, and tried to reach out to them to see if they'd take any sponsors. After a few weeks of tweeting at most of the cast, the Twitter algorithm gods smiled upon me and Liam saw my tweet. We started chatting and he got me in touch with Travis who then put me in contact with Geek & Sundry who was producing the show for them.

​

Smash-cut (like in my commercials) to a month or so later and I was in Los Angeles for the taping of the Umbrasyl episode with Chris Perkins (Season 1 Episode 55) - which was one of the first sponsored episodes of Critical Role! Nowadays they're in very good hands :D

What do the driver manufacturers think of your sharing of data with the public? Sometimes you make them look good, other times when reliability is poor quite bad.

Also you have spoken a lot about Enterprise vs Consumer drives. Do you think it annoys them?

Yev here -> It's a mixed bag, like you said, sometimes they like it other times they don't - but I think over the years they've grown to use the stats as a way to dig into their performance. I did an AUA with u/Seagate_Surfer a few months back -> Seagate Scientist IAmA so we're definitely on good terms with all the manufacturers. Overall I think the release of those stats has been good for the industry and has also been good for consumers (granted our use-case is different than 99.99% of people).

What does your tech stack look like? Do you all use any kind of containers, and if so, container orchestration platform like Kubernetes? Do you host on-prem, through a cloud provider, or mixed? If cloud provider which one(s), and what does your deployment pipeline look like? What are your favorite cocktails? Sorry I'm kind of a DevOps/SRE person so I can ask you as many relevant backend questions as you all care to answer :)

Yev here ->

Do you host on-prem, through a cloud provider, or mixed? If cloud provider which one(s), and what does your deployment pipeline look like?

We actually rolled our own cloud, you can read a ton more about the architecture here: Zettabyte-Scale Cloud Storage Architecture. /u/brianwski might be able to speak more to the tech stack as a whole.

What are your favorite cocktails?

I'm a gin fan, so a lot of gimlets or martinis (extra dirty w/ an onion + olives, so kind of a hybrid Gibson) are what I'm drinking a lot of right now!

​

Is it common for the new hellium drives to fail prematurely? I purchased 17 of the Seagate 12TB drives and 4 drives failed after 5 minutes. Ended up switching to WD 10tb drives and have had zero issues so far.

Thanks,

Brandon

Yev here -> Actually we found that they pretty much perform the same over time. We wrote an article almost a year ago about it once we had a bit of experience with helium, you can read it here -> https://www.backblaze.com/blog/helium-filled-hard-drive-failure-rates/.

Why should I choose your platform over something like GDrive, Dropbox or similar service?

I am actually looking for cloud storage for backup so this is good time to do AMA. 😂

Yev here! Great question, I actually wrote an article about that Sync vs. Backup vs. Storage.

TL/DR - Sync is great for working out of one folder or making sure you have access to a subset of data at all times, but if you do not EXCLUSIVELY work out of that folder, the other things aren't getting sync'd. Backups are automatic and take place in the background, giving you access to all the data that is on your computer regardless of location. Cloud Storage is more manual, and is usually what any sync or offsite backup service will use as the back-end (you can write to the APIs and build your own services as well).

What is next for you guys? Anything major?

Also what protocols do you guys have for DDOS attacks?

Favourite video game? Favourite open source tool?

Yev here ->

What is next for you guys?

We're work on a lot of cool stuff all the time, but usually we keep it under wraps until we're ready to release it!

protocols do you guys have for DDOS attacks?

DDOS -> Our CTO had a good DDOS response to a different question.

Favourite video game

Top 3 games of all time for me: Last of Us, Vampire The Masquerade Bloodlines (so hype for the sequel), and Age of Empires (the series, loved it growing up, many hours spent in there).

Favourite open source tool?

Loving BitWarden right now!

​

Do hard drive manufacturers reach out to you folks?

Just curious if they can learn anything from what you folks do.

As we all know, making a product and being the end-user of a product are two completely different observations/expectations of said product.

Yev here -> We're constantly chatting with the different manufacturers, sometimes we run some testing for them on our hardware and other times we just chat about the general state of the backup world. Over time I think that most of the manufacturers found our stats to be insightful, and thus there's no really ill will or bad vibes going on. At the end of the day we're both trying to store data, so being open and helpful is a benefit to all folks!

​

Why should I trust you with my personal data? Please don't tell me because you are nice

Yev here -> We're nice.

Blunt shells or wraps?

Yev here -> Edibles, c'mon.