We are Blue Origin Software Engineers - We Build Software for Rockets and Rocket Scientists - AUA!

Hi all! Thanks for doing this AMA!

Speaking as a lowly web developer, I can honestly say that I rarely implement enough exception/error handling in my code. Thankfully, people's lives don't necessarily depend on my code...

It occurred to me, looking at the Apollo 11 Guidance System code that systems like the ones you write have to have hundreds if not thousands of fallbacks/handlers for any number of things that might go wrong.

• Is this true of your work?
• Given that it'd be impossible to predict every contingency, what practices do you follow to make sure you've been as thorough as possible?
• What kinds of testing does your code have to go through?

Yes, for safety-critical code, we have to plan for and handle every possible failure mode. There is also flight and ground code that isn't safety-critical. And obviously we have a lot of software at the company that supports engineering and analysis. Not everything gets developed to the same rigor because rigor takes time.

You're right that you can't predict every possible failure and typically you also can't test every possible combination of inputs and outputs. The single best way to mitigate that is to architect systems that are inherently simple. That means isolating systems from one another and keeping the safety-critical surface area small. Fewer failure modes means fewer cases to analyze and handle. After that, though, we make sure our systems are really well understood, with documented interfaces, requirements, designs, and tests, in addition to the code -- with review of all of those. We measure code coverage, invest in static analysis, use continuous integration, etc. It's all about making the systems simple and well-understood.

For testing of the flight code, we test at multiple levels -- unit and component testing, integrated simulation, the full hardware-in-the-loop setup, and even some on the vehicle (e.g. we can make the vehicle think it's flying when it's still on the ground). The hard part is making sure we've covered everything that has to be covered. For that we rely primarily on human review and code coverage analysis.

How often do you say CMON GUYS THIS ISN'T ROCKET SCIENCE?

My guess is not enough.

Or maybe exactly the right amount

Do you have to pay for AWS?

Yes, we do :)

Hi! I have fun creating my own autopilots and avionics for Kerbal Space Program.

1. What tools/software do you use for visualizing rocket telemetry during launch? For analyzing after?

2. Are there any good open papers or books on designing avionics interfaces for human use?

3. Reliable systems are hard. What kind of software development processes do you have make this possible?

4. I'm guessing you don't have a single computer running the rocket for reliability reasons. If I recall correctly, SpaceX has said they do a Byzantine generals fault tolerant system with multiple computers. Airbus and Boeing airliners do multiple flight computers, each computer running the same program on multiple CPU architectures and checking the results. How do you guys handle this?

5. Unlike SpaceX's booster landing process which requires nutso math, from watching Blue Origin landing videos, it looks like your landing process is just "stop the rocket in the air, slide over until above pad, then lower down", which a kid could hack together with just PID controllers. Is it really that simple, or are you doing full on motion planning for the final landing? Either way, I like the room for correcting errors that your method provides.

6. I've heard that SpaceX, and other rocket companies, working hours are rough for engineers and families. Have any of you worked on a Saturday in the last month?

It's great to see your rockets fly. Keep up the good work!

Hi aCalculus,

Thanks for your questions,

1. This is a tough one to answer, because we have a huge number of tools for this! As you speculate, they can be divided into two categories: (a) Real-time (data visualization and data reduction that informs flight controllers during a mission) and (b) Post-flight (analyzing telemetry and logs to determine exactly what happened when and why). Most of this software was developed in-house in Python and C++, but we use third-party software in certain places (for example, Grafana for visualizing time series data).

2. I'm not sure if I understand the scope of your question. Are you looking for papers/books specifically about the interfaces through which an astronaut or ground-control operator interacts with flight software? Or are you just looking for more general resources about avionics?

3. Our safety-critical software is designed against DO-178C and additional standards that we have developed in-house. We've answered a few other questions today about our testing process, so there are more details in the other answers too.

4. You guess correctly: our avionics architecture uses multiple flight computers. Many of our systems incorporate redundancy in order to be fault tolerant, including tolerance against Byzantine failures. Running multiple computers is a good idea for reasons other than redundancy, too, for various reasons, it can be beneficial to run control loops on hardware that resides physically near the sensor it is controlling.

5. Nothing involving rockets is that simple, but when attainable, simplicity is good. We would also hire that kid if you could point him or her out to us :)

6. I haven't worked a Saturday in the last month; a few of us have, but it is not something we do without a very good reason. You get diminishing returns when you push too hard for too long.

Have you guys ever played Kerbal Space Program?

Yes, many of us do.

How does one become a software engineer for a rocket/space company? And do any of you have interesting interview stories? Thanks in advance!

I for one never expected to work in the space industry or Blue. I was previously working at Amazon and heard a lot of mentions of Blue. I thought since I didn't have a big mechanical/electrical engineering background I wouldn't be able to find a job in the field.

I saw that Blue had software opening that matched my background(CS Degree) and applied and got on a great team building applications/tools that drives the business of building rockets.

The interview for Blue is different than any other I have been to before as the first hour is you giving a presentation on yourself, job history, and projects.

So don't let your dreams be dreams! Just DO IT!

Blue has seen a ton of growth over the past years. In your mind, what has Blue done well/poorly to handle this growth? How do you see the company managing even more growth in the future, as you shift from a small R&D firm to a company that builds and frequently flies a large fleet of reusable vehicles?

Thanks! I'm a huge fan of BO and the passion everyone there seems to have. I actually am going to be starting an internship with you in Kent in a month, and can't wait to start! Gradatim Ferociter!

This is a big and important question. There's a lot more to the answer than I can do justice to here. First, it's top of mind company-wide and we're constantly on the lookout for signs of scaling pain and ways we can do better. A key part to the scaling is specialization. For example, people are increasingly working on either existing New Shepard operations, the next increments of New Shepard, BE-4, the orbital program, or common tools/frameworks but not all of them simultaneously. We're setting up appropriate leadership and communication structures, creating strong ownership of individual areas, constantly iterating on our onboarding processes, and improving documentation and training.

There's no way to grow quickly without discomfort, but I think we're actually doing pretty well. Hopefully you'll see and feel the same when you start! Definitely stop by and say hi to the people in our proof pic when you arrive.

What is your opinion on what SpaceX has achieved so far, and what do you think will change in the space industry wth such healthy competition between you and SpaceX?

More the merrier!

When there is success anywhere in the industry especially as young as the private space industry it helps everyone. I hope with the increase of private companies related to space encourages more young kids to get inspired and dream of living and working in space.

Have you ever shattered any windows in the window warehouse next door while testing your engines?

We've never had a complaint. We're very good neighbors.

Love what you guys are doing, the space industry is really making phenomenal progress right now!

Congratulations on the successful launches so far!

Questions...

How many on your team, developer:tester:management ratio? How many in GNC, avionics, infrastructure, ground systems etc?

What's your development methodology?

Where does software sit in the pecking order vs propulsion, vehicle etc?

How many hours a week are you guys pulling?

Did any of you transfer from Amazon?

• How many on your team, developer:tester:management ratio? How many in GNC, avionics, infrastructure, ground systems etc?

It is mostly flat structure. Software engineers have full ownership end to end. We are organized in small teams spread among the different software groups. We are hiring!

• What's your development methodology?

We use agile methodologies. Continuous integration and iteration over features is the culture here.

• Where does software sit in the pecking order vs propulsion, vehicle etc?

Software is critical component of making the vehicle fly.

• How many hours a week are you guys pulling?

When you are passionate about what you do, time becomes relative.

How often do you implement new guidance/navigation algorithms? I imagine that testing them in simulation is more common than onboard the flight hardware, but I'm interested in how frequently you improve the GN&C systems, especially since BO has been launching/landing more frequently.

Also, does the vehicle do all of the GN&C onboard, or are parts offloaded to ground systems?

Finally, how much of the GN&C software would you consider to be "intelligent"? Does it all rely on the models you develop for flight conditions, or do you build in the ability to sense evolving conditions and adapt to possibly unknown scenarios?

Thanks!

Hi astropjj,

On New Shepard, we have made GN&C improvements and changes between each flight. There are many of these, but a few examples are listed on our blog: between M2 and M3 we made the landing algorithm less picky about landing in the exact center of the pad, and between M3 and M4 we lowered the altitude of engine restart. We also make changes to support flight tests (for instance, to support the "one-parachute out" test that we flew last mission). As you say, we have multiple layers of tests to verify these changes work as expected, from software simulations to integrated checkouts in our HIL setup and on the vehicle. One reason for the multiple layers is to speed up our development cadence as we don't have to rerun the most demanding tests every time we make a software change: just when we're doing final software qualification in preparation for a flight.

The vehicle is fully autonomous and all GN&C is handled onboard.

Thanks for your questions!

Hey, I recently looked at Internships at Blue Origin and unfortunately found that you only accept US citizens because of federal Export policies (I'm german). So do you know if there is any way for non US citizens to get cleared to work in the space industry and do you have any foreign employees? Thank you

That's correct -- one of our hiring requirements is that a candidate be a US Citizen or Permanent Resident due to export requirements. We do not currently have any foreign national employees.

I'm a US citizen and a comp sci student with experience in c++, java, and python. What can I do to work with you guys because this is basically my dream job.

Apply here https://www.blueorigin.com/careers

Do you ever make dad jokes about how 'it's not rocket science?'

Yes, but what I constantly struggle with is what do I compare hard tasks too now?

It's not brain surgery?

I am pretty sure we have someone in Blue that is qualified to do both brain surgery and rocket science if not now, then eventually and we'll have to ask this question all over agian.

(EDIT) relevant

1. Do you develop in an agile or water fall cycle?

2. Do you follow or base your development process on any Nasa standards or requirements? Npr7150, software safety standard, etc.

We do agile and work in sprints. We've developed our own standards that meet and in many cases exceed industry standards.

Hey thanks for the AMA, i'm really curious

1.- how are you using python? (I would imagine not in the rockets) 2.- it seems that you use js only for your website, is that right?

Python is used extensively in Blue to create tools used by engineers for processing simulation results, interacting with our REST services, automation etc. It is mostly our go to scripting language.

Found this image of your HQ. Looks awesome.

What are your plans for the future with Blue? Just get the Orbital-class launcher running and dominate the market like that?

How many hours do you guys work?

What do you have to say about the supposed 'rivalry' with SpaceX?

As a British teenager, what's the best way to get to work at Blue?

• What are your plans for the future with Blue? Just get the Orbital-class launcher running and dominate the market like that?

Our vision is millions of people living and working in space.

• How many hours do you guys work?

When you are passionate about what you do, time becomes relative.

• As a British teenager, what's the best way to get to work at Blue?

To work for Blue a candidate must be a US Citizen or Permanent Resident (green card holder) due to ITAR requirements.

Awesome works you guys and gals are going over there! How good are y'all at Kerbal Space Program?

As you can imagine, there are many Kerbal fans at Blue. We've had demos of New Shepard-like VTVL rockets. I for one am now flying a much larger VTVL orbital launcher. We have discussed building a Blue mod.

Hey guys! How much of your software would you say is built from scratch? In my experience, older aerospace companies tend to rely heavily on heritage code with only the minimum modifications needed to get the job done.

Does BO follow a similar philosophy by building upon work already done? Or are you building everything from the ground up?

We at Blue always try to use the best of breed software/tools for all the jobs we do, but when you factor in the complexity of building rockets and with flying human passengers we do end up writing a lot of our software. That being said we do build on work that's been done in third party services/libraries.

We always start the discussion around what building the service/tool our self gives us in the long run, be it finer control, change control, or customization beyond what it would normally offer.

How do you guys actively debug your code if the machinery it controls costs millions of dollars? Can you somehow simulate the code running or something?

We can run the code, simulation, and a lot of the tests on a desktop, which is both faster and safer than real hardware. And then we can run on real hardware before we connect hazardous actuators. But at some point you've got to control a real actuator for the first time. At that point you don't plan to be actively debugging: you start logging everything, add safety glass or blast mats, and then step away. Far away.

What was the biggest hurdle or learning curve you faced within the first few months of starting work for Blue Origin?

It takes time to learn the technology stack and get to know your internal customers. This is a hands-on process where the engineer is expected to have full ownership. As a new graduate I rotated through all of the different software teams for the first year of employment, in which there were six extremely challenging projects requiring different technical skills. Passion for what we do is integral to success at Blue.

How long does it take to complete one of your missions?

In the happy case, actual flight time is only about 10 minutes - see an actual timeline on the last webcast (https://www.youtube.com/watch?v=EI-tGVFg7PU). Rollout, prelaunch, safing, and recovery turn that into most of a day. Then there are days on either side of launch day for pre-flight preps and inspections. And then there are scrubs for weather or any issues that arise.

Hi /u/blueoriginsoftware,

I'm a young female at the University of Washington Seattle Campus studying CSE :).

I'm always scared of how the real world works when it comes to big challenges like this! I remember one of my professors saying a Russian (?) rocket failed miserably due to a 64-bit float being converted to 16-bit signed int, and of-course producing the wrong output for whatever API they were using.

Right now I'm interning, but I feel like I don't really do all too much other than basic tasks. I feel so far away from being able to do what I really want which is more simulation type stuff with physics. I'm awed by water simulations, heat simulations, some of the stuff I see from NASA with figuring wind turbulence, etc.

What would you suggest for someone like me who wants to do that to focus on in graduate school (Masters or PhD)? Or would just trying to get into the field directly be a bad option?

Our software engineers have many different academic backgrounds -- Bachelors, Masters and PhD's -- in many different fields. We have also had engineers work for us and get their Masters in parallel. There is no single path we can recommend.

We offer both an intern program and a new graduate rotation program. Both give engineers the opportunity to explore different software development stacks and domains to figure out what they are most passionate about.

Send us your resume and we'll bring you down for a tour. That's always a good first step :)

https://www.blueorigin.com/careers

I (and many others over at /r/BlueOrigin) really enjoyed the last live stream. Do you guys have any plans to provide live video feeds from the rocket in a future live stream? Also, are you guys working on any software for Very Big Brother yet?

We will definitely have streams like the ones we've had so far. We are looking into live streaming from the rocket itself. Yes, Very Big Brother is being worked on.

I recently discovered that you guys have what I believe is the actual Star Trek Enterprise model used in the original motion pictures in your office. Can you confirm this and could you share some pictures of it? As a big Star Trek fan I'd love to see her!

Check out this article for a picture of our lobby: http://spacenews.com/blue-origin-plans-growth-spurt-this-year/

What advice would you give to someone finishing a B.S. in Computer Science who wants to work in the aerospace/general engineering world as a software engineer? What are the specific technologies, skills, and experiences that you'd look for if you were to hire a new graduate just beginning their career?

Thank you!

As a recent CS graduate who started my career at Blue, my advice would be to follow what you are passionate for. As a new graduate, it is important to have a strong understanding of algorithms and data structures, as well as good problem solving skills. It is also important to have a strong desire to learn more about aerospace.

How do you do testing on your software? Oh yea, how do you go about tracking the tests and the results?

Please get down and dirty if you don't mind.

For Enterprise software, we write our own unit tests that run in jenkins automatically. We also run integration tests. We use jmeter for load testing our backend, and Frisby (which is a nodeJS framework) for sanity checking the data we get back.

For Avionics software, we also do unit-testing (GTest) as well as hardware-in-the-loop (HIL) tests in which the flight software controls a simulated mission.

Do you guys write any in-house CFD/FEA code to analyze your system? Or do you primarily stick to commercial/NASA codes?

When it comes to fault tolerant systems, do you guys rely much on error checking at many points within the pipeline or just at critical pathways? Like for example, error checking between the ecu and the main computer or between the sensors and the flight computer or do you still check internal memories of the computers with parity or bit by bit comparisons?

We primarily use third-party CFD tools.

Fault tolerance is interesting. Generally you're employing it where you need high reliability. However, simpler systems are often more reliable. So while we won't go into the details of our computing architectures, we definitely aim for the simplest systems that meet our fault tolerance needs.

Do you plan on growing significantly for the development of Very Big Brother or is the design already maturing quickly without the need for more growth?

We are growing significantly! We have many openings available: https://www.blueorigin.com/careers

I've always believed that programming languages are like tools and you have to use the right tool for the job. Like you wouldn't use a hammer to put in a screw. Where do you use each of your languages?

I completely agree. We use Java for internal distributed tooling, services, and web applications. C/C++ for safety critical applications. Python/MATLAB for developer tools.

Thanks for the AMA, appreciate the hard work and sweat you guys put into BO! Unto the questions:

1. What are the set of backgrounds that you guys came from to get to where you're at now?

2. If any, what are the different divisions / teams of the software engineers at BO?

3. Where do you see ITAR restrictions and the likes ( preventing non-lawful-American citizens from working in the industry ) in the coming years? ( X-posted this from the latest AMA in /r/spacex because it was dodged, I was hoping you'd answer this one because it's particular important to me :] )

• What are the set of backgrounds that you guys came from to get to where you're at now?

Most have formal education in Computer Science or Electrical Engineering but many come from related fields in Mathematics, Physics and Mechanical Engineering. If you are passionate about software, then we would be interested in talking to you.

• If any, what are the different divisions / teams of the software engineers at BO?

Our software teams work on designing and building:

• Systems to supports all engineering activities including design, manufacturing, test and operations
• Systems and tools to automate simulations and data analysis
• Software to control the rocket, space vehicles and ground systems

What's the hardest thing you had to do in school to get your degree, also what is your degrees? All computer science? Was it difficult to get a job at BO?

I'm a huge face of space exploration, and I am looking forward to Spacex and NASA heading toMars, but I believe that space tourism is going to be the thing that will capture the public's eye and help push space exploration incredibly fast and farther than ever before. Keep up the amazing work, every time I see New Shep launch and land it instills an amazing feeling inside me and I cannot wait to work there.

Most of the people on the team have a computer science degree. I personally come from Electrical Engineering background. I spent a lot of my free time playing with different software, building my own projects and also getting into the industry to learn from the best. There wasn't one particular thing that was the hardest in school, it was all solving different problems. Problem solving skill and passion is very important in the interview.

Just how complex is the software that controls a rocket?

Very. It is rocket science!

What is your background? Have you all studied software engineering or some of you come from math/physics/etc?

Mostly computer science. However we also have many software team members with backgrounds in math, physics, mechanical engineering, and electrical engineering.

I'm currently a BA for a Fortune 200 company based in the renewable energy sector on the East Coast. I want to work for BO-- It's a goal of mine. How many resumes can I submit before I am asked to stop?

The best way to find out is to start sending resumes to Blue. :) https://www.blueorigin.com/careers

Are you working on code for the new bigger launch vehicle you'll be building and launching out of Florida? Any hints about what it'll be called? ;)

Yes and no :)

How do you handle radiation hardening of you code?

Fortunately, code itself isn't subject to radiation. The hardware it runs on is a different matter...

where do I apply?

https://www.blueorigin.com/careers

Hi what sort of info are you storing on mongo? Why non relational db?

We use MongoDB internally in some of our services to store meta data about various things. We always think of what is the right tool for the job. Do we need transactions? What is the data access pattern? How do we plan to scale and on which dimensions? What is the cost of development and operations. All these questions and more guide our choices.