IamA Software Engineer who worked on Juno (satellite going into Jupiter orbit today) AMA!

What software and coding languages do you primarily work with? I'm attending college to become a software engineer right now and would love to work for NASA or spacex one day

Great question! We developed the flight software in C++, and the drivers in C. The flight software was well established and flown on multiple missions before Juno/GRAIL, so it was more customizing the existing system for our mission. If you're looking to work on embedded systems, C++ is a great place to start. I also have done a lot of work in Java since. I sincerely believe that as long as you learn good fundamentals, you can learn any syntax.

Thank you! I've been working with Java for a year now and have started to dabble around with c# and Python. Additionally I also like to work with html, Css and JavaScript although web development isn't what I would like to pursue. Hopefully one day I can do what you do. It is my dream after all. Thank you for your time and response. Good luck with your future work.

Good luck to you too!

What is your favorite language?

It's hard to say-- every language has a purpose. I'm a big fan of Java and C# for back-end projects, but I've used some really slick javascript libraries for front-end stuff.

What's an overview of the fault detection and protection architecture? How are SEUs handled?

As I mentioned in another thread, it's a 2-sided spacecraft, so there is redundant software and hardware on each side, with redundant paths to each instrument. So, fault protection detects faulty conditions (failures to communicate with an antenna, for example) and will swap to the alternate path. If that still doesn't work, it will swap to the other side and use the other hardware.

SEUs (we call them bit flips) are handled by running constant checksums on the 4 copies (2 on each side) of the flight software. If a checksum fails, that copy is marked as bad, and is overwritten with a good copy. All this was thoroughly tested.

I'm going to be majoring in software engineering starting this fall. Any tips that will help me along the way?

Don't be afraid to ask for extra help on things you don't understand. I learned some of my best stuff at office hours and study sessions. Don't sweat the syntax as much as the core concepts and data structures. Syntax will change, but "if"s and "loop"s are in every language.

What did you do to fill the last 5 years?

I worked for Lockheed Martin for a while afterwards, but after sequestration, my program got cut and I decided to do commercial software. I consulted for a couple years, and am now working for an ecommerce/e-learning startup here in Denver called Craftsy. It's one of the things I love most about software engineering, you can be in pretty much any industry.

What was your speciality as an undergrad, where did you graduate from, and how did you got that job? Thanks, Current Computer Engineer senior-undergrad.

I got my Bachelors in Computer Science from Penn State. I interned at Lockheed Martin (in Owego NY) the summers before Junior and Senior years. They offered me a job after I graduated before I went back for Senior year which took a lot of pressure off. Once I was in, I found out they built spacecraft here in the Denver area, and wanted to come here, so I put in for a transfer, interviewed, and got it!

We are!

I'm at psu as well (Mech Eng '17). I will be interning at nasa's JSC this fall. Could you go into the pros/cons of working for NASA vs. one of their contractors?

Thanks for doing this AMA. Good luck tonight.

Unfortunately, I can't, really. I've only ever worked as a government contractor, so I'm not really sure what the differences are. I think probably the main advantage is you're less constrained by all the lovely government workplace regulations that are in place.

Enjoy the internship, and I'm always happy to help out fellow PSUers-- PM me if I can help!

What's the most fantastic but still possible outcome of this mission?

There are lots, I think-- finding an interesting amount of water in the atmosphere, finding a core and whether it's solid or liquid... I think any new things we learn about Jupiter will be fantastic. Plus we should get some pretty sweet pictures from JunoCam.

Any tips for a person that would want to become a software engineer?

Start as soon as you can, and always be ready for change. I've used lots of different languages throughout my career, and some have worked dramatically different than others. So, whatever you learn, make sure you're learning good core values rather than the nitty-gritty stuff. Buy lots of books, always be reading up on new technologies.

5 years is a long time between launch, and the major events that happen near and soon after JOI. Are you "on call" for a period of time near major mission events? Did you need to brush up on old code, in case of an "oh, shit!" event of some sort where only a software engineer with deep knowledge of the code could save the day? I know that if I go back to code I wrote 5 years ago, it takes me some time to reacquaint myself with the details of it.

I haven't been at the company for over 3 years now (in that division for over 4). Once development and testing is finished, the mission is handed off to an ATLO team to get it launched, then to an operations team to actually run the mission. Some engineers stay on for these phases, but most roll off onto another program. I did carry a pager (yes, an actual pager) during my tenure on this program, though, for emergencies. For JOI specifically, this is very much an automated process. They send the command to start, and the rest is done by the spacecraft. We just have to wait and hope everything goes right. It's not practical to do otherwise-- light time to jupiter is over 40 mins each way.

Interesting about the team hand off, thanks. Understood about the light delay; I was thinking more like the orbital capture works, but over the next day or two there is Some Kind Of Problem, and no one knows what to do, but then Bat-Engineer swoops in after seeing NASA's Bat-Engineer Signal in the sky, quick git commits are made, an emergency code fix is pushed to the spacecraft, the whole mission is saved, there are ticker tape parades, phone calls from the president, massive cheering crowds in Times Square, million dollar offers to star on reality TV shows, etc.

It could happen. I'm pretty sure.

Anyway, I hope everything goes great with Juno! Very exciting to have more outer solar system orbiters! And having your code out there is certifiably badass.

This software has undergone the most thorough and rigorous testing I have ever encountered, so while there's always a possibility for bugs, I think that possibility is extremely low, and the possibility of it being a mission-ending bug is even less likely. But it would make for a pretty cool story or movie :)

How accurate are the statistics on salary earnings for computer engineers as of 2017, according to you?

Depends on the source. It really comes down to experience and where you live. Denver's a pretty hot tech market right now.

Is there any backup equipment or redundancies in place in the event that something goes wrong or stops working, or is the available space and weight restriction too limiting for such things?

There are very many backups and redundancies. The spacecraft is "two sided", there's redundant hardware on each side, and 2 copies of the flight software on each side (4 total). The system is constantly running checksums since Jupiter's magnetic field is so strong, and the possibilities of bit flips is very high. If a copy doesn't pass the checksum, it's marked bad, and overwritten with a good copy.

If hardware or software fails on one site, it autonomously reports the fault and fails over to the other side. This is primarily the software I worked on (the detection and handling of faults). There are also redundant paths, for example an "A" and "B" path to the communications arrays, so if a path fails, it's marked bad, and fails over to the alternate path.

Plus the whole thing is in a titanium vault :)

IMHO this is the most interesting reply from the OP in this thread! Is some sort of similar redundancy used as well for the science data as well as the flight software? I assume there is typically a large amount of data buffered on the craft before it's able to be piped down through DSN.

Can you / are you allowed to say whether the spacecraft is running some well known embedded OS such as Wind River or QNX?

I'm not as sure on the science data, but I do know that after JOI, the spacecraft will be pointed towards the earth (and the sun, for power) as much as possible, so data will be sent down quite often. It would make sense to make multiple copies of the data and run checksums on it, again, due to the potential magnetic interference.

I don't believe it's running a traditional OS (I honestly don't remember), it's bootstrapped to just start up and run the flight software.

Congratulations on your project! What do you think will be the most surprising thing Juno could find in Saturn?

Ewoks. No, actually, it's studying Jupiter-- and I think the most exciting thing to find would be a solid or liquid core. Nobody really knows what's in there!

First of all, congratulations! :D

OK, this might sound like a stupid question but I'm curious...

The phrase "oh come on, it's not rocket science!" has entered everyday language. Apparently, what you do is the toughest type of mathematical engineering to understand.

But how difficult is rocket science for you? Is it something that leaves you constantly scratching your head and popping painkillers for headaches or does solving equations and manipulating complicated software come naturally to you?

Thanks for the well-wishes-- I'm still very nervous about JOI tonight, but so far everything has been going well!

I totally agree that astrophysics is one of the most complex fields out there. I actually wanted to minor in astronomy/astrophysics, but I just couldn't cut it. I have lots of friends who did make it through the program, and I think they're brilliant (one was just on Jeopardy, in fact).

Software feels more like solving puzzles to me-- I would do crosswords and sudokus in my spare time to keep sharp.

Hi! I'm currently finishing my degrees in Comp Sci and Mathematics and I was wondering: how much domain knowledge do you need to get into an industry? If I wanted to work for nasa or spacex, would I need any sort of degree in physics?

Nope. Code is code. You learn the business as you're in the industry. If you're passionate about it, all the better, but you certainly don't need a degree in astrophysics to work on a spacecraft.

Is Juno a metric or imperial spacecraft, from a coding perspective? I.e. are you coding in foot-pounds of thrust or Newton meters?

Ever since that very embarrassing Mars incident, everything is in metric so that situation will never arise again.

How much do you get paid?

I won't give exact numbers, but I made a lot more when I left government contracting. Cool jobs will always be long hours at not super good pay, because lots of people want to do it, so you're totally replaceable with some other person who will accept it. I didn't get a pay increase for the 2.5 years I worked on this project.

Hey, thanks for doing this AMA!

I have a general question. Lately I've noticed that a lot of people who major in some form of engineering, like biomedical, or in your case software, are working in fields way out of their comfort zone. Working on a space mission doesn't seem like the kind of job I'd expect from a software engineer (which, by the way, is absolutely amazing!).

How broad are my options when majoring in a certain field of engineering? Do I have plenty of options, rather than just a few like I've always thought?

The beauty of software engineering is that everything uses computers now. You can be developing websites, educational software, medical software, ridesharing apps... where there's a computer, there's a need for someone to code it. I was very fortunate that my interests aligned with the opportunities I had. I think it's very rewarding that I can jump from industry to industry and learn more about each one, while still speaking the common language of software.

The best way to figure out what jobs are available for a field is by googling it. If you're gonna be a petroleum engineer, search for petroleum engineering jobs. Same for mechanical, computer, biomedical... a base knowledge of these fields can be applied in many industries.

Are you working on some other spacecraft at the moment?

I'm not, I've moved into commercial software. It was a great time, but I've had a very fulfilling career outside the space industry. I still have friends who are in the industry, and I'd never say I wouldn't reconsider going back.

Is the mission of Juno to get pictures of the surface of Jupiter? Or is it to get pictures of the storms on the surface? If I'm not right with either, please fill me in on what exactly it will do! I'm curious and trying to become more knowledgeable of the mission, as well as space in general.

The funny thing is, Juno was originally not even going to have a camera. They added it towards the end when they realized it would be pretty silly to not put one on. So, while we will get some really cool pictures of Jupiter, that is pretty much the lowest priority part of the mission. The mission is to figure out as much about Jupiter as we can with Juno's suite of instruments, so we can learn how it may have formed. The press kit I linked to at the top has a TON of info about everything you'd ever want to know about Juno.

I'm very jealous, what was it like working on Ellen Page?

She's built like a steakhouse, but she handles like a bistro!

This may be a silly question and yet it is one that concerns me. As the articles on the internet as well as the movie documentary "The Lego Brickumentry" tell, there are 3 Lego Mini Figs on board the Juno space craft that are made of aluminum. I read that at the end of Juno's mission it is set for a self collision right into Jupiter. What will happen to the 3 Lego Mini Figs on the space craft, will their aluminum bodies survive forever somewhere in the center or Jupiter or will they disintegrate upon crashing?

Rest assured, their tiny lego bodies will melt and incinerate with the rest of Juno in Jupiter's atmosphere.

SWE here as well (I don't work on space stuff though). This is awesome. Can you say more about what the testing frameworks were like? I'd imagine unit/integration testing is really insanely important to you guys.

What is the flight software stack like? What platform are the drivers written for? How do you do software updates above the cloud?

How do you deal with communication delay to the spacecraft? Do you have to keep a tmux session open like we do here on earth when our wifi gets shitty?

Okay, that last one was a joke. Mostly.

Every piece of code was unit tested, then we integration tested the spacecraft in many different scenarios. Every requirement is verified by a test.

The FSW has many subsystems (thermal, communications, fault protection, etc), and they all work in congress together. Software updates are sent as patches, which are loaded, checked for validity, applied, and then the spacecraft restarts.

The spacecraft is very autonomous, so we tend to be pretty "hands off" when trying to drive the spacecraft. We send very broad commands, and the spacecraft knows how to do the rest :)

What sort of testing is done on this sort of software? Im sure it's extensive.

Are you still nervous you missed something and a null parameter exception is going to cause complete mission failure? :P

I am dreading that very thing right now. Funny story, the software was... exercised... unexpectedly when going around earth. Someone had applied the wrong parameters and the power level dropped when it went into earth's shadow, so the spacecraft went into space mode. This is the correct behavior, but before they had found out the root cause, I was beside myself thinking I screwed something up.

Hey Phil! Congrats on today, I bet it must be very exciting to see this take place.

What kind of scenarios were used in testing the software, and how long does one QA a piece of software for such a complex mission like Juno?

Thanks!

There were a bunch of mission phases; launch, inner cruise, outer cruise, JOI (now), and science. Each critical path was tested in each mission phase. I'd say at least a dozen of us worked on testing for many months (at least 3). It took a lot of work and lab time.

Thanks for all of the insight into the mission!

As a software developer/I.T. guy, I'm curious about the automatic restart that is supposed to happen if it "crashes" during the burn... it's been highlighted a few times, but I haven't caught any of the details.

Is that something "simple" like a persistent variable (e.g. seconds_left_to_burn) that picks up when it reboots? Or is it something more sophisticated like high-precision guidance systems that can re-calculate the burn parameters "on-the-fly"?

It's more of the former. We have an accurate clock of how long we've burned, and how long we were down, so we can compensate by burning a little longer. Thankfully, that hasn't happened, and most (if not all) fault protection is turned off during JOI since it's so critical.

What systems did you create code for on Juno?

I was responsible for the fault protection system, which is the autonomous recovery system for the spacecraft.

You've spoken about how you were working on the fault protection aspects of the software, but what were some of the spacecraft's other software requirements/functionalities? Was there another area that you thought seemed interesting that you would have liked to have been working on?

The coolest part about fault protection is that you worked a little bit with every subsystem, so I feel like it was the best place to be.

Did you ever smoke pot at work?

AT work?! Is there any job where you CAN do that?

What was your biggest problem doing the software?

Getting everything done in time!

Are you able to (even at this distance) update the software if needed, or are you pretty much at the mercy of what you sent up?

Also, what programming language was used?

We can totally patch the software. C and C++ were used.

Any interesting side projects? Or do you prefer to separate work from leisure time?

You know, I have a couple ideas for some fun side projects, but I never feel like I have time to do them (which is how lots of programmers feel, from what I understand). The one I like the most is a social integration for stack overflow, so you can find and follow your friends' activity on stack overflow.

How open is NASA's recruiting for software engineers? Or do they find people they need?

After the Juno mission, what will you be doing? Will you be maintaining code for future spacecraft?

I can't speak for NASA, I worked for a contractor. But I would imagine they need software engineers as much as anyone else :)

Do you think we'll be able to use the information we are learning from Hubble and Juno about solar wind affects/auroras on Jupiter to better understand the possible impact of changes past/present/future with-in Earth's atmosphere?

disclaimer, it's not my field; but yes, I do think some of the specific experiments should tell us much more about the atmosphere in Jupiter, and since the largest planets form first in a solar system, we should get a good idea of how all the other planets formed.

largest planets form first in a solar system

Why is this?

The short answer is "Gravity." The current accepted theory is that planets formed via stuff sticking together and being pulled into each other by gravity. Bigger bodies attract more stuff, making them even bigger, attracting more stuff, and so on. So, the biggest planets have been around for the longest time, pulling together the most stuff. See https://en.wikipedia.org/wiki/Formation_and_evolution_of_the_Solar_System

Hi there, thanks for doing this AMA! First off, congratulations on your accomplishments and I hope this mission turns up some very interesting answers for you.

I just recently graduated with a BS in Computer Science and have been looking into the job market with no luck. I would love to work for NASA or a similar agency that is involved in space-related technologies, but am not entirely sure where to even begin besides applying directly. Unfortunately, college internships have passed for me but do you have any advice on a possible plan or route I should take? I have heard some people suggest that I try to get a few years of industry experience first, but I'm not entirely sure.

Also, I have an obligatory question. Do you think Pluto should be a planet or not? :) Thank you so much!

Thanks for the questions and the congratulations. I'll feel much better once Juno's safely in orbit.

NASA does not build many things itself, as far as I understand. Most of the work is contracted out to government contractors. I worked for Lockheed Martin, but there are plenty of other contractors (Raytheon, Northrop Grumman, etc) and local ones (some out here are ASI, Red Canyon, and more).

If you can't get directly onto a mission, it's easier to get your foot in the door and then transfer.

As for Pluto, I gotta side with NDT on this one. Dwarf planet.

Thanks for the reply! I recently got a call from a Boeing recruiter for an entry-level software engineer position so hopefully that can open up some doors for me in that area.

Best of luck with your future endeavors and hopefully we learn a lot of great things from Juno!

Boeing does some very cool stuff, good luck!

What is the hardware platform on Juno, for your C++ / C to run on? Is it some derivative of x86 (rad hardened), or a totally different architecture?

What OS are you running?

I wish I remembered more about the hardware, but it was a long time ago and I didn't have much exposure to it. I do believe it was x86, and there isn't really an OS to speak of, it's just "turn on, run program"-- that part might be unix based. We definitely developed on unix machines. So no OSX or Windows :P

Do you have any advice for aspiring software engineers?

Yep! https://www.reddit.com/r/IAmA/comments/4r8zvi/iama_software_engineer_who_worked_on_juno/d4zak3z

I assume you needed TS clearance to work on the spacecraft? What was involved in getting it?

This is a scientific satellite, totally unclassified.

Broncos or Bills Phil?

Both. But if you make me choose, still Bills.

Thanks for doing this AMA! I have a question about bugs.

I'm a web developer, and I create bugs all the time. My code isn't bad, but I work fast and refactor later, etc etc.

Is there a concept of "tech debt" in your line of work? How extensive is the testing? Do things just take a really, really long time to build?

Now, things may have changed in the past 5 years, but when I was there it was very much waterfall. Requirements->Design->Code->Test. Every requirement had to have a test verifying it. We would also discover more bugs in integration, and fix those. It was very VERY well documented. And yes, things generally just take a really long time to build :)

If you had an opportunity to join a risky 6 month mission to another body, say Ceres or Mars, would you take it?

Nah, I'm old and out of shape. I'd love to do one of those space tours though, where you go up into super high atmosphere.

Did all the developers agree to use EITHER km or miles exclusively when calculating deceleration ?

The mission requires us to use SI units (metric).

The future of space travel is still uncertain, but the idea of private exploration has already been put into action. What are your personal thoughts on private space exploration? Also, how do feel about NASA vs. SpaceX? If you could work for one or the other, which would you choose?

I think the market has already shown that private companies can make a competitive, desirable product, and I think it will expand in partnership with NASA in years to come.

At this point, I'd probably work for SpaceX, since they can probably pay better and are trying more innovative techniques. I do hear they work their engineers very hard, though.

Is there somewhere I can buy copies of the Lego minifigures sent up with Juno?

I don't believe so, looks like they're one of a kind!

What types of projects would you recommend for an beginner - intermediate programmer? What sort of stuff did you do when you were first learning how to program?

lots of exercises out of books. I think the most valuable exercise I did, though, was creating a linked list from scratch. Teaches you a lot about pointers and data structures.

What happens if Juno is just facing the wrong way? Do you have software to use thrusters or something?

Juno always knows how to right itself and point back towards the earth.

What is Juno's IP address?

192.168.0.1

So, my dream job is basically a software engineer/developer at NASA. Do you have any tips?

Also, huge congratulations on Juno! Must feel awesome. 😀

Study hard in software engineering, work for a defense contractor, and keep a passion for space.

How much of those 5 years were spent naming variables?

How many of your team members' commit messages are "some changes"?

How many bugs did you find in the source code after Juno launched?

Lots. Few. Fewer. :)

whats your opinion on seL4? from a laymen perspective it seems well suited for space applications because of the mathematically veryfied code and the realtime capability of this microkernel

I'd honestly not heard of it until now, but it looks pretty cool. The space business, though, is VERY risk averse because of the astronomical costs associated with sending things up. So, unless there is measurable proof it's better than what's currently status-quo, I don't see it changing.