My short bio: I was a space nerd growing up, and my 2nd assignment out of undergraduate was a dream come true: working on the fault protection systems for the Juno and GRAIL spacecraft (which were being developed simultaneously). I also worked on testing the software for the missions once development was complete.

Today, almost 5 years after I watched in awe of my work roaring away from a launch pad in Florida, Juno has finally arrived at Jupiter and is ready to begin its mission! Watch JOI (Jupiter Orbit Insertion) coverage on NASA TV.

If I can't answer your question (especially if it's about the science-y stuff), JPL has a great press kit containing lots of detailed information about the mission and the science it'll be doing. A couple flight engineers also did an AMA a few days ago, and your question may have been asked and answered there.

Finally, the mission has its own subreddit- check it out at /r/junomission (although I'm sure /r/space will give it plenty of attention, too).

My Proof: - note some of the proof is GRAIL related.

  • 8:42 Eastern - gonna take a break for dinner and enjoy some of the holiday with friends, but I'll be back later to answer more questions. I'm having a great time, and thank you all for the questions and being excited about space!
  • 10:20 Eastern - I'm back-- less than an hour to go to JOI! IT'S HAPPENING!
  • 11:55 Eastern - BURN COMPLETE! WE DID IT! I am extremely happy/relieved. I'm going to have an adult beverage or three to celebrate. Thanks for participating, everyone. I'm happy to keep answering questions, but I'm gonna call it a night for the AMA.

Hey, all, a former coworker with me on Juno (who remembers a lot more than I do) has asked me to post some comments on his behalf. I'm prefixing them with "LM" -- he wants to remain anonymous.

Comments: 150 • Responses: 55  • Date: 

basedgodCookie17 karma

What software and coding languages do you primarily work with? I'm attending college to become a software engineer right now and would love to work for NASA or spacex one day

pennstatephil27 karma

Great question! We developed the flight software in C++, and the drivers in C. The flight software was well established and flown on multiple missions before Juno/GRAIL, so it was more customizing the existing system for our mission. If you're looking to work on embedded systems, C++ is a great place to start. I also have done a lot of work in Java since. I sincerely believe that as long as you learn good fundamentals, you can learn any syntax.

basedgodCookie8 karma

Thank you! I've been working with Java for a year now and have started to dabble around with c# and Python. Additionally I also like to work with html, Css and JavaScript although web development isn't what I would like to pursue. Hopefully one day I can do what you do. It is my dream after all. Thank you for your time and response. Good luck with your future work.

pennstatephil7 karma

Good luck to you too!

jorgehn123 karma

What is your favorite language?

pennstatephil10 karma

It's hard to say-- every language has a purpose. I'm a big fan of Java and C# for back-end projects, but I've used some really slick javascript libraries for front-end stuff.

thezerg19 karma

What's an overview of the fault detection and protection architecture? How are SEUs handled?

pennstatephil8 karma

As I mentioned in another thread, it's a 2-sided spacecraft, so there is redundant software and hardware on each side, with redundant paths to each instrument. So, fault protection detects faulty conditions (failures to communicate with an antenna, for example) and will swap to the alternate path. If that still doesn't work, it will swap to the other side and use the other hardware.

SEUs (we call them bit flips) are handled by running constant checksums on the 4 copies (2 on each side) of the flight software. If a checksum fails, that copy is marked as bad, and is overwritten with a good copy. All this was thoroughly tested.

Larsonrb6 karma

I'm going to be majoring in software engineering starting this fall. Any tips that will help me along the way?

pennstatephil19 karma

Don't be afraid to ask for extra help on things you don't understand. I learned some of my best stuff at office hours and study sessions. Don't sweat the syntax as much as the core concepts and data structures. Syntax will change, but "if"s and "loop"s are in every language.

theskuffy6 karma

What did you do to fill the last 5 years?

pennstatephil17 karma

I worked for Lockheed Martin for a while afterwards, but after sequestration, my program got cut and I decided to do commercial software. I consulted for a couple years, and am now working for an ecommerce/e-learning startup here in Denver called Craftsy. It's one of the things I love most about software engineering, you can be in pretty much any industry.

jorgehn125 karma

What was your speciality as an undergrad, where did you graduate from, and how did you got that job? Thanks, Current Computer Engineer senior-undergrad.

pennstatephil10 karma

I got my Bachelors in Computer Science from Penn State. I interned at Lockheed Martin (in Owego NY) the summers before Junior and Senior years. They offered me a job after I graduated before I went back for Senior year which took a lot of pressure off. Once I was in, I found out they built spacecraft here in the Denver area, and wanted to come here, so I put in for a transfer, interviewed, and got it!

NavyOtter5 karma

We are!

I'm at psu as well (Mech Eng '17). I will be interning at nasa's JSC this fall. Could you go into the pros/cons of working for NASA vs. one of their contractors?

Thanks for doing this AMA. Good luck tonight.

pennstatephil6 karma

Unfortunately, I can't, really. I've only ever worked as a government contractor, so I'm not really sure what the differences are. I think probably the main advantage is you're less constrained by all the lovely government workplace regulations that are in place.

Enjoy the internship, and I'm always happy to help out fellow PSUers-- PM me if I can help!

IAmFern5 karma

What's the most fantastic but still possible outcome of this mission?

pennstatephil6 karma

There are lots, I think-- finding an interesting amount of water in the atmosphere, finding a core and whether it's solid or liquid... I think any new things we learn about Jupiter will be fantastic. Plus we should get some pretty sweet pictures from JunoCam.

Maltazar165 karma

Any tips for a person that would want to become a software engineer?

pennstatephil11 karma

Start as soon as you can, and always be ready for change. I've used lots of different languages throughout my career, and some have worked dramatically different than others. So, whatever you learn, make sure you're learning good core values rather than the nitty-gritty stuff. Buy lots of books, always be reading up on new technologies.

Liquid_Reality4 karma

5 years is a long time between launch, and the major events that happen near and soon after JOI. Are you "on call" for a period of time near major mission events? Did you need to brush up on old code, in case of an "oh, shit!" event of some sort where only a software engineer with deep knowledge of the code could save the day? I know that if I go back to code I wrote 5 years ago, it takes me some time to reacquaint myself with the details of it.

pennstatephil5 karma

I haven't been at the company for over 3 years now (in that division for over 4). Once development and testing is finished, the mission is handed off to an ATLO team to get it launched, then to an operations team to actually run the mission. Some engineers stay on for these phases, but most roll off onto another program. I did carry a pager (yes, an actual pager) during my tenure on this program, though, for emergencies. For JOI specifically, this is very much an automated process. They send the command to start, and the rest is done by the spacecraft. We just have to wait and hope everything goes right. It's not practical to do otherwise-- light time to jupiter is over 40 mins each way.

Liquid_Reality2 karma

Interesting about the team hand off, thanks. Understood about the light delay; I was thinking more like the orbital capture works, but over the next day or two there is Some Kind Of Problem, and no one knows what to do, but then Bat-Engineer swoops in after seeing NASA's Bat-Engineer Signal in the sky, quick git commits are made, an emergency code fix is pushed to the spacecraft, the whole mission is saved, there are ticker tape parades, phone calls from the president, massive cheering crowds in Times Square, million dollar offers to star on reality TV shows, etc.

It could happen. I'm pretty sure.

Anyway, I hope everything goes great with Juno! Very exciting to have more outer solar system orbiters! And having your code out there is certifiably badass.

pennstatephil3 karma

This software has undergone the most thorough and rigorous testing I have ever encountered, so while there's always a possibility for bugs, I think that possibility is extremely low, and the possibility of it being a mission-ending bug is even less likely. But it would make for a pretty cool story or movie :)

jorgehn123 karma

How accurate are the statistics on salary earnings for computer engineers as of 2017, according to you?

pennstatephil8 karma

Depends on the source. It really comes down to experience and where you live. Denver's a pretty hot tech market right now.

Rechamber3 karma

Is there any backup equipment or redundancies in place in the event that something goes wrong or stops working, or is the available space and weight restriction too limiting for such things?

pennstatephil13 karma

There are very many backups and redundancies. The spacecraft is "two sided", there's redundant hardware on each side, and 2 copies of the flight software on each side (4 total). The system is constantly running checksums since Jupiter's magnetic field is so strong, and the possibilities of bit flips is very high. If a copy doesn't pass the checksum, it's marked bad, and overwritten with a good copy.

If hardware or software fails on one site, it autonomously reports the fault and fails over to the other side. This is primarily the software I worked on (the detection and handling of faults). There are also redundant paths, for example an "A" and "B" path to the communications arrays, so if a path fails, it's marked bad, and fails over to the alternate path.

Plus the whole thing is in a titanium vault :)

Liquid_Reality2 karma

IMHO this is the most interesting reply from the OP in this thread! Is some sort of similar redundancy used as well for the science data as well as the flight software? I assume there is typically a large amount of data buffered on the craft before it's able to be piped down through DSN.

Can you / are you allowed to say whether the spacecraft is running some well known embedded OS such as Wind River or QNX?

pennstatephil3 karma

I'm not as sure on the science data, but I do know that after JOI, the spacecraft will be pointed towards the earth (and the sun, for power) as much as possible, so data will be sent down quite often. It would make sense to make multiple copies of the data and run checksums on it, again, due to the potential magnetic interference.

I don't believe it's running a traditional OS (I honestly don't remember), it's bootstrapped to just start up and run the flight software.

danielsk13 karma

Congratulations on your project! What do you think will be the most surprising thing Juno could find in Saturn?

pennstatephil8 karma

Ewoks. No, actually, it's studying Jupiter-- and I think the most exciting thing to find would be a solid or liquid core. Nobody really knows what's in there!

TheRadioKingQueen3 karma

First of all, congratulations! :D

OK, this might sound like a stupid question but I'm curious...

The phrase "oh come on, it's not rocket science!" has entered everyday language. Apparently, what you do is the toughest type of mathematical engineering to understand.

But how difficult is rocket science for you? Is it something that leaves you constantly scratching your head and popping painkillers for headaches or does solving equations and manipulating complicated software come naturally to you?

pennstatephil9 karma

Thanks for the well-wishes-- I'm still very nervous about JOI tonight, but so far everything has been going well!

I totally agree that astrophysics is one of the most complex fields out there. I actually wanted to minor in astronomy/astrophysics, but I just couldn't cut it. I have lots of friends who did make it through the program, and I think they're brilliant (one was just on Jeopardy, in fact).

Software feels more like solving puzzles to me-- I would do crosswords and sudokus in my spare time to keep sharp.

Ace_Emerald3 karma

Hi! I'm currently finishing my degrees in Comp Sci and Mathematics and I was wondering: how much domain knowledge do you need to get into an industry? If I wanted to work for nasa or spacex, would I need any sort of degree in physics?

pennstatephil7 karma

Nope. Code is code. You learn the business as you're in the industry. If you're passionate about it, all the better, but you certainly don't need a degree in astrophysics to work on a spacecraft.

ADSWNJ2 karma

Is Juno a metric or imperial spacecraft, from a coding perspective? I.e. are you coding in foot-pounds of thrust or Newton meters?

pennstatephil9 karma

Ever since that very embarrassing Mars incident, everything is in metric so that situation will never arise again.

caillumknowles2 karma

How much do you get paid?

pennstatephil10 karma

I won't give exact numbers, but I made a lot more when I left government contracting. Cool jobs will always be long hours at not super good pay, because lots of people want to do it, so you're totally replaceable with some other person who will accept it. I didn't get a pay increase for the 2.5 years I worked on this project.

Hifya2 karma

Hey, thanks for doing this AMA!

I have a general question. Lately I've noticed that a lot of people who major in some form of engineering, like biomedical, or in your case software, are working in fields way out of their comfort zone. Working on a space mission doesn't seem like the kind of job I'd expect from a software engineer (which, by the way, is absolutely amazing!).

How broad are my options when majoring in a certain field of engineering? Do I have plenty of options, rather than just a few like I've always thought?

pennstatephil4 karma

The beauty of software engineering is that everything uses computers now. You can be developing websites, educational software, medical software, ridesharing apps... where there's a computer, there's a need for someone to code it. I was very fortunate that my interests aligned with the opportunities I had. I think it's very rewarding that I can jump from industry to industry and learn more about each one, while still speaking the common language of software.

The best way to figure out what jobs are available for a field is by googling it. If you're gonna be a petroleum engineer, search for petroleum engineering jobs. Same for mechanical, computer, biomedical... a base knowledge of these fields can be applied in many industries.

PatyxEU2 karma

Are you working on some other spacecraft at the moment?

pennstatephil4 karma

I'm not, I've moved into commercial software. It was a great time, but I've had a very fulfilling career outside the space industry. I still have friends who are in the industry, and I'd never say I wouldn't reconsider going back.

berb982 karma

Is the mission of Juno to get pictures of the surface of Jupiter? Or is it to get pictures of the storms on the surface? If I'm not right with either, please fill me in on what exactly it will do! I'm curious and trying to become more knowledgeable of the mission, as well as space in general.

pennstatephil5 karma

The funny thing is, Juno was originally not even going to have a camera. They added it towards the end when they realized it would be pretty silly to not put one on. So, while we will get some really cool pictures of Jupiter, that is pretty much the lowest priority part of the mission. The mission is to figure out as much about Jupiter as we can with Juno's suite of instruments, so we can learn how it may have formed. The press kit I linked to at the top has a TON of info about everything you'd ever want to know about Juno.

GorillaBallet2 karma

I'm very jealous, what was it like working on Ellen Page?

pennstatephil12 karma

She's built like a steakhouse, but she handles like a bistro!

KidROFL1 karma

This may be a silly question and yet it is one that concerns me. As the articles on the internet as well as the movie documentary "The Lego Brickumentry" tell, there are 3 Lego Mini Figs on board the Juno space craft that are made of aluminum. I read that at the end of Juno's mission it is set for a self collision right into Jupiter. What will happen to the 3 Lego Mini Figs on the space craft, will their aluminum bodies survive forever somewhere in the center or Jupiter or will they disintegrate upon crashing?

pennstatephil2 karma

Rest assured, their tiny lego bodies will melt and incinerate with the rest of Juno in Jupiter's atmosphere.

ohme21 karma

SWE here as well (I don't work on space stuff though). This is awesome. Can you say more about what the testing frameworks were like? I'd imagine unit/integration testing is really insanely important to you guys.

What is the flight software stack like? What platform are the drivers written for? How do you do software updates above the cloud?

How do you deal with communication delay to the spacecraft? Do you have to keep a tmux session open like we do here on earth when our wifi gets shitty?

Okay, that last one was a joke. Mostly.

pennstatephil1 karma

Every piece of code was unit tested, then we integration tested the spacecraft in many different scenarios. Every requirement is verified by a test.

The FSW has many subsystems (thermal, communications, fault protection, etc), and they all work in congress together. Software updates are sent as patches, which are loaded, checked for validity, applied, and then the spacecraft restarts.

The spacecraft is very autonomous, so we tend to be pretty "hands off" when trying to drive the spacecraft. We send very broad commands, and the spacecraft knows how to do the rest :)

iwiggums1 karma

What sort of testing is done on this sort of software? Im sure it's extensive.

Are you still nervous you missed something and a null parameter exception is going to cause complete mission failure? :P

pennstatephil1 karma

I am dreading that very thing right now. Funny story, the software was... exercised... unexpectedly when going around earth. Someone had applied the wrong parameters and the power level dropped when it went into earth's shadow, so the spacecraft went into space mode. This is the correct behavior, but before they had found out the root cause, I was beside myself thinking I screwed something up.

datduce1 karma

Hey Phil! Congrats on today, I bet it must be very exciting to see this take place.

What kind of scenarios were used in testing the software, and how long does one QA a piece of software for such a complex mission like Juno?

pennstatephil1 karma


There were a bunch of mission phases; launch, inner cruise, outer cruise, JOI (now), and science. Each critical path was tested in each mission phase. I'd say at least a dozen of us worked on testing for many months (at least 3). It took a lot of work and lab time.

luxliquidus1 karma

Thanks for all of the insight into the mission!

As a software developer/I.T. guy, I'm curious about the automatic restart that is supposed to happen if it "crashes" during the burn... it's been highlighted a few times, but I haven't caught any of the details.

Is that something "simple" like a persistent variable (e.g. seconds_left_to_burn) that picks up when it reboots? Or is it something more sophisticated like high-precision guidance systems that can re-calculate the burn parameters "on-the-fly"?

pennstatephil2 karma

It's more of the former. We have an accurate clock of how long we've burned, and how long we were down, so we can compensate by burning a little longer. Thankfully, that hasn't happened, and most (if not all) fault protection is turned off during JOI since it's so critical.

RollerJesus1 karma

What systems did you create code for on Juno?

pennstatephil1 karma

I was responsible for the fault protection system, which is the autonomous recovery system for the spacecraft.

billian921 karma

You've spoken about how you were working on the fault protection aspects of the software, but what were some of the spacecraft's other software requirements/functionalities? Was there another area that you thought seemed interesting that you would have liked to have been working on?

pennstatephil2 karma

The coolest part about fault protection is that you worked a little bit with every subsystem, so I feel like it was the best place to be.

tomeyk1 karma

Did you ever smoke pot at work?

pennstatephil1 karma

AT work?! Is there any job where you CAN do that?

rafaelima1 karma

What was your biggest problem doing the software?

pennstatephil2 karma

Getting everything done in time!

Lyianx1 karma

Are you able to (even at this distance) update the software if needed, or are you pretty much at the mercy of what you sent up?

Also, what programming language was used?

pennstatephil1 karma

We can totally patch the software. C and C++ were used.

FourtySevenLions1 karma

Any interesting side projects? Or do you prefer to separate work from leisure time?

pennstatephil2 karma

You know, I have a couple ideas for some fun side projects, but I never feel like I have time to do them (which is how lots of programmers feel, from what I understand). The one I like the most is a social integration for stack overflow, so you can find and follow your friends' activity on stack overflow.

chazzeromus1 karma

How open is NASA's recruiting for software engineers? Or do they find people they need?

After the Juno mission, what will you be doing? Will you be maintaining code for future spacecraft?

pennstatephil2 karma

I can't speak for NASA, I worked for a contractor. But I would imagine they need software engineers as much as anyone else :)

cozypozy1 karma

Do you think we'll be able to use the information we are learning from Hubble and Juno about solar wind affects/auroras on Jupiter to better understand the possible impact of changes past/present/future with-in Earth's atmosphere?

pennstatephil4 karma

disclaimer, it's not my field; but yes, I do think some of the specific experiments should tell us much more about the atmosphere in Jupiter, and since the largest planets form first in a solar system, we should get a good idea of how all the other planets formed.

ryan45884 karma

largest planets form first in a solar system

Why is this?

pennstatephil6 karma

The short answer is "Gravity." The current accepted theory is that planets formed via stuff sticking together and being pulled into each other by gravity. Bigger bodies attract more stuff, making them even bigger, attracting more stuff, and so on. So, the biggest planets have been around for the longest time, pulling together the most stuff. See

bmurphey1 karma

Hi there, thanks for doing this AMA! First off, congratulations on your accomplishments and I hope this mission turns up some very interesting answers for you.

I just recently graduated with a BS in Computer Science and have been looking into the job market with no luck. I would love to work for NASA or a similar agency that is involved in space-related technologies, but am not entirely sure where to even begin besides applying directly. Unfortunately, college internships have passed for me but do you have any advice on a possible plan or route I should take? I have heard some people suggest that I try to get a few years of industry experience first, but I'm not entirely sure.

Also, I have an obligatory question. Do you think Pluto should be a planet or not? :) Thank you so much!

pennstatephil4 karma

Thanks for the questions and the congratulations. I'll feel much better once Juno's safely in orbit.

NASA does not build many things itself, as far as I understand. Most of the work is contracted out to government contractors. I worked for Lockheed Martin, but there are plenty of other contractors (Raytheon, Northrop Grumman, etc) and local ones (some out here are ASI, Red Canyon, and more).

If you can't get directly onto a mission, it's easier to get your foot in the door and then transfer.

As for Pluto, I gotta side with NDT on this one. Dwarf planet.

bmurphey3 karma

Thanks for the reply! I recently got a call from a Boeing recruiter for an entry-level software engineer position so hopefully that can open up some doors for me in that area.

Best of luck with your future endeavors and hopefully we learn a lot of great things from Juno!

pennstatephil4 karma

Boeing does some very cool stuff, good luck!

ADSWNJ1 karma

What is the hardware platform on Juno, for your C++ / C to run on? Is it some derivative of x86 (rad hardened), or a totally different architecture?

What OS are you running?

pennstatephil6 karma

I wish I remembered more about the hardware, but it was a long time ago and I didn't have much exposure to it. I do believe it was x86, and there isn't really an OS to speak of, it's just "turn on, run program"-- that part might be unix based. We definitely developed on unix machines. So no OSX or Windows :P

Lethargic_Bork1 karma

Do you have any advice for aspiring software engineers?

ADSWNJ1 karma

I assume you needed TS clearance to work on the spacecraft? What was involved in getting it?

pennstatephil5 karma

This is a scientific satellite, totally unclassified.

coach4261 karma

Broncos or Bills Phil?

pennstatephil2 karma

Both. But if you make me choose, still Bills.

jordanscales1 karma

Thanks for doing this AMA! I have a question about bugs.

I'm a web developer, and I create bugs all the time. My code isn't bad, but I work fast and refactor later, etc etc.

Is there a concept of "tech debt" in your line of work? How extensive is the testing? Do things just take a really, really long time to build?

pennstatephil5 karma

Now, things may have changed in the past 5 years, but when I was there it was very much waterfall. Requirements->Design->Code->Test. Every requirement had to have a test verifying it. We would also discover more bugs in integration, and fix those. It was very VERY well documented. And yes, things generally just take a really long time to build :)

Exxmorphing1 karma

If you had an opportunity to join a risky 6 month mission to another body, say Ceres or Mars, would you take it?

pennstatephil3 karma

Nah, I'm old and out of shape. I'd love to do one of those space tours though, where you go up into super high atmosphere.

google_academic1 karma

Did all the developers agree to use EITHER km or miles exclusively when calculating deceleration ?

pennstatephil1 karma

The mission requires us to use SI units (metric).

throwawaythrowawayfa1 karma

The future of space travel is still uncertain, but the idea of private exploration has already been put into action. What are your personal thoughts on private space exploration? Also, how do feel about NASA vs. SpaceX? If you could work for one or the other, which would you choose?

pennstatephil1 karma

I think the market has already shown that private companies can make a competitive, desirable product, and I think it will expand in partnership with NASA in years to come.

At this point, I'd probably work for SpaceX, since they can probably pay better and are trying more innovative techniques. I do hear they work their engineers very hard, though.

neobowman1 karma

Is there somewhere I can buy copies of the Lego minifigures sent up with Juno?

pennstatephil1 karma

I don't believe so, looks like they're one of a kind!

imthedewg1 karma

What types of projects would you recommend for an beginner - intermediate programmer? What sort of stuff did you do when you were first learning how to program?

pennstatephil2 karma

lots of exercises out of books. I think the most valuable exercise I did, though, was creating a linked list from scratch. Teaches you a lot about pointers and data structures.

Stockman_1 karma

What happens if Juno is just facing the wrong way? Do you have software to use thrusters or something?

pennstatephil1 karma

Juno always knows how to right itself and point back towards the earth.

mordecai981 karma

What is Juno's IP address?

pennstatephil3 karma

VRtual1 karma

So, my dream job is basically a software engineer/developer at NASA. Do you have any tips?

Also, huge congratulations on Juno! Must feel awesome. 😀

pennstatephil2 karma

Study hard in software engineering, work for a defense contractor, and keep a passion for space.

AlphaWhelp1 karma

How much of those 5 years were spent naming variables?

How many of your team members' commit messages are "some changes"?

How many bugs did you find in the source code after Juno launched?

pennstatephil1 karma

Lots. Few. Fewer. :)

elypter0 karma

whats your opinion on seL4? from a laymen perspective it seems well suited for space applications because of the mathematically veryfied code and the realtime capability of this microkernel

pennstatephil4 karma

I'd honestly not heard of it until now, but it looks pretty cool. The space business, though, is VERY risk averse because of the astronomical costs associated with sending things up. So, unless there is measurable proof it's better than what's currently status-quo, I don't see it changing.