pgoupee

This particular data set used a lot of complex joins to sort the data in real time depending on what was available in the data when the query was run.

There were around 1000 different sources a day being fed in to the system so new data could be added to existing records and thus change the query results. They just couldn't get it. Even flew out to see them multiple times over a five year period to explain it. I'd walk them through it...we'd all be on the same page....fly home and then they call me saying "We don't get it". Their biggest problem was not using proper joins. Granted they were some of the most complex I've worked on but it all made sense. Ultimately it fell apart when they claimed that a huge number of records had a problem and copied their bosses on the email. I replied asking how that was possible when I'd only provided 1/3 of that amount.

Word the wise....learn how to properly use joins people.....all of them.

That's fine. I'm not here to prove anything to anyone. Just a guy that works with big data daily and giving my two cents.

To me excel is used for small reports to sales and management....that's about it. SQL does all the heavy lifting here. Everything I build is 100% custom built to order with the ability to scale.

I don't know about Excels abilities because I shouldn't be building products with it. Especially when it can't scale past a million rows.

Get really good with understanding how databases are structured and more specifically, WHY they are structured that way. There is a huge need competent data scientist but a serious lack of quality ones. I work with large data companies every day and I recently saw an entire team fired because not one of them could understand the data we send them despite the data being designed very well. And these people were supposed to be the best of the best at a huge company that you've all heard of.

I stand corrected...I guess I haven't really used Excel since the 2003 version...nice to see that they stepped it up but still.....a million rows is nothing when talking big data.

True that...can't think of a good use for cross joins lol.