I'm a IT Engineer and Manager at a Managed Service Provider (MSP) doing outsourced IT throughout the pandemic AMA!

How do you feel about msp’s exploiting their salaried workers putting them on call and not paying them appropriately?

It's a nasty practice that I've previously been on the receiving end of. It's not just MSPs either, there's a lot of industries that do that.

At a previous employer I was disciplined for not checking my emails at 3am, because apparently I wasn't allowed to sleep while oncall?

When they threatened to fire me for refusing to comply I quoted them the labor law and showed them how much overtime they legally owed me and they changed their tune pretty fast.

You shouldn't be offering 24/7 service if you aren't willing to have 24/7 shifts.

I've tried real hard to avoid these practices while being a manager. You can't avoid some after hours work in IT, but you can automate away a lot of things, be flexible with schedules, pay OT, and set it up so your staff have infrequent oncall rotations instead of 'all the time' or every couple of weeks.

If I was on an on-call rotation at a previous employer could I sue them for back pay? I was salary and didn't know about overtime for on-call. State was Colorado.

\r\legaladvice

I'm 30 and still doing entry level IT roles, never went to university.

I started in IT when I was 19 and left 3 years later to travel and such and entered back into IT 2 years ago.

What field of IT should I specialise in for the moneys and job security etc, Netowroking? coding ? Security ?

If you're in it for the money and job security I suggest you learn Mainframes... like the old IBM type.

I'm not joking, mainframe guys are super in demand and it's garaunteed job security. You'll be the mainframe guy at a major insurance company or whatever because you'll be the only one that knows have any of it works.

Otherwise IT security is lucrative, but very very stressful.

Cybersecurity. Sounds sexy really tedious.

"60 hours worth of meetings next week to make sure we pass our PCI"

Security, we can't find qualified people.

Or dismiss their resumes because they fail the drug test

When companies decide to outsource infrastructure services, what’s their common complaint with MSPs. That is, what do they wish their MSPs should do better? In your experience, What should MSPs focus on to keep a client “happy”?

TLDR: If you have good people, you pay them well, and they don't get burned out then they'll pay more attention, care about the customers, and won't be as prone to mistakes.

The most consistent complaints I hear from existing MSP customers wanting to switchover to us is not being able to get hold of techs on demand, delays in getting support, pushy sales people + constant upselling, regular maintenance not getting done, and straight up incompetence.

For the first two staff are often used to having a dedicated IT person on staff that can show up and teach you to use excel at the drop of a hat. When you're with an MSP you have to deal with the fact that we have a lot of customers and we have to prioritize who gets service when.

We have a servicedesk to provide quick service, but staff often hate calling for support so it's give and take. Having high quality people and good training for the service desk helps, but you also need to have a good escalation path so that people don't waste time on the phone with issues the tech on the line can't fix. Being able to do regular onsite visits helps with the touchy-feely stuff.

We also offer fulltime onsite techs if a company wants, but they have to pay extra for that.

For the pushy sales people, that's a cultural problem within the MSP. We have sales people dedicated to MSP accounts that aren't on the same commission structure as our enterprise sales guys. That helps a lot. The MSP Sales guys get commission based on new contracts and retention so they are more interested in keeping customers happy than upselling them on something they don't need to make their sales targets.

As for incompetence and not getting stuff done... have good people, good managers, and pay your techs what they are worth. Give your techs a chance to learn, listen to their ideas, and innovate. The biggest complaints I get from MSP techs I talk too is dealing with burn out, refusal to pay industry average wages, and not enough time for hands on training. Too many MSPs are only interested in the bottom line rather than developing people, so they become meat grinder shops that go through a revolving door of techs. I've worked very hard not to be like that with my teams...

For the first two staff are often used to having a dedicated IT person on staff that can show up and teach you to use excel at the drop of a hat. When you're with an MSP you have to deal with the fact that we have a lot of customers and we have to prioritize who gets service when.

Work in a MSP and this is the big one. Whenever we onboard a new client it is always the same where they had internal IT and were used to just seeing the IT dude walking around and asking him a question.

On the flip side you also have those that came from another MSP that was ran awful. First call with the department manager and you get a list of everything that was ignored by the previous MSP and you get it all resolved and then fix some other shit you noticed while fixing it and you have that department manager loving you for life.

The biggest complaints I get from MSP techs I talk too is dealing with burn out, refusal to pay industry average wages, and not enough time for hands on training.

This has been my biggest complaint. MSPs tend to focus on if you can't fix it asap just escalate and move on to the next call/email instead of do some research and figure it out or reach out to a tier 2 to get some ideas and learn how to fix it in the future.

yup, too much focus on grinding away on tickets and not nearly enough on training and developing your techs.

"Write your certs if you want to get promoted"

Yeah and btw we have no training budget and study on your own time.

I'm glad my current employer has a very different approach.

Are you going to DattoCon?

Nope, Veeam all the way baby

I'm a small fry Network Engineer and really need to up my experience. What kind of home lab setup would you suggest to increase knowledge?

\r\homelab

You can easily find Cisco 3750's switches and 2800 series routers on Ebay. Older stuff, but good stuff for a CCNA lab that won't cost you much.

If you could wave a magic wand to automate a particularly annoying routine task, what would it be?

If I had a magic wand I would wave printers out of existence.

Were you impacted by the recent Kaseya vulnerability? I used to work for a large MSP that used Kaseya (wonderful tool, but super-dangerous if compromised) so I'm well aware of what could happen. On that note, I absolutely love what Kaseya can do if you have the necessary voodoo-level knowledge of command line operations. I'm an old-school command line wizard so I appreciate that.

As for me, I got out of MSP work and into the data center side of things, and honestly never want to work for an MSP again. I'm just glad I no longer have to deal with constantly ringing phones and irate customers.

I left the consulting world for the private sector for nearly a year.

Like many other career consultants I needed a break from the grind, but after months at a company I started to hate it, couldn't deal with the rigid team structure, got bored and went back to consulting and I'm much happier now.

Just more mindful of burnout now.

How did you get into the industry? Is there anything you would change in your approach/strategy? Any other tips appreciated, looking to start my career in IT through an apprenticeship.

I always wanted to work with computers. I went to University for a degree in Computer Science but programming wasn't for me, so I got a trade school degree instead.

From there I just worked my way through the ranks.

If your the type that prefers fixed hours and looking for benefits and pension try to get in with a large company or multinational. It will take you longer to move up, and you will have to specialize but you'll get benefits over time.

Where-as if you're the type to think on your feet like me and prefer to learn on the job MSPs and SMBs will probably be more to your liking. You'll have more freedom, and you'll learn more, but the pay and benefits won't be as good. Every major pay raise I've ever gotten in this sector has been from changing jobs, so don't be afraid to apply for stuff if you're feeling stuck.

Recent CS graduate and I've been applying to many Security positions. Dont have professional certs but I've been doing Tryhackme learning pathways. Any further advice to increase the chances of a job in the field?

Get on the CISSP track, when COVID allows go to hacker conferences to learn and network.

Don't smoke Marijuana if you want any chance of working for the US govt.

Work towards your CCISP, and when COVID eases up get to some hacker conferences to build up your knowledge

How do you feel all the changes made to accommodate the pandemic and work from home has affected cybersecurity?

AAAAAAAAAAAAAHHHHHHHHHHHHHHHH

A lot of companies weren't caught off guard and just had to spin up additional VPN and Teams/Zoom licenses, and buy a ton of laptops. Working from home was just an extension of the existing work from home policy and various IT departments handled it easily enough despite the initial panic buying of hardware + licensing and setups.

But many organizations have done crazy things like enable RDP servers open to the web, do BYOD (get users to use their own PCs etc), let users take home data on USB drives, no centralized monitoring of Anti-Virus, just utter madness.

Zoom was another problem with all the various security holes that cropped up shortly after it became the defacto work-from-home conference calling platform on a budget.

It's no wonder ransomware and phishing attacks went up by a significant percentage during the pandemic.

The upside of having an MSP like ours is we were able to use our experience to transition over our customers to work from home relatively easily (although we did work 3x 60 hours weeks at the start of the pandemic) because most of what was required was either already in place or copy+paste to implement.

I was at Build IT and Kyle Hanslovan asked a seemingly innocuous question that resonated with me and I think is much more nuanced than people are giving it credit for. The question was "Are you increasing the security risk of your clients simply by being their MSP?"

I'm curious what you do to instill a security first mind set at all levels of your company to ensure that you are increasing your clients' security posture and not compromising it in any way?

EDIT: When I say "all levels of your company" I mean all levels including ownership, sales, marketing, etc.

Our CEO reinforces a mentality of being pretty risk adverse. In other words "don't do anything that will get us sued"

With that in mind it's really up to the senior techs and the managers to reinforce a risk adverse mentality in terms of IT security.

Another key thing pointed out by my operations manager is to be willing to stop doing business with customers that repeatedly take risks, or don't listen to your advice. You have to avoid the bad press.

EDIT: You should also define and enforce certain basic standards on all your customers like Strong passwords, MFA, etc. If they can't even comply with those basic things then your shouldn't be doing business with them.

… aaand when the client asks their MSP that question, it’s time for the MSP to bring in the MSSP heh.

lol, yes

Do you work from home? I haven't been to the office since covid started. I'm just a network engineer, no management experience.

It's been 50:50 for us. Several of my techs work from home, others prefer to be in the office.

We were 100% working from home for several months last year and moved back to a hybrid environment as the restrictions were laxed.

We can't avoid onsite visits for some things as a number of my customers are required to stay open.

If you were running your own business, what would you insource vs outsource to an MSP, given your experience?

Depends on the size of your business and the complexity of your infrastructure.

We deal primarily with SMB's and non-profits that either can't afford their own dedicated IT people, or can only afford a full time Tier 1. So having an MSP behind them means they have access to a much larger and more diverse team.

Some customers use us as a full IT department, while others use us to supplement 1 or 2 in house techs so they have an escalation point, or just outsource the daily riff raff to our servicedesk.

Once your environment reaches a certain amount of complexity and number of users + devices it makes sense to have in house people. At that point an MSP can switch to more of a consulting role or take over certain IT jobs like Service Desk, helping with automation and planning, or handling Tier 3 + 4 escalations.

But to answer your question, if I started a small business (being a tech) I would do my own IT until it became too much for me to handle then I'd hire an MSP to take over. Once the business grew to a size where hiring a dedicated tech made sense, I'd hire a dedicate tech and scale down the MSP contract.

We do that with customers all the time.

How do you guys handle your internal documentation?

Do you guys deal with any self-resolution systems like user facing documentation?

With being a remote MSP do you guys do anything to vet the person calling in is who they say they are for things like password resets?

We have canned documentation platform for most things including network info, contracts, contacts, and procedures. Plus the usual mix of Calendars, Teams, and Visio.

We have a repo of pre-made PDF docs that we can send to customers on demand for self-resolution stuff like "how to setup active sync on your phone" which we tend to lean on during bigger projects and migrations.

We have a verification process for confirm user identity, and we enforce MFA for customers as well so even if a password gets stolen there's an extra layer of protection

How do you feel about people saying IT is not engineer?

In my career I've been called an Engineer and an Architect.

What does that have to do with IT? I have no clue, it just goes on my business card and translates to my pay grade.

How do you deal with ridiculous SLAs?

For me, I deal with them with a lot of stress.

At my old job with a lot of caffeine and overtime

At my current job I have the power to not put us in that position in the first place.

How do you find clients?

We have an office filled with Type A sales people that take care of that for us.

The usual mix of networking, cold calls, trade shows (when not dealing with COVID), follow ups, and references.

What's the biggest IT mess up you have had to deal with?

I am not at liberty to discus the details

REDACTED

I've cleaned up after a lot of Ransomware over the years. That's never fun.

If you regularly have to clean up ransomware from networks you manage, you've got some problems to address...

I was waste deep in it when Ransomware first appeared years ago and I still have PTSD about it.

Nowadays security practices have come a very long way and ransomware prevention is part of our daily lives, but we still get a fair number of walk-up clients begging us for help with ransomware out of the blue.

Are you one of those MSPs I hear stories about on Darknet Diaries? Like having a client with a domain controller with RDP open to the internet for all the users to log into?

I like to think of us as the guys that steal customers from those MSPs and fix them.

What is your current workstation setup? What tools are you using daily for work? And personal thanks!

I have a mediocre laptop with a docking station at the office with 2x additional monitors.

Nothing too fancy, just a Core i5 with 16gb of RAM and an SSD. Adequate, light weight, and good battery life.

I have a gaming laptop for personal use, my work laptop doesn't have to be a powerhouse.

Tool wise in my bag I carry an electric screw driver, bits, colored tape, my console cable, USB ethernet adapter (for a 2nd LAN plug), USB hub, a purple patch cord (so people don't steal it), flashlight, and a Patroit Supersonic thumbdrive.

Hi, As a MSP do you provide the product with the same price the vendor are offering? Or you are more expensive than the vendor, The second question let's say you manage EPP (endpoint security) for 100 clients how many people you need to manage them ?

Thanks

We are the vendor/reseller

For endpoint security it depends on the platform, the number of endpoints, and your incident count. Is your endpoint security person only doing monitoring? or remediation as well?

1 person can monitor thousands of endpoints easily enough, but remediation depends on the daily ticket volume.

Is it possible to progress your IT career at an MSP? Or should you be looking to job hop to an internal position within a year or two of experience?

[deleted]

I've have companies reject to interview me in the past because I worked for MSPs and they didn't believe anyone that's worked for an MSP would want to go onto a note relaxed environment. Crazy...

A lot of more senior techs in the consulting world (myself included) end up burning out on occasion and go into private sector. We end up hating it and going back into consulting before the end of the year.

Companies end up not wanting to hire former consultants because we don't last that long.

Dealing with siloed job roles, unions, change process, etc drives us absolutely crazy. Honestly we get bored!

But I'm the type that the stress doesn't bother me 99% of the time, I thrive under pressure.

Depends on the MSP, how big it is, and what the culture is like.

If it's a big team and they emphasize training and give you chances to learn, get certs, and shadow other techs then yes you can progress.

But if it's a 5-man show and they make you do the same thing every day then I'd say no...

I will say that almost every major raise I've ever gotten has been as the result of changing jobs.

As someone who deals with storage, how much do you hate EMC? IBM?

EMC - Evil Machine Company

I LOATH EMC, overpriced and under performing junk

What does the average day look like for someone who has worked in the MSP space for so long?

Come the office

Make coffee

Go through a ton of emails, review my tickets

Join various meetings with customers + vendors

Follow up on tasks with the team, take escalations as they come up

Work on whatever projects I've got on the go

Order varies based on how many fires I have to put out that day

I actually am a bit like you, I saw in a previous comment you started out in Programming but dropped it and pursed something still in computers. I remember doing the programming course for about a year and realized this really isn't my speed.

I have a degree in IT but kinda having trouble finding a job. Most places tell me I'm too green or when they ask for help desk I don't really have much experience in it. I did work for an MSP a few months before COVID happened and got laid off when it really hit the fan in the spring. My only experience so far is doing deployment both for that MSP and some contract work I had this past Spring.

Are there any tips or maybe something I can pursue with what experience I have? It's been really frustrating trying to get a job personally, I've tried recruiting agencies, Indeed/Ziprecruiter, etc but I feel like overall I just lack something to land a job.

Everyone starts on the helpdesk. Just keep applying and expect that you'll have to put your time in to build experience.

If there's something in IT that you want to specialize in or are curious about like Cloud, 365, Linux, Storage, networking, security, etc start researching, watch videos, do some lab stuff, anything to get some knowledge and experience. Even a little bit can go a long way in an interview as it shows that you could be worth pushing down that track.

"Looking for someone with all these certs, 10 years experience, willing to pay less than starting wage for a tier 1 tech" - more common than you think.

[removed]

TLDR: I've got a learning disability (ADHD) and it's extremely difficult for me to get through the mandatory courses. I'm honestly just really terrible at studying and writing exams.

I hold a ton of Networking, Storage, Veeam, Sales certs etc that I don't bother to put on my profile fyi. Mostly because they're expired.

Oh I've GONE for certs... but actually passing is a different problem.

I've got a learning disability and I can't learn in classrooms, I actually tend to fall asleep or lose attention with 30 minutes. So any exam that has a mandatory classroom phase or e-learning like Vmware, Cisco, lots of HP+Dell stuff, etc I have no hope in hell of passing. I often need multiple attempts at the exam to pass, so my employers learned to focus on the techs more likely to pass on the first try and let me focus on fixing things instead.

I have to learn hands on. Give me a new product and I'll figure out the ins and outs in an hour, but go to a class? snore

what's your background in this field? what would you say is different between standard business/residential IT vs engineer level positions?

I have an extensive background in residential and small business IT and am currently going to school for a BA in computer engineering, haven't considered "IT Engineer" because I've never actually heard of that term before, but it sounds like it may be more in line with my experience

My background was 7 years working at a small business that grew into an Enterprise due to various acquisitions. I started as a desktop tech and became a server generalist eventually becoming the senior engineer in the department.

Today I mostly manage the team, but I also work as a consultant to design and build IT solutions.

As a consultant I specialize in Full-stack Virtualization, Networking, Storage, Backup and Disaster Recovery, and I'm an Active Directory guru.

The major difference between SBS (Small Business) and the Enterprise world is the kind of technologies you get to work with.

Working at an SBS you have to be a generalist and know a bit of everything, while in the Enterprise you need to specialize more. That and most SBSs can't afford all the bells and whistles in terms of software and hardware, but it's not like they would take advantage of them either.

At the Enterprise level you can afford to pay someone that deals with something specific full time.

You've got Desktop Technicians, Network Engineers, Voice Engineers, Datacenter Engineers, Database Administrators, Storage Engineers, Application specialists, Mainframe, Email, Cloud, Security people. The list goes on.

As an IT Engineer/Manager at an MSP what are the biggest “blind spots” that you face when is comes to monitoring end-to-end infrastructure/performance/security/traffic etc? What tools do you use to monitor? What does your boss care about the most? What are you personally measured on?

As an IT Engineer/Manager at an MSP what are the biggest “blind spots” that you face when is comes to monitoring end-to-end infrastructure/performance/security/traffic etc?

It's not so much blind spots as information overload. It's easy to collect data, it's another to be able to filter it and turn it into something useful. So much of what we collect becomes noise.

What tools do you use to monitor?

We use a purpose made tool but I won't say which one.

What does your boss care about the most?

How much revenue are we generating? are the customers happy with the service?

What are you personally measured on?

How many contracts we sign over the course of the year, our retention rate, how much additional business (sales) have we generated, how many billable hours do I average in a week.

What's your automation strategy? And if you have one, which tools/tech are you using, and are you paying for them?

Automation is key, the more you can automate the more your technicians can do on a daily basis because they aren't busy dealing with the little things. The other side is more automation means you need less eyes paying attention to certain things.

Alerting, reporting, performance monitoring, patch management, etc is mostly automated like 90%+

We use a dedicated suite for most of the automation. I won't say which one.