Senior Software Engineer - Incident Insights & Readiness

About the position

The Incident Insights & Readiness SRE team at Datadog fosters a resilient culture by using incidents as learning opportunities and catalysts for growth. Our users are Datadog engineers, and we build the software, tooling, and operational frameworks that help them prepare for, respond to, and learn from incidents. We work closely with engineering teams across Datadog to analyze incidents and turn those insights into better tools, stronger incident response, and organizational learning. Our efforts empower Datadog to navigate unexpected failures confidently, efficiently, and with a commitment to continuous learning and systems improvement. At Datadog, we place value in our office culture - the relationships and collaboration it builds and the creativity it brings to the table. We operate as a hybrid workplace to ensure our Datadogs can create a work-life harmony that best fits them.

Responsibilities

  • Own and improve the on-call experience for the company by establishing best practices and building platforms to support on-call rotations and compensation.
  • Define how we respond to incidents, lead the design and implementation of software to streamline the process, and collaborate with product teams to improve incident response across Datadog. Our aim is to fully support our incident responders in dealing with complexity.
  • Contribute to the post-mortem process for the company, collaborating with teams on writing them, and identifying opportunities to reduce friction and enhance learning value for the organization. Our team also runs a weekly postmortem reading group.
  • Support various teams in facilitating incident reviews that emphasize learning and blamelessness. Help them share their learnings across the organization to improve the resilience of our people.
  • Provide technical leadership and day-to-day coaching to team members, accelerating their growth through design reviews, collaborative problem-solving and operational excellence best practices.
  • Train our on-callers in incident and post-mortem processes, sharing expertise in incident management best practices. This involves both introducing newcomers to on-call responsibilities and refreshing the knowledge of existing engineers.
  • Lead cross-functional initiatives in engineering organizations across Datadog, embedding with teams to understand their challenges and drive lasting improvements to reliability and operational excellence.

Requirements

  • At least 5 years of experience building software that solves real user problems. Experience designing new features and collaborating on code and technical design reviews. We primarily develop in Go and Python, with a bit of TypeScript.
  • Experience building or operating distributed systems, with familiarity with Kubernetes and an understanding of complex failure modes.
  • Demonstrated ability to independently own ambiguous technical problems from design through delivery while balancing long-term engineering quality with pragmatic execution.
  • Experience analyzing incidents, identifying systemic risks, and driving engineering improvements informed by operational learnings.
  • Experience participating in on-call rotations and improving incident response processes.
  • Empathy, collaboration, and communication skills in English to cultivate strong relationships across various teams in the organization
  • Experience mentoring engineers, driving cross-functional initiatives, and influencing technical direction without relying on organizational authority.

Nice-to-haves

  • Experience serving as an incident commander or incident coordinator is a plus.

Benefits

  • New hire stock equity (RSUs) and employee stock purchase plan (ESPP)
  • Continuous professional development, product training, and career pathing
  • Intradepartmental mentor and buddy program for in-house networking
  • An inclusive company culture, ability to join our Community Guilds (Datadog employee resource groups)
  • Access to Inclusion Talks, our internal panel discussions
  • Free, global mental health benefits for employees and dependents age 6+
  • Competitive global benefits
  • healthcare
  • dental
  • parental planning
  • mental health benefits
  • a 401(k) plan and match
  • paid time off
  • fitness reimbursements
  • a discounted employee stock purchase plan
Back to blog

Common Interview Questions And Answers

1. HOW DO YOU PLAN YOUR DAY?

This is what this question poses: When do you focus and start working seriously? What are the hours you work optimally? Are you a night owl? A morning bird? Remote teams can be made up of people working on different shifts and around the world, so you won't necessarily be stuck in the 9-5 schedule if it's not for you...

2. HOW DO YOU USE THE DIFFERENT COMMUNICATION TOOLS IN DIFFERENT SITUATIONS?

When you're working on a remote team, there's no way to chat in the hallway between meetings or catch up on the latest project during an office carpool. Therefore, virtual communication will be absolutely essential to get your work done...

3. WHAT IS "WORKING REMOTE" REALLY FOR YOU?

Many people want to work remotely because of the flexibility it allows. You can work anywhere and at any time of the day...

4. WHAT DO YOU NEED IN YOUR PHYSICAL WORKSPACE TO SUCCEED IN YOUR WORK?

With this question, companies are looking to see what equipment they may need to provide you with and to verify how aware you are of what remote working could mean for you physically and logistically...

5. HOW DO YOU PROCESS INFORMATION?

Several years ago, I was working in a team to plan a big event. My supervisor made us all work as a team before the big day. One of our activities has been to find out how each of us processes information...

6. HOW DO YOU MANAGE THE CALENDAR AND THE PROGRAM? WHICH APPLICATIONS / SYSTEM DO YOU USE?

Or you may receive even more specific questions, such as: What's on your calendar? Do you plan blocks of time to do certain types of work? Do you have an open calendar that everyone can see?...

7. HOW DO YOU ORGANIZE FILES, LINKS, AND TABS ON YOUR COMPUTER?

Just like your schedule, how you track files and other information is very important. After all, everything is digital!...

8. HOW TO PRIORITIZE WORK?

The day I watched Marie Forleo's film separating the important from the urgent, my life changed. Not all remote jobs start fast, but most of them are...

9. HOW DO YOU PREPARE FOR A MEETING AND PREPARE A MEETING? WHAT DO YOU SEE HAPPENING DURING THE MEETING?

Just as communication is essential when working remotely, so is organization. Because you won't have those opportunities in the elevator or a casual conversation in the lunchroom, you should take advantage of the little time you have in a video or phone conference...

10. HOW DO YOU USE TECHNOLOGY ON A DAILY BASIS, IN YOUR WORK AND FOR YOUR PLEASURE?

This is a great question because it shows your comfort level with technology, which is very important for a remote worker because you will be working with technology over time...

Other Jobs To Apply

Human Resources Manager (Full-Time, Remote) (Remote)

REMOTE ENTRY-LEVEL POSITION - Full/Part-Time Available| Beginner Friendly - Start ASAP

Flight Attendant (PT or FT with Premium Pay)

Aetna Remote Jobs, Aetna Careers Remote @ Joblagii ( Manager Account Management )

Software Engineer Principal (Remote)

Airlines Customer Service Remote Jobs, Remote Airline Jobs Atlanta Apply In USA ( Customer Assi[...]

Amazon Delivery Driver

Senior SOC Analyst - 2nd Shift Weekends

Portuguese (Brazilian or European) Remote Interpreter

Amazon Content Moderator Job (Remote) $22/H ?? Part-Time ?? US

Regulatory and Start Up Specialist (Portuguese speaker) (Fully Remote)

Remote Cyber Security Engineer - Flexible, Part-Time

Tier 1 SOC Analyst - REMOTE

Chat Agent

Remote Amazon Data Entry Jobs - No Experience - Part-Time

Flexible Sortation Warehouse Associate(Smyrna)

Amazon Customer Service - Work From Home $16-$35/hr

Philadelphia Overnight Warehouse Associate with $21-$25/hr Pay

Warehouse Associate (PT/FT with 1st/2nd/3rd shift openings)

Data Entry / Medical Billing Work at Jobs

Registered Nurse Triage and Transitions Team PRN

Administrative Assistant (Remote)

Remote Customer Service Specialist

Warehouse General Labor & Forklift

Remote Call Center Representative

Area Manager II

*New - Infusion Prior Authorization Specialist – Remote Position – (Experienced)

Amazon Work From Home Data Entry Jobs, Amazon Looking To Fill 250 Work-From-Home Jobs Apply For[...]

Earn Extra Cash as a Local Delivery Driver

Sr. Ad Operations Manager, Social Marketing

Warehouse Operations

Warehouse / Supply Assistant

Abbeville,LA:XL Warehouse Associate at Amazon | Hiring

Part-Time Application Reader, Undergraduate Admissions

Amazon Fulfillment Center - Warehouse

Amazon Data Entry Jobs from home - No Experience Needed

Amazon Delivery Driver

Scalehouse Attendant

Resource Liaison

Cashier Assistant (Front End)

Remote Product Tester - $25-45 per hour

amazon flex $15+/ Hour (Sign on Bonus)!

Software Engineer - Prime Video

Entry Level Customer Service - Work From Home ™

Amazon Package Delivery Driver - Earn $20.00 - $28.00/hr

Student Warehouse Operative, Entry Level (No Experience Needed)

Walmart Data Entry Work From Home (Entry Level/No Experience Required)

Clinical Quality PM, One Medical Senior Health

Weekend Remote Chat Support Jobs – Work Saturdays & Sundays Only | $25–$35/hr

Online Proofreader