On-Call Support Process: Tier 1 and Tier 2 Roles
Purpose
This document outlines the procedure for handling on-call responsibilities for the Tier 1 Support team and defines the process for escalation and collaboration with the Tier 2 Support team when additional assistance is required. The goal is to ensure seamless communication, efficient incident resolution, and consistent coverage during on-call periods.
Scope
This process applies to all Tier 1 and Tier 2 Support team members participating in the on-call rotation for after-hours and weekend support.
Roles & Responsibilities
Tier 1 (Primary On-Call)
- Acts as the first point of contact for all alerts, incidents, or pages received during their on-call shift (10AM Thursday to the following Thursday at 10AM).
- Responsible for triaging incidents, gathering initial details from incoming cases, and taking immediate action (calling an incident, paging the appropriate engineering group) based on the issue.
- Communicates with Leadership if there is potential impact to services or customers.
- Maintains engagement if an incident occurs during the day; opens a parent case and provides institution updates as communicated through the incident channel.
- Engages Tier 2 Support when assistance is needed or when the incident/case requires escalation beyond Tier 1 scope.
Tier 2 (Secondary On-Call / Support Backup)
- Serves as the designated backup to the Tier 1 on-call representative.
- Provides technical guidance and troubleshooting support when Tier 1 requests assistance.
- Assists in resolving escalated cases that exceed Tier 1 technical capabilities, rerouting cases, paging engineers, etc.
- May take ownership of the incident if it transitions into a higher-level category (critical outage).
- Ensures Tier 1 receives troubleshooting steps or information to build knowledge for future cases/incidents.
Escalation Procedure
1. Tier 1 receives an alert or page
- Acknowledge within 5 minutes of receipt.
- Contact the institution to gather needed information.
- Review monitoring details, logs, and alerts to determine impact and priority.
2. If Tier 1 can resolve independently
- Proceed with resolution by sending cases to the appropriate group or paging engineering.
- Provide an update to the institution via phone call or notes in the case.
3. If Tier 1 requires assistance
- Contact the assigned Tier 2 on-call (through a low severity incident and paging “Tier 2 Banno Support”).
- Provide relevant details:
- Case number
- Error messages
- Steps already taken
- Tier 2 should respond within 10–15 minutes to collaborate.
4. If the issue escalates beyond Tier 2
- Tier 2 engages Engineering and/or Leadership as appropriate and increases the incident severity if needed.
- Tier 1 remains informed and retains primary ownership unless otherwise directed.
Role Identification
- Tier 1 On-Call: Identified in the incident.io schedule; should review who their Tier 2 backup is for the same schedule.
- Tier 2 Support: Designated as the assigned backup for the same coverage period.
- Both roles must ensure contact information is current in the JH portal.
Benefits of Tier 2 Support for On-Call Operations
- Improved Efficiency: Reduces time to resolution with additional technical expertise.
- Enhanced Confidence: Provides real-time support for Tier 1 during critical incidents.
- Consistent Coverage: Ensures service quality and customer satisfaction during off-hours.
- Skill Development: Encourages collaboration and knowledge-sharing.
- Reduced Escalations: Minimizes unnecessary handoffs to engineering or management.
Review & Updates
This process will be reviewed regularly to ensure alignment with support team structure, escalation paths, and incident volume trends. Feedback from Tier 1 and Tier 2 team members will be incorporated to improve efficiency and clarity.