6 Lessons Learned Phase
6.1 Overview
In this phase, the Red Teaming Provider should deliver the final red teaming report, which should contain the overall assessment of the Member Organization's resilience against targeted cyber-attacks.
The Blue Team should deliver the blue team report with their observations, findings and recommendations and should focus on the alerts and actions taken as part of the detection and response capabilities of the Member Organization.
Once the final red and blue team reports are distributed to all Teams. The White Team should invite the Red, Blue and Green Teams to participate in a (360 degrees) feedback session in which they share their observations and experiences for learning purposes (of the staff and management involved), to understand what capabilities need to be improved (e.g. prevent, detect and respond) and (enhancing) future exercises.
After the feedback session, a Replay Exercise should be organized, led by the Blue and Red Team. The objective of the joint Replay Exercise is to step through the red team exercise, discussing all the relevant actions and observations, highlighted from both angles, i.e. the Blue and Red Team.
The next step is the overall evaluation of the red teaming exercise processes itself. The outcome of the evaluation may contribute to vital information to enhance the Financial Entities Ethical Red Teaming Framework for future exercises.
The White Team should create a remediation plan based on the detailed observations and recommendations.
To ensure that all Member Organizations within the Financial Sector benefit from these red teaming exercises, an anonymized summary report of the executed red teaming test should be provided, and if required presented. The sharing of this report should be limited to the agreed with the closed community (i.e. addresses) and within the boundaries of the agreed communication protocol.
The duration of this phase is approximately four (4) weeks.
Based on the evaluations, feedback and sharing sessions SAMA should review, discuss and initiate adjustments to improve the current Framework, if required.
An overview of the Lessons Learned process is depicted below:
6.2 Debriefing
The Red Teaming Provider finalizes the red teaming report and presents the output to all the White and Blue Team. Simultaneously, the Blue Team should create a blue team report. The blue team report should provide the observations from the perspective of the Blue Team and should include the alert and events detected; the actions initiated and the result of these actions. The blue team reports should also provide the Blue Team's recommendations for improvements.
It is important that all teams (i.e. Green, White and Red) that were directly involved in the red teaming test provide their (360 degrees) feedback on the executed red teaming test. The White Team should initiate and schedule a joint replay session with all the teams.
6.3 Red and Blue Team Replay Exercise
After delivering the red and blue team reports, the White Team should organize a Replay Exercise. During this Replay Exercise, the Blue and Red Team jointly perform a chronological walkthrough of the red teaming exercise and the relevant alerts, events and attack steps that were initiated.
The purpose of the Replay Exercise is to explain and discuss each step and action individually to assess whether the alert or detected event lead to the expected actions. It is important to determine whether the initiated actions led to the expected results and whether the actions were correctly initiated or should be subject for improvement.
Replaying the red teaming exercise should ensure the more comprehensive (in-depth) understanding of the performed attack patterns, the current maturity of the detection and response capabilities and the implemented layered defenses or controls within the tested Member Organization.
Additionally, the White Team may repeat the replay exercise for specific target audiences within the Member Organization. It is strongly suggested to re-perform the replay exercise for:
a. The relevant staff members within the IT organization - the scope of this session can be a very in-depth and technical session in order to provide the relevant insights in the technical and procedural aspects.
Note. When the level of detail is insufficient or the attack steps cannot be demonstrated, then there con be a tendency for members within the IT organization to downplay these attacks or argue that the exercise is just theoretical.
b.
The Senior Management - a high-level replay session with the Senior Management should also seek to raise awareness and educate Senior Management. The replay session should provide an overview and objective of the red teaming exercise, an overview of the performed attacks and responses, an overview of the current detection capabilities and an overview of the suggested improvements required to further improve the cyber resilience.
6.4 Defining the Remediation Plan
The White Team should create a remediation plan based on the recommendations provided by the Red Team and Blue Team. The White Team should:
- analyze the observations and recommendations;
- determine the improvements regarding the detection and response capabilities;
- determine the associated risks and priorities.
The White Team shares the internally agreed and approved remediation plan with the Green Team and periodically track the remediation progress to ensure that the vulnerabilities identified are monitored and mitigated.
Note; The Green Team should not actively share nor distribute the red and blue team reports, nor the evaluation reports, nor the remediation plan unless the Member Organization provides written permission. As stated earlier, the primary objective of this Framework is exercising, learning and sharing. 6.5 Remediate Identified Vulnerabilities
Shortly after finishing the red teaming exercise, the Member Organization, i.e. the White Team should start executing the agreed remediation activities and address the identified vulnerabilities.
The Member Organization should be tracking the actual remediation progress to ensure the timely execution and delivery of the improved capabilities. The Member Organization should ensure that the Cyber Security Committee (and if required the Senior Management) is periodically updated on the progress of the planned remediation actions, and should request support when remediation activities do not progress as expected.
6.6 Sharing of the Lesson Learned
An important activity within the lesson-learned phase is to provide an anonymized summary report of the executed red teaming test, which might be shared with Members Organizations Committees like e.g. BCIS.
Sharing the summarized report and the lessons learned helps other Member Organizations build the knowledge and experience they need to improve their own cyber resilience.
Note. The sharing of the report and lessons learned should be limited to the agreed with the closed community (i.e. addresses) and within the boundaries of the agreed communication protocol. By applying the lessons learned within their own Member Organizations the cyber resilience of the overall Saudi Arabian Financial Sector will improve, regardless of whether the Member Organizations are considered systematic for the sector, or not.
6.7 Enhancing the Red Teaming Framework
Based on the evaluations, feedback and lessons learned sharing sessions SAMA should review, discuss and initiate adjustments to improve the current Framework for future red teaming exercises.