On November 16, 2020, an outage occurred shortly after a major upgrade to ASX’s stock trading platform, ASX Trade, known as the ASX Trade Refresh Project (the Project).
ASIC and the Reserve Bank of Australia (RBA) (the regulators) view operational incidents of this nature with great concern. To examine the issues, regulators have informed ASX that they expect an independent review of the project to be conducted in the first half of 2021. In consultation with regulators, ASX has appointed IBM Australia Limited to undertake a review by an independent expert.
The objective of the independent expert review was to review the project and assess whether it complied with internationally recognized standards or frameworks and relevant securities industry practices.
The independent expert, IBM Australia Limited, reported on their findings and a summary of their findings is attached as an annex.
Overall, the independent expert found that ASX met or exceeded industry best practices in 58 of the 75 capacities assessed, including:
- developing a business case and managing project change, which went beyond accepted practices
- the project received and had access to sufficient financial, time, human and technological resources at all stages of delivery to achieve its objectives
- communications with key stakeholders were managed appropriately, and
- ASX incident management actions were appropriate.
Significantly, the independent expert identified several key gaps in the project, including:
- factors suggesting that the ASX Trade system was not ready to go into service given ASX’s almost zero appetite for downtime. This was the case even though the formal implementation readiness processes were completed and verified by multiple parties without objection to commissioning.
- there were gaps in the rigor applied to the risk management process and problems with project execution expected for a project of this nature, and
- Risk and problem management, project compliance with ASX practices, project requirements, and project testing strategy / planning did not meet accepted industry practices. It was not reasonable to expect that the test plan used would satisfy ASX’s almost zero appetite for downtime.
The independent expert made recommendations in seven key categories: risk, governance, delivery, requirements, supplier management, testing and incident management.
ASX has provided regulators with its high-level response to the independent review. Regulators note that ASX has agreed to respond to the review’s recommendations and apologized for the market disruption.
ASIC President Joe Longo said: “The independent expert found that ASX met or exceeded industry leading practices in most areas, but the conclusion that the project was not ready for commissioning is very disappointing. ASX recognized and accepted the need for improvement. However, we do require assurance that these improvements are implemented effectively and result in an overall improvement in ASX’s project management practices across the enterprise. ‘
Regulators expect ASX to apply IBM knowledge across the ASX group to ensure that existing and proposed projects, including the CHESS replacement program, are managed and implemented appropriately. .
Regulators will review the independent review, including the recommendations and the ASX’s official response. ASIC is also undertaking a separate investigation into the ASX Trade outage to determine whether ASX has fulfilled its obligations under its Australian market license, which is pending.
ASIC continues to engage with market participants, participants, institutional investors and other stakeholders on the impact of the incident and will work with industry to identify broader market adjustments, if any, that may be necessary to reduce the impact of any future incident.
Summary of Independent Review Findings
1. Whether it was reasonable to expect the new trading platform to be ready for successful production implementation and continued availability
Although the formal preparation processes were completed and verified by multiple parties with no objection to commissioning, 7 factors suggested that the platform was not ready for commissioning given the almost zero appetite for commissioning. ‘ASX for downtime. These factors were:
- historical quality indicators of software products
- additional testing needs noted
- the quantity of open defects
- gaps in end-to-end test coverage
- proximity to end-of-year change freeze windows for participants
- risk probability ratings
- lack of evidence of challenges to the risk rating or commissioning
2. Did the project have sufficient resources
The project was provided with and had access to sufficient financial, time, people and technology resources at all stages of delivery to achieve its objectives. However, IBM also concluded that the project could have benefited from additional and independent review.
3. The effectiveness of the change control process
The project change request process was performed in accordance with ASX standards. Project change requests have been reviewed by the appropriate internal governance forums and have all been deemed fit for approval.
4. The robustness and rigor applied to the management of risks and issues
There were gaps in the rigor applied to the risk management process and problems with project execution expected for a project of this nature. These gaps included:
- missed opportunities to identify additional risks
- differences between project delivery risk models and enterprise delivery risk processes
- the project would have benefited from the involvement of risk resources with greater experience in technical projects
- governance was transferred to a group that had a wide range of responsibilities, and the group did not include a key role. The change diluted the focus on the project.
5. Whether it was reasonable to expect the project test plan to be effective, given the risk appetite and criticality of the ASX Trade system
It was not reasonable to expect that the test plan used would meet the near zero appetite for downtime for systemically important national infrastructure, as stated in the ASX test policy. .
6. The implications of the project on stakeholders
The incident had negative implications for market participants and ASIC, but ASX was unable to anticipate these implications for stakeholders to emerge. Communication with key stakeholders was managed appropriately during project delivery, before and after commissioning. The incident management measures taken by ASX were deemed appropriate and resulted in the correct course of action to reduce the impact on project stakeholders.
7. Whether during the 2020 incident, ASX took into consideration the lessons learned from the 2016 incident.
The actions taken by ASX in the 2020 incident were appropriate and reflected lessons learned from the 2016 incident.
8. Aspects that have met or exceeded industry standards, frameworks or practices
Most of the practices of the ASX project met the expectations of the main industry practices. Developing a business case and managing project change emerged as areas that went beyond accepted practices. Evidence of continuous improvement, during and after the project, was also noted in company practices, particularly risk, project delivery and business continuity management.
There was good alignment with accepted practices in governance, risk management and project / program execution. While this is only an indication, of the 75 capabilities included in the scope of the review, 58 have met or exceeded industry practice for a project of this nature.
9. Aspects of the project that did not meet or exceed industry standards, frameworks or practices.
While the majority of ASX project practices met expectations of leading industry practices, IBM concluded that risk and problem management, project compliance with ASX practices, project requirements, and strategy / planning for project testing did not meet accepted industry practice. Regarding the testing practices used by ASX during the delivery of the project, IBM concluded that the ASX test documentation and the implementation of the associated processes were largely not in line with the expectations of the leading practices. Of the industry. Existing governance processes did not act sufficiently in a timely manner, resulting in an extended period before an upgrade was performed, increasing operational risk.
The initial project governance processes did not show sufficient scrutiny by program management on compliance with ASX delivery practices. IBM noted that ASX did not have a formal quality management process and that there were opportunities to improve ASX enterprise-wide project practices.