Written by Jakub Balhar, Chair of the Zowe Technical Steering Committee and Staff Software Engineer at Broadcom 

As anyone working in the field of cyber-security knows first-hand, the world is becoming increasingly more dangerous in the cyber-realm. As the sheer volume of data continues to increase, and along with it the sums of money to potentially be gained by bad actors through cyber-attacks, the effectiveness and sophistication of the attacks themselves is also on the rise. To adapt to this reality, it is essential that Open Source projects face the challenge of developing secure code head-on in order to withstand and deflect any potential cyber-attacks. Zowe, part of the The Open Mainframe Project, which was launched a few years ago, continues to develop in a way that responds to this reality by incorporating the highest level of security standards in its very design.

In response to vulnerabilities in cyber-security, the requirements for development and operation of open source software continue to become more refined and specified in key industry regulations such as the Executive Order on Improving the Nation’s Cybersecurity in the USA, and the Cyber Resilience Act in the European Union. The Zowe open source project not only meets, but goes beyond these requirements both in terms of its development, as well as through its performative organizational operations.

Three years ago, we wrote about Zowe’s cyber-security journey towards Dev Sec Ops in the post: Secure Development Practices within Zowe | by Jakub Balhar | Zowe | Medium. As a follow up, we went into more detail about secure operations in Tips and Tricks: Zowe Secure setup for Production | by Jakub Balhar | Zowe | Medium.

Let’s turn our focus now towards the biggest concerns and questions about project security that should be addressed prior to solution adoption. Based on regulations and feedback from Zowe users, these key questions around security that have come to light:

• If there is a security issue with the project, will it be properly handled in a timely manner, and will we receive notification in time in order to apply a fix?
• Is there a process in place to limit risks coming from dependencies?
• What course should I take if a security issue is found?
• What mechanisms are in place to prevent malicious code from getting into the codebase of the project?
• Who will be able to provide support if I run into a problem?

Vulnerability related processes

Part of the organizational operations of Zowe is holding a bi-weekly Security Working group call, with further communications across a dedicated Slack channel. During the call, the working group investigates any possible vulnerabilities, either directly within the Zowe code, or vulnerabilities within dependencies used across the full spectrum of Zowe projects. This working group also regularly audits violations of policies across all Zowe repositories including any merges to main branches without properly passing all necessary tests or other stated requirements. In addition to investigating and enforcing established security practices across Zowe, this working group is also responsible for initiating general security improvements across the Zowe project.

What happens when one of the scanning tools used by Zowe or one of the vendors introduces a new vulnerability in the dependencies?

In the case that a new vulnerability is identified, the first step is for a new issue to be created in the private security repository. If the severity of the vulnerability is determined to be non-critical, the Security Working Group discusses the issue and next steps to address the issue. In critical cases, the discussion starts directly in the private #zowe-security Slack channel. The issue is assigned to one of the security experts, who then validates whether the issue is or is not exploitable. If the issue is not exploitable, the update and fix can wait for the next planned minor or patch release. The issue gets a milestone assigned for the next release and remains on the board until the release is prepared wherein the issue has been addressed, wherein the issue can subsequently be removed.

In the case where a vulnerability is determined to be exploitable, the Security working group validates the issue’s severity according to the Analysis and Assessment section of Zowe Security Policy. Based on findings, either a new release to fix the issue within a relevant timeframe is proposed, or the fix is to be incorporated in the next scheduled release.

When Zowe releases a new version, the vulnerabilities fixed in the previous release are published in Zowe Docs (Version 2.16.0 (May 2024) | Zowe Docs). The Zowe Technical Steering Committee encourages users to migrate to the one before the latest release whenever there is a new release. When an issue is found within Zowe code, a new Common Vulnerability and Exposure (CVE) for the issue is created and the issue is then disclosed in stages based on the disclosure process. This process begins initially with a notification to Zowe extenders. When the CVE and issue is subsequently fixed and published, a notification is also sent out through the zowe-user mailing list.

Starting with Zowe Version 3, the Security Working group intends to begin active monitoring of release dependencies, whereby all dependencies will be proactively upgraded whenever such upgrades are made available.

How does the Security Working Group discover vulnerabilities?

The Zowe Security Working Group is actively collaborating with LFX on improvements to the LFX Security tool to ultimately serve as the basic utility to analyze security with respect to dependencies. At the time of this blogging, this utility still has some issues with Zowe development processes. In the meantime, Zowe depends on vendors sharing the results of their scans. Among such analytics, the use of Synopsis BlackDuck is notable as most of the reports are made available via this useful tool.

Zowe guidelines describe the process of how new dependencies can be added to the codebase. The outlined process requires squads to discuss and agree on the addition of any and all new dependencies. In practice, however, this cross-squad consensus process has only proven to be marginally effective as the Zowe project has, upon occasion, allowed for new dependencies with only the standard review process. The practice of auditing has also been somewhat insufficient to monitor for all newly introduced dependencies.

As always, the Zowe community continues to strive for transparency, both in terms of its codebase as well as its organizational processes. That said, if you find a security issue with Zowe, we sincerely encourage you to direct the issue to zowe-security@lists.openmainframeproject.org. The Security working group will reach out to you and help resolve the issue.

What is being done to improve code quality?

All the code produced by Zowe is version controlled and lives within GitHub repositories. The main branches used to produce the Zowe artifacts are protected and the code is only merged to the master branches via Pull Requests after satisfying essential review requirements including the approval of one or more reviewers. Additionally, the code needs to go through one of the static analysis tools such as SonarCloud (Projects — Zowe organization — SonarCloud). The automated test suite that is part of every project also needs to be run, in order to validate that regressions are not introduced to the current code. Admittedly, the automated test suite does not cover all cases so far, and Zowe development teams are working on improvements to tests for all areas across Zowe.

Who will provide support if I run into a problem?

If you encounter any problems with your implementation of Zowe, there are a few places you can turn. Firstly, there is an active Zowe community that works on the issues across the various Zowe repositories, including triaging and ultimately addressing issues. For community support, go to zowe.org.

If, however, you’re looking for someone who will have your back when it comes to the SLAs that work for your business, Zowe partners with Open Mainframe Project in the development of the Zowe Conformant Support Provider Program — Open Mainframe Project. Vendors participating in this program are guaranteed assistance with base functionality and capabilities in which they receive both support with their implementation of Zowe as well as support for any specific projects within Zowe.

Based on the answers to the previous questions, Zowe seems to be doing a decent job now, but why should I believe this will continue going forward?

The answer is auditing.

Auditing

It’s great to claim that the contributors to Zowe are following some set of guidelines, but without regular validation, the claim doesn’t hold much weight. With this in mind, we ran an internal audit of the Open SSF Best Practices badge within all of the projects that comprise the Open SSF. The Open SSF Audit results are available in community/OpenSSFBestPractices.md at master · zowe/community (github.com).

To map and validate the projects so they don’t introduce problematic code, we go through the problems and policy violations within GitHub in a bi-weekly Security Workgroup meeting. The workgroup has members from every squad, and so we are able to validate with the squad’s security representative what was happening and whether we need to be concerned.

Conclusion

Core contributors to the Zowe project are well aware of the risks associated with code vulnerabilities, as well as those vulnerabilities introduced through the supply chain (i.e. dependencies). In facing these risks, Zowe and the Open Mainframe Project will continue to take active steps, both in terms of using and developing analytical testing tools, as well as instituting performative organizational practices including the Zowe Security Working Group to actively mitigate these risks. As the Zowe project continues to forge new ground, new tools and practices will be developed and implemented to maintain the highest level of cyber-security within the Zowe platform. This, in combination with continuing to provide expert support to vendors and members of the Zowe community, will help Zowe keep its place as a front-runner in providing a highly-secure platform for mainframe business solutions.

The Zowe Security Policy is publicly available on zowe.org/security. If you enjoyed this blog checkout more Zowe blogs here. Or, ask a question and join the conversation on the Open Mainframe Project Slack Channel #Zowe-help or #Zowe-onboarding. If this is your first time using the OMP slack channel register here.