THE LINUX FOUNDATION PROJECTS
Blog | zopen community

Summer Mentorships 2026: Porting Bazel to IBM z/OS – My Journey Through Compatibility Challenges and Debugging

By | September 30, 2026

Written by Rohini Sankari & Harsh Sagar, Open Mainframe Project Summer Mentorship 2026, mentee researchers, guided by mentor Harita D & Igor Todorovski

Hi, I am Rohini Sankari, and I am writing this blog with my fellow mentee Harsh Sagar. We both joined the LFX Mentorship Program with the same goal: to port Bazel to IBM z/OS, in collaboration with The Open Mainframe Project and the zopen community. 

Neither of us expected to be selected, but getting the acceptance was an amazing moment. This blog documents our project – the challenges we faced, the fixes we implemented, and the lessons we learned along the way. 

What is Bazel?

Bazel is an open-source build and test tool developed by Google. It is designed to build and test software projects efficiently, particularly large and complex projects with many source files and dependencies. 

A build system is responsible for taking source code and turning it into software that can be executed or packaged. This may involve compiling source files, processing generated code, resolving dependencies, linking libraries, and running tests. As projects grow, managing all these steps manually becomes difficult. This is where Bazel comes in. 

What is Bazel used for?

Bazel manages the different steps involved in building and testing a software project. Some of its important capabilities include: 

  • Building software by compiling source code and producing binaries or other build artifacts.
  • Managing dependencies between different parts of a project. 
  • Running tests as part of the build process. 
  • Incremental builds, where Bazel can avoid rebuilding work that has not changed. 
  • Reproducible builds, helping ensure that the same source and build configuration produce consistent results.
  • Supporting multiple programming languages, making it suitable for large projects containing different types of source code. 

One of the important ideas behind Bazel is that a build can be described in terms of actions and dependencies. Bazel analyzes what needs to be done and creates the necessary build actions. This allows it to determine which parts of a project actually need to be rebuilt instead of rebuilding everything from scratch. 

For large software projects, this can make the build process significantly more manageable and efficient. 

Why is Bazel important on IBM z/OS?

IBM z/OS is a platform used for large-scale and business-critical computing. As software development on the platform continues to evolve, having access to modern development and build tools becomes increasingly valuable. 

Many modern open-source projects already use Bazel as their build system. If Bazel can run natively on z/OS, developers working on the platform can use the same build technology and workflows instead of having to depend on a different environment just to perform the build. 

Porting Bazel to z/OS can therefore help: 

  • Bring a modern build system to the z/OS ecosystem. 
  • Make it easier to build open-source projects that depend on Bazel. 
  • Support more consistent and automated build workflows. 
  • Reduce the gap between modern open-source development practices and z/OS development. 
  • Give developers on z/OS access to the same build tooling used by projects on other platforms. 

From Bazel Documentation:

Before making any z/OS-specific changes, I started with Bazel’s official documentation for building Bazel from scratch, also known as bootstrapping. 

The first important detail was that bootstrapping starts with a Bazel distribution archive: bazel-<version>-dist.zip 

rather than simply cloning the GitHub source repository. The distribution archive contains generated files needed for the bootstrap process. 

The documentation also gave us the basic prerequisites: 

  • Bash 
  • C/C++ build tools 
  • zip and unzip 
  • Python 
  • JDK 21 

The standard bootstrap command is:

env EXTRA_BAZEL_ARGS=”–tool_java_runtime_version=local_jdk” bash ./compile.sh

A successful build produces the Bazel binary under:output/bazel 

This gave us a clear starting point. But there was an important difference: the documentation explained how to bootstrap Bazel on supported platforms, not how to do it on IBM z/OS. 

That meant our task was not simply to follow the instructions. We had to find out what happened when each part of the bootstrap process ran on z/OS, understand the failures, and determine what needed to change. 

In simple terms, our process became:

Documentation → Prepare z/OS Environment → Build → Analyze Failure → Apply Fix → Rebuild And that cycle became the foundation of my Bazel porting journey. 

Build Environment and Reproduction Steps

Environment Variables: 

Please set the following environment variables before build: 

export JAVA_HOME=<JDK-21-or-later-path> 

export PATH=$JAVA_HOME/bin:/data/zopen/usr/local/altbin:/data/zopen/usr/local/bin:$PATH

export ZOS_PYTHON_TARBALL=/data/rohini/python_pkg/python-3.14.4.1-s390x-ibm-zos.tar.gz

export ZOS_GO_SDK=<local-go-sdk-path> 

If BAZEL_JAVAC_OPTS are currently set in buildenv, include those too. 

Then show verification commands: 

echo $JAVA_HOME 

java -version 

echo $ZOS_PYTHON_TARBALL 

ls -lh "$ZOS_PYTHON_TARBALL" 

Before starting the build, binary files in the source and dependency trees are explicitly set to binary tagging using chtag -b. 

find . \( \ 

-name “*.jar” -o \ 

-name “*.zip” -o \

-name “*.war” -o \ 

-name “*.ear” -o \ 

-name “*.class” -o \ 

-name “*.so” -o \ 

-name “*.a” -o \ 

-name “*.o” -o \ 

-name “*.dll” -o \ 

-name “*.exe” -o \ 

-name “*.png” -o \ 

-name “*.jpg” -o \ 

-name “*.jpeg” -o \ 

-name “*.gif” -o \ 

-name “*.ico” -o \ 

-name “*.pdf” \ 

\) -exec chtag -b {} \; 

Finally the build command: zopen build -vv

Current State

The Bazel build has progressed through the previously identified Java, platform detection, rules_python, rules_go, JNI, and C/C++ toolchain configuration issues. The z/OS-specific compiler wrapper is also being selected correctly. 

The current failure occurs during native C/C++ compilation, while compiling protobuf: Compiling src/google/protobuf/wire_format_lite.cc [for tool] 

The compilation is invoked through: 

clang_zos_wrapper.sh 

The build terminates with: 

Target //src:bazel_nojdk failed to build 

ERROR: Could not build Bazel 

***ERROR: Make (full) failed. 

The immediate compiler diagnostic is appearing as garbled output in the terminal, so the underlying cause of the protobuf compilation failure still needs to be identified.

Current status: Bazel is reaching the native C/C++ compilation stage, but the bootstrap build is not yet complete. 

Explore the Code

The complete changes from our Bazel porting work, including the fixes and ongoing changes, are available in our GitHub fork. You can explore the code, patches, and commits here: 

GitHub fork Link

Next Steps

The next step is to analyze the protobuf compilation failure, verify file tagging and encoding, and isolate the issue if necessary. The required z/OS-specific fixes will then be applied, followed by a full Bazel build to address any remaining platform-specific issues until bootstrapping completes successfully. 

From here, the blog is divided into two parts, bringing together our individual experiences from the same journey. Part 1 covers my (Rohini’s) work on the Bazel port, while Part 2 shares Harsh’s perspective on the mentorship and the challenges we worked through together.. 

Part 1: What Issues Did I Face During the Porting Process and How Did I Fix Them?

Porting Bazel to z/OS involved a series of compatibility issues across the bootstrap environment, Java runtime, external Bazel rules, and native C/C++ toolchain. Most of these issues were caused by assumptions in the existing implementation about Linux or other Unix-like environments. 

For each issue, I investigated the failure, identified the platform-specific assumption, and introduced a z/OS-specific change while keeping the existing behavior for other platforms unchanged. 

  1. GNU Utility Compatibility

Problem: 

The bootstrap scripts depended on GNU-specific grep options that were not supported by the native z/OS grep. 

FSUMA930 grep: Unknown option -o 

FSUMA930 grep: Unknown option -- 

Although GNU grep was installed through zopen, the build was still resolving /bin/grep. Investigation and Fix: 

I checked: 

which grep 

type grep 

grep –version 

echo $PATH 

This showed that the native z/OS implementation was being selected instead of the GNU version. I updated the build environment to prioritize the zopen utilities and cleared Bash’s command hash:

export PATH=/data/zopen/usr/local/altbin:/data/zopen/usr/local/bin:$PATH

hash -r 

Repository Change 

  • buildenv 

Outcome: 

The build correctly selected GNU grep, and the previous FSUMA930 errors disappeared. 

  1. Missing Java Generated Classes

Problem: 

Java compilation initially produced hundreds of errors involving missing generated classes: cannot find symbol 

AutoValue_… 

AutoOneOf_... 

Investigation: 

I verified that: 

  • the AutoValue dependencies were present; 
  • the annotation processor classes were present; 
  • the processors were registered in the JAR; 
  • generated AutoValue_* sources were not being created during compilation. 

I also enabled annotation-processing diagnostics to determine whether the processors were being executed. 

Fix: 

I explicitly configured the required annotation processors through BAZEL_JAVAC_OPTS so that AutoValue, AutoOneOf, and AutoService processing was performed during bootstrap compilation. 

Repository Change: 

  • buildenv 

Outcome:

The required generated Java sources were produced and the bootstrap progressed beyond this compilation failure. 

  1. Bazel Startup Failure: OpenJ9 Garbage Collector Detection

Problem: 

After Bazel successfully completed compilation, the resulting Bazel binary failed during startup/runtime initialization. 

The failure occurred while MemoryPressureModule attempted to identify the JVM’s garbage collector: 

java.lang.IllegalStateException: Unable to find tenured collector 

Investigation 

I traced the failure to Bazel’s garbage collector detection logic and found that the OpenJ9 JVM on z/OS exposed collector names such as: 

tenured-SOA 

tenured-LOA 

which were not recognized by the existing Bazel detection logic. 

Outcome 

I documented the root cause and shared the findings with the team. The subsequent implementation of the garbage collector fix was continued by Harsh(my fellow mentee).. 

Repository Change 

  • Investigation/documentation only 
  • Final fix continued by another mentee 

Important: This was not a build/compilation failure. The Bazel binary had already been compiled successfully; the failure occurred when the binary was starting and initializing its runtime. 

  1. Adding z/OS Support to rules_python

Problem 

The build initially failed because rules_python did not recognize z/OS as a supported host platform:

No platform declared for host OS z/os on arch 8561 

Investigation 

I downloaded a local copy of rules_python and configured Bazel to use it through local_path_override(). 

To verify that Bazel was actually using my local repository, I temporarily inserted fail() statements into the repository code and confirmed that the build loaded the modified local copy. 

I then traced the host-platform detection and Python toolchain registration

logic. Fix 

I added preliminary z/OS platform support by: 

  • defining zos as a supported operating system; 
  • mapping the z/OS machine type 8561 to s390x; 
  • adding the z/OS platform constraint; 
  • adding metadata for the z/OS Python SDK; 
  • adding the required minor-version mapping. 

Repository Changes 

  • MODULE.bazel 
  • rules_python/python/versions.bzl 
  • rules_python/python/private/toolchains_repo.bzl 
  • rules_python/python/repositories.bzl 
  • platforms/os/BUILD 

Preparing the z/OS Python SDK 

The available Python SDK was distributed as a .pax archive, while the existing rules_python implementation expected a different archive/layout. 

I reorganized the SDK into the expected structure: 

python/ 

├── bin/ 

│ ├── python3 

│ ├── python3.14 

│ └── pip3 

├── lib/ 

│ └── python3.14/

└── include/ 

└── python3.14/ 

and created: 

python-3.14.4.1-s390x-ibm-zos.tar.gz 

Initially, the local archive was referenced using a hardcoded path. I then changed this to use: export ZOS_PYTHON_TARBALL=/path/to/python.tar.gz 

This avoids embedding a machine-specific path in the repository. 

Current Status 

The local Python SDK has been prepared and the ZOS_PYTHON_TARBALL mechanism has been added. However, the current build investigation has shown that the build has not yet reached _python_repository_impl(), where the Python repository implementation actually consumes the archive. 

The current execution reaches python_register_toolchains() and begins creating the platform-specific repositories.Therefore, the local Python archive mechanism has not yet been exercised in this current build path. 

The next step is to trace what triggers _python_repository_impl() and verify the local archive mechanism once the build reaches that stage. After that, an approach based on an existing PYTHON_HOME can be evaluated. 

  1. Adding z/OS Support to rules_go

Problem 

After the rules_python platform changes, the build progressed further but failed during rules_go configuration: 

ERROR: no such target ‘@@rules_go//go/toolchain:z/os’ 

Investigation 

I configured Bazel to use a local copy of rules_go and traced the failure to the platform constraint generation in: 

rules_go/go/private/platforms.bzl 

z/OS was missing from the platform mapping, causing rules_go to generate an invalid toolchain target. 

Fix

I added: 

(“z/os”, “s390x”) 

to the supported platform list and mapped: 

“z/os”: “@platforms//os:zos” 

I also added the corresponding CGO platform mapping. 

Repository Changes 

  • MODULE.bazel 
  • rules_go/go/private/platforms.bzl 

Local Go SDK Support 

Because the z/OS Go SDK was also distributed as a .pax archive, I added support for using a locally available SDK through: 

export ZOS_GO_SDK=/path/to/go-sdk.pax.Z 

The repository rule extracts the SDK using the z/OS pax utility. 

Outcome 

The previous rules_go platform error was resolved and the build progressed to C/C++ toolchain auto-configuration. 

  1. Shell Script Execution on z/OS

Problem 

During C/C++ toolchain configuration, Bazel attempted to execute: 

generate_system_module_map.sh 

directly and received: 

EDC5130I Exec format error 

Investigation 

The failure was traced to: 

bazel_tools/tools/cpp/unix_cc_configure.bzl 

where the script was executed directly instead of explicitly invoking a shell.

Fix 

I changed the execution to locate Bash through Bazel’s repository context: 

bash = repository_ctx.which(“bash”) 

return execute(repository_ctx, [bash, script_path] + dirs) 

Repository Change 

  • bazel_tools/tools/cpp/unix_cc_configure.bzl 

Outcome 

The script was executed successfully through the available zopen Bash installation, allowing the build to proceed. 

  1. z/OS Operating System Detection

Problem 

The build later failed during Bazel’s Java runtime initialization: 

java.lang.IllegalStateException: Unidentified operating system 

Investigation 

I traced the failure to: 

src/main/java/com/google/devtools/build/lib/util/OS.java 

Bazel’s operating-system enumeration did not contain z/OS, so the host was classified as UNKNOWN. 

Fix 

I added: 

ZOS(“zos”, “z/OS”) 

and included ZOS in the POSIX-compatible operating systems. 

Repository Change 

  • src/main/java/com/google/devtools/build/lib/util/OS.java 

Outcome

Bazel successfully recognized z/OS and continued through its existing POSIX command-execution path. 

  1. JNI Loading

Problem 

After fixing OS detection, Bazel encountered: 

java.lang.UnsatisfiedLinkError 

SystemSuspensionModule.registerJNI() 

Investigation 

The JNI loader selected native libraries based on the detected operating system. z/OS was not included in the existing Unix JNI path. 

Fix 

I added z/OS to the existing Unix JNI loading path instead of introducing a separate implementation. Repository Change 

  • src/main/java/com/google/devtools/build/lib/jni/JniLoader.java 

Outcome 

The z/OS build reused Bazel’s existing Unix JNI implementation and progressed to the next stage. 

  1. Native C/C++ Compatibility

Problem 

The native build exposed several z/OS-specific compatibility issues. 

For example: 

  • daemonize.c depended on the BSD err.h interface; 
  • declarations for getopt(), optarg, optind, and optopt were not exposed as expected; ● mkdtemp() was unavailable; 
  • some GNU/Linux linker options were unsupported by IBM Open XL Clang; ● generated dependency files did not match the format expected by Bazel. 

Fixes 

I added z/OS-specific compatibility implementations guarded by __MVS__ so that the existing implementations remained unchanged on other platforms.

For daemonize.c, the BSD-style err() and errx() behavior was reproduced using standard C facilities. 

For getopt(), the required declarations were provided explicitly while leaving the actual implementation to the z/OS runtime. 

For mkdtemp(), a z/OS-compatible implementation based on mkstemp(), unlink(), and mkdir() was added. 

Repository Changes 

  • src/main/tools/daemonize.c 
  • src/main/cpp/util/file_posix.cc 

  1. Native C/C++ Toolchain Integration

Problem 

The larger challenge was integrating Bazel’s native toolchain with IBM Open XL Clang. The compiler successfully generated the dependency file, but Bazel failed while parsing it: Error processing daemonize.d: 

File does not end in a newline 

There were also GNU/Linux linker options such as: 

-B 

-Wl,–as-needed 

-Wl,--gc-sections 

that were not supported by the z/OS compiler. 

Investigation 

I traced the native bootstrap flow and confirmed that: 

  1. compilation of daemonize.c itself succeeded; 
  2. the failure occurred while Bazel parsed the generated .d file; 
  3. IBM Open XL Clang did not add the trailing newline expected by Bazel; 4. several GNU/Linux linker options were incompatible with the z/OS toolchain. 

Fix 

Instead of modifying Bazel’s general dependency parser, I introduced z/OS-specific compiler and archive wrappers. 

The compiler wrapper:

  • invokes IBM Open XL Clang; 
  • removes unsupported GNU linker options; 
  • identifies the generated dependency file; 
  • removes z/OS text tagging; 
  • adds the required trailing newline. 

For example: 

chtag -b “$depfile” 

printf '\n' >> "$depfile" 

This normalizes the file before Bazel parses it. 

I also introduced an archive wrapper and updated Bazel’s C/C++ toolchain configuration to select these wrappers specifically on z/OS. 

Repository Changes 

  • tools/cpp/clang_zos_wrapper.sh 
  • tools/cpp/ar_zos_wrapper.sh 
  • tools/cpp/unix_cc_configure.bzl 
  • tools/cpp/BUILD.tools 
  • tools/cpp/BUILD.tpl 
  • src/main/cpp/util/file_posix.cc 
  • src/main/cpp/util/md5.cc 
  • compile.sh 
  • MODULE.bazel 

Outcome 

The wrapper-based approach allowed the existing Bazel toolchain logic to remain unchanged for Linux while handling z/OS-specific compiler behavior separately. 

Beyond the Code

Working on the Bazel port has been a challenging but rewarding experience. Each compatibility issue pushed me to look beyond the error itself, understand how the system worked, and think about how a solution could fit into an existing codebase. 

I’m grateful to Igor Todorovski and Haritha D for their guidance throughout the journey, and to the Linux Foundation, Open Mainframe Project, and zopen community for giving me the opportunity to contribute to a real-world open-source project. I’m also thankful to Harsh Sagar for the collaboration and the many things we learned along the way. 

The port is still a work in progress, but this journey has given me a much deeper understanding of platform compatibility, debugging, and contributing to open source.

Part 2: How It Started

One of the first things that made the experience different was the people involved. Being part of meetings with my mentors, Haritha D and Igor Tordovski, and working alongside my fellow mentee, Rohini Sankari Byra, gave me my first real exposure to how a larger open-source project operates. The meetings were not just about giving status updates. We discussed what had been tried, what the logs were telling us, what we thought was happening, and what should be investigated next. Even when I did not fully understand something at first, being part of those discussions slowly gave me a better idea of how engineers approach problems that do not have straightforward answers. That was probably the first part of the mentorship that I found genuinely exciting: I was no longer just building something, I was participating in a project. 

Getting Used to z/OS

I was working on a remotely connected IBM Z system running z/OS UNIX System Services, accessing it through SSH from my laptop. Even before getting deep into Bazel, I had to understand how this environment behaved. The tools were familiar because I was working through a Unix-like interface, but the underlying system had its own rules, particularly around file encoding, character tags, and the interaction between ASCII, UTF-8, and EBCDIC. 

The port also meant that I had to stop thinking of “the build” as one single operation. Bazel first has to bootstrap a minimal version of itself and then use that to build the actual Bazel binary. Every time we fixed one problem, the build would move further and reveal another one. 

One Error After Another 

The first major problem I worked through was related to the Java runtime. Bazel’s memory-pressure code expected garbage collector names associated with JVMs such as HotSpot. IBM Semeru uses OpenJ9, and the names exposed by the runtime were different. Instead of the names Bazel expected, OpenJ9 reported pools such as tenured-SOA and tenured-LOA. 

Once the error was traced through Bazel’s memory-management code, the fix turned out to be quite small: the OpenJ9-specific collector names had to be recognised by Bazel. That was my first real experience of fixing a platform compatibility problem from inside a large existing codebase rather than simply changing my own application code. And, more importantly, fixing it did not mean the project was done. It simply meant that Bazel could now get further. 

The next phase involved build-time and runtime problems around Java compilation, AutoService, META-INF/services, archive extraction, JNI loading, and the handling of character encodings on z/OS. This was probably the most confusing part of the work because several of the symptoms looked unrelated. Some logs contained corrupted or unreadable characters, some Java components appeared unable to find resources, and at one point the build reported that it could not find the freshly built Bazel binary. 

A lot of the work during this phase was investigative rather than immediately writing a fix. We examined generated JARs, checked service registration files, looked at the bootstrap scripts, inspected temporary build directories, and compared the behaviour of different environment configurations. One particularly important issue involved JNI. Bazel’s JniLoader did not initially handle z/OS in the way we needed. I added and instrumented the z/OS path to understand exactly what the runtime was doing. That eventually gave us a much clearer result: the loader was being reached correctly, but libunix_jni.so was not present in the bootstrap JAR. That shifted the question from “Why isn’t the code reaching JNI?” to the much more useful question of whether z/OS should actually be using that native path during bootstrap at all. 

The encoding problem was another lesson in how difficult platform-specific debugging can become. z/OS has a bimodal character environment, and variables such as _BPXK_AUTOCVT, file tags, and Java encoding options can affect how data is interpreted. The strangest failure was the corrupted Bazel output path. The error looked almost absurd because the path itself contained unreadable characters, and naturally, the first suspicion was that the build had produced something invalid. 

The important breakthrough came from testing bazel info bazel-bin directly. The path was already being corrupted before the shell script checked for the binary. We then compared the same environment and command with and without IBM_JAVA_OPTIONS. With that variable present, the path could come back as corrupted bytes; when it was unset, the path was normal and the build progressed further.

Working With the Community

The regular meetings with Haritha, Igor, and Rohini were not simply status updates, but rather a place to explain what I had tried, what I had learned from the logs, what I thought was happening, and what I wanted to investigate next. At several points, a discussion during a meeting changed the direction of the investigation completely. 

Working with Rohini also gave me a different perspective on open-source collaboration. We were both working on the same larger problem, but not necessarily on the exact same section of it at the same time. That meant sharing findings, comparing approaches, and making sure that a useful discovery did not remain isolated on one person’s machine. 

Before this mentorship, “open-source community” was mostly an abstract idea to me. During the project, it became repositories, maintainers, build systems, review discussions, mentors, contributors, forks, branches, and a lot of debugging sessions. 

What I’m Taking Away From This Experience

I learned how much of real engineering is investigation: reading logs, comparing two environments, checking what a build script actually does instead of assuming what it does, looking inside generated artifacts, tracing an exception back through several layers of code, making a small change, testing it, and being willing to accept that the change solved only one problem. 

Coming into this mentorship, I had worked on projects where I could define the requirements, write the code, and decide when I was finished. This project has been different. Bazel already exists. z/OS already exists. The zopen ecosystem already exists. I am entering a system that is much larger than my own work and learning how to contribute to it. 

That is probably what makes this experience meaningful to me. I started this mentorship hoping to learn something new, and I’m leaving it with a much better understanding of engineering, collaboration, and what it means to build something together.

 

Stay tuned for more mentee blogs as our Summer Mentorship Program continues!

 

Share