The Mainframe Demystified

From Porting to RAG: Building a Vector Search Engine for z/OS

Mon, 20 Apr 2026 00:00:00 +0000

In a previous blog post, we proved that running a 7B parameter LLM on z/OS was possible. It was a milestone, but performance made it more of a curiosity than a real thing. The real question was no longer whether it could run, but whether it could solve a problem worth solving. On z/OS, that doesn’t always mean generating text. Often, it means retrieving the right context at the right moment like helping system admins triage the thousands of messages streaming across the console and surface the ones that actually matter.

That makes Retrieval-Augmented Generation (RAG) a great fit for z/OS. It works within the platform’s performance limits and respects air-gapped environments. By indexing data locally using efficient embedding models, we can achieve fast semantic search results, turning a slow “curiosity” into a practical, real-time RAG tool.

This blog introduces z-vector-search, a native and open source z/OS engine that allows you to index and query your own data locally. It is also available as a library, so you can embed the same retrieval pipeline directly into your own applications. No cloud dependencies, no data leaving the LPAR, and no more manual flipping through IBM manuals. It’s about building RAG directly where the data lives. It’s also worth noting that much of the code for the project, including a substantial part of the work needed to get llama.cpp embedding support running cleanly on z/OS, was written with the help of IBM Bob, IBM’s AI Coding Assistant.

The scenario that motivated all of this is simple: a z/OS system programmer staring at a console flooded with messages like ABENDs, RACF violations, dataset allocation errors, trying to figure out which ones matter, what they mean, and whether the system has seen anything like this before. Today that means flipping between IBM message manuals, internal runbooks, and ticket histories. What if you could just ask? And what if the answer came from directly on z/OS, not by shipping log data to a cloud LLM, but right there on the LPAR where the data already lives?

This blog covers how we built z-vector-search, the technical decisions behind it, and how z-console, an operator console enrichment tool, serves as one prototype application built on top of it.

z-vector-search running directly on z/OS

Building a RAG system for z/OS

If you’re interested in using the tools and less about learning, go to the Getting Started section.

What RAG Means on z/OS

For a lot of enterprise systems, the missing piece is not generation, it is retrieval. The useful context already exists, but it is scattered across message manuals, procedures, runbooks, ticket histories, and operational logs. A local retrieval engine turns that fragmented knowledge into something you can actually search with natural language, exact identifiers, or a mix of both.

That is the role of z-vector-search. It provides the retrieval layer for on-prem RAG on z/OS: index data locally, search it locally, and return grounded context locally. You can then use that context directly in a CLI, inside an application, or as input to a later generation step. But if retrieval cannot happen securely on the platform where the data lives, the rest of the RAG pipeline is mostly academic.

The RAG idea actually came from a llama.cpp discussion thread about adding embedding model support. Reading through it, I realized that all the pieces I needed to build a z/OS RAG system were already on the table, I just had to wire them up.

But “wiring it up” was only possible because of the stable foundation provided by the zopen llamacpp port. That port was a true community effort, driven by a dedicated group of volunteers and university students who worked tirelessly to bring modern AI tools to the mainframe. Their contributions to the core infrastructure and math optimizations are what allowed us to reach this point.

What’s an embedding, anyway?

If you’ve never worked with them, embeddings are the trick that makes “semantic search” possible. An embedding model takes a piece of text and turns it into a list of numbers, a vector, that captures its meaning. The clever part is that two pieces of text with similar meanings produce vectors that are mathematically close to each other in space, even if they share no words in common.

That means a search for "dataset allocation failure" can find a document that says "IEC070I", because both phrases live near each other in vector space. No keyword matching, no synonyms list, no manual rules. The model has already learned what things mean.

To do search with embeddings, you embed every document once and store the vectors. At query time, you embed the query the same way and find the documents whose vectors are nearest yours.

The Model

The model I chose was Nomic Embed Text v1.5. Quantized to Q4_K_M, it’s just ~84 MB, small enough to run comfortably on z/OS, and well-regarded for retrieval tasks. It’s an encoder-only model (think BERT-style), which means it’s purpose-built for turning text into vectors rather than generating new text.

What It Took to Get Working

llama.cpp’s embedding support is newer than its text generation support, so a few things needed attention to make it behave on z/OS. A substantial part of this implementation work was done with IBM Bob:

Encoder model code path. Encoder-only models like Nomic take a different route through llama.cpp than decoder models like LLaMa. They produce one vector per input rather than streaming tokens, which means a different API (llama_encode() instead of llama_decode()) and slightly different batch handling.
Pooling. The model produces a vector for every token, but you want a single vector per document. Nomic expects MEAN pooling, averaging the per-token vectors together. Getting this wrong produces embeddings that look fine but retrieve nonsense.
Document and query prefixes. Nomic uses a clever convention where you prepend search_document: to text you’re indexing and search_query: to text you’re searching for. This subtly nudges the model to put documents and queries in slightly different regions of the embedding space, which measurably improves retrieval quality. A simple trick, but it makes a real difference.
The endianness problem, again! Just like with the original llama.cpp port, endianness came back to haunt me. Embedding vectors are arrays of 32-bit floats, and a database built on little-endian platforms needs every float byte-swapped before z/OS (big-endian) can read them. I added automatic endianness detection and a --convert-endian flag to enable a high-performance hybrid workflow: you can seed your knowledge base on a fast Linux or macOS box (where indexing thousands of documents takes seconds) and then ship the .db file over to z/OS for production use. This gives you the best of both worlds: massive throughput for the initial data ingestion and secure, local semantic search where it matters most.

After working through these, I had embeddings producing sensible vectors on z/OS, and that was enough to start building something real.

Building the Search Engine

With embeddings now working on z/OS, the next step was obvious: build a persistent vector store so you could index documents once and query them repeatedly.

Storage: SQLite + sqlite-vec

I chose SQLite as the backend, extended with sqlite-vec for vector similarity search. The combination made it extremely simple: no database server to manage, no network dependencies, just a single .db file.

The schema stores each text chunk alongside its embedding and metadata.

Chunking

Large documents can’t be embedded as a single unit: encoder models have a token limit, and long texts lose detail when compressed into one vector. So documents are split into overlapping chunks, 256 tokens each, with 64 tokens of overlap between adjacent chunks. The overlap ensures context at chunk boundaries isn’t lost, and each chunk is independently embedded and stored.

The Tools

To make this process seamless, I created z-setup, a simple initialization tool. It handles the heavy lifting: unpacking the pre-built messages database, performing the necessary big-endian conversion for z/OS, and downloading the optimized Nomic embedding model.

The project is composed of a suite of command-line tools:

Tool	Purpose
`z-index`	Index documents into the persistent vector store
`z-query`	Search the store with natural language queries
`z-vector-search`	One-shot mode: index and query without persistence

A typical workflow:

# Index your documents
z-index *.txt

# Search with natural language
z-query "how do I recover from an IEC070I error"

The query returns the most semantically relevant chunks, ranked by similarity. If your runbook says “dataset allocation failure” and you search for “IEC070I error,” it still finds the right answer.

All tools support --json output for scripting, so you can pipe results into jq:

z-query --json "dataset allocation failure" | jq '.results[0].snippet'

Hybrid Search

Pure semantic search is powerful, but sometimes you know exactly what you’re looking for. If an operator sees ICH408I and wants to look it up, they don’t need semantic similarity, they need an exact match.

So z-query automatically classifies each query:

ICH408I → keyword search (exact message ID via SQL LIKE)
DFH* → keyword search (wildcard)
MSGID:IEF JOB:PAYROLL → keyword search (structured prefix)
why is my CICS transaction failing → semantic search (natural language)
ICH408I unauthorized access → hybrid (both, merged)

When both modes run, results are merged using Reciprocal Rank Fusion (RRF), a technique for combining ranked lists without needing to normalize scores across different methods. The formula is simple:

score = Σ 1/(k + rank)    where k = 60

The IBM z/OS Messages Knowledge Base

A semantic search engine is only as good as its data. To make the tool immediately useful for z/OS operators, I built a pre-packaged knowledge base of 24,565 IBM z/OS messages, covering MVS and system abend/wait codes. Each entry includes the message ID, explanation, system action, and operator response.

The knowledge base ships as a ready-to-use SQLite database, so z-query can answer questions about IBM messages out of the box:

z-query "what does abend S0C4 mean"

This returns the relevant system code documentation explaining that S0C4 is a protection exception, typically caused by a program accessing storage it doesn’t own.

z-console: One Application on Top of the Engine

The IBM messages knowledge base makes z-query useful immediately, but it is only one way to use the engine. To show what a full application built on top of z-vector-search can look like, I also built z-console.

Built on top of z-vector-search, z-console brings the same retrieval pipeline directly to the operator console. It is the answer to the question from the intro: what if a z/OS operator could just ask about a console message?

It comes pre-packaged with the z-vector-search suite. It builds directly on the core z-vector-search engine, using it as a library to perform real-time semantic lookups.

The z/OS operator console is the nerve center of a mainframe system. Messages stream in constantly: job completions, security events, storage allocations, errors, abends. Experienced operators know what to look for, but the volume is overwhelming, and critical messages can be buried in the noise.

z-console reads your console messages and enriches each one with relevant context from both IBM documentation and your system’s own operational history, all by running z-vector-search under the hood.

How It Works

Read, pulls messages from the z/OS SYSLOG via pcon (an IBM ZOAU utility that reads the system log)
Filter, picks out high-value messages: abends (IEF*), data errors (IEC*), RACF violations (ICH*), CICS (DFH*), DB2 (DSN*), MQ (CSQ*), and anything with action/error severity
Look up, for each interesting message, runs a two-phase search:
- Keyword against the IBM messages knowledge base, what does this message ID mean?
- Semantic against your operational history, have we seen something like this before?
Display, presents everything with color-coded severity and ranked context

Input Modes

z-console has three ways to feed it messages:

# Single message, the simplest starting point
z-console "ICH408I USER(BATCH1) GROUP(PROD) LOGON/JOB INITIATION - ACCESS REVOKED"

# Live console via pcon
z-console --pcon -l                    # last hour
z-console --pcon -t 30                 # last 30 minutes
z-console --since 2026-04-06T10:00     # since a specific timestamp

# Pipe from stdin
cat syslog.txt | z-console

Example Output

Running z-console --pcon -l on a system with an access violation might produce:

Parsed 847 messages, 23 interesting, 14 unique IDs to look up.

━━━ ICH408I (severity: E) ━━━
  ICH408I USER(BATCH1) GROUP(PROD) NAME(BATCH JOB)
    LOGON/JOB INITIATION - ACCESS REVOKED

  IBM Documentation (keyword match):
     ICH408I - A RACF-defined user has been revoked. The user's access
     authority has been removed, typically because consecutive incorrect
     password attempts exceeded the SETROPTS PASSWORD limit.
     System Action: The logon or job is rejected.
     Operator Response: Contact the security administrator to reinstate
     access via ALTUSER userid RESUME.
     (distance: 0.12)

  Operational History (semantic match):
     [2026-04-03 14:22] ICH408I,ICH409I, BATCH1 revoked on SYS1,
     resolved by security team reset. Related: RACF password policy
     change ticket INC-4421.
     (distance: 0.31)

In a single glance, the operator knows what the message means and that it’s happened before, with a pointer to how it was resolved last time. That’s the whole pitch for RAG on the console.

Building Operational History

To power the “have we seen this before?” lookups, there’s a companion tool called z-ingest-console. It runs as a background daemon via z-console-daemon.sh (every 5 minutes by default) and continuously indexes console messages into the vector store:

# Start the daemon in the background
nohup ./z-console-daemon.sh &

Messages are grouped into 5-minute time windows and stored with structured metadata, message IDs, highest severity, jobname, system name, timestamps. The longer it runs, the more historical context z-console can draw on.

The Full Picture: Enabling RAG Directly on z/OS

By bringing together embeddings, a persistent vector store, and a hybrid search engine, z-vector-search provides the retrieval core for a complete Retrieval-Augmented Generation (RAG) workflow that works directly on z/OS.

For the air-gapped environments common in finance and healthcare, it means you can build assistants and retrieval-driven tools that understand your specific system configuration and historical data without a single byte leaving your secure LPAR.

Here is the general pipeline that z-vector-search implements, whether the caller is z-query, z-console, or your own embedded application:

Documents / Logs / Runbooks / Message KB
        ↓
   Tokenize (llama.cpp)
        ↓
   Chunk (256 tokens, 64 overlap)
        ↓
   Embed (Nomic Embed v1.5)
        ↓
   L2 Normalize
        ↓
   Store (SQLite + sqlite-vec)
        ↓
   Query → Classify → Keyword / Semantic / Hybrid
        ↓
   Reciprocal Rank Fusion
        ↓
   Top-K Results with Context

Everything runs locally on z/OS. No external API calls, no cloud dependencies, no data leaving the LPAR. The generation step, if you want one, can sit on top later. The important part is that the retrieval foundation already runs where the data lives.

The entire stack is pure C++17 with SQLite, sqlite-vec, and llama.cpp. No Python runtime, no Java, no external dependencies beyond what zopen provides.

Getting Started

Prerequisites: you’ll need the zopen package manager set up on your z/OS system. If you haven’t used zopen before, the QuickStart Guide takes about five minutes.

The simplest way to get going is straight from zopen:

zopen install z-vector-search

Then, regardless of how you installed:

# 3. Run setup, downloads the model and unpacks the IBM messages DB
z-setup

# 4. Query the IBM messages knowledge base
z-query "what does abend S0C4 mean"

# 5. Index your own runbooks or operational docs
z-index /path/to/runbooks/*.txt

# 6. Search them semantically
z-query "how do I recover from an IEC070I error"

# 7. Optional: look up a single console message
z-console "ICH408I USER(BATCH1) GROUP(PROD) LOGON/JOB INITIATION - ACCESS REVOKED"

# 8. Optional: read the last hour of live console
z-console --pcon -l

The source code is available on GitHub.

Conclusion

What started as “can we get embeddings working on z/OS?” turned into a practical retrieval layer for RAG on the mainframe. Embeddings gave us semantic understanding. A vector store made it persistent. Hybrid search made it practical for people who think in message IDs, commands, and procedures, not just natural language. z-console tied it into one concrete operator workflow, but the bigger point is that secure, local retrieval on z/OS is now a real building block you can use elsewhere too.

Thank you to Bill O’Farrell, Chad McIntyre, James Tang, Haritha D, and Sachin T for their support and feedback!

From Porting to RAG: Building a Vector Search Engine for z/OS

Mon, 06 Apr 2026 00:00:00 +0000

In a previous blog post, we proved that running a 7B parameter LLM on z/OS was possible. It was a milestone, but a model in isolation is just a curiosity. The real question is: how do we make it useful? For a z/OS system programmer, utility isn’t found in generating poetry; it’s found in navigating the “data deluge” of the operator console.

Today, we’re moving from simple text generation to semantic retrieval. This post introduces z-vector-search, a native z/OS engine that allows you to index your own documentation and logs locally. No cloud dependencies, no data leaving the LPAR, and no more manual flipping through IBM manuals. We’re building Retrieval-Augmented Generation (RAG) directly where the data lives.

The scenario that motivated all of this is simple: a z/OS system programmer staring at a console flooded with messages — ABENDs, RACF violations, dataset allocation errors — trying to figure out which ones matter, what they mean, and whether the system has seen anything like this before. Today that means flipping between IBM message manuals, internal runbooks, and ticket histories. What if you could just ask? And critically, what if the answer came from directly on z/OS, not by shipping log data to a cloud LLM, but right there on the LPAR where the data already lives?

This post covers how we built z-vector-search, the technical decisions behind it, and how z-console — an operator console enrichment tool — serves as a prototype real-world application on top of it. Along the way, we’ll also look at the SIMD vectorization work that made the whole thing fast enough to actually use on z/OS.

Getting Embeddings Working on z/OS

The idea actually came from a llama.cpp discussion thread about adding embedding model support. Reading through it, I realized that all the pieces I needed to build a z/OS RAG system were already on the table, I just had to wire them up.

What’s an embedding, anyway?

To do search with embeddings, you embed every document once and store the vectors. At query time, you embed the query the same way and find the documents whose vectors are nearest yours. That’s the whole game.

The Model

What It Took to Get Working

llama.cpp’s embedding support is newer than its text generation support, so a few things needed attention to make it behave on z/OS:

Encoder model code path. Encoder-only models like Nomic take a different route through llama.cpp than decoder models like LLaMa. They produce one vector per input rather than streaming tokens, which means a different API (llama_encode() instead of llama_decode()) and slightly different batch handling.
Pooling. The model produces a vector for every token, but you want a single vector per document. Nomic expects MEAN pooling, averaging the per-token vectors together. Getting this wrong produces embeddings that look fine but retrieve nonsense.
Document and query prefixes. Nomic uses a clever convention where you prepend search_document: to text you’re indexing and search_query: to text you’re searching for. This subtly nudges the model to put documents and queries in slightly different regions of the embedding space, which measurably improves retrieval quality. A simple trick, but it makes a real difference.
The endianness problem, again! Just like with the original llama.cpp port, endianness came back to haunt me. Embedding vectors are arrays of 32-bit floats, and a database built on x86 (little-endian) needs every float byte-swapped before z/OS (big-endian) can read them. I added automatic endianness detection and a --convert-endian flag to enable a high-performance hybrid workflow: you can seed your knowledge base on a fast Linux or macOS box (where indexing thousands of documents takes seconds) and then ship the .db file over to z/OS for production use. This gives you the best of both worlds: massive throughput for the initial data ingestion and secure, local semantic search where it matters most.

After working through these, I had embeddings producing sensible vectors on z/OS, and that was enough to start building something real.

Building the Search Engine

With embeddings now working on z/OS, the next step was obvious: build a persistent vector store so you could index documents once and query them repeatedly.

Storage: SQLite + sqlite-vec

I chose SQLite as the backend, extended with sqlite-vec for vector similarity search. The combination is simple and elegant: no database server to manage, no network dependencies, just a single .db file.

The schema stores each text chunk alongside its embedding and metadata.

Chunking

The Tools

The project is composed of a suite of command-line tools:

Tool	Purpose
`z-index`	Index documents into the persistent vector store
`z-query`	Search the store with natural language queries
`z-vector-search`	One-shot mode: index and query without persistence

A typical workflow:

# Index your runbooks
z-index --store ~/my-store.db /path/to/runbooks/*.txt

# Search with natural language
z-query --store ~/my-store.db "how do I recover from an IEC070I error"

The query returns the most semantically relevant chunks, ranked by similarity. No keyword matching needed, if your runbook says “dataset allocation failure” and you search for “IEC070I error,” it still finds the right answer.

All tools support --json output for scripting, so you can pipe results into jq:

z-query --json "dataset allocation failure" | jq '.results[0].snippet'

Hybrid Search

So z-query automatically classifies each query:

ICH408I → keyword search (exact message ID via SQL LIKE)
DFH* → keyword search (wildcard)
MSGID:IEF JOB:PAYROLL → keyword search (structured prefix)
why is my CICS transaction failing → semantic search (natural language)
ICH408I unauthorized access → hybrid (both, merged)

score = Σ 1/(k + rank)    where k = 60

The IBM z/OS Messages Knowledge Base

The knowledge base ships as a ready-to-use SQLite database, so z-query can answer questions about IBM messages out of the box:

z-query "what does abend S0C4 mean"

This returns the relevant system code documentation explaining that S0C4 is a protection exception, typically caused by a program accessing storage it doesn’t own.

z-console: RAG for the Operator Console

z-console is a prototype implementation of a real-world scenario built on top of z-vector-search. It’s the answer to the question from the intro: what if a z/OS operator could just ask about a console message?

The z/OS operator console is the nerve center of a mainframe system. Messages stream in constantly, job completions, security events, storage allocations, errors, abends. Experienced operators know what to look for, but the volume is overwhelming, and critical messages can be buried in noise.

How It Works

Read, pulls messages from the z/OS SYSLOG via pcon (an IBM ZOAU utility that reads the system log)
Filter, picks out high-value messages: abends (IEF*), data errors (IEC*), RACF violations (ICH*), CICS (DFH*), DB2 (DSN*), MQ (CSQ*), and anything with action/error severity
Look up, for each interesting message, runs a two-phase search:
- Keyword against the IBM messages knowledge base, what does this message ID mean?
- Semantic against your operational history, have we seen something like this before?
Display, presents everything with color-coded severity and ranked context

Input Modes

z-console has three ways to feed it messages:

# Single message, the simplest starting point
z-console "ICH408I USER(BATCH1) GROUP(PROD) LOGON/JOB INITIATION - ACCESS REVOKED"

# Live console via pcon
z-console --pcon -l                    # last hour
z-console --pcon -t 30                 # last 30 minutes
z-console --since 2026-04-06T10:00     # since a specific timestamp

# Pipe from stdin
cat syslog.txt | z-console

Example Output

Running z-console --pcon -l on a system with an access violation might produce:

Parsed 847 messages, 23 interesting, 14 unique IDs to look up.

━━━ ICH408I (severity: E) ━━━
  ICH408I USER(BATCH1) GROUP(PROD) NAME(BATCH JOB)
    LOGON/JOB INITIATION - ACCESS REVOKED

  IBM Documentation (keyword match):
     ICH408I - A RACF-defined user has been revoked. The user's access
     authority has been removed, typically because consecutive incorrect
     password attempts exceeded the SETROPTS PASSWORD limit.
     System Action: The logon or job is rejected.
     Operator Response: Contact the security administrator to reinstate
     access via ALTUSER userid RESUME.
     (distance: 0.12)

  Operational History (semantic match):
     [2026-04-03 14:22] ICH408I,ICH409I, BATCH1 revoked on SYS1,
     resolved by security team reset. Related: RACF password policy
     change ticket INC-4421.
     (distance: 0.31)

In a single glance, the operator knows what the message means and that it’s happened before, with a pointer to how it was resolved last time. That’s the whole pitch for RAG on the console.

Summary Mode

Sometimes you don’t need full RAG enrichment, just a quick health check. --summary groups messages by severity and category without loading the embedding model at all:

z-console --summary --pcon -l

=== Console Summary (last hour) ===
Total messages: 847 | Interesting: 23

  CRITICAL/ERROR (3):
    ICH408I  ×2  USER(BATCH1) LOGON/JOB INITIATION - ACCESS REVOKED
    IEC030I  ×1  I/O ERROR, DATASET SYS1.LINKLIB

  WARNING (5):
    IEA404W  ×3  REAL STORAGE SHORTAGE
    CSV028W  ×2  MODULE NOT FOUND IN LINKLIST

  INFORMATIONAL (15):
    DFH1501I ×8  CICS TRANSACTION COMPLETED
    DSN9022I ×7  DB2 COMMAND COMPLETED

Fast enough to run frequently, and gives operators an at-a-glance view of system health.

Building Operational History

# Start the daemon in the background
nohup ./z-console-daemon.sh &

Measuring Performance

How fast is all of this? Rather than hardcoding numbers into this post, where they’d go stale the moment you run on different hardware, both z-query and z-console support a --metrics flag that outputs timing data as JSON on stderr.

For a query:

z-query --metrics "what does abend S0C4 mean" 2>metrics.json

{"mode":"semantic","model_load_ms":2341.5,"embed_ms":287.3,
 "search_ms":42.1,"total_ms":2812.4,"results":5,"store_chunks":34102}

For z-console, the metrics break down timing across all enriched messages, with per-message averages:

z-console --metrics --pcon -l 2>metrics.json

{"total_parsed":847,"interesting":23,"skipped":824,"unique_ids":14,
 "cache_hits":3,"enriched":11,"model_load_ms":2341.5,
 "total_enrich_ms":4892.1,"total_embed_ms":3156.7,"total_search_ms":1204.8,
 "avg_enrich_ms":444.7,"avg_embed_ms":286.9,"avg_search_ms":109.5}

The key insight from the metrics: model load is a one-time cost of a few seconds, and after that each message enrichment takes well under half a second. Keyword-only modes (--summary, pure msgid lookups) skip the model entirely and return in milliseconds. The --metrics output lets you measure exactly what matters on your LPAR, under your workload.

The Full Picture: Enabling RAG Directly on z/OS

By bringing together embeddings, a persistent vector store, and a hybrid search engine, we’ve enabled a complete Retrieval-Augmented Generation (RAG) system that works directly on z/OS.

For the air-gapped environments common in finance and healthcare, this isn’t just a nice-to-have—it’s a hard requirement. It means you can build intelligent assistants that understand your specific system configuration and historical data without a single byte leaving your secure LPAR.

Here’s the full pipeline that runs every time z-console enriches a message:

Console Messages / Documents
        ↓
   Tokenize (llama.cpp)
        ↓
   Chunk (256 tokens, 64 overlap)
        ↓
   Embed (Nomic Embed v1.5)
        ↓
   L2 Normalize
        ↓
   Store (SQLite + sqlite-vec)
        ↓
   Query → Classify → Keyword / Semantic / Hybrid
        ↓
   Reciprocal Rank Fusion
        ↓
   Top-K Results with Context

Everything runs locally on z/OS. No external API calls, no cloud dependencies, no data leaving the LPAR. For the air-gapped environments common in finance and healthcare, that’s not a nice-to-have, it’s a hard requirement.

The entire stack is pure C++17 with SQLite, sqlite-vec, and llama.cpp. No Python runtime, no Java, no external dependencies beyond what zopen provides.

Getting Started

Prerequisites: you’ll need the zopen package manager set up on your z/OS system. If you haven’t used zopen before, the QuickStart Guide takes about five minutes.

The simplest way to get going is straight from zopen:

zopen install z-vector-search

Then, regardless of how you installed:

# 3. Run setup, downloads the model and unpacks the IBM messages DB
z-setup

# 4. Query the IBM messages knowledge base
z-query "what does abend S0C4 mean"

# 5. Look up a single console message
z-console "ICH408I USER(BATCH1) GROUP(PROD) LOGON/JOB INITIATION - ACCESS REVOKED"

# 6. Or read the last hour of live console
z-console --pcon -l

The source code is available on GitHub.

Conclusion

What started as “can we get embeddings working on z/OS?” turned into a full RAG-powered operational assistant. Each step revealed the next problem worth solving. Embeddings gave us semantic understanding. A vector store made it persistent. Hybrid search made it practical for operators who think in message IDs, not natural language. z-console tied it all together. And along the way, a round of SIMD vectorization made the whole thing fast enough to actually use.

The mainframe has always been about running critical workloads reliably. Now it can understand them too.

I’ve Got a Crush on AI: Automating your z/OS Workflows

Tue, 02 Sep 2025 00:00:00 +0000

As a developer in the zopen community, I’m always looking for ways to streamline my workflow on z/OS. With the rise of agentic AI, I started wondering if I could “agentify” my z/OS workflow.

Imagine being able to use natural language to automate tasks with familiar tools. For example, what if I could simply ask: “Show me all of the installed zopen packages that need updating” or “Install everything I need for web development”, and have the AI use its knowledge and the zopen package manager to carry out the task?

Thanks to Large Language Models (LLMs) and the Model Context Protocol (MCP), this is now possible directly on z/OS!

Agentic AI running directly on z/OS!

To make this work, we combine several key technologies:

A custom zopen MCP Server that translates AI requests into real zopen commands.
Ollama or LLama.cpp, running an open-source language model directly on your workstation or on z/OS.
Crush, an open-source terminal-native AI agent that ties everything together.

What is MCP

Before we start, let’s explain MCP. The Model Context Protocol (MCP) is an open standard designed to let AI models safely and reliably interact with external tools (like git, grep and in our case zopen) and data. The way I think of it is as a universal adapter. It defines a standard language that any AI agent can use to “talk” to any tool that also speaks that language.

The design of MCP is heavily inspired by the success of the Language Server Protocol (LSP), which I covered in another blog. Just as LSP created a standard for code editors to communicate with language analysis tools, MCP creates a standard for AI agents to communicate with any tool or data source. This open-standard approach encourages a rich ecosystem where any tool provider can make their service AI-ready, and any AI agent, like Crush, can consume it.

Prerequisites

Before we begin, you’ll need the following:

Go 1.23 or later. For z/OS, you can get Go from the IBM Open Enterprise SDK for Go product page.
An environment with zopen installed (either locally or on a remote z/OS system). Use these instructions if you don’t have zopen installed.

Step 1: Choose Your Language Model

The core of our agent is the Large Language Model. You can leverage any LLM, but given that the zopen community is all about open-source I am going to leverage an open-source model. Whether you run it remotely or on z/OS is up to you.

Option A (Recommended): Run the Model remotely (in my case my Workstation) with Ollama

For the best performance and responsiveness, running the LLM on a workstation (as opposed to directly on z/OS) is the ideal choice as long as you have a GPU or Mac M series. Choosing the right model is important and will impact how well the automation workflow behaves.

For simplicity, I am going to install the LLM on my Mac using Ollama.

If you use Homebrew, you can install it via the command line.

brew install ollama

I’m going to use Qwen3, because it’s a fairly lightweight model (8b parameter model) and it does well at most coding tasks. If you have enough VRAM, then I would suggest Qwen-3-Coder as it is tuned for agentic coding.

In your terminal, run the following command to get the qwen3:8b model:

# On your workstation
ollama serve &
ollama run qwen3:8b

Now leave the Ollama application running in the background on your workstation.

Option B: Run the Model Directly on z/OS with llama.cpp

For a fully self-contained solution, you can now run a model server directly on z/OS. Follow my previous blog on how to set that up. One reason for running LLaMa.cpp locally on z/OS is security. Data on z/OS machines is typically sensitive, and as such, many clients choose to air-gap their systems. (Air-gapping isolates a computer or network from external connections). For sensitive industries like finance and healthcare, local AI models are critical for security by limiting exposure to external threats.

Step 2: The zopen MCP Server - Our Custom Tool Bridge

This is the component we built ourselves. The zopen-mcp-server is a Go application that acts as a translator. It listens for MCP requests from our AI agent (Crush) and converts them into zopen commands. We use the official go mcp sdk to create the MCP server. Its core components are:

The mcp.NewServer Function: Creates the main server instance.
The mcp.AddTool Function: This is the key. It turns a Go function into a capability that the AI can discover and use. The tool’s description is crucial, as it tells the AI what the tool does.
The main Function: This handles startup and configuration, allowing the server to be run as a standalone executable.

The end result is our zopen mcp server, available at https://github.com/IgorTodorovskiIBM/zopen-mcp-server.

To install the server on z/OS, you can use go install:

# On z/OS
go install github.com/IgorTodorovskiIBM/zopen-mcp-server@v1.0.0

Available Tools in the zopen MCP server

The zopen-mcp-server now supports the following commands:

zopen_list: Lists information about zopen community packages.
zopen_query: Lists local or remote info about zopen community packages.
zopen_install: Installs one or more zopen community packages.
zopen_remove: Removes installed zopen community packages.
zopen_upgrade: Upgrades existing zopen community packages.
zopen_info: Displays detailed information about a package.
zopen_version: Displays the installed zopen version.
zopen_init: Initializes the zopen environment.
zopen_clean: Removes unused resources.
zopen_alt: Switches between different versions of a package.

You can also run it remotely if you choose to run your Crush AI agent on your workstation.

Step 3: Crush, Your Terminal Agent - Now on z/OS!

Crush is an AI agent that runs directly in your z/OS terminal.

According to the Crush repository, it offers a rich set of features:

Features

Multi-Model – Choose from a wide range of LLMs or add your own.
Built-in Tools – Crush comes with support for a variety of tools out-of-the-box to interact with your system, including bash, download, edit, grep, ls, rg, and view.
Flexible – Switch LLMs mid-session while preserving context.
Session-Based – Maintain multiple work sessions and contexts per project.
LSP-Enhanced – Leverages LSPs for additional context, just like your editor.
Extensible – Add capabilities via MCPs (http, stdio, and sse). We use the stdio transport for increased security.

And now, thanks to the zopen community, Crush has been ported to z/OS. You can install it using zopen:

# On z/OS
zopen install crush

Before we initialize crush, we need to configure it to use our local LLM.

Configuring Crush: The `crush.json` File

The crush.json file tells Crush how to connect all the pieces. The configuration will differ slightly depending on your chosen deployment model.

Configuration for Workstation Model + z/OS Tools

In this setup, Crush runs on z/OS, talks to your zopen server on z/OS, but gets its knowledge from the Ollama server running back on your workstation.

First, create the file in the standard location:

mkdir  ~/.local/share/crush/
vim  ~/.local/share/crush/crush.json

Copy and adjust these contents appropriately:

{
  "$schema": "https://charm.land/crush.json",
  "providers": {
    "ollama": {
      "name": "Ollama Workstation",
      "base_url": "http://<your_ip>:11434/v1/",
      "type": "openai",
      "models": [{"name": "Qwen3 8B", "id": "qwen3:8b"}]
    }
  },
  "mcp": {
    "zopen": {
      "type": "stdio",
      "command": "/path/on/zos/to/zopen-mcp-server"
    }
  },
}

The Result: An AI-Powered z/OS Workflow!

With your crush.json file in place, simply run crush in your z/OS terminal. Crush will start, connect to your chosen LLM server, and launch your zopen-mcp-server as a subprocess.

From here, you can start giving it tasks to automate!

Can you clone the git repo https://github.com/git/git (with a depth of 1) and discover the build dependencies needed to compile it? Check if zopen already has these dependencies available.

As you can see, Crush already supports many tools, including git and view, which allow it to clone repositories and inspect their contents. It also leverages our zopen-mcp-server to check package availability on z/OS.

In this example, Crush cloned the Git repository, inspected the INSTALL file, and produced a dependency analysis:

Core dependencies found in zopen: zlib, bash, Perl, libcurl, gettext, python.
Optional dependencies missing or partially available: expat, openssl, tcl/tk, xmlto.
Documentation tools not explicitly packaged in zopen: asciidoc, makeinfo, docbook2X, dblatex, docbook-xsl.

Crush was able to summarize this automatically, saving the manual effort of digging through build scripts.

Pretty awesome!

Feel free to experiment. You can ask it anything, from understanding your codebase to making changes to it!

Next Steps: An AI Assistant for Porting Apps

While package management is a great first step, the real power of agentic AI lies in automating complex, knowledge-intensive tasks. My next goal is to tackle one of the most time-consuming parts of my workflow: porting new open-source applications to z/OS.

Agentic AI can also assist with debugging issues during the porting process—and much more. Exciting times ahead!

Thank you to Mike Fulton, Bill O’Farrell and Andrew Sica for reviewing this post and providing valuable feedback!

Faster builds on z/OS with ccache

Fri, 11 Jul 2025 00:00:00 +0000

As a developer in the zopen community (a z/OS Open Source Community), I’ve lost what feels like countless hours waiting for builds to finish. The build-time bottleneck is especially painful in an automated CI/CD pipeline, where I’m waiting on a critical fix to be built, verified, and deployed. I personally don’t enjoy waiting, and neither do our users. This drove me to find a better way, and my search led me to ccache, a tool that caches compiler output to speed up rebuilds. The caching approach intrigued me, so I decided to port ccache to z/OS and integrate it into the zopen community to see if it could help.

In this blog, we’ll cover the basics of ccache, how to get it running on z/OS, how to use it effectively, and most importantly, how it can cut your build times by up to 95%!

What is `ccache` and Why Should I Care?

ccache is a compiler cache. It speeds up recompilation by caching the results of previous compilations. When rebuilding software, ccache provides these cached outputs instead of re-running the compiler. This results in a much faster build turnaround-time.

How `ccache` Works

The beauty of ccache is its simplicity. It acts as a ‘wrapper’ that intercepts calls to your real compiler (in my case, the IBM Open XL C/C++ clang compiler). It then quickly determines whether it has a valid, cached result (usually a compiled object) from a previous identical compilation.

The Process

Analysis and Hashing: When your build process calls ccache, it analyzes the command and runs the C/C++ preprocessor on your source file. This is a key step because ccache hashes the preprocessed output, meaning it’s fully aware of header file changes.
“Cache Hit”: If ccache finds a previously stored object file that matches this exact hash, it copies the cached result to the correct location. This is the fast path.
“Cache Miss”: If no match is found, ccache calls the real compiler to do the work. Upon success, it saves the new object file to its on-disk cache, ready for the next time.

For more in-depth details, visit the official ccache website: https://ccache.dev

Using `ccache` on z/OS

1. Installing `ccache`

Begin by installing or updating ccache on your system using the zopen package manager:

zopen install ccache 

Leveraging Ccache

There are several ways to use ccache. You can use it for a single compilation, integrate it into your own Makefile, or use the seamless support within the zopen build framework.

Simple Usage: Caching a Single File

For a quick test or a one-off compilation, you can simply prefix your compiler command with ccache.

# Instead of: clang -c myprogram.c -o myprogram.o
# Use:
ccache clang -c myprogram.c -o myprogram.o

The first time you run this, ccache will compile and cache the result. The second time, it will be nearly instant.

Integration into Makefiles: Using `ccache` in a Makefile

For any personal or non-zopen project that you build with make, the most common way to use ccache is to set the CC variable in your Makefile. This tells make to use ccache for all C compilations.

# In your Makefile
CC = ccache clang
CFLAGS = -m64 -O3 -g

my_program: main.o utils.o
	$(CC) $(CFLAGS) -o my_program main.o utils.o

main.o: main.c
	$(CC) $(CFLAGS) -c main.c

utils.o: utils.c
	$(CC) $(CFLAGS) -c utils.c

With this setup, every time you run make, the compilation steps will automatically benefit from caching.

Benchmarking `ccache` with the zopen Build Framework

While the manual methods are great, I wanted to test ccache in a fully integrated scenario for zopen. The zopen project provides a build framework (zopen build) that automates the entire build process, and it now has built-in support for ccache through a simple --ccache flag. I used the GNU bash port (bashport) as my test project.

The Setup

My first step was to clone the bashport repository from the zopen community.

git clone https://github.com/zopencommunity/bashport.git
cd bashport

The Experiment

My experiment was designed to mimic a CI/CD workflow where the workspace is cleaned before each run. I performed three builds:

A baseline build without ccache.
A build with --ccache on an empty cache.
A second build with --ccache on a populated cache.

1. The Baseline Test (Without ccache)

First, I timed a standard, full build using zopen build with verbose flags.

echo "Starting baseline 'zopen build'..."
time zopen build -vv

Baseline Full Build Time:

real 8m43.963s (approximately 524 seconds)
The zopen build framework, which times the core build phase separately from configure and install, reported: VERBOSE: build completed in 237 seconds.

This nearly 9-minute full-build time, and more specifically the 237-second build phase, became my baseline.

2. The ccache Test (Cold Cache)

Next, I removed the build artifacts to simulate a clean workspace and ran the build again, this time adding the --ccache flag. Before this step, it’s important to have the necessary tools.

rm -rf bash-* # remove the bash source/build directory
time zopen build -vv --ccache

Cold ccache Build Time:

real 9m16.050s (approximately 556 seconds)
The zopen build framework, which times the core build phase separately from configure and install, reported: VERBOSE: build completed in 251 seconds.

As the results show, the very first build with ccache was slightly slower than the baseline. This is expected! This is the one-time cost of ccache hashing every file and populating its cache for the first time. The real magic happens next.

3. The ccache Test (Warm Cache):

This is the key test. It simulates all subsequent builds in a CI pipeline or a developer’s clean rebuild after the cache is populated. I again removed the build artifacts and re-ran the exact same command.

rm -rf bash-* # remove the bash source/build directory
echo "Starting 'zopen build --ccache' on a warm cache..."
time zopen build -vv --ccache

Warm ccache Build Time:

real 4m2.766s (approximately 243 seconds)
The zopen build framework reported: VERBOSE: build completed in 12 seconds.

The results were excellent! The total time dropped from 8m43s to 4m2s. That’s a 54% reduction (more than 2x faster) in the total time I had to wait.

The framework’s build timer tells a better story: the core build phase, which was 237 seconds in the baseline, dropped to just 12 seconds—a reduction of 95%! This shows that with a warm cache, the time-consuming compilation step was almost entirely eliminated, leaving only the much faster linking step in the build phase. While the configure and install phases took a similar amount of time in each run, the massive speedup in the build phase drove this incredible overall improvement.

Managing and Monitoring Your Cache

Using ccache is great, but knowing how to manage it is also critical.

Checking Statistics: To see how effective ccache is, use the -s flag. It provides a human-readable summary of cache hits, misses, cache size, and more.
```
1
  ccache -s
```
Clearing the Cache: To start completely fresh, you can clear the entire cache.
```
1
  ccache -C
```
Setting Cache Size: You can control the maximum size of the cache. For example, to set a 5 GB limit:
```
1
  ccache -M 5G
```

Future Considerations

While ccache delivers clear performance gains for C/C++ with modern compilers like Clang, traditional z/OS compilers—such as those for COBOL or PL/I—aren’t supported.

This raises important questions:

Should we explore caching strategies for legacy languages on z/OS?
What impact could this have on build speed, especially in integrated CI/CD pipelines?

Optimizing Git Performance on z/OS

Wed, 19 Mar 2025 00:00:00 +0000

Optimizing Git Performance

As an open source developer for the zopen community, I rely heavily on Git for version control. But, as repositories grow in size and history, my git operations often slow down to the point where they can impact my daily workflow, slow down CI/CD pipelines, and ultimately affect my productivity as a developer.

I wanted to make sure I was doing everything I can to optimize my Git worfklow, So recently, I investigated various Git performance optimization techniques, and I decided to put them to the test.

My goal was to understand the practical impact of these strategies on z/OS by using git from IBM Open Enterprise Foundation for z/OS.

We’ll start by exploring the strategies that provided the most significant performance improvements first.

1. Data Reduction Strategies

The idea behind “data reduction” strategies is simple: if we can reduce the amount of data Git has to process, we can improve Git’s performance.

Shallow Clones: Reducing the History (`--depth=1`)

Shallow clones are a great way to speed up clones when you don’t need the full commit history. By cloning with a limited depth, you eliminate the commit history and thereby reduce the amount of data transferred, making cloning much faster. This is extremely useful in CI/CD pipelines or for quickly getting the latest codebase where the commit history is not relevant.

Benefits:

Reduced Cloning Time: Save time, especially for repositories with a long history.
Lower Bandwidth: Less data to transfer
Reduced Disk Footprint: Shallow clones take up significantly less disk space.

Experiment on Perl 5: As an experiment, I cloned the Perl 5 repository (https://github.com/Perl/perl5). This repository contains the source code for the latest Perl language implementation. First I cloned perl5 with a full clone and then with a shallow clone (–depth=1 which downloads the latest commit only). I ran these tests on a z/OS 3.1 z15 machine.

# Baseline: Full Clone
time git clone https://github.com/Perl/perl5 perl5-full-clone

# Shallow Clone with depth 1
time git clone --depth=1 https://github.com/Perl/perl5 perl5-shallow-clone

The results were as follows:

# Baseline: Full Clone
real     0m46.611s
user     1m16.479s
sys      0m25.493s

# Shallow Clone with depth 1
real     0m4.447s
user     0m2.328s
sys      0m0.776s

Shallow cloning the Perl 5 repository showed a dramatic improvement! The clone time was reduced from over 46 seconds to just over 4 seconds, a reduction of over 90%! The repository size was also significantly reduced as only the latest commit was downloaded, instead of the entire history. For scenarios where commit history isn’t immediately needed, shallow clones make a clear difference, especially in CI/CD pipelines.

Inspecting the Size of a Shallow Clone To gain further insight into the volume of data being processed during your shallow clones, you can execute the following command within your Git repository: git count-objects -vH.

After performing a clone, navigate into the newly created repository directory (e.g., cd perl5-shallow-clone) and run the command:

git count-objects -vH 

This command will provide a human-readable summary of the number and size of the various Git objects in your repository’s object database.

Now let’s compare the output from a full clone vs a shallow clone:

# Full clone output:
count: 0
size: 0 bytes
in-pack: 530133
packs: 1
size-pack: 319.96 MiB
prune-packable: 0
garbage: 0
size-garbage: 0 bytes

# Shallow clone output:
count: 0
size: 0 bytes
in-pack: 7703
packs: 1
size-pack: 22.09 MiB
prune-packable: 0
garbage: 0
size-garbage: 0 bytes

As shown, a shallow clone drastically reduces the data downloaded by fetching only recent commits instead of the entire history. This is reflected in the in-pack count, which drops from 530,133 objects in a full clone to just 7,703 objects in a shallow clone. Since fewer commits, trees, and blobs are included, the total size-pack also shrinks significantly, from 319.96 MiB to 22.09 MiB.

This reduction happens because Git stores repository data in packfiles, which group and compress objects for efficiency. In a full clone, all historical commits and their associated objects are included, leading to a large size-pack. In contrast, a shallow clone fetches only recent objects, leaving out older history, resulting in fewer packed objects and a much smaller repository footprint.

Metric	Full Clone	Shallow Clone	Reduction
Clone Time	46s	4s	91%
Size	319MB	22MB	93%
Objects	530,133	7,703	98.5%

Avoiding Downloading Large Binaries (`--filter=blob:none`)

Repositories can sometimes accumulate large binary files over time. If these binaries (e.g pax files, object files, executables) are not needed for every operation or during development, you can instruct Git to filter these out during the cloning process. This can help reduce bandwidth, disk space and clone time.

The --filter=blob:none option prevents Git from downloading any large file blobs.

I cloned the Perl 5 repository again, this time using the --filter=blob:none option to exclude binary blobs.

time git clone --filter=blob:none  git@github.com:Perl/perl5.git

The results were as follows:

real     0m15.333s
user     0m9.057s
sys      0m3.019s

Using --filter=blob:none reduced the clone time to approximately 15 seconds, significantly faster than the baseline full clone of 46 seconds. This is a great way to reduce clone time, while maintaining history for repositories where binaries are not essential for maintenance.

Cloning Only a Single Branch and Avoiding Tags (`--no-tags` and `--single-branch`)

By default, git clone clones all branches and tags. If you are only interested in a single branch, and tags are not immediately necessary, cloning only a single branch and excluding tags can reduce the amount of data transferred and speed up the clone operation. The --no-tags option skips downloading tags, and --single-branch clones only the specified branch.

I cloned the Perl 5 repository again, this time using the --no-tags and --single-branch options.

time git clone  --no-tags --single-branch git@github.com:Perl/perl5.git

The results were:

real     0m34.867s
user     1m2.789s
sys      0m20.930s

Cloning a single branch without tags took approximately 35 seconds, which is faster than the full clone baseline, but not as fast as --filter=blob:none. However, when combined with --filter=blob:none, the total clone time was reduced to 13s.

Sparse Checkouts: Clone What You Need (`git sparse-checkout`)

Sparse checkouts allow you to download only a specific subset of directories from a repository into your working directory. This is great for monorepos or large projects where you only work within certain modules or components.

Benefits:

Reduced Working Directory Size: Save disk space by only checking out necessary files.
Faster git status and Checkout Operations: Git has less data to track and manage in your working directory, improving command speed.

Using a sparse checkout on the Perl 5 repository, I wanted to exclude the cpan/ directory, which contains CPAN module-related code and is approximately 26mb in size. I used the following script:

# sparse-checkout.sh

# Initialize an empty repository
rm -rf perl5-sparse-empty-test
mkdir perl5-sparse-empty-test
cd perl5-sparse-empty-test
git init

# Add the Perl 5 repository as a remote
git remote add origin [https://github.com/Perl/perl5](https://github.com/Perl/perl5)

# Ignore the cpan directory
git config core.sparseCheckoutCone false
git sparse-checkout init --no-cone
git sparse-checkout set "/*"
git sparse-checkout add '!cpan' # ignore cpan dir

git fetch origin blead

git checkout blead

Running this script with sparse checkout commands yielded a total time of:

real     0m42.361s
user     1m7.524s
sys      0m22.508s

When the sparse-checkout commands are commented out, essentially performing a full clone, the time taken was:

real     0m43.325s
user     1m8.081s
sys      0m22.694s

While the time difference isn’t dramatically significant in this specific test, sparse checkout successfully saved approximately 26mb of data by excluding the cpan directory.

Working-Tree-Encoding: Consider the Conversion Costs (`.gitattributes`)

If your project uses working-tree-encoding or zos-working-tree-encoding, be aware of the potential performance overhead. Encoding conversions handled by iconv can add CPU processing time, especially when global wildcards are used.

To test this, I created a fork of the Perl 5 repository with a .gitattributes file that would apply zos-working-tree-encoding=ibm-1047 to all text files:

https://github.com/IgorTodorovskiIBM/perl5

# .gitattributes
* text zos-working-tree-encoding=ibm-1047

Cloning this repository:

time git clone git@github.com:IgorTodorovskiIBM/perl5.git

The encoding conversion, when applied globally with the * wildcard, added about 3 seconds to the total clone time compared to cloning the original repository without the encoding attribute. While 3 seconds might seem small, it’s another factor to consider, especially in scenarios involving large repositories or frequent Git operations. For projects using working-tree-encoding, consider the scope of encoding conversions to reduce potential performance impacts.

However, an important counterpoint to skipping conversion at the Git level means each compile step must handle encoding conversion instead. For languages like C, which support auto-conversion of source files, this can introduce overhead every time a file is compiled. This ongoing cost might outweigh the one-time Git conversion delay.

Profiling and Diagnostics (`GIT_TRACE=1` and `GIT_PERFORMANCE=1`)

Git’s diagnostic environment variables such as GIT_TRACE and GIT_PERFORMANCE are great ways for identifying performance bottlenecks within Git operations. By enabling these variables, Git generates detailed logs that can reveal which specific steps are consuming the most time and resources.

To enable tracing and performance logging, set the corresponding environment variables before running your Git commands.

export GIT_TRACE=1
export GIT_PERFORMANCE=1

After setting these variables, execute the Git command you want to analyze. Git will output detailed trace information to stderr.

For example, to trace a clone operation:

export GIT_TRACE_PERFORMANCE=1 GIT_TRACE=1
12:33:53.398786 exec-cmd.c:267          trace: resolved executable dir: /home/itodoro/zopen/usr/local/zopen/git/git-heads.v2.48.1.20250131_080517.zos/bin
12:33:53.399026 git.c:476               trace: built-in: git clone https://github.com/Perl/perl5
Cloning into 'perl5'...
12:33:53.406725 run-command.c:667       trace: run_command: git remote-https origin https://github.com/Perl/perl5
12:33:53.406752 run-command.c:759       trace: start_command: /home/itodoro/zopen/usr/local/zopen/git/git-heads.v2.48.1.20250131_080517.zos/libexec/git-core/git remote-https origin https://github.com/Perl/perl5
12:33:53.433854 exec-cmd.c:267          trace: resolved executable dir: /home/itodoro/zopen/usr/local/zopen/git/git-heads.v2.48.1.20250131_080517.zos/libexec/git-core
12:33:53.434307 git.c:769               trace: exec: git-remote-https origin https://github.com/Perl/perl5
12:33:53.434315 run-command.c:667       trace: run_command: git-remote-https origin https://github.com/Perl/perl5
12:33:53.434338 run-command.c:759       trace: start_command: /home/itodoro/zopen/usr/local/zopen/git/git-heads.v2.48.1.20250131_080517.zos/libexec/git-core/git-remote-https origin https://github.com/Perl/perl5
12:33:53.468060 exec-cmd.c:267          trace: resolved executable dir: /home/itodoro/zopen/usr/local/zopen/git/git-heads.v2.48.1.20250131_080517.zos/libexec/git-core
12:33:53.938639 run-command.c:667       trace: run_command: git index-pack --stdin -v --fix-thin '--keep=fetch-pack 33562439 on ZOSCAN2B' --check-self-contained-and-connected
12:33:53.938676 run-command.c:759       trace: start_command: /home/itodoro/zopen/usr/local/zopen/git/git-heads.v2.48.1.20250131_080517.zos/libexec/git-core/git index-pack --stdin -v --fix-thin '--keep=fetch-pack 33562439 on ZOSCAN2B' --check-self-contained-and-connected
12:33:53.965929 exec-cmd.c:267          trace: resolved executable dir: /home/itodoro/zopen/usr/local/zopen/git/git-heads.v2.48.1.20250131_080517.zos/libexec/git-core
12:33:53.966507 git.c:476               trace: built-in: git index-pack --stdin -v --fix-thin '--keep=fetch-pack 33562439 on ZOSCAN2B' --check-self-contained-and-connected
remote: Enumerating objects: 613690, done.
remote: Counting objects: 100% (16/16), done.
remote: Compressing objects: 100% (16/16), done.
remote: Total 613690 (delta 5), reused 5 (delta 0), pack-reused 613674 (from 1)
12:34:02.103539 trace.c:417             performance: 8.669476706 s: git command: /home/itodoro/zopen/usr/local/zopen/git/git-heads.v2.48.1.20250131_080517.zos/libexec/git-core/git remote-https origin https://github.com/Perl/perl5
Receiving objects: 100% (613690/613690), 340.21 MiB | 42.31 MiB/s, done.
Resolving deltas: 100% (464433/464433), done.
12:34:31.897026 trace.c:417             performance: 37.930887480 s: git command: /home/itodoro/zopen/usr/local/zopen/git/git-heads.v2.48.1.20250131_080517.zos/libexec/git-core/git index-pack --stdin -v --fix-thin '--keep=fetch-pack 33562439 on ZOSCAN2B' --check-self-contained-and-connected
12:34:31.898222 run-command.c:667       trace: run_command: git rev-list --objects --stdin --not --all --quiet --alternate-refs '--progress=Checking connectivity'
12:34:31.898281 run-command.c:759       trace: start_command: /home/itodoro/zopen/usr/local/zopen/git/git-heads.v2.48.1.20250131_080517.zos/libexec/git-core/git rev-list --objects --stdin --not --all --quiet --alternate-refs '--progress=Checking connectivity'
12:34:31.932503 exec-cmd.c:267          trace: resolved executable dir: /home/itodoro/zopen/usr/local/zopen/git/git-heads.v2.48.1.20250131_080517.zos/libexec/git-core
12:34:31.933121 git.c:476               trace: built-in: git rev-list --objects --stdin --not --all --quiet --alternate-refs '--progress=Checking connectivity'
12:34:31.933623 trace.c:417             performance: 0.000879117 s: git command: /home/itodoro/zopen/usr/local/zopen/git/git-heads.v2.48.1.20250131_080517.zos/libexec/git-core/git rev-list --objects --stdin --not --all --quiet --alternate-refs '--progress=Checking connectivity'
12:34:31.999553 unpack-trees.c:2010     performance: 0.022947339 s:   traverse_trees
Updating files: 100% (6895/6895), done.
12:34:33.150223 unpack-trees.c:512      performance: 1.150647878 s:   check_updates
12:34:33.157202 cache-tree.c:492        performance: 0.006916968 s:   cache_tree_update
12:34:33.157222 unpack-trees.c:2107     performance: 1.180711913 s:  unpack_trees
12:34:33.160126 read-cache.c:3115       performance: 0.002893479 s:  write index, changed mask = 2a
12:34:33.160601 trace.c:417             performance: 39.761582911 s: git command: git clone https://github.com/Perl/perl5

real    0m39.793s
user    1m12.404s
sys     0m24.135s

You can now examine the log output to identify performance bottlenecks and areas for potential optimization.

Conclusion

Whenever possible, use shallow clones with a depth of 1 as it provides a significant performance benefit.

For those working within z/OS, a more detailed guide outlining specific performance recommendations can be found in the Git on z/OS guide: https://github.com/zopencommunity/gitport?tab=readme-ov-file#git-performance-considerations.

Git also provides various performance-related configuration settings, such as adjusting the number of parallel threads and other optimizations: Git Configuration Options

If you have any other tips or strategies for improving Git performance, I’d love to hear them!

A special thanks to Mike Fulton for his contributions to this article.

Additional Resources

Performance and Memory Insights: Tracing with Clang and Perfetto

Thu, 19 Sep 2024 00:00:00 +0000

Enabling Tracing with Clang on z/OS

As a member of the zopen community, I get the opportunty to port a wide range of open-source tools to z/OS. It’s always exciting when you get a tool running smoothly on z/OS, especially when it makes your workflow easier. However, even in cases where everything seems to be working fine, performance issues or memory leaks can still creep in. While IPCS is great for deep-dive debugging, I’ve never been able to become proficient at it. I wanted something that was easier to use and more visual. In this post, I’ll walk you through how I addressed this using Clang’s instrumentation features and Perfetto’s trace visualization and show how you can easily apply them in your projects.

I start first with a dive into how I generated the trace data. If you’re not interested in the tracing implementation, you can skip over to Visualizing the Trace data using Perfetto for how to leverage it.

For the impatient, here’s a quick look at the end result:

Compiler Tracing with Clang

Before I started on this project, I did some research and discovered Clang’s -finstrument-functions feature, available in IBM Open XL C/C++. This compiler option automatically inserts instrumentation code at each function entry and exit point, making it possible to collect detailed tracing metrics. With the instrumented code, I could easily capture function names, timestamps, and other data, all of which are essential for profiling performance and understanding call flow. Another route I could have taken would have been to port the LLVM XRay instrumentation library, but that seemed like a lot of effort for an experimental project.

Integrating with zoslib

Clang’s -finstrument-functions feature only requires two functions to be implemented: __cyg_profile_func_enter and __cyg_profile_func_exit. By leveraging these, I was able to capture the function names using the Program Prologue Area (PPA1), along with the timestamps, process IDs (PIDs), and thread IDs (TIDs).

My initial implementation was simple and fairly effective:

__attribute__((no_instrument_function))
static void write_perf_data(const char* name, const char* phase) {
  pthread_mutex_lock(&__prof_mutex);
  fprintf(___file, "\"name\": \"%s\", \"ph\": \"%s\", \"pid\": %d, \"tid\": \"%d\", \"ts\": %lu\n", name, phase, getpid(), gettid(), get_timens());
  pthread_mutex_unlock(&__prof_mutex);
}

__attribute__((no_instrument_function))
void __cyg_profile_func_enter(void* this_fn, void* call_site) {
  __stack_info si;
  void *cur_dsa = dsa();

  __iterate_stack_and_get(cur_dsa, &si);
  write_perf_data(si.entry_name, "B");
}

__attribute__((no_instrument_function))
void __cyg_profile_func_exit(void* this_fn, void* call_site) {
  __stack_info si;
  void *cur_dsa = dsa();

  __iterate_stack_and_get(cur_dsa, &si);
  write_perf_data(si.entry_name, "E");
}

After successfully testing the prototype, I integrated it into zoslib, a foundational open-source library for building C/C++ projects on z/OS.

Zoslib is linked into 99% of the C/C++ projects within the zopen community. If you want to learn more about zoslib, check out my other blog here.

Standardizing Trace Data Format

While the prototype was definitely a step up over what I had before, it was still difficult and time-consuming to analyze the data. I needed to generate the trace data into a more consumable format. One such standardized format is the Chrome Tracing JSON format. This format offers several advantages:

Readability on z/OS via the jq command since the trace data will be in json.
Compatibility with Chrome’s built-in tracing feature and Perfetto open source.

Trace Data Structure

The Chrome Tracing format stores each function call as a JSON object with these attributes:

Function name
Timestamp
PID
TID
Phase (“B” for function entry, “E” for function exit)

To make the traces more manageable, especially for long-running applications, I implemented automatic gzip compression. After the trace is generated, a check ensures that gzip is available, and if so, the trace file is compressed. This keeps the file sizes under control, but if gzip fails for some reason, the trace file remains uncompressed, and the application continues as normal.

Next Step - Tracking Memory Allocations

Another pain point for me on z/OS was tracing memory allocations. IPCS is powerful, but again I needed something more accessible for memory footprint analysis. So, I figured why not extend this tracing to handle memory management too?

So that’s what I did. I implemented custom overrides for malloc and free to trace and track heap memory spikes and allocation patterns.

Implementation

The current implementation is available in zoslib here

And here’s an example of the generated json for git on z/OS:

{ "traceEvents": [
  {"cat": "PERF", "name": "zoslib_env_hook", "ph": "B", "pid": 83893699, "tid": 1, "ts": 582917421},
  {"cat": "malloc", "name": "Memory", "ph": "C", "pid": 83893699, "tid": 1, "ts": 582922816, "args": {"delta_memory": 1024, "total_memory": 1024}},
  {"cat": "free", "name": "Memory", "ph": "C", "pid": 83893699, "tid": 1, "ts": 582928703, "args": {"delta_memory": -1024, "total_memory": 0}},
  {"cat": "malloc", "name": "Memory", "ph": "C", "pid": 83893699, "tid": 1, "ts": 582929642, "args": {"delta_memory": 1024, "total_memory": 1024}},
  {"cat": "free", "name": "Memory", "ph": "C", "pid": 83893699, "tid": 1, "ts": 582934379, "args": {"delta_memory": -1024, "total_memory": 0}},
  {"cat": "malloc", "name": "Memory", "ph": "C", "pid": 83893699, "tid": 1, "ts": 582935145, "args": {"delta_memory": 1024, "total_memory": 1024}},
  {"cat": "free", "name": "Memory", "ph": "C", "pid": 83893699, "tid": 1, "ts": 582940290, "args": {"delta_memory": -1024, "total_memory": 0}},
  {"cat": "malloc", "name": "Memory", "ph": "C", "pid": 83893699, "tid": 1, "ts": 582941020, "args": {"delta_memory": 1024, "total_memory": 1024}},
  {"cat": "free", "name": "Memory", "ph": "C", "pid": 83893699, "tid": 1, "ts": 582942273, "args": {"delta_memory": -1024, "total_memory": 0}},
  {"cat": "malloc", "name": "Memory", "ph": "C", "pid": 83893699, "tid": 1, "ts": 582942975, "args": {"delta_memory": 1024, "total_memory": 1024}},
  {"cat": "free", "name": "Memory", "ph": "C", "pid": 83893699, "tid": 1, "ts": 582949534, "args": {"delta_memory": -1024, "total_memory": 0}},
  {"cat": "PERF", "name": "zoslib_env_hook", "ph": "E", "pid": 83893699, "tid": 1, "ts": 582950330},
  {"cat": "PERF", "name": "main", "ph": "B", "pid": 83893699, "tid": 1, "ts": 582970044},
  ...

Visualizing the trace data using Perfetto

Once I had the mechanism in place to generate trace data into a standardized format, the next step was analysis. This is what led me to Perfetto, an open source trace visualization tool that offers an interactive UI to inspect your program’s execution in detail.

Perfetto is really easy to use. It lets you zoom in on specific timeframes, analyze function call stacks visually, and filter events by category or thread. It even supports SQL queries on your trace data, which opens up a lot of possibilities for custom analysis.

Here’s a look at the performance visualization for git status:

With Perfetto, you can visualize your program’s entire call flow and easily track memory spikes.

How to leverage the tracing in zopen tools

Using zopen build –instrument for Easy Tracing

While this was great, I now wanted a way to make it easy for everyone contributing to zopen to leverage this tracing capability.

So, I incorporated this functionality into the zopen build framework by adding a --instrument option:

zopen build --instrument

With this option, the open source project is built with instrumentation enabled (applies only to C/C++ projects).

Then, when the instrumented application runs, it generates a -.json.gz file that contains the trace data. Note, you can override this location and filename by setting the ZOSLIB_PROF_PATH environment variable. You may want to do this in if space is an issue on your system.

Viewing Trace Data

Now that you have a trace file, the next step is to perform the analysis.

To view the trace data, there are two approaches:

Manual inspection of the json
Using perfetto (preferred)

Manual inspection:

Using zcat and jq: You can decompress and inspect the JSON trace data directly from the command line:

zcat <application>-<timestamp>.json.gz | jq .

Using Perfetto UI

The json file can be loaded into the Perfetto UI for a detailed view of the execution.

Steps to view traces in Perfetto UI:

Visit the Perfetto UI using a browser.
Download the generated json.gz file from z/OS and then click on ‘Open Trace file’ in the Perfetto UI.
Explore the timeline to see detailed function calls, filter by specific processes or threads, and identify bottlenecks.

You can use the W/A/S/D keys to maneuver around.

Using Perfetto, I could now visually identify spikes in memory footprint and identify the culprit functions:

Using the SQL Query engine for custom analysis

Perfetto also offers a Python library, which I haven’t had the chance to explore yet, but it opens up the possibility for custom analysis and even integrating with large language models (LLMs) for deeper insights.

Additionally, Perfetto’s built-in SQL query engine allows you to easily locate long-running tasks and perform advanced data queries. Here’s a simple example to find the longest-running functions:

SELECT ts, dur, name FROM slice ORDER BY dur DESC

This makes it easy to dig into specific performance bottlenecks and get a more granular view of your application’s behavior.

Next Steps

This project started as an experiment but has proven to be valuable (for me at least) in quickly detecting performance and memory issues in zopen tools. Moving forward, I’m excited to explore how we can further improve this approach and track additional metrics that could enhance our analysis and optimization efforts.

If there’s enough interest, I hope to present this in the next zopen guild meeting!

And of course, there’s always more to do - like figuring out why Git is allocating 80KB of memory!

Conclusion

Thank you to Mike Fulton, Haritha D and everyone from the zopen community, and especially to the IBM Open XL C/C++ team for providing an awesome C/C++ compiler for z/OS!

Enhancing Vim on z/OS UNIX with Language Server Protocol (LSP) support

Thu, 18 Apr 2024 00:00:00 +0000

As a long-time Vim user who develops directly on z/OS UNIX (yes, call me old school!), one feature that I’ve always wanted is the support for Language Server Protocols.

If you’re not aware of Language Servers and the Language Server Protocol (LSP), it’s a protocol that standardizes the communication between editors (VS Code, Neovim, Emacs, VIM) and language servers.

Microsoft’s LSP website summarizes it well:

The Language Server Protocol (LSP) defines the protocol used between an editor or IDE and a language server that provides language features like auto complete, go to definition, find all references etc. The goal of the Language Server Index Format (LSIF, pronounced like “else if”) is to support rich code navigation in development tools or a Web UI without needing a local copy of the source code.

Language servers use JSON-RPC to communicate between the development tool and the language server. Using JSON-RPC allows the server to be implemented in any language. The protocol itself defines a set of features that can be supported by the server, such as code completion and go-to-definition.

Vim on z/OS with the Go Language Server

For more information on the LSP architecture, visit Microsoft’s docs. To find out what languages implement language servers, check out this page.

Why LSP support is great for editors

LSP’s enable code editors to support the following features without having to write their own Langauge parsers and analyzers:

Code Completion: Auto-complete suggestions can signifiantly speed up coding because you no longer need to look up the APIs by browsing the manual or system interfaces.
Error Diagnostics: This feature helps you maintain clean code as you are actively writing new code.
Code Navigation: LSPs understand your code and eanbles you to easily navigate to code definitions and call sites.

Ideally, to get the best support for LSPs in a terminal editor, we would want to port Neovim to z/OS. Unfortuantely, the neovim port is still in progress!

So in the meantime, I’m going to describe how we can provide LSP support to Vim on z/OS.

Adding LSP Support to Vim

1. Installing Vim

Begin by installing or updating Vim on your system using the zopen package manager:

zopen install vim

Note: you’ll need the latest Vim release which includes terminal support (required by the plugins we’ll install below).

2. Installing the vim-plug plugin manager

Before we set up the LSP support, we’ll need to install a vim plugin manager. As of this writing, vim-plug appears to be the most popular. Install it with the following commands:

# You will need curl for this operation (you can install it via zopen as above)
curl -fLo ~/.vim/autoload/plug.vim --create-dirs https://raw.githubusercontent.com/junegunn/vim-plug/master/plug.vim

Then, create a ~/.vimrc file (if it doesn’t exist), and add these lines to enable vim-plug:

call plug#begin()
" Add plugins here
call plug#end()

In the next step, we’ll install the LSP and auto-complete plugins through vim-plug.

3. Installing LSP and Autocompletion Plugins

We’ll now use vim-plug to install the following plugins:

vim-lsp/vim-lsp-settings: These plugins simplify LSP configuration in Vim.
asyncomplete.vim/asyncomplete-lsp: These plugins provide the core functionality for autocompletion in Vim.

Update your ~/.vimrc file with the following changes:

call plug#begin()
Plug 'prabirshrestha/vim-lsp'
Plug 'IgorTodorovskiIBM/vim-lsp-settings', { 'branch': 'zos' }

Plug 'prabirshrestha/asyncomplete.vim'
Plug 'prabirshrestha/asyncomplete-lsp.vim'
call plug#end()

Note: We’re using my fork of vim-lsp-settings since some of the LSP settings provided in the official repo do not currently work.

Restart Vim and run :PlugInstall to install plugins.

4. Configuring LSP for Python

Now let’s explore how to set up an LSP for Python.

Note: The Python LSP requires python3 and clang in the PATH.

First, open a Python file with Vim. vim-lsp-settings associates LSPs with the language extension, so make sure the extension is .py.

vim hello.py

Use the following command to install the Python language server (pyls).

:LspInstallServer

After installing, restart vim or reload the file. You should now have auto-complete support via the Python LSP:

5. Configuring LSP for Go

Now let’s explore how to set up the language server for Go.

Note: The Go language server requires the go compiler in the PATH. Also make sure to disable CGO with export CGO_ENABLED=0.

First, open a Go file with Vim. vim-lsp-settings associates LSPs with the language extension, so make sure the extension is .go.

vim main.go

Use the following command to install the Go language server (gopls).

:LspInstallServer

After installing, restart vim or reload the file. You should now have auto-complete support via the Go language server.

Next Steps

Test out more languages!
Neovim! Port LuaJit so that we get all of the Neovim features enabled for z/OS!

Thank you

Thanks for reading and thanks to Mike Fulton, Haritha D, Peter Haumer and the zopen community contributors!

Porting tmux to z/OS

Tue, 12 Mar 2024 00:00:00 +0000

I’m excited to share my journey of porting tmux, a popular terminal multiplexer, to z/OS! This project builds upon my previous experience successfully porting GNU screen and making it available under the z/OS Open Tools umbrella. Following the success of GNU screen, the z/OS community expressed a strong desire for tmux in a recent poll, making it one of the most requested tools for z/OS.

So what is Tmux?

tmux is a terminal multiplexer. It lets you switch easily between several programs in one terminal, detach them (they keep running in the background) and reattach them to a different terminal.

Tmux running on z/OS

Tmux has many advantages over GNU Screen, including mouse integration, supporting terminal splitting into both vertical and horizontal panes, and generally many more configuration options.

Why Tmux on z/OS?

For z/OS users, particularly system administrators and developers, efficient management of multiple shell sessions is critical for efficiency.

Tmux provides the following features:

Multi-pane Workspaces: Organize your terminal by splitting the window into panes, each running an independent program. This is great when you’re building and editing at the same time like me!
Detachable Sessions: Leave your work running on z/OS even after disconnecting. Reattach seamlessly from any terminal later. This means you can finish working, sleep and then resume where you left off the following morning.
Enhanced Workflow: Easily switch between tasks, monitor multiple processes concurrently, and maintain a clutter-free terminal environment. This is one vital for me since I sometimes work on 5+ different tasks during the day.

Why not run tmux on Linux? You can certainly run tmux remotely from a Linux machine to manage your screen sessions, but being able to run it locally on z/OS can reduce latency and is especially good for air-gapped environments.

How do I install it?

Grab it from z/OS Open Tools using the zopen package manager:

zopen install tmux -y

Then run it:

tmux

To learn more about the shortcuts, I recommend you read this page.

Porting Tmux to z/OS

Porting tmux to z/OS presented unique challenges due to the underlying differences between z/OS and traditional Unix-like systems. Let’s explore some key areas and the solutions implemented:

Missing dependencies: Tmux in itself does not have that many library dependencies. It depends on libevent and ncurses. Fortunately ncurses has already been ported for other projects like Vim and Less so I only had to focus on getting libevent ported.
I/O Handling: Standard Unix I/O operations might require special handling on z/OS. We leveraged functionalities provided by ZOSLIB (introduced in the previous blog post) to simplify I/O handling.

Conquering Dependency Challenges:

By leveraging the zopen build framework, I was able to port libevent fairly easily. The port is available at https://github.com/zopencommunity/libeventport. Since libevent is a library, I had to make sure I exposed it as a library when I was adding it as a dependency for Tmux with the following additions to its buildenv file:

  export ZOPEN_EXTRA_CFLAGS="\${ZOPEN_EXTRA_CFLAGS} -I\$PWD/include"
  export ZOPEN_EXTRA_CXXFLAGS="\${ZOPEN_EXTRA_CXXFLAGS} -I\$PWD/include"
  export ZOPEN_EXTRA_LDFLAGS="\${ZOPEN_EXTRA_LDFLAGS} -L\$PWD/lib"
  export ZOPEN_EXTRA_LIBS="\${ZOPEN_EXTRA_LIBS} -levent"

For a detailed look on how I built it, check out the entire buildenv configuration here https://github.com/zopencommunity/libeventport/blob/main/buildenv.

Now I was ready to build Tmux.

Building tmux

tmux’s buildenv configuration was relatively simple. We added the known dependencies (libevent, ncurses, make, and a few others) and chose the Clang compiler for building. To build it, I used the zopen build framework.

zopen build -v

Unfortunately, tmux had a number of issues at build and runtime and I’ll go through a few of them.

Overcoming the `forkpty` hurdle

forkpty is needed by tmux but is absent on z/OS, so I employed an alternative approach utilizing existing C LE apis. This involved creating a pseudo terminal by opening specific devices and manipulating terminal attributes to simulate a terminal environment for tmux sessions. You can find the patch here.

Testing and Debugging the Build

After testing the build of tmux, I was presented with garbled output immediately! To understand the issue, I leveraged tmux’s -v option for logging. After further investigation, I realized that the content written to the screen was actually readable and in ASCII format but it wasn’t being converted back to IBM-1047 for the terminal.

The culprit? writev was not respecting file tags and auto-conversion. This meant the text going to standard output wasn’t being automatically converted to the terminal’s CCSID (IBM-1047).

To resolve this, I modified the libevent code to use write instead of writev. This meant I had to create a patch for libevent here.

This enabled tmux to render text correctly, but I wasn’t done yet. The newline characters didn’t seem to register. This resulted in the prompt line being printed without a newline!

After comparing the stty output from a working terminal to that of the tmux created terminal, I noticed that the flags OPOST and ONLCR were missing. Adding them back in the tmux C code fixed it!

The last major issue was with regards to socketpair. Tmux uses socketpairs instead of pipes for interprocess communication, but socketpairs (as of z/OS 2.4) do not support auto-conversion. Changing the code to use pipes instead of socketpairs fixed this issue!

There’s still a lingering issue when using /bin/sh as the default shell. So for now I set my tmux default shell to bash to overcome it in my $HOME/.tmux.conf tmux config file as follows:

vim $HOME/.tmux.conf
# Now add the following line
set-option -g default-shell $ZOPEN_PKGINSTALL/bash/bash/bin/bash

For a more involved tmux.config file, you can check mine out here.

Next Steps

There’s still a few issues remaining, but for the most part tmux is pretty much functional. Here’s what remains:

Resolve issues when using /bin/sh as the default shell
Migrating the forkpty and writev changes to the common zoslib library

Thank you

Thanks for reading and thanks to Mike Fulton, Haritha D and the z/OS Open Tools contributors!

Exploring ZOSLIB: An open source z/OS library for porting

Mon, 12 Feb 2024 00:00:00 +0000

Ever tried porting your favorite Linux tool to z/OS? As we’ll see, it’s not as simple as grabbing the source and building it on z/OS!

My role at the zopen community involves the task of porting open-source tools to the IBM z/OS operating system. This requires navigating through the various differences between Linux and z/OS, such as:

C runtime differences (GNU C Library (Glibc) vs z/OS C Runtime Library)
File system differences (z/OS UNIX/datasets on z/OS)
Endianness differences
Security differences (RACF/SAF on z/OS)
Codepages differences (EBCDIC/ASCII/UTF8 program support in z/OS).

One significant hurdle that we continue to encounter is the absence of essential C runtime APIs on z/OS. While we can implement workarounds at the application source-code level, it doesn’t seem like a sustainable model. Can we do better?

Moreover, adding to the complexity of porting to z/OS, z/OS UNIX supports file tag metadata that describe a file’s encoding content. File tag metadata doesn’t exist in Linux, but it’s crucial in z/OS because programs can run in various character modes, including EBCDIC/ASCII/UTF8 mode. What makes file tags interesting is that, combined with the z/OS Enhanced ASCII services, z/OS can perform automatic conversion of files to and from the program’s encoding upon read and write. This enables EBCDIC/ASCII/UTF8 programs to interpret file data correctly, irregardless of the program’s character mode. This feature greatly simplifies handling files of various codepages in applications. However, although file tags are great, many z/OS services generate files with no file tag metadata (these are known as “untagged files”). How are applications supposed to handle such “untagged” files?

That’s where ZOSLIB jumps in — it addresses many of these complexities and makes porting to z/OS a little bit simpler!

What is ZOSLIB?

At its core, ZOSLIB is an open source extension of the z/OS LE C Runtime Library, offering the following additional functionalities:

Extended POSIX Support: ZOSLIB includes a subset of POSIX APIs not present in the LE C Runtime Library, addressing critical functionality gaps.
Enhanced File Operations: Improved implementations for C open, pipe, and more which facilitate seamless integration with z/OS auto-conversion facilities.
Data Conversion Utilities: C APIs for EBCDIC to ASCII conversions and vice versa simplify the handling of different character encodings.
Diagnostic Reporting: ZOSLIB provides APIs for enhanced diagnostic reporting, aiding in troubleshooting.

ZOSLIB was originally introduced as a library to aid in porting Node.js and its components to z/OS. It has since been open-sourced and is now used by over 100 open-source projects under the zopen community organization.

The ZOSLIB project is available on GitHub at https://github.com/ibmruntimes/zoslib. It is licensed under the Apache 2.0 license, which imposes few restrictions and facilitates modification and redistribution.

ZOSLIB is a 64-bit ASCII library and is meant to be leveraged by ASCII-based applications and libraries.

Now, let’s explore the gaps that ZOSLIB aims to fill.

Gap 1: How do we handle untagged files?

File tags are very useful metadata for determining the underlying codepage of a file. However, z/OS does not make this metadata field mandatory. z/OS UNIX files can lack a tag, and such files are considered untagged. Complicating matters further is that by default, C LE open() and creat() functions generate untagged files even when the program is running in ASCII mode. Additional code is required to tag newly created files.

So why are untagged files an issue? They’re a problem because the application can not determine the true encoding of untagged files. This means that an EBCDIC-based program expecting text data will interpret the data one way and an ASCII-based program will interpret the data another way. Many applications address this by introducing an option or environment variable to toggle how untagged files are interpreted.

To address this issue, in zoslib, we implemented an override for the C open() function that incorporates an optimized heuristic (toggleable if needed by an environment variable), that determines the true encoding of untagged files. Additionally, since z/OS creates untagged files (and pipes) by default, and there is no toggle to control this behavior, we decided to override C open(), ‘pipe()’, and ‘creat()’ to additionally tag all files/pipes to ASCII ISO8859-1 by default. This override can be adjusted via an environment variable (documented in the zoslib manpages).

When zoslib is added to your project and the macro -DZOSLIB_OVERRIDE_CLIB=1 is set, all references to open(), ‘pipe()’, and ‘creat()’ are mapped to a new zoslib function. In the case of open, it’s mapped to __open_ascii function (defined here).

This enhancement has enabled zoslib applications to seamlessly work with both legacy untagged data and tagged data. For our Git to z/OS port, this entire support reduced the number of changes by close to 50%!

Gap 2: Missing C Runtime APIs

The ZOSLIB project also enables developers to implement missing C runtime functions once and then make them available to all projects that leveraging zoslib. For instance, several open source projects have transitioned to using posix_spawn rather than fork/exec. Rather than implementing posix_spawn for each individual application, we implemented it once in zoslib and leverage it across many projects. Additional functions added to zoslib include mkdtemp, execvpe, strpcpy, strsignal and many more.

Additionally, there are instances where a header file may be missing. For example, on z/OS, sys/param.h does not exist, and the macro definitions that are typically in sys/param.h are present in other headers. However, the assumption by open source projects is that this header exists across all operating systems. Without guarding out the header include via a #ifdef check, the compiler would trigger an error. However, this means making an application specific change. By adding a sys/param.h header to zoslib, we avoid yet another change to the application’s source code.

The good news is that many of these APIs are currently being implemented in the z/OS LE C runtime library, but zoslib enables us to adopt an agile approach to development, allowing us to promptly address these gaps as we encounter them.

Gap 3: Provide System-Level Interfaces for z/OS

Operating systems like Linux also provide system-level functions, such as sysconf(_SC_NPROCESSORS_ONLN);, which retrieves the number of online CPUs for the system. Such system-level functions vary from platform to platform, and offering interfaces to capture equivalent implmentations aids in porting.

For instance, the following functions have been added to zoslib:

__get_num_online_cpus - capture the number of online cpus
__get_os_level - return the operating system level
__get_cpu_model - returns the cpu model

These are just a few examples of what ZOSLIB has to offer and I encourage to read the source and examine the zoslib documentation for more information.

How we’re porting tools with ZOSLIB

Now that we’ve examined what ZOSLIB has to offer, let’s explore how you can seamlessly integrate ZOSLIB into your application or library.

To leverage ZOSLIB, you can either compile it from source as described in the README or you can download a release from the zopen zoslibport repository.

This release provides the libraries (static and dynamic), header files, man pages, as well as environment variables for use in the zopen framework.

We’ll first cover how to integrate zoslib into your code manually, and then describe how we’re automatically integrating zoslib into the 100+ zopen community projects.

Integrating zoslib manually

First, download a stable release of zoslib here. If you have the zopen package manager, zopen, on your system (details here), you can simply run:

zopen install zoslib

Alternatively, you can download the pax.Z asset and extract it onto your system.

Let’s assume you have a simple hello world program that writes a file to disk.

//a.c
#include <fcntl.h>
#include <unistd.h>

int main() {
    const char *filename = "output.txt";
    const char *message = "Hello, World!\n";

    int file = open(filename, O_WRONLY | O_CREAT | O_TRUNC, 0666);

    if (file == -1) {
        perror("Error opening file");
        return 1; 
    }

    ssize_t bytes_written = write(file, message, strlen(message));

    if (bytes_written == -1) {
        perror("Error writing to file");
        close(file);
        return 1; 
    }

    close(file);

    return 0; 
}

The below example uses the clang compiler, available here. If you do not have clang on your z/OS system, you can use xlc or xlclang.

Now let’s compile this C program using clang as follows:

clang -fzos-le-char-mode=ascii -m64 -o a.out a.c
./a.out # run it
ls -lT output.txt # check the tag

The output:

- untagged    T=off -rw-rw-r--   1 ITODORO  CDEV          14 Feb 12 15:09 output.txt

You will notice that the output.txt file is untagged.

Now let’s incorporate zoslib. Set ZOSLIB_HOME to the path to your zoslib installation.

clang -fzos-le-char-mode=ascii -m64 -DZOSLIB_OVERRIDE_CLIB=1 -isystem $ZOSLIB_HOME/include -L $ZOSLIB_HOME/lib -o a.out a.c -lzoslib $ZOSLIB_HOME/lib/celquopt.s.o -lzoslib-supp
./a.out
ls -lT output.txt

The output:

t ISO8859-1   T=on  -rw-rw-r--   1 ITODORO  CDEV          14 Feb 12 14:52 output.txt

As you can see, the output.txt file is now tagged as ISO8859-1. This is because open() is now overriden with __open_ascii() from zoslib as instructed by the macro -DZOSLIB_OVERRIDE_CLIB=1.

Now let’s examine a simpler way of integrating ZOSLIB, and that is by leveraging the zopen framework. This is how zoslib is currently being leveraged in over 100+ zopen projects.

Leveraging the zopen build framework

Examine the following code snippet:

export ZOPEN_BUILD_LINE="STABLE"
export ZOPEN_STABLE_URL="http://www.greenwoodsoftware.com/less/less-${LESS_VERSION}.tar.gz"
export ZOPEN_STABLE_DEPS="make curl gzip tar ncurses zoslib"

The above snippet, taken from lessport describes how simple it is to leverage zoslib using the zopen build framework. Simply adding zoslib as a dependency (in ZOPEN_STABLE_DEPS) will instruct the build to add zoslib as a dependency and pass in the zoslib compiler flags.

Note: The great news is that zoslib is now always added as a dependency. As such, you no longer need to add zoslib as a dependency in ZOPEN_STABLE_DEPS.

These flags automatically instruct the compiler to pick up the necessary header files and to link to the ZOSLIB static archives. If you’re wondering, these flags are passed in via the .appenv file from your ZOSLIB_HOME installation. The file is generated by the zoslibport build configuration.

The zopen build framework sets the CPPFLAGS, CXXFLAGS, CFLAGS, LIBS, LDFLAGS accordingly. Fortunately, almost every projects we deal with rely on these flags, meaning that we can easily pass integrate ZOSLIB as an additional library to any port.

Since late 2022, several z/OS ported projects, including Git, Curl, Bash and many more have been successfully integrated with zoslib. Open Source communities have also adopted our dependency on zoslib, including GNU Make, Bash, Vim and more!

If you’re interested in porting to zopen and would like to learn how, check out our documentation here.

Contributing to ZOSLIB

Contributions to ZOSLIB are encouraged, and the process involves opening pull requests at https://github.com/ibmruntimes/zoslib/pulls.

Future considerations for ZOSLIB

Consider creating routines to expose z/OS services like SAF / CMS APIs
Add more missing POSIX functions!
Support for building on V2R5 and V2R3 targets

Special Thanks

Thank you to Mike Fulton, Gaby Baghdadi, Wayne Zhang, CW Cheung, Sean Perry, Eric Janssen, Haritha D and many others for their contributions to ZOSLIB.

AI on z/OS: LLaMa and LLaMa.cpp

Tue, 22 Aug 2023 00:00:00 +0000

In the rapidly evolving field of AI, Large Language Models (LLM)’s like LLaMa and the open source inference engine, LLaMa.cpp, are quickly becoming instrumental in bridging the gap between cutting-edge AI models and their practical deployment on common architectures. We’ll describe LLaMa’s significance, uncover the benefits of LLaMa.cpp, and examine their relevance to z/OS, as well as explore the challenges of adapting the LLaMa model and the LLaMa.cpp inference engine to the z/OS platform.

LLaMa.cpp's server program running on z/OS

The LLaMa Model

LLaMa, or Large Language Model Meta AI, was introduced in February 2023 by Meta. The LLaMa set of large language models (LLMs) quickly gained attention for its ability to generate human-like text and comprehend vast language data, with very similar accuracy to ChatGPT.

Unlike OpenAI’s ChatGPT/GPT-4 models, which have remained closed, Meta opened up its LLaMa v1 model to the community under a noncommercial license.

Then, in July 2023, Meta unveiled LLaMa 2, introducing models with 7 billion, 13 billion, 33 and 70 billion parameters, and this time made its LLM models available for commercial use as well. Additionally, Llama 2 models were trained with 40% more data, leading to much better results than before.

Meta also released models that were fine-tuned for chat, including Llama-2-7b-chat and Llama-2-13b-chat, which were fine-tuned using reinforcement learning from human feedback.

LLaMa Models and Performance

The LLaMa models achieved great scores across the board, as evidenced by Hugging Face’s LLM leaderboard. The Hugging Face leaderboard measures and tracks the top open LLM models.

The model with the best trade-off, though, is LLaMa’s 13-billion parameter model, which outperformed much larger models like GPT-3 (with 175 billion parameters) in various language tasks, but with the benefit of being able to run it on lower cost systems!

Can we run it on a CPU Based Architecture?

Until now, it has been widely assumed that a high-powered GPU is necessary for performing efficient transformations and inferencing. However, LLaMa.cpp has changed the game by enabling CPU-based architectures to run LLM models at a reasonable speed!

Introducing LLaMa.cpp - A Game Changer in AI

LLaMa.cpp, an open source LLaMa inference engine, is a new groundbreaking C++ inference engine designed to run LLaMa models efficiently.

It achieves this through its use of quantization. Quantization is used to reduce the precision of the model weights. This saves memory and makes calculations much faster. For LLaMa models, such as Llama 2-7B, quantization reduces the memory requirements from 13Gb to 4gb and for the LLaMa-13B model, from 24 GB to 7.8gb.

If you’re interested in learning more about the quantization method used in LLaMa.cpp, the article How is LLaMa.cpp possible? does a great job explaining it.

Additionally, LLaMa.cpp is written in C++ and provides C headers to its interfaces. This means that you can bind it to your favourite languages, such as Node.js, Go, and Python!

Performance Across Devices

LLaMa Benchmark reports reveal significant improvements over the original models, in some cases offering up to 3-4 times speedup from quanitization.

Their github page also reports the following statistics across the various quantization methods:

Model	Measure	F16	Q4_0	Q4_1	Q5_0	Q5_1	Q8_0
7B	perplexity	5.9066	6.1565	6.0912	5.9862	5.9481	5.9070
7B	file size	13.0G	3.5G	3.9G	4.3G	4.7G	6.7G
7B	ms/tok @ 4th	127	55	54	76	83	72
7B	ms/tok @ 8th	122	43	45	52	56	67
7B	bits/weight	16.0	4.5	5.0	5.5	6.0	8.5
13B	perplexity	5.2543	5.3860	5.3608	5.2856	5.2706	5.2548
13B	file size	25.0G	6.8G	7.6G	8.3G	9.1G	13G
13B	ms/tok @ 4th	-	103	105	148	160	131
13B	ms/tok @ 8th	-	73	82	98	105	128
13B	bits/weight	16.0	4.5	5.0	5.5	6.0	8.5

As you can see, going from F16 (16-bit float) to Q4_0 quantization gets you a 1.5 times speedup. Also, perplexity which is a common metric for measuring how effective an LLM is, does not suffer greatly from quantization! Awesome!

The power of LLaMa.cpp has already been evidenced in the field with these examples:

LLaMa Meets z/OS

z/OS, the operating system for IBM Z mainframes, serves as the backbone for critical operations across many industries. Data is everything, and z/OS systems process a lot of data. Merging AI with z/OS opens new doors for business and financial intelligence. IBM is already doing a fantastic job at this by introducing the Python AI Toolkit for IBM z/OS as well as WatsonX, which also plans to make the LLaMa 2 models available on its platform!

LLaMa.cpp, is yet another option and it aligns well with z/OS’s CPU-based architecture and large memory cache.

One other reason for running LLaMa.cpp locally on z/OS is security. Data on z/OS machines is typically sensitive, and as such, many clients choose to air-gap their systems. (Air-gapping isolates a computer or network from external connections). For sensitive industries like finance and healthcare, local AI models are critical for security by limiting exposure to external threats.

I’m convinced! let’s port LLaMa.cpp to z/OS

Let’s now discuss how we can get LLaMa.cpp to work on z/OS.

Porting LLaMa.cpp to z/OS isn’t without its difficulties. Challenges include:

C Runtime Adaptation: Porting C/C++ code involves dealing with a different C runtime on z/OS, known as the Language Environment (LE). Fortunately, in the case of LLaMa.cpp, we were able to leverage the zoslib library, which bridges the gap between the Linux C runtime and the z/OS C runtime. ✅
Missing Tools: LLaMa.cpp has dependencies on common build tools such as GNU Make and CMake. These tools are not natively available on z/OS. However, we were able to leverage the work from the zopen community and use these tools to build LLaMa.cpp with no issues. ✅
Endianness Issues: 99% of the open LLM models, including LLaMa’s models, are in little endian format. IBM Z is a big endian platform. Therefore, we need a mechanism to convert models from little endian to big endian. Either the models need to be converted as a pre-step prior to inferencing, or the inferencing engine needs to be able to auto convert little endian models to big endian. Let’s examine both approaches in more detail.

For a look at the patches applied to get LLaMa.cpp to work, visit llamacppport. (Note, this was hacked in a weekend. :) I list the future consideration at the end of this blog).

Approach 1 - Converting models from little endian to big endian

LLaMa.cpp requires models to be in its own unique GGJT format (soon to be updated to GGUF), and it provides a handy script, convert.py that can convert any LLaMa model to this format. Fortunately, this provided a way for us to additionally write out the content in big endian format after conversion.

For example, the conversion script, which was written in Python, writes out 32-bit integer values as follows:

self.fout.write(struct.pack("i", len(text)))

Adding an “!” swapped the bytes so that it would be written out in big endian order.

self.fout.write(struct.pack("!i", len(text)))

I ran the conversion script on Linux x86, but if we were to run it on IBM Z, we would swap the bytes at read-time, as opposed to write-time.

However, I wasn’t very happy with this approach because it meant that I couldn’t directly download a model from huggin face and have it just work.

Approach 2 - Modifing the LLaMa.cpp code to dynamically convert models to big endian

When the LLaMa.cpp inference engine starts up, it reads the model into memory. It provides routines to read float and integer data. Therefore, after reading integer or float data, we could swap the bytes to match the native endian format, big endian. This required some knowledge of the tensor weight format, but with a little bit of work, we wre able to get it to work!

For a closer look at how this was achieved, view the patches here.

LLaMa running on z/OS!

After successfully applying these changes, we are now able to run LLaMa.cpp on z/OS!

If you’re interested in installing LLaMa.cpp, you can download it using zopen as follows:

zopen install llamacpp

or download the pax.Z release.

In my example below, I used the model “Llama-2-7b-chat”. I downloaded the little endian model that was already converted and quantized from https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGML.

I used zopen’s curl to download it to z/OS as follows:

curl -L -O https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGML/resolve/main/llama-2-7b-chat.ggmlv3.q4_0.bin

Note, to proceed to the next steps, you will need at least 4GB of above the bar memory to proceed. Adjust your MEMLIMIT if necessary.

Now we’re ready to try it out!

To demonstrate a typical financial scenario, I asked it to calculate the total expenses as follows:

I have 1000 employees. Each one makes $80k/year. I also pay $100k/year for my property. What are my yearly expenses?

I issued it as follows:

$PATH_TO_LLAMACPP/main -m llama-2-7b-chat.ggmlv3.q4_0.bin -n 128 "I have 1000 employees. Each one makes \$80k/year. I also pay \$100k/year for my property. What are my yearly expenses?"

The result was:

system_info: n_threads = 12 / 24 | AVX = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 0 | NEON = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 0 | VSX = 0 |
sampling: repeat_last_n = 64, repeat_penalty = 1.100000, presence_penalty = 0.000000, frequency_penalty = 0.000000, top_k = 40, tfs_z = 1.000000, top_p = 0.950000, typical_p = 1.000000, temp = 0.800000, mirostat = 0, mirostat_lr = 0.100000, mirostat_ent = 5.000000
generate: n_ctx = 512, n_batch = 512, n_predict = 128, n_keep = 0


 I have 1000 employees. Each one makes $80k/year. I also pay $100k/year for my property. What are my year expenses?
$100,000 (property) + ($80,000 x 1000 employees) = $8,100,000

So your yearly expenses are $8,100,000. [end of text]

llama_print_timings:        load time = 22215.58 ms
llama_print_timings:      sample time =    98.91 ms /    60 runs   (    1.65 ms per token,   606.60 tokens per second)
llama_print_timings: prompt eval time = 24075.94 ms /    41 tokens (  587.22 ms per token,     1.70 tokens per second)
llama_print_timings:        eval time = 34295.69 ms /    59 runs   (  581.28 ms per token,     1.72 tokens per second)
llama_print_timings:       total time = 58579.91 ms

Terrific! It worked!

Now let’s make it an interactive session! And let’s see if it can write code for us!

Let’s ask it to write a C program that prints out Hello World.

./bin/main -m ../llama-2-7b-chat.ggmlv3.q4_0.bin -n 125 -i -p "[INST] <<SYS>> You are a helpful, respectful and honest assistant. <</SYS>> Write a C program that prints Hello World. [/INST]"

The -i option make it interactive.

This is the output that we got:

INST] <<SYS>> You are a helpful, respectful and honest assistant. <</SYS>> Write a Java program that prints Hello World. [/INST]  Of course! Here is a simple Java program that prints "Hello World":

public class HelloWorld {
  public static void main(String[] args) {
    System.out.println("Hello World!");
  }
}

To compile and run this program, you will need a Java compiler (such as `javac`) and a Java runtime environment (such as `java`). You can also use an Integrated Development Environment (IDE) such as Eclipse or NetBeans to write, compile, and run your program.

Here is how you could compile and runCan you continue

To compile the program, you can use the following command in the terminal or command prompt:

javac HelloWorld.java

This will create a `HelloWorld.class` file in the current directory, which you can then run using the following command:

java HelloWorld

This will print "Hello World!" to the console.

I hope this helps! Let me know if you have any questions or need further assistance.

It worked! On a heavily used LPAR with 24 CPUs, I was able to get a token rate of 500ms/token, which was not too bad.

Now, let’s ask it to turn our unstructured data (text) into structured data (JSON).

[INST] <<SYS>> You are a helpful, respectful and honest assistant. <</SYS>> Can you create a json file representing the following information: IBM revenue for the twelve months ending June 30, 2023 was 5.25B, a 1.42% increase year-over-year. IBM annual revenue for 2022 was 5.3B, a 5.54% increase from 2021. IBM annual revenue for 2021 was 7.35B, a 3.93% increase from 2020. IBM annual revenue for 2020 was 5.179B, a 4.39% decline from 2019.

We get the following output:

{
  "Revenue": {
     "Annual Revenue": [
      {
        "Year": 2023,
        "Revenue": 5.25B
      },
      {
        "Year": 2022,
        "Revenue": 5.3B
      },
...

Impressive!

LLaMa.cpp also features an example chatbot that can be run via it’s server executable.

Run it as follows:

server -m llama-2-7b-chat.ggmlv3.q4_0.bin

Here’s an example dialogue with z/OS running the inference engine!

LLaMa.cpp's server program running on z/OS

Future Work

In order to perform the dynamic convertion from little endian to big endian, we had to disable the mmap. Enabling mmap support would allow for the model to load much faster. Revisit Approach 1 so that we can optimize start-up time.
Performance improvement considerations: LLaMa.cpp supports the BLAS library for math speedup. It also supports SIMD (AVX2), but this needs to be enabled on s390x.
Add an option to convert models to big endian, rather than always dynamically converting to big endian.
Upstream the effort to the LLaMa.cpp community

Special Thanks

Thank you to Haritha D and Mike Fulton for contributing to this effort!

The Mainframe Demystified

From Porting to RAG: Building a Vector Search Engine for z/OS

Building a RAG system for z/OS

What RAG Means on z/OS

What’s an embedding, anyway?

The Model

What It Took to Get Working

Building the Search Engine

Storage: SQLite + sqlite-vec

Chunking

The Tools

Hybrid Search

The IBM z/OS Messages Knowledge Base

z-console: One Application on Top of the Engine

How It Works

Input Modes

Example Output

Building Operational History

The Full Picture: Enabling RAG Directly on z/OS

Getting Started

Conclusion

From Porting to RAG: Building a Vector Search Engine for z/OS

Getting Embeddings Working on z/OS

What’s an embedding, anyway?

The Model

What It Took to Get Working

Building the Search Engine

Storage: SQLite + sqlite-vec

Chunking

The Tools

Hybrid Search

The IBM z/OS Messages Knowledge Base

z-console: RAG for the Operator Console

How It Works

Input Modes

Example Output

Summary Mode

Building Operational History

Measuring Performance

The Full Picture: Enabling RAG Directly on z/OS

Getting Started

Conclusion

I’ve Got a Crush on AI: Automating your z/OS Workflows

What is MCP

Prerequisites

Step 1: Choose Your Language Model

Step 2: The zopen MCP Server - Our Custom Tool Bridge

Available Tools in the zopen MCP server

Step 3: Crush, Your Terminal Agent - Now on z/OS!

Configuring Crush: The crush.json File

Configuration for Workstation Model + z/OS Tools

The Result: An AI-Powered z/OS Workflow!

Next Steps: An AI Assistant for Porting Apps

Faster builds on z/OS with ccache

What is ccache and Why Should I Care?

How ccache Works

Using ccache on z/OS

1. Installing ccache

Leveraging Ccache

Simple Usage: Caching a Single File

Integration into Makefiles: Using ccache in a Makefile

Benchmarking ccache with the zopen Build Framework

Managing and Monitoring Your Cache

Future Considerations

Optimizing Git Performance on z/OS

Optimizing Git Performance

1. Data Reduction Strategies

Shallow Clones: Reducing the History (--depth=1)

Avoiding Downloading Large Binaries (--filter=blob:none)

Cloning Only a Single Branch and Avoiding Tags (--no-tags and --single-branch)

Sparse Checkouts: Clone What You Need (git sparse-checkout)

Working-Tree-Encoding: Consider the Conversion Costs (.gitattributes)

Profiling and Diagnostics (GIT_TRACE=1 and GIT_PERFORMANCE=1)

Conclusion

Additional Resources

Performance and Memory Insights: Tracing with Clang and Perfetto

Enabling Tracing with Clang on z/OS

Compiler Tracing with Clang

Integrating with zoslib

Standardizing Trace Data Format

Configuring Crush: The `crush.json` File

What is `ccache` and Why Should I Care?

How `ccache` Works

Using `ccache` on z/OS

1. Installing `ccache`

Integration into Makefiles: Using `ccache` in a Makefile

Benchmarking `ccache` with the zopen Build Framework

Shallow Clones: Reducing the History (`--depth=1`)

Avoiding Downloading Large Binaries (`--filter=blob:none`)

Cloning Only a Single Branch and Avoiding Tags (`--no-tags` and `--single-branch`)

Sparse Checkouts: Clone What You Need (`git sparse-checkout`)

Working-Tree-Encoding: Consider the Conversion Costs (`.gitattributes`)

Profiling and Diagnostics (`GIT_TRACE=1` and `GIT_PERFORMANCE=1`)

Overcoming the `forkpty` hurdle