Run Oneshot Notebooks in GHA

Overview

Recently, a customer asked if they could have our notebooks run automatically rather than interactively. We looked at each other and said: "let us think about it and get back to you." After a bit of hacking, we came up with the notion of Oneshot Runtimes. These ephemeral runtimes are still Docker containers you can bring up with simple docker run commands, just like normal runtimes. However, they take a different set of arguments. Check out https://docs.siftd.ai/guide/oneshot-runtime for examples and details.

The same customer then asked if they could run these notebooks in a GitHub workflow. To make it easy, we decided to create a dedicated GitHub Action called spin-oneshot-action. This article walks through the steps for embedding a oneshot notebook that analyzes logs from all the jobs in a given workflow.

Example Notebook

In this example, our notebook will grab the logs for jobs in the current workflow and do a simple grep for errors.

Shell Cell: Use GitHub API to Grab Logs

Note that the code accesses GHA_TOKEN, GHA_REPOSITORY, and GHA_RUN_ID. The framework forwards these environment variables from the GitHub runner to the notebook.

curl -s -H "Authorization: Bearer $GHA_TOKEN" \
      -H "Accept: application/vnd.github+json" \
      "https://api.github.com/repos/$GHA_REPOSITORY/actions/runs/$GHA_RUN_ID/jobs" \
      | jq -r '.jobs[] | select(.status == "completed" and .conclusion != "skipped") | "\(.id) \(.name)"' \
      | while read job_id job_name; do
          echo "=== Fetching logs for: $job_name (ID: $job_id) ==="
          curl -sL -H "Authorization: Bearer $GHA_TOKEN" \
              "https://api.github.com/repos/$GHA_REPOSITORY/actions/jobs/$job_id/logs"
          echo ""
        done | tee /tmp/gha_logs.txt
 
echo "=== Logs saved to /tmp/gha_logs.txt ==="
echo "Total size: $(wc -c < /tmp/gha_logs.txt) bytes"

Shell Cell: Summarize Errors and Pipe to $GITHUB_STEP_SUMMARY

For this example, we simply grep the log file generated from previous cell and pipe the output to $GITHUB_STEP_SUMMARY. This environment variable points to a file that is mounted into the oneshot runtime from the GitHub runner. GHA uses the contents of this file to populate a summary section of the workflow run.

errors=$(grep -i -n -B2 -A5 'error\|exception\|fatal\|panic\|failed' /tmp/gha_logs.txt 2>/dev/null | head -200)
 
if [ -z "$errors" ]; then
    echo "No error patterns found in logs"
 
    if [ -n "$GITHUB_STEP_SUMMARY" ]; then
        echo "## No Errors Found" >> "$GITHUB_STEP_SUMMARY"
        echo "No error patterns (error, exception, fatal, panic, failed) detected in job logs." >> "$GITHUB_STEP_SUMMARY"
    fi
else
    echo "Found errors:"
    echo "$errors"
 
    if [ -n "$GITHUB_STEP_SUMMARY" ]; then
        echo "## Error Analysis" >> "$GITHUB_STEP_SUMMARY"
        echo "" >> "$GITHUB_STEP_SUMMARY"
        echo "The following error patterns were found in the job logs:" >> "$GITHUB_STEP_SUMMARY"
        echo "" >> "$GITHUB_STEP_SUMMARY"
        echo '```' >> "$GITHUB_STEP_SUMMARY"
        echo "$errors" >> "$GITHUB_STEP_SUMMARY"
        echo '```' >> "$GITHUB_STEP_SUMMARY"
    fi
fi
 
# Also show summary location for debugging
if [ -n "$GITHUB_STEP_SUMMARY" ]; then
    echo ""
    echo "Results written to GITHUB_STEP_SUMMARY"
fi

Use spin-oneshot-action

Now we are ready to use the spin-oneshot-action to invoke a notebook in the workflow. In this example, we add a new job that runs after the other jobs have completed since we will be calling APIs against completed jobs to retreive logs.

NOTE: For your use case, it may make sense to embed the notebook action in an existing job.

Gotchas

• place actual notebook-url value in yaml below
  • you can simply navigate to the notebook in your borwsesr and copy/paste.
• place actual spin-token value in yaml below
  • you can create oneshot spin-token by navigating to the “Runtimes” management page, click on the “Oneshot Token” button on top and follow instructions from there.

jobs:
 
  your-existing-ci-jobs:
    # existing ci stuff
 
  # ci job added that will call oneshot notebook
  spin-oneshot-failure-analysis:
    runs-on: ubuntu-latest
    permissions:
      actions: read      # Required to call Actions API to fetch job/step logs
      contents: read     # Required for general repo access
    steps:
      - name: Analyze failure with SPIN notebook (fetches logs via GitHub API)
        id: failure-analysis
        uses: siftd/spin-oneshot-action@v1
        with:
          notebook-url: <<ENTER NOTEBOOK URL HERE>>
          spin-token: <<ENTER ONESHOT TOKEN HERE>>
          max-duration: 10m
          output-format: logs
        continue-on-error: true

Bonus Step - see the GHA oneshot session in browser

With the above steps, your GHA oneshot notebook will execute. However, what if you want to view the details of the session that it just ran. The runtimes are ephemeral, so we need a way to save away the session before the runtime goes away. We do this by configuring Session Store in the workspace.

• Make sure you are in the workspace that your notebook belongs to
• Click on the workspace, you can find this on top of the left pane menu
• Click on the 'Session Store' tab and fill in the required configuration.
  • You'll have to provide a connection that has permission to read/write to object store of choice

Now when you run the oneshot, you should see last line in the console output that looks something like:

You can view the SPIN session in your browser: https://spin.siftd.ai/acmeorg/default/notebooks/f7rpyid0nutsryummygmmczk?session=ss_thlut65tj1thy30830olux6k

You can simply copy/paste that into a browser window and it should bring up the session. One caveat is you need to be connected to a runtime or you'll see some errors around not being able to retreive data for the session.

Resources

• SiftD SPIN Documentation
• SPIN Oneshot Github Action
• Contact Us