Tracing and Monitoring

Event callbacks provide in-depth tracing of aiconfig operations, and allow hooks for debuggability and monitoring.

The AIConfig SDK has a CallbackManager class which can be used to register callbacks that trace prompt resolution, serialization, deserialization, and model inference, as well as intermediate steps.

This lets you get a stack trace of what's going on under the covers, which is especially useful for complex control flow operations.

note

Anyone can register a callback, and filter for the events you care about. You can subsequently use these callbacks to integrate with your own monitoring and observability systems.

Creating custom callbacks

Custom callbacks are functions that conform to the Callback type. They receive a CallbackEvent object containing event details, and return a Promise. Here's an example of a simple logging callback:

Python

app.py

from aiconfig import AIConfigRuntime, CallbackEvent, CallbackManager

async def my_custom_callback(event: CallbackEvent) -> None:
  print(f"Event triggered: {event.name}", event)

TypeScript

app.ts

import { Callback, CallbackEvent } from "aiconfig";

const myCustomCallback: Callback = async (event: CallbackEvent) => {
  console.log(`Event triggered: ${event.name}`, event);
};

Registering Callbacks

info

By default, a CallbackManager is set up which logs all events to to aiconfig.log. This is useful for local debugging.

Python

app.py

from aiconfig import AIConfigRuntime, CallbackEvent, CallbackManager
config = AIConfigRuntime.load('aiconfig.json')

async def my_custom_callback(event: CallbackEvent) -> None:
  print(f"Event triggered: {event.name}", event)

callback_manager = CallbackManager([my_custom_callback])
config.set_callback_manager(callback_manager)

await config.run("prompt_name")

TypeScript

app.ts

import * as path from "path";
import {
  AIConfigRuntime,
  Callback,
  CallbackEvent,
  CallbackManager,
} from "aiconfig";

const config = AIConfigRuntime.load(path.join(__dirname, "aiconfig.json"));

const myCustomCallback: Callback = async (event: CallbackEvent) => {
  console.log(`Event triggered: ${event.name}`, event);
};

const callbackManager = new CallbackManager([myCustomCallback]);
config.setCallbackManager(callbackManager);

await config.run("prompt_name");

Structure of a CallbackEvent

Each CallbackEvent is an object of the CallbackEvent type, containing:

• name: The name of the event (e.g., "on_resolve_start").
• file: The source file where the event is triggered
• data: An object containing relevant data for the event, such as parameters or results.
• ts_ns: An optional timestamp in nanoseconds.

Sample output:

Event triggered: on_resolve_start
CallbackEventModel(name='on_resolve_start', file='/Users/John/Projects/aiconfig/python/src/aiconfig/Config.py', data={'prompt_name': 'get_activities', 'params': None}, ts_ns=1700094936363867000)

Triggering Callbacks

Callbacks are automatically triggered at specific points in the AIConfigRuntime flow. For example, when the resolve method is called on an AIConfigRuntime instance, it triggers on_resolve_start and on_resolve_end events, which are then passed to the CallbackManager to execute any associated callbacks.

Similarly, ModelParsers should trigger their own events when serializing, deserializing, and running inference. These events are also passed to the CallbackManager to execute any associated callbacks.

Handling Callbacks with Timers

The CallbackManager uses a timeout mechanism to ensure callbacks do not hang indefinitely. If a callback does not complete within the specified timeout, it is aborted, and an error is logged. This timeout can be adjusted in the CallbackManager constructor and defaults to 5s if not specified.

Python

custom_timeout = 10; # 10 seconds
callback_manager = CallbackManager([my_logging_callback], custom_timeout)

TypeScript

const customTimeout = 10;
const callbackManager = new CallbackManager(callbacks, customTimeout);

Error Handling

Custom callbacks should include error handling to manage exceptions. Errors thrown within callbacks are caught by the CallbackManager and can be logged or handled as needed.