Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: processor refactor #45

Merged
merged 5 commits into from
Jan 25, 2025
Merged

feat: processor refactor #45

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
2e3b338
processor refactor
ponderingdemocritus Jan 25, 2025
3b7d0a7
fix
ponderingdemocritus Jan 25, 2025
22417b4
processor
ponderingdemocritus Jan 25, 2025
2a32366
room content ids
ponderingdemocritus Jan 25, 2025
3a39f48
cleanp
ponderingdemocritus Jan 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 3 additions & 4 deletions examples/example-api.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ import { Orchestrator } from "../packages/core/src/core/orchestrator";
import { HandlerRole } from "../packages/core/src/core/types";
import { RoomManager } from "../packages/core/src/core/room-manager";
import { ChromaVectorDB } from "../packages/core/src/core/vector-db";
import { Processor } from "../packages/core/src/core/processor";
import { MessageProcessor } from "../packages/core/src/core/processor";
import { LLMClient } from "../packages/core/src/core/llm-client";
import { LogLevel } from "../packages/core/src/core/types";
import chalk from "chalk";
Expand Down Expand Up @@ -39,8 +39,7 @@ async function main() {
});

// Initialize processor with default character personality
const processor = new Processor(
vectorDb,
const processor = new MessageProcessor(
llmClient,
defaultCharacter,
loglevel
Expand All @@ -61,7 +60,7 @@ async function main() {
const orchestrator = new Orchestrator(
roomManager,
vectorDb,
processor,
[processor],
scheduledTaskDb,
{
level: loglevel,
Expand Down
7 changes: 3 additions & 4 deletions examples/example-twitter.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ import { HandlerRole } from "../packages/core/src/core/types";
import { TwitterClient } from "../packages/core/src/core/io/twitter";
import { RoomManager } from "../packages/core/src/core/room-manager";
import { ChromaVectorDB } from "../packages/core/src/core/vector-db";
import { Processor } from "../packages/core/src/core/processor";
import { MessageProcessor } from "../packages/core/src/core/processor";
import { LLMClient } from "../packages/core/src/core/llm-client";
import { env } from "../packages/core/src/core/env";
import { LogLevel } from "../packages/core/src/core/types";
Expand Down Expand Up @@ -41,8 +41,7 @@ async function main() {
});

// Initialize processor with default character personality
const processor = new Processor(
vectorDb,
const processor = new MessageProcessor(
llmClient,
defaultCharacter,
loglevel
Expand All @@ -63,7 +62,7 @@ async function main() {
const core = new Orchestrator(
roomManager,
vectorDb,
processor,
[processor],
scheduledTaskDb,
{
level: loglevel,
Expand Down
85 changes: 74 additions & 11 deletions packages/core/src/core/orchestrator.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
import { Logger } from "./logger";
import { RoomManager } from "./room-manager";
import { TaskScheduler } from "./task-scheduler";
import type { Processor } from "./processor";
import type { VectorDB } from "./types";
import type { BaseProcessor, MessageProcessor } from "./processor";
import type { ProcessedResult, VectorDB } from "./types";
import { HandlerRole, LogLevel, type LoggerConfig } from "./types";
import type { IOHandler } from "./types";
import type { ScheduledTaskMongoDb } from "./scheduled-db";
Expand All @@ -21,6 +21,8 @@ export class Orchestrator {

private pollIntervalId?: ReturnType<typeof setInterval>;

private processors: Map<string, BaseProcessor> = new Map();

/**
* A TaskScheduler that only schedules and runs input handlers
*/
Expand All @@ -29,7 +31,6 @@ export class Orchestrator {
>;

private readonly logger: Logger;
private readonly processor: Processor;

private readonly scheduledTaskDb: ScheduledTaskMongoDb;

Expand All @@ -40,12 +41,16 @@ export class Orchestrator {
constructor(
private readonly roomManager: RoomManager,
vectorDb: VectorDB,
processor: Processor,
processors: BaseProcessor[],
scheduledTaskDb: ScheduledTaskMongoDb,
config?: LoggerConfig
) {
this.vectorDb = vectorDb;
this.processor = processor;
this.processors = new Map(
processors.map((p) => {
return [p.getName(), p];
})
);
Comment on lines +49 to +53
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Add validation for duplicate processor names.

When initializing the processors Map, there's no validation to ensure processor names are unique. If two processors return the same name, the latter will silently override the former.

 this.processors = new Map(
-    processors.map((p) => {
-        return [p.getName(), p];
-    })
+    processors.map((p) => {
+        const name = p.getName();
+        if (this.processors.has(name)) {
+            throw new Error(`Duplicate processor name: ${name}`);
+        }
+        return [name, p];
+    })
 );
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
this.processors = new Map(
processors.map((p) => {
return [p.getName(), p];
})
);
const processorNames = new Set();
processors.forEach((p) => {
const name = p.getName();
if (processorNames.has(name)) {
throw new Error(`Duplicate processor name: ${name}`);
}
processorNames.add(name);
});
this.processors = new Map(
processors.map((p) => {
return [p.getName(), p];
})
);

this.scheduledTaskDb = scheduledTaskDb;
this.logger = new Logger(
config ?? {
Expand All @@ -63,6 +68,56 @@ export class Orchestrator {
this.startPolling();
}

public async processContent(
content: any,
source: string
): Promise<ProcessedResult | null> {
const room = await this.roomManager.ensureRoom(source, "core");

// 1) Find the first processor that can handle it
const processor = Array.from(this.processors.values()).find((p) =>
p.canHandle(content)
);

const canHandle = processor !== undefined;

if (!canHandle || !processor) {
// No processor found; fallback
console.log("No suitable processor found for content:", content);
return null;
}

const availableOutputs = Array.from(this.ioHandlers.values()).filter(
(h) => h.role === HandlerRole.OUTPUT
);

const availableActions = Array.from(this.ioHandlers.values()).filter(
(h) => h.role === HandlerRole.ACTION
);

// 2) Process
const processedResult = await processor.process(content, room, {
availableOutputs,
availableActions,
});

// 3) If you want to store memory, do it here
// if (!processedResult.alreadyProcessed) {
// await this.roomManager.addMemory(
// room.id,
// JSON.stringify(processedResult.content),
// {
// source,
// type: "input",
// ...processedResult.metadata,
// ...processedResult.enrichedContext,
// }
// );
// }

return processedResult;
}

/**
* Primary method to register any IOHandler (input or output).
* - If it's an input with an interval, schedule it for recurring runs.
Expand All @@ -80,8 +135,6 @@ export class Orchestrator {
}
this.ioHandlers.set(handler.name, handler);

this.processor.registerIOHandler(handler);

this.logger.info(
"Orchestrator.registerIOHandler",
`Registered ${handler.role}`,
Expand Down Expand Up @@ -248,15 +301,25 @@ export class Orchestrator {
// 1) Ensure there's a room
const room = await this.roomManager.ensureRoom(source, "core");

// 2) Process with your existing processor logic
const processed = await this.processor.process(data, room);
// 2) Find the first processor that can handle it
const processor = Array.from(this.processors.values()).find((p) =>
p.canHandle(data)
);

if (!processor) {
// this.logger.warn("No processor found for data", { source });
continue;
}

// 3) Process with the found processor
const processed = await processor.process(data, room);

// If the processor thinks we've already processed it, we skip
if (processed.alreadyProcessed) {
continue;
}

// 3) Save to memory (like you do in processInputTask)
// 4) Save to memory (like you do in processInputTask)
await this.roomManager.addMemory(
room.id,
JSON.stringify(processed.content),
Expand All @@ -278,7 +341,7 @@ export class Orchestrator {
}
}

// 4) For each suggested output, see if it’s an action or an output
// 5) For each suggested output, see if it's an action or an output
for (const output of processed.suggestedOutputs) {
const handler = this.ioHandlers.get(output.name);
if (!handler) {
Expand Down
Loading
Loading