Skip to content

Ollama classifier with llama3.1

Welcome to the Ollama Classifier guide! This example will walk you through creating an Ollama classifier and integrating it into your Multi-Agent Orchestrator System. Let’s dive in!

πŸ“š Prerequisites:

  • Basic knowledge of Python
  • Familiarity with the Multi-Agent Orchestrator System
  • Ollama installed on your machine

πŸ’Ύ 1. Ollama installation:

First, let’s install the Ollama Python package:

Terminal window
pip install ollama

🧬 2. Create the Ollama Classifier class:

Now, let’s create our OllamaClassifier class. This class extends the Classifier abstract class from the Multi-Agent Orchestrator. The process_request method must be implemented by the OllamaClassifier

from typing import List, Dict, Optional, Any
from multi_agent_orchestrator.classifiers import Classifier, ClassifierResult
from multi_agent_orchestrator.types import ConversationMessage, ParticipantRole
from multi_agent_orchestrator.utils import Logger
import ollama
class OllamaClassifierOptions:
def __init__(self,
model_id: Optional[str] = None,
inference_config: Optional[Dict[str, Any]] = None,
host: Optional[str] = None
):
self.model_id = model_id
self.inference_config = inference_config or {}
self.host = host
class OllamaClassifier(Classifier):
def __init__(self, options: OllamaClassifierOptions):
super().__init__()
self.model_id = options.model_id or 'llama3.1'
self.inference_config = options.inference_config
self.streaming = False
self.temperature = options.inference_config.get('temperature', 0.0)
self.client = ollama.Client(host=options.host or None)
async def process_request(self,
input_text: str,
chat_history: List[ConversationMessage]) -> ClassifierResult:
messages = [
{"role": msg.role, "content": msg.content[0]['text']}
for msg in chat_history
]
self.system_prompt = self.system_prompt + f'\n question: {input_text}'
messages.append({"role": ParticipantRole.USER.value, "content": self.system_prompt})
try:
response = self.client.chat(
model=self.model_id,
messages=messages,
options={'temperature':self.temperature},
tools=[{
'type': 'function',
'function': {
'name': 'analyzePrompt',
'description': 'Analyze the user input and provide structured output',
'parameters': {
'type': 'object',
'properties': {
'userinput': {
'type': 'string',
'description': 'The original user input',
},
'selected_agent': {
'type': 'string',
'description': 'The name of the selected agent',
},
'confidence': {
'type': 'number',
'description': 'Confidence level between 0 and 1',
},
},
'required': ['userinput', 'selected_agent', 'confidence'],
},
}
}]
)
# Check if the model decided to use the provided function
if not response['message'].get('tool_calls'):
Logger.get_logger().info(f"The model didn't use the function. Its response was:{response['message']['content']}")
raise Exception(f'Ollama model {self.model_id} did not use tools')
else:
tool_result = response['message'].get('tool_calls')[0].get('function', {}).get('arguments', {})
return ClassifierResult(
selected_agent=self.get_agent_by_id(tool_result.get('selected_agent', None)),
confidence=float(tool_result.get('confidence', 0.0))
)
except Exception as e:
Logger.get_logger().error(f'Error in Ollama Classifier :{str(e)}')
raise e

Now that we have our OllamaClassifier, let’s use it in the Multi-Agent Orchestrator:

πŸ”— 3. Use OllamaClassifier in the orchestrator:

If you have used the quickstarter sample program, you can use the Ollama classifier and run it like this:

from ollamaClassifier import OllamaClassifier, OllamaClassifierOptions
from multi_agent_orchestrator.orchestrator import MultiAgentOrchestrator
classifier = OllamaClassifier(OllamaClassifierOptions(
model_id='llama3.1',
inference_config={'temperature':0.0}
))
orchestrator = MultiAgentOrchestrator()
# Use our newly created classifier within the orchestrator
orchestrator.set_classifier(classifier)

And you are done!

πŸƒ 4. Run Your Ollama Model Locally:

Before running your program, make sure to start the Ollama model locally:

Terminal window
ollama run llama3.1

If you haven’t downloaded the Llama3.1 model yet, it will be downloaded automatically before running.

πŸŽ‰ You’re All Set!

Congratulations! You’ve successfully integrated an Ollama classifier into your Multi-Agent Orchestrator System. Now you can start classifiying user requests and leveraging the power of Llama3.1 in your applications!

6.πŸ’‘ Next Steps:

  • Experiment with different Ollama models
  • Build a complete multi agent system in an offline environment

Happy coding! πŸš€