Using Any Language Model

A cornerstone philosophy of AI JSON is that it is platform-agnostic.

To achieve that in the llm action of the aijson-ml package, we use litellm.

Any models supported by litellm can be used in AI JSON, even if not shown by auto-complete. Please note that many litellm models do not have async support, and will block the event loop during inference.

Inferring the Default Model

If no language model is specified in the config, the llm action will guess what model to use based on the environment:

• if an OPENAI_API_KEY is found in the environment, it will use OpenAI’s GPT-4o;
• if an ANTHROPIC_API_KEY is found in the environment, it will use Anthropic’s Claude Sonnet 3.5;
• if an Ollama server is found running on http://localhost:11434, it will use the first downloaded model.
• if an AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are found in the environment, it will use AWS Bedrock LLama3.1 405B.

Otherwise, an error is thrown.

We would love to accept contributions to extend this inference mechanism to other models! Check out the infer_default_model function.

Specifying a Model

You can set the default model on the default_model key in the top-level of the config:

default_model:
  model: gpt-4o
  api_base: ...
  auth_token: ...
  temperature: ...
  max_output_tokens: ...
  max_prompt_tokens: ...
  top_p: ...
  frequency_penalty: ...
flow: ...

You can also set the model on individual llm action invocations:

flow:
  catchphrase:
    action: llm
    model:
      model: gpt-4o
      temperature: 0.5
    prompt: Come up with a cool catchphrase about {{ thing }}