Custom LLM

starting turbine 26 1 2 release you can use the custom llm tab under account settings to route hero ai through a provider you manage instead of default platform routing the custom llm tab appears only when the custom llm feature flag is enabled for your account and you have account administrator or super administrator access to enable custom llm, contact swimlane support changing custom llm settings while hero ai work is in progress (plan generation, playbook builder prompts, companion chat, and similar) applies the new routing to subsequent llm requests in that operation wait for long running jobs to finish before switching providers if you need a single model throughout one task section use when turbine cloud docid\ deoy k3jmcpdm c ikrq4 you use hosted turbine cloud and configure litellm turbine platform — litellm docid\ deoy k3jmcpdm c ikrq4 you use on premises turbine platform with a litellm proxy turbine platform — custom bedrock docid\ deoy k3jmcpdm c ikrq4 you use on premises turbine platform with your own amazon bedrock api key turbine cloud the custom llm tab on turbine cloud connects hero ai to your litellm proxy you enter a url, an api key, and the model names or aliases prerequisites admin panel > settings > account > custom llm is visible litellm base url, api key, and the model identifiers your proxy exposes configure litellm open admin panel > settings > account > custom llm turn use custom llm provider on enter litellm url (https base url for your litellm compatible api) enter api key (credential turbine sends to litellm) enter all the model fields and click save field role medium model name primary model for medium tier hero ai workloads (companion, agents) medium model fallback name fallback for the medium tier small model name primary model for the smallest tier (lighter or faster paths) small model fallback name fallback for the small tier tip when your litellm gateway publishes claude aliases, swimlane often recommends claude sonnet 4 5 for medium model name and claude haiku 4 5 for small model name use the identifiers your proxy actually exposes turbine on prem the custom llm tab on turbine on prem supports two providers litellm or custom bedrock use select your provider to choose which form the tab shows llm routing can also be defined in the turbine platform installer (for example llm model id medium) confirm with your swimlane contact whether installer values or account settings take precedence in your cluster configure litellm prerequisites admin panel > settings > account > custom llm is visible litellm base url, api key, and the model names or aliases your proxy exposes steps open admin panel > settings > account > custom llm turn use custom llm provider on set select your provider to litellm enter litellm url enter api key enter all the model fields using litellm model names or aliases and click save field role medium model name used for the hero ai companion and ai soc plan generation medium model fallback name (optional) fallback for the medium tier small model name used for the hero ai native action and some ai soc features small model fallback name (optional) fallback for the small tier litellm url , api key , and all the model fields are required when custom llm is enabled tip use sonnet class aliases for medium tiers and haiku class aliases for small tiers when your litellm catalog supports them configure custom bedrock prerequisites admin panel > settings > account > custom llm is visible bedrock runtime url, bedrock api key (when using api key authentication), and cross region inference profile ids for each tier steps open admin panel > settings > account > custom llm turn use custom llm provider on set select your provider to custom bedrock turn use bedrock api key on when you authenticate with a bedrock api key (bearer token) to the bedrock runtime enter bedrock url (runtime endpoint for your region, for example https //bedrock runtime us west 2 amazonaws com) enter bedrock api key enter inference profile ids in each model field (see below) click save long term amazon bedrock api keys can be configured to last longer than 12 hours swimlane recommends long term keys only for exploratory use to reduce security risk in production for custom bedrock without a bedrock api key (for example iam credentials configured at deployment), follow guidance from your swimlane contact inference profile ids the ui prompts you to enter an inference profile arn for each model use a cross region inference profile id, not a plain foundation model id example global anthropic claude 3 5 sonnet 20241022 v2 0 field role medium model name used for the hero ai companion and ai soc plan generation medium model fallback name (optional) fallback for the medium tier small model name used for the hero ai native action and some ai soc features small model fallback name (optional) fallback for the small tier the inference profile prefix (us , eu , jp , or so on ) must match the region group of your bedrock url endpoint bedrock api keys are valid for the bedrock runtime only; mapping from a foundation model id to an inference profile requires iam control plane access and does not run on the api key path all the model fields are required on save when use custom llm provider is on validation and errors turbine cloud model and url fields cannot contain characters blocked for injection safety angle brackets, double quotation marks, apostrophe, ampersand, parentheses, curly braces, or semicolons litellm url must be a valid url pattern when use custom llm provider is on, litellm url , api key , and all the model fields must be valid before save succeeds turbine platform the same character and url rules apply to litellm and custom bedrock fields when use custom llm provider is on, all the model fields and the connection fields for your selected provider must be valid before save succeeds both deployments if save fails, read the notification and correct the indicated fields