Custom LLM

starting turbine 26 1 2 release you can use the custom llm tab under account settings to route hero ai through a provider you manage instead of default platform routing the custom llm tab appears only when the custom llm feature flag is enabled for your account and you have account administrator or super administrator access to enable custom llm, contact swimlane support changing custom llm settings while hero ai work is in progress (plan generation, playbook builder prompts, companion chat, and similar) applies the new routing to subsequent llm requests in that operation wait for long running jobs to finish before switching providers if you need a single model throughout one task hero ai native action and provider changes when you turn use custom llm provider on or off, or switch between default swimlane routing and litellm (or custom bedrock on turbine platform ), existing hero ai native action docid\ c4ckmt7fhxgjbtlxxooc2 steps may still store a modelid from the previous configuration starting in 26 2 0 at run time , if the saved model is not in the current available models list, the action uses the small model (the default for hero ai native action) so playbook runs are not blocked in the action dialog, model selection shows previously selected model is not available in red until you pick a valid model or use use default model review playbooks after changing custom llm settings see hero ai native action docid\ c4ckmt7fhxgjbtlxxooc2 in hero ai native action docid\ c4ckmt7fhxgjbtlxxooc2 tooltip on use custom llm provider starting in 26 2 0 , hover the info icon next to use custom llm provider on admin panel > settings > account > custom llm for guidance similar to when using this setting, hero ai traffic is routed to your custom llm if your custom llm is not available at run time, or if you enable this setting and later disable it, hero ai native actions fall back to the default small model so workflows are not disrupted review your automations and update model selections where needed exact tooltip text may vary slightly in the product ui section use when turbine cloud https //docs swimlane com/custom llm#turbine cloud you use hosted turbine cloud and configure litellm turbine platform — litellm https //docs swimlane com/custom llm#turbine platform you use on premises turbine platform with a litellm proxy turbine platform — custom bedrock https //docs swimlane com/custom llm#configure custom bedrock you use on premises turbine platform with your own amazon bedrock api key turbine cloud the custom llm tab on turbine cloud connects hero ai to your litellm proxy you enter a url and api key, fetch the model catalog from your proxy, then choose models from dropdown lists turbine cloud does not show select your provider ; litellm is the only option prerequisites admin panel > settings > account > custom llm is visible litellm base url, api key, and a running litellm compatible endpoint that exposes a models list configure litellm open admin panel > settings > account > custom llm turn use custom llm provider on enter litellm url (https base url for your litellm compatible api) enter api key (credential turbine sends to litellm) click fetch models turbine loads the models your proxy exposes and fills the model dropdowns below select medium model , medium model fallback (optional), small model , and small model fallback (optional) click save field role fetch models retrieves the model list from your litellm proxy after litellm url and api key are set medium model primary model for medium tier hero ai workloads (companion, agents) medium model fallback optional fallback for the medium tier small model primary model for the smallest tier (lighter or faster paths) small model fallback optional fallback for the small tier fetch models is disabled until litellm url and api key are entered when you reopen saved settings, the ui fetches models automatically after you change the url or api key, click fetch models again before you save if the request fails, turbine shows failed to fetch models and the dropdowns stay empty until you correct the connection and fetch again tip choose sonnet class models for medium model and haiku class models for small model when your litellm catalog includes them (for example sonnet45 and haiku45 aliases) turbine platform the custom llm tab on turbine platform supports two providers litellm or custom bedrock use select your provider to choose which form the tab shows llm routing can also be defined in the turbine platform installer (for example llm model id medium) confirm with your swimlane contact whether installer values or account settings take precedence in your cluster configure litellm prerequisites admin panel > settings > account > custom llm is visible litellm base url, api key, and a running litellm compatible endpoint that exposes a models list steps open admin panel > settings > account > custom llm turn use custom llm provider on set select your provider to litellm enter litellm url enter api key click fetch models select medium model , medium model fallback (optional), small model , and small model fallback (optional) click save field role fetch models loads models from litellm into the dropdowns below medium model used for the hero ai companion and ai soc plan generation medium model fallback optional fallback for the medium tier small model used for the hero ai native action and some ai soc features small model fallback optional fallback for the small tier litellm url , api key , medium model , and small model are required when custom llm is enabled tip use sonnet class models for medium tiers and haiku class models for small tiers when your litellm catalog supports them configure custom bedrock custom bedrock supports two configuration paths on turbine platform path use bedrock api key model entry iam / deployment credentials off click fetch models , then select models from dropdowns (same pattern as litellm) bedrock api key (bearer token) on enter bedrock url , bedrock api key , and inference profile ids manually in text fields ( fetch models is not shown) custom bedrock with iam or deployment credentials prerequisites admin panel > settings > account > custom llm is visible bedrock access configured at deployment (iam or cluster settings per your swimlane contact) steps open admin panel > settings > account > custom llm turn use custom llm provider on set select your provider to custom bedrock leave use bedrock api key off click fetch models select medium model , medium model fallback (optional), small model , and small model fallback (optional) click save custom bedrock with bedrock api key prerequisites admin panel > settings > account > custom llm is visible bedrock runtime url, bedrock api key, and cross region inference profile ids for each tier steps open admin panel > settings > account > custom llm turn use custom llm provider on set select your provider to custom bedrock turn use bedrock api key on enter bedrock url (runtime endpoint for your region, for example https //bedrock runtime us west 2 amazonaws com ) enter bedrock api key (masked after save; leave blank to keep the stored key, or enter a new value to replace it) enter inference profile ids in each medium model , small model , and optional fallback field (see below) click save long term amazon bedrock api keys can be configured to last longer than 12 hours swimlane recommends long term keys only for exploratory use to reduce security risk in production for custom bedrock without a bedrock api key (for example iam credentials configured at deployment), follow guidance from your swimlane contact inference profile ids the ui prompts you to enter an inference profile arn for each model use a cross region inference profile id, not a plain foundation model id example global anthropic claude 3 5 sonnet 20241022 v2 0 field role medium model used for the hero ai companion and ai soc plan generation medium model fallback optional fallback for the medium tier small model used for the hero ai native action and some ai soc features small model fallback optional fallback for the small tier the inference profile prefix (us , eu , jp , or so on ) must match the region group of your bedrock url endpoint bedrock api keys are valid for the bedrock runtime only; mapping from a foundation model id to an inference profile requires iam control plane access and does not run on the api key path when use bedrock api key is on, medium model , small model , and the connection fields for your selected provider must be valid before save succeeds validation and errors turbine cloud model and url fields cannot contain characters blocked for injection safety angle brackets, double quotation marks, apostrophe, ampersand, parentheses, curly braces, or semicolons litellm url must be a valid url pattern when use custom llm provider is on, litellm url , api key , medium model , and small model must be valid before save succeeds if fetch models fails, correct litellm url and api key , fetch again, then save turbine platform the same character and url rules apply to litellm and custom bedrock fields for litellm and for custom bedrock without use bedrock api key , select models from the lists populated by fetch models when use custom llm provider is on, all required connection and model fields for your selected provider must be valid before save succeeds both deployments if save fails, read the notification and correct the indicated fields