Offer Vertex-AI as a provider

[hellovai]

Vertex-ai is another commonly offered interface for speaking to LLMs on-top of gemini directly. It has issues in the way we get the authorization token (needs to be refreshed frequently w/ oauth). We can support this if many people require it.

[Foxicution]

We'd like this feature.

[anish-palakurthi]

We'll add support for this by end of the week!

[hellovai]

@Foxicution: @anish-palakurthi has got a prototype of this working (in the playground and runtime)! We should be pushing out the PR and merging soon!

To ensure we support what you need, how do you currently do authentication for vertex-ai?

[Foxicution]

Currently we use the google-cloud-sdk to authenticate using account credentials, and then we do

import vertexai

vertexai.init(project="***", location="***")
model = vertexai.generative_models.GenerativeModel("***")

generation_config = {
    "max_output_tokens": 8192,
    "temperature": 1,
    "top_p": 0.95,
}

safety_settings = {
    vertexai.preview.generative_models.HarmCategory.HARM_CATEGORY_HATE_SPEECH: vertexai.preview.generative_models.HarmBlockThreshold.BLOCK_ONLY_HIGH,
    vertexai.preview.generative_models.HarmCategory.HARM_CATEGORY_DANGEROUS_CONTENT: vertexai.preview.generative_models.HarmBlockThreshold.BLOCK_ONLY_HIGH,
    vertexai.preview.generative_models.HarmCategory.HARM_CATEGORY_SEXUALLY_EXPLICIT: vertexai.preview.generative_models.HarmBlockThreshold.BLOCK_ONLY_HIGH,
    vertexai.preview.generative_models.HarmCategory.HARM_CATEGORY_HARASSMENT: vertexai.preview.generative_models.HarmBlockThreshold.BLOCK_ONLY_HIGH,
}
responses = mode.generate_content(["message"], generation_config=generation_config, safety_settings=safety_settings)

[hellovai]

We'll confirm and ensure it works!

[anish-palakurthi]

@Foxicution
We are getting ready to ship this today!
Take a look and see if this fits your needs!
https://github.com/BoundaryML/baml/blob/vertex-ai/docs/docs/snippets/clients/providers/vertex.mdx

Here is a sample BAML client with your configuration:

client<llm> Vertex {
  provider vertex-ai

  options {
    model gemini-1.5-pro
    project_id my-project-id
    location us-central1
    safetySettings  [
        { 
            category HARM_CATEGORY_HATE_SPEECH
            threshold BLOCK_ONLY_HIGH
        },
        {
            category HARM_CATEGORY_DANGEROUS_CONTENT
            threshold BLOCK_ONLY_HIGH
        },
        {
            category HARM_CATEGORY_SEXUALLY_EXPLICIT
            threshold BLOCK_ONLY_HIGH
        },
        {
            category HARM_CATEGORY_HARASSMENT
            threshold BLOCK_ONLY_HIGH 
        }
      ]
 
    generationConfig {
      maxOutputTokens 10
      temperature 1
      topP 0.95
    }
  }
}

[Foxicution]

Read through the docs, looks good I think. Will try it out.