Problems with calling Gemini API from Vertex training job

[FLB1]

I am trying to run Vertex training job which contains calls to Gemini API (GenerativeModel -> generate_content()), however it fails on error "PermissionDenied: 403 Request had insufficient authentication scopes". I tried to generate service account key in container using "gcloud iam service-accounts keys create", but this fails on error "Your current active account [] does not have any valid credentials" -> I only found mention from 2014 that setting CLOUDSDK_PYTHON_SITEPACKAGES=1 might fix this, but it didn't.

So how this should be done. In project which is used in vertexai.init() service account attached to training jobs has following rights: Vertex AI Custom Code Service Agent, Vertex AI Extension Custom Code Service Agent, Vertex AI User

[macato97]

I am having the same issue with Document-AI in place of Gemini. I think the problem is that, when the custom job runs, it uses the priviledges associated with your Vertex AI Custom Code Service Agent (see here). It looks like this Service Agent has by default access only to BigQuery and Storage, so you need to enable this service account to the proper Gemini role.