Image Analyzer for Home Assistant using GPT Vision

🌟 Features · 📖 Resources · ⬇️ Installation · ▶️ Usage · 🧠 Model Overview · 🪲 How to report Bugs

gpt4vision is a Home Assistant integration that allows you to analyze images and camera feeds using GPT-4 Vision.
Supported providers are OpenAI, Anthropic, Google Gemini, LocalAI and Ollama.

• Compatible with OpenAI, Anthropic Claude, Google Gemini, LocalAI and Ollama
• Takes images and camera entities as input as well as image files
• Images can be downscaled for faster processing
• Can be installed and updated through HACS and can be set up in the Home Assistant UI

Check the 📖 wiki for examples on how you can integrate gpt4vision into your Home Assistant setup or join the 🗨️ discussion in the Home Assistant Community.

1. Search for GPT-4 Vision in Home Assistant Settings/Devices & services
2. Select your provider
3. Follow the instructions to complete setup

1. Download and copy the gpt4vision folder into your custom_components folder.
2. Add integration in Home Assistant Settings/Devices & services
3. Provide your API key or IP address and port of your self-hosted server

Simply obtain an API key from OpenAI and enter it in the Home Assistant UI during setup.
A pricing calculator is available here: https://openai.com/api/pricing/.

Obtain an API key from Anthropic and enter it in the Home Assistant UI during setup. Pricing is available here: Anthropic image cost. Images can be downscaled with the built-in downscaler.

To use Google Gemini you need to have a Google account and obtain an API key from the AI Studio. Depending on your region, you may need to enable billing. Pricing is available here: Gemini Pricing

To use LocalAI you need to have a LocalAI server running. You can find the installation instructions here. During setup you'll need to provide the IP address of your machine and the port on which LocalAI is running (default is 8000).

To use Ollama you need to have an Ollama server running. You can download it from here. Once installed you need to run the following command to download the llava model:

ollama run llava

If your Home Assistant is not running on the same computer as Ollama, you need to set the OLLAMA_HOST environment variable.

On Linux:

1. Edit the systemd service by calling systemctl edit ollama.service. This will open an editor.
2. For each environment variable, add a line Environment under section [Service]:

[Service]
Environment="OLLAMA_HOST=0.0.0.0"

3. Save and close the editor.
4. Reload systemd and restart Ollama

systemctl daemon-reload
systemctl restart ollama

On Windows:

1. Quit Ollama from the system tray
2. Open File Explorer
3. Right click on This PC and select Properties
4. Click on Advanced system settings
5. Select Environment Variables
6. Under User variables click New
7. For variable name enter OLLAMA_HOST and for value enter 0.0.0.0
8. Click OK and start Ollama again from the Start Menu

On macOS:

1. Open Terminal
2. Run the following command

launchctl setenv OLLAMA_HOST "0.0.0.0"

3. Restart Ollama

After restarting, the gpt4vision.image_analyzer service will be available. You can test it in the developer tools section in home assistant. To get OpenAI gpt-4o's analysis of a local image, use the following service call.

service: gpt4vision.image_analyzer
data:
  provider: OpenAI
  message: Describe what you see?
  max_tokens: 100
  model: gpt-4o
  image_file: |-
    /config/www/tmp/example.jpg
    /config/www/tmp/example2.jpg
  image_entity:
    - camera.garage
    - image.front_door_person
  target_width: 1280
  detail: low
  temperature: 0.5
  include_filename: true

Image 1:
<base64 encoded image>
Image 2:
<base64 encoded image>
...
<Your prompt>

Front Door:
<base64 encoded image>
front_door_2024-12-31_23:59:59:
<base64 encoded image>
...
<Your prompt>

Data is based on the MMMU Leaderboard²

gpt4vision is compatible with multiple providers, each of which has different models available. Some providers run in the cloud, while others are self-hosted.
To see which model is best for your use case, check the figure below. It visualizes the averaged MMMU¹ scores of available cloud-based models. The higher the score, the better the model performs.

Benchmark will be updated regularly to include new models.

¹ MMMU stands for "Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark". It assesses multimodal capabilities including image understanding.
² The data is based on the MMMU Leaderboard

To enable debugging, add the following to your configuration.yaml:

logger:
  logs:
    custom_components.gpt4vision: debug