Embeddings

Vertex AI offers wide ranging support for embedding text, images and videos. Portkey provides a standardized interface for embedding multiple modalities.

Gemini Embedding Models

The gemini-embedding-2-preview model supports embedding across multiple modalities — text, image, video, and audio — through a single unified endpoint.

Input Type	Supported Formats
Text	Plain string or structured object
Image	GCS URI, HTTPS URL, base64, data URI
Video	GCS URI, HTTPS URL, base64
Audio	GCS URI, HTTPS URL, base64

Additional supported parameters: task_type, dimensions

Embedding Text

from portkey_ai import Portkey

client = Portkey(
    api_key="YOUR_PORTKEY_API_KEY",
    provider="@PROVIDER",
    vertex_region="us-central1",
)

embeddings = client.embeddings.create(
    model="gemini-embedding-2-preview",
    input="What is the meaning of life?",
)

import { Portkey } from 'portkey-ai';

const portkey = new Portkey({
    apiKey: "YOUR_API_KEY",
    provider: "@YOUR_PROVIDER",
    vertexRegion: "us-central1",
});

const embedding = await portkey.embeddings.create({
    input: "What is the meaning of life?",
    model: "gemini-embedding-2-preview",
});

console.log(embedding);

curl --location 'https://api.portkey.ai/v1/embeddings' \
--header 'Content-Type: application/json' \
--header 'x-portkey-api-key: PORTKEY_API_KEY' \
--header 'x-portkey-provider: @PORTKEY_PROVIDER' \
--header 'x-portkey-vertex-region: us-central1' \
--data '{
    "model": "gemini-embedding-2-preview",
    "input": "What is the meaning of life?"
}'

from openai import OpenAI
from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders

portkey_client = OpenAI(
    api_key="NOT_REQUIRED",
    base_url=PORTKEY_GATEWAY_URL,
    default_headers=createHeaders(
        provider="@PORTKEY_PROVIDER",
        api_key="PORTKEY_API_KEY",
        vertex_region="us-central1",
    )
)

embeddings = portkey_client.embeddings.create(
    model="gemini-embedding-2-preview",
    input="What is the meaning of life?",
)

import OpenAI from 'openai';
import { PORTKEY_GATEWAY_URL, createHeaders } from 'portkey-ai'

const portkeyClient = new OpenAI({
  apiKey: "NOT_REQUIRED",
  baseURL: PORTKEY_GATEWAY_URL,
  defaultHeaders: createHeaders({
    apiKey: "PORTKEY_API_KEY",
    provider: "@PORTKEY_PROVIDER",
    vertexRegion: "us-central1",
  })
});

const embedding = await portkeyClient.embeddings.create({
    input: "What is the meaning of life?",
    model: "gemini-embedding-2-preview",
});

console.log(embedding);

Embedding Images

from portkey_ai import Portkey

client = Portkey(
    api_key="YOUR_PORTKEY_API_KEY",
    provider="@PROVIDER",
    vertex_region="us-central1",
)

embeddings = client.embeddings.create(
    model="gemini-embedding-2-preview",
    input=[
        {
            "image": {
                "url": "gs://your-bucket/image.png",
                "mime_type": "image/png"
            }
        }
    ],
)

import { Portkey } from 'portkey-ai';

const portkey = new Portkey({
    apiKey: "YOUR_API_KEY",
    provider: "@YOUR_PROVIDER",
    vertexRegion: "us-central1",
});

const embedding = await portkey.embeddings.create({
    input: [
        {
            image: {
                url: "gs://your-bucket/image.png",
                mime_type: "image/png"
            }
        }
    ],
    model: "gemini-embedding-2-preview",
});

console.log(embedding);

curl --location 'https://api.portkey.ai/v1/embeddings' \
--header 'Content-Type: application/json' \
--header 'x-portkey-api-key: PORTKEY_API_KEY' \
--header 'x-portkey-provider: @PORTKEY_PROVIDER' \
--header 'x-portkey-vertex-region: us-central1' \
--data '{
    "model": "gemini-embedding-2-preview",
    "input": [
        {
            "image": {
                "url": "gs://your-bucket/image.png",
                "mime_type": "image/png"
            }
        }
    ]
}'

from openai import OpenAI
from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders

portkey_client = OpenAI(
    api_key="NOT_REQUIRED",
    base_url=PORTKEY_GATEWAY_URL,
    default_headers=createHeaders(
        provider="@PORTKEY_PROVIDER",
        api_key="PORTKEY_API_KEY",
        vertex_region="us-central1",
    )
)

embeddings = portkey_client.embeddings.create(
    model="gemini-embedding-2-preview",
    input=[
        {
            "image": {
                "url": "gs://your-bucket/image.png",
                "mime_type": "image/png"
            }
        }
    ],
)

import OpenAI from 'openai';
import { PORTKEY_GATEWAY_URL, createHeaders } from 'portkey-ai'

const portkeyClient = new OpenAI({
  apiKey: "NOT_REQUIRED",
  baseURL: PORTKEY_GATEWAY_URL,
  defaultHeaders: createHeaders({
    apiKey: "PORTKEY_API_KEY",
    provider: "@PORTKEY_PROVIDER",
    vertexRegion: "us-central1",
  })
});

const embedding = await portkeyClient.embeddings.create({
    input: [
        {
            image: {
                url: "gs://your-bucket/image.png",
                mime_type: "image/png"
            }
        }
    ],
    model: "gemini-embedding-2-preview",
});

console.log(embedding);

You can also pass images as base64:

{
    "input": [
        {
            "image": {
                "base64": "iVBORw0KGgoAAAANSUhEUgAA...",
                "mime_type": "image/png"
            }
        }
    ]
}

Or as a data URI:

{
    "input": [
        {
            "image": {
                "url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA..."
            }
        }
    ]
}

Embedding Videos

from portkey_ai import Portkey

client = Portkey(
    api_key="YOUR_PORTKEY_API_KEY",
    provider="@PROVIDER",
    vertex_region="us-central1",
)

embeddings = client.embeddings.create(
    model="gemini-embedding-2-preview",
    input=[
        {
            "video": {
                "url": "gs://cloud-samples-data/generative-ai/video/pixel8.mp4",
                "mime_type": "video/mp4"
            }
        }
    ],
)

import { Portkey } from 'portkey-ai';

const portkey = new Portkey({
    apiKey: "YOUR_API_KEY",
    provider: "@YOUR_PROVIDER",
    vertexRegion: "us-central1",
});

const embedding = await portkey.embeddings.create({
    input: [
        {
            video: {
                url: "gs://cloud-samples-data/generative-ai/video/pixel8.mp4",
                mime_type: "video/mp4"
            }
        }
    ],
    model: "gemini-embedding-2-preview",
});

console.log(embedding);

curl --location 'https://api.portkey.ai/v1/embeddings' \
--header 'Content-Type: application/json' \
--header 'x-portkey-api-key: PORTKEY_API_KEY' \
--header 'x-portkey-provider: @PORTKEY_PROVIDER' \
--header 'x-portkey-vertex-region: us-central1' \
--data '{
    "model": "gemini-embedding-2-preview",
    "input": [
        {
            "video": {
                "url": "gs://cloud-samples-data/generative-ai/video/pixel8.mp4",
                "mime_type": "video/mp4"
            }
        }
    ]
}'

from openai import OpenAI
from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders

portkey_client = OpenAI(
    api_key="NOT_REQUIRED",
    base_url=PORTKEY_GATEWAY_URL,
    default_headers=createHeaders(
        provider="@PORTKEY_PROVIDER",
        api_key="PORTKEY_API_KEY",
        vertex_region="us-central1",
    )
)

embeddings = portkey_client.embeddings.create(
    model="gemini-embedding-2-preview",
    input=[
        {
            "video": {
                "url": "gs://cloud-samples-data/generative-ai/video/pixel8.mp4",
                "mime_type": "video/mp4"
            }
        }
    ],
)

import OpenAI from 'openai';
import { PORTKEY_GATEWAY_URL, createHeaders } from 'portkey-ai'

const portkeyClient = new OpenAI({
  apiKey: "NOT_REQUIRED",
  baseURL: PORTKEY_GATEWAY_URL,
  defaultHeaders: createHeaders({
    apiKey: "PORTKEY_API_KEY",
    provider: "@PORTKEY_PROVIDER",
    vertexRegion: "us-central1",
  })
});

const embedding = await portkeyClient.embeddings.create({
    input: [
        {
            video: {
                url: "gs://cloud-samples-data/generative-ai/video/pixel8.mp4",
                mime_type: "video/mp4"
            }
        }
    ],
    model: "gemini-embedding-2-preview",
});

console.log(embedding);

Embedding Audio

from portkey_ai import Portkey

client = Portkey(
    api_key="YOUR_PORTKEY_API_KEY",
    provider="@PROVIDER",
    vertex_region="us-central1",
)

embeddings = client.embeddings.create(
    model="gemini-embedding-2-preview",
    input=[
        {
            "audio": {
                "url": "gs://cloud-samples-data/generative-ai/audio/Chirp-3-Docs-Dive.mp3",
                "mime_type": "audio/mpeg"
            }
        }
    ],
)

import { Portkey } from 'portkey-ai';

const portkey = new Portkey({
    apiKey: "YOUR_API_KEY",
    provider: "@YOUR_PROVIDER",
    vertexRegion: "us-central1",
});

const embedding = await portkey.embeddings.create({
    input: [
        {
            audio: {
                url: "gs://cloud-samples-data/generative-ai/audio/Chirp-3-Docs-Dive.mp3",
                mime_type: "audio/mpeg"
            }
        }
    ],
    model: "gemini-embedding-2-preview",
});

console.log(embedding);

curl --location 'https://api.portkey.ai/v1/embeddings' \
--header 'Content-Type: application/json' \
--header 'x-portkey-api-key: PORTKEY_API_KEY' \
--header 'x-portkey-provider: @PORTKEY_PROVIDER' \
--header 'x-portkey-vertex-region: us-central1' \
--data '{
    "model": "gemini-embedding-2-preview",
    "input": [
        {
            "audio": {
                "url": "gs://cloud-samples-data/generative-ai/audio/Chirp-3-Docs-Dive.mp3",
                "mime_type": "audio/mpeg"
            }
        }
    ]
}'

from openai import OpenAI
from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders

portkey_client = OpenAI(
    api_key="NOT_REQUIRED",
    base_url=PORTKEY_GATEWAY_URL,
    default_headers=createHeaders(
        provider="@PORTKEY_PROVIDER",
        api_key="PORTKEY_API_KEY",
        vertex_region="us-central1",
    )
)

embeddings = portkey_client.embeddings.create(
    model="gemini-embedding-2-preview",
    input=[
        {
            "audio": {
                "url": "gs://cloud-samples-data/generative-ai/audio/Chirp-3-Docs-Dive.mp3",
                "mime_type": "audio/mpeg"
            }
        }
    ],
)

import OpenAI from 'openai';
import { PORTKEY_GATEWAY_URL, createHeaders } from 'portkey-ai'

const portkeyClient = new OpenAI({
  apiKey: "NOT_REQUIRED",
  baseURL: PORTKEY_GATEWAY_URL,
  defaultHeaders: createHeaders({
    apiKey: "PORTKEY_API_KEY",
    provider: "@PORTKEY_PROVIDER",
    vertexRegion: "us-central1",
  })
});

const embedding = await portkeyClient.embeddings.create({
    input: [
        {
            audio: {
                url: "gs://cloud-samples-data/generative-ai/audio/Chirp-3-Docs-Dive.mp3",
                mime_type: "audio/mpeg"
            }
        }
    ],
    model: "gemini-embedding-2-preview",
});

console.log(embedding);

Multimodal Embedding (Mixed Inputs)

You can combine multiple input types in a single request:

from portkey_ai import Portkey

client = Portkey(
    api_key="YOUR_PORTKEY_API_KEY",
    provider="@PROVIDER",
    vertex_region="us-central1",
)

embeddings = client.embeddings.create(
    model="gemini-embedding-2-preview",
    input=[
        {
            "video": {
                "url": "gs://cloud-samples-data/generative-ai/video/pixel8.mp4",
                "mime_type": "video/mp4"
            }
        },
        {
            "audio": {
                "url": "gs://cloud-samples-data/generative-ai/audio/Chirp-3-Docs-Dive.mp3",
                "mime_type": "audio/mpeg"
            }
        }
    ],
)

import { Portkey } from 'portkey-ai';

const portkey = new Portkey({
    apiKey: "YOUR_API_KEY",
    provider: "@YOUR_PROVIDER",
    vertexRegion: "us-central1",
});

const embedding = await portkey.embeddings.create({
    input: [
        {
            video: {
                url: "gs://cloud-samples-data/generative-ai/video/pixel8.mp4",
                mime_type: "video/mp4"
            }
        },
        {
            audio: {
                url: "gs://cloud-samples-data/generative-ai/audio/Chirp-3-Docs-Dive.mp3",
                mime_type: "audio/mpeg"
            }
        }
    ],
    model: "gemini-embedding-2-preview",
});

console.log(embedding);

curl --location 'https://api.portkey.ai/v1/embeddings' \
--header 'Content-Type: application/json' \
--header 'x-portkey-api-key: PORTKEY_API_KEY' \
--header 'x-portkey-provider: @PORTKEY_PROVIDER' \
--header 'x-portkey-vertex-region: us-central1' \
--data '{
    "model": "gemini-embedding-2-preview",
    "input": [
        {
            "video": {
                "url": "gs://cloud-samples-data/generative-ai/video/pixel8.mp4",
                "mime_type": "video/mp4"
            }
        },
        {
            "audio": {
                "url": "gs://cloud-samples-data/generative-ai/audio/Chirp-3-Docs-Dive.mp3",
                "mime_type": "audio/mpeg"
            }
        }
    ]
}'

from openai import OpenAI
from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders

portkey_client = OpenAI(
    api_key="NOT_REQUIRED",
    base_url=PORTKEY_GATEWAY_URL,
    default_headers=createHeaders(
        provider="@PORTKEY_PROVIDER",
        api_key="PORTKEY_API_KEY",
        vertex_region="us-central1",
    )
)

embeddings = portkey_client.embeddings.create(
    model="gemini-embedding-2-preview",
    input=[
        {
            "video": {
                "url": "gs://cloud-samples-data/generative-ai/video/pixel8.mp4",
                "mime_type": "video/mp4"
            }
        },
        {
            "audio": {
                "url": "gs://cloud-samples-data/generative-ai/audio/Chirp-3-Docs-Dive.mp3",
                "mime_type": "audio/mpeg"
            }
        }
    ],
)

import OpenAI from 'openai';
import { PORTKEY_GATEWAY_URL, createHeaders } from 'portkey-ai'

const portkeyClient = new OpenAI({
  apiKey: "NOT_REQUIRED",
  baseURL: PORTKEY_GATEWAY_URL,
  defaultHeaders: createHeaders({
    apiKey: "PORTKEY_API_KEY",
    provider: "@PORTKEY_PROVIDER",
    vertexRegion: "us-central1",
  })
});

const embedding = await portkeyClient.embeddings.create({
    input: [
        {
            video: {
                url: "gs://cloud-samples-data/generative-ai/video/pixel8.mp4",
                mime_type: "video/mp4"
            }
        },
        {
            audio: {
                url: "gs://cloud-samples-data/generative-ai/audio/Chirp-3-Docs-Dive.mp3",
                mime_type: "audio/mpeg"
            }
        }
    ],
    model: "gemini-embedding-2-preview",
});

console.log(embedding);

Setting Task Type and Dimensions

You can optionally specify task_type and dimensions to control the embedding behavior:

{
    "model": "gemini-embedding-2-preview",
    "input": "What is the meaning of life?",
    "task_type": "RETRIEVAL_DOCUMENT",
    "dimensions": 768
}

Legacy Embedding Models

The following sections cover the older Vertex AI embedding models like textembedding-gecko@003 and multimodalembedding@001.

Embedding Text

from portkey_ai import Portkey

client = Portkey(
    api_key="YOUR_PORTKEY_API_KEY", # defaults to os.environ.get("PORTKEY_API_KEY")
    provider="@PROVIDER",
)

embeddings = client.embeddings.create(
  model="textembedding-gecko@003",
  input_type="classification",
  input="The food was delicious and the waiter...",
  # input=["text to embed", "more text to embed"], # if you would like to embed multiple texts
)

import { Portkey } from 'portkey-ai';

const portkey = new Portkey({
    apiKey: "YOUR_API_KEY",
    provider:"@YOUR_PROVIDER"
});

const embedding = await portkey.embeddings.create({
    input: 'Name the tallest buildings in Hawaii',
    // input: ['text to embed', 'more text to embed'], // if you would like to embed multiple texts
    model: 'textembedding-gecko@003'
});

console.log(embedding);

curl --location 'https://api.portkey.ai/v1/embeddings' \
--header 'Content-Type: application/json' \
--header 'x-portkey-api-key: PORTKEY_API_KEY' \
--header 'x-portkey-provider: PORTKEY_PROVIDER' \
--data-raw '{
    "model": "textembedding-gecko@003",
    "input": [
        "A HTTP 246 code is used to signify an AI response containing hallucinations or other inaccuracies",
        "246: Partially incorrect response"
    ],
    # "input": "Name the tallest buildings in Hawaii",
    "input_type": "classification"
}'

from openai import OpenAI
from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders

portkey_client = OpenAI(
    api_key='NOT_REQUIRED',
    base_url=PORTKEY_GATEWAY_URL,
    default_headers=createHeaders(
        provider="openai",
        api_key="PORTKEY_API_KEY"
    )
)

embeddings = portkey_client.embeddings.create(
  model="textembedding-gecko@003",
  input_type="classification",
  input="The food was delicious and the waiter...",
  # input=["text to embed", "more text to embed"], # if you would like to embed multiple texts
)

import OpenAI from 'openai'; // We're using the v4 SDK
import { PORTKEY_GATEWAY_URL, createHeaders } from 'portkey-ai'

const portkeyClient = new OpenAI({
  apiKey: 'NOT_REQUIRED', // defaults to process.env["OPENAI_API_KEY"],
  baseURL: PORTKEY_GATEWAY_URL,
  defaultHeaders: createHeaders({
    provider: "vertex-ai",
    apiKey: "PORTKEY_API_KEY", // defaults to process.env["PORTKEY_API_KEY"]
    provider:"@PORTKEY_PROVIDER"
  })
});

const embedding = await portkeyClient.embeddings.create({
    input: 'Name the tallest buildings in Hawaii',
    // input: ['text to embed', 'more text to embed'], // if you would like to embed multiple texts
    model: 'textembedding-gecko@003'
});

console.log(embedding);

Embeddings Images

from portkey_ai import Portkey

client = Portkey(
    api_key="YOUR_PORTKEY_API_KEY", # defaults to os.environ.get("PORTKEY_API_KEY")
    provider="@PROVIDER",
)

embeddings = client.embeddings.create(
  model="multimodalembedding@001",
  input=[
          {
              "text": "this is the caption of the image",
              "image": {
                  "base64": "UklGRkacAABXRUJQVlA4IDqcAACQggKdASqpAn8B.....",
                  # "url": "gcs://..." # if you want to use a url
              }
          }
      ]
)

import { Portkey } from 'portkey-ai';

const portkey = new Portkey({
    apiKey: "YOUR_API_KEY",
    provider:"@YOUR_PROVIDER"
});

const embedding = await portkey.embeddings.create({
    input: [
          {
              "text": "this is the caption of the image",
              "image": {
                  "base64": "UklGRkacAABXRUJQVlA4IDqcAACQggKdASqpAn8B.....",
                  // "url": "gcs://..." // if you want to use a url
              }
          }
      ],
    model: 'multimodalembedding@001'
});

console.log(embedding);

curl --location 'https://api.portkey.ai/v1/embeddings' \
--header 'Content-Type: application/json' \
--header 'x-portkey-api-key: PORTKEY_API_KEY' \
--header 'x-portkey-provider: PORTKEY_PROVIDER' \
--data-raw '{
    "model": "multimodalembedding@001",
    "input": [
                  {
                      "text": "this is the caption of the image",
                      "image": {
                          "base64": "UklGRkacAABXRUJQVlA4IDqcAACQggKdASqpAn8B....."
                          # "url": "gcs://..." # if you want to use a url
                      }
                  }
              ]
}'

from openai import OpenAI
from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders

portkey_client = OpenAI(
    api_key='NOT_REQUIRED',
    base_url=PORTKEY_GATEWAY_URL,
    default_headers=createHeaders(
        provider="openai",
        api_key="PORTKEY_API_KEY"
    )
)

embeddings = portkey_client.embeddings.create(
  model="multimodalembedding@001",
  input=[
          {
              "text": "this is the caption of the image",
              "image": {
                  "base64": "UklGRkacAABXRUJQVlA4IDqcAACQggKdASqpAn8B.....",
                  # "url": "gcs://..." # if you want to use a url
              }
          }
        ]
)

import OpenAI from 'openai'; // We're using the v4 SDK
import { PORTKEY_GATEWAY_URL, createHeaders } from 'portkey-ai'

const portkeyClient = new OpenAI({
  apiKey: 'NOT_REQUIRED', // defaults to process.env["OPENAI_API_KEY"],
  baseURL: PORTKEY_GATEWAY_URL,
  defaultHeaders: createHeaders({
    apiKey: "PORTKEY_API_KEY", // defaults to process.env["PORTKEY_API_KEY"]
    provider:"@PORTKEY_PROVIDER"
  })
});

const embedding = await portkeyClient.embeddings.create({
    input: [
              {
                  "text": "this is the caption of the image",
                  "image": {
                      "base64": "UklGRkacAABXRUJQVlA4IDqcAACQggKdASqpAn8B.....",
                      // "url": "gcs://..." // if you want to use a url
                  }
              }
            ],
    model: 'multimodalembedding@001'
});

console.log(embedding);

Embeddings Videos

from portkey_ai import Portkey

client = Portkey(
    api_key="YOUR_PORTKEY_API_KEY", # defaults to os.environ.get("PORTKEY_API_KEY")
    provider="@PROVIDER",
)

embeddings = client.embeddings.create(
  model="multimodalembedding@001",
  input=[
          {
              "text": "this is the caption of the video",
              "video": {
                  "base64": "UklGRkacAABXRUJQVlA4IDqcAACQggKdASqpAn8B.....",
                  "start_offset": 0,
                  "end_offset": 10,
                  "interval": 5,
                  # "url": "gcs://..." # if you want to use a url
              }
          }
      ]
)

import { Portkey } from 'portkey-ai';

const portkey = new Portkey({
    apiKey: "YOUR_API_KEY",
    provider:"@YOUR_PROVIDER"
});

const embedding = await portkey.embeddings.create({
    input: [
          {
              "text": "this is the caption of the video",
              "video": {
                  "base64": "UklGRkacAABXRUJQVlA4IDqcAACQggKdASqpAn8B.....",
                  "start_offset": 0,
                  "end_offset": 10,
                  "interval": 5,
                  // "url": "gcs://..." // if you want to use a url
              }
          }
      ],
    model: 'multimodalembedding@001'
});

console.log(embedding);

curl --location 'https://api.portkey.ai/v1/embeddings' \
--header 'Content-Type: application/json' \
--header 'x-portkey-api-key: PORTKEY_API_KEY' \
--header 'x-portkey-provider: PORTKEY_PROVIDER' \
--data-raw '{
    "model": "multimodalembedding@001",
    "input": [
                  {
                      "text": "this is the caption of the video",
                      "video": {
                          "base64": "UklGRkacAABXRUJQVlA4IDqcAACQggKdASqpAn8B.....",
                          "start_offset": 0,
                          "end_offset": 10,
                          "interval": 5
                          # "url": "gcs://..." # if you want to use a url
                      }
                  }
              ]
}'

from openai import OpenAI
from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders

portkey_client = OpenAI(
    api_key='NOT_REQUIRED',
    base_url=PORTKEY_GATEWAY_URL,
    default_headers=createHeaders(
        provider="openai",
        api_key="PORTKEY_API_KEY"
    )
)

embeddings = portkey_client.embeddings.create(
  model="multimodalembedding@001",
  input=[
          {
              "text": "this is the caption of the video",
              "video": {
                  "base64": "UklGRkacAABXRUJQVlA4IDqcAACQggKdASqpAn8B.....",
                  "start_offset": 0,
                  "end_offset": 10,
                  "interval": 5,
                  # "url": "gcs://..." # if you want to use a url
              }
          }
        ]
)

import OpenAI from 'openai'; // We're using the v4 SDK
import { PORTKEY_GATEWAY_URL, createHeaders } from 'portkey-ai'

const portkeyClient = new OpenAI({
  apiKey: 'NOT_REQUIRED', // defaults to process.env["OPENAI_API_KEY"],
  baseURL: PORTKEY_GATEWAY_URL,
  defaultHeaders: createHeaders({
    apiKey: "PORTKEY_API_KEY", // defaults to process.env["PORTKEY_API_KEY"]
    provider:"@PORTKEY_PROVIDER"
  })
});

const embedding = await portkeyClient.embeddings.create({
    input: [
              {
                  "text": "this is the caption of the video",
                  "video": {
                      "base64": "UklGRkacAABXRUJQVlA4IDqcAACQggKdASqpAn8B.....",
                      "start_offset": 0,
                      "end_offset": 10,
                      "interval": 5,
                      // "url": "gcs://..." // if you want to use a url
                  }
              }
            ],
    model: 'multimodalembedding@001'
});

console.log(embedding);

Ecosystem

LLM Integrations

Cloud Platforms

Guardrails

Plugins

Vector Databases

Agents

AI Apps

Libraries

Tracing Providers

MCP Clients

MCP Servers

Gemini Embedding Models

Embedding Text

Embedding Images

Embedding Videos

Embedding Audio

Multimodal Embedding (Mixed Inputs)

Setting Task Type and Dimensions

Legacy Embedding Models

Embedding Text

Embeddings Images

Embeddings Videos

​Gemini Embedding Models

​Embedding Text

​Embedding Images

​Embedding Videos

​Embedding Audio

​Multimodal Embedding (Mixed Inputs)

​Setting Task Type and Dimensions

​Legacy Embedding Models

​Embedding Text

​Embeddings Images

​Embeddings Videos

Gemini Embedding Models

Embedding Text

Embedding Images

Embedding Videos

Embedding Audio

Multimodal Embedding (Mixed Inputs)

Setting Task Type and Dimensions

Legacy Embedding Models

Embedding Text

Embeddings Images

Embeddings Videos