Build Smarter AI Pipelines with Typed, Multi-Dimensional Vectors

CocoIndex has introduced comprehensive support for typed vector data, encompassing everything from basic numeric arrays to complex, deeply nested multi-dimensional vectors. This enhancement is designed to facilitate seamless integration with high-performance vector databases such as Qdrant, enabling advanced indexing, embedding, and retrieval workflows across diverse data modalities.

The platform now accommodates a range of vector types with robust typing guarantees. While CocoIndex can automatically infer types for most data flows, explicit type specification is available for custom functions, providing greater control.

Supported Vector Types

1. One-dimensional vectors: These represent simple sequences of numbers. CocoIndex supports dynamic dimensions, where the vector length can vary, or fixed dimensions, such as Vector[Float32, Literal[384]] for a vector always containing 384 elements. Internally, these can correspond to NumPy arrays (numpy.typing.NDArray[np.float32]) or Python lists (list[float]). Supported number types include native Python float and int, CocoIndex type aliases (Float32, Float64, Int64 for precise bit-size control), and NumPy types (numpy.float32, numpy.float64, numpy.int64).

2. Two-dimensional vectors (Multi-vectors): Represented as Vector[Vector[Float32, Literal[3]], Literal[2]], these signify a vector composed of other vectors—for instance, two vectors, each with three elements. This structure is ideal for scenarios requiring multiple embeddings per data point, multi-view representations, or batched encodings. Their underlying Python representation is typically a nested list (list[list[float]]) or a multi-dimensional NumPy array (numpy.typing.NDArray).

Understanding Multi-Dimensional Vectors

A multi-dimensional vector is fundamentally a vector whose elements are themselves vectors, akin to a matrix or a nested list. CocoIndex represents this structure using nested Vector types, such as Vector[Vector[T, N], M], where M denotes the number of inner vectors, each of dimension N. While M and N can be dynamic within CocoIndex, certain target systems like Qdrant may require a fixed inner dimension (N). This concept is vital in deep learning and multimodal applications, allowing for richer data representations, such as an image as a collection of patch-level embeddings, a document as paragraph-level vectors, or a user session as a sequence of behavioral vectors. CocoIndex supports these complex representations natively, avoiding the need for flattening.

Applications of Multi-Vector Embeddings

Multi-vector embeddings are particularly beneficial in several advanced use cases:

• Vision: Patch Embeddings: In computer vision, a Vision Transformer (ViT) might output multiple vectors for an image, one for each patch (e.g., 196 patches, each with a 768-dimensional embedding). This enables local feature matching and region-aware retrieval, represented as Vector[Vector[Float32, Literal[768]]].
• Text: Document with Paragraphs: For long documents, splitting into paragraph or sentence-level vectors (e.g., 10 paragraph vectors of 384 dimensions each) preserves context and structure better than averaging all vectors. This allows for searching within sub-parts of a document, retrieving documents based on a single relevant paragraph, and implementing hierarchical search. This can be typed as Vector[Vector[Float32, Literal[384]]].
• User Behavior: Sessions or Time Series: A user session can be modeled as a sequence of actions or states, where each step is a vector (e.g., 20 steps, each a 16-dimensional vector). This is valuable in e-commerce click sequences, financial time series, or UX analytics for multi-step interactions, represented as Vector[Vector[Float32, Literal[16]]].
• Scientific & Biomedical: Multi-view Embeddings: Complex entities like molecules or proteins might have multiple conformations or measurement modalities. Representing these as multiple vectors (e.g., 5 different views, each 128-dimensional) allows for comprehensive comparisons across their latent spaces, typed as Vector[Vector[Float32, Literal[128]]].

Benefits Over Flat Vectors

While it's possible to flatten a multi-vector (e.g., converting [[1,2,3],[4,5,6]] to [1,2,3,4,5,6]), doing so results in the loss of semantic boundaries. True multi-vectors preserve the meaning of each sub-vector, enabling matching and querying at the sub-vector level. Vector databases like Qdrant can leverage this by scoring across multiple embeddings, identifying the best match from a set of related vectors (e.g., the closest image patch or relevant paragraph). Crucially, the inner vector can maintain a fixed dimension even if the outer dimension varies, a common requirement for efficient vector indexing in most databases.

CocoIndex to Qdrant Type Mapping

Qdrant, a widely used vector database, supports both dense vectors and multi-vectors. CocoIndex intelligently maps its vector types to Qdrant-compatible formats:

• A CocoIndex Vector[Float32, Literal[N]] is mapped to a Qdrant Dense Vector.
• A CocoIndex Vector[Vector[Float32, Literal[N]]] is mapped to a Qdrant MultiVector.
• Any other vector type, or vectors with dynamic dimensions unsupported by Qdrant, are stored as part of Qdrant’s JSON payload.

Qdrant specifically requires vectors to have a fixed dimension for indexing. CocoIndex automatically detects vector shapes and manages this mapping, ensuring compatibility.

CocoIndex Data Model Overview

Within CocoIndex, data is organized into rows and fields. Each row corresponds to a Qdrant point. A field can either be a named vector, if it adheres to Qdrant’s vector constraints, or it can be part of the payload for non-conforming types. This hybrid approach offers the flexibility of structured metadata combined with the power of vector search.

By supporting deeply typed vectors and multi-dimensional embeddings, CocoIndex provides structure and semantic clarity to vector data pipelines. Its typing system ensures compatibility, debuggability, and correctness at scale, whether indexing images, text, audio, or abstract graph representations.