Weaviate#

One can use Weaviate as the document store for DocumentArray. It is useful when one wants to have faster Document retrieval on embeddings, i.e. .match(), .find().

Tip

This feature requires weaviate-client. You can install it via pip install "docarray[weaviate]".

Here is a video tutorial that guides you to build a simple image search using Weaviate and Docarray.

Usage#

Start Weaviate service#

To use Weaviate as the storage backend, it is required to have the Weaviate service started. Create docker-compose.yml as follows:

---
version: '3.4'
services:
  weaviate:
    command:
      - --host
      - 0.0.0.0
      - --port
      - '8080'
      - --scheme
      - http
    image: semitechnologies/weaviate:1.11.0
    ports:
      - "8080:8080"
    restart: on-failure:0
    environment:
      QUERY_DEFAULTS_LIMIT: 25
      AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
      PERSISTENCE_DATA_PATH: '/var/lib/weaviate'
      DEFAULT_VECTORIZER_MODULE: 'none'
      ENABLE_MODULES: ''
      CLUSTER_HOSTNAME: 'node1'
...

Then

docker compose up

Create DocumentArray with Weaviate backend#

Assuming service is started using the default configuration (i.e. server address is http://localhost:8080), one can instantiate a DocumentArray with Weaviate storage as such:

from docarray import DocumentArray

da = DocumentArray(storage='weaviate')

The usage would be the same as the ordinary DocumentArray.

To access a DocumentArray formerly persisted, one can specify the name, the host, the port and the protocol to connect to the server. name is required in this case but other connection parameters are optional. If they are not provided, then it will connect to the Weaviate service bound to http://localhost:8080.

Note, that the name parameter in config needs to be capitalized.

from docarray import DocumentArray

da = DocumentArray(
    storage='weaviate', config={'name': 'Persisted', 'host': 'localhost', 'port': 1234}
)

da.summary()

Other functions behave the same as in-memory DocumentArray.

Config#

The following configs can be set:

Name Description Default
host Hostname of the Weaviate server 'localhost'
port port of the Weaviate server 8080
protocol protocol to be used. Can be 'http' or 'https' 'http'
name Weaviate class name; the class name of Weaviate object to presesent this DocumentArray None
serialize_config Serialization config of each Document None
ef The size of the dynamic list for the nearest neighbors (used during the search). The higher ef is chosen, the more accurate, but also slower a search becomes. None, defaults to the default value in Weaviate*
ef_construction The size of the dynamic list for the nearest neighbors (used during the construction). Controls index search speed/build speed tradeoff. None, defaults to the default value in Weaviate*
timeout_config Set the timeout configuration for all requests to the Weaviate server. None, defaults to the default value in Weaviate*
max_connections The maximum number of connections per element in all layers. None, defaults to the default value in Weaviate*
dynamic_ef_min If using dynamic ef (set to -1), this value acts as a lower boundary. Even if the limit is small enough to suggest a lower value, ef will never drop below this value. This helps in keeping search accuracy high even when setting very low limits, such as 1, 2, or 3. None, defaults to the default value in Weaviate*
dynamic_ef_max If using dynamic ef (set to -1), this value acts as an upper boundary. Even if the limit is large enough to suggest a lower value, ef will be capped at this value. This helps to keep search speed reasonable when retrieving massive search result sets, e.g. 500+. None, defaults to the default value in Weaviate*
dynamic_ef_factor If using dynamic ef (set to -1), this value controls how ef is determined based on the given limit. E.g. with a factor of 8, ef will be set to 8*limit as long as this value is between the lower and upper boundary. It will be capped on either end, otherwise. None, defaults to the default value in Weaviate*
vector_cache_max_objects For optimal search and import performance all previously imported vectors need to be held in memory. However, Weaviate also allows for limiting the number of vectors in memory. By default, when creating a new class, this limit is set to 2M objects. A disk lookup for a vector is orders of magnitudes slower than memory lookup, so the cache should be used sparingly. None, defaults to the default value in Weaviate*
flat_search_cutoff Absolute number of objects configured as the threshold for a flat-search cutoff. If a filter on a filtered vector search matches fewer than the specified elements, the HNSW index is bypassed entirely and a flat (brute-force) search is performed instead. This can speed up queries with very restrictive filters considerably. Optional, defaults to 40000. Set to 0 to turn off flat-search cutoff entirely. None, defaults to the default value in Weaviate*
cleanup_interval_seconds How often the async process runs that “repairs” the HNSW graph after deletes and updates. (Prior to the repair/cleanup process, deleted objects are simply marked as deleted, but still a fully connected member of the HNSW graph. After the repair has run, the edges are reassigned and the datapoints deleted for good). Typically this value does not need to be adjusted, but if deletes or updates are very frequent it might make sense to adjust the value up or down. (Higher value means it runs less frequently, but cleans up more in a single batch. Lower value means it runs more frequently, but might not be as efficient with each run). None, defaults to the default value in Weaviate*
skip There are situations where it doesn’t make sense to vectorize a class. For example if the class is just meant as glue between two other class (consisting only of references) or if the class contains mostly duplicate elements (Note that importing duplicate vectors into HNSW is very expensive as the algorithm uses a check whether a candidate’s distance is higher than the worst candidate’s distance for an early exit condition. With (mostly) identical vectors, this early exit condition is never met leading to an exhaustive search on each import or query). In this case, you can skip indexing a vector all-together. To do so, set "skip" to "true". skip defaults to false; if not set to true, classes will be indexed normally. This setting is immutable after class initialization. None, defaults to the default value in Weaviate*

*You can read more about the HNSW parameters and their default values here

Minimum Example#

The following example shows how to use DocArray with Weaviate Document Store in order to index and search text Documents.

First, let’s run the create the DocumentArray instance (make sure a Weaviate server is up and running):

from docarray import DocumentArray

da = DocumentArray(
    storage="weaviate", config={"name": "Persisted", "host": "localhost", "port": 8080}
)

Then, we can index some Documents:

from docarray import Document

da.extend(
    [
        Document(text='Persist Documents with Weaviate.'),
        Document(text='And enjoy fast nearest neighbor search.'),
        Document(text='All while using DocArray API.'),
    ]
)

Now, we can generate embeddings inside the database using BERT model:

from transformers import AutoModel, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
model = AutoModel.from_pretrained('bert-base-uncased')


def collate_fn(da):
    return tokenizer(da.texts, return_tensors='pt', truncation=True, padding=True)


da.embed(model, collate_fn=collate_fn)

Finally, we can query the database and print the results:

results = da.find(
    DocumentArray([Document(text='How to persist Documents')]).embed(
        model, collate_fn=collate_fn
    ),
    limit=1,
)

print(results[0].text)
Persist Documents with Weaviate.