docarray.array.memory module#

class docarray.array.memory.DocumentArrayInMemory(*args, storage='memory', **kwargs)[source]#

Bases: StorageMixins, DocumentArray

Default DocumentArray that stores Documents in memory. With this implementation, match() and find() perform exact (exhaustive) vector search.

Example usage:

from docarray import Document, DocumentArray

da = DocumentArray(
    [Document(text='The cake is a lie'), Document(text='Do a barrel roll!')]
)
da.apply(Document.embed_feature_hashing)

query = Document(text='Can i have some cake?').embed_feature_hashing()
query.match(da, metric='jaccard', use_scipy=True)

print(query.matches[:, ('text', 'scores__jaccard__value')])

[['The cake is a lie', 'Do a barrel roll!'], [0.9, 1.0]]

See also

For further details, see our user guide.

append(value, **kwargs)#

Append doc to the end of the array.

Parameters:: value (Document) – The doc needs to be appended.

apply(*args, **kwargs)#: # noqa: DAR102 # noqa: DAR101 # noqa: DAR201 :rtype: T :return: a new DocumentArray

apply_batch(*args, **kwargs)#: # noqa: DAR102 # noqa: DAR101 # noqa: DAR201 :rtype: T :return: a new DocumentArray

batch(batch_size, shuffle=False)#

Creates a Generator that yields DocumentArray of size batch_size until docs is fully traversed along the traversal_path. The None docs are filtered out and optionally the docs can be filtered by checking for the existence of a Document attribute. Note, that the last batch might be smaller than batch_size.

Parameters:

batch_size (int) – Size of each generated batch (except the last one, which might be smaller, default: 32)
shuffle (bool) – If set, shuffle the Documents before dividing into minibatches.

Yield:

a Generator of DocumentArray, each in the length of batch_size

Return type:

Generator[DocumentArray, None, None]

batch_ids(batch_size, shuffle=False)#

Creates a Generator that yields lists of ids of size batch_size until self is fully traversed. Note, that the last batch might be smaller than batch_size.

Parameters:

batch_size (int) – Size of each generated batch (except the last one, which might be smaller)
shuffle (bool) – If set, shuffle the Documents before dividing into minibatches.

Yield:

a Generator of list of IDs, each in the length of batch_size

Return type:

Generator[List[str], None, None]

property blobs: Optional[List[bytes]]#

Get the blob attribute of all Documents.

Return type:: Optional[List[bytes]]
Returns:: a list of blobs

clear()#: Clear the data of DocumentArray

static cloud_delete(name)#

Delete a DocumentArray from the cloud. :type name: str :param name: the name of the DocumentArray to delete.

Return type:: None

static cloud_list(show_table=False)#

List all available arrays in the cloud.

Parameters:: show_table (bool) – if true, show the table of the arrays.
Return type:: List[str]
Returns:: List of available DocumentArray’s names.

classmethod cloud_pull(cls, name, show_progress=False, local_cache=True, *args, **kwargs)#

Pulling a DocumentArray from Jina Cloud Service to local.

Parameters:

name (str) – the upload name set during push()
show_progress (bool) – if to show a progress bar on pulling
local_cache (bool) – store the downloaded DocumentArray to local folder

Return type:

Returns:

a DocumentArray object

cloud_push(name, show_progress=False, public=True, branding=None)#

Push this DocumentArray object to Jina Cloud which can be later retrieved via push()

Note

Push with the same name will override the existing content.
Kinda like a public clipboard where everyone can override anyone’s content. So to make your content survive longer, you may want to use longer & more complicated name.
The lifetime of the content is not promised atm, could be a day, could be a week. Do not use it for persistence. Only use this full temporary transmission/storage/clipboard.

Parameters:

name (str) – a name that later can be used for retrieve this DocumentArray.
show_progress (bool) – if to show a progress bar on pulling
public (bool) – by default anyone can pull a DocumentArray if they know its name. Setting this to False will allow only the creator to pull it. This feature of course you to login first.
branding (Optional[Dict]) – a dict of branding information to be sent to Jina Cloud. {“icon”: “emoji”, “background”: “#fff”}

Return type:

Dict

property contents: Optional[Union[Sequence[DocumentContentType], ArrayType]]#

Get the content of all Documents.

Return type:: Union[Sequence[DocumentContentType], ArrayType, None]
Returns:: a list of texts, blobs or ArrayType

count(value) → integer -- return number of occurrences of value#

classmethod dataloader(path, func, batch_size, protocol='protobuf', compress=None, backend='thread', num_worker=None, pool=None, show_progress=False)#

Load array elements, batches and maps them with a function in parallel, finally yield the batch in DocumentArray

Parameters:

path (Union[str, Path]) – Path or filename where the data is stored.
func (Callable[[DocumentArray], T]) – a function that takes DocumentArray as input and outputs anything. You can either modify elements in-place (only with thread backend) or work later on return elements.
batch_size (int) – Size of each generated batch (except the last one, which might be smaller)
protocol (str) – protocol to use
compress (Optional[str]) – compress algorithm to use
backend (str) –
if to use multi-process or multi-thread as the parallelization backend. In general, if your func is IO-bound then perhaps thread is good enough. If your func is CPU-bound then you may use process. In practice, you should try yourselves to figure out the best value. However, if you wish to modify the elements in-place, regardless of IO/CPU-bound, you should always use thread backend.

Warning

When using process backend, you should not expect func modify elements in-place. This is because the multiprocessing backing pass the variable via pickle and work in another process. The passed object and the original object do not share the same memory.
num_worker (Optional[int]) – the number of parallel workers. If not given, then the number of CPUs in the system will be used.
pool (Union[Pool, ThreadPool, None]) – use an existing/external pool. If given, backend is ignored and you will be responsible for closing the pool.
show_progress (bool) – if set, show a progressbar

Return type:

Generator[DocumentArray, None, None]

Returns:

embed(embed_model, device='cpu', batch_size=256, to_numpy=False, collate_fn=None)#

Fill embedding of Documents inplace by using embed_model

Parameters:

embed_model (AnyDNN) – the embedding model written in Keras/Pytorch/Paddle
device (str) – the computational device for embed_model, can be either cpu or cuda.
batch_size (int) – number of Documents in a batch for embedding
to_numpy (bool) – if to store embeddings back to Document in numpy.ndarray or original framework format.
collate_fn (Optional[CollateFnType]) – create a mini-batch of Input(s) from the given DocumentArray. Default built-in collate_fn is to use the tensors of the documents.

Return type:

Returns:

itself after modified.

property embeddings: Optional[ArrayType]#

Return a ArrayType stacking all the embedding attributes as rows.

Return type:: Optional[ArrayType]
Returns:: a ArrayType of embedding

classmethod empty(size=0, *args, **kwargs)#

Create a DocumentArray object with size empty Document objects.

Parameters:: size (int) – the number of empty Documents in this container
Return type:: T
Returns:: a DocumentArray object

evaluate(other, metric, hash_fn=None, metric_name=None, strict=True, **kwargs)#

Compute ranking evaluation metrics for a given DocumentArray when compared with a groundtruth.

This implementation expects to provide a groundtruth DocumentArray that is structurally identical to self. It is based on comparing the matches of documents inside the `DocumentArray.

This method will fill the evaluations field of Documents inside this DocumentArray and will return the average of the computations

Parameters:

other (DocumentArray) – The groundtruth DocumentArray` that the DocumentArray compares to.
metric (Union[str, Callable[..., float]]) – The name of the metric, or multiple metrics to be computed
hash_fn (Optional[Callable[[Document], str]]) – The function used for identifying the uniqueness of Documents. If not given, then Document.id is used.
metric_name (Optional[str]) – If provided, the results of the metrics computation will be stored in the evaluations field of each Document. If not provided, the name will be computed based on the metrics name.
strict (bool) – If set, then left and right sides are required to be fully aligned: on the length, and on the semantic of length. These are preventing you to evaluate on irrelevant matches accidentally.
kwargs – Additional keyword arguments to be passed to metric_fn

Return type:

Optional[float]

Returns:

The average evaluation computed or a list of them if multiple metrics are required

extend(values, **kwargs)#

S.extend(iterable) – extend sequence by appending elements from the iterable

Return type:: None

find(query=None, metric='cosine', limit=20, metric_name=None, exclude_self=False, filter=None, only_id=False, index='text', on=None, **kwargs)#

Returns matching Documents given an input query. If the query is a DocumentArray, Document or ArrayType, exhaustive or approximate nearest neighbor search will be performed depending on whether the storage backend supports ANN. Furthermore, if filter is not None, pre-filtering will be applied along with vector search. If the query is a dict object or, query is None and filter is not None, Documents will be filtered and all matching Documents that match the filter will be returned. In this case, query (if it’s dict) or filter will be used for filtering. The object must follow the backend-specific filter format if the backend supports filtering or DocArray’s query language format. In the latter case, filtering will be applied in the client side not the backend side. If the query is a string or list of strings, a search by text will be performed if the backend supports indexing and searching text fields. If not, a NotImplementedError will be raised.

Parameters:

query (Union[DocumentArray, Document, ArrayType, Dict, str, List[str], None]) – the input query to search by
limit (Union[int, float, None]) – the maximum number of matches, when not given defaults to 20.
metric_name (Optional[str]) – if provided, then match result will be marked with this string.
metric (Union[str, Callable[[ArrayType, ArrayType], ndarray]]) – the distance metric.
exclude_self (bool) – if set, Documents in results with same id as the query values will not be considered as matches. This is only applied when the input query is Document or DocumentArray.
filter (Optional[Dict]) – filter query used for pre-filtering or filtering
only_id (bool) – if set, then returning matches will only contain id
index (str) – if the query is a string, text search will be performed on the index field, otherwise, this parameter is ignored. By default, the Document text attribute will be used for search, otherwise the tag field specified by index will be used. You can only use this parameter if the storage backend supports searching by text.
on (Optional[str]) – specifies a subindex to search on. If set, the returned DocumentArray will be retrieved from the given subindex.
kwargs – other kwargs.

Return type:

Union[DocumentArray, List[DocumentArray]]

Returns:

a list of DocumentArrays containing the closest Document objects for each of the queries in query.

flatten()#

Flatten all nested chunks and matches into one DocumentArray.

Note

Flatten an already flattened DocumentArray will have no effect.

Return type:: DocumentArray
Returns:: a flattened DocumentArray object.

classmethod from_base64(data, protocol='pickle-array', compress=None, _show_progress=False, *args, **kwargs)#

Return type:: T

classmethod from_bytes(data, protocol='pickle-array', compress=None, _show_progress=False, *args, **kwargs)#

Return type:: T

classmethod from_csv(*args, **kwargs)#

# noqa: DAR101 # noqa: DAR102 # noqa: DAR201

Return type:: T

classmethod from_dataframe(df, *args, **kwargs)#

Import a DocumentArray from a pandas.DataFrame object.

Parameters:: df (DataFrame) – a pandas.DataFrame object.
Return type:: T
Returns:: a DocumentArray object

classmethod from_dict(values, protocol='jsonschema', **kwargs)#

Return type:: T

classmethod from_files(*args, **kwargs)#

# noqa: DAR101 # noqa: DAR102 # noqa: DAR201

Return type:: T

classmethod from_huggingface_datasets(*args, **kwargs)#

# noqa: DAR101 # noqa: DAR102 # noqa: DAR201

Return type:: T

classmethod from_json(file, protocol='jsonschema', **kwargs)#

Return type:: T

classmethod from_lines(*args, **kwargs)#

# noqa: DAR101 # noqa: DAR102 # noqa: DAR201

Return type:: T

classmethod from_list(values, protocol='jsonschema', **kwargs)#

Return type:: T

classmethod from_ndarray(*args, **kwargs)#

# noqa: DAR101 # noqa: DAR102 # noqa: DAR201

Return type:: T

classmethod from_ndjson(*args, **kwargs)#

# noqa: DAR101 # noqa: DAR102 # noqa: DAR201

Return type:: T

classmethod from_protobuf(pb_msg)#

Return type:: T

classmethod from_pydantic_model(model)#

Convert a list of PydanticDocument into DocumentArray

Parameters:: model (List[BaseModel]) – the list of pydantic data model objects that represents a DocumentArray
Return type:: T
Returns:: a DocumentArray

classmethod from_strawberry_type(model)#

Convert a list of Strawberry into DocumentArray

Parameters:: model (List[StrawberryDocument]) – the list of strawberry type objects that represents a DocumentArray
Return type:: T
Returns:: a DocumentArray

classmethod get_json_schema(indent=2)#

Return a JSON Schema of DocumentArray class.

Return type:: str

get_vocabulary(min_freq=1, text_attrs=('text',))#

Get the text vocabulary in a dict that maps from the word to the index from all Documents.

Parameters:

text_attrs (Tuple[str, ...]) – the textual attributes where vocabulary will be derived from
min_freq (int) – the minimum word frequency to be considered into the vocabulary.

Return type:

Dict[str, int]

Returns:

a vocabulary in dictionary where key is the word, value is the index. The value is 2-index, where 0 is reserved for padding, 1 is reserved for unknown token.

index(value[, start[, stop]]) → integer -- return first index of value.#

Raises ValueError if the value is not present.

Supporting start and stop arguments is optional, but recommended.

insert(index, value)#

Insert doc at index.

Parameters:

index (int) – Position of the insertion.
value (Document) – The doc needs to be inserted.

classmethod load(file, file_format='binary', encoding='utf-8', **kwargs)#

Load array elements from a JSON or a binary file, or a CSV file.

Parameters:

file (Union[str, TextIO, BinaryIO]) – File or filename to which the data is saved.
file_format (str) – json or binary or csv. JSON and CSV files are human-readable, but binary format gives much smaller size and faster save/load speed. CSV file has very limited compatability, complex DocumentArray with nested structure can not be restored from a CSV file.
encoding (str) – encoding used to load data from a file (it only applies to JSON and CSV format). By default, utf-8 is used.

Return type:

Returns:

the loaded DocumentArray object

classmethod load_binary(file, protocol='pickle-array', compress=None, _show_progress=False, streaming=False, *args, **kwargs)#

Load array elements from a compressed binary file.

Parameters:

file (Union[str, BinaryIO, bytes, Path]) – File or filename or serialized bytes where the data is stored.
protocol (str) – protocol to use
compress (Optional[str]) – compress algorithm to use
_show_progress (bool) – show progress bar, only works when protocol is pickle or protobuf
streaming (bool) – if True returns a generator over Document objects.

In case protocol is pickle the Documents are streamed from disk to save memory usage :rtype: Union[DocumentArray, Generator[Document, None, None]] :return: a DocumentArray object

Note

If file is str it can specify protocol and compress as file extensions. This functionality assumes file=file_name.$protocol.$compress where $protocol and $compress refer to a string interpolation of the respective protocol and compress methods. For example if file=my_docarray.protobuf.lz4 then the binary data will be loaded assuming protocol=protobuf and compress=lz4.

classmethod load_csv(file, field_resolver=None, encoding='utf-8')#

Load array elements from a binary file.

Parameters:

file (Union[str, TextIO]) – File or filename to which the data is saved.
field_resolver (Optional[Dict[str, str]]) – a map from field names defined in JSON, dict to the field names defined in Document.
encoding (str) – encoding used to read a CSV file. By default, utf-8 is used.

Return type:

Returns:

a DocumentArray object

classmethod load_json(file, protocol='jsonschema', encoding='utf-8', **kwargs)#

Load array elements from a JSON file.

Parameters:

file (Union[str, TextIO]) – File or filename or a JSON string to which the data is saved.
protocol (str) – jsonschema or protobuf
encoding (str) – encoding used to load data from a JSON file. By default, utf-8 is used.

Return type:

Returns:

a DocumentArrayLike object

map(func, backend='thread', num_worker=None, show_progress=False, pool=None)#

Return an iterator that applies function to every element of iterable in parallel, yielding the results.

See also

To process single element, please use map();
To return DocumentArray, please use apply_batch().

Parameters:

batch_size (int) – Size of each generated batch (except the last one, which might be smaller, default: 32)
shuffle (bool) – If set, shuffle the Documents before dividing into minibatches.
func (Callable[[DocumentArray], T]) – a function that takes DocumentArray as input and outputs anything. You can either modify elements in-place (only with thread backend) or work later on return elements.
backend (str) –
if to use multi-process or multi-thread as the parallelization backend. In general, if your func is IO-bound then perhaps thread is good enough. If your func is CPU-bound then you may use process. In practice, you should try yourselves to figure out the best value. However, if you wish to modify the elements in-place, regardless of IO/CPU-bound, you should always use thread backend.

Warning

When using process backend, you should not expect func modify elements in-place. This is because the multiprocessing backing pass the variable via pickle and work in another process. The passed object and the original object do not share the same memory.
num_worker (Optional[int]) – the number of parallel workers. If not given, then the number of CPUs in the system will be used.
show_progress (bool) – show a progress bar
pool (Union[Pool, ThreadPool, None]) – use an existing/external pool. If given, backend is ignored and you will be responsible for closing the pool.

Yield:

anything return from func

Return type:

Generator[T, None, None]

match(darray, metric='cosine', limit=20, normalization=None, metric_name=None, batch_size=None, exclude_self=False, filter=None, only_id=False, use_scipy=False, device='cpu', num_worker=1, on=None, **kwargs)#

Compute embedding based nearest neighbour in another for each Document in self, and store results in matches. .. note:

'cosine', 'euclidean', 'sqeuclidean' are supported natively without extra dependency.
You can use other distance metric provided by ``scipy``, such as `braycurtis`, `canberra`, `chebyshev`,
`cityblock`, `correlation`, `cosine`, `dice`, `euclidean`, `hamming`, `jaccard`, `jensenshannon`,
`kulsinski`, `mahalanobis`, `matching`, `minkowski`, `rogerstanimoto`, `russellrao`, `seuclidean`,
`sokalmichener`, `sokalsneath`, `sqeuclidean`, `wminkowski`, `yule`.
To use scipy metric, please set ``use_scipy=True``.

To make all matches values in [0, 1], use dA.match(dB, normalization=(0, 1))
To invert the distance as score and make all values in range [0, 1],
use dA.match(dB, normalization=(1, 0)). Note, how normalization differs from the previous.
If a custom metric distance is provided. Make sure that it returns scores as distances and not similarity, meaning the smaller the better.

Parameters:

darray (DocumentArray) – the other DocumentArray to match against
metric (Union[str, Callable[[ArrayType, ArrayType], ndarray]]) – the distance metric
limit (Union[int, float, None]) – the maximum number of matches, when not given defaults to 20.
normalization (Optional[Tuple[float, float]]) – a tuple [a, b] to be used with min-max normalization, the min distance will be rescaled to a, the max distance will be rescaled to b all values will be rescaled into range [a, b].
metric_name (Optional[str]) – if provided, then match result will be marked with this string.
batch_size (Optional[int]) – if provided, then darray is loaded in batches, where each of them is at most batch_size elements. When darray is big, this can significantly speedup the computation.
exclude_self (bool) – if set, Documents in darray with same id as the left-hand values will not be considered as matches.
filter (Optional[Dict]) – filter query used for pre-filtering
only_id (bool) – if set, then returning matches will only contain id
use_scipy (bool) – if set, use scipy as the computation backend. Note, scipy does not support distance on sparse matrix.
device (str) – the computational device for .match(), can be either cpu or cuda.
num_worker (Optional[int]) –
the number of parallel workers. If not given, then the number of CPUs in the system will be used.

Note

This argument is only effective when batch_size is set.
on (Optional[str]) – specifies a subindex to search on. If set, the returned DocumentArray will be retrieved from the given subindex.
kwargs – other kwargs.

Return type:

None

plot_embeddings(title='MyDocumentArray', path=None, image_sprites=False, min_image_size=16, channel_axis=-1, start_server=True, host='127.0.0.1', port=None, image_source='tensor')#

Interactively visualize embeddings using the Embedding Projector.

Parameters:

title (str) – the title of this visualization. If you want to compare multiple embeddings at the same time, make sure to give different names each time and set path to the same value.
host (str) – if set, bind the embedding-projector frontend to given host. Otherwise localhost is used.
port (Optional[int]) – if set, run the embedding-projector frontend at given port. Otherwise a random port is used.
image_sprites (bool) – if set, visualize the dots using uri and tensor.
path (Optional[str]) – if set, then append the visualization to an existing folder, where you can compare multiple embeddings at the same time. Make sure to use a different title each time .
min_image_size (int) – only used when image_sprites=True. the minimum size of the image
channel_axis (int) – only used when image_sprites=True. the axis id of the color channel, -1 indicates the color channel info at the last axis
start_server (bool) – if set, start a HTTP server and open the frontend directly. Otherwise, you need to rely on return path and serve by yourself.
image_source (str) – specify where the image comes from, can be uri or tensor. empty tensor will fallback to uri

Return type:

str

Returns:

the path to the embeddings visualization info.

plot_image_sprites(output=None, canvas_size=512, min_size=16, channel_axis=-1, image_source='tensor', skip_empty=False, show_progress=False, show_index=False, fig_size=(10, 10), keep_aspect_ratio=False)#

Generate a sprite image for all image tensors in this DocumentArray-like object.

An image sprite is a collection of images put into a single image. It is always square-sized. Each sub-image is also square-sized and equally-sized.

Parameters:

output (Optional[str]) – Optional path to store the visualization. If not given, show in UI
canvas_size (int) – the size of the canvas
min_size (int) – the minimum size of the image
channel_axis (int) – the axis id of the color channel, -1 indicates the color channel info at the last axis
image_source (str) – specify where the image comes from, can be uri or tensor. empty tensor will fallback to uri
skip_empty (bool) – skip Document who has no .uri or .tensor.
show_index (bool) – show the index on the top-right corner of every image
fig_size (Optional[Tuple[int, int]]) – the size of the figure
show_progress (bool) – show a progressbar while plotting.
keep_aspect_ratio (bool) – preserve the aspect ratio of the image by using the aspect ratio of the first image in self.

Return type:

None

pop([index]) → item -- remove and return item at index (default last).#: Raise IndexError if list is empty or index is out of range.

post(host, show_progress=False, batch_size=None, parameters=None, **kwargs)#

Posting itself to a remote Flow/Sandbox and get the modified DocumentArray back

Parameters:

host (str) – a host string. Can be one of the following: - grpc://192.168.0.123:8080/endpoint - ws://192.168.0.123:8080/endpoint - http://192.168.0.123:8080/endpoint - jinahub://Hello/endpoint - jinahub+docker://Hello/endpoint - jinahub+docker://Hello/v0.0.1/endpoint - jinahub+docker://Hello/latest/endpoint - jinahub+sandbox://Hello/endpoint
show_progress (bool) – if to show a progressbar
batch_size (Optional[int]) – number of Document on each request
parameters (Optional[Dict]) – parameters to send in the request

Return type:

DocumentArray

Returns:

the new DocumentArray returned from remote

classmethod pull(cls, name, show_progress=False, local_cache=True, *args, **kwargs)#

Pulling a DocumentArray from Jina Cloud Service to local.

Parameters:

name (str) – the upload name set during push()
show_progress (bool) – if to show a progress bar on pulling
local_cache (bool) – store the downloaded DocumentArray to local folder

Return type:

Returns:

a DocumentArray object

push(name, show_progress=False, public=True, branding=None)#

Push this DocumentArray object to Jina Cloud which can be later retrieved via push()

Note

Push with the same name will override the existing content.
Kinda like a public clipboard where everyone can override anyone’s content. So to make your content survive longer, you may want to use longer & more complicated name.
The lifetime of the content is not promised atm, could be a day, could be a week. Do not use it for persistence. Only use this full temporary transmission/storage/clipboard.

Parameters:

name (str) – a name that later can be used for retrieve this DocumentArray.
show_progress (bool) – if to show a progress bar on pulling
public (bool) – by default anyone can pull a DocumentArray if they know its name. Setting this to False will allow only the creator to pull it. This feature of course you to login first.
branding (Optional[Dict]) – a dict of branding information to be sent to Jina Cloud. {“icon”: “emoji”, “background”: “#fff”}

Return type:

Dict

reduce(other)#: Reduces other and the current DocumentArray into one DocumentArray in-place. Changes are applied to the current DocumentArray. Reducing 2 DocumentArrays consists in adding Documents in the second DocumentArray to the first DocumentArray if they do not exist. If a Document exists in both DocumentArrays, the data properties are merged with priority to the first Document (that is, to the current DocumentArray’s Document). The matches and chunks are also reduced in the same way. :type other: T :param other: DocumentArray :rtype: T :return: DocumentArray

reduce_all(others)#

Reduces a list of DocumentArrays and this DocumentArray into one DocumentArray. Changes are applied to this DocumentArray in-place.

Reduction consists in reducing this DocumentArray with every DocumentArray in others sequentially using DocumentArray.:method:reduce. The resulting DocumentArray contains Documents of all DocumentArrays. If a Document exists in many DocumentArrays, data properties are merged with priority to the left-most DocumentArrays (that is, if a data attribute is set in a Document belonging to many DocumentArrays, the attribute value of the left-most DocumentArray is kept). Matches and chunks of a Document belonging to many DocumentArrays are also reduced in the same way. Other non-data properties are ignored.

Note

Matches are not kept in a sorted order when they are reduced. You might want to re-sort them in a later
step.
The final result depends on the order of DocumentArrays when applying reduction.

Parameters:: others (List[T]) – List of DocumentArrays to be reduced
Return type:: T
Returns:: the resulting DocumentArray

remove(value)#: S.remove(value) – remove first occurrence of value. Raise ValueError if the value is not present.

reverse()#: S.reverse() – reverse IN PLACE

sample(k, seed=None)#

random sample k elements from DocumentArray without replacement.

Parameters:

k (int) – Number of elements to sample from the document array.
seed (Optional[int]) – initialize the random number generator, by default is None. If set will save the state of the random function to produce certain outputs.

Return type:

DocumentArray

Returns:

A sampled list of Document represented as DocumentArray.

save(file, file_format='binary', encoding='utf-8')#

Save array elements into a JSON, a binary file or a CSV file.

Parameters:

file (Union[str, TextIO, BinaryIO]) – File or filename to which the data is saved.
file_format (str) – json or binary or csv. JSON and CSV files are human-readable, but binary format gives much smaller size and faster save/load speed. Note that, CSV file has very limited compatability, complex DocumentArray with nested structure can not be restored from a CSV file.
encoding (str) – encoding used to save data into a file (it only applies to JSON and CSV format). By default, utf-8 is used.

Return type:

None

save_binary(file, protocol='pickle-array', compress=None)#

Save array elements into a binary file.

Parameters:

file (Union[str, BinaryIO]) – File or filename to which the data is saved.
protocol (str) – protocol to use
compress (Optional[str]) –
compress algorithm to use

Note

If file is str it can specify protocol and compress as file extensions. This functionality assumes file=file_name.$protocol.$compress where $protocol and $compress refer to a string interpolation of the respective protocol and compress methods. For example if file=my_docarray.protobuf.lz4 then the binary data will be created using protocol=protobuf and compress=lz4.

Comparing to save_json(), it is faster and the file is smaller, but not human-readable.

Note

To get a binary presentation in memory, use bytes(...).

Return type:: None

save_csv(file, flatten_tags=True, exclude_fields=None, dialect='excel', with_header=True, encoding='utf-8')#

Save array elements into a CSV file.

Parameters:

file (Union[str, TextIO]) – File or filename to which the data is saved.
flatten_tags (bool) – if set, then all fields in Document.tags will be flattened into tag__fieldname and stored as separated columns. It is useful when tags contain a lot of information.
exclude_fields (Optional[Sequence[str]]) – if set, those fields wont show up in the output CSV
dialect (Union[str, Dialect]) – define a set of parameters specific to a particular CSV dialect. could be a string that represents predefined dialects in your system, or could be a csv.Dialect class that groups specific formatting parameters together.
encoding (str) – encoding used to save the data into a CSV file. By default, utf-8 is used.

Return type:

None

save_embeddings_csv(file, encoding='utf-8', **kwargs)#

Save embeddings to a CSV file

This function utilizes numpy.savetxt() internal.

Parameters:

file (Union[str, TextIO]) – File or filename to which the data is saved.
encoding (str) – encoding used to save the data into a file. By default, utf-8 is used.
kwargs – extra kwargs will be passed to numpy.savetxt().

Return type:

None

save_gif(output, channel_axis=-1, duration=200, size_ratio=1.0, inline_display=False, image_source='tensor', skip_empty=False, show_index=False, show_progress=False)#

Save a gif of the DocumentArray. Each frame corresponds to a Document.uri/.tensor in the DocumentArray.

Parameters:

output (str) – the file path to save the gif to.
channel_axis (int) – the color channel axis of the tensor.
duration (int) – the duration of each frame in milliseconds.
size_ratio (float) – the size ratio of each frame.
inline_display (bool) – if to show the gif in Jupyter notebook.
image_source (str) – the source of the image in Document atribute.
skip_empty (bool) – if to skip empty documents.
show_index (bool) – if to show the index of the document in the top-right corner.
show_progress (bool) – if to show a progress bar.

Return type:

None

Returns:

save_json(file, protocol='jsonschema', encoding='utf-8', **kwargs)#

Save array elements into a JSON file.

Comparing to save_binary(), it is human-readable but slower to save/load and the file size larger.

Parameters:

file (Union[str, TextIO]) – File or filename to which the data is saved.
protocol (str) – jsonschema or protobuf
encoding (str) – encoding used to save data into a JSON file. By default, utf-8 is used.

Return type:

None

shuffle(seed=None)#

Randomly shuffle documents within the DocumentArray.

Parameters:: seed (Optional[int]) – initialize the random number generator, by default is None. If set will save the state of the random function to produce certain outputs.
Return type:: DocumentArray
Returns:: The shuffled list of Document represented as DocumentArray.

split_by_tag(tag)#

Split the DocumentArray into multiple DocumentArray according to the tag value of each Document.

Parameters:: tag (str) – the tag name to split stored in tags.
Return type:: Dict[Any, DocumentArray]
Returns:: a dict where Documents with the same value on tag are grouped together, their orders are preserved from the original DocumentArray.

Note

If the tags of Document do not contains the specified tag, return an empty dict.

summary()#: Print the structure and attribute summary of this DocumentArray object.

Warning

Calling {meth}`.summary` on large DocumentArray can be slow.

property tensors: Optional[ArrayType]#

Return a ArrayType stacking all tensor.

The tensor attributes are stacked together along a newly created first dimension (as if you would stack using np.stack(X, axis=0)).

Warning

This operation assumes all tensors have the same shape and dtype. All dtype and shape values are assumed to be equal to the values of the first element in the DocumentArray

Return type:: Optional[ArrayType]
Returns:: a ArrayType of tensors

property texts: Optional[List[str]]#

Get text of all Documents

Return type:: Optional[List[str]]
Returns:: a list of texts

to_base64(protocol='pickle-array', compress=None, _show_progress=False)#

Return type:: str

to_bytes(protocol='pickle-array', compress=None, _file_ctx=None, _show_progress=False)#

Serialize itself into bytes.

For more Pythonic code, please use bytes(...).

Parameters:

_file_ctx (Optional[BinaryIO]) – File or filename or serialized bytes where the data is stored.
protocol (str) – protocol to use
compress (Optional[str]) – compress algorithm to use
_show_progress (bool) – show progress bar, only works when protocol is pickle or protobuf

Return type:

bytes

Returns:

the binary serialization in bytes

to_dataframe(**kwargs)#

Export itself to a pandas.DataFrame object.

Parameters:: kwargs – the extra kwargs will be passed to pandas.DataFrame.from_dict().
Return type:: DataFrame
Returns:: a pandas.DataFrame object

to_dict(protocol='jsonschema', **kwargs)#

Convert the object into a Python list.

Parameters:: protocol (str) – jsonschema or protobuf
Return type:: List
Returns:: a Python list

to_json(protocol='jsonschema', **kwargs)#

Convert the object into a JSON string. Can be loaded via load_json().

Parameters:: protocol (str) – jsonschema or protobuf
Return type:: str
Returns:: a Python list

to_list(protocol='jsonschema', **kwargs)#

Convert the object into a Python list.

Parameters:: protocol (str) – jsonschema or protobuf
Return type:: List
Returns:: a Python list

to_protobuf(ndarray_type=None)#

Convert DocumentArray into a Protobuf message.

Parameters:: ndarray_type (Optional[str]) – can be list or numpy, if set it will force all ndarray-like object from all Documents to List or numpy.ndarray.
Return type:: DocumentArrayProto
Returns:: the protobuf message

to_pydantic_model()#

Convert a DocumentArray object into a Pydantic model.

Return type:: PydanticDocumentArray

to_strawberry_type()#

Convert a DocumentArray object into a Pydantic model.

Return type:: List[StrawberryDocument]

traverse(traversal_paths, filter_fn=None)#

Return an Iterator of :class:TraversableSequence of the leaves when applying the traversal_paths. Each :class:TraversableSequence is either the root Documents, a ChunkArray or a MatchArray.

Parameters:

traversal_paths (str) – a comma-separated string that represents the traversal path
filter_fn (Optional[Callable[[Document], bool]]) – function to filter docs during traversal

Yield:

:class:TraversableSequence of the leaves when applying the traversal_paths.

Example on traversal_paths:

r: docs in this TraversableSequence

m: all match-documents at adjacency 1

c: all child-documents at granularity 1

r.[attribute]: access attribute of a multi modal document

cc: all child-documents at granularity 2

mm: all match-documents at adjacency 2

cm: all match-document at adjacency 1 and granularity 1

r,c: docs in this TraversableSequence and all child-documents at granularity 1

r[start:end]: access sub document array using slice

Return type:: Iterable[T]

traverse_flat(traversal_paths, filter_fn=None)#

Returns a single flattened :class:TraversableSequence with all Documents, that are reached via the traversal_paths.

Warning

When defining the traversal_paths with multiple paths, the returned :class:Documents are determined at once and not on the fly. This is a different behavior then in :method:traverse and :method:traverse_flattened_per_path!

Parameters:

traversal_paths (str) – a list of string that represents the traversal path
filter_fn (Optional[Callable[[Document], bool]]) – function to filter docs during traversal

Return type:

DocumentArray

Returns:

a single :class:TraversableSequence containing the document of all leaves when applying the traversal_paths.

traverse_flat_per_path(traversal_paths, filter_fn=None)#

Returns a flattened :class:TraversableSequence per path in traversal_paths with all Documents, that are reached by the path.

Parameters:

traversal_paths (str) – a comma-separated string that represents the traversal path
filter_fn (Optional[Callable[[Document], bool]]) – function to filter docs during traversal

Yield:

:class:TraversableSequence containing the document of all leaves per path.

TYPE_MAP: Dict[str, TypeMap]#