Module

vectorize.py

import goldener.vectorize

Vectorization of the embeddings into 2D tensors of vectors.

An embedding is a tensor of at least 3 dimensions (batch, channels and e.g. tokens or pixels). The algorithms of Goldener work on 2D tensors of vectors, so the embeddings are flattened into vectors and, optionally, some of these vectors are filtered out (e.g. keeping only the vectors of a region of interest given by a target).

GoldTensorVectorizationTool transforms a tensor into vectors, using Filter2DWithCount and FilterLocation to filter them, and returns a Vectorized object keeping track of the sample of each vector.
GoldVectorizer applies the vectorization on a whole dataset in batches and stores the vectors locally, without recomputing already vectorized samples.

The vectors are the input of the clustering (goldener.clusterize) and selection (goldener.select) steps.

On this page 5 classes2 functions

FilterLocation Filter2DWithCount Vectorized GoldTensorVectorizationTool GoldVectorizer unwrap_vectors_in_batch() vectorize_and_unwrap_in_batch()

logger = <Logger goldener.vectorize (INFO)>

class FilterLocation(enum.Enum):

Enum defining filter location strategies for filtering 2D tensor rows.

START: Filter from the start of the tensor. END: Filter from the end of the tensor. RANDOM: Filter randomly from the tensor.

START = <FilterLocation.START: 'start'>

END = <FilterLocation.END: 'end'>

RANDOM = <FilterLocation.RANDOM: 'random'>

class Filter2DWithCount:

Filter 2D tensor rows based on specified criteria.

Attributes:

_filter_location: Location to filter from (start, end, random).
_filter_count: Number of rows to filter.
_keep: Whether to keep or remove the filtered rows.
_random_sampler: Sampler for random filtering.

Filter2DWithCount( filter_count: int = 1, filter_location: FilterLocation = <FilterLocation.RANDOM: 'random'>, keep: bool = False, generator: torch._C.Generator | None = None)

Initialize the Filter2DWithCount.

Arguments:

filter_count: Number of rows to filter.
filter_location: Location to filter from (start, end, random).
keep: Whether to keep or remove the filtered rows.
generator: Random number generator for random filtering.

Raises:

ValueError: If filter_count is not greater than 0.

is_random: bool

Check if the filter is random.

Returns:

True if the filter uses random sampling, False otherwise.

is_keeping: bool

Check if the filter is set to keep the filtered rows.

Returns:

True if the filter is set to keep the filtered rows, False if it is set to remove them.

filter_count: int

Get the number of rows to filter.

Returns:

The number of rows that the filter is set to filter.

def filter(self, x: torch.Tensor) -> torch.Tensor:

Filter a single 2D tensor.

Arguments:

x: Input 2D tensor to filter.

Returns:

Filtered tensor.

def filter_tensors(self, x: dict[str, torch.Tensor]) -> dict[str, torch.Tensor]:

Filter the input tensor or dictionary of tensors.

Arguments:

x: Input 2D tensor or dictionary of 2D tensors to filter. In case of a dictionary,
all tensors must have the same number of rows and this number must be greater than filter_count.

Returns:

Filtered tensor or dictionary of tensors. If the batch size is less than filter_count, the input is returned unchanged.

Raises:

ValueError: If input tensors do not have the same batch size or are not 2 dimensional.

@dataclass

class Vectorized:

Dataclass to hold vectorized tensors and the corresponding batch indices.

Attributes:

vectors: 2D tensor of vectorized data.
batch_indices: 1D tensor containing information about the origin of each vector.

Vectorized(vectors: torch.Tensor, batch_indices: torch.Tensor)

vectors: torch.Tensor

batch_indices: torch.Tensor

class GoldTensorVectorizationTool:

Transform input as 2D tensor and filter based on target tensor.

Attributes:

keep: Filter2DWithCount instance to keep specific rows in the input x of filter.
remove: Filter2DWithCount instance to remove specific rows in the input x of filter.
random: Random Filter2DWithCount instance to randomly filter vectors after
applying keep, remove and the target y on the input x of filter.
fusion_strategy: Optional strategy to fuse vectors when more than 1 vector is kept for a sample after all filtering steps.
transform_y: Optional callable to transform the target tensor before transforming it to 2D.
channel_pos: position of the channel dimension in the input tensor to vectorize.
in_selection_tool: Optional smart selection tool applied after all other input filters.
in_selection_size: Number or fraction of vectors to keep with in_selection_tool.

GoldTensorVectorizationTool( keep: Filter2DWithCount | None = None, remove: Filter2DWithCount | None = None, random: Filter2DWithCount | None = None, fusion_strategy: goldener.embed.EmbeddingFusionStrategy | None = None, transform_y: Optional[Callable[[torch.Tensor], torch.Tensor]] = None, channel_pos: int = 1, in_selection_tool: goldener.select.GoldSelectionTool | None = None, in_selection_size: int | float = 1.0)

Initialize the GoldTensorVectorizationTool.

Arguments:

keep: Optional filter to keep specific rows in the input.
remove: Optional filter to remove specific rows from the input.
random: Optional random filter to apply after keep/remove filters.
fusion_strategy: Optional strategy to fuse vectors when more than 1 vector is kept for a sample after all filtering steps.
transform_y: Optional transformation to apply to the target tensor.
channel_pos: Position of the channel dimension in the input tensor. Defaults to 1.
in_selection_tool: Optional selection tool to choose the most useful remaining vectors.
in_selection_size: Number or fraction of vectors for in_selection_tool to keep.

Raises:

ValueError: If keep, remove and random filters are not having the expected properties.
See the checks in the code for details.

transform_y

channel_pos

keep

remove

random

in_selection_tool

in_selection_size

fusion_strategy

def vectorize( self, x: torch.Tensor, y: torch.Tensor | None = None) -> Vectorized:

Vectorize input tensor and filter based on target tensor.

If y is not provided, only filtering and vectorization are performed. If y is provided, it is used to filter the vectorized x tensor, only if at least 1 vector location is non zero in the target.

Arguments:

x: Input tensor to vectorize.
y: Optional target tensor to filter the input tensor.

Returns:

Vectorized and filtered input tensor with the information about which element of the batch it corresponds to.

Raises:

ValueError: If x and y shapes are incompatible. See check_x_and_y_shapes for details.

class GoldVectorizer(goldener.do.GoldDoerWithTable):

Extract and flatten vectors from dataset samples and store results in a PixelTable table.

The GoldVectorizer processes a dataset or PixelTable table to extract and flatten vectors using a GoldTensorVectorizationTool. The computed vectors are stored in a local PixelTable table (specified by table_path) so that the vectorization process is idempotent: calling the same operation multiple times will not duplicate or recompute vectors that are already present in the table.

Assuming all the data will not fit in memory, the dataset is processed in batches. All torch tensors will be converted to numpy arrays before saving.

The vectorization can operate in a sequential (single-process) mode or a distributed mode (not implemented). The table schema is created/validated automatically and will include minimal indexing columns (idx, idx_vector) required to link vectors back to their originating samples.

Attributes:

table_path: Path to the PixelTable table where vectorized outputs will be stored locally.
vectorizer: GoldTensorVectorizationTool instance for transforming batched inputs into vectors.
collate_fn: Optional function to collate dataset samples into batches composed of dictionaries with at least the key specified by data_key returning a PyTorch Tensor. If None, collate_keeping_sequences_as_sequences is used, which preserves sequence-valued fields as one sequence per sample.
data_key: Key in the batch dictionary that contains the data to vectorize. Default is "embeddings".
target_key: Optional key in the batch dictionary containing the target used to filter vectors. Default is "target".
vectorized_key: Column name to store the resulting vectors in the PixelTable table. Default is "vectorized".
label_key: Optional key for labels in the batch dictionary. Default is None.
target_to_label: Optional mapping from target values to label strings. Default is None.
merge_multilabels: Whether to merge multiple labels into a single label. Default is False.
exclude_full_zero_target: Whether to exclude samples with a target tensor containing only zeros (in case of multi target). Default is False.
exclude_labels: Optional set of label strings to exclude from vectorization. Default is None.
to_keep_schema: Optional dictionary defining additional columns to keep from the original dataset/table into the vectorized table. The keys are the column names and the values are the PixelTable types.
min_pxt_insert_size: Minimum number of rows to accumulate before inserting into the PixelTable table. Default is 100.
batch_size: Batch size used when iterating over the data.
num_workers: Number of workers for the PyTorch DataLoader during iteration on data.
allow_existing: If False, an error will be raised when the table already exists. Default is True.
distribute: Whether to use distributed processing for vectorization and table population. Not implemented yet. Default is False.
drop_table: Whether to drop the vectorized table after creating the dataset with vectorized outputs. It is only applied when using vectorize_in_dataset. Default is False.
max_batches: Optional maximum number of batches to process. Useful for testing on a small subset of the dataset.

GoldVectorizer( table_path: str, vectorizer: GoldTensorVectorizationTool, collate_fn: Optional[Callable] = None, data_key: str = 'embeddings', target_key: str = 'target', vectorized_key: str = 'vectorized', label_key: str | None = None, target_to_label: dict[tuple[int, ...], str] | None = None, merge_multilabels: bool = False, exclude_labels: set[str] | None = None, exclude_full_zero_target: bool = False, to_keep_schema: dict[str, type] | None = None, min_pxt_insert_size: int = 100, batch_size: int = 1, num_workers: int = 0, allow_existing: bool = True, distribute: bool = False, drop_table: bool = False, max_batches: int | None = None, estimation_time_batch_count: int = 2)

Initialize the GoldVectorizer.

Arguments:

table_path: Path to the PixelTable table for storing vectors.
vectorizer: GoldTensorVectorizationTool instance for transforming tensors.
collate_fn: Optional collate function for preparing batches. If None, collate_keeping_sequences_as_sequences is used.
data_key: Key for data in the batch dictionary. Defaults to "embeddings".
target_key: Key for target in the batch dictionary. Defaults to "target".
vectorized_key: Column name for storing vectors. Defaults to "vectorized".
label_key: Optional key for labels in the batch dictionary. Default is None.
target_to_label: Optional mapping from target values to label strings. Default is None.
merge_multilabels: Whether to merge multiple labels into a single label. Default is False.
exclude_labels: Optional set of label strings to exclude from vectorization. Default is None.
exclude_full_zero_target: Whether to exclude samples with a target tensor containing only zeros (in case of multi target). Default is False.
to_keep_schema: Optional schema for additional columns to preserve.
min_pxt_insert_size: Minimum number of rows to accumulate before inserting into the PixelTable table. Default is 100.
batch_size: Batch size used when iterating over the data.
num_workers: Number of workers for the PyTorch DataLoader during iteration on data.
allow_existing: Whether to allow using an existing table. Defaults to True.
distribute: Whether to use distributed processing. Defaults to False.
drop_table: Whether to drop the table after dataset creation. Defaults to False.
max_batches: Optional maximum number of batches to process.
estimation_time_batch_count: Number of initial batches used to estimate the total computation time. Defaults to 2.

Raises:

NotImplementedError: If distribute is True.
ValueError: If estimation_time_batch_count is not a positive integer.

table_path

vectorizer

data_key

target_key

label_key

target_to_label

exclude_labels

exclude_full_zero_target

merge_multilabels

vectorized_key

estimation_time_batch_count

distribute: bool

Whether distributed processing is enabled.

def vectorize_in_dataset( self, to_vectorize: torch.utils.data.dataset.Dataset | pixeltable.catalog.table.Table, restrict_to: set[int] | None = None) -> goldener.pxt_utils.GoldPxtTorchDataset:

Extract and flatten vectors from samples and return results as a GoldPxtTorchDataset.

The vectorization process extracts and flattens vectors from the provided dataset or table using the GoldTensorVectorizationTool instance and stores them in a PixelTable table specified by table_path. The resulting vectors are stored in the column specified by vectorized_key.

This is a convenience wrapper that runs vectorize_in_table to populate (or resume populating) the PixelTable table, then wraps the table into a GoldPxtTorchDataset for downstream consumption. If drop_table is True, the table will be removed after the dataset is created.

Arguments:

to_vectorize: Dataset or Table to be vectorized. If a Dataset is provided, each item should be a dictionary with at least the key specified by data_key after applying the collate_fn. If a Table is provided, it should contain both 'idx' and data_key columns.
restrict_to: Optional set of sample indices (idx) to restrict the vectorization to. If provided, only the selected samples will be vectorized.

Returns:

A GoldPxtTorchDataset containing at least the vectorized data in the vectorized_key key and idx (index of the sample) and idx_vector (index of the vector) keys as well.

def vectorize_in_table( self, to_vectorize: torch.utils.data.dataset.Dataset | pixeltable.catalog.table.Table, restrict_to: set[int] | None = None) -> pixeltable.catalog.table.Table:

Extract and flatten vectors from samples and store results in a PixelTable table.

This method is idempotent (i.e., failure proof), meaning that if it is called multiple times on the same dataset or table, it will not duplicate or recompute the vectorization already present in the PixelTable table.

Arguments:

to_vectorize: Dataset or Table to be vectorized. If a Dataset is provided, each item should be a dictionary with at least the key specified by data_key after applying the collate_fn. If a Table is provided, it should contain both 'idx' and data_key columns.
restrict_to: Optional set of sample indices (idx) to restrict the vectorization to. If provided, only the selected samples will be vectorized.

Returns:

A PixelTable Table containing at least the vectorized data in the vectorized_key column and idx (index of the sample) and idx_vector (index of the vector) columns as well.

Raises:

ValueError: If allow_existing is False and the table at table_path already exists, or if an existing vectorized table is missing the required idx_vector primary key.

def estimate_computation_time(self, dataset: torch.utils.data.dataset.Dataset) -> float | None:

Estimate the total computation time for vectorizing a dataset.

Times self.estimation_time_batch_count initial batches — including data loading, preprocessing, and vectorization — running the exact same steps as _sequential_vectorize (via _compute_batch), then extrapolates if the dataset's size is known.

Inherited Members

goldener.do.GoldDoerWithTable: collate_fn; to_keep_schema; min_pxt_insert_size; batch_size; num_workers
goldener.do.GoldDoer: allow_existing; drop_table; max_batches

def unwrap_vectors_in_batch( vectorized: Vectorized, vectorized_key: str, batch: dict[str, typing.Any], starts: int = 0, to_keep: list[str] | None = None, label_key: str | None = None, exclude_labels: set[str] | None = None) -> dict[str, typing.Any]:

Unwrap vectorized output into individual rows for table insertion.

This converts the Vectorized dataclass (which contains batched vectors and their batch indices) into a dictionary format suitable for inserting into the PixelTable table, with one entry per vector.

Arguments:

vectorized: Vectorized output containing vectors and batch indices.
vectorized_key: Key under which to store the vectors in the output dictionary.
batch: Original batch dictionary with metadata.
starts: Starting index for assigning new idx_vector values.
to_keep: Optional list of additional keys to preserve from the batch.
label_key: Optional key for labels in the batch dictionary. Default is None.
exclude_labels: Optional set of label strings to exclude from vectorization. Default is None.

Returns:

Dictionary with unwrapped vectors and metadata, ready for table insertion.

Raises:

ValueError: If exclude_labels is provided but label_key is None.

def vectorize_and_unwrap_in_batch( batch: dict[str, typing.Any], vectorizer: GoldTensorVectorizationTool, data_key: str, vectorized_key: str, target_key: str | None = None, label_key: str | None = None, target_to_label: dict[tuple[int, ...], str] | None = None, merge_multilabels: bool = False, exclude_labels: set[str] | None = None, to_keep: list[str] | None = None, starts: int = 0, exclude_full_zero_target: bool = True) -> dict[str, typing.Any]:

Vectorize a batch and insert the results into a PixelTable table.

This function vectorizes the data in the provided batch using the specified GoldTensorVectorizationTool and inserts the resulting vectors into the given PixelTable table.

Arguments:

batch: The batch dictionary containing data to vectorize.
vectorizer: The GoldTensorVectorizationTool instance to use for vectorization.
data_key: Key in the batch dictionary that contains the data to vectorize.
vectorized_key: Column name to store the resulting vectors in the PixelTable table.
target_key: Optional key in the batch dictionary containing the target used to filter vectors.
label_key: Optional key in the batch dictionary containing the labels.
target_to_label: Optional mapping from target values to labels for binarization.
merge_multilabels: Whether to merge multiple labels into a single label. Default is False.
exclude_labels: Optional set of label strings to exclude from vectorization. Default is None.
to_keep: Optional list of additional keys to preserve from the batch.
starts: Starting index for assigning new idx_vector values.
exclude_full_zero_target: Whether to exclude samples with a full zero target when binarizing
targets based on the target_to_label mapping. Default is True.