Module

clusterize.py

import goldener.clusterize

Clustering of vectorized samples.

Clustering groups the vectorized samples in clusters, gathering similar content unless a random clustering is used. The clusters are used to keep the variability of the data when selecting or splitting it, and to balance the batches during training (goldener.organize).

GoldClusteringTool is the abstract interface of the clustering algorithms.
GoldSKLearnClusteringTool wraps a scikit-learn clustering estimator providing predict and n_clusters.
GoldRandomClusteringTool chunks the data randomly into clusters of almost equal size.
GoldClusterizer runs the clustering on a whole dataset of vectors, possibly in chunks, and stores the cluster assignments locally (one per vector by default, a sample with multiple vectors can then have multiple clusters).

On this page 4 classes1 function

GoldClusteringTool GoldRandomClusteringTool get_random_chunk_assignment() GoldSKLearnClusteringTool GoldClusterizer

logger = <Logger goldener.clusterize (INFO)>

class GoldClusteringTool(abc.ABC):

Run clustering on input vectors.

@abstractmethod

def fit(self, x: torch.Tensor, n_clusters: int) -> torch.Tensor:

Fit the clustering tool from a given set of vectors into n clusters.

Arguments:

x: Input vectors to select from. Must be a 2D tensor of shape (num_vectors, feature_dim).
n_clusters: Number of clusters to form.

Returns: The cluster assignments for each input vector as a 1D tensor of cluster indices.

@abstractmethod

def predict(self, x: torch.Tensor) -> torch.Tensor:

Assign the input vectors to the clusters.

Arguments:

x: Input vectors to transform. Must be a 2D tensor of shape (num_vectors, feature_dim).

Returns: The cluster assignments for each input vector as a 1D tensor of cluster indices.

class GoldRandomClusteringTool(GoldClusteringTool):

Chunk data randomly into clusters of almost equal size.

GoldRandomClusteringTool(random_state: int | None = None)

random_state

def fit(self, x: torch.Tensor, n_clusters: int) -> torch.Tensor:

Randomly assign each input vector to clusters of roughly the same size.

Arguments:

x: Input vectors to select from. Must be a 2D tensor of shape (num_vectors, feature_dim).
n_clusters: Number of clusters to form.

Returns: The cluster assignments for each input vector as a 1D tensor of cluster indices.

Raises:

ValueError: If n_clusters is greater than the number of samples in x.

def predict(self, x: torch.Tensor) -> torch.Tensor:

Assign the input vectors to the clusters.

Arguments:

x: Input vectors to transform. Must be a 2D tensor of shape (num_vectors, feature_dim).

Returns: The cluster assignments for each input vector as a 1D tensor of cluster indices.

def get_random_chunk_assignment( to_chunk: list[int], max_chunk_size: int | None = None, random_state: int | None = None) -> list[list[int]]:

Assign indices to random chunks.

Arguments:

to_chunk: List of indices to chunk.
max_chunk_size: Maximum size of each chunk. If None, all indices are returned in a single chunk.
random_state: Optional random state for reproducibility during chunk assignment.

Returns:

A list of chunks, where each chunk is a list of indices from to_chunk randomly assigned.

Raises:

ValueError: If to_chunk is empty or if max_chunk_size is not a positive integer or None.

class GoldSKLearnClusteringTool(GoldClusteringTool):

Cluster data using a wrapped scikit-learn clustering estimator.

GoldSKLearnClusteringTool(tool: sklearn.base.ClusterMixin)

tool

def fit(self, x: torch.Tensor, n_clusters: int) -> torch.Tensor:

Fit the scikit-learn clustering tool on the input vectors and return cluster assignments.

Arguments:

x: Input vectors to select from. Must be a 2D tensor of shape (num_vectors, feature_dim).
n_clusters: Number of clusters to form.

Returns: The cluster assignments for each input vector as a 1D tensor of cluster indices.

def predict(self, x: torch.Tensor) -> torch.Tensor:

Assign the input vectors to clusters using the fitted scikit-learn clustering tool.

Arguments:

x: Input vectors to assign to clusters. Must be a 2D tensor of shape (num_vectors, feature_dim).

Returns:

A 1D tensor of cluster indices predicted for each input vector.

class GoldClusterizer(goldener.do.GoldDoerWithTable):

Cluster data points from vectorized samples.

The GoldClusterizer processes a dataset or PixelTable table to perform clustering using a clustering algorithm on already vectorized representations. The clustering results are stored in a local PixelTable table (specified by table_path) so that the clustering process is idempotent: calling the same operation multiple times will not duplicate or recompute clustering that are already present in the table.

If the dataset is too big to fit into memory or the clustering algorithm is too computationally expensive, the clustering can be performed in chunks. All torch tensors will be converted to numpy arrays before saving.

The clustering can operate in a sequential (single-process) mode or a distributed mode (not implemented). The table schema is created/validated automatically and will include minimal indexing columns (idx, idx_vector) required to link clustered samples back to their originating data points.

Attributes:

table_path: Path to the PixelTable table where clustering results will be stored locally.
clustering_tool: The GoldClusteringTool implementing the clustering algorithm.
reducer: Optional GoldReductionTool instance for dimensionality reduction before clustering.
chunk: Optional chunk size for processing data in chunks to reduce memory consumption.
collate_fn: Optional function to collate dataset samples into batches composed of dictionaries with at least the key specified by vectorized_key returning a PyTorch Tensor. If None, collate_keeping_sequences_as_sequences is used, which preserves sequence-valued fields as one sequence per sample.
vectorized_key: Key in the batch dictionary that contains the vectorized data for the clustering. Default is "vectorized".
include_vectorized_in_table: Whether to keep the vectorized data in the cluster table. It is only applied if the cluster table is created from a Table (it is forced anyway for Dataset). Default is False.
cluster_key: Column name to store the clustering value in the PixelTable table. Default is "cluster".
label_key: Optional key for label-based stratified clustering.
to_keep_schema: Optional dictionary defining additional columns to keep from the original dataset/table into the cluster table. The keys are the column names and the values are the PixelTable types.
min_pxt_insert_size: Minimum number of rows to accumulate before inserting into PixelTable. Default is 100.
batch_size: Batch size used when iterating over the data.
num_workers: Number of workers for the PyTorch DataLoader during iteration on data.
allow_existing: If False, an error will be raised when the table already exists. Default is True.
distribute: Whether to use distributed processing for clustering and table population. Not implemented yet. Default is False.
drop_table: Whether to drop the cluster table after creating the dataset with clustering results. It is only applied when using cluster_in_dataset. Default is False.
max_batches: Optional maximum number of batches to process. Useful for testing on a small subset of the dataset.
random_state: Optional random state for reproducibility during chunk assignment.
force_same_cluster: If True, all vectors from the same sample (same idx) are assigned to the same cluster. The assignment to a cluster depends on the last clusterized vector of the sample (suboptimal but easiest solution). Default is False.

GoldClusterizer( table_path: str, clustering_tool: GoldClusteringTool, reducer: goldener.reduce.GoldReductionTool | None = None, chunk: int | None = None, collate_fn: Optional[Callable] = None, vectorized_key: str = 'vectorized', include_vectorized_in_table: bool = False, cluster_key: str = 'cluster', label_key: str | None = None, to_keep_schema: dict[str, type] | None = None, min_pxt_insert_size: int = 100, batch_size: int = 1, num_workers: int = 0, allow_existing: bool = True, distribute: bool = False, drop_table: bool = False, max_batches: int | None = None, random_state: int | None = None, force_same_cluster: bool = False)

Initialize the GoldClusterizer with configuration parameters.

Attributes:

table_path: Path to the PixelTable table where clustering results will be stored locally.
clustering_tool: The GoldClusteringTool implementing the clustering algorithm.
reducer: Optional GoldReductionTool instance for dimensionality reduction before clustering.
chunk: Optional chunk size for processing data in chunks to reduce memory consumption.
collate_fn: Optional function to collate dataset samples into batches composed of dictionaries with at least the key specified by vectorized_key returning a PyTorch Tensor. If None, collate_keeping_sequences_as_sequences is used, which preserves sequence-valued fields as one sequence per sample.
vectorized_key: Key in the batch dictionary that contains the vectorized data for the clustering. Default is "vectorized".
include_vectorized_in_table: Whether to keep the vectorized data in the cluster table. Defaults to False. It is only applied if the cluster table is created from a Table (it is forced anyway for Dataset).
cluster_key: Column name to store the clustering value in the PixelTable table. Default is "cluster".
label_key: Optional key for label-based stratified clustering.
to_keep_schema: Optional dictionary defining additional columns to keep from the original dataset/table into the cluster table. The keys are the column names and the values are the PixelTable types.
min_pxt_insert_size: Minimum number of rows to accumulate before inserting into PixelTable. Default is 100.
batch_size: Batch size used when iterating over the data.
num_workers: Number of workers for the PyTorch DataLoader during iteration on data.
allow_existing: If False, an error will be raised when the table already exists. Default is True.
distribute: Whether to use distributed processing for clustering and table population. Not implemented yet. Default is False.
drop_table: Whether to drop the cluster table after creating the dataset with clustering results. It is only applied when using cluster_in_dataset. Default is False.
max_batches: Optional maximum number of batches to process. Useful for testing on a small subset of the dataset.
random_state: Optional random state for reproducibility during chunk assignment.
force_same_cluster: If True, all vectors from the same sample (same idx) are assigned to the same cluster. The assignment to a cluster depends on the last clusterized vector of the sample (suboptimal but easiest solution). Default is False.

Raises:

ValueError: If chunk is not a positive integer or None.
NotImplementedError: If distribute is True.

table_path

clustering_tool

reducer

chunk

vectorized_key

include_vectorized_in_table

cluster_key

label_key

random_state

force_same_cluster

distribute: bool

Whether distributed processing is enabled.

def cluster_in_dataset( self, cluster_from: torch.utils.data.dataset.Dataset | pixeltable.catalog.table.Table, n_clusters: int, restrict_to: set[int] | None = None, restriction_idx_key: str = 'idx_vector') -> goldener.pxt_utils.GoldPxtTorchDataset:

Cluster the data and return results as a GoldPxtTorchDataset.

The clustering process applies a clustering algorithm on already vectorized representations of the data points and stores results in a PixelTable table specified by table_path. When the chunk attribute is set, the clustering is performed in chunks to reduce memory consumption. If a reducer is provided, the vectors are reduced in dimension before applying the clustering.

This is a convenience wrapper that runs cluster_in_table to populate (or resume populating) the PixelTable table, then wraps the table into a GoldPxtTorchDataset for downstream consumption. If drop_table is True, the table will be removed after the dataset is created.

Arguments:

cluster_from: Dataset or Table to cluster. If a Dataset is provided, each item should be a dictionary with at least the vectorized_key, idx_vector and idx keys after applying the collate_fn. If a Table is provided, it should contain at least the vectorized_key and idx columns.
n_clusters: Number of clusters to create.
restrict_to: Optional set of indices to restrict the clustering to. If provided, only the selected indices in restriction_idx_key will be used for the clustering.
restriction_idx_key: Column name used to get sample indices for restriction.

Returns:

A GoldPxtTorchDataset containing at least the clustering information in the cluster_key key and idx (index of the sample) and idx_vector (index of the vector) keys as well.

def cluster_in_table( self, cluster_from: torch.utils.data.dataset.Dataset | pixeltable.catalog.table.Table, n_clusters: int, restrict_to: set[int] | None = None, restriction_idx_key: str = 'idx_vector') -> pixeltable.catalog.table.Table:

Cluster the data and store results in a PixelTable table.

This method is idempotent (i.e., failure proof), meaning that if it is called multiple times on the same dataset or table, it will restart the clustering process based on the vectors already present in the PixelTable table.

Arguments:

cluster_from: Dataset or Table to select from. If a Dataset is provided, each item should be a dictionary with at least the vectorized_key and idx keys after applying the collate_fn. If a Table is provided, it should contain at least the vectorized_key, idx and idx_vector columns.
n_clusters: Number of clusters to create.
restrict_to: Optional set of indices to restrict the clustering to. If provided, only the selected indices in restriction_idx_key will be used for the clustering.
restriction_idx_key: Column name used to get sample indices for restriction.

Returns:

A PixelTable Table containing at least the clustering information in the cluster_key column and idx (index of the sample) and idx_vector (index of the vector) columns as well.

Raises:

ValueError: If restriction_idx_key is not "idx" or "idx_vector", or if n_clusters is less than or equal to 1.

@staticmethod

def get_cluster_pxt_query( table: pixeltable.catalog.table.Table, cluster_key: str, cluster_idx: int | None = None, label_key: str | None = None, label_value: str | None = None, idx_key: str = 'idx_vector', restrict_to: set[int] | None = None, restriction_idx_key: str = 'idx_vector') -> pixeltable._query.Query:

Get the Pixeltable query to access samples clustered in a given cluster.

Arguments:

table: PixelTable table to query.
cluster_key: Column name used to store the clustering values.
cluster_idx: Value in the cluster column to filter samples.
label_key: Optional column name used to filter samples by label.
label_value: Optional label value to filter samples by label.
idx_key: Column name used to get sample indices.
restrict_to: Optional set of indices to restrict the search to. If provided, only the selected indices in restriction_idx_key will be used for the clustering.
restriction_idx_key: Column name used to get sample indices for restriction.

Returns: A Pixeltable Query object that can be used to access the samples in the specified cluster (and label if specified).

Raises:

ValueError: If only one of label_key or label_value is provided (both must be set together).

@staticmethod

def get_cluster_indices( table: pixeltable.catalog.table.Table, cluster_key: str = 'cluster', cluster_idx: int | None = None, label_key: str | None = None, label_value: str | None = None, idx_key: str = 'idx_vector', restrict_to: set[int] | None = None, restriction_idx_key: str = 'idx_vector') -> set[int]:

Get the indices of samples clustered in a given cluster.

Arguments:

table: PixelTable table to query.
cluster_key: Column name used to store the clustering values.
cluster_idx: Value in the cluster column to filter samples.
label_key: Optional column name used to filter samples by label.
label_value: Optional label value to filter samples by label.
idx_key: Column name used to get sample indices.
restrict_to: Optional set of indices to restrict the search to. If provided, only the selected indices in restriction_idx_key will be used for the clustering.
restriction_idx_key: Column name used to get sample indices for restriction.

Returns: A set of indices corresponding to the samples in the specified cluster (and label if specified).

Raises:

ValueError: If only one of label_key or label_value is provided (both must be set together).

@staticmethod

def get_cluster_count( table: pixeltable.catalog.table.Table, cluster_key: str, cluster_idx: int | None = None, label_key: str | None = None, label_value: str | None = None, idx_key: str = 'idx_vector', restrict_to: set[int] | None = None, restriction_idx_key: str = 'idx_vector') -> int:

Get the number of samples clustered in a given cluster.

Arguments:

table: PixelTable table to query.
cluster_key: Column name used to store the clustering values.
cluster_idx: Value in the cluster column to filter samples.
label_key: Optional column name used to filter samples by label.
label_value: Optional label value to filter samples by label.
idx_key: Column name used to get sample indices.
restrict_to: Optional set of indices to restrict the search to. If provided, only the selected indices in restriction_idx_key will be used for the clustering.
restriction_idx_key: Column name used to get sample indices for restriction.

Returns: The number of samples in the specified cluster (and label if specified).

Raises:

ValueError: If only one of label_key or label_value is provided (both must be set together).

Inherited Members

goldener.do.GoldDoerWithTable: collate_fn; to_keep_schema; min_pxt_insert_size; batch_size; num_workers
goldener.do.GoldDoer: allow_existing; drop_table; max_batches