Module

organize.py

import goldener.organize
View source

Organization of the data during training.

With a random sampler, the content of the batches can vary a lot, which can harm the convergence of a model. The GoldClusterizedBatchSampler forces every batch to contain at least one sample of each cluster of the data (see goldener.clusterize), while keeping randomness in the selection of the samples within the clusters. When there are more clusters than the batch size, every batch contains samples from different clusters instead. ExhaustedClusterStrategy defines what to do when all the samples of a cluster have been used.

On this page 2 classes1 function
get_indices_per_cluster_for_subset() ExhaustedClusterStrategy GoldClusterizedBatchSampler
logger = <Logger goldener.organize (INFO)>
def get_indices_per_cluster_for_subset( indices_per_cluster: dict[int, list[int]], indices_in_subset: Sequence[int]) -> dict[int, list[int]]:

Get the indices per cluster corresponding to the subset of a dataset.

The initial dataset does not have any duplicate indices. A Subset of a dataset is storing the dataset and the sequence of indices corresponding to the subset. The same dataset index can be present multiple times in the subset. This function is adapted to this situation and include all the subset locations in the output list of indices for each cluster.

Arguments:
  • indices_per_cluster: The list of indices pointing toward samples in the initial dataset for each cluster.
  • indices_in_subset: The indices specifying the subset of the initial dataset.

Returns: The converted indices_per_cluster mapping the cluster with the indices in the subset

class ExhaustedClusterStrategy(enum.Enum):

Strategy to adopt when a cluster is exhausted within a clusterized Batch sampler.

Inside the batch sampler with clustering, the clusters might be of different size. Thus, some clusters might be exhausted before the others. Here the different strategy for the batch sampler:

  • RESTART: The exhausted cluster is reinitialized and still used for next batch sampling. This means that the samples of the exhausted cluster will be oversampled compared to the other clusters.
  • STOP: The batch sampler is stopped when a cluster is exhausted. This means that not all the samples of the other clusters will be used.
  • EXCLUDE: The exhausted cluster is excluded from the next batch sampling. This means that the samples of the exhausted cluster will be used only once, and that the batches will contain less and less clusters until all the clusters are exhausted.
RESTART = <ExhaustedClusterStrategy.RESTART: 'restart'>
EXCLUDE = <ExhaustedClusterStrategy.EXCLUDE: 'exclude'>
class GoldClusterizedBatchSampler(typing.Generic[+_T_co]):

Batch sampler forcing the presence of all clusters in each batch.

With a random batcher, the content distribution within the batches might vary a lot. During the training of a model, this variation can influence badly its convergence leading to an underperforming model.

In this batcher, the data is first clustered in order to gather together the samples showing a similar content. Then, every batch is forced to contain at least 1 element of each cluster.

The randomness is still available in the process of selection among the elements of all clusters.

Attributes:
  • shuffle: if True, the order of the samples of all clusters will be shuffled before sampling, the batch is shuffled to change the cluster order, and once exhausted a cluster is shuffled again. If False, the order of the samples and clusters will be preserved.
  • generator: optional generator to manage the random shuffling. If None, a new generator will be created with a random seed.
GoldClusterizedBatchSampler( dataset: torch.utils.data.dataset.Dataset, batch_size: int, clusterizer: goldener.clusterize.GoldClusterizer, n_clusters: int | None = None, descriptor: goldener.describe.GoldDescriptor | None = None, vectorizer: goldener.vectorize.GoldVectorizer | None = None, force_same_size: bool = False, shuffle: bool = True, generator: torch._C.Generator | None = None, strategy: ExhaustedClusterStrategy = <ExhaustedClusterStrategy.RESTART: 'restart'>)

Initialize the batch sampler based on clustering results.

Arguments:
  • dataset: dataset to sample from.
  • batch_size: batch size specifying the size of each batch. Fix the number of clusters if n_clusters is None.
  • clusterizer: clusterizer to cluster the data into clusters.
  • n_clusters: Number of clusters to create. If None, it is set to the batch size. It can be different from the batch size if we want to have more clusters than the batch size, and thus not all the clusters are present in each batch.
  • descriptor: optional descriptor to describe the dataset before clusterization.
  • vectorizer: optional vectorizer to vectorize the dataset before clusterization.
  • force_same_size: if True, all the clusters are required to have the same size. If False, the sampler will cycle through the clusters until all the samples are exhausted. It will then oversample the smallest clusters.
  • shuffle: if True, the order of the samples of all clusters will be shuffled before sampling, and once exhausted a cluster is shuffled again. If False, the order of the samples and clusters will be preserved.
  • generator: optional generator to manage the random shuffling. If None, a new generator will be created with a random seed.
  • strategy: strategy to apply when a cluster is exhausted. See ExhaustedClusterStrategy for more details.
shuffle
generator