Module

pxt_utils.py

import goldener.pxt_utils
View source

Utilities to store and read the results of Goldener.

All the steps of Goldener store their results locally, in tables. This module gathers the helpers to create and query these tables, and GoldPxtTorchDataset, a PyTorch dataset reading such a table while restoring the original shape of the array columns.

On this page 1 class14 functions
get_expr_from_column_name() create_pxt_dirs_for_path() create_pxt_table_from_sample() set_value_to_idx_rows() GoldPxtTorchDataset get_array_column_shapes_from_table() get_array_column_shapes_from_query() get_distinct_value_and_count_in_column() get_column_distinct_ratios() get_sample_row_from_idx() get_valid_table() make_batch_ready_for_table() get_pxt_table_primary_keys() check_pxt_table_has_primary_key() get_max_value_in_column()
def get_expr_from_column_name( table: pixeltable.catalog.table.Table, column_name: str) -> pixeltable.exprs.expr.Expr:

Get the expression object for a given column name in a PixelTable table.

Arguments:
  • table: The PixelTable table.
  • column_name: The name of the column.
Returns:

pxt.Expr: The expression object corresponding to the column name.

Raises:
  • ValueError: If the column name does not exist in the table schema.
def create_pxt_dirs_for_path(table_path: str) -> None:

Create necessary PixelTable directories for a given table path.

Arguments:
  • table_path: The full path of the PixelTable table (e.g., 'dir1.dir2.table_name').
def create_pxt_table_from_sample( table_path: str, sample: dict, unwrap: bool = False, add: dict[str, typing.Any] | None = None, if_exists: Literal['error', 'replace_force'] = 'error') -> pixeltable.catalog.table.Table:

Create a PixelTable table from a sample dictionary.

Arguments:
  • table_path: The full path of the PixelTable table to create (e.g., 'dir1.dir2.table_name').
  • sample: A dictionary representing a single sample, where keys are column names and values are the corresponding data.
  • unwrap: Whether to unwrap (only take first element) tensors/lists in the sample.
  • add: Additional key-value pairs to add to the sample before creating the table.
  • if_exists: Behavior if the table already exists. Options are 'error' or 'replace_force'.
Raises:
  • ValueError: If a key in add already exists in sample.
def set_value_to_idx_rows( table: pixeltable.catalog.table.Table, col_expr: pixeltable.exprs.expr.Expr, idx_expr: pixeltable.exprs.expr.Expr, indices: set[int], value: int | float | str | None) -> None:

Set a column to a specific value for rows with given indices in a PixelTable table.

Arguments:
  • table: The PixelTable table. Must contain an 'idx' column.
  • col_expr: The column expression to be set.
  • idx_expr: The column expression for the index.
  • indices: List of row indices to update.
  • value: The value to set the column to.
class GoldPxtTorchDataset(torch.utils.data.dataset.Dataset[+_T_co], typing.Iterable[+_T_co]):

A Pixeltable PyTorch Dataset that reshapes array columns to their original shapes.

This class intends to solve a current issue in Pixeltable dataset where array columns are flattened when converted to PyTorch datasets. This class reshapes those columns back to their original shapes during iteration.

Attributes:
  • shapes: A dictionary mapping column names of arrays to their original shapes.
  • keep_cache: Whether to keep the cached parquet files after the dataset is garbage collected.
  • path: Path to directory containing cached parquet files.
  • image_format: Image format ('np' or 'pt').
GoldPxtTorchDataset( source: pixeltable.catalog.table.Table | pixeltable._query.Query, keep_cache: bool = False)

Initialize the GoldPxtTorchDataset.

Arguments:
  • source: The PixelTable Table or Query to convert to a PyTorch dataset.
  • keep_cache: Whether to keep the cached parquet files after the dataset is garbage collected.
shapes
keep_cache
def get_array_column_shapes_from_table(table: pixeltable.catalog.table.Table) -> dict[str, tuple[int, ...]]:

Get the shapes of array columns in a PixelTable table.

Arguments:
  • table: The PixelTable table.
Returns:

A dictionary mapping column names of array columns to their shapes.

def get_array_column_shapes_from_query(query: pixeltable._query.Query) -> dict[str, tuple[int, ...]]:

Get the shapes of array columns in a PixelTable Query.

Arguments:
  • query: The PixelTable Query.
Returns:

A dictionary mapping column names of array columns to their shapes.

def get_distinct_value_and_count_in_column( table: pixeltable.catalog.table.Table, col_expr: pixeltable.exprs.expr.Expr) -> dict[typing.Any, int]:

Get distinct values and their counts in a specified column of a PixelTable table.

Arguments:
  • table: The PixelTable table.
  • col_expr: The column expression to analyze.
Returns:

A dictionary mapping distinct values to their counts in the specified column.

def get_column_distinct_ratios( table: pixeltable.catalog.table.Table, col_expr: pixeltable.exprs.expr.Expr) -> dict[str, float]:

Get the ratios of distinct values in a specified column of a PixelTable table.

Arguments:
  • table: The PixelTable table.
  • col_expr: The column expression to analyze.
Returns:

A dictionary mapping distinct values to their ratios in the specified column.

def get_sample_row_from_idx( table: pixeltable.catalog.table.Table, idx: int = 0, idx_key: str = 'idx', collate_fn: Optional[Callable] = None, expected_keys: list[str] | None = None) -> dict[str, typing.Any]:

Get a sample row from a PixelTable table.

Arguments:
  • table: The PixelTable table.
  • idx: The index of the row to retrieve.
  • idx_key: The name of the column representing the index in the table.
  • collate_fn: An optional collate function to apply to the sample.
  • expected_keys: An optional list of expected keys in the sample.
Returns:

A dictionary representing a sample row from the table.

Raises:
  • ValueError: If no sample is found at the specified index.
  • ValueError: If multiple samples are found at the specified index.
  • ValueError: If the sample is missing any of the expected_keys.
def get_valid_table( table: pixeltable.catalog.table.Table | str, minimal_schema: dict[str, type], primary_key: str | None = None) -> pixeltable.catalog.table.Table:

Get a valid PixelTable view for a given table.

Arguments:
  • table: The PixelTable table.
  • minimal_schema: A dictionary representing the minimal expected schema (keys are column names, values are expected types).
Returns:

A valid PixelTable view for the given table.

Raises:
  • ValueError: If the table is missing expected keys, has excluded keys,
  • or if the existing view does not point to the provided table.
def make_batch_ready_for_table( batch: dict[str, typing.Any], to_insert: list[str], index_key: str = 'idx') -> list[dict[str, typing.Any]]:

Unwrap a batch of data into a list of data ready to be included in the table.

Arguments:
  • table: The PixelTable table to include the batch into.
  • batch: A dictionary representing a batch of data (each key corresponds to stacked information).
  • to_insert: A list of keys from the batch to insert or update in the table.
  • index_key: The key in the batch that represents the index for matching rows in the table.
Returns:

A list of dictionaries, each representing a row to be inserted or updated in the table.

Raises:
  • ValueError: If the index key is not found in the batch or if it is included in the to_insert list.
def get_pxt_table_primary_keys(table: pixeltable.catalog.table.Table) -> set[str]:

Get the primary key columns of a PixelTable table.

Arguments:
  • table: The PixelTable table.
Returns:

A set of primary key column names.

def check_pxt_table_has_primary_key(table: pixeltable.catalog.table.Table, cols: set[str]) -> None:

Check if some columns are defined as primary keys in the PixelTable table.

Arguments:
  • table: The PixelTable table.
  • cols: A set of column names to check.
Raises:
  • ValueError: If any of the specified columns are not primary keys in the table.
def get_max_value_in_column( table: pixeltable.catalog.table.Table, col_expr: pixeltable.exprs.expr.Expr) -> int:

Get the maximum value in a specified column of a PixelTable table.

Arguments:
  • table: The PixelTable table.
  • col_expr: The column expression to analyze.
Returns:

The maximum value in the specified column.

Raises:
  • ValueError: If the maximum value is not an integer.