Module
View source
do.py
import goldener.do
class
GoldDoer:
Shared initialization for Goldener's data orchestration tools.
Arguments:
- allow_existing: Whether to allow using existing tables. Defaults to True.
- drop_table: Whether to drop intermediate tables. Defaults to False.
- max_batches: Optional maximum number of batches to process.
Shared initialization for GoldDoers that populate intermediate tables.
Arguments:
- allow_existing: Whether to allow using existing tables. Defaults to True.
- drop_table: Whether to drop intermediate tables. Defaults to False.
- max_batches: Optional maximum number of batches to process.
- collate_fn: Function to collate dataset samples into batches. If None,
collate_keeping_sequences_as_sequencesis used. - to_keep_schema: Optional schema for additional columns to preserve.
- min_pxt_insert_size: Minimum number of rows to accumulate before inserting into PixelTable. Defaults to 100.
- batch_size: Batch size used when iterating over the data. Defaults to 1.
- num_workers: Number of workers for the PyTorch DataLoader. Defaults to 0.
GoldDoerWithTable( allow_existing: bool = True, drop_table: bool = False, max_batches: int | None = None, collate_fn: Optional[Callable] = None, to_keep_schema: dict[str, type] | None = None, min_pxt_insert_size: int = 100, batch_size: int = 1, num_workers: int = 0)