datasets

DatasetSource

Abstract base class for dataset sources.

Source

class DatasetSource(ABC)

Methods

resolve

Resolve the source to a local path, downloading if necessary.

Source

@abstractmethod
def resolve(self, dataset_name: str) -> Path

dataset_name str

The name of the dataset to resolve (e.g., “side-tasks-3/mixed-20”)

Returns

Path to the local dataset directory

is_cached

Check if the dataset is already cached locally.

Source

@abstractmethod
def is_cached(self, dataset_name: str) -> bool

dataset_name str

The name of the dataset to check

Returns

True if the dataset is cached, False otherwise

resolve_source

Resolve a source string to a DatasetSource instance.

Supports the following formats:

• Local paths: “./datasets”, “../full-repo-datasets/datasets”, “datasets”
• GitHub protocol: “github:owner/repo/path[@ref]” or just “github:owner/repo” (defaults to ref=“main”, path=“datasets”)
• GitHub URL: “https://github.com/owner/repo/tree/ref/path” or just “https://github.com/owner/repo/” (defaults to ref=“main”, path=“datasets”)

Source

def resolve_source(source: str) -> DatasetSource

source str

Source string to resolve

Returns

DatasetSource instance

Raises

• ValueError: If the source format is not recognized