Skip to main content
A dataset is a named set of example files in Auremi. Each file is one example, and each example belongs to a split: train, validation, or test. Every time files are added, Auremi saves a new version of the dataset, and each training run uses one version. A model can train on any dataset linked to it. A dataset you add from a model is linked to that model automatically. All datasets also sit in a shared library, so you can link one dataset to several models.

​
Add a dataset in the console

1

Open the model's datasets

In the Auremi console, open your model and click the Datasets tab.
2

Create the dataset

Click Add uploaded dataset, then Create new. Enter a Dataset name, choose a Split (Train, Validation, or Test), and select the files. You can select many files at once.
3

Upload

Click Add train dataset. The button is named for the split you chose. The dataset is linked to this model and appears in its list.
To add more files later, click View on the dataset, then + Add. Auremi saves a new version with the existing examples plus the new files. A validation dataset needs two notes for each file: Expected output annotation, and What should validation catch or avoid?

​
From an S3 bucket

Connect an S3 account first, under Cloud Compute on the Storage tab. Then, on the model’s Datasets tab, open S3 bucket and click Add S3 dataset. Choose the S3 account and click Browse prefix. Under Folder rules, files in train/, validation/ and test/ go to those splits. You can change the split of any file in the list. On the model’s Datasets tab, click Add uploaded dataset, then Use existing. Tick the datasets you want and click Add selected. A dataset added from Datasets in the sidebar goes into the library without a model, so link it this way. On a model’s tab, Remove unlinks a dataset from that model and keeps it in the library. Delete on the sidebar’s Datasets page deletes the dataset and all its versions.

​
Add a dataset through your agent

Put the files in one folder. Each file becomes one example. A file inside a folder named test goes to the test split, one inside validation or val goes to validation, and every other file goes to train.

Upload a dataset

The agent pushes the folder to Auremi, which links it to your model. If the upload is interrupted, the agent can resume it where it stopped.

Link a dataset

auremi dataset list prints each dataset’s ID, with its version IDs indented underneath. auremi dataset link takes a dataset ID. Training takes a version ID.

​
Train on a dataset

A training run uses one dataset version. If the model has more than one version with training examples, the CLI won’t choose, so the agent lists them and asks you which to use. Name a dataset, or tell it to use the newest.

Train on a dataset

--dataset newest picks the most recently created version. In the console, click + New Training on the model’s Overview and choose a Dataset version. In a notebook, a training cell has a Train dataset menu. See Notebooks.