> ## Documentation Index
> Fetch the complete documentation index at: https://docs.auremi.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Datasets

> Add a dataset to Auremi and link it to a model, from the console or through your agent.

A dataset is a named set of example files in Auremi. Each file is one example, and each example belongs to a split: train, validation, or test. Every time files are added, Auremi saves a new version of the dataset, and each training run uses one version.

A model can train on any dataset linked to it. A dataset you add from a model is linked to that model automatically. All datasets also sit in a shared library, so you can link one dataset to several models.

## Add a dataset in the console

<Steps>
  <Step title="Open the model's datasets">
    In the [Auremi console](https://auremi.ai), open your model and click the **Datasets** tab.
  </Step>

  <Step title="Create the dataset">
    Click **Add uploaded dataset**, then **Create new**. Enter a **Dataset name**, choose a **Split** (**Train**, **Validation**, or **Test**), and select the files. You can select many files at once.
  </Step>

  <Step title="Upload">
    Click **Add train dataset**. The button is named for the split you chose. The dataset is linked to this model and appears in its list.
  </Step>
</Steps>

To add more files later, click **View** on the dataset, then **+ Add**. Auremi saves a new version with the existing examples plus the new files.

A validation dataset needs two notes for each file: **Expected output annotation**, and **What should validation catch or avoid?**

### From an S3 bucket

Connect an S3 account first, under **Cloud Compute** on the **Storage** tab. Then, on the model's **Datasets** tab, open **S3 bucket** and click **Add S3 dataset**. Choose the **S3 account** and click **Browse prefix**. Under **Folder rules**, files in `train/`, `validation/` and `test/` go to those splits. You can change the split of any file in the list.

## Link an existing dataset to a model

On the model's **Datasets** tab, click **Add uploaded dataset**, then **Use existing**. Tick the datasets you want and click **Add selected**.

A dataset added from **Datasets** in the sidebar goes into the library without a model, so link it this way. On a model's tab, **Remove** unlinks a dataset from that model and keeps it in the library. **Delete** on the sidebar's **Datasets** page deletes the dataset and all its versions.

## Add a dataset through your agent

Put the files in one folder. Each file becomes one example. A file inside a folder named `test` goes to the test split, one inside `validation` or `val` goes to validation, and every other file goes to train.

```text theme={null}
photos/
  train/
    cat-001.png
    cat-002.png
  test/
    cat-900.png
```

<Prompt description="Upload a dataset">
  Upload the images in ./data/photos as a dataset for this model.
</Prompt>

The agent pushes the folder to Auremi, which links it to your model. If the upload is interrupted, the agent can resume it where it stopped.

<Accordion title="What the agent runs">
  ```bash theme={null}
  auremi notebook status   # prints the project ID
  auremi dataset push ./data/photos --project <projectId> --name photos
  auremi dataset list --project <projectId>
  ```
</Accordion>

## Link an existing dataset through your agent

<Prompt description="Link a dataset">
  Use the photos-v2 dataset for this model too.
</Prompt>

<Accordion title="What the agent runs">
  ```bash theme={null}
  auremi dataset list
  auremi dataset link <datasetId> --project <projectId>
  ```
</Accordion>

`auremi dataset list` prints each dataset's ID, with its version IDs indented underneath. `auremi dataset link` takes a dataset ID. Training takes a version ID.

## Train on a dataset

A training run uses one dataset version. If the model has more than one version with training examples, the CLI won't choose, so the agent lists them and asks you which to use. Name a dataset, or tell it to use the newest.

<Prompt description="Train on a dataset">
  Train on the photos dataset.
</Prompt>

<Accordion title="What the agent runs">
  ```bash theme={null}
  auremi dataset list --project <projectId>
  auremi train --dataset <datasetVersionId> --no-run
  auremi train --dataset <datasetVersionId> --wait
  ```
</Accordion>

`--dataset newest` picks the most recently created version.

In the console, click **+ New Training** on the model's **Overview** and choose a **Dataset version**. In a notebook, a training cell has a **Train dataset** menu. See [Notebooks](/notebooks).