#531

# dataset #531
path: /capstor/store/cscs/swissai/infra01/vision-datasets/tokenized/sft/google_rsrcc
kind: manual | status: active
samples: n/a | size: 3.0 GB | files: 68

fields:
- document_id: int
- component_index: int
- kind: int
- token_offset: int
- token_length: int
- token_hash: text
- resize_height: int
- resize_width: int

tree (truncated):
```
dry_run_stats.json
merged.bin
merged.idx
rank_0000/_SUCCESS
rank_0000/components.000000.parquet
rank_0000/tokens.000000.bin
rank_0000/worker_stats.json
rank_0000_checkpoint.pt
rank_0000_chunk_0000.bin
rank_0000_chunk_0000.idx
rank_0000_stats.json
rank_0001/_SUCCESS
rank_0001/components.000000.parquet
rank_0001/tokens.000000.bin
rank_0001/worker_stats.json
rank_0001_checkpoint.pt
rank_0001_chunk_0000.bin
rank_0001_chunk_0000.idx
rank_0001_stats.json
rank_0002/_SUCCESS
rank_0002/components.000000.parquet
rank_0002/tokens.000000.bin
rank_0002/worker_stats.json
rank_0002_checkpoint.pt
rank_0002_chunk_0000.bin
... (+43 more)
```

samples

1

0

0

1

0

60

fe526c0d5ff7a90a

0

0

2

0

2

1

240

1

e1814151631433b9

0

0

3

0

4

1

244

242

a41775d269a1010a

0

0

4

0

1

0

1212

1065

506335091d3e7d39

512

512

5

0

3

0

5472

1065

e6dfa1fd85dec4ea

512

512

6

1

0

1

9732

60

fe526c0d5ff7a90a

0

0

7

1

2

1

9972

1

e1814151631433b9

0

0

8

1

4

1

9976

132

eb35738a5726bdfa

0

0

9

1

1

0

10504

1065

f82caed94906eea4

512

512

10

1

3

0

14764

1065

b21ed1d1f8c322f0

512

512