Introduction

This page introduces Toyota Smarthome dataset. Smarthome has been recorded in an apartment equipped with 7 Kinect v1 cameras. It contains the common daily living activities of 18 subjects. The subjects are senior people in the age range 60-80 years old. The dataset has a resolution of 640×480 and offers 3 modalities: RGB + Depth + 3D Skeleton. The 3D skeleton joints were extracted from RGB. For privacy-preserving reasons, the face of the subjects is blurred. Currently,  two versions of the dataset are provided: Toyota Smarthome Trimmed and Toyota Smarthome Untrimmed.

Toyota Smarthome Trimmed has been designed for the activity classification task of 31 activities. The videos were clipped per activity, resulting in a total of 16,115 short RGB+D video samples.  activities were performed in a natural manner. As a result, the dataset poses a unique combination of challenges: high intra-class variation, high-class imbalance, and activities with similar motion and high duration variance. Activities were annotated with both coarse and fine-grained labels. These characteristics differentiate Toyota Smarthome Trimmed from other datasets for activity classification. [Paper Link][Supp]

The data loader and evaluation codes for Toyota Smarthome Trimmed can be found here:

Toyota Smarthome Untrimmed (TSU) is targeting the activity detection task in long untrimmed videos. Therefore, in TSU, we kept the entire recording when the person is visible. The dataset contains 536 videos with an average duration of 21 mins. Since this dataset is based on the same footage video as Toyota Smarthome Trimmed version, it features the same challenges and introduces additional ones. To better tackle the real-world challenges in the untrimmed video, we densely annotate the dataset with 51 activities. [Paper Link], [TPAMI version]

A demo of the proposed action detection method on the TSU dataset is available at this [link].

The pre-extracted feature and working pipeline for Toyota Smarthome Untrimmed can be found below. The features are pre-trained on the training set of the data:

The visualization for the skeleton data on Smarthome is provided in this [repo].

Both datasets are available on request. Please fill out the form below to obtain the original videos. For more details mail us at toyotasmarthome@inria.fr

News

  • 2019/11/01  The Toyota Smarthome Trimmed is accepted to ICCV’19. The data is available for request.
  • 2020/12/01  The Toyota Smarthome Untrimmed (TSU) dataset is released. The data is available for request.
  • 2021/01/05  We update the skeleton data (V1.2) for the Toyota Smarthome Trimmed. The new skeleton is based on our Pose Refinement method.
  • 2022/04/11  The Toyota Smarthome Untrimmed (TSU) has been accepted to T-PAMI. Some new features for this dataset will be updated soon. Stay tuned!
  • 2022/04/24  We released the pipeline codes including the data loader and evaluation metrics for Toyota Smarthome Trimmed: for RGB version (I3D): [link], for Skeleton version (2S-AGCN) [link].
  • 2022/05/16  We released the Toyota Smarthome Untrimmed’s pre-extracted feature and the running pipeline.
  • 2022/07/18  To address the different research interests for the action detection task, we released the Balanced and Joint-view TSU in this repo. By default, the TSU dataset is unbalanced and unsynchronized.
  • 2022/09/01 We have released a repo for the pose visualization on the Smarthome dataset.
  • 2022/09/19 We have released the I3D feature extractor along with the pre-trained model for TSU (link)

Samples of Activities in Toyota Smarthome Trimmed

Sample of Activities and Videos in Toyota Smarthome Untrimmed

License

The dataset is provided for academic research only. The full license can be found here. Please read carefully the terms and conditions of the license and any accompanying documentation before you download and/or use the Toyota Smarthome dataset. By downloading and/or using the Data, you acknowledge that you have read these terms and conditions, understand them, and agree to be bound by them. If you do not agree with these terms and conditions, you must not download and/or use the Software. Any infringement of the terms of this agreement will automatically terminate your rights under this License.

Toyota Smathome Data Request

Toyota Smathome Data Request Form

Please fill-up the form below to download the dataset. By clicking the submit button you acknowledge that you have read the license (link), understand it and agree to be bound by it. To ensure academic usage, please fill in your work email address, the request from the personal email address will be ignored (e.g. gmail). If your work/school email address is using Gmail, please provide your supervisor's name and email in the project description. Before your submission, please double check your email address !! If you do not receive the data download link or feedback in 3 days, please contact toyotasmarthome@inria.fr or fill in this google form.

Agreement *

*This dataset complies with GDPR European Regulation.

Comments are closed.

  • Bibtex of Toyota Smarthome Trimmed Dataset:

    @InProceedings{Das_2019_ICCV,
    author = {Das, Srijan and Dai, Rui and Koperski, Michal and Minciullo, Luca and Garattoni, Lorenzo and Bremond, Francois and Francesca, Gianpiero},
    title = {Toyota Smarthome: Real-World Activities of Daily Living},
    booktitle = {The IEEE International Conference on Computer Vision (ICCV)},
    month = {October},
    year = {2019}
}

  • Bibtex of Toyota Smarthome Untrimmed Dataset:

    @misc{Dai_2020_arxiv,
    author = {Dai, Rui and Das, Srijan and Sharma, Saurav and Minciullo, Luca and Garattoni, Lorenzo and Bremond, Francois and Francesca, Gianpiero},
    title = {Toyota Smarthome Untrimmed: Real-World Untrimmed Videos for Activity Detection}, 
    year = {2020}, 
    eprint = {2010.14982}, 
    archivePrefix = {arXiv}, 
    primaryClass = {cs.CV}
}


    @ARTICLE{Dai_2022_PAMI,
  author={Dai, Rui and Das, Srijan and Sharma, Saurav and Minciullo, Luca and Garattoni, Lorenzo and Bremond, Francois and Francesca, Gianpiero},
  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence}, 
  title={Toyota Smarthome Untrimmed: Real-World Untrimmed Videos for Activity Detection}, 
  year={2022},
  volume={},
  number={},
  pages={1-1},
  doi={10.1109/TPAMI.2022.3169976}}