Skip to content

xiangjieSui/Awesome-360-processing

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

97 Commits
 
 

Repository files navigation

Awesome-360-processing

Awesome works and resources relevant to 360 processing. Insipred by awesome-360-vision.

Welcome to pull.

360 saliency prediction

Image

Scanpath prediction

  • SaltiNet: Scan-path prediction on 360 degree images using saliency volumes
    Marc Assens, Kevin McGuinness, Xavier Giro-i-Nieto, Noel E. O'Connor
    IEEE International Conference on Computer Vision Workshops (ICCVW), 2017.
    [pdf] [code]

  • The prediction of head and eye movement for 360 degree images
    Zhu, Yucheng, Guangtao Zhai, and Xiongkuo Min
    Signal Processing: Image Communication (SPIC), 2018.
    [pdf]

  • PathGAN: Visual scanpath prediction with generative adversarial networks
    Marc Assens, Xavier Giro-i-Nieto, Kevin McGuinness, and Noel E. O’Connor
    European Conference on Computer Vision (ECCV), 2018.
    [pdf] [code]

  • ScanGAN360: A generative model of realistic scanpaths for 360° images
    Daniel Martin, Ana Serrano, Alexander W. Bergman, Gordon Wetzstein, Belen Masia
    IEEE Transactions on Visualization and Computer Graphics (TVCG), 2022.
    [project]

  • ScanDMM: A deep Markov model of scanpath prediction for 360° Images
    Xiangjie Sui, Yuming Fang, Hanwei Zhu, Shiqi Wang, Zhou Wang
    IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
    [pdf][project]

Salient regions prediction

  • Saliency in VR: How do people explore virtual environments?
    Vincent Sitzmann, Ana Serrano, Amy Pavel, Maneesh Agrawala, Diego Gutierrez, Belen Masia, Gordon Wetzstein
    IEEE Transactions on Visualization and Computer Graphics (TVCG), 2018.
    [pdf][code]

  • SalGAN360: Visual saliency prediction on 360 degree images with generative adversarial networks
    Fang-Yi Chao, Lu Zhang, Wassim Hamidouche, and Olivier Deforges
    IEEE International Conference on Multimedia & Expo Workshops (ICMEW), 2018.
    [pdf] [code]

  • SalNet360: Saliency maps for omnidirectional images with CNN
    Rafael Monroy, Sebastian Lutz, Tejo Chalasani, and Aljosa Smolic
    Signal Processing: Image Communication (SPIC), 2018.
    [pdf] [code]

  • A novel superpixel-based saliency detection model for 360-degree images
    Fang, Yuming, Xiaoqiang Zhang, and Nevrez Imamoglu
    Signal Processing: Image Communication (SPIC), 2018.
    [pdf] [code]

  • SalGCN: Saliency prediction for 360-degree images based on spherical graph convolutional networks
    Haoran Lv, Qin Yang, Chenglin Li, Wenrui Dai, Junni Zou, Hongkai Xiong
    ACM International Conference on Multimedia (ACMM), 2020.
    [pdf]

  • Stage-wise salient object detection in 360° omnidirectional image via object-level semantical saliency ranking
    Guangxiao Ma, Shuai Li, Chenglizhao Chen, Aimin Hao, Hong Qin
    IEEE Transactions on Visualization and Computer Graphics (TVCG), 2020.
    [pdf][code + dataset]

  • A multi-FoV viewport-based visual saliency model using adaptive weighting losses for 360◦ images
    Fang-Yi Chao, Lu Zhang, Wassim Hamidouche, Olivier Déforges
    IEEE Transactions on Multimedia (TMM), 2020.
    [pdf] [code]

  • SalBiNet360: Saliency prediction on 360° images with local-global bifurcated deep network
    Dongwen Chen, Chunmei Qing, Xiangmin Xu, Huansheng Zhu
    IEEE Conference on Virtual Reality and 3D User Interfaces (VR), 2020.
    [pdf]

Dataset or Tools

  • A dataset of head and eye movements for 360 degree images [salient360!]
    Yashas Rai, Jesús Gutiérrez, and Patrick Le Callet
    ACM Multimedia Systems Conference (ACMMSys), 2017.
    [pdf] [project]

  • A fixation-based 360° benchmark dataset for salient object detection
    Yi Zhang, Lu Zhang, Wassim Hamidouche, Olivier Deforges
    IEEE International Conference on Image Processing (ICIP), 2020.
    [pdf][dataset]


Video

Scanpath prediction

  • Pano2vid: Automatic cinematography for watching 360° videos
    Yu-Chuan Su, Dinesh Jayaraman, and Kristen Grauman
    Asian Conference on Computer Vision (ACCV), 2016.
    [pdf] [code] [dataset]

  • Making 360° video watchable in 2D: learning videography for click free viewing
    Yu-Chuan Su, Kristen Grauman
    IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
    [pdf] [code]

  • Deep 360 pilot: Learning a deep agent for piloting through 360° sports video
    Hou-Ning Hu, Yen-Chen Lin, Ming-Yu Liu, Hsien-Tzu Cheng, Yung-Ju Chang, Min Sun
    IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
    [pdf] [code + dataset]

  • Tell me where to look: Investigating ways for assisting focus in 360° video
    Yen-Chen Lin, Yung-Ju Chang, Hou-Ning Hu, Hsien-Tzu Cheng, Chi-Wen Huang, Min Sun
    Conference on Human Factors in Computing Systems (CHI), 2017.
    [pdf] [project]

  • Your attention is unique: Detecting 360-degree video saliency in head-mounted display for head movement prediction
    Nguyen, Anh, Zhisheng Yan, and Klara Nahrstedt
    ACM international conference on Multimedia (ACMM), 2018.
    [pdf] [code]

  • Predicting head movement in panoramic video: A deep reinforcement learning approach
    Mai Xu, Yuhang Song, Jianyi Wang, Minglang Qiao, Liangyu Huo, and Zulin Wang
    IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2019.
    [pdf] [code]

  • Interactive and automatic navigation for 360° video playback
    Kyoungkook Kang and Sunghyun Cho
    ACM Transactions on Graphics (TOG), 2019.
    [pdf] [project]

  • Attention-based deep reinforcement learning for virtual cinematography of 360° videos
    Jianyi Wang, Mai Xu, Lai Jiang, Yuhang Song
    IEEE Transactions on Multimedia (TMM), 2020.
    [pdf]

  • DGaze: CNN-based gaze prediction in dynamic scenes
    Zhiming Hu, Sheng Li, Congyi Zhang, Kangrui Yi, Guoping Wang, Dinesh Manocha
    IEEE Transactions on Visualization and Computer Graphics (TVCG), 2020.
    [pdf] [code] [project]

  • A spherical convolution approach for learning long term viewport prediction in 360 immersive video
    Chenglei Wu, Rui-Xiao Zhang, Zhi Wang, Lifeng Sun
    AAAI Conference on Artificial Intelligence (AAAI), 2020.
    [pdf] [code]

  • Transitioning360: Content-aware NFoV virtual camera paths for 360° video playback
    Miao Wang, Yi-Jun Li, Wen-Xuan Zhang
    International Symposium on Mixed and Augmented Reality (ISMAR), 2020.
    [pdf]

  • Scanpath prediction in panoramic videos via expected code length minimization
    Mu Li, Kanglong Fan, and Kede Ma
    arXiv:2305.02536, 2023.
    [pdf]

Salient regions prediction

  • Cube padding for weakly-supervised saliency prediction in 360° videos
    Hsien-Tzu Cheng, Chun-Hung Chao, Jin-Dong Dong, Hao-Kai Wen, Tyng-Luh Liu, Min Sun
    IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
    [pdf] [code]

  • V-BMS360: A video extention to the BMS360 image saliency model
    Lebreton, Pierre, Stephan Fremerey, and Alexander Raake
    IEEE International Conference on Multimedia & Expo Workshops (ICMEW), 2018.
    [pdf] [code]

  • Saliency detection in 360° videos
    Ziheng Zhang, Yanyu Xu, Jingyi Yu
    European Conference on Computer Vision (ECCV), 2018.
    [pdf] [code]

  • Video Saliency Prediction Based on Spatial Temporal Two-Stream Network
    Kao Zhang and Zhenzhong Chen
    IEEE Transactions on Circuits and Systems for Video Technology (TCSVT), 2019.
    [pdf] [code]

  • Viewport-dependent saliency prediction in 360° video
    Minglang Qiao, Mai Xu, Zulin Wang, Ali Borji
    IEEE Transactions on Multimedia (TMM), 2020.
    [pdf]

  • ATSal: An attention based architecture for saliency prediction in 360° videos
    Yasser Dahou, Marouane Tliba, Kevin McGuinness, Noel O'Connor
    arxiv, 2020.
    [pdf] [code]

Dataset or Tools

  • A saliency dataset for 360-degree videos
    Anh Nguyen and Zhisheng Yan
    ACM Multimedia Systems Conference (ACMMSys), 2019.
    [pdf] [project]

360 visual quality assessment

2D-Plane-based

  • [WS-PSNR] Weighted-to-spherically-uniform quality evaluation for omnidirectional video
    Yule Sun, Ang Lu, Lu Yu
    IEEE Signal Processing Letters (SPL), 2017.
    [pdf]

  • [W-SSIM/WMS-SSIM] Subjective and objective quality assessment of omnidirectional video
    Francisco Lopes, João Ascenso, António Rodrigues, Maria Paula Queluz
    Applications of Digital Image Processing, 2018.
    [pdf]

  • [CPP-PSNR] Quality metric for spherical panoramic video
    Vladyslav Zakharchenko, Kwang Pyo Choi, Jeong Hoon Park
    Optics and Photonics for Information Processing, 2016.
    [pdf]

  • Deep virtual reality image quality assessment with human perception guider for omnidirectional image
    Hak Gu Kim, Heoun-Taek Lim, Yong Man Ro
    IEEE Transactions on Circuits and Systems for Video Technology (TCSVT), 2019.
    [pdf]

Shpere-based

  • [S-PSNR] A framework to evaluate omnidirectional video coding schemes
    Matt Yu, Haricharan Lakshman, Bernd Girod
    IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 2015.
    [pdf]

  • [S-SSIM] Spherical structural similarity index for objective omnidirectional video quality assessment
    Sijia Chen, Yingxue Zhang, Yiming Li, Zhenzhong Chen, Zhou Wang
    IEEE International Conference on Multimedia and Expo (ICME), 2018.
    [pdf]

Viewport-based

  • Saliency-driven omnidirectional imaging adaptive coding: Modeling and assessment
    Guilherme Luz, João Ascenso, Catarina Brites, Fernando Pereira
    International Workshop on Multimedia Signal Processing (MMSP), 2017.
    [pdf]

  • Assessing visual quality of omnidirectional videos
    Mai Xu, Chen Li, Zhenzhong Chen, Zulin Wang, Zhenyu Guan
    IEEE Transactions on Circuits and Systems for Video Technology (TCSVT), 2019.
    [pdf]

  • Quality assessment of stereoscopic 360-degree images from multi-viewports
    Jiahua Xu, Ziyuan Luo, Wei Zhou, Wenyuan Zhang, Zhibo Chen
    IEEE Picture Coding Symposium (PCS), 2019.
    [pdf]

  • Bridge the gap between VQA and human behavior on omnidirectional video: A large-scale dataset and a deep learning model
    Chen Li, Mai Xu, Xinzhe Du, Zulin Wang
    ACM international conference on Multimedia (ACMM), 2018.
    [pdf] [dataset]

  • Viewport proposal CNN for 360° video quality assessment
    Chen Li, Mai Xu, Lai Jiang, Shanyi Zhang, Xiaoming Tao
    IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
    [pdf] [code]

  • MC360IQA: A multi-channel CNN for blind 360-degree image quality assessment
    Wei Sun, Weike Luo, Xiongkuo Min, Guangtao Zhai, Xiaokang Yang, Ke Gu, Siwei Ma
    IEEE Journal of Selected Topics in Signal Processing (JSTSP), 2019.
    [pdf]

  • Blind omnidirectional image quality assessment with viewport oriented graph convolutional networks
    Jiahua Xu, Wei Zhou, Zhibo Chen
    IEEE Transactions on Circuits and Systems for Video Technology (TCSVT), 2020.
    [pdf]

  • Perceptual quality assessment of omnidirectional images as moving camera videos
    Xiangjie Sui, Kede Ma, Yiru Yao, Yuming Fang
    IEEE Transactions on Visualization and Computer Graphics (TVCG), 2021.
    [pdf] [code]

  • Viewport-based omnidirectional video quality assessment: Database, modeling and inference
    Yu Meng, Zhan Ma
    IEEE Transactions on Circuits and Systems for Video Technology (TCSVT), 2021.
    [pdf] [project]

  • Omnidirectional image quality assessment by distortion discrimination assisted multi-stream network
    Yu Zhou, Yanjing Sun, Leida Li, Ke Gu, and Yuming Fang
    IEEE Transactions on Circuits and Systems for Video Technology (TCSVT), 2022. [pdf]

  • Perceptual quality assessment of omnidirectional images
    Yuming Fang, Liping Huang, Jiebin Yan1, Xuelin Liu, Yang Liu
    AAAI Conference on Artificial Intelligence (AAAI), 2022.
    [pdf]

  • Subjective quality assessment of user-generated videos
    Yuming Fang, Yiru Yao, Xiangjie Sui, and Kede Ma
    IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), 2023.
    [pdf][Database]

  • Perceptual quality assessment of virtual reality videos in the wild
    Wen Wen, Mu Li, Yiru Yao, Xiangjie Sui, Yabin Zhang, Long Lan, Yuming Fang, Kede Ma
    arXiv:2206.08751, 2022.
    [pdf]

  • Assessor360: Multi-sequence network for blind omnidirectional image quality assessment
    Tianhe Wu, Shuwei Shi, Haoming Cai, Mingdeng Cao, Jing Xiao, Yinqiang Zheng and Yujiu Yang
    arXiv:2305.10983, 2023.
    [pdf][code]

  • Perceptual quality assessment of 360° images based on generative scanpath representation
    Xiangjie Sui, Hanwei Zhu, Xuelin Liu, Yuming Fang, Shiqi Wang, Zhou Wang
    arXiv:2309.03472, 2023.
    [pdf][code]

360 sickness

  • VRSA net: VR sickness assessment considering exceptional motion for 360° VR video
    Hak Gu Kim, Heoun-Taek Lim, Sangmin Lee, Yong Man Ro
    IEEE Transactions on Image Processing (TIP), 2018.
    [pdf]

  • VR sickness versus VR presence: A statistical prediction model
    Woojae Kim, Sanghoon Lee, Alan Conrad Bovik
    IEEE Transactions on Image Processing (TIP), 2020.
    [pdf]

  • Assessing individual VR sickness through deep feature fusion of VR video and physiological response
    Sangmin Lee, Seongyeop Kim, Hak Gu Kim, Yong Man Ro
    IEEE Transactions on Circuits and Systems for Video Technology (TCSVT), 2021.
    [pdf]

360 transmission

  • Graph learning based head movement prediction for interactive 360 video streaming
    Xue Zhang, Gene Cheung, Yao Zhao, Patrick Le Callet, Chunyu Lin, and Jack Z. G. Tan
    IEEE Transactions on Image Processing (TIP), 2021.
    [pdf]

  • QoE evaluation methods for 360-degree VR video transmission
    Zesong Fei, Fei Wang, Jing Wang, Xiang Xie
    IEEE Journal of Selected Topics in Signal Processing (JSTSP), 2020.
    [pdf]

  • A Log-Rectilinear transformation for foveated 360-degree video streaming
    David Li, Ruofei Du, Adharsh Babu, Camelia Brumar, and Amitabh Varshney
    IEEE Transactions on Visualization and Computer Graphics (TVCG Honorable Mentions), 2021.
    [pdf] [project] [code]

About

Awesome works and resources are relevant to 360 processing.

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published