Effects of Data Enrichment with Image Transformations on the Performance of Deep Networks
Main Article Content
Abstract
Images cannot always be expected to come in a certain standard format and orientation. Deep networks need to be trained to take into account unexpected variations in orientation or format. For this purpose, training data should be enriched to include different conditions. In this study, the effects of data enrichment on the performance of deep networks in the super resolution problem were investigated experimentally. A total of six basic image transformations were used for the enrichment procedures. In the experiments, two deep network models were trained with variants of the ILSVRC2012 dataset enriched by these six image transformation processes. Considering a single image transformation, it has been observed that the data enriched with 180 degree rotation provides the best results. The most unsuccessful result was obtained when the models were trained on the enriched data generated by the flip upside down process. Models scored highest when trained with a mix of all transformations.
Downloads
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
References
U. A. Patel and S. Priya, “Development of a student attendance management system using RFID and face recognition: a review,” Int. J. Adv. Res. Comput. Sci. Manag. Stud., vol. 2, no. 8, pp. 109–119, 2014.
H. Temiz and H. S. Bilge, “Super Resolution of B-mode Ultrasound Images with Deep Learning,” IEEE Access, p. 1, 2020, doi: 10.1109/ACCESS.2020.2990344. DOI: https://doi.org/10.1109/ACCESS.2020.2990344
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2016, doi: 10.1109/CVPR.2016.90. DOI: https://doi.org/10.1109/CVPR.2016.90
J. Kim, J. K. Lee, and K. M. Lee, “Deeply-Recursive Convolutional Network for Image Super-Resolution,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, pp. 1637–1645, doi: 10.1109/CVPR.2016.181.
J. Kim, J. K. Lee, and K. M. Lee, “Accurate Image Super-Resolution Using Very Deep Convolutional Networks,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 1646–1654. DOI: https://doi.org/10.1109/CVPR.2016.182
Y. Tai, J. Yang, and X. Liu, “Image Super-Resolution via Deep Recursive Residual Network,” in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 2790–2798, doi: 10.1109/CVPR.2017.298.
G. Huang, Z. Liu, K. Q. Weinberger, and L. Van Der Maaten, “Densely Connected Convolutional Networks,” in CVPR, 2017. DOI: https://doi.org/10.1109/CVPR.2017.243
Y. Zhang, Y. Tian, Y. Kong, B. Zhong, and Y. Fu, “Residual Dense Network for Image Super-Resolution,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018. DOI: https://doi.org/10.1109/CVPR.2018.00262
S. Li, R. Fan, G. Lei, G. Yue, and C. Hou, “A two-channel convolutional neural network for image super-resolution,” Neurocomputing, vol. 9, no. 275, pp. 267–277, 2018, doi: 10.1016/j.neucom.2017.08.041. DOI: https://doi.org/10.1016/j.neucom.2017.08.041
J. Yamanaka, S. Kuwashima, and T. Kurita, “Fast and Accurate Image Super Resolution by Deep CNN with Skip Connection and Network in Network,” Neural Inf. Process., pp. 217–225, Jul. 2017. DOI: https://doi.org/10.1007/978-3-319-70096-0_23
J. Kim, J. K. Lee, and K. M. Lee, “Deeply-recursive convolutional network for image super-resolution,” in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2016, doi: 10.1109/CVPR.2016.181. DOI: https://doi.org/10.1109/CVPR.2016.181
Y. Tai, J. Yang, and X. Liu, “Image super-resolution via deep recursive residual network,” in Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, 2017, doi: 10.1109/CVPR.2017.298. DOI: https://doi.org/10.1109/CVPR.2017.298
W. Shi et al., “Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 1874–1883. DOI: https://doi.org/10.1109/CVPR.2016.207
B. Lim, S. Son, H. Kim, S. Nah, and K. M. Lee, “Enhanced Deep Residual Networks for Single Image Super-Resolution,” in 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2017, pp. 1132–1140, doi: 10.1109/CVPRW.2017.151. DOI: https://doi.org/10.1109/CVPRW.2017.151
R. Timofte et al., “NTIRE 2017 Challenge on Single Image Super-Resolution: Methods and Results,” IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. Work., vol. 2017-July, pp. 1110–1121, 2017, doi: 10.1109/CVPRW.2017.149. DOI: https://doi.org/10.1109/CVPRW.2017.149
C. Dong, C. C. Loy, and X. Tang, “Accelerating the Super-Resolution Convolutional Neural Network,” in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 9906 LNCS, 2016, pp. 391–407. DOI: https://doi.org/10.1007/978-3-319-46475-6_25
Y. Tai, J. Yang, X. Liu, and C. Xu, “Memnet: A persistent memory network for image restoration,” in Proceedings of the IEEE international conference on computer vision, 2017, pp. 4539–4547. DOI: https://doi.org/10.1109/ICCV.2017.486
W.-S. Lai, J.-B. Huang, N. Ahuja, and M.-H. Yang, “Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution,” in IEEE Conference on Computer Vision and Pattern Recognition, 2017, vol. 2 (3), p. 5. DOI: https://doi.org/10.1109/CVPR.2017.618
M. S. M. Sajjadi, B. Scholkopf, and M. Hirsch, “EnhanceNet: Single Image Super-Resolution Through Automated Texture Synthesis,” in Proceedings of the IEEE International Conference on Computer Vision, 2017, vol. 2017-Octob, pp. 4501–4510, doi: 10.1109/ICCV.2017.481. DOI: https://doi.org/10.1109/ICCV.2017.481
B. Liu and D. Ait-Boudaoud, “Effective image super resolution via hierarchical convolutional neural network,” Neurocomputing, vol. 374, pp. 109–116, 2020. DOI: https://doi.org/10.1016/j.neucom.2019.09.035
J. Tang, C. Huang, J. Liu, and H. Zhu, “Image Super-Resolution Based on CNN Using Multilabel Gene Expression Programming,” Appl. Sci., vol. 10, no. 3, p. 854, 2020. DOI: https://doi.org/10.3390/app10030854
C. Ledig et al., “Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network,” 2016, doi: 10.1109/CVPR.2017.19. DOI: https://doi.org/10.1109/CVPR.2017.19
X. Wang et al., “ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks,” in The European Conference on Computer Vision (ECCV) Workshops, 2018. DOI: https://doi.org/10.1007/978-3-030-11021-5_5
X. Hu, X. Liu, Z. Wang, X. Li, W. Peng, and G. Cheng, “RTSRGAN: Real-Time Super-Resolution Generative Adversarial Networks,” in 2019 Seventh International Conference on Advanced Cloud and Big Data (CBD), 2019, pp. 321–326. DOI: https://doi.org/10.1109/CBD.2019.00064
M. Zareapoor, M. E. Celebi, and J. Yang, “Diverse adversarial network for image super-resolution,” Signal Process. Image Commun., vol. 74, pp. 191–200, 2019. DOI: https://doi.org/10.1016/j.image.2019.02.008
M. E. A. Seddik, M. Tamaazousti, and J. Lin, “Generative collaborative networks for single image super-resolution,” Neurocomputing, 2019. DOI: https://doi.org/10.1016/j.neucom.2019.02.068
X. Zhang, H. Song, K. Zhang, J. Qiao, and Q. Liu, “Single image super-resolution with enhanced Laplacian pyramid network via conditional generative adversarial learning,” Neurocomputing, 2019. DOI: https://doi.org/10.1016/j.neucom.2019.04.097
Y. Zhang, S. Liu, C. Dong, X. Zhang, and Y. Yuan, “Multiple cycle-in-cycle generative adversarial networks for unsupervised image super-resolution,” IEEE Trans. Image Process., vol. 29, pp. 1101–1112, 2019. DOI: https://doi.org/10.1109/TIP.2019.2938347
F. Wu, B. Wang, D. Cui, and L. Li, “Single Image Super-Resolution Based on Wasserstein GANs,” in 2018 37th Chinese Control Conference (CCC), 2018, pp. 9649–9653, doi: 10.23919/ChiCC.2018.8484039. DOI: https://doi.org/10.23919/ChiCC.2018.8484039
Z. Zhang, Z. Wang, Z. Lin, and H. Qi, “Image super-resolution by neural texture transfer,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 7982–7991. DOI: https://doi.org/10.1109/CVPR.2019.00817
S. Lian, H. Zhou, and Y. Sun, “FG-SRGAN: A Feature-Guided Super-Resolution Generative Adversarial Network for Unpaired Image Super-Resolution,” in International Symposium on Neural Networks, 2019, pp. 151–161. DOI: https://doi.org/10.1007/978-3-030-22796-8_17
X. Zhu, L. Zhang, L. Zhang, X. Liu, Y. Shen, and S. Zhao, “Generative Adversarial Network-based Image Super-Resolution with a Novel Quality Loss,” in 2019 International Symposium on Intelligent Signal Processing and Communication Systems (ISPACS), 2019, pp. 1–2. DOI: https://doi.org/10.1109/ISPACS48206.2019.8986250
Z. Wang, B. Chen, H. Zhang, and H. Liu, “Variational probabilistic generative framework for single image super-resolution,” Signal Processing, vol. 156, pp. 92–105, 2019. DOI: https://doi.org/10.1016/j.sigpro.2018.10.004
P. Shamsolmoali, M. Zareapoor, R. Wang, D. K. Jain, and J. Yang, “G-GANISR: Gradual generative adversarial network for image super resolution,” Neurocomputing, vol. 366, pp. 140–153, Nov. 2019, doi: 10.1016/j.neucom.2019.07.094. DOI: https://doi.org/10.1016/j.neucom.2019.07.094
X.-J. Mao, C. Shen, and Y.-B. Yang, “Image Restoration Using Very Deep Convolutional Encoder-Decoder Networks with Symmetric Skip Connections,” Adv. Neural Inf. Process. Syst., pp. 2802–2810, Mar. 2016.
C. Dong, C. C. Loy, K. He, and X. Tang, “Image Super-Resolution Using Deep Convolutional Networks,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 38, no. 2, pp. 295–307, 2016, doi: 10.1109/TPAMI.2015.2439281. DOI: https://doi.org/10.1109/TPAMI.2015.2439281
H. Temiz, “An Experimental Study on Hyper Parameters for Training Deep Convolutional Networks,” in 2020 4th International Symposium on Multidisciplinary Studies and Innovative Technologies (ISMSIT), 2020, pp. 1–8. DOI: https://doi.org/10.1109/ISMSIT50672.2020.9254621
Y. Lu, Z. Yang, J. Kannala, and S. Kaski, “Learning Image Relations with Contrast Association Networks.” 2019. DOI: https://doi.org/10.1109/IJCNN.2019.8852344
C. Dong, C. C. Loy, K. He, and X. Tang, “Learning a Deep Convolutional Network for Image Super-Resolution,” in 13th European Conference on Computer Vision, ECCV 2014, 2014, pp. 184–199, doi: 10.1007/978-3-319-10593-2_13. DOI: https://doi.org/10.1007/978-3-319-10593-2_13
A. Krizhevsky, I. Sutskever, and H. Geoffrey E., “ImageNet Classification with Deep Convolutional Neural Networks,” Adv. Neural Inf. Process. Syst. 25, pp. 1–9, 2012, doi: 10.1109/5.726791. DOI: https://doi.org/10.1109/5.726791
F. Chollet, “Keras: The Python Deep Learning library,” Keras.Io, 2015.
M. Abadi et al., “Tensorflow: A system for large-scale machine learning,” in 12th Symposium on Operating Systems Design and Implementation, 2016, pp. 265–283.