Tomato Maturity Detection Method Based on Visual and Haptic Double Transfer Learning

ZHANG Peng, DU Dongfeng, LI Shuang, SHAN Dongri, CHEN Zhenxue

Abstract

Aiming at the problem that tomato ripeness cannot be accurately recognized by relying only on visual technology in the current automated picking process, a tomato ripeness detection method based on visual-haptic dual migration learning was proposed. The method firstly adopted the visual and haptic double transfer learning fusion algorithm as the feature extraction fusion module to solve the problem of not effectively extracting tomato feature information. Secondly, the soft parameter sharing-multilabel classification method was used as the classification module to avoid overfitting by increasing the correlation between different classification tasks. Focusing on tomato varieties that ripened to a single color, such as red and yellow fruits, experimental studies on a newly developed visual and tactile dataset were conducted. The experiments showed that the parameter count of the soft parameter sharing-multilabel detection model was 1.882×107, and the ripeness AUC score reached 0.977 3. Compared with the detection models of uncertainty weighted loss, adaptive hard parameter sharing, cross-stitch network, and soft parameter sharing, the parameter counts dropped by 3.08×106, 6.16×106, 3.08×106, and 3.08×106, and the ripeness AUC scores increased by 0.017 5, 0.017 9, 0.026 7 and 0.008 9, respectively. This indicated that the method improved the detection of tomato ripeness during automated picking to a certain extent and provided an effective solution to the tomato ripeness detection problem.

 

Keywords: tomato maturity, machine vision, machine haptics, double transfer learning, soft parameter sharing-multilabel

 

Download Full Text:

PDF


References


HUANG Yuping, LU Renfu, QI Chao, et al. Tomato maturity classification based on spatially resolved spectra [J]. Spectroscopy and Spectral Analysis, 2018, 38(7) ; 2183 -2188.

MA Zenghong, TAN Li, ZENG Wei, et al. Analytical method and evaluation of tomato size and posture based on visual and tactile perception[J]. Transactions of the Chinese Society for Agricultural Machinery, 2024, 55(1) ;223 -232.

YANG Xinting, LIU Tong, HAN Jiawei, et al. Low temperature storage tomato maturity recognition and time series prediction based on Swin Transformer - GRU [J]. Transactions of the Chinese Society for Agricultural Machinery, 2024, 55(3) :213 - 220. (in Chinese)

BEGUM N, HAZARIKA M K. Maturity detection of tomatoes using transfer learning [J]. Measurement; Food, 2022, 7; 100038.

PANGILINAN J R, LEGASPI J, LINSANGAN N. InceptionV3, ResNet50, and VGG19 performance comparison on tomato ripeness classification [С] // 2022 5th International Seminar on Research of Information Technology and Intelligent Systems (ISRITI). IEEE, 2022; 619 -624.

RAFIQ A, MAKROO H A, HAZARIKA M K. Artificial neural network-based image analysis for evaluation of quality attributes of agricultural produce[J]. Journal of Food Processing and Preservation, 2016, 40(5) : 1010 - 1019.

SHAO Yuanyuan, WANG Yongxian, XUAN Guantao, et al. Visual detection of SSC and firmness and maturity prediction for Feicheng peach by using hyperspectral imaging [J] . Transactions of the Chinese Society for Agricultural Machinery, 2020, 51(8); 344 - 350.

HUANG Y, XIONG J, JIANG X, et al. Assessment of firmness and soluble solids content of peaches by spatially resolved spectroscopy with a spectral difference technique [J]. Computers and Electronics in Agriculture, 2022,200; 107212.

VALEO C, CRRISOSTO С H, SLAUGHTER D. Relationship between nondestructive firmness measurements and commercially important ripening fruit stages for peaches, nectarines and plums [J]. Postharvest Biology and Technology, 2007, 44(3); 248 -253.

SCIMECA L, MAILINO P, CARDIN-CATALAN D, et al. Non-destructive robotic assessment of mango ripeness via multi¬point soft haptics[ C] // 2019 International Conference on Robotics and Automation (ICRA). IEEE, 2019; 1821 - 1826.

HE L, TAO L, MA Z, et al. Cherry tomato firmness detection and prediction using a vision-based tactile sensor [J]. Journal of Food Measurement and Characterization, 2024, 18(2); 1053 - 1064.

WANG Junping, XU Gang. Research on tomato maturity detection method based on machine vision and electronic nose fusion [J]. Food & Machinery ,2022(2) ; 148 - 152.

ALOM M Z, ТАНА T M, YAKOPCIC C, et al. The history began from alexnet; a comprehensive survey on deep learning approaches [J]. arXiv Preprint, arXiv; 1803. 01164, 2018.

TANG P, WANG H, KWONG S. G-MS2F; GoogLeNet based multi-stage feature fusion of deep CNN for scene recognition [J]. Neurocomputing, 2017, 225 : 188 -197.

WANG L, GUO S, HUANG W, et al. Places205-VGGNet models for scene recognition [J]. arXiv Preprint, arXiv;1508. 01667, 2015.

TARG S, ALMEIDAI D, LYMANY K. Resnet in resnet: generalizing residual architectures[J]. arXiv Preprint, arXiv; 1603. 08029, 2016.

IANDOLA F, MOSKEWICZ M, KARAYEV S, et al. DenseNet: implementing efficient convnet descriptor pyramids [J]. arXiv Preprint, arXiv:1404. 1869, 2014.

CHU V, MCMAHON I, RIANO L. Robotic learning of biological objectives through physical interaction [J]. Robotics and Autonomous Systems, 2015, 63; 279 -292.

BEDNAREK M, KICKI P, WALAS K. On robustness of multi-modal fusion-robotics perspective [J]. Electronics, 2020, 9(7): H52.2023, 40(1): 91 -96.

FENG Xingjie, ZHANG Xinyang, FENG Xiaorong. Joint event extraction method based on soft parameter sharing [J]. Application Research of Computers, 2023, 40(1) ; 91 -96. (in Chinese)

SEHGAL P, GOEL N. Auto-annotation of tomato images based on ripeness and firmness classification for multimodal retrieval [C] / / 2016 International Conference on Advances in Computing, Communications and Informatics (ICACCI). IEEE, 2016: 1084-1091.

MIAO Ronghui, LI Zhiwei, WU Jinlong. Lightweight maturity detection of cherry tomato based on improved YOLO v7 [J]. Transactions of the Chinese Society for Agricultural Machinery, 2023, 54( 10) ; 225 -233. (in Chinese)

GAO Y, HENDRICKS L A, KUCHENBECKER К J, et al. Deep learning for tactile understanding from visual and haptic data [C]//2016 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2016: 536 -543.

WANG H, JIN X, DU Y, et al. Adaptive hard parameter sharing method based on multi-task deep learning [J]. Mathematics, 2023, 11(22): 4639.

KENDALL A, GAL Y, CIOLLA R. Multi-task learning using uncertainty to weigh losses for scene geometry and senmntics [C] // Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018: 7482 -7491.

MISRA I, SHIVASTAVAH A, GUPTA A, et al. Cross-stitch networks for multi-task learning [С] //Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,2016: 3994 -4003.


Refbacks

  • There are currently no refbacks.