Classifying Operator Experience from Electric Screwdriving Signals: A BiLSTM-Based Study with External Validation

Abstract

This study presents a deep learning–based approach to objectively classify operator experience levels (Novice–Intermediate–Expert) from multivariate signals and user interactions obtained during electric screwdriving operations. The dataset comprises 64 participant-specific files, each containing multiple tightening trials. Windowing was performed independently per file; short segments unsuitable for windowing were excluded, yielding 3,326 time windows (2,958 for training/testing; 368 for independent validation). A two-layer Bidirectional LSTM (BiLSTM) architecture was employed and evaluated on both the train–test split and an external validation set constructed from 12 previously unseen files. On the test set, the model achieved 76% overall accuracy with macro-averaged precision/recall/F1 of 77%/76%/76%. Class-wise analysis indicated stronger separability for the Expert class (recall ≈ 84%) and comparatively lower performance for Intermediate (recall ≈ 66%). On the hold-out validation set, accuracy was 75.00%, with a mean predicted probability of 85.0%, indicating moderate-to-high confidence. The findings show that while BiLSTM provides a solid foundation for time-series classification, its effectiveness may be limited for complex patterns without a convolutional front end.