We present a new approach to segment and classify bacterial spore layers from Transmission Electron Microscopy (TEM) images using a hybrid Convolutional Neural Network (CNN) and Random Forest (RF) classifier algorithm. This approach utilizes deep learning, with the CNN extracting features from images, and the RF classifier using those features for classification. The proposed model achieved 73% accuracy, 64% precision, 46% sensitivity, and 47% F1-score with test data. Compared to other classifiers such as AdaBoost, XGBoost, and SVM, our proposed model demonstrates greater robustness and higher generalization ability for non-linear segmentation. Our model is also able to identify spores with a damaged core as verified using TEMs of chemically exposed spores. Therefore, the proposed method will be valuable for identifying and characterizing spore features in TEM images, reducing labor-intensive work as well as human bias.