In this paper, we propose a Pixel-Training Image-Testing (PTIT) algorithm for joint object segmentation and classification. PTIT consists of four processing stages: sample-pixel (human) labeling, pixel-wise training, image generalization and uniform region refinement. Compared to the state-of-the-art semantic image segmentation and classification algorithms, PTIT significantly reduces the time required for labeling and is capable of handling more complex scenes (cluttered background and large amount of objects of interest). PTIT performance is tested on coral images to show its advantage over the typical segmentation followed by classification image processing framework. For visual analysis, our algorithmically derived class shows class structure better, while also displaying most of time better receiver operating character at the pixel level.