Pancreatic ductal adenocarcinomas often cause marked pancreatic duct dilatation and associated parenchymal atrophy. We present the case of a small pancreatic neuroendocrine tumor with upstream pancreatic duct dilatation and severe parenchymal atrophy. A small enhancing tumor was observed at the head of the pancreas on computed tomography (CT). Endoscopic ultrasonography-guided fine-needle aspiration was negative for malignancy. We performed a pylorus-preserving pancreatoduodenectomy since we could not exclude the presence of pancreatic ductal adenocarcinoma. The pathological and immunohistochemical examination revealed a serotonin-positive neuroendocrine tumor, measured 1.0 x 0.5 cm. The pathological specimen was remarkable for the marked stromal fibrosis in the area of the tumor, which resulted in narrowing of the main pancreatic duct. Here, we report a rare small pancreatic neuroendocrine tumor, the CT image of which resembled that of pancreatic ductal adenocarcinoma, in which the expression of serotonin and associated fibrosis might be a possible mechanism for the marked main pancreatic duct dilatation.