Aiming at the problems of small target size and variable shape in steel surface defect detection tasks， low efficiency of traditional detection methods， and difficulty in accurately capturing their feature information with general algorithms， an improved model based on YOLOv8 is proposed. Replacing some standard convolutions with deformable convolutions enables the model to better learn the offset of sampling positions， adapt to geometric transformations of surface defects， and thus improve detection accuracy. Introducing an Inner IoU loss function based on auxiliary bounding boxes， which can self adjust the scaling factor according to different detection tasks to control the generation of auxiliary bounding boxes， is suitable for handling surface defects with variable shapes， and improves model generalization. The experimental results on the benchmark dataset NEU-DET show that the improved model improves detection accuracy by 2.3% compared to the baseline model with the same parameter count， and outperforms other compared algorithms. In addition， on the GC10-DET dataset， the detection accuracy improved by 4.0% compared to the baseline model， indicating that the model has good generalization ability.