I think you should better describe your observation, because the sun's apparent size can depend on several factors. But I understand that on a clear day, the sun will appear smallest when it is overhead (around noon), and largest at sunrise and sunset.
My thinking for this is that our clearest view of the sun is at midday, when it is mostly overhead. As it moves further away, there is diffraction of sunlight due to the additional atmosphere it has to penetrate to reach you (atmosphere is thicker closer to the ground, so is easier to penetrate top-down, than horizontally), which can actually make the sun appear larger despite it being further away. Note also it is usually easier to look at the sun at sunset rather than midday, for the same reasons.