I think a good apology comes from sincerity. That and actually being sorry for the thing that happened as opposed to being caught.
And as the example shows, sometimes you have to apologize because the person I question feels wronged. You may not understand why, but you apologize because the person is hurt and you know it’s important to THEM.