Scripture does not specify that deacons are to attend to the "temporal affairs". Deacons are servants; that's what the word means. Elders are also, by extension, servants, but their term encompasses leadership. Also, in Scripture, pastors are never identified as elders.