The General Introduction distinguishes between parish priests who have the responsibility of pastor, and priests and deacons who do not.
13. It is the duty of parish priests (pastors) to assist the bishop in the instruction and baptism of the adults entrusted to their care, unless the bishop makes other provisions. Parish priests (pastors), with the assistance of catechists or other qualified laypersons, have the duty of preparing the parents and godparents of children through appropriate pastoral guidance and of baptizing the children.
14. Other priests and deacons, since they are co-workers in the ministry of bishops and parish priests (pastors), also prepare candidates for baptism and, by the invitation or consent of the bishop or parish priest (pastor), celebrate the sacrament.
It’s all very hierarchical, but it does give an orderly landscape for responsibility. Bishops share with pastors. Pastors share with lay catechists. Note the reference in GICI 13 is for the baptism of children. Take that as you will.
In order for an ordinary minister to celebrate baptism (or any sacrament, really) that “invitation or consent” from the ordinary pastor is needed.