Why aren't my Domain Controllers Global Catalogs?

Today was interesting.  A customer just promoted some new Server 2019 DCs, but those DCs wouldn't become global catalogs.  They were logging this event in the Applications and Services >> Directory Service Log.

Event 1578, ActiveDirectory_DomainService: "Promotion of the local domain controller to a global catalog has been delayed because the directory partition occupancy requirements have not been met."

In another event...

"As a precondition to becoming a global catalog server, a domain controller must host a read-only replica of all directory partitions in the forest. This event occurs because a Knowledge Consistency Checker (KCC) task has not been completed or because the domain controller cannot add a replica of the directory partition because of unavailable source domain controllers. An attempt to add the replica will occur again at the next KCC interval."

Digging in, the underlying problem is they had an old "abandoned" child domain in AD.  All of the domain controllers had been removed from the child domain, but for the last one they didn't check the "This is the last domain controller in the domain" box.  That left the domain partition and some other cruft laying around.  The new GCs couldn't replicate this partition, there were no DCs to replicate from.  QED they couldn't become global catalogs.

The solution was to use ntdsutil to perform metadata cleanup for the busted domain.  That's documented here How to remove orphaned domains from Active Directory - Windows Server | Microsoft Learn and it usually works.

This time it didn't work.  It failed with 

DsRemoveDsDomainW error 0x2015(The directory service can perform the requested operation only on a leaf object.)

Some digging later we found the problem was the domainDnsZones naming context was a child object of the orphaned domain NC and it wasn't possible to remove the parent without removing the child first.  ntdsutil was able to do that too.  After you connect to the domain naming FSMO, quit back to the ntdsutil main menu, select Partition Management, and remove the DomainDnsZones Partition from there.  With that done you can quit back to the ntdsutil main menu and run metadata cleanup in the normal way to remove the orphaned domain.

With the orphaned domain tidied up, the Server 2019 DCs became Global Catalogs automagically on the next reboot.