Why is it recommended to use minimum of 3 APIC Server?
To understand why three APICs is the recommended minimum you must understand how the APICs distribute information between the three. All parts of ACI are datasets generated and processed by the Distributed Policy Repository and that data for those APICs functions are partitioned into logically banded subsets called shards (like DB shard). a Shard is then broken into three replicas or copies. each APIC has a replica for every shard but only 1 APIC is the master for a particular replica/shard. This is a way to distribute the workload evenly and load balance processing across the cluster of 3 as well as a fail safe in case an APIC goes down.
Now that the theory is out of the way, imagine one of your three APICs goes down. the remaining two will negotiate who will now be the master for the shards that the down APIC was in charge of. Workload is then load balance to the two and the cluster becomes fully fit again. Working with 2 APICs is really unadvised due to the split brain condition. This occurs when APIC 1 and APIC 2 thing they are both leaders for a shard and cannot agree so the shard is in contention and the cluster is unfit/”data layer partially diverged”. with the cluster in this state it is unadvised to make changes in the GUI, i don’t remember if its even allowed.
With the case of only 1 APIC, that APIC does all the work, it is the leader for all shards but if it goes down then you can not make any changes at all. data plane will continue forwarding but since no APIC, theres no way to create new policies or changes.
1. Current Mode (ACI/NX)
2. LLDP is Enabled on APIC
1. Leaf/Spine Mode
ACI N9K Firmware
Converting from Cisco NX-OS to ACI Boot Mode
Verify: Show Version
2. LLDP should be DISABLED on VIC of the APIC
Log into the CIMC web interface and under the system settings look at the VIC configuration. Make sure that Enable LLDP is not checked.
After rebooting the APIC should be able to detect the leaf switches. Once you are able to detect the directly connected switches add them to the fabric and then the rest of the fabric switches should start being detected. Once you have added all fabric switches to the fabric, you can then run the setup of the additional APIC’s.
FYI – This is a known issue with certain APICs manufactured within a certain date range. A few hundred units were incorrectly configured with LLDP enabled on the VIC.
This is tracked as: https://bst.cloudapps.cisco.com/bugsearch/bug/CSCva28506
Verify: Verify (Fabric – Inventory – Fabric membership).
Other Solution for troubleshooting:
1. Cables should properly connected
2. Move SFP+ transceivers on the Leaf switches to different ports
3. Rebooted both Leaf and APIC server