BPN-NVMe4-216N-S24
==================
old:
----
Rev: 1.00
S/N: HB22CS005285
CPLD: V09

new:
Rev: 1.00
S/N: HB241S002146
CPLD: V10


NOTES:
------
	First SSD bay == 0, Last SSD bay = 23.
	Cabling: see AS-2124US-TNRP USER’S MANUAL Revision 2.0
	off: echo 0 >/sys/bus/pci/slots/${Slot}/power
	on: echo 1 >/sys/bus/pci/slots/${Slot}/power
	reboot: init 6
	pull: NVME manually removed from Bay slot N
	push: NVME manually inserted into Bay slot N


Bay population: 0-2,12-23	(15 NVMEs)
--------------------------------------
off 23-20	green
off 19		orange, but should be green
off 18-16	green
off 15-12	green

swap cables: 17,19 <-> 21,23
off 23		short time orange 23 AND 19, than both back to blue
off 22		short time orange 22 AND 18, than both back to blue
off 21		short time orange 21 AND 17, than both back to blue
off 20		short time orange 20 AND 16, than both back to blue
off 19		short time orange 19 AND 23, than both back to blue
off 18		short time orange 18 AND 22, than both back to blue
off 17		short time orange 17 AND 21, than both back to blue
off 16		short time orange 16 AND 20, than both back to blue

Increases smart errors

swap cables back 21,23 <-> 17,19

off 19      orange, but should be green
reboot
			LED off
				- OS: drive n/a
				- BIOS: all drives available/green
					1,7 
						- Blink works: red LED blinking
					    - Insert: Selected action is not applicable to all or some of the rows.
						- Eject: => N/A => LED green, but OS still doesn't see it
						- Insert: Status gray, Infos N/A, but LED now orange

on 19		stays orange, n/a
reboot		LED off
reboot		LED blue
off 19		green
reboot		LED off
off 18		green
reboot		LED 18 off, LED 19 blue
off 23,12	both green
reboot		LED 12 blue, LED 23 off
pull 23
push 23		LED green, still n/a for the OS, BIOS: N/A (Insert does not work)


off 22-19	orange, 3x green, LED 23 switched to blue
mv 19-22 => 4-7
off 18,17	green
off 16		orange => green, LED 17,18 blue
off 18,17	green


Bay population: 0-13,23	(15 NVMEs)
--------------------------------------
off 11-7	green
off 6		green, but LED 7 blue
off 7		green
off 5-4		green
off 3		no effect
off 3		green
off 2-0		green
reboot		all blue (ok, expected)
off 11-0	all green (ok, expected)
on 11-0		all blue (ok, expected)
off 11-0	all green (ok, expected)
on 0-11		all blue (ok, expected)


Bay population: 0-6,11,12-18 (15 NVMEs)
---------------------------------------
poweron
off 18,17	green
off 16		green, but LED 17,18 back to blue
off 18,17	green
on 16-18	blue
off 18-16	green
reboot		all blue (ok, expected)


Summary:
========
As soon as the last 8 bays are used, backplane behaves odd wrt. PCI hotplug.
However in contrast to V09 it behaves more predictable. 
Lessons to learn:
1) Do not reboot when one or more NVMEs in bay 15-23 are in offline (green)
   or ??? (orange) state. If needed reboot twice in a row or poweroff/poweron
   the system. Otherwise NVMEs in those bays are N/A.
2) When one offlines a NVME, always check whether other already offlined NVMEs
   are still in this state (green). Backplane occasionally tends to turn on
   offlined drives.
3) Never ever swap NVME controllers. It makes BP behave in an unexpected manner,
   e.g. like turning off more than the switched off (unplugged) drive and thus
   one may really loose complete filesystems, because redundancy is not
   sufficient to compensate the unavailability of 1+ drives.



SSD 16-19: SXB1B WIO-L RSC-W2-66G4 S2 (slot 6)