bigzfs status

We have a purchase order out to Thinkmate for a new drive shelf into which to move 45 drives.
We expect it to be ready on Monday -- someone from staff will go to Walpole to pick it up.

In addition, Aberdeen, the vendor of the server, is sending us an additional empty shelf.
We are waiting for a tracking number.   

It will probably be late Monday or early Tuesday before
the server is available again.

The underlying file system, ZFS, is very robust and resiliant.
We don't expect any data loss.

All files on the first big pool (230 TB capacity) are visible.
/cores -- mbcf
/centers -- blaise  ccsb  cmop   cytof  macf-web
/labs -- kslab      lane       mitsiades  segallab   treon

The second big pool is offline.

The third pool is online but empty. 


Yesterday morning, shelf two of bigzfs1 suddenly went off line.
This was unexpected and a highly unusual circumstance.
Nikos George and I spent time diagnosing what had had happened, swapping
out the connecting cables, calling our tech support at Aberdeen (and also
with one of our common vendors, Thinkmate.)

What we found was this: 
The server has four disk controller cards.
Each controller card connects to a box with 45 drives, and 
each of those 45-drive boxes connects to another 45-drive box.
(Kind of a long chain.)

The chain having a problem is this:

Disk Controller 1 --> Disk Box 1 --- Disk Box 2

The problem:  Disk controller 1 can see Disk Box 1, but could
not see Disk Box 2.   That would lead us to suspect disk box 2
But when we attempted to turn disk lights on around the shelves
connected to Controller 1, we found that Disk Controller1 reported clearly
incorrect information from half of the disk drives in Disk Box 1 
(wrong or missing serial numbers, etc.)

That makes things look like a problem with disk box 1.

So, right now, we don't know if the problem resides in Disk Box 1 which
is cascading the problem to Disk Box 2, or if /both/ Disk Box 1 AND Disk
Box 2 are problematic.

We want two known-working shelves in waiting to complete testing.

-- Matt Temple