Dell - EMC - Isilon - Notes

Dell - EMC - Isilon - Notes

Maxx Lee - Isilon tips
FastStorage - Isilon Firmware Upgrade HowTo
StorageMojo - Isilon Systems Price List [Deprecated]
reddit - FreeNAS - Isilon IQ12000X: Anyone tried re-using for FreeNAS?
Sys Admin Pocket Survival Guide - isilon

Dell EMC Community - Hyperterminal or Putty Settings for Serial Connection In short: DB9 null modem cable, 115200 baud, Data Bits 8, Stop Bits 1, Parity None, Flow Control hardware.
Dell EMC Community - Data Domain: Charging NVRAM Batteries mentions NVRAM batteries can take up to 3 hours to fully charge.
Dell EMC Community - OneFS CLI Mapping (Nov. 2014)

“The charge time was a couple of hours and left the card connected to the motherboard so the batteries could charge over night. The discharge rate according to my voltage meter was around 0,03V per night (I left the computer off for the night). Fully charged the load was 4.18V, after one night it was 4.15V. In theory you should be able to hold the NVRAM contents in memory for a month at minimum if you let the batteries discharge.”
Source:ARM Blog - Ext4 journal on NVRAM card

Commands

Get cluster status

Execute:

isi status

Example output:

Cluster Name: cluster01
Cluster Health:     [ATTN]
Cluster Storage:  HDD                 SSD            
Size:             1.7T (1.7T Raw)     0 (0 Raw)      
VHS Size:         0                   
Used:             20M (< 1%)          0 (n/a)        
Avail:            1.7T (> 99%)        0 (n/a)        

                   Health Throughput (bps)    HDD Storage      SSD Storage
ID |IP Address     |DASR|  In   Out  Total| Used / Size      |Used / Size
---+---------------+----+-----+-----+-----+------------------+-----------------
  1|10.0.2.1       |-A-R|    0| 158K| 158K|   20M/ 1.7T(< 1%)|    (No SSDs)    
------------------------+-----+-----+-----+------------------+-----------------
 Cluster Totals:        |    0| 158K| 158K|   20M/ 1.7T(< 1%)|    (No SSDs)    

     Health Fields: D = Down, A = Attention, S = Smartfailed, R = Read-Only

Critical Events:

 03/07 21:23   1 NVRAM Battery problem.  (Battery Status: Battery1: (Failed)...
 03/07 21:24   1 Redundant power supply failure

Cluster Job Status:
isi_job_d is not running.

No running jobs.

No paused or waiting jobs.

No failed jobs.

This node has an issue with the NVRAM battery being discharged/defective and the second power supply being disconnected/defective.

Password reset for root

Tested on an IQ1920i with OneFS 6.5.5.26.

Power off the Isilon (press the green power button on the back 3 times within 4 seconds and it should power down properly).
Connect a serial terminal with speed 115200 to the COM port on the Isilon.
Power on the Isilon.
After the loading of the .ko kernel modules the following text will be displayed:
```
Hit [Enter] to boot immediately, or any other key for command prompt.
```
Press spacebar (or any other key besides Enter).
You should get a prompt:
```
OK 
```
Execute the following command:
```
boot -s
```

The system will boot single user:

GDB: debug ports: sio
GDB: current port: sio
KDB: debugger backends: ddb gdb
KDB: current backend: ddb
(c) 2011 Isilon Systems LLC.  All rights reserved.  Isilon and
        Isilon Systems are registered trademarks of Isilon Systems LLC.
Copyright (c) 1992-2011 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
Isilon OneFS v6.5.5.26 B_6_5_5_268(RELEASE): 0x605050001A0010C: Tue Oct  8 15:00:48 PDT 2013
    root@fastbuild-02.west.isilon.com:/build/mnt/obj.RELEASE/build/mnt/src/sys/IQ.amd64.release amd64
Isilon: model 4 family 15 stepping 1; highmem disabled.
MEMGUARD DEBUGGING ALLOCATOR INITIALIZED:
        MEMGUARD map base: 0xffffff8000400000
        MEMGUARD map limit: 0xffffff8186ae6000
        MEMGUARD map size: 6396824 KBytes
<...>
twe0: command interrupt
ndmp_sa: version 20090116
Initializing compression for panic dumps.
currdev=disk8s1d:
disk_id = 1073742338, partition = d
 /dev/ad0s1: f0x0 p0 id=0
 /dev/twed0s1: f0x0 p0 id=105
 /dev/twed1s1: f0x0 p0 id=101
 /dev/twed2s1: f0x0 p0 id=104
 /dev/twed3s1: f0x0 p0 id=103
 /dev/twed4s1: f0x0 p0 id=107
 /dev/twed5s1: f0x0 p0 id=65
 /dev/twed6s1: f0x0 p0 id=106
 /dev/twed7s1: f0x0 p0 id=1073742338

Disk to mount from is: /dev/twed7s1d
Trying to mount root from ufs:/dev/imdd0a
WARNING: / was not properly dismounted
Enter full pathname of shell or RETURN for /bin/sh:

Press Enter to get the default /bin/sh shell.
By default /dev/imdd0a is mounted read-only. Execute the following command to mount imdd0a as read-write:
```
mount /dev/imdd0a /mnt/
```

If the following appears:

WARNING: R/W mount of / denied.  Filesystem is not clean - run fsck
mount: /dev/imdd0a : Operation not permitted

Run an fsck:
```
fsck /dev/imdd0a
```

It might take a few minutes but should mark the file system as clean:

** /dev/imdd0a
** Last Mounted on /
** Root file system
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
37666 files, 437678 used, 70465 free (593 frags, 8734 blocks, 0.1% fragmentation)

***** FILE SYSTEM MARKED CLEAN *****

Now mount the file system:
```
mount /dev/imdd0a /mnt/
```

Verify that the file system is mounted by executing the command:
```
mount
```

It should show imdd0a mounted on /mnt without read-only:

/dev/imdd0a on / (ufs, local, read-only)
devfs on /dev (devfs, local)
/dev/imdd0a on /mnt (ufs, local)

Change the root directory to /mnt by executing the command:
```
chroot /mnt
```

Now use passwd to change the password for root:

# passwd
Changing local password for root
New Password:
Retype New Password:

And reboot the system by executing the command:
```
reboot
```
You should now be able to login as root with the password you just configured.

Reset node to factory default

Tested on an IQ1920i with OneFS 6.5.5.26.

Login via the serial console and reset the node to unconfigured by executing the following command (WARNING: this will delete all data and configuration on the node!!):
```
isi_reformat_node
```

It will ask for confirmation twice, answer yes both times:

Reformatting node and destroying filesystem
Are you sure (yes/no) [no]? yes

This could destroy data
Are you really sure (yes/no) [no]? yes

The system will reformat the 12 disks and reboot.

After reboot you should be greeted by the configuration wizard:

<...>
Checking Isilon Journal integrity...
Journal new
Welcome to the Isilon IQ configuration wizard.
(c) 2011 Isilon Systems LLC.  All rights reserved.
Enter 'help' at any prompt for additional information on that step.
Enter 'back' at any prompt to return to previous step.
Enter 'quit' at any prompt to discard all changes.
Enter 'manual' at any prompt to switch to manual mode (console).

        Node build: Isilon OneFS v6.5.5.26 B_6_5_5_268(RELEASE)
        Node serial number: 07190232S

Select an option:
        [ 1] Create a new cluster
        [ 2] Join an existing cluster
        [ 3] Exit wizard and configure manually
Wizard >>>

Now you can reconfigure the node to your specifications.

Troubleshooting

Join to cluster fails

Error message:

Error doing verified node join to cluster cluster01 (retried 4 times): Failed to grab patch/update lock: The file system does not have quorum

This can be caused by the cluster file system being marked Read-only. See the output of

isi status

on another node in the cluster.

Table of Contents