Operation Daily Magic Error

3 locks for your SSH door

2011-09-08T10:00:00.004+02:00

I have found this great article on Developerworks. Great stuff, especially part 3, about port knocking. Read it, use it.

Resetting unsuccessful logins

2011-09-06T13:36:00.000+02:00

AIX keeps records of the unsuccessful logins to every account. Too many login failures may cause you the get locked out. To reset, try these:

chsec -f /etc/security/lastlog -a "unsuccessful_login_count=0" -s login_name

chuser "account_locked=false" login_name

The chrctcp command

2011-09-01T17:44:00.000+02:00

The chrctcp command in not documented in AIX but you can still use it to do nice things, especially when you are scripting. Some examples:

chrctcp -S -a xntpd enable and start xntpd
chrctcp -S -d xntpd disable and stop xntpd
chrctcp -a xntpd enable but do not start xntpd
chrctcp -d xntpd disable but do not stop xntpd

So, instead of manually editing /etc/rc.tcpip you can use chrctcp to enable (uncomment), disable (comment) some services, and start and stop them in a single command. Nice.

LED 0517

2011-04-07T16:31:00.000+02:00

Your LPAR/server stopping at 0517? Check your bootlist, it may still be wanting to boot from network instead of disk.

Numbering scheme for dual-VIOS configurations

2011-03-31T12:43:00.001+02:00

The question of virtual adapter numbering keeps popping up every time you are about to introduce a larger dual-VIOS configuration. In this example I am going to give you a numbering scheme which may help you find your way amongst the ever growing number of virtual devices.

Please mind that it is the way I usually set up my systems. None of the things described here are rules, rather they are guidelines for those who need them. You may use it as is or develop your own based on it, or even forget it all and come up with something completely different. But the point I can't emphasize enough is that you should have a plan of some sort before taking the dive.

Suppose you have a dual-VIOS setup, the VIO servers are called vio1 and vio2. We want to have virtual ethernet, virtual SCSI, and virtual FC (NPIV) connections. For the virtual ethernet we want multiple port-based VLANs and SEA failover (with a control channel). Both vSCSI and vFC will be redundant. We will have less than 100 LPARs including the two VIOS partitions.

General VIOS considerations

Before you start setting up your VIOS and client partitions, make sure you have your layout well planned. With a good plan, setting up the system is like a breeze. With no plan, you may soon find yourself in the middle of some great mess.
Look at your plans and establish the maximum number of virtual devices per partition.

vSCSI

vSCSI uses a one-to-one relationship between a server SCSI adapter and a client SCSI adapter. To easily pair the server and client adapters, both will have the same virtual slot ID. (For now I will assume that there will always be a maximum of one pair of server and client SCSI adapters between any VIO server and client.)
All adapter pairs between a client and vio1 will have the virtual slot ID 1xx, where xx is the client LPAR ID with a leading zero if needed. Example: adapter slot ID 107 will be used between vio1 (LPAR ID 1) and client7 (LPAR ID 7) for both virtual server and client adapters.
All adapter pairs between a client and vio2 will have the virtual slot ID 2xx, where xx is the client LPAR ID with a leading zero if needed. Example: adapter slot ID 229 will be used between vio2 (LPAR ID 2) and client29 (LPAR ID 29) for both virtual server and client adapters.

vEthernet

Virtual ethernet uses the one-to-many layout. The VIOS usually has one virtual adapter, a bridge, that is connected to (on the internal end) the virtual ethernet switch provided by the hypervisor and (on the external end) the shared ethernet adapter and the physical ethernet adapter port.
A virtual ethernet adapter acting as a bridge adapter must have its access external network flag set.
For virtual Ethernet slot IDs I usually use the 300-499 range. The 300-399 range is reserved for VLAN that are used either between LPARs (internal networks) or between LPARs and the outside world. While the 400-499 range is reserved for control channel VLANs.
Each external VLAN ID is mapped to a number between 300 and 399. The mapping can be some mathematical formula to make it easy for the administrator to identify a VLAN just by looking at the virtual slot ID of the adapter (see Example #1), or it can be an relationship between the VLAN ID and the virtual slot ID usually in the form of a table (see Example #2).
Each VLAN defined in the 300-399 range has one corresponding controll channel VLAN ID in the 400-499 range. VLAN 359 has a control channel VLAN ID of 459.

vFC

Virtual FC adapters use the same one-to-one layout as vSCSI adapters. There is one server adapter defined in the VIOS and it is connected to exactly one client adapter defined in the client LPAR. Both members of one pair will use the exact same virtual slot ID.
With NPIV it often happens that one channel between a VIO server and client is insufficient, so in this example I will use two pairs.
The system of assigning virtual slot IDs for vFC adapter is the exact same as for vSCSI adapters, but in this case we will use the 500-599 range for the first pair, 700-799 for the second pair on vio1.
The range 600-699 and 800-899 will be used for the first and second pair of adapters on vio2.
If you need more adapter pairs, the logic is clear. Hundreds starting with an odd number starting at 5 mean the pair belongs to vio1, hundreds starting with even numbers starting at 6 mean the pair belongs to vio2.

Examples

Suppose you have VLAN IDs 111, 112, 184, and 191. The formula you could use to calculate the virtual slot IDs for the bridge adapters for these VLANs is this: vSlotID=vlanId+200. Thus 111, 112, 184, and 191 become 311, 312, 384, and 391 respectively. And the corresponding control channel virtual adapter slot IDs are 411, 412, 484 and 491.
Suppose you have VLAN IDs 14, 99, 134, and 254. In this case you have two choices. (a) You can use the two rightmost digits of the VLAN IDs to generate the virtual slot IDs, such as 314, 399, 334, and 354, with control channel virtual adapter clost IDs 414, 499, 434, and 454, respectively. Or (b) you can map the VLAN IDs to arbitrary virtual slot IDs: 14->300; 99->301; 134->302; 254->303. The respective control channel virtual slot IDs then are 400, 401, 402, and 403.

Removing all failed paths

2010-09-30T11:23:00.001+02:00

To remove all the failed paths use the following command:

lspath -s failed -F"name connection" | while read dev con; do rmpath -l $dev -w $con -d; done

Unwanted RMC traffic

2010-09-25T12:33:00.004+02:00

In an HMC controlled LPAR environment a misconfiguration may exist that causes the RMC daemon (ctrmcd) to try to communicate with the HMC private network interface (usually eth0). This communication may never succeed because the private network can never be reached from an LPAR. The traffic often bothers some customers. To get rid of the unwanted network activity, read this: Partitions communicating with HMC private network

Make sure you are logged in on the console while changing the gateway device, otherwise you may lose connection to the HMC and will not be able to reboot the HMC remotely.

"cldare: Not able to discover the name of the local node."

2010-05-16T13:01:00.000+02:00

In case you get the error message in the title, make sure your base addresses are added to your HACMP/PowerHA configuration. Adding them only as communication interfaces at initial cluster setup may not be sufficient.

Useful NIM commands

2010-05-06T13:33:00.006+02:00

To create an updated NIM lpp_source and SPOT while keeping the originals:

(1) cd /nim/lppsource
(2) cp -rp aix531200 aix531201
(3) nim -o define -t lpp_source -a server=master -a location=/nim/lppsource/aix531201 aix531201
(4) nim -o update -a packages=all -a source=/swi/AIX/53/tl12sp1 aix531201
(5) nim -o lppmgr -a lppmgr_flags="-bu -x -r -e" aix531201
(6) nim -o define -t spot -a server=master -a location=/nim/spot -a source=aix531201 spot531201

You will only need step 5 if your source is a cummulative fix pack containing both a full TL and the SP above it. If it is only an SP (does not contain the fixes up to the full TL) you may skip step 5. However, it won't hurt runnig it.

To create a new lpp_source from an install DVD, copy the DVD contents (DVD #1 is sufficient) into a directory, and then:

nim -o define -t lpp_source -a server=master -a location=/nim/lppsource/lpp6100-05-00 -a source=/swi/AIX/61/6100-05-00-DVD/1 lpp6100-05-00

PowerHA problems

2009-11-27T18:09:00.004+01:00

A defect exists in PowerHA 5.5 (base level) that causes a PowerHA node to lose the default gateway after a reboot. To fix the defect install SP04.

But SP04 also has a defect that causes a Power HA node to lose the default gateway after a cluster stop. To correct, install IZ63771. Until the standard fix is available, there is an e-fix available at ftp://public.dhe.ibm.com/aix/efixes/iz63771/.

NTP on VIOS

2009-10-02T11:28:00.001+02:00

Edit /home/padmin/config/ntp.conf and start NTP using startnetsvc xntpd.

DS5000 default IP addresses

2009-08-28T17:35:00.005+02:00

Controller A port 1: 192.168.128.101
Controller A port 2: 192.168.129.101
Controller B port 1: 192.168.128.102
Controller B port 2: 192.168.129.102

Summary

2009-06-21T20:13:00.002+02:00

fscsi device tuning

2009-05-06T20:11:00.003+02:00

To enable dynamic tracking and faster path error discovery (if you have multiple paths to your devices) use the following settings on your fscsi devices:

chdev -l fscsi0 -a dyntrk=yes -a fc_err_recov=fast_fail

mircmp tool

2009-04-23T10:42:00.004+02:00

This tool can be used to compare copies of a mirrored LV. Comes in handy sometimes.


#!/bin/ksh

#------- constants ----------
WORKDIR=/tmp/mircmp$$
WORKDIR2=/tmp/mircmp
STOPFILE_GLOBAL=/tmp/stop_mircmp
STOPFILE_LOCAL=$WORKDIR/stop_mircmp
MBSIZE=1048576  # 1MB.  The -S flag takes the io size in MB,
 # so this is a useful value.  We immediately
 # convert this value into the size in 128KB
 # units, but most users probably find it easier
 # to think in terms of MB.
LTGSIZE=131072  # 128KB.  This is actually the min possible
 # ltg size for a vg, and has nothing to do
 # with the current ltgsize value for the vg.
 # This constant is useful since readlvcopy
 # takes io size and offset in 128KB chunks
 # rather than DBSIZE (512 byte) chunks.  This
 # is the finest granularity of io that readlvcopy
 # supports, so its the finest granularity we
 # can support in this script.


#------- global variables --------

# flags and flag values.  For documentation of each
# flag please see the Usage function.
cFLAG=
lFLAG=
lVAL=
sFLAG=
sVAL=
SFLAG=
SVAL=
vFLAG=
vVAL=
zFLAG=
wFLAG=
wVAL=

# general global variables
LVNAME=
LVID=
VGNAME=
VGID=
PPSIZE=
PPSIZE_P2=
NLTGS_PERPP=
NMB_PERPP=
IO_SIZE=



#------- utility functions --------

# Documents the script.  Called for any syntax errors.
Usage()
{
echo "Usage: mircmp -l lvname | -v vgname"
echo "  [-s iosize in 128KB | -S iosize in MB]"
echo "  [-c] [-b begin lp#] [-e end lp# | -n numlps]"
echo ""
echo "Purpose: Verify mirror consistency."
echo ""
echo "Required Flags:"
echo " -l lvname examine this lv."
echo " -v vgname each lv within the vg will be "
echo "   examined.  Any non-mirrored lvs"
echo "   within the vg will be skipped."
echo ""
echo "Other options:"
echo " -c   display cksum of comparison files as"
echo "   we perform each io.  The naming"
echo "   convention for the comparison files"
echo "   is lvname.lp#.io#.a/b/c/s,"
echo "   where a=primary mirror copy, b=secondary"
echo "   mirror copy, c=tertiary mirror copy, and"
echo "   s=second pass at the primary mirror copy"
echo "   to check for io in flight."
echo " -b begin begin with this lp#. By default this"
echo "   is the start of the lv."
echo " -e end  end at this lp#.  By default this is"
echo "   the end of the lv.  The -e and -n"
echo "   flags are mutually exclusive."
echo " -n numlps number of lps to examine.  The -e"
echo "   and -n flags are mutually exclusive."
echo " -s 128KB io size specified in 128KB units."
echo " -S 1MB  io size specified in MB."
echo " -z  saves the inconsistent blocks into /tmp/mircmp"
echo " -w workdir use specified workdir instead of /tmp"
echo ""
echo "Description:"
echo "  Verify mirror consistency by reading each mirror"
echo "  copy of a logical volume and comparing the data"
echo "  against the other copies.  Report whether or not"
echo "  the mirrors for that lp have matching copies."
echo "  Ignore the vgsa stale bits which tell lvm whether"
echo "  the lp is stale or fresh, and just compare the actual"
echo "  data.  If the mirrors don't match a second pass"
echo "  attempts to determine if io is in flight."
echo ""
echo "False Failures:"
echo "  One important thing to note is that if an lv is"
echo "  mirrored at creation time (with the mklv -c flag)"
echo "  then the mirror copies won't match until that"
echo "  particular 512 byte block is written to.  This is"
echo "  done because it saves a lot of time during the"
echo "  creation of a large lv (we don't have to do io"
echo "  to the logical partitions, we just have to generate"
echo "  a map of the allocations).  In short, lvs that were"
echo "  mirrored at creation time will show mirror"
echo "  mismatches unless each 512 byte block of the lv"
echo "  has been written to.  These are false failures"
echo "  and do not indicate a problem with the mirrors."
echo ""
echo "  If an lv is created without mirroring and then"
echo "  mklvcopy is used to mirror, lvm will once again"
echo "  not do any io to the new mirror copies, and the"
echo "  new mirror copies will be marked stale.  Once"
echo "  syncvg is run the new mirror copies will be"
echo "  marked fresh and the mirror copies will match."
echo ""
echo "  Stale partitions will show up as mirror miscompares"
echo "  because mircmp completely ignores the vgsa stale bits."
echo ""
echo "Graceful Termination:"
echo "  To stop all running instances of the mircmp process"
echo "  create a file called /tmp/stop_mircmp.  To stop a"
echo "  particular mircmp process, create a file called"
echo "  /tmp/mircmp/stop_mircmp.  These stopfiles"
echo "  are used for graceful termination.  There is a signal"
echo "  handling function as well but you'd rather use that."
echo "  If mircmp is terminated with a signal it will not exit"
echo "  gracefully and the /tmp/mircmp will be left"
echo "  behind."
echo ""
echo "Space Considerations:"
echo "  Each /tmp/mircmp will contain N+1 files,"
echo "  N is the number of mirrors for the lv.  The +1"
echo "  comes from the way mircmp performs a second pass"
echo "  to check for io in flight if the first pass detects"
echo "  a mirror mismatch.  In the second pass we re-read"
echo "  from the primary mirror copy and compare against"
echo "  the first pass primary mirror copy data.  If the"
echo "  first pass and second pass find equal data we"
echo "  assume io is not in fight and this is a true mirror"
echo "  mismatch."
echo ""
echo "  The size of each file is known as the iosize and"
echo "  can be specified with either the -s or -S flag."
echo "  By default the iosize=ppsize for the vg that"
echo "  mircmp is operating on.  If the user specifies an"
echo "  iosize > ppsize then we set iosize=ppsize.  As"
echo "  the iosize decreases, the /tmp space requirement"
echo "  decreases and the performance slows down."
echo ""
echo "  If the -z flag is used, /tmp/mircmp will contain"
echo "  saved versions of every block who's copies do not match."
echo "  This could require significant space in /tmp, as described"
echo "  above."
echo ""

cleanup_exit
}

# Called if we get a signal.
cleanup_signal()
{
cleanup_exit
}

# Gracefully cleanup our work directory
# and exit.
cleanup_exit()
{
trap '' 0 1 2 15
rm -rf $WORKDIR
exit
}

# Useful function for checking return codes.
# Errors are fatal.  First parameter is the
# rc, second parameter is the error string.
test_return()
{
if [ "$1" != 0 ]
then
echo "mircmp: fatal error: $2"
cleanup_exit
fi
}

# Check return code from readlvcopy.  This is a
# special case where a read error shouldn't be
# fatal unless /tmp is full.
test_io_return()
{
if [ "$1" != 0 ]
then
TIO_RC=`df /tmp | grep -v "%Used" | awk '{print $4}'`
if [ "$TIO_RC" = "100%" ]; then
 echo "mircmp: fatal error: $2"
 echo ""
 echo "mircmp: /tmp is full.  Please choose"
 echo "  a smaller iosize using the -s or"
 echo "  -S parameter and try again."
else
 echo "mircmp: fatal error: $2"
fi
cleanup_exit
fi
}

# Check and see if the user has specified either
# a global or local stopfile.  If so then
# gracefully cleanup our work directory and exit.
check_stopfile()
{
if [ -f "$STOPFILE_GLOBAL" ]; then
echo "global stopfile detected"

trap '' 0 1 2 15
rm -rf $WORKDIR
exit
elif [ -f "$STOPFILE_LOCAL" ]; then
echo "local stopfile detected"

trap '' 0 1 2 15
rm -rf $WORKDIR
exit
fi
}

# This is the function that is called on each logical
# partition.  Loop reading io size chunks of the
# partition and comparing that chunk across all
# mirrors.  Once the entire lp has been examined,
# report results to stdout.
process_lp()
{
LP=$1

if [ -z "$LP" ]; then
echo "mircmp: process_lp: no lp specified"
cleanup_exit
fi

# Compute the start of the lp in 128KB units.
((LP_OFF = $LP - 1))
((LP_SKIP = $LP_OFF * $NLTGS_PERPP))

NLP_IOEQUAL=0
NLP_IOFLIGHT=0
NLP_IOMISMATCH=0
CUR_IO=1
while [ "$CUR_IO" -le "$NIOS_PERPP" ];
do
# Check for global and local stopfiles
# each time through the loop so that we
# can fail gracefully if necessary.
check_stopfile

# Compute the distance from the start of
# the lp to the current io.  Once again
# this is in 128KB units.
((TMP_IO_OFF = $CUR_IO - 1))
((IO_OFF = $TMP_IO_OFF * $IO_SIZE))
((IO_SKIP = $LP_SKIP + $IO_OFF))

if [ $IO_SKIP -eq 0 ]; then
 SKIP_STRING=""
else
 SKIP_STRING="-s $IO_SKIP"
fi

# Naming convention for the comparison
# files is lp#.io#.a/b/c/s, where
# a=first mirror copy, b=second mirror copy,
# c=third mirror copy, and s=second pass against
# the first mirror copy.

readlvcopy -d /dev/r$LVNAME -c 1 $SKIP_STRING \
  -n $IO_SIZE -o $WORKDIR/$LP.$CUR_IO.a >/dev/null 2>&1
test_io_return $? "readlvcopy $LP.$CUR_IO.a"

readlvcopy -d /dev/r$LVNAME -c 2 $SKIP_STRING \
  -n $IO_SIZE -o $WORKDIR/$LP.$CUR_IO.b >/dev/null 2>&1
test_io_return $? "readlvcopy $LP.$CUR_IO.b"

cmp -s $WORKDIR/$LP.$CUR_IO.a $WORKDIR/$LP.$CUR_IO.b
CMP_B_RC=$?

if [ $NCOPIES -eq 3 ]; then
 readlvcopy -d /dev/r$LVNAME -c 3 $SKIP_STRING \
   -n $IO_SIZE -o $WORKDIR/$LP.$CUR_IO.c >/dev/null 2>&1
 test_io_return $? "readlvcopy $LP.$CUR_IO.c"

 cmp -s $WORKDIR/$LP.$CUR_IO.a $WORKDIR/$LP.$CUR_IO.c
 CMP_C_RC=$?
else
 CMP_C_RC=0
fi

# If mirrors match increment NLP_IOEQUAL.
#
# If mirrors don't match we perform a second pass
# which reads mirror "a" again and compares it to
# mirror a data from the first pass.  If mirror
# "a" has changed between the first and second pass
# we increment NLP_IOFLIGHT, otherwise we increment
# NLP_IOMISMATCH.
if [[ $CMP_B_RC -eq 0 ]] && [[ $CMP_C_RC -eq 0 ]]; then
 SECOND_PASS=0
 ((NLP_IOEQUAL = $NLP_IOEQUAL + 1))
else
 SECOND_PASS=1
 readlvcopy -d /dev/r$LVNAME -c 1 $SKIP_STRING \
   -n $IO_SIZE -o $WORKDIR/$LP.$CUR_IO.s >/dev/null 2>&1
 test_io_return $? "readlvcopy $LP.$CUR_IO.s"

 cmp -s $WORKDIR/$LP.$CUR_IO.a $WORKDIR/$LP.$CUR_IO.s
 CMP_S_RC=$?

 if [ $CMP_S_RC -eq 0 ]; then
   ((NLP_IOMISMATCH = $NLP_IOMISMATCH + 1))
 else
   ((NLP_IOFLIGHT = $NLP_IOFLIGHT + 1))
 fi
fi

# Display the cksum for the comparison files if the
# -c flag is specified.  Then cleanup the comparison files
if [ -n "$cFLAG" ]; then
  cksum $WORKDIR/$LP.$CUR_IO.a $WORKDIR/$LP.$CUR_IO.b
  if [ $NCOPIES -eq 3 ]; then
   cksum $WORKDIR/$LP.$CUR_IO.c
  fi
  if [ $SECOND_PASS -eq 1 ]; then
   cksum $WORKDIR/$LP.$CUR_IO.s
  fi
fi

# Remove all the files related to this particular
# io_size chunk.  Note that the .c and .s files
# may not exist but because we're using rm -f
# we don't care if the files don't exist.
if [ -n "$zFLAG" -a "$CMP_B_RC" -ne "0" -a "$CMP_S_RC" -eq "0" ] ; then
 mv $WORKDIR/$LP.$CUR_IO.a $WORKDIR2/${LVNAME}_LP${LP}_${CUR_IO}.1
 mv $WORKDIR/$LP.$CUR_IO.b $WORKDIR2/${LVNAME}_LP${LP}_${CUR_IO}.2
fi

if [ -n "$zFLAG" -a "$CMP_C_RC" -ne "0" -a "$CMP_S_RC" -eq "0" ] ; then
 if [ -e "$WORKDIR/$LP.$CUR_IO.a" ]; then
  mv $WORKDIR/$LP.$CUR_IO.a $WORKDIR2/${LVNAME}_LP${LP}_${CUR_IO}.1
 fi
 mv $WORKDIR/$LP.$CUR_IO.c $WORKDIR2/${LVNAME}_LP${LP}_${CUR_IO}.3
fi

rm -f $WORKDIR/$LP.$CUR_IO.a $WORKDIR/$LP.$CUR_IO.b \
  $WORKDIR/$LP.$CUR_IO.c $WORKDIR/$LP.$CUR_IO.s

((CUR_IO = $CUR_IO + 1))
done

# Tally the results and report them to stdout.
((TALLY_IOS = $NLP_IOEQUAL + $NLP_IOFLIGHT + $NLP_IOMISMATCH))
if [ $TALLY_IOS -ne $NIOS_PERPP ]; then
echo "$LVNAME.LP$LP internal error $NLP_IOS / $NIOS_PERPP"
elif [ $NLP_IOMISMATCH -gt 0 ]; then
echo "$LVNAME.LP$LP mismatch"
elif [ $NLP_IOFLIGHT -gt 0 ]; then
echo "$LVNAME.LP$LP inflight"
else
echo "$LVNAME.LP$LP equal"
fi
}


# This is the function that is called on each logical
# partition.  Query the number of lps and copies
# for this lv.  Determine the range of lps to scan,
# depending on what the user asked for (by default
# we scan every lp in the lv).  Loop scanning the lps.
# If the lv is non-mirrored or if this particular lv
# has no lps within the range the user was asking about,
# simply print a message and continue.
process_lv()
{
LVNAME=$1

if [ -z "$LVNAME" ]; then
echo "mircmp: process_lv: no lvname specified"
cleanup_exit
fi

# use lvname to get lvid, vgname, vgid.
LVID=`getlvodm -l $LVNAME`
test_return $? "getlvodm -l $LVNAME"

      CHARS=`lquerylv -L $LVID -cC`
test_return $? "lquerylv -L $LVID -cC"
      if [ -n "$CHARS" ]; then
              set $CHARS
              LVSIZE_LPS=$1
              NCOPIES=$2
else
echo "mircmp: process_lv: lquerylv failed"
cleanup_exit
      fi

# Determine the range of lps to examine.  If the
# user asked to scan lps 300-500 and this lv in
# the vg only has 5 lps then we print the invalid
# range error message and move on to the next lv.
#
# If -b was specified, make sure the -b value is
# not past the end of this lv.  If -e or -n was
# specified, make sure that we stop at the end of
# the lv.  It is perfectly valid to specify -b
# without -e or -n.  In that case we use the last
# lp in the lv as the stopping point.  It is also
# perfectly valid to specify -b without -e or -n.
# In that case we use the first lp in the lv as
# the starting point. 
PLV_INVALID_RANGE=0
if [ -n "$bFLAG" ]; then
if [ $bVAL -gt $LVSIZE_LPS ]; then
 PLV_INVALID_RANGE=1
else
 CUR_LP=$bVAL
fi
else
CUR_LP=1
fi

if [ -n "$eFLAG" ]; then
if [ $eVAL -gt $LVSIZE_LPS ]; then
 END_LP=$LVSIZE_LPS
else
 END_LP=$eVAL
fi
elif [ -n "$nFLAG" ]; then
((TMP_nVAL = $nVAL - 1))
((nEND = $CUR_LP + $TMP_nVAL))
if [ $nEND -gt $LVSIZE_LPS ]; then
 END_LP=$LVSIZE_LPS
else
 END_LP=$nEND
fi
else
END_LP=$LVSIZE_LPS
fi

# Check for global and local stopfiles
# so that we can fail gracefully if necessary.
check_stopfile

if [ $NCOPIES -lt 2 ]; then
echo "===$LVNAME is not mirrored==="
elif [ $PLV_INVALID_RANGE -eq 1 ]; then
echo "===$LVNAME invalid lp range specified==="
else
echo "===starting $LVNAME==="
while [ "$CUR_LP" -le "$END_LP" ]
do
 process_lp $CUR_LP

 ((CUR_LP = $CUR_LP + 1))
done
fi
}

process_vg()
{
PVG_VGNAME=$1

PVG_LVLIST=`lsvg -l $PVG_VGNAME | \
awk '$1 != "LV" && $1 !~ ":" {print $1}'`

for PVG_LV in $PVG_LVLIST
do
process_lv $PVG_LV
done
}


# ------ main function -------

# Setup signal handler.
trap 'cleanup_signal' 0 1 2 15

# Parse input and deterine if there is a syntax error.
set -- `getopt b:ce:l:n:s:S:v:zw: "$@"`
if [ $? != 0 ]; then
Usage
fi

# For more detail on flag specific information see
# the Usage function.
while  [ $1  !=  -- ]
do
case $1 in
-b) bFLAG='-b';  bVAL=$2;  shift;  shift;;   # begin lp
-c) cFLAG='-c';  shift;;       # show cksum
-e) eFLAG='-e';  eVAL=$2;  shift;  shift;;   # end lp
-l) lFLAG='-l';  lVAL=$2;  shift;  shift;;   # lvname
-n) nFLAG='-n';  nVAL=$2;  shift;  shift;;   # numlps
-s) sFLAG='-s';  sVAL=$2;  shift;  shift;;   # io_size_in_128KB
-S) SFLAG='-S';  SVAL=$2;  shift;  shift;;   # io_size_in_MB
-v) vFLAG='-v';  vVAL=$2;  shift;  shift;;   # vgname
-z) zFLAG='-z';  shift;;                     # save readlvcopy output
-w) wFLAG='-w';  wVAL=$2;  shift;  shift;;   # working dir
esac
done

# Must have a target device.
if [[ -z "$lFLAG" ]] && [[ -z "$vFLAG" ]]; then
echo "mircmp: no lvname or vgname specified"
Usage
fi

if [ -n "$lFLAG" ]; then
# The -l lvname flag was specified.  Derive the
# vgname and vgid so that we can determine the ppsize
# of the vg.  This allows us to validate the -s or -S
# value early on in the script if they are specified.
LVNAME=$lVAL

LVID=`getlvodm -l $LVNAME`
test_return $? "getlvodm -l $LVNAME"

VGNAME=`getlvodm -b $LVID`
test_return $? "getlvodm -b $LVID"

VGID=`getlvodm -v $VGNAME`
test_return $? "getlvodm -v $VGNAME"

PPSIZE_P2=`lqueryvg -g $VGID -s`
test_return $? "lqueryvg -g $VGID -s"

((PPSIZE = 1 << $PPSIZE_P2))  ((NLTGS_PERPP = $PPSIZE / $LTGSIZE))  ((NMB_PERPP = $PPSIZE / $MBSIZE)) elif [ -n "$vFLAG" ]; then  # The -v vgname flag was specified.  Derive the   # vgid so that we can determine the ppsize of the   # vg.  This allows us to validate the -s or -S   # value early on in the script if they are specified.     VGNAME=$vVAL;   VGID=`getlvodm -v $VGNAME`  test_return $? "getlvodm -v $VGNAME"    PPSIZE_P2=`lqueryvg -g $VGID -s`  test_return $? "lqueryvg -g $VGID -s"   ((PPSIZE = 1 << $PPSIZE_P2))  ((NLTGS_PERPP = $PPSIZE / $LTGSIZE))  ((NMB_PERPP = $PPSIZE / $MBSIZE)) fi  # Sanity check -s and -S values, so that we  # can determine the io size.   if [[ -n "$sFLAG" ]] && [[ -n "$SFLAG" ]]; then  echo "mircmp: The -s and -S flags are mutually exclusive."  Usage fi  if [ -n "$sFLAG" ]; then  if [ $sVAL -eq 0 ]; then   echo "mircmp: -s value must be > 0"
Usage
fi

# See the comments before NIOS_PERPP_LEFTOVER
# for an explanation of why we have this
# "divisible by 2" restriction.
if [ $sVAL -ne 1 ]; then
((TMP_sVAL = $sVAL % 2))
if [ $TMP_sVAL -ne 0 ]; then
 echo "mircmp: -s value must be evenly divisible by 2"
 Usage
fi
fi

# If the io_size > pp_size then io_size = pp_size.
if [ $sVAL -gt $NLTGS_PERPP ]; then
IO_SIZE=$NLTGS_PERPP
else
IO_SIZE=$sVAL
fi
elif [ -n "$SFLAG" ]; then
if [ $SVAL -eq 0 ]; then
echo "mircmp: -S value must be > 0"
Usage
fi

# Convert from MB to 128KB units.
((IO_SIZE = $SVAL * 8))
else
# If no io size is specified, io_size = pp_size.
IO_SIZE=$NLTGS_PERPP
fi

if [ -n "$wFLAG" ]; then
if [ -d "$wVAL" ]; then
WORKDIR=$wVAL/mircmp$$
WORKDIR2=$wVAL/mircmp
else
echo "mircmp: $wVAL is not a directory! "
Usage
fi
fi

# The ppsize is always a multiple of 128K.  Valid
# ppsizes are 1MB to 128GB incrementing by power
# of 2.
#
# We ensure that the io_size is evenly divisible
# by 2 so that we don't have to worry about any
# case where there are leftovers on the last
# io for the pp.  Here we have a sanity check to
# verify that we don't have any leftovers (ie
# all io's will be the same size).
#
# An example of leftovers would be if we allowed
# a -s value of 3.  Lets assume the ppsize is 1MB.
# Each io would be 384KB until the final io of the
# pp which would be 128KB.  For the sake of keeping
# things simple, we don't allow ourselves to get into
# that state.
((NIOS_PERPP = $NLTGS_PERPP / $IO_SIZE))
((NIOS_PERPP_LEFTOVER = $NLTGS_PERPP % $IO_SIZE))

if [ $NIOS_PERPP_LEFTOVER -ne 0 ]; then
echo "mircmp: non-zero nios_perpp_leftover=$NIOS_PERPP_LEFTOVER"
cleanup_exit
fi

if [[ -n "$eFLAG" ]] && [[ -n "$nFLAG" ]]; then
echo "mircmp: The -e and -n flags are mutually exclusive."
Usage
fi

if [ -n "$bFLAG" ]; then
if [ $bVAL -eq 0 ]; then
echo "mircmp: -b value must be > 0."
Usage
fi
fi

if [ -n "$eFLAG" ]; then
if [ $eVAL -eq 0 ]; then
echo "mircmp: -e value must be > 0."
Usage
fi
fi

if [[ -n "$bFLAG" ]] && [[ -n "$eFLAG" ]]; then
if [ $bVAL -gt $eVAL ]; then
echo "mircmp: -b value > -e value."
Usage
fi
fi

if [ -n "$nFLAG" ]; then
if [ $nVAL -eq 0 ]; then
echo "mircmp: -n value must be > 0."
Usage
fi
fi

# setup work directory
mkdir $WORKDIR

if [ -n "$zFLAG" -a ! -e "$WORKDIR2" ]; then
mkdir $WORKDIR2
fi

# pass control to a vg or lv specific function.
if [ -n "$lFLAG" ]; then
process_lv $lVAL
elif [ -n "$vFLAG" ]; then
process_vg $vVAL
fi

# Reset signals so that we don't go into
# the cleanup_signal function when we exit.
trap '' 0 1 2 15

# cleanup work directory
rm -rf $WORKDIR
exit

TS7520/DS4200 default password

2009-02-11T21:40:00.003+01:00

warning2use

ifor_ls.html.en_US.base.cli does not get updated

2009-01-24T17:50:00.005+01:00

ifor_ls.html.en_US.base.cli remains at 5.3.7.0 when upgrading from TL7 to TL8. Check out this.

Accessing the ASMI

2008-10-20T13:41:00.005+02:00

Detailed instructions available at this page.

Service processor	Server connector	Subnet mask	IP address
Service processor A	HMC1 HMC2	255.255.255.0	192.168.2.147 192.168.3.147
Service processor B (if installed)	HMC1 HMC2	255.255.255.0	192.168.2.146 192.168.3.146

Recovering a lost service processor IP address

2008-10-08T20:19:00.006+02:00

While upgrading/reinstalling an HMC you may end up in a situation where some or all of your operating p5/p6 systems (service processors) lose contact with the HMC. The systems are up and running fine, but your HMC has no idea where to look for them because you failed to retain the IP address range used by the service processors. Hmmm.

Fortunately there is a way to recover the IP addresses of the individual service processors...through the operator panel. If you are not familiar with the operator panel, check this PDF.

So, here it goes.

Choose menu 02.
Change the N to M, that is normal to manual. This will enable menus that are normally invisible .
Scroll down to menu 30.
Enter and you'll see 30**.
Enter and you'll have two submenus 3000 and 3001 (in a single service processor configuration). Submenu 3000 will tell you the actual IP address of the HMC1 interface, 3001 will tell you the same of HMC2.
When you have noted them down, go back to menu 02 and reset the M to N. Failing to reset this flag WILL result in your server booting into diag next time it is restarted.

HMC console issues

2008-10-08T19:48:00.004+02:00

When you attach an HMC to an originally HMC-less (fullpar) p5/p6 server you may experience issues with the HMC's terminal emulation (regardless of HMC version and system firmware version). When the server and the HMC are connected, everything looks good until you try to open a serial connection when you may see one of the following: (1) the terminal opens and you get an error about the serial port not being available; (2) the terminal opens and you get an error about the serial port being occupied by another connection; (3) the terminal opens, you get a decent login prompt but nothing you type reaches the server, the connection is read-only.

The phenomenon is caused by the MDC (Machine Default Configuration) flag. You can check this flag by opening your server's properties, if set to true, this also applies to you.

Here's how you can rectify the situation.

The removal of the MDC flag is implicit. You can do it by creating a new dummy LPAR and immediately removing it. Of course, if you need it, you can keep it. The creation itself toggles the flag. Or you can set Service Authority from the default partition to unassigned. That'll do it too.
Stop LPAR #1, the original one.
Activate LPAR #1 to reload the MDC-less partition profile.
Hopefully you have some TCP/IP connection to LPAR #1 because at this point you still lack a working serial connection. Log in and remove the exisiting vty and vsa devices. rmdev -dl vsa0 -R; remdev -dl vsa1 -R; rmdev -dl vsa2 -R
Run cfgmgr to reconfigure the serial devices. You should have a new vty0 and a vsa0.
You can check your console setting by issuing lscons -a. You will see NULL as your console device, which is causing all your problems. Now type chcons -a login=enable -a login_logname='/var/adm/ras/conslog' -a console_logsize='32768' -a console_logverb='1' /dev/vty0
Reboot LPAR #1 (no need to reactivate it since the profile has not been changed).

And you're done!

0511-171 Reached end of media

2008-07-08T18:47:00.002+02:00

An error exists in AIX 5.3 TL7 that causes tar to produce a similar message:

tar: There are 20 blocks on -.
tar: 0511-171 Reached end of media.

To overcome this error, install TL8SP2.

The relevant APAR description is here.

"Release through application and retry"

2008-06-17T15:37:00.002+02:00

When you encounter a locked DS4000 LUN, that is you want to delete it using SM or SMcli and you get refused because it is reserved (or so the storage says) by a host OS that usually does not exist any more or has terminated the FC/SCSI connection abnormally, here's what you can do.

Use SM LUN properties to get the SSID. Log in to each of the controllers using either rlogin or serial connection and check the LUN in question (LUN 0 in this example):

-> hlmTestLunShow 0x0
LunNumber:0x0 LunInfo :0x15f35d4 State:0x0 QuiescenceFlag:0x0
Owner:0x1 IsReady :0x1
reserveId:0xe resv3rdId:0xffff
value = 131 = 0x83

If reserveId (0xe in this example) is not 0xffff, the LUN is reserved. To release the lun, type:

-> hlmTestRelease 0xe,0x0

You should be okay now. Now check the other controller as well.

It seems to be a controller firmware problem. Mine is at 06.12.16.00. Newer versions may differ.

Link to forum article.

About virtual SCSI devices

2008-04-24T14:45:00.003+02:00

Concurrent access to virtualized SCSI disks is possible starting VIOS 1.5. Assign the disks (or, rather, their adapter) to the VIOS partition, create the virtual SCSI server adapters (vhost0...vhostn), and map the disks to the vhosts like this (as padmin):

$ mkvdev -dev hdisk2 -vadapter vhost0 -dev conc1dev
$ mkvdev -f -dev hdisk2 -vadapter vhost1 -dev conc2dev

Concurrent access to logical volume backed virtual SCSI disks is not yet supported at this level of VIOS.

The latest OpenSSL for AIX...

2008-04-24T14:38:00.000+02:00

...can be found here. IBM ID required.

No console after fresh install

2008-04-22T20:50:00.004+02:00

You have a fresh install of AIX , you're finished with the Installation Assistant, try to log in and...there is no login prompt? Here's why.

The root case problem here is with the cable/terminal. The AIX system is expecting CD (Carrier Detect) to be true before the port will open. This is the expected and behavior. The typical cable would connect DTR (Data Terminal Ready) from the terminal to CD of the AIX serial port. When the terminal is powered on, DTR on the terminal is asserted thus presenting CD to the AIX system. If the terminal is powered off, the terminals DTR drops, thus dropping CD at the AIX system. With the loss of CD, a login session would be killed and a new login prompt would be presented when the terminal is powered back on.

AIX (and POSIX UNIX systems) support the CLOCAL option. This option effectively ignores the lack of CD. AIX offers two states at which time CLOCAL is applied. The first is: STTY attributes for RUN time and the second is: STTY attributes for LOGIN If you enable CLOCAL only for LOGIN, the typical situation will be that you will be presented the login prompt, you enter a login and before the prompt appears, you are again presented the login prompt. What just happened is that CLOCAL on the LOGIN attribute allows the login prompt to be presented and we transition into run time mode. There is no CD or CLOCAL, so the process is killed and a new login prompt is presented.

Setting CLOCAL for both LOGIN as well as RUN time will resolve this, with one huge potential problem and security risk. If the terminal is powered cycled there is no way for the AIX system to detect this, and when theterminal is powered back on, the previous process will still be active.The only true solution is to provide the proper cable and RS-232 signaling.

How to create virtual devices, a quick summary of commands

2008-04-21T17:54:00.028+02:00

Virtual SCSI disks

Make sure the disk you intend to virtualize either in whole or divided into LV's are available and ready to be used on the VIOS.
Define a sufficient number of virtual SCSI server adapters for the VIOS using the HMC. Make sure they are available on the VIOS.
```
$ cfgdev
```
or
```
$ shutdown -reboot
```
Then check your devices with
```
$ lsdev -virtual
```
You should see vhost0...vhostn.

If you want to slice up your disks, here's the time. Create a VG with
```
$ mkvg -f -vg clientrootvg hdisk2
```
Slice it up with
```
$ mklv -lv clientrootlv clientrootvg 10G
```
If you want to pass on the entire disk to your client OS, skip this step, you already have the device you need, it's hdisk2 in this example.

Check you virtual SCSI server adapters:
```
$ lsdev -vpd | grep vhost
```

Map the virtualized devices to your server adapters:

$ mkvdev -vdev clientrootlv -vadapter vhost0
-dev clientdev

$ mkvdev -vdev hdisk2 -vadapter vhost0 -dev clientdev

Check what you've done:
```
$ lsdev -virtual
$ lsmap -all
```

Virtual SCSI optical drive

Minimum required code level for VIOS is 1.5 (1.5.1.1 is even better).

Check if you already have any storage pools or repositories:
```
$ lssp
$ lsrep
```

If you don't have a storage pool, create one:
```
$ mksp sp hdisk3
```
to create a logical volume backed storage pool, or
```
$ mksp -fb sp -sp rootvg -size 30G
```
to create a mirrored file backed storage pool inside a logical volume backed storage pool.

Now create a repository:
```
$ mkrep -sp sp -size 15G
```

If you have an ISO image, upload it to the padmin home and create a virtual optical disk using
```
$ mkvopt -name aix5307
-file /home/padmin/aix5307.iso -ro
```
or if you have the media loaded and ready in cd0, issue
```
$ mkvopt -name aix5304 -dev cd0 -ro
```

Create a virtual optical drive and map it your client partition using
```
$ mkvdev -fbo -vadapter vhost0
-dev clientdvd
```

To load a media into the virtual drive, use
```
$ loadopt -vtd clientdvd -disk aix5307
```

To unload a virtual drive, use
```
$ unloadopt -vtd clientdvd
```

Virtual ethernet to access external networks

First you need to have a physical ethernet adapter connected to the external network you want to access, say, it's ent0.
You have to define a virtual ethernet adapter for the VIOS using the HMC, don't forget to check the access external network checkbox. Make sure the virtual adapter is available on the VIOS
```
$ lsdev -virtual
```
Check for ent devices with l-lan in their description, take ent1 for this example).
Now you have to define a Shared Ethernet Adapter or SEA to connect the physical network and the virtual ethernet adapter that will bridge packets to the external network.
```
$ mkvdev -sea ent0 -vadapter ent1 -default ent1
-defaultid 1
```
Check what you've done:
```
$ lsmap -all -net
```

For further details, refer to this Redpiece.

fcparray to mpio

2008-02-27T11:40:00.003+01:00

Ever wondered how you could switch your DS4K storage server from RDAC to native AIX MPIO?

Here's how:
http://www-03.ibm.com/systems/storage/disk/ds4000/pdf/aix.pdf

HMC tricks, part two

2007-10-25T11:17:00.005+02:00

To change the default language of the remote wsm client to English instead of your local language here's a hint.

Remote wsm clients are usually started from
c:\program files\websm\bin\wsm.bat. In that file modify the line that starts the wsm Java process by adding this "-Duser.language=en". Mine now looks like this:


java -Duser.language=en -Dremote.auiml="true"
-Xms%W_HEAP_MIN_SIZE% -Xmine%W_HEAP_INC_SIZE%
-Xmx%W_HEAP_MAX_SIZE% -ss128k
-DIDebug.enabled=%DEBUGENABLE%
-DWEBSM_NO_REMOTE_CLASS_LOADING=%WNRCL% 
-DWEBSM_NO_SECURITY_MANAGER=%WNSM% %WSMSSL%
-Djava.security.policy=..\config\websm.policy
-Dawt.appletWarning="Remote class Window" 
-DWEBSM_ALL_PERMISSIONS_FOR_SECURE=true
-DWSMDIR="%WSMDIR%"
-Dremote.auiml="true"
%JAVA_ARG_ACCESSIBLE% %JAVA_ARG_SVK%
com.ibm.websm.console.WConsole

20EE000B on a partitioned 6M2

2007-08-02T11:41:00.000+02:00

After upgrading a 6M2 partition to AIX 5.3 TL6 we found that the LPAR stops booting with firmware error code 20EE000B. We called IBM and they suggested that we upgrade the system firmware. The original version was 3K061030, and we upgraded to 3K070425. The upgrade solved the issue.

HMC tricks, part one

2007-08-02T10:58:00.001+02:00

I am currently working on a system with Release 3 Version 2.6 HMC. I have tried to download the Java-based WebSM client connecting to URL http://hmc/remote_client.html but I have been unsuccesful. For now, the reason is not known and I have had no time to investigate so I have to resort to other methods. Time for command line!

To list all LPARS and the managed system:

[hscroot@hmc hscroot]$ lslpars
=========================================================================
Managed System/LPAR   State           Operator Panel    Activated Profile
=========================================================================
651234A               Ready           LPAR...
 FullSystemPartit... Not Available
 oradev2             Running                           Normal
 rstorat1            Running                           Normal
 rstwast1            Running                           Normal
 iasdev1             Running                           Normal
-------------------------------------------------------------------------

To start a partition:

[hscroot@hmc hscroot]$ start_partition -p "iasdev1" -f "Normal" \
-m "651234A"
Starting partition...
Partition successfully activated

To open a virtual term to an LPAR (vtmenu):


[hscroot@hmc hscroot]$ vtmenu
Retrieving name of managed system(s) . . . 651234A
----------------------------------------------------------
 Partitions On Managed System:  651234A
----------------------------------------------------------
  1)    oradev2                              Running:
  2)    rstorat1                             Running:
  3)    rstwast1                             Running:
  4)    iasdev1                              Running:


Enter Number of Running Partition (q to quit): 4

   Opening Virtual Terminal On Partition iasdev1 . . .

NVTS hmc.domain.hu 9734 001*7038-6M2*651234A 1 001*7038-6M2*651234A _VT_

AIX Version 5
(C) Copyrights by IBM and by others 1982, 2007.
Console login:

To open a virtual terminal to an LPAR (mkvterm):


[hscroot@hmc hscroot]$ mkvterm -m "651234A" -p "iasdev1"
NVTS hmc.domain.hu 9734 001*7038-6M2*651234A 1 001*7038-6M2*651234A _VT_

AIX Version 5
(C) Copyrights by IBM and by others 1982, 2007.
Console login:

Note: Other HMC releases and versions may differ, do your reading before trying any of these commands.

False report on reserved disks

2007-07-19T23:35:00.000+02:00

HACMP 5.2 sometimes reports a false disk reservation on concurrent disks while synchronizing the cluster. It looks like this:

WARNING: serverp1: Read on disk /dev/hdisk2 failed.
Check cables and connections.
A reserve may be set on that disk by another node.

Examining the disks we have found absolutely no reason why they would have a reserve on them, so, for now, it's a bug in HACMP (latest fixes as of today). It must be pointed out, however, that this is only a warning, not an error, and it has no visible effect on cluster function. IBM is called in to investigate. Updates later.

--- UPDATE/August 1st, 2007

An error exists in the cluster.es.server.diag.5.2.0.8 fileset. The error causes the verification to print these warnings. APAR IZ01478 has been opened to correct the issue.

Missing concurrent disks

2007-07-16T19:13:00.000+02:00

Ever wondered why you couldn't add a new disk to your enhanced concurrent VG using C-SPOC in HACMP 5.2? Are you getting a "1800-051 There are no items of this type" message from the smit selector menu?

Well, here's the explanation. C-SPOC uses /usr/es/sbin/cluster/utilities/cllsvg -c -v to list the exisiting concurrent resource groups/volume groups in this SMIT selector and the cllsvg in cluster.es.cspoc.utils.5.2.0.8 has a bug. Because of the bug, cllsvg returns an empty string when you try to list any enhanced concurrent VGs. Here's the bug:

#!/bin/ksh
# IBM_PROLOG_BEGIN_TAG
# This is an automatically generated prolog.
#
# 51haes_r520 src/43haes/lib/libclres/cllsvg.sh 1.5
#
# Licensed Materials - Property of IBM
#
# (C) COPYRIGHT International Business Machines Corp. 1990,2006
# All Rights Reserved
#
# US Government Users Restricted Rights - Use, duplication or
# disclosure restricted by GSA ADP Schedule Contract with IBM Corp.
#
# IBM_PROLOG_END_TAG
# @(#)82 1.5 src/43haes/lib/libclres/cllsvg.sh, hacmp.libcluster,
# 51haes_r520, 0705A_hacmp520 5/16/06 14:27:41
#################################################
# COMPONENT_NAME: UTILITIES
#
# FUNCTIONS: usage
#
# Name: cllsvg
#
# This program list VG's in the HA configuration
#
# Arguments: -v display only active VGs
# -n display the nodes that share the VG
# -s display shared VGs
# -c display concurrent VGs
# -g display VGs for one resource group
#
# Returns: 0 - Success
# 1 - Invalid Arguments
#
# Environment:
#
#################################################

export PATH="$(/usr/es/sbin/cluster/utilities/cl_get_path all)"

[[ "$VERBOSE_LOGGING" = "high" ]] && set -x
[[ "$VERBOSE_LOGGING" = "high" ]] && version='1.5'

usage()
{
echo
echo "Usage: cllsvg [-g resource group] [-v] [-n] [-c or -s]\n"
echo
echo "-v display only active VGs"
echo "-n display the nodes that share each VG"
echo "-s display shared VGs"
echo "-c display concurrent VGs"
echo "-g display VGs for one resource group"
echo
exit 1
}

################################################
# Main
################################################

# get the command line parameters.

if [ -z "$*" ]
then
usage
fi

set -f
set -- `getopt g:vnsc $*`
if [ "$?" != 0 ]
then
usage
fi

ACTIVE=0
NODES=0
SHARED=0
CONC=0
GROUP=""

while [ "$1" != -- ]
do
case $1 in
-v) ACTIVE=1 # Display only active VGs
shift;;
-n) NODES=1 # List nodes that share VG
shift;;
-s) if [ "$CONC" = "1" ]
then
usage
fi
SHARED=1 # List shared VGs
shift;;
-c) if [ "$SHARED" = "1" ]
then
usage
fi
CONC=1 # List concurrent VGs
shift;;
-g) GROUP=$2
shift;shift;;
*) usage

esac
done

if [ "$SHARED" = "$CONC" ]
then
usage
fi

# Retrieve VG info based on command line parameters

lastvg="" # variable used to eliminate redundant lines from cl_getpv
cl_getpv 2>/dev/null | egrep "active|concurrent" | cut -d: -f4- | sort |\
while read vgstring
do
if [ "$ACTIVE" = "1" ]
then
vgstring=`echo $vgstring | grep -v inactive`
fi

vg=`echo $vgstring | cut -d: -f1`
vgtype=`odmget -q "name=$vg" CuAt 2>/dev/null| grep conc_capable | cut -d\" -f 2`

# If CONC=1, we are looking for VGs of type concurrent in HACMP
# terminology. Old style concurrent VG's will have a value of 16.
# Enhanced concurrent VG's have a value of 32. Shared VG's have
# a value of 0. So a non-zero vgtype means the VG is concurrent.

if [ "$CONC" = "1" ]
then
if [ "$vgtype" != "conc_capable" ]
then
vgstring=""
fi
fi

if [ -z "$vgstring" ]
then
continue # Does not match VG type we are looking for
fi

# Because cl_getpv generates a line of output for each PV in a VG, use lastvg and # sorted
# output from cl_getpv to generate only one line of ouput from cllsvg for each VG

if [ "$vg" != "$lastvg" ]
then
lastvg=$vg
rg=`odmget -q "value=$vg and name=VOLUME_GROUP"
HACMPresource 2>/dev/null | grep group | cut -d\" -f 2`
if [ -n "$rg" ]
then
if [[ -n "$GROUP" && "$GROUP" != "$rg" ]]
then
continue # Not part of the RG we are
looking for
fi

# If CONC=0, we are looking for VGs of type 'shared' in
# HACMP terminology, i.e., they belong to non-concurrent
# resource groups. Those can be enhanced concurrent.
# Hence, filter out VGs that belong to concurrent RGs.

pref=`odmget -q "group=$rg" HACMPgroup 2>/dev/null| grep startup_pref | cut -d \" -f 2`
rgtype=`odmget -q "group=$rg" HACMPgroup 2>/dev/null| grep "type ="| grep -v stype | cut -d \" -f 2`

# For non-concurrent RG's we want to see all VG's, whether they
# are concurrent or not, because the RG may be using enhanced
# concurrent VG's for the purpose of fast failover. In HA 5.2
# and above, a startup_pref of "OAAN" (On Any Active # Node) is
# analogous to the pre-5.2 concurrent RG.

if [[ "$CONC" = "0" &&amp;amp;amp;amp;amp; "$rgtype" != "concurrent" && "$pref" != "OAAN" ]] \
|| [[ "$CONC" = "1" &&amp;amp;amp;amp;amp; "$vgtype" = "conc_capable" && ("$rgtype" = "concurrent" || "$pref" = "OAAN") ]]
then
if [ "$NODES" = "1" ]
then
nodes=`odmget -q "group=$rg" HACMPgroup 2>/dev/null| grep "nodes =" | cut -d\" -f 2`
nodes=`echo "$nodes" | tr " " ","`
echo $rg $nodes $vg
else
echo $rg $vg
fi
fi
fi
fi
done

exit 0

Delete the section marked in boldface and you'll be fine.

A7004730

2007-05-24T23:06:00.000+02:00

When your partitioned p5 system starts with one processor and you get A7004730 on your HMC status line, go here, type your machine type and serial number, receive your CuOD key and enable the missing feature on your system. :)

For the record, I got this on a 9133-55A this afternoon.

Tape, no go

2007-05-15T22:14:00.000+02:00

It seems that the PCI-X Ultra320 Dual-channel SCSI RAID controller (F/C 5703; 2 ext. VHDCI connectors) does not support the connection to a 3580 standalone tape drive. Anyone has an idea why?

By the way, the server is a 9111-520.

No ESCape!

2007-05-15T20:20:00.000+02:00

Today I had to configure a DS4300 (FAStT600) through a serial console. It proved to be harder than I had anticipated.

We started from scratch and wanted to configure the ethernet interfaces to be able to use Storage Manager to setup the box. What normally happens is this:

(1) You connect your null-modem cable between your terminal (emulator) and one controller of the DS4000.
(2) You initiate the connection (turn on your terminal).
(3) You keep sending the BREAK signal (Ctrl-Break, or Alt-b in TeraTerm) until you get a message similar to this: "Press <space> for baud rate within 5 seconds".
(4) You press the space key and the baud rate gets set.
(5) You send the BREAK signal a second time and you get this: "Press within 5 seconds: <esc> for SHELL, <break> for baud rate."
(6) You press the ESC key and get the login prompt for the shell.
(7) You enter your password (default: infiniti)
(8) You are in, hoorray!
(9) You start setting up the IP parameters using netCfgSet.
(A) ...

Now, what I saw today was a bit different. In step (3) I got an unusual message:

"Press within 5 seconds: <s> for Service Interface, <break> for baud rate".

I sent a BREAK, the baud rate got set and I saw this:

"Enter Password to access Service Interface (60 sec timeout)".

Has anyone heard of a "Service Interface" within a DS4000 box? No one? Me neither. Even the "DS4000 Best Practices and Performance Tuning Guide" (dated May 9, 2007) has no info on it. This got me a little confused. I tried "infiniti" for a password but it failed authentication. I also tried a few other obvious passwords, like "password" or "service", to no avail.

After 15 minutes I called IBM and asked for help. They told me to press ESC after a second "Press within 5 seconds: <s> for Service Interface, <break> for baud rate"! One of the firmware updates has changed the behavior of the serial port and now the instruction to press ESC is missing! Damn! By the way, the box now has the newst firmware to date, which is v6.23.05 (and DS4000 Storage Manager v9.23 to manage it)

Also, I have found this. I haven't had the chance to test it yet, but when I'm done, I'll share.

One legged beast

2007-05-15T00:08:00.000+02:00

Here's today's hint. Whenever you plan to extend a logical disk in a FAStT 600 (or probably any older series [and maybe new, too] DS disk storage subsystem) and your server (in this case, a W2K3 node) has only a single path to the storage, make sure you disconnect the server somehow (or simply shut it down if you can) before starting the extension or be prepared to be thrown at some unfriendly error messages. I only fixed this through a phone call, so I do not have the exact error text.

In short, single HBA configurations to a FAStT or DS box are not supported and may result in unexpected errors or other strange phenomena. This does not apply to the FAStT 200; this box supports single HBA server connections.

NFS -1

2007-05-13T09:47:00.000+02:00

One of my customers implemented some changes in their domain name system, after which the encountered this:

sapft# mount pr1srvc:/saptrans /usr/sap/trans
nfsmnthelp: 1831-019 pr1srvc: System call error number -1.
mount: 1831-008 giving up on:
pr1srvc:/saptrans
System call error number -1.

I suspected that the error was somewhere in the new domain name system but I was not able to pinpoint the exact cause. I did forward and reverse lookups on all the names and IP addresses involved in the mount process and evereything worked fine as long as I used the host command. However, as soon as I switched to nslookup, the story started to unfold. Since nslookup disregards the settings in the /etc/netsvc.conf file and the NSORDER environment variable, it did not get fooled by the entries in /etc/hosts and I soon figured out that while implementing the new name space the guys had forgotten to add the reverse zones to the static IP addresses (the dynamic, ie. DHCP, addresses were doing fine).

Adding the reverse zones to the config immediately solved the NFS mount issue, all mounts are fine now.

One port up, one port down

2007-05-10T21:54:00.000+02:00

Next time you install an F/C 5759 4Gb dual port FC adapter in a 9113-550 (and probably any other p5 model), make sure you have at least AIX 5.2 TL8 (5200-08). Any lower fix level will result in one of the ports on the adapter to remain unconfigured, even with the latest driver filesets.

This info is from IBM support. We have the upgrade scheduled for tomorrow. I'll let you know when we're done. Oh, one more thing, upgrading the system firmware prior to the AIX upgrade can't hurt either. :)

---UPDATE ONE---
I failed at the very first step: I could not install the firmware file (rpm -ivh --ignoreos ...), I got a strange error. The rest of the update, of course, had to be cancelled. Details later.

---UPDATE TWO---
Here is the exact error text:

rsvoratst:/tmp/fwupdate# rpm -ivh --ignoreos
01SF230_158_120.rpm
stty: tcgetattr: A specified file does not
support the ioctl system call.
execution of 01SF230_158_120-1.1-1 script failed,
exit status 2

The error occurs the exact same way on other 5.2 ML4 servers, even with the latest rpm.rte (3.0.5.39). However, the error does not occur on 5.2TL9 or any 5.3 I have access to now. So, I guess it's an AIX bug. No official IBM answer yet.

---UPDATE THREE---
I have received an answer from IBM, from which it is obvious that they did not even understand the problem in the first place. They suggest that I should replace the blind swap cage that holds the dual-port FC adapter. Hmmm. The model 550 does not use blind swap cages at all. Quite sad.

Seeing that we cannot expect much help from IBM in this particular case, we decided to take the situation in our own hands. First, I unpacked two firmware files (SF230_158 and the latest SF240_320) on another system. Then ftp'd the files to the 550 and...

/usr/lpp/diagnostics/bin/update_flash -f /tmp/fwupdate/01SF230_158_120

Guess what, it worked perfectly. After about 20 minutes the box rebooted from the temporary side and showed SF230_158 as the current microcode level (lsmcode). So far so good. I took the next step:

/usr/lpp/diagnostics/bin/update_flash -f /tmp/fwupdate/01SF240_320_201

This worked great, too. And here comes the funny part. The dual-port FC adapter started working on both ports. fcs2 and fcs3 came up available! And, of course, seeing this, we decided not to upgrade AIX to TL9. We were very near to the end of our maintenance window anyway.

Details unavailable

2007-04-28T12:59:00.001+02:00

When you see something like this on an FC switch Java GUI:

--- start of commit (Enable Config) at: Sat Apr 28 2001, 10:34:02 AM
Commit changes:
Failed to commit changes to fabric: details unavailable.
Commit Failed.
--- end of commit at: Sat Apr 28 2001, 10:34:11 AM

Have no fear! The problem is probably not fatal, it seems to be a Java error somewhere; either in your browser, or in the applet that's loaded from the switch, which means a Fabric OS error. We did not have a chance to further test it or upgrade the FOS, so we took the other way. We telneted into the switch and used the command line method.

Example for adding new aliases and zones to a config:

SWITCHA:admin> aliCreate "srvFotoA", "21:00:01:0e:b0:47:21"
SWITCHA:admin> aliCreate "sto3583D1", "50:00:00:61:b1:22:f3"
SWITCHA:admin> zoneCreate "zne_FotoA_3583D1", "srvFotoA; sto3583D1"
SWITCHA:admin> cfgAdd "cfgProdA", "zne_FotoA_3583D1"
SWITCHA:admin> cfgEnable "cfgProdA"

If you still have the error, contact your next level of support. And by the way, the switch was an IBM 2109-F16.

E170 and 0xdeadbeef

2007-04-27T23:38:00.000+02:00

I had a routine AIX maintenance the other day. We went from 5200-02 to 5200-09 on a 7028-6C4 (ie. eServer pSeries 630 Model 6C4). The update went just fine; however, when we rebooted the server, it stopped at E170. It took us some time to figure out what had gone wrong, so here's a little help for you.

When you encounter an E170 and you see loads of 0xdeadbeef and an Open Firmware prompt on your serial console, you're running on backlevel firmware and should upgrade to go past E170. The current level of firmware for the aforementioned server at the time of writing this post is 3R061030, dated 12/08/2006. You can get it here.

Somewhere between ML02 and TL09 the size of the kernel increased beyond the 12MB limit and the old firmware we had could not handle it. If you're reading this because you're preparing for such an upgrade, you're lucky to be in a position where you can simply use the AIX command line method to upgrade and save some trouble for yourself. On the other hand, if you're reading this because you smell 0xdeadbeef, you'd better get your floppies ready for action. Good luck.

Ps.: In some cases we saw a Service Processor reset after E170 (B0FF, then back to OK). Also, if you have done the firmware upgrade and still can't get past E170, try a Service Processor reset from the SP menu (14900 and choose the safe method, option 1, I think). If you're still stuck, remove the SP battery for a few minutes. If this does not help, contact your next level of IBM support, you may have a hardware issue.

Nothing happened, or my Day One

2007-04-26T22:55:00.000+02:00

We wanted to migrate a 2.8 TB Orcale database (residing on one single LV/FS plus one jfs2log) from a set of three DS8100 logical disks to another set of 28 DS8100 logical disks in another storage complex. The original three-disk VG contained hdisk14, hdisk15, and hdisk16. We added the new 28 disks to the VG (by the way, it was the scalable type) and started the migration from one source disk at a time like this:

migratepv hdisk16 hdisk1 hdisk2 hdisk4 hdisk5 ...

The command ran for about 24 hours and gave us an error, something like this:

0516-132 lmigratelv: Incorrect entry in partition map file.
0516-134 lmigratelv: Received: 000a122aff313e72 6008
0516-136 lmigratelv: Error reading input map.
0516-812 migratepv: Warning, migratepv did not completely succeed;
all physical partitions have not been moved off the PV.

Examinig the migration status of the LV, we saw that not a single partition had been moved during the 24-hour run of migratepv. How strange, we thought to ourselves. Something was not right. So we started to test the situation a bit more. We ran the migratepv command again and checked disk I/O activity closely using nmon and iostat and we found that practically no I/O was present. The iostat showed all zeroes for the destination disks and very little I/O for the source disks, which was normal since the Oracle database was up and running while we were testing.

The AIX error log was empty, pcmpath query adapter showed some errors but we knew they had been caused by a prior incident, unrelated to our present situation. fcstat on both HBAs also showed some errors but, again, unrelated.
We found a case that suggested that the problem may have been caused by a full /tmp (migratepv uses some temporary tables that are built in /tmp) but our /tmp filesystem was 2GB in size and only 6% full. Unlikely, unless the temp tables were huge. Further investigation showed they weren't, less than 20MB both. Dead end.

We decided to run another test. We tried to migrate the data using mklvcopy/syncvg/rmlvcopy. We thought this way we could see what was happening in more detail. Well, we were wrong. We got stuck at the very first step:

mklvcopy oradatalv 2 hdisk1 hdisk2 hdisk4 hdisk5...

Even without any data synchronization, the mklvcopy never exited (well, we only waited for about 30 minutes before we hit ctrl-c) and generated no disk I/O whatsoever. Mind you, that during this time the database was up and running fine on disks that were in a similar DS8100, connected through the same HBAs, using the exact same SDDPCM driver.

At this point we were out of ideas and decided to upgrade all AIX components. We went from AIX 5300-05-04 to 5300-05-06 and upgraded SDDPCM from 2.1.0.8 to 2.1.2.5, rebooted and waited for the miracle. But...nothing happened.

Since we were having time constraints on the migration, we came up with a plan to migrate the data without having to use any LV commands which might be faulty (migratepv and/or lmigratelv). We created an new scalable VG and started moving the data using a number of parallel cp commands. Performance is nothing extraordinary but we expect it to finish in about 12 to 20 hours. Thank God, it's only a test LPAR and we can afford to have such a long downtime for the database. IBM has alredy been called in to investigate. I will update this post as soon as they have found anything.

To be continued...

---UPDATE ONE---
Well, I have no good news for you, guys. Due to space constraints the disks with the funny VG on it had to be recycled and all the evidence with it. IBM Support had the intention to solve the issue but time was not on our side. We copied all data to a newly created scalable VG and had to destroy the problematic one. Sorry, no solution for this one.
If you happen to encounter the same phenomenon and have a solution, please let me know so I can share it here. Thanks.