Operation Daily Magic Error

This tool can be used to compare copies of a mirrored LV. Comes in handy sometimes.

#!/bin/ksh

#------- constants ----------
WORKDIR=/tmp/mircmp$$
WORKDIR2=/tmp/mircmp
STOPFILE_GLOBAL=/tmp/stop_mircmp
STOPFILE_LOCAL=$WORKDIR/stop_mircmp
MBSIZE=1048576  # 1MB.  The -S flag takes the io size in MB,
 # so this is a useful value.  We immediately
 # convert this value into the size in 128KB
 # units, but most users probably find it easier
 # to think in terms of MB.
LTGSIZE=131072  # 128KB.  This is actually the min possible
 # ltg size for a vg, and has nothing to do
 # with the current ltgsize value for the vg.
 # This constant is useful since readlvcopy
 # takes io size and offset in 128KB chunks
 # rather than DBSIZE (512 byte) chunks.  This
 # is the finest granularity of io that readlvcopy
 # supports, so its the finest granularity we
 # can support in this script.


#------- global variables --------

# flags and flag values.  For documentation of each
# flag please see the Usage function.
cFLAG=
lFLAG=
lVAL=
sFLAG=
sVAL=
SFLAG=
SVAL=
vFLAG=
vVAL=
zFLAG=
wFLAG=
wVAL=

# general global variables
LVNAME=
LVID=
VGNAME=
VGID=
PPSIZE=
PPSIZE_P2=
NLTGS_PERPP=
NMB_PERPP=
IO_SIZE=



#------- utility functions --------

# Documents the script.  Called for any syntax errors.
Usage()
{
echo "Usage: mircmp -l lvname | -v vgname"
echo "  [-s iosize in 128KB | -S iosize in MB]"
echo "  [-c] [-b begin lp#] [-e end lp# | -n numlps]"
echo ""
echo "Purpose: Verify mirror consistency."
echo ""
echo "Required Flags:"
echo " -l lvname examine this lv."
echo " -v vgname each lv within the vg will be "
echo "   examined.  Any non-mirrored lvs"
echo "   within the vg will be skipped."
echo ""
echo "Other options:"
echo " -c   display cksum of comparison files as"
echo "   we perform each io.  The naming"
echo "   convention for the comparison files"
echo "   is lvname.lp#.io#.a/b/c/s,"
echo "   where a=primary mirror copy, b=secondary"
echo "   mirror copy, c=tertiary mirror copy, and"
echo "   s=second pass at the primary mirror copy"
echo "   to check for io in flight."
echo " -b begin begin with this lp#. By default this"
echo "   is the start of the lv."
echo " -e end  end at this lp#.  By default this is"
echo "   the end of the lv.  The -e and -n"
echo "   flags are mutually exclusive."
echo " -n numlps number of lps to examine.  The -e"
echo "   and -n flags are mutually exclusive."
echo " -s 128KB io size specified in 128KB units."
echo " -S 1MB  io size specified in MB."
echo " -z  saves the inconsistent blocks into /tmp/mircmp"
echo " -w workdir use specified workdir instead of /tmp"
echo ""
echo "Description:"
echo "  Verify mirror consistency by reading each mirror"
echo "  copy of a logical volume and comparing the data"
echo "  against the other copies.  Report whether or not"
echo "  the mirrors for that lp have matching copies."
echo "  Ignore the vgsa stale bits which tell lvm whether"
echo "  the lp is stale or fresh, and just compare the actual"
echo "  data.  If the mirrors don't match a second pass"
echo "  attempts to determine if io is in flight."
echo ""
echo "False Failures:"
echo "  One important thing to note is that if an lv is"
echo "  mirrored at creation time (with the mklv -c flag)"
echo "  then the mirror copies won't match until that"
echo "  particular 512 byte block is written to.  This is"
echo "  done because it saves a lot of time during the"
echo "  creation of a large lv (we don't have to do io"
echo "  to the logical partitions, we just have to generate"
echo "  a map of the allocations).  In short, lvs that were"
echo "  mirrored at creation time will show mirror"
echo "  mismatches unless each 512 byte block of the lv"
echo "  has been written to.  These are false failures"
echo "  and do not indicate a problem with the mirrors."
echo ""
echo "  If an lv is created without mirroring and then"
echo "  mklvcopy is used to mirror, lvm will once again"
echo "  not do any io to the new mirror copies, and the"
echo "  new mirror copies will be marked stale.  Once"
echo "  syncvg is run the new mirror copies will be"
echo "  marked fresh and the mirror copies will match."
echo ""
echo "  Stale partitions will show up as mirror miscompares"
echo "  because mircmp completely ignores the vgsa stale bits."
echo ""
echo "Graceful Termination:"
echo "  To stop all running instances of the mircmp process"
echo "  create a file called /tmp/stop_mircmp.  To stop a"
echo "  particular mircmp process, create a file called"
echo "  /tmp/mircmp/stop_mircmp.  These stopfiles"
echo "  are used for graceful termination.  There is a signal"
echo "  handling function as well but you'd rather use that."
echo "  If mircmp is terminated with a signal it will not exit"
echo "  gracefully and the /tmp/mircmp will be left"
echo "  behind."
echo ""
echo "Space Considerations:"
echo "  Each /tmp/mircmp will contain N+1 files,"
echo "  N is the number of mirrors for the lv.  The +1"
echo "  comes from the way mircmp performs a second pass"
echo "  to check for io in flight if the first pass detects"
echo "  a mirror mismatch.  In the second pass we re-read"
echo "  from the primary mirror copy and compare against"
echo "  the first pass primary mirror copy data.  If the"
echo "  first pass and second pass find equal data we"
echo "  assume io is not in fight and this is a true mirror"
echo "  mismatch."
echo ""
echo "  The size of each file is known as the iosize and"
echo "  can be specified with either the -s or -S flag."
echo "  By default the iosize=ppsize for the vg that"
echo "  mircmp is operating on.  If the user specifies an"
echo "  iosize > ppsize then we set iosize=ppsize.  As"
echo "  the iosize decreases, the /tmp space requirement"
echo "  decreases and the performance slows down."
echo ""
echo "  If the -z flag is used, /tmp/mircmp will contain"
echo "  saved versions of every block who's copies do not match."
echo "  This could require significant space in /tmp, as described"
echo "  above."
echo ""

cleanup_exit
}

# Called if we get a signal.
cleanup_signal()
{
cleanup_exit
}

# Gracefully cleanup our work directory
# and exit.
cleanup_exit()
{
trap '' 0 1 2 15
rm -rf $WORKDIR
exit
}

# Useful function for checking return codes.
# Errors are fatal.  First parameter is the
# rc, second parameter is the error string.
test_return()
{
if [ "$1" != 0 ]
then
echo "mircmp: fatal error: $2"
cleanup_exit
fi
}

# Check return code from readlvcopy.  This is a
# special case where a read error shouldn't be
# fatal unless /tmp is full.
test_io_return()
{
if [ "$1" != 0 ]
then
TIO_RC=`df /tmp | grep -v "%Used" | awk '{print $4}'`
if [ "$TIO_RC" = "100%" ]; then
 echo "mircmp: fatal error: $2"
 echo ""
 echo "mircmp: /tmp is full.  Please choose"
 echo "  a smaller iosize using the -s or"
 echo "  -S parameter and try again."
else
 echo "mircmp: fatal error: $2"
fi
cleanup_exit
fi
}

# Check and see if the user has specified either
# a global or local stopfile.  If so then
# gracefully cleanup our work directory and exit.
check_stopfile()
{
if [ -f "$STOPFILE_GLOBAL" ]; then
echo "global stopfile detected"

trap '' 0 1 2 15
rm -rf $WORKDIR
exit
elif [ -f "$STOPFILE_LOCAL" ]; then
echo "local stopfile detected"

trap '' 0 1 2 15
rm -rf $WORKDIR
exit
fi
}

# This is the function that is called on each logical
# partition.  Loop reading io size chunks of the
# partition and comparing that chunk across all
# mirrors.  Once the entire lp has been examined,
# report results to stdout.
process_lp()
{
LP=$1

if [ -z "$LP" ]; then
echo "mircmp: process_lp: no lp specified"
cleanup_exit
fi

# Compute the start of the lp in 128KB units.
((LP_OFF = $LP - 1))
((LP_SKIP = $LP_OFF * $NLTGS_PERPP))

NLP_IOEQUAL=0
NLP_IOFLIGHT=0
NLP_IOMISMATCH=0
CUR_IO=1
while [ "$CUR_IO" -le "$NIOS_PERPP" ];
do
# Check for global and local stopfiles
# each time through the loop so that we
# can fail gracefully if necessary.
check_stopfile

# Compute the distance from the start of
# the lp to the current io.  Once again
# this is in 128KB units.
((TMP_IO_OFF = $CUR_IO - 1))
((IO_OFF = $TMP_IO_OFF * $IO_SIZE))
((IO_SKIP = $LP_SKIP + $IO_OFF))

if [ $IO_SKIP -eq 0 ]; then
 SKIP_STRING=""
else
 SKIP_STRING="-s $IO_SKIP"
fi

# Naming convention for the comparison
# files is lp#.io#.a/b/c/s, where
# a=first mirror copy, b=second mirror copy,
# c=third mirror copy, and s=second pass against
# the first mirror copy.

readlvcopy -d /dev/r$LVNAME -c 1 $SKIP_STRING \
  -n $IO_SIZE -o $WORKDIR/$LP.$CUR_IO.a >/dev/null 2>&1
test_io_return $? "readlvcopy $LP.$CUR_IO.a"

readlvcopy -d /dev/r$LVNAME -c 2 $SKIP_STRING \
  -n $IO_SIZE -o $WORKDIR/$LP.$CUR_IO.b >/dev/null 2>&1
test_io_return $? "readlvcopy $LP.$CUR_IO.b"

cmp -s $WORKDIR/$LP.$CUR_IO.a $WORKDIR/$LP.$CUR_IO.b
CMP_B_RC=$?

if [ $NCOPIES -eq 3 ]; then
 readlvcopy -d /dev/r$LVNAME -c 3 $SKIP_STRING \
   -n $IO_SIZE -o $WORKDIR/$LP.$CUR_IO.c >/dev/null 2>&1
 test_io_return $? "readlvcopy $LP.$CUR_IO.c"

 cmp -s $WORKDIR/$LP.$CUR_IO.a $WORKDIR/$LP.$CUR_IO.c
 CMP_C_RC=$?
else
 CMP_C_RC=0
fi

# If mirrors match increment NLP_IOEQUAL.
#
# If mirrors don't match we perform a second pass
# which reads mirror "a" again and compares it to
# mirror a data from the first pass.  If mirror
# "a" has changed between the first and second pass
# we increment NLP_IOFLIGHT, otherwise we increment
# NLP_IOMISMATCH.
if [[ $CMP_B_RC -eq 0 ]] && [[ $CMP_C_RC -eq 0 ]]; then
 SECOND_PASS=0
 ((NLP_IOEQUAL = $NLP_IOEQUAL + 1))
else
 SECOND_PASS=1
 readlvcopy -d /dev/r$LVNAME -c 1 $SKIP_STRING \
   -n $IO_SIZE -o $WORKDIR/$LP.$CUR_IO.s >/dev/null 2>&1
 test_io_return $? "readlvcopy $LP.$CUR_IO.s"

 cmp -s $WORKDIR/$LP.$CUR_IO.a $WORKDIR/$LP.$CUR_IO.s
 CMP_S_RC=$?

 if [ $CMP_S_RC -eq 0 ]; then
   ((NLP_IOMISMATCH = $NLP_IOMISMATCH + 1))
 else
   ((NLP_IOFLIGHT = $NLP_IOFLIGHT + 1))
 fi
fi

# Display the cksum for the comparison files if the
# -c flag is specified.  Then cleanup the comparison files
if [ -n "$cFLAG" ]; then
  cksum $WORKDIR/$LP.$CUR_IO.a $WORKDIR/$LP.$CUR_IO.b
  if [ $NCOPIES -eq 3 ]; then
   cksum $WORKDIR/$LP.$CUR_IO.c
  fi
  if [ $SECOND_PASS -eq 1 ]; then
   cksum $WORKDIR/$LP.$CUR_IO.s
  fi
fi

# Remove all the files related to this particular
# io_size chunk.  Note that the .c and .s files
# may not exist but because we're using rm -f
# we don't care if the files don't exist.
if [ -n "$zFLAG" -a "$CMP_B_RC" -ne "0" -a "$CMP_S_RC" -eq "0" ] ; then
 mv $WORKDIR/$LP.$CUR_IO.a $WORKDIR2/${LVNAME}_LP${LP}_${CUR_IO}.1
 mv $WORKDIR/$LP.$CUR_IO.b $WORKDIR2/${LVNAME}_LP${LP}_${CUR_IO}.2
fi

if [ -n "$zFLAG" -a "$CMP_C_RC" -ne "0" -a "$CMP_S_RC" -eq "0" ] ; then
 if [ -e "$WORKDIR/$LP.$CUR_IO.a" ]; then
  mv $WORKDIR/$LP.$CUR_IO.a $WORKDIR2/${LVNAME}_LP${LP}_${CUR_IO}.1
 fi
 mv $WORKDIR/$LP.$CUR_IO.c $WORKDIR2/${LVNAME}_LP${LP}_${CUR_IO}.3
fi

rm -f $WORKDIR/$LP.$CUR_IO.a $WORKDIR/$LP.$CUR_IO.b \
  $WORKDIR/$LP.$CUR_IO.c $WORKDIR/$LP.$CUR_IO.s

((CUR_IO = $CUR_IO + 1))
done

# Tally the results and report them to stdout.
((TALLY_IOS = $NLP_IOEQUAL + $NLP_IOFLIGHT + $NLP_IOMISMATCH))
if [ $TALLY_IOS -ne $NIOS_PERPP ]; then
echo "$LVNAME.LP$LP internal error $NLP_IOS / $NIOS_PERPP"
elif [ $NLP_IOMISMATCH -gt 0 ]; then
echo "$LVNAME.LP$LP mismatch"
elif [ $NLP_IOFLIGHT -gt 0 ]; then
echo "$LVNAME.LP$LP inflight"
else
echo "$LVNAME.LP$LP equal"
fi
}


# This is the function that is called on each logical
# partition.  Query the number of lps and copies
# for this lv.  Determine the range of lps to scan,
# depending on what the user asked for (by default
# we scan every lp in the lv).  Loop scanning the lps.
# If the lv is non-mirrored or if this particular lv
# has no lps within the range the user was asking about,
# simply print a message and continue.
process_lv()
{
LVNAME=$1

if [ -z "$LVNAME" ]; then
echo "mircmp: process_lv: no lvname specified"
cleanup_exit
fi

# use lvname to get lvid, vgname, vgid.
LVID=`getlvodm -l $LVNAME`
test_return $? "getlvodm -l $LVNAME"

      CHARS=`lquerylv -L $LVID -cC`
test_return $? "lquerylv -L $LVID -cC"
      if [ -n "$CHARS" ]; then
              set $CHARS
              LVSIZE_LPS=$1
              NCOPIES=$2
else
echo "mircmp: process_lv: lquerylv failed"
cleanup_exit
      fi

# Determine the range of lps to examine.  If the
# user asked to scan lps 300-500 and this lv in
# the vg only has 5 lps then we print the invalid
# range error message and move on to the next lv.
#
# If -b was specified, make sure the -b value is
# not past the end of this lv.  If -e or -n was
# specified, make sure that we stop at the end of
# the lv.  It is perfectly valid to specify -b
# without -e or -n.  In that case we use the last
# lp in the lv as the stopping point.  It is also
# perfectly valid to specify -b without -e or -n.
# In that case we use the first lp in the lv as
# the starting point. 
PLV_INVALID_RANGE=0
if [ -n "$bFLAG" ]; then
if [ $bVAL -gt $LVSIZE_LPS ]; then
 PLV_INVALID_RANGE=1
else
 CUR_LP=$bVAL
fi
else
CUR_LP=1
fi

if [ -n "$eFLAG" ]; then
if [ $eVAL -gt $LVSIZE_LPS ]; then
 END_LP=$LVSIZE_LPS
else
 END_LP=$eVAL
fi
elif [ -n "$nFLAG" ]; then
((TMP_nVAL = $nVAL - 1))
((nEND = $CUR_LP + $TMP_nVAL))
if [ $nEND -gt $LVSIZE_LPS ]; then
 END_LP=$LVSIZE_LPS
else
 END_LP=$nEND
fi
else
END_LP=$LVSIZE_LPS
fi

# Check for global and local stopfiles
# so that we can fail gracefully if necessary.
check_stopfile

if [ $NCOPIES -lt 2 ]; then
echo "===$LVNAME is not mirrored==="
elif [ $PLV_INVALID_RANGE -eq 1 ]; then
echo "===$LVNAME invalid lp range specified==="
else
echo "===starting $LVNAME==="
while [ "$CUR_LP" -le "$END_LP" ]
do
 process_lp $CUR_LP

 ((CUR_LP = $CUR_LP + 1))
done
fi
}

process_vg()
{
PVG_VGNAME=$1

PVG_LVLIST=`lsvg -l $PVG_VGNAME | \
awk '$1 != "LV" && $1 !~ ":" {print $1}'`

for PVG_LV in $PVG_LVLIST
do
process_lv $PVG_LV
done
}


# ------ main function -------

# Setup signal handler.
trap 'cleanup_signal' 0 1 2 15

# Parse input and deterine if there is a syntax error.
set -- `getopt b:ce:l:n:s:S:v:zw: "$@"`
if [ $? != 0 ]; then
Usage
fi

# For more detail on flag specific information see
# the Usage function.
while  [ $1  !=  -- ]
do
case $1 in
-b) bFLAG='-b';  bVAL=$2;  shift;  shift;;   # begin lp
-c) cFLAG='-c';  shift;;       # show cksum
-e) eFLAG='-e';  eVAL=$2;  shift;  shift;;   # end lp
-l) lFLAG='-l';  lVAL=$2;  shift;  shift;;   # lvname
-n) nFLAG='-n';  nVAL=$2;  shift;  shift;;   # numlps
-s) sFLAG='-s';  sVAL=$2;  shift;  shift;;   # io_size_in_128KB
-S) SFLAG='-S';  SVAL=$2;  shift;  shift;;   # io_size_in_MB
-v) vFLAG='-v';  vVAL=$2;  shift;  shift;;   # vgname
-z) zFLAG='-z';  shift;;                     # save readlvcopy output
-w) wFLAG='-w';  wVAL=$2;  shift;  shift;;   # working dir
esac
done

# Must have a target device.
if [[ -z "$lFLAG" ]] && [[ -z "$vFLAG" ]]; then
echo "mircmp: no lvname or vgname specified"
Usage
fi

if [ -n "$lFLAG" ]; then
# The -l lvname flag was specified.  Derive the
# vgname and vgid so that we can determine the ppsize
# of the vg.  This allows us to validate the -s or -S
# value early on in the script if they are specified.
LVNAME=$lVAL

LVID=`getlvodm -l $LVNAME`
test_return $? "getlvodm -l $LVNAME"

VGNAME=`getlvodm -b $LVID`
test_return $? "getlvodm -b $LVID"

VGID=`getlvodm -v $VGNAME`
test_return $? "getlvodm -v $VGNAME"

PPSIZE_P2=`lqueryvg -g $VGID -s`
test_return $? "lqueryvg -g $VGID -s"

((PPSIZE = 1 << $PPSIZE_P2))  ((NLTGS_PERPP = $PPSIZE / $LTGSIZE))  ((NMB_PERPP = $PPSIZE / $MBSIZE)) elif [ -n "$vFLAG" ]; then  # The -v vgname flag was specified.  Derive the   # vgid so that we can determine the ppsize of the   # vg.  This allows us to validate the -s or -S   # value early on in the script if they are specified.     VGNAME=$vVAL;   VGID=`getlvodm -v $VGNAME`  test_return $? "getlvodm -v $VGNAME"    PPSIZE_P2=`lqueryvg -g $VGID -s`  test_return $? "lqueryvg -g $VGID -s"   ((PPSIZE = 1 << $PPSIZE_P2))  ((NLTGS_PERPP = $PPSIZE / $LTGSIZE))  ((NMB_PERPP = $PPSIZE / $MBSIZE)) fi  # Sanity check -s and -S values, so that we  # can determine the io size.   if [[ -n "$sFLAG" ]] && [[ -n "$SFLAG" ]]; then  echo "mircmp: The -s and -S flags are mutually exclusive."  Usage fi  if [ -n "$sFLAG" ]; then  if [ $sVAL -eq 0 ]; then   echo "mircmp: -s value must be > 0"
Usage
fi

# See the comments before NIOS_PERPP_LEFTOVER
# for an explanation of why we have this
# "divisible by 2" restriction.
if [ $sVAL -ne 1 ]; then
((TMP_sVAL = $sVAL % 2))
if [ $TMP_sVAL -ne 0 ]; then
 echo "mircmp: -s value must be evenly divisible by 2"
 Usage
fi
fi

# If the io_size > pp_size then io_size = pp_size.
if [ $sVAL -gt $NLTGS_PERPP ]; then
IO_SIZE=$NLTGS_PERPP
else
IO_SIZE=$sVAL
fi
elif [ -n "$SFLAG" ]; then
if [ $SVAL -eq 0 ]; then
echo "mircmp: -S value must be > 0"
Usage
fi

# Convert from MB to 128KB units.
((IO_SIZE = $SVAL * 8))
else
# If no io size is specified, io_size = pp_size.
IO_SIZE=$NLTGS_PERPP
fi

if [ -n "$wFLAG" ]; then
if [ -d "$wVAL" ]; then
WORKDIR=$wVAL/mircmp$$
WORKDIR2=$wVAL/mircmp
else
echo "mircmp: $wVAL is not a directory! "
Usage
fi
fi

# The ppsize is always a multiple of 128K.  Valid
# ppsizes are 1MB to 128GB incrementing by power
# of 2.
#
# We ensure that the io_size is evenly divisible
# by 2 so that we don't have to worry about any
# case where there are leftovers on the last
# io for the pp.  Here we have a sanity check to
# verify that we don't have any leftovers (ie
# all io's will be the same size).
#
# An example of leftovers would be if we allowed
# a -s value of 3.  Lets assume the ppsize is 1MB.
# Each io would be 384KB until the final io of the
# pp which would be 128KB.  For the sake of keeping
# things simple, we don't allow ourselves to get into
# that state.
((NIOS_PERPP = $NLTGS_PERPP / $IO_SIZE))
((NIOS_PERPP_LEFTOVER = $NLTGS_PERPP % $IO_SIZE))

if [ $NIOS_PERPP_LEFTOVER -ne 0 ]; then
echo "mircmp: non-zero nios_perpp_leftover=$NIOS_PERPP_LEFTOVER"
cleanup_exit
fi

if [[ -n "$eFLAG" ]] && [[ -n "$nFLAG" ]]; then
echo "mircmp: The -e and -n flags are mutually exclusive."
Usage
fi

if [ -n "$bFLAG" ]; then
if [ $bVAL -eq 0 ]; then
echo "mircmp: -b value must be > 0."
Usage
fi
fi

if [ -n "$eFLAG" ]; then
if [ $eVAL -eq 0 ]; then
echo "mircmp: -e value must be > 0."
Usage
fi
fi

if [[ -n "$bFLAG" ]] && [[ -n "$eFLAG" ]]; then
if [ $bVAL -gt $eVAL ]; then
echo "mircmp: -b value > -e value."
Usage
fi
fi

if [ -n "$nFLAG" ]; then
if [ $nVAL -eq 0 ]; then
echo "mircmp: -n value must be > 0."
Usage
fi
fi

# setup work directory
mkdir $WORKDIR

if [ -n "$zFLAG" -a ! -e "$WORKDIR2" ]; then
mkdir $WORKDIR2
fi

# pass control to a vg or lv specific function.
if [ -n "$lFLAG" ]; then
process_lv $lVAL
elif [ -n "$vFLAG" ]; then
process_vg $vVAL
fi

# Reset signals so that we don't go into
# the cleanup_signal function when we exit.
trap '' 0 1 2 15

# cleanup work directory
rm -rf $WORKDIR
exit
Service processor	Server connector	Subnet mask	IP address
Service processor A	HMC1 HMC2	255.255.255.0	192.168.2.147 192.168.3.147
Service processor B (if installed)	HMC1 HMC2	255.255.255.0	192.168.2.146 192.168.3.146
Operation Daily Magic Error

Summary

fscsi device tuning

mircmp tool

TS7520/DS4200 default password

ifor_ls.html.en_US.base.cli does not get updated

Accessing the ASMI

Recovering a lost service processor IP address

HMC console issues

0511-171 Reached end of media

"Release through application and retry"

useful::links

about::me
