Dan Rumney DCR January 22, 2009 With the introduction of Multiple-Target FlashCopy and Cascaded FlashCopy to IBM System Storage SAN Volume Controller (SVC) , it can become difficult to keep track of which Virtual Disks are mapped to which, and how FlashCopy Mappings (FCM) depend upon one another. While submitting svcinfocommands or using the SVC Console can provide this information in a textual format, there are times when a diagram provides all the information you need in an easy to understand format. This paper outlines a method for generating diagrams that link VDisks and FlashCopy Mappings. It also serves as a worked example of automation on the SVC Command Line. This paper assumes familiarity with SVC and the FlashCopy functionality.

When SVC was introduced in 2003 it included a number of Copy Services: FlashCopy and Remote Copy. FlashCopy is a Point-In-Time Copy Service, whereby the contents of a Source Virtual Disk (Source) is copied to a Target Virtual Disk (Target), such that the Target is an exact copy of the Source, at that point in time. The relationship between a Source and a Target is called a FlashCopy Mapping (FCM). The original implementation was such that a Source could only ever have one active Target. In addition to this, a Target of one FCM could not be the Source of another FCM. A number of FCMs can be gathered together into a FlashCopy Consistency Group (FCG) and managed as a single entity, to ensure that all Target VDisks in the FCG represent the exact same point in time. With the release of SVC 4.2.1, new types of FlashCopy arrangements can be created. A single VDisk can act as the Source to multiple Targets. In addition to this, a VDisk which is acting as the Target of one FCM can also act as the Source of a different FCM. For more details on Copy Services in SVC, see the Redbook SVC 4.2.1 Advanced Copy Services SG24-7574-00 . In a complex environment, the interactions between FCMs and VDisks can get quite involved. Dependencies between FCMs stem from internal data structures within the cluster rather than the logical connections between VDisks. All of these dependencies can be discovered by submitting the appropriate svcinfo commands, but the information is presented in a purely textual way; this provides no insight into the interaction between cluster objects. shows how this output looks. svcinfo lsfcmap -delim : id:name:source_vdisk_id:source_vdisk_name:target_vdisk_id:target_vdisk_name: group_id:group_name:status:progress:copy_rate:clean_progress:incremental 0:fcmap0:1039:vdisk1039:1040:vdisk1040:::idle_or_copied:100:93:100:on 1:fcmap1:1041:vdisk1041:1042:vdisk1042:::idle_or_copied:100:30:100:off 2:fcmap2:1043:vdisk1043:1044:vdisk1044:::idle_or_copied:100:88:100:on 3:fcmap3:1045:vdisk1045:1046:vdisk1046:::idle_or_copied:100:36:100:off 4:fcmap4:1046:vdisk1046:1047:vdisk1047:::idle_or_copied:100:96:100:on IBM_2145:cluster_name:admin> IBM_2145:cluster_name:admin>svcinfo lsfcmapdependentmaps -delim : 2 fc_id:fc_name 1:fcmap1 3:fcmap3]]> The output from the svcinfo command does not lend itself to a swift overview of the cluster state. We look to the DOT language to generate a graphical representation of this information.

The DOT Language is a language used to describe directed and undirected graphs. Once written, the DOT can then be processed by an appropriate program, to render the graph on screen. A directed graph consists of nodes and edges. A node is a graphical shape which may or may not contain some text. The nodes are interconnected by lines, called edges. In a directed graph, the edges have arrows at one or both ends. shows a sample directed graph. Ellipses a,b, c and d are all nodes.

Sample directed graph generated by a DOT parser

The DOT language is quite straightforward; for instance, the DOT required to generate can be seen in . The nodes are represented by the various letters. In each line in the example, you can see the 'arrow' notation which indicates a directed edge, joining two nodes. digraph EXAMPLE { a -> c; b -> c; c -> d; d -> a; } A DOT file only describes the graph. You must pass this file to a rendering program in order to generate a graphical representation. Once such program is dot, which is part of the Graphviz package. Once installed, the invocation shown in will generate a GIF file rendering of the sample directed graph described in . dot -T gif -o sampleDigraph.gif -K dot -v sampleDiGraph.dot The capabilities of the DOT language lend themselves directly to the challenge of visualizing FlashCopy Mapping and VDisk relationships. The challenge is to generate the DOT necessary to generate our required visualization. In order to do this, we can use SVC Command Line Scripting.

The SVC Command Line Interface is based on a restricted Bash Shell. This provides us with a double-edged opportunity. On one hand, the Bash Shell means that we have the opportunity to execute scripts while logged in to the SVC Command Line. On the other hand, the restricted aspect strongly limits what we can do and results in the need for some imaginative scripting. The Bash Shell includes a number of 'built-in' commands that can be used, such as: if while for read echo The restrictions on the Shell mean that there is no access to scripting standards such as sed, awk and grep. In addition, IO cannot be redirected to files. However, command output can be redirected to other commands, via pipes. contains the script that we'll be discussing for the remainder of this paper. This script creates a CLI function which, when executed, generates DOT language which describes the connections between a set of VDisks and FCMs. The CLI function has one parameter, which is the id of one of the cluster's VDisks. Given a VDisk ID, this command follows the procedure below. Search for any FCMs which have the provided VDisk as a Source or Target For any FCMs found, identify the counterpart VDisks and repeat step 1 for each of the new VDisks For all of the FCMs found, search for all dependant FCMs Generate the DOT language, describing the VDisk and FCM interactions shows example output from this function. When rendered by dot, the graphic shown in makeFCMapTree 1212 digraph F { 1210 [style=filled,fillcolor=green] 514 [shape=box,height=0.4,width=0.4,fillcolor=green,style=filled] 1210 -> 514 514 -> 1212 515 [shape=box,height=0.4,width=0.4,fillcolor=green,style=filled] 1210 -> 515 515 -> 1213 1212 [style=filled,fillcolor=green] 1213 [style=filled,fillcolor=green] 1211 [style=filled,fillcolor=green] 512 [shape=box,height=0.4,width=0.4,fillcolor=green,style=filled] 1211 -> 512 512 -> 1209 513 [shape=box,height=0.4,width=0.4,fillcolor=green,style=filled] 1211 -> 513 513 -> 1210 1209 [style=filled,fillcolor=green] }]]> The ellipses represent VDisks. The rectangles represent FlashCopy Mappings. The colours of the shapes represent the state of the FlashCopy Mappings and VDisks. In one glance, the interactions between VDisks and FlashCopy Mappings are clear and any issues are immediately obvious.

Sample FlashCopy graph

In the next section, we will go through this script line by line and explain what is being done. The techniques used in this script can be used in scripts of your own to perform whatever actions you require. If you prefer, you can skip to to learn how to use the script.

When the script in is executed, it creates a new function called makeFCMapTree in the active SVC CLI session. The script itself has no output. Once the function has been created, it is invoked by providing the function name and a VDisk ID. It generates output in the DOT language which can be captured and saved for later rendering. In this section, we analysis the script section by section and explain its function.

The first 18 lines are concerned with setting up variables for use during the main loop. The fifteen first lines simply start the function and set up some variables assigning colours to FCM and VDisk states. These can be changed to suit your needs, with the requirement that all 10 states are present and the colours that are selected are part of the DOT language. shows the colours supported by DOT. This line outputs the first line of DOT, which indicates that we'll be describing a directed graph. We call it F for FlashCopy, but the name is entirely arbitrary (within the constraints of the DOT language) In this line, we create an array called $possSrcs. This array will be treated as a FIFO stack. It will contain a list of VDisks which are possibly acting as the Sources of an FCM. At this point in the execution, we add the VDisk ID that was provided as the sole parameter to the function. In subsequent passes through the main loop, new VDisk IDs may be added, as required. Here, we create an empty array which will keep track of all VDisk IDs which have been processed. This is to ensure that the script processes each VDisk ID once and once only.

These lines start the main loop and gather information about the next VDisk ID in the $possSrcs stack. This starts the main loop of this script. The loop will execute as long as there are VDisk IDs in $possSrcs. This loop ends on line 56. Lines 20-21 perform the pop operation of a stack, removing the first element from $possSrcs and assigning it to $currSrc. Lines 22-26 show a technique that will be repeated a number of times in this script. This technique is to run an svcinfo command and execute a series of commands based on each line of output from the svcinfo command. The technique has the following form: svcinfo xxxx | while read var1 var2 var3 rest; do some commands using $var1, $var2, etc done This technique will take each line of output from the svcinfo command and pass it to the read built-in command. The read command works in the following way: read name The read command takes a line from STDIN and splits it into words separated according to the Internal Field Separator (IFS). By default, IFS is set to whitespace. The first word is assigned to the first name, the second to the second name and so on. Any leftover words are assigned to the final name with the intervening IFS included. In the technique shown above, the variables $var1, $var2, etc can now be used by commands inside the while loop. The loop will repeat once per line of output from the svcinfo command. The variable $rest is needed to capture any remaining values at the end of the line of output. The command svcinfo lsvdisk -nohdr -filtervalue id=$currSrc will generate zero or one line of output. (depending on whether the value in $currSrc is an actual VDisk ID). This line acts to 'dereference' the VDisk status. The code $(eval echo \$$vStatus) will take the value stored in $vStatus and treat it as the name of a variable and try to find the value stored in that. For instance, if $vStatus is equal to 'preparing' then this line will look for the value found in $preparing and, in this case, return 'yellow'. In this way, we can assign colours to VDisk states. This generates lines of DOT which define a node in the graph. In this instance a VDisk node is generated. The VDisk node is an ellipse, filled in a colour representing the VDisk's state.

These two lines generate an array of FCM Target VDisk IDs and mark the current VDisk ID (held in $currentSrc) as having been processed. Lines 27-29 create a new array that holds all of the FCMs which have the current VDisk (represented by $currSrc) as the Source VDisk. This array is called $newTgts. These three lines shows a method of generating an array directly from the output of an svcinfo command. In this particular instance, each line of svcinfo output generates 3 array elements: $newTgts[L] FlashCopy Mapping ID $newTgts[L+1] Target VDisk ID $newTgts[L+2] FlashCopy Mapping status where L increases by one per line of svcinfo output. Line 30 keeps track of the fact that we have now processed the VDisk ID(as an FCM Source... it may appear later as an FCM Target).

These lines process the FCMs that were placed into $newTgts and generate the appropriate DOT to represent them. Lines 31 creates the loop to process $newTgts. It creates an index variable which is incremented by 3 for each pass (since 3 elements in an array represent one FC Mapping). Lines 32 to 34 simply collect the relevant array elements in to clearer variable names. Line 35 decodes the FC Mapping status into a colour, much like line 24. Lines 36-39 generate DOT language; lines 36 and 37 creates an FC Mapping node, which is a square filled with a colour that represents its state. Lines 38 and 39 generate the edges that link the FC Mapping node with its Source and Target VDisks. Lines 40-42 look at the FCM Target and determine whether or not it has been processed as a Source. If it has not, it is added to our list of possible Sources: $possSrcs. The code in line 41 acts to place this new VDisk ID at the end of the stack. Lines 43-45 look to see if there are any dependencies between this FCM and other FCMs. If there are, a new edge is generated to indicate this.

Lines 48-55 perform a similar task to line 27 and the loop that follows it. Line 48 generates an array of FCM Source VDisk IDs, which have $currentSrc as their Target. This array is called $newSrcs. Instead of generating any DOT in the loop, however, this loop simply adds the indicated VDisk IDs to the $possSrcs array, if they have not previously been processed.

All that remains at this point is to complete the loops, close out the DOT language with a curly bracket and the function is complete.

There are two simple ways to use the script. It can be executed directly from an interactive CLI session, or it can be used as part of a batch session.

Adding the makeFCTreeMap function to an interactive session is very straightforward. Simply copy the full text of the script into the clipboard and then paste it into the terminal. Once the script has executed, the makeFCTreeMap function will be available to you for the remainder of that CLI session. Once you've passed a VDisk ID to the function, you will need to copy the output from the CLI session and place it into a dot file for rendering.

Adding the makeFCTreeMap function to a batch session depends on your SSH client. Here, we will discuss PuTTY for the Windows operating system and ssh for Linux or AIX. Whichever operating system you use, the output from the script will be returned to the STDOUT stream on your local system. You can redirect this output to a dot file, and then pass it to your rendering application.

When a script is submitted to the SVC Cluster in this way, STDIN is replaced by the contents of the script and executed as if it was typed in manually. Once the end of the file is reached, control returns to the local command line and not the SVC command line. Since the normal function of the makeFCTreeMap script is to create a new function in the CLI session (and nothing more), the following modifications are needed to generate output: Delete lines 1 and 2 Delete line 58 Replace $1 in line 17 with the ID of the VDisk that you're interested in. Once these changes have been made, the resulting script_file should be submitted to the cluster using one of the methods shown in the next subsections.

The plink command comes as part of the PuTTY application. shows the command to use to submit a script to an SVC cluster. plink -l admin -m script_file -i private_key_file cluster_name script_file The file containing the makeFCTreeMap script private_key_file An SSH private key which corresponds to a public key that has been uploaded to the SVC cluster in question cluster_name The IP address or DNS name

The ssh command comes with most (if not all) *nix operating systems. shows the invocation required to submit a script to an SVC cluster using ssh. ssh -i private_key_file -T admin@cluster_name < script_file script_file The file containing the makeFCTreeMap script private_key_file An SSH private key which corresponds to a public key that has been uploaded to the SVC cluster in question cluster_name The IP address or DNS name

The script in functions correctly for all possible VDisk IDs, including ones that are not present on the cluster. However, there are some interesting changes that could be made to enhance the script. These are offered as suggestions and are left to the reader to implement: Handle Multiple VDisk IDs The current version of the makeFCTreeMap only accepts a single VDisk ID. It would be a fairly simple task to change the script to allow the function to accept any number of VDisk IDs. It would be important to check for duplicate VDisk IDs appearing in the $possSrcs stack. Handle FlashCopy Mapping IDs Expanding this script to support FCM IDs instead of VDisk IDs is the straightforward task of taking the FCM ID, determining the Source VDisk's ID and placing this into the $possSrcs stack and then proceeding as before. The challenge lies in making the one script support VDisk IDs and FCM IDs. Handle FlashCopy Consistency Group IDs Handling FlashCopy Consistency Groups is the natural combination of handling multiple VDisk IDs and handling FlashCopy Mappings, since an FCG is simply a group of FCMs. The following code will turn an FCG id into an array of FCM ids:

The script below has been formatted so that it will fit onto the page. As a result, line continuation operators have been used on lines 22, 27, 28, 36 and 48. fc$fcm"; echo "fc$fcm -> $tgt"; if [ "${processed[$tgt]}" != "y" ]; then possSrcs=(${possSrcs[@]} $tgt); fi; svcinfo lsfcmapdependentmaps -nohdr $fcm | while read fcId fcName; do echo "$fcm -> $fcId [style=dotted]" done done; newSrcs=(`svcinfo lsfcmap -nohdr -filtervalue target_vdisk_id=$currSrc| \ while read id name srcId junk; do echo "$srcId "; done`); for src in ${newSrcs[@]}; do if [ "${processed[$src]}" != "y" ]; then possSrcs=(${possSrcs[@]} $src); fi; done; done; echo "}"; }]]> The following are acceptable colours in the DOT language: This list can also be found at Graphviz Color Names aliceblue antiquewhite antiquewhite1 antiquewhite2 antiquewhite3 antiquewhite4 aquamarine aquamarine1 aquamarine2 aquamarine3 aquamarine4 azure azure1 azure2 azure3 azure4 beige bisque bisque1 bisque2 bisque3 bisque4 black blanchedalmond blue blue1 blue2 blue3 blue4 blueviolet brown brown1 brown2 brown3 brown4 burlywood burlywood1 burlywood2 burlywood3 burlywood4 cadetblue cadetblue1 cadetblue2 cadetblue3 cadetblue4 chartreuse chartreuse1 chartreuse2 chartreuse3 chartreuse4 chocolate chocolate1 chocolate2 chocolate3 chocolate4 coral coral1 coral2 coral3 coral4 cornflowerblue cornsilk cornsilk1 cornsilk2 cornsilk3 cornsilk4 crimson cyan cyan1 cyan2 cyan3 cyan4 darkgoldenrod darkgoldenrod1 darkgoldenrod2 darkgoldenrod3 darkgoldenrod4 darkgreen darkkhaki darkolivegreen darkolivegreen1 darkolivegreen2 darkolivegreen3 darkolivegreen4 darkorange darkorange1 darkorange2 darkorange3 darkorange4 darkorchid darkorchid1 darkorchid2 darkorchid3 darkorchid4 darksalmon darkseagreen darkseagreen1 darkseagreen2 darkseagreen3 darkseagreen4 darkslateblue darkslategray darkslategray1 darkslategray2 darkslategray3 darkslategray4 darkslategrey darkturquoise darkviolet deeppink deeppink1 deeppink2 deeppink3 deeppink4 deepskyblue deepskyblue1 deepskyblue2 deepskyblue3 deepskyblue4 dimgray dimgrey dodgerblue dodgerblue1 dodgerblue2 dodgerblue3 dodgerblue4 firebrick firebrick1 firebrick2 firebrick3 firebrick4 floralwhite forestgreen gainsboro ghostwhite gold gold1 gold2 gold3 gold4 goldenrod goldenrod1 goldenrod2 goldenrod3 goldenrod4 gray gray0 gray1 gray2 gray3 gray4 gray5 gray6 gray7 gray8 gray9 gray10 gray11 gray12 gray13 gray14 gray15 gray16 gray17 gray18 gray19 gray20 gray21 gray22 gray23 gray24 gray25 gray26 gray27 gray28 gray29 gray30 gray31 gray32 gray33 gray34 gray35 gray36 gray37 gray38 gray39 gray40 gray41 gray42 gray43 gray44 gray45 gray46 gray47 gray48 gray49 gray50 gray51 gray52 gray53 gray54 gray55 gray56 gray57 gray58 gray59 gray60 gray61 gray62 gray63 gray64 gray65 gray66 gray67 gray68 gray69 gray70 gray71 gray72 gray73 gray74 gray75 gray76 gray77 gray78 gray79 gray80 gray81 gray82 gray83 gray84 gray85 gray86 gray87 gray88 gray89 gray90 gray91 gray92 gray93 gray94 gray95 gray96 gray97 gray98 gray99 gray100 green green1 green2 green3 green4 greenyellow grey grey0 grey1 grey2 grey3 grey4 grey5 grey6 grey7 grey8 grey9 grey10 grey11 grey12 grey13 grey14 grey15 grey16 grey17 grey18 grey19 grey20 grey21 grey22 grey23 grey24 grey25 grey26 grey27 grey28 grey29 grey30 grey31 grey32 grey33 grey34 grey35 grey36 grey37 grey38 grey39 grey40 grey41 grey42 grey43 grey44 grey45 grey46 grey47 grey48 grey49 grey50 grey51 grey52 grey53 grey54 grey55 grey56 grey57 grey58 grey59 grey60 grey61 grey62 grey63 grey64 grey65 grey66 grey67 grey68 grey69 grey70 grey71 grey72 grey73 grey74 grey75 grey76 grey77 grey78 grey79 grey80 grey81 grey82 grey83 grey84 grey85 grey86 grey87 grey88 grey89 grey90 grey91 grey92 grey93 grey94 grey95 grey96 grey97 grey98 grey99 grey100 honeydew honeydew1 honeydew2 honeydew3 honeydew4 hotpink hotpink1 hotpink2 hotpink3 hotpink4 indianred indianred1 indianred2 indianred3 indianred4 indigo ivory ivory1 ivory2 ivory3 ivory4 khaki khaki1 khaki2 khaki3 khaki4 lavender lavenderblush lavenderblush1 lavenderblush2 lavenderblush3 lavenderblush4 lawngreen lemonchiffon lemonchiffon1 lemonchiffon2 lemonchiffon3 lemonchiffon4 lightblue lightblue1 lightblue2 lightblue3 lightblue4 lightcoral lightcyan lightcyan1 lightcyan2 lightcyan3 lightcyan4 lightgoldenrod lightgoldenrod1 lightgoldenrod2 lightgoldenrod3 lightgoldenrod4 lightgoldenrodyellow lightgray lightgrey lightpink lightpink1 lightpink2 lightpink3 lightpink4 lightsalmon lightsalmon1 lightsalmon2 lightsalmon3 lightsalmon4 lightseagreen lightskyblue lightskyblue1 lightskyblue2 lightskyblue3 lightskyblue4 lightslateblue lightslategray lightslategrey lightsteelblue lightsteelblue1 lightsteelblue2 lightsteelblue3 lightsteelblue4 lightyellow lightyellow1 lightyellow2 lightyellow3 lightyellow4 limegreen linen magenta magenta1 magenta2 magenta3 magenta4 maroon maroon1 maroon2 maroon3 maroon4 mediumaquamarine mediumblue mediumorchid mediumorchid1 mediumorchid2 mediumorchid3 mediumorchid4 mediumpurple mediumpurple1 mediumpurple2 mediumpurple3 mediumpurple4 mediumseagreen mediumslateblue mediumspringgreen mediumturquoise mediumvioletred midnightblue mintcream mistyrose mistyrose1 mistyrose2 mistyrose3 mistyrose4 moccasin navajowhite navajowhite1 navajowhite2 navajowhite3 navajowhite4 navy navyblue oldlace olivedrab olivedrab1 olivedrab2 olivedrab3 olivedrab4 orange orange1 orange2 orange3 orange4 orangered orangered1 orangered2 orangered3 orangered4 orchid orchid1 orchid2 orchid3 orchid4 palegoldenrod palegreen palegreen1 palegreen2 palegreen3 palegreen4 paleturquoise paleturquoise1 paleturquoise2 paleturquoise3 paleturquoise4 palevioletred palevioletred1 palevioletred2 palevioletred3 palevioletred4 papayawhip peachpuff peachpuff1 peachpuff2 peachpuff3 peachpuff4 peru pink pink1 pink2 pink3 pink4 plum plum1 plum2 plum3 plum4 powderblue purple purple1 purple2 purple3 purple4 red red1 red2 red3 red4 rosybrown rosybrown1 rosybrown2 rosybrown3 rosybrown4 royalblue royalblue1 royalblue2 royalblue3 royalblue4 saddlebrown salmon salmon1 salmon2 salmon3 salmon4 sandybrown seagreen seagreen1 seagreen2 seagreen3 seagreen4 seashell seashell1 seashell2 seashell3 seashell4 sienna sienna1 sienna2 sienna3 sienna4 skyblue skyblue1 skyblue2 skyblue3 skyblue4 slateblue slateblue1 slateblue2 slateblue3 slateblue4 slategray slategray1 slategray2 slategray3 slategray4 slategrey snow snow1 snow2 snow3 snow4 springgreen springgreen1 springgreen2 springgreen3 springgreen4 steelblue steelblue1 steelblue2 steelblue3 steelblue4 tan tan1 tan2 tan3 tan4 thistle thistle1 thistle2 thistle3 thistle4 tomato tomato1 tomato2 tomato3 tomato4 transparent turquoise turquoise1 turquoise2 turquoise3 turquoise4 violet violetred violetred1 violetred2 violetred3 violetred4 wheat wheat1 wheat2 wheat3 wheat4 white whitesmoke yellow yellow1 yellow2 yellow3 yellow4 yellowgreen