Advanced features¶
Both directory and its child items coexist in litefile table¶
As described in the above chapters, we can add the files/directories to litefile table. Sometimes, it is necessary to put one directory and also its child item(s) into the litefile table. Due to the implementation of the statelite on Linux, some scenarios works, but some doesn’t work.
Here are some examples of both directory and its child items coexisting:
Both the parent directory and the child file coexist:
"ALL","/root/testblank/",,, "ALL","/root/testblank/tempfschild","tempfs",,One more complex example:
"ALL","/root/",,, "ALL","/root/testblank/tempfschild","tempfs",,Another more complex example, but we don’t intend to support such one scenario:
"ALL","/root/",,, "ALL","/root/testblank/",,, "ALL","/root/testblank/tempfschild","tempfs",,
For example, in scenario 1, the parent is /root/testblank/
, and the child is /root/testblank/tempfschild
.
In scenario 2, the parent is /root/
, and the child is /root/testblank/tempfschild
.
In order to describe the hierarchy scenarios we can use , P
to denote parent, and C
to denote child.
Option | Example | Remarks |
---|---|---|
P:tmpfs | “ALL”,”/root/testblank/”,,, “ALL”,”/root/testblanktempfschild”,”tempfs”,, | Both the parent and the child are mounted to tmpfs on the booted node following their respective options. Only the parent are mounted to the local file system. |
P:tmpfs C:persistent | “ALL”,”/root/testblank/”,,, “ALL”,”/root/testblank/testpersfile”,”persistent”,, | Both parent and child are mounted to tmpfs on the booted node following their respective options. Only the parent is mounted to the local file |
P:persistent C:tmpfs | “ALL”,”/root/testblank/”,”persistent”,, “ALL”,”/root/testblank/tempfschild”,,, | Not permitted now. But plan to support it. |
P:persistent C:persistent | “ALL”,”/root/testblank/”,”persistent”,, “ALL”,”/root/testblank/testpersfile”,”persistent”,, | Both parent and child are mounted to tmpfs on the booted node following their respective options. Only the parent is mounted to local file system. |
P:ro C:any | Not permitted | |
P:tmpfs C:ro | Both parent and child are mounted to tmpfs on the booted node following their respective options. Only the parent is mounted to local file system. | |
P:tmpfs C:con | Both parent and child are mounted to tmpfs on the booted node following their respective options. Only the parent is mounted to local file system. | |
P:link C:link | “ALL”,”/root/testlink/”,”link”,, “ALL”,”/root/testlink/testlinkchild”,”link”,, | Both parent and child are created in tmpfs on the booted node following their respective options; there’s only one symbolic link of the parent is created in the local file system. |
P: link C: link, persistent | “ALL”,”/root/testlinkpers/”,”link”,, “ALL”,”/root/testlink/testlinkchild”,, “link,persistent” | Both parent and child are created in tmpfs on the booted node following their respective options; there’s only one symbolic link of the parent is created in the local file system. |
C: link |
“ALL”,”/root/testlinkpers/”,”link,persistent”,, “ALL”,”/root/testlink/testlinkchild”,”link” |
NOT permitted |
|
“ALL”,”/root/testlinkpers/”,”link,persistent”,, “ALL”,”/root/testlink | Both parent and child are created in tmpfs on the booted node following “link,persistent” way; there’s only one symbolic link of the parent is created in the local file system. |
P:link C:link,ro | “ALL”,”/root/testlink/”,”link”,, “ALL”,”/root/testlink/testlinkro”,”link,ro”,, | Both parent and child are created in tmpfs on the booted node, there’s only one symbolic link of the parent is created in the local file system. |
P:link C:link,con | “ALL”,”/root/testlink/”,”link”,, “ALL”,”/root/testlink/testlinkconchild”,”link,con”,, | Both parent and child are created in tmpfs on the booted node, there’s only one symbolic link of the parent in the local file system. |
C:link,ro |
NOT Permitted | |
C:link,con |
NOT Permitted | |
P:tmpfs C:link | NOT Permitted | |
P:link C:persistent | NOT Permitted |
litetree table¶
The litetree table controls where the initial content of the files in the litefile table come from, and the long term content of the ro
files. When a node boots up in statelite mode, it will by default copy all of its tmpfs files from the /.default
directory of the root image, so there is not requirement to setup a litetree table. If you decide that you want some of the files pulled from different locations that are different per node, you can use this table.
See litetree man page for description of attributes.
For example, a user may have two directories with a different /etc/motd
that should be used for nodes in two locations:
10.0.0.1:/syncdirs/newyork-590Madison/rhels5.4/x86_64/compute/etc/motd
10.0.0.1:/syncdirs/shanghai-11foo/rhels5.4/x86_64/compute/etc/motd
You can specify this in one row in the litetree table:
1,,10.0.0.1:/syncdirs/$nodepos.room/$nodetype.os/$nodetype.arch/$nodetype.profile
When each statelite node boots, the variables in the litetree table will be substituted with the values for that node to locate the correct directory to use. Assuming that /etc/motd
was specified in the litefile table, it will be searched for in all of the directories specified in the litetree table and found in this one.
You may also want to look by default into directories containing the node name first:
$noderes.nfsserver:/syncdirs/$node
The litetree prioritizes where node files are found. The first field is the priority. The second field is the image name (ALL for all images) and the final field is the mount point.
Our example is as follows:
1,,$noderes.nfsserver:/statelite/$node
2,,cnfs:/gpfs/dallas/
The two directories /statelite/$node
on the node’s $noderes.nfsserver and the /gpfs/dallas
on the node cnfs contain root tree structures that are sparsely populated with files that we want to place in those nodes. If files are not found in the first directory, it goes to the next directory. If none of the files can be found in the litetree hierarchy, then they are searched for in /.default
on the local image.
Installing a new Kernel in the statelite image¶
Obtain you new kernel and kernel modules on the MN, for example here we have a new SLES kernel.
Copy the kernel into /boot :
cp **vmlinux-2.6.32.10-0.5-ppc64**/boot
Copy the kernel modules into
/lib/modules/<new kernel directory>
/lib/modules # ls -al total 16 drwxr-xr-x 4 root root 4096 Apr 19 10:39 . drwxr-xr-x 17 root root 4096 Apr 13 08:39 .. drwxr-xr-x 3 root root 4096 Apr 13 08:51 2.6.32.10-0.4-ppc64 **drwxr-xr-x 4 root root 4096 Apr 19 10:12 2.6.32.10-0.5-ppc64**
Run genimage to update the statelite image with the new kernel
genimage -k 2.6.32.10-0.5-ppc64 <osimage_name>
Then after a nodeset command and netbooti, shows the new kernel:
uname -a
Enabling the localdisk option¶
Note
You can skip this section if not using the localdisk
option in your litefile
table.
Define how to partition the local disk¶
When a node is deployed, the local hard disk needs to be partitioned and formatted before it can be used. This section explains how provide a configuration file that tells xCAT to partition a local disk and make it ready to use for the directories listed in the litefile table.
The configuration file needs to be specified in the partitionfile
attribute of the osimage definition. The configuration file includes several sections:
- Global parameters to control enabling or disabling the function
[disk]
section to control the partitioning of the disk[localspace]
section to control which partition will be used to store thelocaldisk
directories listed in thelitefile
table[swapspace]
section to control the enablement of the swap space for the node.
An example localdisk
configuration file:
enable=yes
enablepart=no
[disk]
dev=/dev/sda
clear=yes
parts=10,20,30
[disk]
dev=/dev/sdb
clear=yes
parts=100M-200M,1G-2G
[disk]
dev=/dev/sdc
ptype=gpt
clear=yes
parts=10,20,30
[localspace]
dev=/dev/sda1
fstype=ext3
[swapspace]
dev=/dev/sda2
The two global parameters enable
and enablepart
can be used to control the enabling/disabling of the functions:
- enable: The
localdisk
feature only works whenenable
is set toyes
. If it is set tono
, thelocaldisk
configuration will not be run.- enablepart: The partition action (refer to the
[disk]
section) will be run only whenenablepart=yes
.
The [disk]
section is used to configure how to partition a hard disk:
- dev: The path of the device file.
- clear: If set to
yes
it will clear all the existing partitions on this disk.- ptype: The partition table type of the disk. For example,
msdos
orgpt
, andmsdos
is the default.- fstype: The file system type for the new created partitions.
ext3
is the default.- parts: A comma separated list of space ranges, one for each partition that will be created on the device. The valid format for each space range is
<startpoint>-<endpoint>
or<percentage of the disk>
. For example, you could set it to100M-10G
or50
. If set to50
, 50% of the disk space will be assigned to that partition.
The [localspace]
section is used to specify which partition will be used as local storage for the node.
- dev: The path of the partition.
- fstype: The file system type on the partition.
the [swapspace]
section is used to configure the swap space for the statelite node.
- dev: The path of the partition file which will be used as the swap space.
To enable the local disk capability, create the configuration file (for example in /install/custom
) and set the path in the partitionfile
attribute for the osimage:
chdef -t osimage <osimage> partitionfile=/install/custom/cfglocaldisk
Now all nodes that use this osimage (i.e. have their provmethod
attribute set to this osimage definition name), will have its local disk configured.
Configure the files in the litefile table¶
For the files/directories to store on the local disk, add an entry in the litefile
table:
"ALL","/tmp/","localdisk",,
Note
you do not need to specify the swap space in the litefile table. Just putting it in the partitionfile
config file is enough.
Add an entry in policy table to permit the running of the getpartition
command from the node
chtab priority=7.1 policy.commands=getpartition policy.rule=allow
Run genimage
and packimage
for the osimage
If Using the RAMdisk-based Image¶
If you want to use the local disk option with a RAMdisk-based image, remember to follow the instructions in Switch to the RAMdisk based solution.
If your reason for using a RAMdisk image is to avoid compute node runtime dependencies on the service node or management node, then the only entries you should have in the litefile table should be files/dirs that use the localdisk option.
Debugging techniques¶
When a node boots up in statelite mode, there is a script that runs called statelite that is in the root directory of
$imageroot/etc/init.d/statelite
. This script is not run as part of the rc scripts, but as part of the pre-switch root environment. Thus, all the linking is done in this script. There is aset x
near the top of the file. You can uncomment it and see what the script runs. You will then see lots of mkdirs and links on the console.You can also set the machine to shell. Just add the word
shell
on the end of the pxeboot file of the node in the append line. This will make the init script in the initramfs pause 3 times before doing a switch_root.When all the files are linked they are logged in
/.statelite/statelite.log
on the node. You can get into the node after it has booted and look in the/.statelite
directory.