Road to P.H.D

Samba Server on Centos HOWTO

2009-08-02T23:02:00.008-04:00

TDB database
root: > smbd -b | grep PRIVATE_DIR
/etc/samba

root: >smbd -b | grep LOCKDIR
/var/cache/samba

use tdbbackup utility to backup those tdb files

mkdir a directory and change its ownship
chown account.group /path/to/directory

Samba Configruation File
root: >smbd -b | grep smb.conf
/etc/samba/smb.conf

Dry run Samba Configurations
root: >testparm /etc/samba/smb.conf
root: >smbclient -L xxx.xxx.xxx.xxx(SambaServerIP) -U%

Samba valid user setup
root: >smbpasswd -a csguest
New SMB password:
Retype new SMB password:
startsmbfilepwent_internal: file /etc/samba/smbpasswd did not exist. File successfully created.
Added user csguest.

chown user.group /path/to/sambashare

Run/start/stop/restart Samba Server
root: >service smb reload
root: >service smb start
root: >service smb stop
root: >service smb condrestart

Mount Samba Share in Linux
root: >apt-get install smbfs
root: >smbmount //SambaServerIP/SambaShare /mnt/Dir -o username=account, password=pwd

Access Samba Share in WindowsXP
Control Panel -> User Accounts -> yourUserAccount
Click Manage my network passwords
Add button to enter Server, User name and Password
Server: xxx.xxx.xxx.xxx(SambaServerIP)
Username: xxx.xxx.xxx.xxx\AccountName
Password:******

My Computer -> Tools -> Map Network Drive
Folder: //SambaServerIP/SambaShare

Centos: nilfs compiling from src

2009-05-01T20:49:00.003-04:00

Download source:
2.6.18 centos kernel source code (kernel-devel-2.6.18-128.el5.centos.plus.i686.rpm)
http://rpm.pbone.net/index.php3/stat/4/idpl/12390365/com/kernel-devel-2.6.18-128.el5.centos.plus.i686.rpm.html
kernel module source code
http://www.nilfs.org/download/nilfs-2.0.12.tar.bz2
patch for Centos
http://www.nilfs.org/pub/centos/5/SRPMS/nilfs-kmod-2.0.12-r.src.rpm
userland utilities source code
http://www.nilfs.org/download/nilfs-utils-2.0.12.tar.bz2

Required packages:
gcc
yum install gcc
e2fsprogs-devel
yum install e2fsprogs-devel

default installed: make binutils

Installation Instructions:
I. use Centos rpm installer to unpackage the kernel source code
The source code will be in the directory :
/usr/src/kernels/2.6.18-128.1.6.el5-i686

2. untar the nilfs kernel module source code and userland utilities source code
kernel module is in the directory: nilfs-2.0.12
utilities is in the directory: nilfs-utils-2.0.12

3. uncompress the nilfs-kmod-2.0.12-r.src.rpm package and copy the patch file nilfs-rhel5-kern-feature.patch to the kernel module src directory nilfs-2.0.12
_______________________________________________________

--- a/fs/kern_feature.h 2008-02-12 08:38:22.000000000 -0000
+++ b/fs/kern_feature.h 2009-04-09 12:27:24.000000000 -0000
@@ -30,6 +30,25 @@
/*
* Please define as 0/1 here if you want to override
*/
+/* for Red Hat Enterprise Linux / CentOS 5.x */
+#if defined(RHEL_MAJOR) && (RHEL_MAJOR == 5)
+# define NEED_MOUNT_SEMAPHORE 1
+# define NEED_INODE_GENERIC_IP 0
+# define NEED_INODE_BLKSIZE 0
+# define HAVE_PAGE_MKWRITE 1
+# if (RHEL_MINOR < 1)
+# define PageChecked PageFsMisc
+# define SetPageChecked SetPageFsMisc
+# define ClearPageChecked ClearPageFsMisc
+# endif
+# if (RHEL_MINOR > 0)
+# define NEED_INC_NLINK 0
+# if (RHEL_MINOR > 2)
+# define NEED_DROP_NLINK 0
+# define HAVE_LE32_64_ADD_CPU 1
+# endif
+# endif
+#endif

/*
* defaults dependent to kernel versions
_______________________________________________________

4. apply the patch to the kernel module source code
root: ~> patch -p1 < nilfs-rehel-kern-feature.patch

5. compile the kernel module source code root: ~> make
root: ~> make install

6. compile the utils source code
root: ~> make
root: ~> make install

Userful resource:
http://www.nilfs.org/download/READMEv2
http://www.cyberciti.biz/faq/appy-patch-file-using-patch-command/
http://rpm.pbone.net/index.php3/stat/4/idpl/12390365/com/kernel-devel-2.6.18-128.el5.centos.plus.i686.rpm.html
http://www.nilfs.org/en/faq.html
http://www.nilfs.org/en/pkg_centos.html

NilFS: tutorial

2009-04-21T20:53:00.009-04:00

Nilfs is a LFS that NTT(Nippon Telegraph and Telephone Corp.) Group distributed to the Linux world.

It has been merged into Linux mainline 2.6.30 rc-1

Below is the tutorial of how to set up a LFS partition in a VMware VM.

1. Attach multiple virtual disks to the VM in VMware

Editing virutal machine settings -> add -> add a hard disk

2. Install a CentOS5.1 distro in the VM and run the more than 260 updates

(It is said CentOS5.1/5.2 are both OK. But CentOS5.0 needs to be patched.)

root : centos

3. Download the NilFS kmod and utils rpms from nilfs.org site.

i386 utilities package: http://www.nilfs.org/pub/centos/5/RPMS/i386/

i386 kernel module package: http://www.nilfs.org/pub/centos/5/RPMS/i686/

Look for the nilfs utilities : nilfs-utils-2.0.12-1.i386.rpm

Look for the exact kernel version 2.8.18_128 : kmod-nilfs-2.0.12-2.2.6.18_128.1.6.el5.i686.rpm

the utilities and the kernel module version should match the same nilfs version and also the centos kernel version.

(It also includes PAE and Xen rpm packages in the same directory)

4. Install the NilFS rpm package

root: ~>rpm -Uvh kmod-nilfs-2.0.12-2.6.18_128.1.6.el5.i686.rpm

5. Format the virtual disk as nilFS and mount to the system.

(The extra disk is /dev/hda)

Format the disk by mkfs command

root: ~>mkfs -t nilfs2 /dev/hda

mount the LFS disk by mount command

root: ~>mount -t nilfs2 /dev/hda /root/nilfs

6. Use the nilFS to recover the data

Before you make some changes, you can set a checkpoint by mkcp

root: ~>mkcp

root: ~>lscp (lscp will show what checkpoints you have)

After you make the changes, you sjhould make the previous checkpoint to be a snapshot and then mount that previous snapshot to a new spot

root: ~>chcp ss 5

root: ~>mount -t nilfs2 -r -0 cp=5 /dev/hda /root/nilfs-cp

nilfs.org website instructions are at the link http://www.nilfs.org/en/about_nilfs.html (how to use nilfs --incomplete)

7. Umount nilFS disk

root: ~>umount /root/nilfs-cp

root: ~>umount /root/nilfs

Next --> Setup NilFS in a Xen Guest

gcc linking error

2009-01-19T13:42:00.004-05:00

Sympton:
/usr/lib/gcc/i586-suse-linux/4.1.2/../../../../i586-suse-linux/bin/ld: errno: TLS definition in /lib/libc.so.6 section .tbss mismatches non-TLS reference in \
/tmp/ccox5xu2.o
/lib/libc.so.6: could not read symbols: Bad value

Diagnosis:
Link error

Analysis:
/usr/include/unistd.h includes features.h which is using errno.h

So errno.h should be included.

Solution:
(Final)
#include before #include

(Temporary)
gcc -include /usr/include/errno.h code.c

Reference:

http://www.robotercoding.com/blog/?p=36
http://blog.chinaunix.net/u1/36607/showart_313647.html

Lab build

2009-01-19T10:21:00.005-05:00

Ubuntu Hardy 8.04 LTS

Sun java
sudo apt-get install sun-java6-jdk - For the JDK (Developer)
sudo apt-get install sun-java6-jre - For the JRE (User)
sudo update-alternatives --config java

Acrobat Reader 8
sudo wget http://www.medibuntu.org/sources.list.d/hardy.list -O /etc/apt/sources.list.d/medibuntu.list
sudo apt-get update && sudo apt-get install medibuntu-keyring && sudo apt-get update
sudo apt-get install acroread

emacs
sudo apt-get install emacsx
smart-compile tag
/usr/share/emacs/site-lisp/smart-compile.el

Solaris xVM ---PV Linux domU

2008-05-25T23:34:00.010-04:00

Solaris system is reinstalled and vanilla. Xen is working. But still something needs to do from the beginning.

VNC configuration
virt-manager

To configure VNC to let it support xen display by the following command:
# svccfg -s xvm/xend setprop config/vncpaswd = astring:\"passwd\"
# svccfg -s xvm/xend setprop config/vnc-listen=astring:\"0.0.0.0\"
# svcadm refresh xvm/xend; svcadm restart xvm/xend

To install virt-manager, download the package and use the following command:
#pkgadd -d. SUNWvirt-manager

The one thing I left undone yet is paravirtual Linux domU. Generally, we have three methods to create a PV Linux domU: virt-install, virt-manager, and xm create.
virt-manager is GUI installlation and supports Fedora well. It is network install.
virt-install is command-line style but works substantially in diverse ways: netinstall, isoinstall, cdrom, etc.
xm create is command-line style too but mainly work in iso or cdrom install, which required configuration file prehand.

First, virt-manager. Due to our specific box, only centos works and DNS in guest still doesn't work properly. So I use the path address: http://128.153.145.19/pub/centos/5/os/i386/
The screenshot for installation can be seen as below:
I am just waiting for its completion and when it restarts, it looks like this:
While it is installing, it can not probe the video card. Either the driver is not ported or the virt-manager is not complete yet.
Note: so far I have no idea of where virt-manager has stored the configuration file for the guest. As I know, in virt-manager log it did log the xml format guest configuration.

To boot the installed CentOS domU guest, you need the following configuration file
name="centosP" #kernel="/var/lib/xen/virtinst-vmlinuz.UAktlH" #ramdisk="/var/lib/xen/virtinst-initrd.img.XxNCk0" disk=['file:/export/home/newbie/centosP.img,xvda,w'] memory=1024 vif=[''] root="/dev/xvda"
Note: virtinst-vmlinuz.UAktlH and virtinst-initrd.img.XxNCk0 are created by virt-manager and when booting the guest, those two lines should be removed. Grub in the image will take over and boot up the guest. BTW, xvda is required in disk option rather than 0,w]

Graphics doesn't work for Paravirtualization in Solaris yet. Also even using virt-manager, CentOS cannot detect properly the video card type, so only text mode network installation works, using a generic video driver. ParaVirtualization is still black-and-white world. Some drivers are missing and need to be ported but they are in progress and almost there.

When we tried virt-install and xm create. It cannot install from iso because cdrom is not paravirtual ported. In another word, we didn't have cdrom paravirtual drivers there. We can have the CentOS boot up, but it will keep asking you the CD-ROM device driver. When booting a Linux-made , XENBUS said block device driver, nic device driver, and the console missing. The details can be seen as below:

It is said that one way to circle around CDROM is to use NFS. But I am not familiy

Solaris DNS

2008-05-25T20:52:00.003-04:00

After saving the files from the last broken solaris, I need to reinstall the Solaris again. But this time, I didn't get that luck so that everything was smoothy. The network is broken. I have dual network cards in that box. One plugin for external, and the other integrated for internal. It only gets an IP 10.0.0.x and cannot work on the external one. I bet the plugg-in network card is broken. So I hook the external cable to the integrated network card. It can get an external IP now, but the DNS server address 2.35 is wrong. It can browse google by 61.233.169.104. But it can not go any further with DNS broken.

Without network, Solaris is a lonely isolated island. Even the sky is grey.

The solaris is default using the facility NWAM to auto configure the network. I have tried to switch to GNOME network configuration tool, it doesn't work that well. So I have to research in NWAM and make it work right.

For more details you can check the pages:
http://opensolaris.org/os/project/nwam/
and
http://docs.sun.com/app/docs/doc/806-1386/6jam5ahnd?a=view

Generally speaking, most of the websites introduce the way to configure DNS in solaris is to modify the related configuration file: /etc/resolv.conf and /etc/nsswitch.conf.
/etc/resolv.conf is to specify the DNS server, search domain, etc.
sample:
search clarkson.edu nameserver 128.153.128.2 nameserver 128.153.4.2

/etc/nsswitch.conf is to specify the resolution order. To resolve an address, it has many ways: files, dns, mdns, etc.
key part of a sample
hosts: files dns mdns ipnodes: files dns mdns
Originally, hosts option only has files, we need to add dns and mdns after it to provide more resolution methods.

How does DNS work in Solaris?
When there is an web address to be resolved, the DNS client in solaris FIRST consults the /etc/nsswitch.conf file to see which to look into first. In this example, the presumed order is local file first, DNS server second.
THEN, if the DNS client consults the local file and does not find an entry, the client consults the /etc/resolv.conf file to look through the name resolution search list and the address of the local DNS server.

It seems that as long as we make these two configure files right, then it is done. However, Solaris doesn't work as Linux does. Linux working style is to modify the configuration, and then restart the service, then all will be good. Solaris style works differently. Everything is in framework. No manually modifying configuration files but there are always tools or facilities to provide consistent modification.

Once the facility nwam can obtain IP from DHCP server, then the DNS should also work properly now. But there is another problem for Clarkson Network. The DHCP server in Clarkson gives me the 2.35 which is not the DNS server at all(I also tried ubuntu, the same result). The key problem is that Clarkson has multiple DNS servers. If it can broadcast and collect all the DNS server responses, then 2.35 is not a problem any more.

Solaris has the facility mdnsd (multicast DNS daemon)but it doesn't automatically run after booting. It is to broadcast to discover the DNS servers.

To make it work, first /etc/nsswitch.conf should be modified to add mdns after hosts option.
Then, to start the service by the following command:

# svcadm enable svc:/network/mdns:default

As the DNS daemon will cache the file. So once the configuration is done, Solaris needs to REBOOT.
-----------------------------------------ADDON-----------------------------------------------------------------------------------
The network interfaces in Solaris is stored in the file /etc/nwam/llp (LLP)
# cat /etc/nwam/llp bge0 dhcp rge0 dhcp
The host name and domain is stored in the file /etc/hosts
# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain solaris

solaris broken

2008-05-25T10:53:00.004-04:00

Solaris is broken again. Maybe it is because I can't wait till the system successfully shut down. So some system files are lost and I was asked to /lib/svc/bin/restore_repository.

But I can have the root console. The FS is read-only. I need to use the command below to change it to be read/write:
/lib/svc/method/fs-root
/lib/svc/method/fs-usr

For more details, you can check the page:
http://www.sun.com/msg/SMF-8000-MY

Now the situation is that I can execute some commands and I have some import files in my home directory, and I don't have network at all. How can I get those files out? The day without network is horribly dark~!

The system recommended to use fsck. I tried, but it doesn't help. It did can find some inconsistency but the Solaris seems to lose some system file.

I try to attach a USB drive to the box and hope it can recognize it. I use the command iostat as below to figure out if the disk can be recognized.
#iostat -En
It shows that the disk is there but it cannot recognize it as it appears as sda1 but not as c1d0...

Fortunately, that box has an The next that is worth trying is to burn those files into a CD. Then I checked the page
http://sysunconfig.net/unixtips/cdr.html
and
http://blogs.sun.com/reed/entry/how_to_burn_cd_dvd
sounds good.

I first package those files in the directory Desktop into ISO file, using the command as below:

# mkisofs -l -L -r -o /desk.iso  /export/home/newbie/Desktop

Then I will burn it into the CD by the following configuration and the command:
1) first check the CD-RW device number

#  cdrecord -scanbus

1,0,0 is the device number

2)configure the CD-RW
create a file as /etc/default/cdrecord, inside are

   CDR_FIFOSIZE=8m
 CDR_SPEED=8
 CDR_DEVICE=USCSI:1,0,0

3)burn the iso image into CD

# cdrecord -v /desk.iso

But it has burn error. Failed again.

Last strike is Live CD. First, I tried Ubuntu Live CD but it cannot recognize the FS. It requires UFS package. Then I have to try Solaris Live CD. Solaris Live CD has 2008.05 Live and installation CD, Benelix, and Schillix. I used 2008.05 release. It doesn't show the disk directly.
Still use the command iostat, you can see the disk is c5d0. Then use the mount command to mount it

# mount -F ufs /desk.iso /mnt

Then go to the directory /mnt, you can file there are three solaris directory there. Go to each one, there is one that is newbie's home directory. So move the ISO file into the home directory, and then copy the ISO from the home directory onto the LiveCD desktop. Then burn it into CD. It works finally.

---------------------------------------------------------ADDON---------------------------------------------------------------------------------------
Then reinstall the Solaris, it turned out that the network is broken. It kept get a wrong address as DNS server. The same to Ubuntu.
In Solaris, it has two styles to configure the network.
One is NWAM facility(automata network Management), the other is GNOME network configuration. You can use svc to alternate those two services.
For more details, man nwam or http://opensolaris.org/os/project/nwam/.

To see the services that are running, use
#svcs

To enable the default GNOME network configuration

     % svcadm enable svc:/network/physical:default

To disable the Solaris Network auto-magic

     % svcadm enable svc:/network/physical:nwam

To start the Solaris DNS server serivce

     % /lib/svc/method/dns-server start|stop

Solaris Virtualbox

2008-05-23T22:26:00.006-04:00

Virtualbox is another full virtualization product, now bought by SUN and can work in solaris. But it cannot work in Solaris xVM. It will report "VirtualBox Kernel Driver not Loaded" error.

I am using OpenSolaris Express B87 and virtualbox1.6.

Virtualbox can be downloaded from the website http://virtualbox.org/wiki/Downloads

Virtualbox is available for all the platforms. For Solaris, it has 32-bit and 64-bit-AMD which corresponds to x-86 or AMD64. You can check by the command
bash-3.2# isainfo
amd64 i386

Then install the package by the pkgadd command
bash-3.2# pkgadd -d VirtualBox-1.6.0-SunOS-amd64-r30421.pkg
Package filename is VirtualBox-1.6.0-SunOS-amd64-r30421.pkg, which will install the package

For installation details, you can refer to READMe.txt, which contains all the instructions.

Run the Virtualbox by the command:
bash-3.2# VirtualBox
Note: VirtualBox V and B are capitalized.

Below is Virtualbox's application interface.

From now on, we will create a Solaris image in Virtualbox. Its image file is vmdk or vdi(virtual disk image). The image is created in the default directory /root/.VirtualBox/VDI. So if the image name is solarisV, it will be solarisV.vid

Once the solaris installation is done, the screenshot can be seen as below:
I saw a file looked like configuration file in the directory ~/.VirtualBox/Machines/solarisV/solarisV.xml.

Next, I tried windows on VirtualBox. It's just the same to follow the normal Windows XP installation steps.
The screenshot can be see as below:
Generally speaking, VirtualBox won't have any problem in installing any guest. It is full virtualization, and it is easy and simple to use and install. No tricky techniques there.

Solaris xVM (con't) --HVM guest

2008-05-21T12:56:00.009-04:00

Still following the instructions on the page http://opensolaris.org/os/community/xen/docs/virtinstall/

Before going on to create the solaris HVM guest, we need first configure VNC. Facility svc will be used.
svccfg is used to configure the service

# svccfg -s xvm/xend setprop config/vncpasswd = astring: \"passwd\"

-s xvm/xend

-s specifies the service

setprop config/vncpasswd = astring: \"passwd\"

setprop specifies the command
Note: You can see that password is quoted by ""

Similiarly,

# svccfg -s xvm/xend setprop config/vnc-listen = astring: \"0.0.0.0\"

This command is to specify the vnc-listen

Another one is svcadm, which is used to start or stop service. It can "enable", "disable", "refresh", and "restart" the service. refresh is to reload the configuration file.
Here the service is xend, which is in xvm/xend

# svcadm refresh xvm/xend; svcadm restart xvm/xend

It restarts the xen daemon xend.

OK~~~~~! Now, let's create the HVM solaris guest.

machine:root>#virt-install -n solarisH --hvm -r 1024 --vnc -f /solarisH.img -s 8 -c /export/home/newbie/sol-nv-b87.iso

Still almost the same options as creating the paravirt solaris guest, -n solarisH specifies domain name, -r 1024 specifies ram size, -f /solarisH.img specifies guest image filename, -s 8 specifies the guest image size of 8G, -c /export/home/newbie/sol-nv-b87.iso indicates installing from ISO file.

The only differences are --hvm which specifies the guest works in HVM mode and --vnc which specifies the guest will have a VNC connection.

When you run the virt-install command, it will first prompt up a window asking VNC password which will be passwd as we have set up before. The rest installation process will look exactly like installing solaris on a real machine. The snapshot can be seen as below.
Note: The virt-install still has the problem with libvirt as below
libvir: Xen Daemon error : GET operation failed:WARNING: virt-install doesn't create the configuration file for the guest image. It only creates an image with an OS system in it. The user himself needs to create the configuration file on their own. One simple configuration file can be as below:
A good reference is here: http://opensolaris.org/os/community/xen/docs/HVMdomains.htm
Once the installation is done, You can see the solaris HVM guest looks like below.

However, virt-install doesn't provide the configuration file for the created image. So we need to make on our own. One way to ask the system to create its configuration for us is to use the command virsh while the guest is running. The instructions can be seen on the page http://docs.sun.com/app/docs/doc/819-2450/6n4o5me9r?a=view
It will produce xml-format configuration file
#virsh dumpxml solarisH > solarisH.xml
Then we also need to use the facility virsh to boot the guest by xml configuration file
#virsh create solarisH.xml
Next is to connect to the VNC. We need to use the tool virt-manage.
#virt-manage
Then select the guest domain and press the button open who then will ask for the vnc password. Then a VNC window is popped up and the interface looks like below:
Another way is to manually create a configuration file as below whose name is solarisH.py and file type is python script.
##########################################################
# -*- mode: python; -*-
#============================================================================
# Python configuration setup for 'xm create'.
# This script sets the parameters used when a domain is created using 'xm create'.
# You use a separate script for each domain you want to create, or
# you can set the parameters for the domain on the xm command line.
#============================================================================

import os, re
arch = os.uname()[4]
if re.search('64', arch):
arch_libdir = 'lib64'
else:
arch_libdir = 'lib'

#----------------------------------------------------------------------------
# Kernel image file.
kernel = "/usr/lib/xen/boot/hvmloader"

# The domain build function. HVM domain uses 'hvm'.
builder='hvm'

# Initial memory allocation (in megabytes) for the new domain.
#
# WARNING: Creating a domain with insufficient memory may cause out of
# memory errors. The domain needs enough memory to boot kernel
# and modules. Allocating less than 32MBs is not recommended.
memory = 1024

# Shadow pagetable memory for the domain, in MB.
# Should be at least 2KB per MB of domain memory, plus a few MB per vcpu.
shadow_memory = 8

# A name for your domain. All domains must have different names.
name = "solarisH"

# 128-bit UUID for the domain. The default behavior is to generate a new UUID
# on each call to 'xm create'.
#uuid = "06ed00fe-1162-4fc4-b5d8-11993ee4a8b9"

#-----------------------------------------------------------------------------
# the number of cpus guest platform has, default=1
vcpus=1

# enable/disable HVM guest PAE, default=0 (disabled)
#pae=0

# enable/disable HVM guest ACPI, default=0 (disabled)
#acpi=1

# enable/disable HVM guest APIC, default=0 (disabled)
#apic=1

# List of which CPUS this domain is allowed to use, default Xen picks
#cpus = "" # leave to Xen to pick
#cpus = "0" # all vcpus run on CPU0
#cpus = "0-3,5,^1" # run on cpus 0,2,3,5

# Optionally define mac and/or bridge for the network interfaces.
# Random MACs are assigned if not given.

vif = [ 'type=ioemu' ]

#----------------------------------------------------------------------------
# Define the disk devices you want the domain to have access to, and
# what you want them accessible as.
# Each disk entry is of the form phy:UNAME,DEV,MODE
# where UNAME is the device, DEV is the device name the domain will see,
# and MODE is r for read-only, w for read-write.

#disk = [ 'file:/export/home/mydisk.raw,hdc,w', 'file:/export/home/install.iso,hda:cdrom,r' ]
disk = [ 'file:/solarisH.img,hda,w']
#disk = [ 'phy:/dev/dsk/c1d0p0,hdc,w', 'file:/export/home/install.iso,hda:cdrom,r' ]

#disk = [ 'phy:/dev/zvol/dsk/mypool/mydisk,hdc,w', 'file:/export/home/install.iso,hda:cdrom,r' ]

#----------------------------------------------------------------------------
# Configure the behaviour when a domain exits. There are three 'reasons'
# for a domain to stop: poweroff, reboot, and crash. For each of these you
# may specify:
#
# "destroy", meaning that the domain is cleaned up as normal;
# "restart", meaning that a new domain is started in place of the old
# one;
# "preserve", meaning that no clean-up is done until the domain is
# manually destroyed (using xm destroy, for example); or
# "rename-restart", meaning that the old domain is not cleaned up, but is
# renamed and a new domain started in its place.
#
# The default is
#
# on_poweroff = 'destroy'
# on_reboot = 'restart'
# on_crash = 'restart'
#
# For backwards compatibility we also support the deprecated option restart
#
# restart = 'onreboot' means on_poweroff = 'destroy'
# on_reboot = 'restart'
# on_crash = 'destroy'
#
# restart = 'always' means on_poweroff = 'restart'
# on_reboot = 'restart'
# on_crash = 'restart'
#
# restart = 'never' means on_poweroff = 'destroy'
# on_reboot = 'destroy'
# on_crash = 'destroy'

on_poweroff = 'destroy'
on_reboot = 'restart'
on_crash = 'preserve'

#============================================================================

# New stuff
device_model = '/usr/' + arch_libdir + '/xen/bin/qemu-dm'

#-----------------------------------------------------------------------------
# boot on floppy (a), hard disk (c) or CD-ROM (d)
# default: hard disk, cd-rom, floppy
boot="cda"
#boot='d'

#-----------------------------------------------------------------------------
# write to temporary files instead of disk image files
#snapshot=1

#----------------------------------------------------------------------------
# enable SDL library for graphics, default = 0
sdl=0

#----------------------------------------------------------------------------
# enable VNC library for graphics, default = 1
vnc=1

#----------------------------------------------------------------------------
# address that should be listened on for the VNC server if vnc is set.
# default is to use 'vnc-listen' setting from /etc/xen/xend-config.sxp
vnclisten="0.0.0.0"

#----------------------------------------------------------------------------
# set VNC display number, default = domid
#vncdisplay=1

#----------------------------------------------------------------------------
# try to find an unused port for the VNC server, default = 1
#vncunused=1

#----------------------------------------------------------------------------
# enable spawning vncviewer for domain's console
# (only valid when vnc=1), default = 0
vncconsole=1

vncpasswd=''

#----------------------------------------------------------------------------
# no graphics, use serial port
#nographic=0

#----------------------------------------------------------------------------
# enable stdvga, default = 0 (use cirrus logic device model)
stdvga=0

#-----------------------------------------------------------------------------
# serial port re-direct to pty deivce, /dev/pts/n
# then xm console or minicom can connect
serial='pty'
#serial='stdio'
#serial='file:/tmp/blah'
#serial='/dev/pts/0'
serial='null'

#-----------------------------------------------------------------------------
# enable sound card support, [sb16|es1370|all|..,..], default none
#soundhw='sb16'

#-----------------------------------------------------------------------------
# set the real time clock to local time [default=0 i.e. set to utc]
#localtime=1

#-----------------------------------------------------------------------------
# start in full screen
#full-screen=1

#-----------------------------------------------------------------------------
# Enable USB support (specific devices specified at runtime through the
# monitor window)
usb=1

# Enable USB mouse support (only enable one of the following, `mouse' for
# PS/2 protocol relative mouse, `tablet' for
# absolute mouse)
usbdevice='mouse'
usbdevice='tablet'
##########################################################
Note: two things in the script needs to be taken care of.
1) disk = [ 'file:/solarisH.img,hda,w']
Here you will see the disk is defined by file location, disk type, and access rights.
rather than in the pv one that you may see disk = [ 'file:/solarisH.img,0,w']
2) serial='null'
The console is none, which means when booting the guest, we cannot access by the console but we can use vnc.
Then we can boot the guest by the command
#xm create solarisH.xml
Note: here we didn't have the option -c for xm create. Neither can we use the command xm console to connect to the guest. VNC will be automatically called to display the graph of the guest.

References:
Here Sun provides a diversity of domUs ready for use: http://www.opensolaris.org/os/community/xen/How-To-8-15-06/install/AugDomUs/

Here Sun provides the sample of HOWTO make a HVM domU configuration file and HOWTO make a Solaris domU:
http://opensolaris.org/os/community/xen/docs/HVMdomains.htm
http://opensolaris.org/os/community/xen/docs/install-solaris-domu-iso.htm
http://blogs.sun.com/shalon/entry/a_summary_of_creation_of
--------------------------------------------------------ADDON-----------------------------------------------------------------------
How to choose VM ram and storage?
The machine I am using is
CPU
memory 4G
.....
When I am creating an solaris HVM guest, how much memory should it have and how much storage should be allocated for it?
Try 2G ram and 8G image
Problem 1: 2G memory is too much for a guest
it makes dom0 run slow.
Problem 2: 8G image is not enough for Solaris
when you installs Solaris express development edition, it automatically installs all the required packages and requires more than 8G, so I have installation failure. I guess it is because of running out of space.

Solution:
I choose 1G ram and 12G image for the guest, it works. You can see the working solaris guest as below:

Solaris xVM (xen on Solaris)

2008-05-20T21:28:00.013-04:00

Xen comes along with Solaris Community Express Edition(SCXE).

I am using Nevada version 87 (SNV b87).

In the grub, it has Solaris, Solaris xVm, Solaris Failsafe. If no, you can follow the instructions on the page: http://opensolaris.org/os/community/xen/docs/setupsoldom0/
Facility bootadm will help you to operate on menu.lst
# bootadm list-menu = cat /boot/grub/menu.lst
# bootadm -m upgrade = automatically search the boot directory and modify the menu.lst

It is using this one as below, so it is 64bit xen compatible
#---------- ADDED BY BOOTADM - DO NOT EDIT ----------
title Solaris xVM
kernel$ /boot/$ISADIR/xen.gz
module$ /platform/i86xpv/kernel/$ISADIR/unix /platform/i86xpv/kernel/$ISADIR/unix [*]
module$ /platform/i86pc/$ISADIR/boot_archive
#---------------------END BOOTADM--------------------

Note: in solaris 2008.5 released version, xen doesn't come along with it. But you can install the package using the package management application. However, bootadm doesn't work.

Once it is booted from Solaris xVM, you can use the command xm list to see dom0 is working as below:

Next is to create a Solaris domU. The instruction is on the page:http://opensolaris.org/os/community/xen/docs/virtinstall/

First, I am trying to create a Solaris guest in paravirtual mode. I found only the command below works:

machine:root> virt-install --nographics -n SolarisUU --paravirt -f /xvm/SolarisU.img -r 1024 --mac=aa:ff:bb:aa:28:16 -s 8 -l /export/home/newbie/sol-nv-b87.iso

--nographics

installation has no graphic mode, purely in console command mode

-n SolarisUU

the guest domain name will be SolarisUU

--paravirt

the guest will be created in paravirt mode
Note: on the webpage, the option paravirt is missing 'a'

-f /SolarisU.img

the guest will be in a file, the filename is SolarisU.img created in the root directory

-r 1024

the guest domain will use 1024M memory

--mac=aa:ff:bb:aa:28:16

the guest domain's network mac address is aa:ff:bb:aa:28:16

-s 8

the file size is 8G

-l /export/home/newbie/sol-nv-b87.iso

the guest domain will install from iso image

The rest is to follow the solaris installation process and it will automatically reboot and run the guest domain. Finally, you will see the guest domain is running as below. The guest ip is 128.153.144.100. So far, I didn't find anywhere to choose the guest domain network mode. The default is bridge mode. I didn't find where the virt-install has put the guest configure file.

Comments:

1) when creating paravirt guest, it will complain libvir: Xen Daemon error : GET operation failed:

2)Simple command # virt-install doesn't work.

3) The command for VNC doesn't work. It claims that authentication failed.

# svccfg -s xvm/xend setprop \
config/vncpasswd = astring: \"somepwd\"
# svcadm refresh xvm/xend; svcadm restart xvm/xend

4)virt-manager can be installed in xVM, but I can not have it to display the guest or create the guest. So I am doubt about it. Here are the instructions to install virt-manager in xVM

The page for virt-manager project is at http://opensolaris.org/os/project/jds/tasks/virt-manager/.
1)Download the binary package for SNV83 or later on the page http://opensolaris.org/os/project/jds/tasks/virt-manager/virt-manager-0.4.0-04-09.tar.gz.
2)Untar it and in the directory pkgs-04-09 and use the package installation command bash-3.2# pkgadd -d. SUNWvirt-manager to install.
Note: it has a dot right after -d and no space between them.
3) on the command line, type virt-manager, it will show up as below

TODO list:

1. Figure out where the configure file are stored by virt-install

2. Figure out how to run an existed guest

3. Make VNC works

4. Create HVM guest: solaris and windows

5. Create paravirt Fedora guest

----------------------------------------------------ADDON-----------------------------------------------------------------------------------
virt-install is a tool that works and can create both para and HVM domU images, but after OS installation and image creation, it doesn't provide a configuration file for the guest to run. So the user need to make up one for their own. Here is the one I am using and you can also refer to http://opensolaris.org/os/community/xen/docs/install-solaris-domu-iso.htm
and
http://blogs.sun.com/mrj/entry/installing_opensolaris_on
http://bderzhavets.blogspot.com/2008/03/install-solaris-0108-hvm-domu-64-bit-at.html\

a python configuration script pv.py
name = "solarisU" vcpus = 1 memory = "1024" disk = ['file:/solarisU.img,0,w'] vif = [''] on_shutdown = "destroy" on_reboot = "restart" on_crash = "destroy"

Note: in the disk option, you need to specify where your guest image is.

To run the guest once you have the image and the configuration file, you can use the command below to boot and connect to it.
#xm create -c pv.py
If you wants to get out of the domU, you can use the combination keys ctrl+] (same as linux xen) to exit the guest. But if you already have the guest running, but outside of it, you can use the command below to get into it by specifying its domain name or domain ID.
#xm console solarisU

Reinstall SPECweb

2008-03-07T02:12:00.007-05:00

java 1.5 installation

uninstall java gcj fcs because specwebclient cannot be complied and run

sun download page
http://java.sun.com/javase/downloads/index_jdk5.jsp
https://sdlc5c.sun.com/ECom/EComActionServlet/DownloadPage:~:com.sun.sunit.sdlc.content.DownloadPageInfo;jsessionid=FDCF352AF4C6A49B5CCEBCE7129FAF0E;jsessionid=FDCF352AF4C6A49B5CCEBCE7129FAF0E
instruction
http://java.sun.com/j2se/1.4.2/install-linux.html

change execuatable
chmod +x jre-1_5_0_15-linux-i586-rpm.bin
./jre-1_5_0_15-linux-i586-rpm.bin

install sun java
rpm -iv jre-1_5_0_15-linux-i586.rpm
http://liquidat.wordpress.com/2007/05/31/howto-sun-java-on-fedora-7/

install SUN java

#sh ./jre-version-linux-i586-rpm.bin

#/usr/sbin/alternatives --install /usr/bin/java java /usr/java/jreversion/bin/java priority
demo:   #alternatives --install /usr/bin/java java /usr/java/jre1.5.0_15/bin/java 2

#/usr/sbin/alternatives --config java

mod_fastcgi module installation:

downloald mod-fastcgi package from
http://www.fastcgi.com/dist/

mod_fastcgi-2.4.6.tar.gz

check the instruction INSTALL using the mothod of DSO
then install httpd-devel to have apsx working

http://jimthompson.org/wp/2005/05/29/building-modfastcgi-for-apache2/
use the command:

apxs -n mod_fastcgi -i -a -c mod_fastcgi.c fcgi_buf.c fcgi_config.c fcgi_pm.c fcgi_protocol.c fcgi_util.c

Load module in httpd.conf

Selinux prevent httpd to bind 81
http://danwalsh.livejournal.com/9275.html
use the following commands:

semanage port -a -t http_port_t -p tcp 81

SELinux prevented httpd reading and writing access to http files.
use the following commands:
setsebool -P httpd_unified=1

config SPECweb
Test.config
BESIM_INIT_SCRIPT = "/fcgi-bin/besim_fcgi.fcgi"
rather than /var/www/html/fcgi-bin/besim_fcgi.fcgi

The error information is
2008-03-07 05:32:55:776 RemoteLoadGen: Adding host 128.153.144.162
2008-03-07 05:32:55:936 Connection: [ERROR] Bad status: 404
2008-03-07 05:32:55:936 Connection: bad status ======= buf_len=494 headerEnd=179 contentLen=315 chunkEnd-1========
2008-03-07 05:32:55:936 Connection: HTTP/1.1 404 Not Found
Date: Fri, 07 Mar 2008 10:32:55 GMT
Server: Apache/2.2.6 (Fedora)
Content-Length: 315
Connection: close
Content-Type: text/html; charset=iso-8859-1

404 Not Found

Not Found

The requested URL /var/www/html/fcgi-bin/besim_fcgi.fcgi was not found on this server.

Apache/2.2.6 (Fedora) Server at 128.153.144.162 Port 80

2008-03-07 05:32:55:936 HttpRequestSched: [ERROR] Bad response (-1). Request was GET /var/www/html/fcgi-bin/besim_fcgi.fcgi?3&0&1204886909&5 HTTP/1.1
Accept: */*
Host: 128.153.144.162

IF you GOT 500 INTERNAL ERROR in specweb and in the apache it indicates error:
PHP Warning: fopen(http://128.153.144.162:81/fcgi-bin/besim_fcgi.fcgi?3&6&2821) function.fopen]: failed to open stream: No route to host in /var/www/support/inc.common.php on line 44
Then it probably means your specweb machine's 80 is blocked and only 81 works.

IN SPECWEB, you need both 80 and 81 working on the same machine. SO if firewall and selinux are disabled, that will save you a lot of time.

OS: syscall

2008-01-25T14:39:00.000-05:00

1. show the linux architecture for system call;
2. show the steps of modification, each meaning;
3. how to return a value from system call, how to ask a system call to do something for us;
4. how to compile a kernel, load a kernel, use a system call;
5. put up the lab far before the lab, so students can start or have some idea of it before hand;
5. before the lab, run it again to make sure it works.

OS: Shell Tutorial

2008-01-18T14:08:00.000-05:00

1 introduce the special points of C language
printf: %d %f %s %c
symbol: \n \t
cstring: string functions reference/link
single quote and double quote
command line parser: int argc char *argv[]

2. The example fits in the shell program
Set up the whole program architecture for the student
Help to understand the parser.c code

3. Good reference on C Language, System call
C Language tutorial online
Linux system call introduction online
request the kernel to do something for you. send back to user
POSIX standard

4. Shell parser
makefile
compile the program
gcc -ggdb

4. Unix concept
what are pipes? How to use them one output piped into a command's input
ps aux grep jobs grep ps
top &
jobs
kill %2
< into standard input program
> into a file

Doubts on Traditional FS

2007-11-08T20:16:00.000-05:00

metadata is inode and inodetable[ ].
Q1. How to postion an address in a file?
They are linked by direct/indirect pointers. If small writes or inserting some contents, where to put them?

Q2. There are fragements, how bad will it hurt the fs performance?
There are cleaner algorithm. How often do they need to be done? In which way? daemon or command?

Q3. Where is root directory inode located?
At block 2 or some default block or specified somewhere?

LFS thoughts

2007-11-01T23:29:00.001-04:00

How many files can be supported?
How large a file size can be supported?

Ext2 file analysis
http://homepage.smc.edu/morgan_david/cs40/analyze-ext2.htm

FS summary
http://www.science.unitn.it/~fiorella/guidelinux/tlk/node99.html

xen that I was blocked on

2007-08-07T21:19:00.000-04:00

HVM options
nic deprecated
vif rate options

checking help_config

Tests running in xen express

2007-08-07T15:53:00.000-04:00

baseline with 0.3% degradation
fork with misbehaved one 59.93%good 60.29%tolerant. Specweb survives and ctrl+c can bring the system back.
out of memory: killed process 11740(apache2)
Next is memory, the same as fork, the specweb survives and ctrl+c can bring the system back.
malloc.sh line7 killed ./malloc.sh -i
out of memory: killed process 3090(mallocBomb) score 15538 and children.
Then is DISK IO, the guest sending the packets, and then the guest receiving the packets.

Preparation for Defense
get Completion sign sheet for commitee, Course Form.
also need to book the room and send the abstract to get an announcement.
But the most important thing is to have the draft done.

gcc in debian

2007-08-07T12:42:00.000-04:00

debian sarge with xfce in Xen express doesn't have gcc yet. To compile the bomb,
apt-get install gcc to get the gcc
apt-get install make to get the make

xen express problem

2007-08-06T21:39:00.000-04:00

1 xen express cd
go to /client-install
rpm -ivh *.rpm
2 xenserver-client
input master-key
host IP and password

Xen Express only supports RedHat 4.1 and 4.4, RedHat5 needs VT supports, debian sarge, Windows XP sp2, Windows server 2003. No fedora or CentOS.

It can install from network(repository) or install from physical machine(p2v), but all limited to the few OSes.

Debian sarge is installed, apache install instruction:
1. apt-get update
to update the package sources
2. apt-get upgrade update
upgrade debian
3. apt-get install apache2
apache -v display the version
install webserver apache2.0.54,
The web directory is /var/www
apachectl restart restarting the webserver, or /etc/init.d/apache2 restart
3 apt-get install libapache2-mod-php4 php4-cli php4-common php4-cgi
The php version is 4.3.10

Set a static IP in debian
edit /etc/network/interfaces
auto eth0 lo
#iface eth0 inet dhcp
iface eth0 inet static
address xxx.xxx.xxx.xxx
netmask 255.255.255.0
network xxx.xxx.xxx.0
broadcast xxx.xxx.xxx.255
gateway xxx.xxx.xxx.1

update the interface
/etc/init.d/networking restart

cannot afford

2007-08-01T02:45:00.000-04:00

It sounds like that ESX's network performance is not good at all.
The more weird thing is that when running network sending from one guest, the specweb clients are all broken. And the first time that I can catch a chance to evaluate the transmiting rate which is around 68Mbps.

first time, all four down because kernel network panic.
Second time, two broken, complaining exceeded max allowed overthink time of 20 sec. Please make sure that neither the server or client are overloaded. If server is overloaded, consider the nunber of SIMULTANEOUS_SESSIONS requested. If clients appear overloaded, add more clients. The rest two: one 6.71%good, 16.78%tolerant, Total errors: 66; the other 0%good, 0%tolerant, Total error: 207.

I ping google, it has sent 14 packets, 6 received, the packet lost 57% , the response time is 19.0ms.

In ESX, the support sessions are limited to 20, when with 5 sessions, it still have some degradation here. But when fewer guests running there, the degradation will be relieved in some extent.

The side effect is that when the network goes wrong, the client will also be broken. So it is hard to do anything to solve the problem.

The kernel also reports a network panic there. Lost Internet connections

All these may due to:
1. ESX performance overhead is too high
2. Bladecenter network card or infrastructure isn't well designed
3. Network driver is not well written
4. The other blade network usage has affected the testbed.

-------------------------------------------------------------------
Summary
guest sending packets 4 broken, none suvives
guest receving packets the misbehaved broken at the 3rd iteration, the degradation is 60%, the good ones survive with 40% degradations.
The receiving rate is 10Mbps
ping google while guest receiving packets, 7packets , 5 received, 28% packet loss, the response time is 20 ms /40ms

Test going

2007-07-30T16:49:00.000-04:00

The day before yesterday, Baseline with 5 sessions, 96% good each
Yesterday, CPU. Today, Fork.
For the fork one, the first VM is broken
Have Taking snapshot with CPU usage.

Needs to backup all the data, name the test results.

VMware folk has replied with our ESX configration:
Memory 512 each is OK, but CPU maybe overcommite. Fortunately, we have two dual core cpus there, but no idea if the license will support two processors.

Virtual Machine

2007-07-28T09:52:00.000-04:00

Today, coma across a special programming language: Erlang which is designed for OS design. It is open source and developed by Ericssion :). Sometime, I need to study its syntax and its compiler grammar to see how it works :p. More detail please see:
http://www.erlang.org/white_paper.html

Virtual Machines's roles in Computer Science:
versatile platforms of systems and processes.
Hardware: hard, inflexible.
VM wraps a layer of software around this hardware, and suddently computers become flexibile, malleable, and start doing new tricks:
running multiple operating systems, //compatiable different implementation and approaches
executing serveral instruction sets, //compatiable different sets of instructions
allowing running programs to switch machines, // multiple machines serving at the same time
or even rejecting unsafe code, //checking layer, provide more security at a lower layer.

Virtual machines are changing the way in which computer architects, operating system designers, prgramming language implementers, and security experts think about computers and computing.

A computer to imitate another computer, imitate multiple comupters

Time taken more than you expect.

2007-07-26T17:15:00.000-04:00

Like software development, writing is also taking more time than you planed or thought it should be. It always takes double time than you think they would. And there are also maybe new ideas, modifications and patches coming up beyond your expectation.

So start as early as possible, get it done as early as possible at one strike first. Then leave the rest time to improve it and release brandy new versions.

Now the Thesis only gets a skeleton there. It needs Flesh and Blood. Todd has reminded me that i should get the draft done today, every minute, every hour working on it. But I am still thinking about the expression.

Now: each guest is running 20 sessions there, got 20 degradation on the good, and 5% degradation on the tolerance. How to optimize the webserver is a big issue there. The second iteration is still exceeds 1.5% for Request Distribution Errors.

Performance Isolation

2007-07-25T21:23:00.001-04:00

Virtualization is trying to use software to implement a machine function.
Performance isolation is trying to evaluate how much close this software machine function as the hardware machine.

Urgent Call: Thesis draft is TOP priority NOW!!!!

Current status

2007-07-25T03:16:00.000-04:00

Thesis not started yet at all.

Finish the security books, get the idea that staying open is better than securing it. Because no matter what kind of security means, it will always be conquerable. But being aware and tracable will be a human's game.

Got stuck in ESX, the possible valid run is 20 sessions for one guest. But the degradation is really low, around 95%. Disabled all the services in FC, but still nothing changed.

Need working hard now.

Innovation: Battery and Wireless

2007-07-20T13:46:00.001-04:00

What's the annoyance for wireless electronic device? Battery!
Wireless keyboard or blutetooth mouse all needs to have battery althought no bothering by the wire any more. But battery may go die quickly. To bring with an extra battery is kinda annoy and inconvenience.
Think about currently it is so popular for mp3 player and bluetooth earphone to use USB for charging and data transfer.
So why can not wireless keyboard or mouse? Since usb keyboard and mouse is also popular. So use USB to charge and also send the key signals to Computer is one-stone-two-birds success. But I prefer bluetooth keyboard/mouse since wireless keyboard may introduce another connector at the box side.

Done, Just time

2007-07-13T23:57:00.000-04:00

The first SPECweb test has been done on one ESX virtual machine. Only QoS does not meet the requirement with 20 sessions but on Solaris Container within TC it is qualified. So it comes with the next problem: how to optimize Apache and the webserver to provide the best performance.

What have been learned from Research on Performance Isolation Propterties on Virtualization System
This is an initial approach towards the study of performance isolation properties.
We've touched base with many current popular virtualization systems---how to set up and how to configure.
We obtained the popular industry web server benchmark---SPECweb

More than that:
Proficient in setting up Linux, actually fedora. The Linux installation, the network setup, the webserver setup.
Familiar with Apache operations and configurations, especially apache configuration. The module combined, the access controlled, the syntax pattern.
Manual fast_cgi. How to patch and how to add it as a new module

A lot to dig in:
Apache access control, multiple web directory
SElinux configuration and control
Fast_cgi working mechanism, how to execute script in web server
Optimize web serivce and minimize Linux Kernel

stupid mistakes

2007-07-13T21:31:00.000-04:00

1. No need change to root ownership
2. No need change ScriptAlias cgi-bin to Alias cgi-bin
3. No need change php.conf

Just need to change /var/www/html to /var/www in two places of httpd.conf, then put support and Smarty in /var/www, then it is done.

SPECweb that needs to change:
Test.config
1. Connections
SIMULANEOUS SESSION =100
2. Type
TEST_TYPE=SPECweb_Support
3. Webserver IP
WEB_SERVER = xxx.xxx.xxx.135
4. Smarty Directory
SMARTY_DIR="/var/www/Smarty-2.6.7/libs"
SMARTY_SUPPORT_DIR="/var/www/support/"

SPECweb_Support.config
1. padding directory
PADDING_DIR="/var/www/support/dynamic_padding/"
Comments: if missing to change it, then
in /etc/httpd/logs/error_logs:
File does not exist: /var/www/favicon
attemp to invoke directory as script: /var/www/cgi-bin
Directory index forbidden by Options directive: /var/www/error
on the SPECweb:
[Error] State 0: Expected 51700 bytes appened to response but found 0
[Error] State 0: RESPONSE INVALID on GET /support/index.php HTTP/1.1
if the simultaneous session value is less than the dataset can support
it will display:
HTTPRequestSched: [ERROR] Bad response (-1). Request was GET /support/downloads/dir0000000019/download5_0 HTTP/1.1
SPECweb_Support: [ERROR] STATE 6; makeHttpRequest() failed.
Connection: [ERROR] Bad status: 404
Connection: bad status ====buf_len=500 headerEnd=179 contentLen=321 chunkEnd-1===
Connection: HTTP/1.1 404 Not Found
if /var/www's access priviledges were not changed, the SPECweb error will be
Error: Could no twrite to init_vars.php
HttpRequestSched: [ERROR] Repair reset code for conforming run of set COND_GET_REQ_PERCENT to 0 for non-conforming run. Run halted.
SPECweb_Support: [ERROR] Error! setServerDate() failed.
RemoteLoadGen: [ERROR] Unable to successfully initialize workload variables. Termininating

Finally got it work, ESX to go

2007-07-13T17:02:00.000-04:00

Apache2.2.3 + Php5.1.6 + cgi problem
Symptom:
As corrected before, directory listing works. But it doesn't work for cgi-bin.
SPECweb complains:
attemp to invoke directory as script: /var/www/cgi-bin/
On the browser, it complains:
403 Forbidden
You don't have permission to access /cgi-bin/ on this server
But actually it has the full permission 777 and it has the apache:apache ownership
(it sounds like specweb cannot write script in cgi-bin direcotry)
To make php work properly in Apache

Solution:
1. in php.conf
add
AddType application/x-httpd-php .php
comment out
AddType application/x-httpd-php-source .phps

2.To make cgi work properly in Apache
change
ScriptAlias /cgi-bin/ "/var/www/cgi-bin"
Directory "/var/www/cgi-bin"
AllowOverride None
Options None
Order allow, deny
Allow from all
/Directory
to
Alias /cgi-bin/ "/var/www/cgi-bin"
Directory "/var/www/cgi-bin"
AllowOverride None
Options ExecCGI Indexes
Order allow, deny //no space between allow and deny
AddHandler cgi-script cgi pl // if in the cgi-bin/ is perl scripts , if python, then add py
Directory

3. SElinux conflicts with apache
Sometimes SElinux also brings trouble without ideas. So disable SElinux and reboot is more safe and saves your trouble

Also two helpful commands:
1)ls -aZ
to display the SElinux security configuration
--- system_u:object_r:httpd_sys_script_exec_t:s0
--- system_u:object_r:httpd_sys_content_t:s0
use chcron to change it

2) /sbin/service httpd configtest
Syntax OK
to validate the http.conf syntax error

Now testing with 20 sessions in 131.......
---------------------------------
Here is how to make fastcgi working in Apache2.2.3 (When I am free, I will translate it in English)
http://hi.baidu.com/coffeefoam/blog/item/1446493be749f3e814cecbb8.html

On the road, ESX

2007-07-11T09:47:00.001-04:00

1. Create dataset
support_downloads_props.rc is segmented in two parts: section A and section B (B is for specweb running, recommended not to change)
in A
SIMULTANEOUS_SESSIONS=40
base directory number = basenumber + sessions* discaling = 0 + 40 *.25 = 10
the rest are for configuring the directory not to exceed the system limitation

support_image_props.rc only has section A:
SIMULTANEOUS_SESSIONS=40

#./Wafgen support_downloads_props.rc
#./Wafgen support_image_props.rc
DOCROOT is in /var/www, go to /var/www to pick up the dataset and add the scripts in it.

setup->network configuration
XXX.XXX.XXX.13X
255.255.255.0
XXX.XXX.XXX.1

#/etc/init.d/network restart
#service httpd restart

documentroot "/var/www"

------------------------------------------------------
Apache lame
1. 404 Not found
You cannot access / on this server
But display index.php as normal

forgot to change to /var/www

2. 403 Forbidden
You do not have permission to access / on this server
Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.
In /etc/httpd/logs/error.log, Directory index forbidden by options directive /var/www
/var/www/Error has been moved out of the directory, so it should display the directory list, but nope.

In apache2.2.3 besides httpd.conf configuration file for apache, it also has configuration files in conf.d/ .
1) With error/ directory moved out, directory listing doesn't work.
Go to php.conf,
#Add index.php to the list of files that will be served as directory indexes
DirectoryIndex index.php
this should be commented out
2) Browser the subdirectory of localhost/subdir , the directory can be displayed but localhost/ still doesn't work
It keeps saying "Directory index forbidden by options directive /var/www", checked allowoverride, options +Indexes, problems can never be solved, but move error/ back to /var/www, it works. Page can be displayed.
where does that page come from?
check conf.d/ again, welcome.conf defines the welcome page.
In it, saying "Options -Indexes". Catch it, which has override the /var/www options!

Comments:
Apache access rules definition is a good one. It has the options defining the access patter, it has allowoverride defining the access priority, it has Allow defining the access clients. Maybe it can be applied to security system if the access follows if owning theh access to one directory, then also heritages the subdirectory.
It had worked fine in apache2.2. But apache2.3 gave me pains. In apache2.3, it has scattered many sub configurations around but no hints or instructions for the change. So each new version of Linux, it is a new system which you have to be familiar with again. Also sticking to the old ways may be a good solution, also when using the new style, knowing its working mechanism and its style first before get into using it.

Ready to Go, ESX

2007-07-11T00:03:00.000-04:00

Done:
1. uninstall server configuration tool
2. setup-> disable yum update and enable httpd
yum update must be disabled, if updating, gimp and gaim patches will break the update.
3. change /etc/httpd/conf/httpd.conf documentroot=/var/www
4. create index.php to display php information in /var/www
php already installed, also apache 2.2.3
Todo:
1. Change IP to static ip
2. Use wafgen to create support dataset with 40 sessions
sessions > the session value in Test.config
3. put the dataset in /var/www of each image
4. change the ownership of directory /var/www and its sub dirs
chown -R apache:apache /var/www
5. chagne the rights of files /var/www and its sub files
chmod -R 777 /var/www

Undecided:
1. swap impact, current no swap
2. how much memory is approprieate, current 512, used to be 128 and almost stuck in it
3. maybe compiling issues

All four images are running with memory 512M. php info displayed in nycrr host.

Cursed server configuration tool

2007-07-10T21:57:00.000-04:00

http://www.redhat.com/docs/manuals/enterprise/RHEL-5-manual/Deployment_Guide-en-US/s1-apache-config-ui.html
Caution
Do not edit the /etc/httpd/conf/httpd.conf configuration file by hand if you wish to use this tool. The HTTP Configuration Tool generates this file after you save your changes and exit the program. If you want to add additional modules or configuration options that are not available in HTTP Configuration Tool, you cannot use this tool.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
When changing Documentroot or Listen port, all goes wrong. It said
404 Not Found
The reuested URL / was not found on this server

Then when using the server configuration tool GUI stuff
change the virtual host web root there, it messed up http.conf, inserting a lot of empty lines at the beginning and at the end adding scattered grammars there without comments.

Solution:
uninstall server configuration tool.

Comments:
Linux configuration is based on configuration file---text parsing. Windows configuration is based on register table---table. Now Linux is trying to emulate windows GUI configuration tool but their underlying is still text, not format controllable. And configuration GUI actually is another interpretation of configuration grammar rule. Not necessary at all. Linux strength is at its developing power. It is compiler style.
Either totally convert to XML or an uniform format to all applications or just stay where it originally is.

Sample of simple index file

index.html
html Hello World! /html

index. php
?php phpinfo(); ?

Good Reference: http://www.google.com/codesearch

Reasons for LFS

2007-07-10T17:28:00.000-04:00

Disk volumn is not the criticle point any more. 500G only costs $100.

The harddriver has developed from ATA to SATA, whose speed is now competitive to the server level data access SCSI.

Now problems come as how to well manage large data, how to well index the data and fast search the data, how to support large number of directories and large size of files. All circle around file management.

LFS is targeting at roll back, backup and recovery, involving maintaining multiple copies of edited version. So how to most efficiently save the space and quickly access the right spot is a good topic.

Also IBM proposed a paper on maintaining multiple versions of updated applications, trying to solve version dependency which is also a good hint to be adopted into LFS on the application's layout. Besides that, CVS may be a good example of maintaining multiple version. Its implementation details are good to know.

Apache2.2.3 stuck

2007-07-10T15:26:00.001-04:00

Changing server port number leads to startup Failed. only 80 works
Changing documentroot leads to 404 Not Found
The requested URL / was not found on this server
file:///var/www/index.html works

#httpd -t //run configuration syntax test
Syntax OK
#service httpd start restart stop
#htmlview index.html

Kernel, OS

2007-07-10T13:43:00.000-04:00

In Data Network equipment market, CISCO is the giant, who is the first one getting into the market and who owns the biggest share in this market. Huawei is catching him up. But the latest prediction in Wallstreet said that CISCO is still unbeatable in OS and router.
So come back to the old topic, OS is the Core value of Information Era.
It is the most complicating project, similiar to compiler.
It is the most basic layer for all applications, no matter desktop, server, or router.
It is the most key element in performance providing.

OS involves Scheduler, File system, and communication. Scheduler determines computing, FS determines storage, and communicaiton determines exchanging. It almost covers all the aspects of information technologies. It has diversial patterns due to how much computing power you have and how much data you will handle. So it is a really amazing kaleidoscope.

Think about Google: why is so great? Because it is neat philosophy: simpilicity, friendly, convenience. Google is always designed in the way that tries to get rid of annoyance as much as possible, make your operations shortcut and simple straight.
But on the other hand, that's actually not all. It is excellent more based on its quick response. But how can it make so quick response instead of waiting and waiting in dead cycle? It is great power of computing. Where does their great computing power come from? personal desktops. How do they make personal desktops provide so great power? The OS with efficient algorithm manages the distributed PCs.

So for google, its core value is still OS and their GFS that are so delicated to be efficient.

Image Package

2007-07-10T11:09:00.000-04:00

The specweb turns out to be a valid run with 20 sessions running and 2 10-session specwebs running in the other two container. But there is some degradation of good performance
Good Tolerant Fail
99.7% 100% 0%
99.6% 99.9% 0.1%
99.7% 100% 0%

The second image is done.
Image Package:
no Games and Entertainment
no Graphics
no Office/Productivity
no GNOME Software Development
but Server Configuration Tools
no Dial-up Networking Support
But System Tools

Firewall:
ssh, http
SElinux: enforcing

The third and fourth are also done. A test image is on-going as a sample to make SPECweb working in it.

SPECweb reply

2007-07-09T22:19:00.000-04:00

The invalid run in our system is due to too low of a request rate instead of overloading the server.
Official proposed solution: increasing the simultaneous sessions until QoS or Vadlidation Error. Then try to solve the QoS or Validation error instead of Request Distribution Error.

Create the dataset of 20 simultaneous sessions with wafgen, and put it in 119 solaris container.
After 119 has finished 1 iteration with 20 sessions, start another 2 with 10 sessions
It reports
Sum of weighted percentage difference (1.82%) exceeds 1.5% for Iteration 1
So it is almost there.
----------------------------------------------------
One thing to mention: for the valid submission, it should have support, commerce and bank three results ready. And also raw files should be backup as reference.

The first ESX image is going with temorary memory 512M without swap. ESX_1 baby is born, but php not ready yet, static ip is not ready yet. DHCP ip is a temperary solution. httpd is OK.
The second one is sharing the same CDROM with the first one. Disconnect the first one from CDROM.
CDROM is ejected, so the second one cannot find operating system to boot up
Now the second is on the way... ...

ESX License issue

2007-07-09T14:17:00.000-04:00

insufficient license:
Host-Based License: the license files are stored on individual ESSX server hosts
Server-Based License: licenses are stored on a license server, available to multiple hosts.
Host-Based License is enough, per-machine basis, placed on the individual ESX Server hosts

issues:
Go to Inventory -> localhost.localdomain ->Configuration -> License Features
License resource (Edit on the right)
Edit -> Use Host License File -> Upload local file
change VI_SingleHost.Lic to vmware.lic
Host License File: Installed
ESX Server License (Edit on the right)
Edit -> radio button ESX Server Standard (ESX Server Starter grey)
ESX Server Standard Licensed for 1 CPU
SAN Usage
Up to 4-way virtual SMP
NAS Usage
ONLY 1 CPU License, how can it be 2 CPU Licensed? Will it be any impact there?

Image Creation
Partition:
no swap (should study the impact on no swap, do you need to?)
IP range:
128.153.145.131~134
MAC:
00 0C 29 AA F1 07
Next step: populate the rest three images, ESX_2, 3, 4 and install apache and php in it

Invalid Run

2007-07-09T11:49:00.000-04:00

Run only one client with 5 sessions in Solaris:
Aggregate QoS Compliance are all 100% good, no QoS Errors, no Validation Errors.
But request distribution errors are high:

Weighted percentage difference (-1.50%) for search in Iteration 1 is too high. Expected: 109 requests, Actual: 122
Weighted percentage difference (-2.46%) for search in Iteration 2 is too high. Expected: 106 requests, Actual: 127
Weighted percentage difference (-3.47%) for product in Iteration 2 is too high. Expected: 208 requests, Actual: 238
Weighted percentage difference (-2.28%) for fileCatalog in Iteration 2 is too high. Expected: 189 requests, Actual: 209
Sum of weighted percentage difference (-9.18%) exceeds 1.5% for Iteration 2
Weighted percentage difference (-1.39%) for search in Iteration 3 is too high. Expected: 109 requests, Actual: 122

Running specwebclient with option -lo to record the reports and errors. An email has been sent to specweb attached with the standout. Waiting for reply.
---------------------------------------------------------------------------------------
VMware License
Failed twice last night. Using lib's gmail got one. It has been forwarded to my gmail. Time to register and create four images, unsure of what new problems may come up. Maybe only support one host. Time to prove.

Image installation

2007-07-08T19:34:00.000-04:00

Sunday, from 7PM to 12PM, there are 5 hours to go
Launch VMware ESX
IBM eServer BladeCenter LS20
2CPU X 1.993 GHz
Dual Core AMD Opteron(tm) Processor 270
Memory: 441M used/ 4G total
Storage: 60.14G free/60.75G capacity
Active Adapters: vmnic0 1000full 128.153.144.192~195
SCSI adapter: 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI

Add License:
Inventory --> Configuration tab> License Source Edit

waiting for new license
----------------------------------------------------------------------------------------------------
SPECweb working review:
put Directory "support" and "Smarty" into the directory var/www
change the Documentroot option in /etc/httpd/conf/httpd.conf
previliges:
chmod -R 777 www/
ownership:
chown -R apache:apache www/

Resutls backup:
111, 114, 115, 116 are four clients to test OpenVZ
116 is the last client where the bombs run in it.
the results should be tarballed so that the date can be kept as original, or it will be updated as the date when copying.

ESX on blade14

2007-07-07T23:00:00.002-04:00

It is a temporary solution due to scsi no working yet on TC.
Following VMware folks proposal, tests will run on VMware ESX. As a comparision, also Xen Express. They are all enterprise level virtualization system.
Blade 14:
used to be Ubuntu bladexen and erased.
AMD Opton Dual Core 2.0G
Memeory 4096MB

ESX Server 3.0.1
Version Date:
5/31/07
Harddisk:
SCSI DISK sda IBM-ESXS ST973401LC FN 70002MB
NIC:
X 2:1:0 tg3 NetXtreme BCM5704S Gigabyte
_ 3:1:0
_ 2:1:1
_ 3:1:1
DHCP:
128.153.144.119
Default set up virtual network

Todo tomorrow:
Create Virtual Image
License
SPECweb running

SPECweb news:
Tune the simultaneous session value from 1 to 10, but never get valid run
Use the specwebclient option -lo
#java specwebclient -lo
output will write to "specwebclient1099.out", no direct clue yet. An email inquiries on "invalid run" should be sent to the specweb support.

The inconsistency is that the dataset produced by wafgen is 10 sessions, but the specweb is using 5 sessions.
When having tuned to 10 sessions for specweb to run, there were 40 sessions running on the TC, there were 1% degradation for good, 1% for tolerance.

Thoughts from MAC

2007-07-06T20:18:00.000-04:00

Put each application in a conexecutive and complete block of space, each update or change is put in the log. When cleaning, it is time to integrate them all. In that way, the file modified or changed, may lay in different place which may introduce overhead, it also keep the application versions.

Pros: version control, integration, independent packages.

LOG Opening

2007-07-06T18:16:00.000-04:00

This blog is opened due to graduate study in USA. Science is a serious and interesting wanderland.
This blog is mainly logging the ideas coming up and the steps taking for the experiements.
Here it will record the failure and success, the frustration and cheers, the basic and classic. But on the road to the top of Computing and Internet, we need a calm and persistent heart to hear the nature of CS, a delligent and delicated hand to practice every possibility of IT.
To keep the mind of deeping into the research, I would like to quote the words from Noble Prize winner Luise Alvarez
"Why don't you look at this this as research. When you're doing real research, you never know what it'll cost, how much time it'll take, or what you'll find. You just know theres unexplored territory and a chnace to discover what's out there. ... ...You're following a fascinating scent. You're an explorer. Think of 'what' might be behind it. ... ...Well then, forget who's causing the problems, Don't try to be a cop, be a scientist; Research the connections, the techniques, the holes. Apply physical principles. Find new methods to solve problem. Compile statistics, publish your results, and only trust what you can prove. But don't exclude improbable solutions---keep your mind open. .. ... Dead ends are illusory. When did you ever let a 'Do Not Enter' sign keep you away from anything? Go around the brick walls. When you can;t go around, climb over or dig under. Just don't give up. ... ...Nobody will pay for research; hey're only interested in results... ... Don't wait for someone else, do it yourself. Keep your boss happy, but don't let him tie you down. "
So,
It is the logbook of how I pursue for the degree
It is the diary of what I am doing everyday in academic
It is the report of weekday research work
It is the try of learning something new
It is the note of repeating something old
It is the blueprint of the random thoughts, ideas, and talks
It is the bookmark of the proposals, presentations, paper, and thesis.

It is a LOGBOOK in Internet.