Test going

The day before yesterday, Baseline with 5 sessions, 96% good each
Yesterday, CPU. Today, Fork.
For the fork one, the first VM is broken
Have Taking snapshot with CPU usage.

Needs to backup all the data, name the test results.

VMware folk has replied with our ESX configration:
Memory 512 each is OK, but CPU maybe overcommite. Fortunately, we have two dual core cpus there, but no idea if the license will support two processors.

Virtual Machine

Today, coma across a special programming language: Erlang which is designed for OS design. It is open source and developed by Ericssion :). Sometime, I need to study its syntax and its compiler grammar to see how it works :p. More detail please see:
http://www.erlang.org/white_paper.html

Virtual Machines's roles in Computer Science:
versatile platforms of systems and processes.
Hardware: hard, inflexible.
VM wraps a layer of software around this hardware, and suddently computers become flexibile, malleable, and start doing new tricks:
running multiple operating systems, //compatiable different implementation and approaches
executing serveral instruction sets, //compatiable different sets of instructions
allowing running programs to switch machines, // multiple machines serving at the same time
or even rejecting unsafe code, //checking layer, provide more security at a lower layer.

Virtual machines are changing the way in which computer architects, operating system designers, prgramming language implementers, and security experts think about computers and computing.

A computer to imitate another computer, imitate multiple comupters

Time taken more than you expect.

Like software development, writing is also taking more time than you planed or thought it should be. It always takes double time than you think they would. And there are also maybe new ideas, modifications and patches coming up beyond your expectation.

So start as early as possible, get it done as early as possible at one strike first. Then leave the rest time to improve it and release brandy new versions.

Now the Thesis only gets a skeleton there. It needs Flesh and Blood. Todd has reminded me that i should get the draft done today, every minute, every hour working on it. But I am still thinking about the expression.

Now: each guest is running 20 sessions there, got 20 degradation on the good, and 5% degradation on the tolerance. How to optimize the webserver is a big issue there. The second iteration is still exceeds 1.5% for Request Distribution Errors.

Performance Isolation

Virtualization is trying to use software to implement a machine function.
Performance isolation is trying to evaluate how much close this software machine function as the hardware machine.

Urgent Call: Thesis draft is TOP priority NOW!!!!

Current status

Thesis not started yet at all.

Finish the security books, get the idea that staying open is better than securing it. Because no matter what kind of security means, it will always be conquerable. But being aware and tracable will be a human's game.

Got stuck in ESX, the possible valid run is 20 sessions for one guest. But the degradation is really low, around 95%. Disabled all the services in FC, but still nothing changed.

Need working hard now.

Innovation: Battery and Wireless

What's the annoyance for wireless electronic device? Battery!
Wireless keyboard or blutetooth mouse all needs to have battery althought no bothering by the wire any more. But battery may go die quickly. To bring with an extra battery is kinda annoy and inconvenience.
Think about currently it is so popular for mp3 player and bluetooth earphone to use USB for charging and data transfer.
So why can not wireless keyboard or mouse? Since usb keyboard and mouse is also popular. So use USB to charge and also send the key signals to Computer is one-stone-two-birds success. But I prefer bluetooth keyboard/mouse since wireless keyboard may introduce another connector at the box side.

Done, Just time

The first SPECweb test has been done on one ESX virtual machine. Only QoS does not meet the requirement with 20 sessions but on Solaris Container within TC it is qualified. So it comes with the next problem: how to optimize Apache and the webserver to provide the best performance.

What have been learned from Research on Performance Isolation Propterties on Virtualization System
This is an initial approach towards the study of performance isolation properties.
We've touched base with many current popular virtualization systems---how to set up and how to configure.
We obtained the popular industry web server benchmark---SPECweb

More than that:
Proficient in setting up Linux, actually fedora. The Linux installation, the network setup, the webserver setup.
Familiar with Apache operations and configurations, especially apache configuration. The module combined, the access controlled, the syntax pattern.
Manual fast_cgi. How to patch and how to add it as a new module

A lot to dig in:
Apache access control, multiple web directory
SElinux configuration and control
Fast_cgi working mechanism, how to execute script in web server
Optimize web serivce and minimize Linux Kernel

stupid mistakes

1. No need change to root ownership
2. No need change ScriptAlias cgi-bin to Alias cgi-bin
3. No need change php.conf

Just need to change /var/www/html to /var/www in two places of httpd.conf, then put support and Smarty in /var/www, then it is done.

SPECweb that needs to change:
Test.config
1. Connections
SIMULANEOUS SESSION =100
2. Type
TEST_TYPE=SPECweb_Support
3. Webserver IP
WEB_SERVER = xxx.xxx.xxx.135
4. Smarty Directory
SMARTY_DIR="/var/www/Smarty-2.6.7/libs"
SMARTY_SUPPORT_DIR="/var/www/support/"

SPECweb_Support.config
1. padding directory
PADDING_DIR="/var/www/support/dynamic_padding/"
Comments: if missing to change it, then
in /etc/httpd/logs/error_logs:
File does not exist: /var/www/favicon
attemp to invoke directory as script: /var/www/cgi-bin
Directory index forbidden by Options directive: /var/www/error
on the SPECweb:
[Error] State 0: Expected 51700 bytes appened to response but found 0
[Error] State 0: RESPONSE INVALID on GET /support/index.php HTTP/1.1
if the simultaneous session value is less than the dataset can support
it will display:
HTTPRequestSched: [ERROR] Bad response (-1). Request was GET /support/downloads/dir0000000019/download5_0 HTTP/1.1
SPECweb_Support: [ERROR] STATE 6; makeHttpRequest() failed.
Connection: [ERROR] Bad status: 404
Connection: bad status ====buf_len=500 headerEnd=179 contentLen=321 chunkEnd-1===
Connection: HTTP/1.1 404 Not Found
if /var/www's access priviledges were not changed, the SPECweb error will be
Error: Could no twrite to init_vars.php
HttpRequestSched: [ERROR] Repair reset code for conforming run of set COND_GET_REQ_PERCENT to 0 for non-conforming run. Run halted.
SPECweb_Support: [ERROR] Error! setServerDate() failed.
RemoteLoadGen: [ERROR] Unable to successfully initialize workload variables. Termininating

Finally got it work, ESX to go

Apache2.2.3 + Php5.1.6 + cgi problem
Symptom:
As corrected before, directory listing works. But it doesn't work for cgi-bin.
SPECweb complains:
attemp to invoke directory as script: /var/www/cgi-bin/
On the browser, it complains:
403 Forbidden
You don't have permission to access /cgi-bin/ on this server
But actually it has the full permission 777 and it has the apache:apache ownership
(it sounds like specweb cannot write script in cgi-bin direcotry)
To make php work properly in Apache

Solution:
1. in php.conf
add
AddType application/x-httpd-php .php
comment out
AddType application/x-httpd-php-source .phps

2.To make cgi work properly in Apache
change
ScriptAlias /cgi-bin/ "/var/www/cgi-bin"
Directory "/var/www/cgi-bin"
AllowOverride None
Options None
Order allow, deny
Allow from all
/Directory
to
Alias /cgi-bin/ "/var/www/cgi-bin"
Directory "/var/www/cgi-bin"
AllowOverride None
Options ExecCGI Indexes
Order allow, deny //no space between allow and deny
AddHandler cgi-script cgi pl // if in the cgi-bin/ is perl scripts , if python, then add py
Directory

3. SElinux conflicts with apache
Sometimes SElinux also brings trouble without ideas. So disable SElinux and reboot is more safe and saves your trouble

Also two helpful commands:
1)ls -aZ
to display the SElinux security configuration
--- system_u:object_r:httpd_sys_script_exec_t:s0
--- system_u:object_r:httpd_sys_content_t:s0
use chcron to change it

2) /sbin/service httpd configtest
Syntax OK
to validate the http.conf syntax error


Now testing with 20 sessions in 131.......
---------------------------------
Here is how to make fastcgi working in Apache2.2.3 (When I am free, I will translate it in English)
http://hi.baidu.com/coffeefoam/blog/item/1446493be749f3e814cecbb8.html

On the road, ESX

1. Create dataset
support_downloads_props.rc is segmented in two parts: section A and section B (B is for specweb running, recommended not to change)
in A
SIMULTANEOUS_SESSIONS=40
base directory number = basenumber + sessions* discaling = 0 + 40 *.25 = 10
the rest are for configuring the directory not to exceed the system limitation

support_image_props.rc only has section A:
SIMULTANEOUS_SESSIONS=40

#./Wafgen support_downloads_props.rc
#./Wafgen support_image_props.rc
DOCROOT is in /var/www, go to /var/www to pick up the dataset and add the scripts in it.

setup->network configuration
XXX.XXX.XXX.13X
255.255.255.0
XXX.XXX.XXX.1

#/etc/init.d/network restart
#service httpd restart

documentroot "/var/www"

------------------------------------------------------
Apache lame
1. 404 Not found
You cannot access / on this server
But display index.php as normal

forgot to change to /var/www

2. 403 Forbidden
You do not have permission to access / on this server
Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.
In /etc/httpd/logs/error.log, Directory index forbidden by options directive /var/www
/var/www/Error has been moved out of the directory, so it should display the directory list, but nope.

In apache2.2.3 besides httpd.conf configuration file for apache, it also has configuration files in conf.d/ .
1) With error/ directory moved out, directory listing doesn't work.
Go to php.conf,
#Add index.php to the list of files that will be served as directory indexes
DirectoryIndex index.php
this should be commented out
2) Browser the subdirectory of localhost/subdir , the directory can be displayed but localhost/ still doesn't work
It keeps saying "Directory index forbidden by options directive /var/www", checked allowoverride, options +Indexes, problems can never be solved, but move error/ back to /var/www, it works. Page can be displayed.
where does that page come from?
check conf.d/ again, welcome.conf defines the welcome page.
In it, saying "Options -Indexes". Catch it, which has override the /var/www options!

Comments:
Apache access rules definition is a good one. It has the options defining the access patter, it has allowoverride defining the access priority, it has Allow defining the access clients. Maybe it can be applied to security system if the access follows if owning theh access to one directory, then also heritages the subdirectory.
It had worked fine in apache2.2. But apache2.3 gave me pains. In apache2.3, it has scattered many sub configurations around but no hints or instructions for the change. So each new version of Linux, it is a new system which you have to be familiar with again. Also sticking to the old ways may be a good solution, also when using the new style, knowing its working mechanism and its style first before get into using it.

Ready to Go, ESX

Done:
1. uninstall server configuration tool
2. setup-> disable yum update and enable httpd
yum update must be disabled, if updating, gimp and gaim patches will break the update.
3. change /etc/httpd/conf/httpd.conf documentroot=/var/www
4. create index.php to display php information in /var/www
php already installed, also apache 2.2.3
Todo:
1. Change IP to static ip
2. Use wafgen to create support dataset with 40 sessions
sessions > the session value in Test.config
3. put the dataset in /var/www of each image
4. change the ownership of directory /var/www and its sub dirs
chown -R apache:apache /var/www
5. chagne the rights of files /var/www and its sub files
chmod -R 777 /var/www

Undecided:
1. swap impact, current no swap
2. how much memory is approprieate, current 512, used to be 128 and almost stuck in it
3. maybe compiling issues

All four images are running with memory 512M. php info displayed in nycrr host.

Cursed server configuration tool

http://www.redhat.com/docs/manuals/enterprise/RHEL-5-manual/Deployment_Guide-en-US/s1-apache-config-ui.html
Caution
Do not edit the /etc/httpd/conf/httpd.conf configuration file by hand if you wish to use this tool. The HTTP Configuration Tool generates this file after you save your changes and exit the program. If you want to add additional modules or configuration options that are not available in HTTP Configuration Tool, you cannot use this tool.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
When changing Documentroot or Listen port, all goes wrong. It said
404 Not Found
The reuested URL / was not found on this server

Then when using the server configuration tool GUI stuff
change the virtual host web root there, it messed up http.conf, inserting a lot of empty lines at the beginning and at the end adding scattered grammars there without comments.

Solution:
uninstall server configuration tool.

Comments:
Linux configuration is based on configuration file---text parsing. Windows configuration is based on register table---table. Now Linux is trying to emulate windows GUI configuration tool but their underlying is still text, not format controllable. And configuration GUI actually is another interpretation of configuration grammar rule. Not necessary at all. Linux strength is at its developing power. It is compiler style.
Either totally convert to XML or an uniform format to all applications or just stay where it originally is.


Sample of simple index file

index.html
html Hello World! /html

index. php
?php phpinfo(); ?

Good Reference: http://www.google.com/codesearch

Reasons for LFS

Disk volumn is not the criticle point any more. 500G only costs $100.

The harddriver has developed from ATA to SATA, whose speed is now competitive to the server level data access SCSI.

Now problems come as how to well manage large data, how to well index the data and fast search the data, how to support large number of directories and large size of files. All circle around file management.

LFS is targeting at roll back, backup and recovery, involving maintaining multiple copies of edited version. So how to most efficiently save the space and quickly access the right spot is a good topic.

Also IBM proposed a paper on maintaining multiple versions of updated applications, trying to solve version dependency which is also a good hint to be adopted into LFS on the application's layout. Besides that, CVS may be a good example of maintaining multiple version. Its implementation details are good to know.

Apache2.2.3 stuck

Changing server port number leads to startup Failed. only 80 works
Changing documentroot leads to 404 Not Found
The requested URL / was not found on this server
file:///var/www/index.html works

#httpd -t //run configuration syntax test
Syntax OK
#service httpd start restart stop
#htmlview index.html

Kernel, OS

In Data Network equipment market, CISCO is the giant, who is the first one getting into the market and who owns the biggest share in this market. Huawei is catching him up. But the latest prediction in Wallstreet said that CISCO is still unbeatable in OS and router.
So come back to the old topic, OS is the Core value of Information Era.
It is the most complicating project, similiar to compiler.
It is the most basic layer for all applications, no matter desktop, server, or router.
It is the most key element in performance providing.

OS involves Scheduler, File system, and communication. Scheduler determines computing, FS determines storage, and communicaiton determines exchanging. It almost covers all the aspects of information technologies. It has diversial patterns due to how much computing power you have and how much data you will handle. So it is a really amazing kaleidoscope.

Think about Google: why is so great? Because it is neat philosophy: simpilicity, friendly, convenience. Google is always designed in the way that tries to get rid of annoyance as much as possible, make your operations shortcut and simple straight.
But on the other hand, that's actually not all. It is excellent more based on its quick response. But how can it make so quick response instead of waiting and waiting in dead cycle? It is great power of computing. Where does their great computing power come from? personal desktops. How do they make personal desktops provide so great power? The OS with efficient algorithm manages the distributed PCs.

So for google, its core value is still OS and their GFS that are so delicated to be efficient.

Image Package

The specweb turns out to be a valid run with 20 sessions running and 2 10-session specwebs running in the other two container. But there is some degradation of good performance
Good Tolerant Fail
99.7% 100% 0%
99.6% 99.9% 0.1%
99.7% 100% 0%

The second image is done.
Image Package:
no Games and Entertainment
no Graphics
no Office/Productivity
no GNOME Software Development
but Server Configuration Tools
no Dial-up Networking Support
But System Tools

Firewall:
ssh, http
SElinux: enforcing

The third and fourth are also done. A test image is on-going as a sample to make SPECweb working in it.

SPECweb reply

The invalid run in our system is due to too low of a request rate instead of overloading the server.
Official proposed solution: increasing the simultaneous sessions until QoS or Vadlidation Error. Then try to solve the QoS or Validation error instead of Request Distribution Error.

Create the dataset of 20 simultaneous sessions with wafgen, and put it in 119 solaris container.
After 119 has finished 1 iteration with 20 sessions, start another 2 with 10 sessions
It reports
Sum of weighted percentage difference (1.82%) exceeds 1.5% for Iteration 1
So it is almost there.
----------------------------------------------------
One thing to mention: for the valid submission, it should have support, commerce and bank three results ready. And also raw files should be backup as reference.

The first ESX image is going with temorary memory 512M without swap. ESX_1 baby is born, but php not ready yet, static ip is not ready yet. DHCP ip is a temperary solution. httpd is OK.
The second one is sharing the same CDROM with the first one. Disconnect the first one from CDROM.
CDROM is ejected, so the second one cannot find operating system to boot up
Now the second is on the way... ...

ESX License issue

insufficient license:
Host-Based License: the license files are stored on individual ESSX server hosts
Server-Based License: licenses are stored on a license server, available to multiple hosts.
Host-Based License is enough, per-machine basis, placed on the individual ESX Server hosts

issues:
Go to Inventory -> localhost.localdomain ->Configuration -> License Features
License resource (Edit on the right)
Edit -> Use Host License File -> Upload local file
change VI_SingleHost.Lic to vmware.lic
Host License File: Installed
ESX Server License (Edit on the right)
Edit -> radio button ESX Server Standard (ESX Server Starter grey)
ESX Server Standard Licensed for 1 CPU
SAN Usage
Up to 4-way virtual SMP
NAS Usage
ONLY 1 CPU License, how can it be 2 CPU Licensed? Will it be any impact there?

Image Creation
Partition:
no swap (should study the impact on no swap, do you need to?)
IP range:
128.153.145.131~134
MAC:
00 0C 29 AA F1 07
Next step: populate the rest three images, ESX_2, 3, 4 and install apache and php in it

Invalid Run

Run only one client with 5 sessions in Solaris:
Aggregate QoS Compliance are all 100% good, no QoS Errors, no Validation Errors.
But request distribution errors are high:

• Weighted percentage difference (-1.50%) for search in Iteration 1 is too high. Expected: 109 requests, Actual: 122
• Weighted percentage difference (-2.46%) for search in Iteration 2 is too high. Expected: 106 requests, Actual: 127
• Weighted percentage difference (-3.47%) for product in Iteration 2 is too high. Expected: 208 requests, Actual: 238
• Weighted percentage difference (-2.28%) for fileCatalog in Iteration 2 is too high. Expected: 189 requests, Actual: 209
• Sum of weighted percentage difference (-9.18%) exceeds 1.5% for Iteration 2
• Weighted percentage difference (-1.39%) for search in Iteration 3 is too high. Expected: 109 requests, Actual: 122

Running specwebclient with option -lo to record the reports and errors. An email has been sent to specweb attached with the standout. Waiting for reply.
---------------------------------------------------------------------------------------
VMware License
Failed twice last night. Using lib's gmail got one. It has been forwarded to my gmail. Time to register and create four images, unsure of what new problems may come up. Maybe only support one host. Time to prove.

Image installation

Sunday, from 7PM to 12PM, there are 5 hours to go
Launch VMware ESX
IBM eServer BladeCenter LS20
2CPU X 1.993 GHz
Dual Core AMD Opteron(tm) Processor 270
Memory: 441M used/ 4G total
Storage: 60.14G free/60.75G capacity
Active Adapters: vmnic0 1000full 128.153.144.192~195
SCSI adapter: 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI

Add License:
Inventory --> Configuration tab> License Source Edit

waiting for new license
----------------------------------------------------------------------------------------------------
SPECweb working review:
put Directory "support" and "Smarty" into the directory var/www
change the Documentroot option in /etc/httpd/conf/httpd.conf
previliges:
chmod -R 777 www/
ownership:
chown -R apache:apache www/

Resutls backup:
111, 114, 115, 116 are four clients to test OpenVZ
116 is the last client where the bombs run in it.
the results should be tarballed so that the date can be kept as original, or it will be updated as the date when copying.

ESX on blade14

It is a temporary solution due to scsi no working yet on TC.
Following VMware folks proposal, tests will run on VMware ESX. As a comparision, also Xen Express. They are all enterprise level virtualization system.
Blade 14:
used to be Ubuntu bladexen and erased.
AMD Opton Dual Core 2.0G
Memeory 4096MB

ESX Server 3.0.1
Version Date:
5/31/07
Harddisk:
SCSI DISK sda IBM-ESXS ST973401LC FN 70002MB
NIC:
X 2:1:0 tg3 NetXtreme BCM5704S Gigabyte
_ 3:1:0
_ 2:1:1
_ 3:1:1
DHCP:
128.153.144.119
Default set up virtual network

Todo tomorrow:
Create Virtual Image
License
SPECweb running

SPECweb news:
Tune the simultaneous session value from 1 to 10, but never get valid run
Use the specwebclient option -lo
#java specwebclient -lo
output will write to "specwebclient1099.out", no direct clue yet. An email inquiries on "invalid run" should be sent to the specweb support.

The inconsistency is that the dataset produced by wafgen is 10 sessions, but the specweb is using 5 sessions.
When having tuned to 10 sessions for specweb to run, there were 40 sessions running on the TC, there were 1% degradation for good, 1% for tolerance.

Thoughts from MAC

Put each application in a conexecutive and complete block of space, each update or change is put in the log. When cleaning, it is time to integrate them all. In that way, the file modified or changed, may lay in different place which may introduce overhead, it also keep the application versions.

Pros: version control, integration, independent packages.

LOG Opening

This blog is opened due to graduate study in USA. Science is a serious and interesting wanderland.
This blog is mainly logging the ideas coming up and the steps taking for the experiements.
Here it will record the failure and success, the frustration and cheers, the basic and classic. But on the road to the top of Computing and Internet, we need a calm and persistent heart to hear the nature of CS, a delligent and delicated hand to practice every possibility of IT.
To keep the mind of deeping into the research, I would like to quote the words from Noble Prize winner Luise Alvarez
"Why don't you look at this this as research. When you're doing real research, you never know what it'll cost, how much time it'll take, or what you'll find. You just know theres unexplored territory and a chnace to discover what's out there. ... ...You're following a fascinating scent. You're an explorer. Think of 'what' might be behind it. ... ...Well then, forget who's causing the problems, Don't try to be a cop, be a scientist; Research the connections, the techniques, the holes. Apply physical principles. Find new methods to solve problem. Compile statistics, publish your results, and only trust what you can prove. But don't exclude improbable solutions---keep your mind open. .. ... Dead ends are illusory. When did you ever let a 'Do Not Enter' sign keep you away from anything? Go around the brick walls. When you can;t go around, climb over or dig under. Just don't give up. ... ...Nobody will pay for research; hey're only interested in results... ... Don't wait for someone else, do it yourself. Keep your boss happy, but don't let him tie you down. "
So,
It is the logbook of how I pursue for the degree
It is the diary of what I am doing everyday in academic
It is the report of weekday research work
It is the try of learning something new
It is the note of repeating something old
It is the blueprint of the random thoughts, ideas, and talks
It is the bookmark of the proposals, presentations, paper, and thesis.

It is a LOGBOOK in Internet.