you can still create a def file using the output from the objdump program (from the MinGW distribution). Here's an example.
objdump -p mydll.dll > mydll.fil
Search for "[Ordinal/Name Pointer] Table" in mydll.fil and use the list of functions following it to create your def file.
Monday, November 26, 2007
Wednesday, November 21, 2007
Upload Video to Youtube
I am finally on the verge to integrate a youtube uploader tool to one of my side project. I just need to figure out how to log to gmail instead of youtube since there is two ways to authenticate with youtube.
About youtube :
When you upload a video to YouTube, it automatically converts it to a flash video file (.flv) with specific settings (see below, standard column). The contrast of the video is increased and video noise is suppressed. YouTube appears to respect the framerate of the uploaded file; Framerates will varie between 15 (minimum framerate to tell it is a video...), 24 frames (pal) and 30 frames per second (ntsc with no drop frame). It is hard to tell how long you have to wait for the video to be available but it is pretty fast !!. Once the video has been uploaded you can log to your account and from there you can check the status of your video. Usually you will see Uploaded (processing, please wait).
Video Profile
standard recommended possible
file extension : .flv (flash video)
total bitrate : ~320 kbits/sec 804.7 kbits/sec 2147.2 kbits/sec
audio codec : MPEG layer 3 (MP3)
audio format : Mono, 22.050 kHz Stereo, 44.100 kHz Stereo, 44.100 kHz
audio bitrate : ~67 kpbs/sec 128 kpbs/sec 128 kpbs/sec
video codec : Sorenson Spark (H.263)
video format : 320 x 240 px, 15-30 fps 320 x 240 px, 29.97 fps 640 x 480 px, 29.97 fps
video bitrate : ~250 kbps/sec 677.7 kbps/sec 2019.2 kbps/sec
You can save your own .flv files and upload them to YouTube. You can encode video with higher bitrates and resolution than YouTube compresses with. The only limitations I can tell are that the file must be under 100 MB and you must use the Sorenson Spark video codec and MP3 audio codec.
About youtube :
When you upload a video to YouTube, it automatically converts it to a flash video file (.flv) with specific settings (see below, standard column). The contrast of the video is increased and video noise is suppressed. YouTube appears to respect the framerate of the uploaded file; Framerates will varie between 15 (minimum framerate to tell it is a video...), 24 frames (pal) and 30 frames per second (ntsc with no drop frame). It is hard to tell how long you have to wait for the video to be available but it is pretty fast !!. Once the video has been uploaded you can log to your account and from there you can check the status of your video. Usually you will see Uploaded (processing, please wait).
Video Profile
standard recommended possible
file extension : .flv (flash video)
total bitrate : ~320 kbits/sec 804.7 kbits/sec 2147.2 kbits/sec
audio codec : MPEG layer 3 (MP3)
audio format : Mono, 22.050 kHz Stereo, 44.100 kHz Stereo, 44.100 kHz
audio bitrate : ~67 kpbs/sec 128 kpbs/sec 128 kpbs/sec
video codec : Sorenson Spark (H.263)
video format : 320 x 240 px, 15-30 fps 320 x 240 px, 29.97 fps 640 x 480 px, 29.97 fps
video bitrate : ~250 kbps/sec 677.7 kbps/sec 2019.2 kbps/sec
You can save your own .flv files and upload them to YouTube. You can encode video with higher bitrates and resolution than YouTube compresses with. The only limitations I can tell are that the file must be under 100 MB and you must use the Sorenson Spark video codec and MP3 audio codec.
Monday, November 19, 2007
How to take advantage of Silverlight Marker to create hypervideo
Interesting article from Jesse Liberty also i worked on Hypervideo back in 1999.... Silverlight is finally well suited to turn HD video into interactive sequences.
http://silverlight.net/blogs/jesseliberty/archive/2007/10/24/hypervideo-2.aspx
http://silverlight.net/blogs/jesseliberty/archive/2007/10/24/hypervideo-2.aspx
Wednesday, November 14, 2007
silent install of the VC 8.0 runtime (vcredist) packages
if you have downloaded the standalone VC 8.0 redistributable packages, you will need to modify the command lines slightly. The following command lines can be used to install the original release of the standalone VC 8.0 redistributable packages:
For x86: vcredist_x86.exe /q:a /c:"VCREDI~1.EXE /q:a /c:""msiexec /i vcredist.msi /qn"" "
For x64: vcredist_x64.exe /q:a /c:"VCREDI~2.EXE /q:a /c:""msiexec /i vcredist.msi /qn"" "
For ia64: vcredist_ia64.exe /q:a /c:"VCREDI~3.EXE /q:a /c:""msiexec /i vcredist.msi /qn"" "
The following command lines can be used to install the Visual Studio 2005 SP1 release of the standalone VC 8.0 redistributable packages:
For x86: vcredist_x86.exe /q:a /c:"VCREDI~3.EXE /q:a /c:""msiexec /i vcredist.msi /qn"" "
For x64: vcredist_x64.exe /q:a /c:"VCREDI~2.EXE /q:a /c:""msiexec /i vcredist.msi /qn"" "
For ia64: vcredist_ia64.exe /q:a /c:"VCREDI~1.EXE /q:a /c:""msiexec /i vcredist.msi /qn"" "
If you would like to install the VC runtime packages in unattended mode (which will show a small progress bar but not require any user interaction), you can change the /qn switch above to /qb. If you would like the progress bar to not show a cancel button, then you can change the /qn switch above to /qb!
Some folks ask me what vcredist.msi..
Iy means that you want to extract the embedded msi from the self exe package and be able to run msiexec which is used to install the package. msiexec is the Microsoft Installer runtime.
MSI is the prefered format for microsoft installer. Installshield and Visual Studio 2005 and higher can generate MSI. Using the msiexec command. You can even log all the installer session to a file and you can verify and explore the content of the MSI using ORCA in order to look at the installer sequence, string table and in some cases embedded Visual Basic Script. Yes Visual Basic is not dead. Still used and convnient enough for scripting installer code :)
If you want to install the runtime from a Null Soft Installer NSI script with minimal UI and progress bar, here is what you want to do
;extract selft exe to temp
SetOutPath "$TEMP"
; uncompress
File "${RUNTIME_VCREDIST}"
;-------------------------------
; Test if Visual Studio Redistributables 2005+ SP1 installed
; Returns -1 if there is no VC redistributables intstalled
Function CheckVCRedist
Push $R0
ClearErrors
ReadRegDword $R0 HKLM "SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\{7299052b-02a4-4627-81f2-1818da5d550d}" "Version"
; if VS 2005+ redist SP1 not installed, install it
IfErrors 0 VSRedistInstalled
StrCpy $R0 "-1"
VSRedistInstalled:
Exch $R0
FunctionEnd
Section "Installer Sections" SecDummy
SetOverwrite ifnewer
Call CheckVCRedist
;IntCmp $0 5 is5 lessthan5 morethan5
IntCmp $R0 -1 noVCRuntime noVCRuntime VCRuntime
;force install
; IntCmp $R0 -1 noVCRuntime noVCRuntime noVCRuntime
noVCRuntime:
;extract VCRuntime to Temp
SetOutPath "$TEMP"
File "${RUNTIME_VCREDIST}"
;/qn for complete silent install
;/qb small progress dialog but does not require any user interaction
;StrCpy $VC_RUNTIME "$\"$TEMP\vcredist_x86.exe$\" /q:a /c:$\"VCREDI~3.EXE /q:a /c:$\"$\"msiexec /i vcredist.msi /qb!$\""
StrCpy $VC_RUNTIME "$\"$TEMP\vcredist_x86.exe$\" /q:a"
;MessageBox MB_ICONINFORMATION 'Need to install VCRuntime {$R0} {$VC_RUNTIME}'
ExecWait '$VC_RUNTIME' $0
;MessageBox MB_ICONINFORMATION 'Return Code {$0}'
;IntCmp $0 0 done
;MessageBox MB_ICONINFORMATION 'install complete'
Delete "$TEMP\vcredist_x86.exe"
Goto VCRuntime
VCRuntime:
SectionEnd
http://blogs.msdn.com/astebner/archive/2007/02/07/update-regarding-silent-install-of-the-vc-8-0-runtime-vcredist-packages.aspx
For x86: vcredist_x86.exe /q:a /c:"VCREDI~1.EXE /q:a /c:""msiexec /i vcredist.msi /qn"" "
For x64: vcredist_x64.exe /q:a /c:"VCREDI~2.EXE /q:a /c:""msiexec /i vcredist.msi /qn"" "
For ia64: vcredist_ia64.exe /q:a /c:"VCREDI~3.EXE /q:a /c:""msiexec /i vcredist.msi /qn"" "
The following command lines can be used to install the Visual Studio 2005 SP1 release of the standalone VC 8.0 redistributable packages:
For x86: vcredist_x86.exe /q:a /c:"VCREDI~3.EXE /q:a /c:""msiexec /i vcredist.msi /qn"" "
For x64: vcredist_x64.exe /q:a /c:"VCREDI~2.EXE /q:a /c:""msiexec /i vcredist.msi /qn"" "
For ia64: vcredist_ia64.exe /q:a /c:"VCREDI~1.EXE /q:a /c:""msiexec /i vcredist.msi /qn"" "
If you would like to install the VC runtime packages in unattended mode (which will show a small progress bar but not require any user interaction), you can change the /qn switch above to /qb. If you would like the progress bar to not show a cancel button, then you can change the /qn switch above to /qb!
Some folks ask me what vcredist.msi..
Iy means that you want to extract the embedded msi from the self exe package and be able to run msiexec which is used to install the package. msiexec is the Microsoft Installer runtime.
MSI is the prefered format for microsoft installer. Installshield and Visual Studio 2005 and higher can generate MSI. Using the msiexec command. You can even log all the installer session to a file and you can verify and explore the content of the MSI using ORCA in order to look at the installer sequence, string table and in some cases embedded Visual Basic Script. Yes Visual Basic is not dead. Still used and convnient enough for scripting installer code :)
If you want to install the runtime from a Null Soft Installer NSI script with minimal UI and progress bar, here is what you want to do
;extract selft exe to temp
SetOutPath "$TEMP"
; uncompress
File "${RUNTIME_VCREDIST}"
;-------------------------------
; Test if Visual Studio Redistributables 2005+ SP1 installed
; Returns -1 if there is no VC redistributables intstalled
Function CheckVCRedist
Push $R0
ClearErrors
ReadRegDword $R0 HKLM "SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\{7299052b-02a4-4627-81f2-1818da5d550d}" "Version"
; if VS 2005+ redist SP1 not installed, install it
IfErrors 0 VSRedistInstalled
StrCpy $R0 "-1"
VSRedistInstalled:
Exch $R0
FunctionEnd
Section "Installer Sections" SecDummy
SetOverwrite ifnewer
Call CheckVCRedist
;IntCmp $0 5 is5 lessthan5 morethan5
IntCmp $R0 -1 noVCRuntime noVCRuntime VCRuntime
;force install
; IntCmp $R0 -1 noVCRuntime noVCRuntime noVCRuntime
noVCRuntime:
;extract VCRuntime to Temp
SetOutPath "$TEMP"
File "${RUNTIME_VCREDIST}"
;/qn for complete silent install
;/qb small progress dialog but does not require any user interaction
;StrCpy $VC_RUNTIME "$\"$TEMP\vcredist_x86.exe$\" /q:a /c:$\"VCREDI~3.EXE /q:a /c:$\"$\"msiexec /i vcredist.msi /qb!$\""
StrCpy $VC_RUNTIME "$\"$TEMP\vcredist_x86.exe$\" /q:a"
;MessageBox MB_ICONINFORMATION 'Need to install VCRuntime {$R0} {$VC_RUNTIME}'
ExecWait '$VC_RUNTIME' $0
;MessageBox MB_ICONINFORMATION 'Return Code {$0}'
;IntCmp $0 0 done
;MessageBox MB_ICONINFORMATION 'install complete'
Delete "$TEMP\vcredist_x86.exe"
Goto VCRuntime
VCRuntime:
SectionEnd
http://blogs.msdn.com/astebner/archive/2007/02/07/update-regarding-silent-install-of-the-vc-8-0-runtime-vcredist-packages.aspx
Sunday, November 11, 2007
Friday, October 12, 2007
Parsing XML in PHP
Traverse the document tree
In this example we have a XML file which consists of a document with a specific version. The document contains persons with the attributes firstname, lastname and description. The code snippet finds the document version and prints it. It's pretty straight forward you just check all the top nodes of the document and look for the one named document.
foreach ( $tree->children as $document )
{
// parse the document
if ( $document->name == "document" )
{
// get the document version attribute
foreach ( $document->attributes as $documentAttr )
{
if ( $documentAttr->name == "version" )
{
print( "Found document with version: " . $documentAttr->content . "
" );
}
}
// find persons here
}
}
When you've found the document node you can start looking for persons. This is done in the same manner, check the children nodes and look for person.
To make the process of getting the attribute values simpler we write a helper function to fetch the attribute value from a node.
function getAttrValue( $node, $attrName )
{
$ret = false;
foreach ( $node->attributes as $nodeAttr )
{
if ( $nodeAttr->name == $attrName )
{
$ret = $nodeAttr->content;
}
}
return $ret;
}
Now we're ready to parse the information describing the persons in this example. When we've found the person node we check all the subnodes and look for the nodes we want and fetch the information from these nodes.
// parse all persons
foreach ( $document->children as $person )
{
if ( $person->name == "person" )
{
print( "Found a new person
" );
$firstName = "";
$lastName = "";
$descriptionName = "";
// get the name and description
foreach ( $person->children as $personAttribute )
{
switch ( $personAttribute->name )
{
case "firstname" :
{
$firstName = getAttrValue( $personAttribute, "value" );
}break;
case "lastname" :
{
$lastName = getAttrValue( $personAttribute, "value" );
}break;
case "description" :
{
// get the description text
foreach ( $personAttribute->children as $description )
{
if ( $description->type == 3 )
{
$description = $description->content;
}
}
}break;
}
}
print( "The persons firstname is: $firstName
" );
print( "The persons lastname is: $lastName
" );
print( "The persons description is: $description
" );
}
}
include_once( "ezxml/classes/ezxml.php" );
$xmlDocument =
"
Coder.
Coder.
";
$tree =& eZXML::domTree( $xmlDocument, array( "TrimWhiteSpace" => true ) );
foreach ( $tree->children as $document )
{
// parse the document
if ( $document->name == "document" )
{
// get the document version attribute
foreach ( $document->attributes as $documentAttr )
{
if ( $documentAttr->name == "version" )
{
print( "Found document with version: " . $documentAttr->content . "
" );
}
}
// parse all persons
foreach ( $document->children as $person )
{
if ( $person->name == "person" )
{
print( "Found a new person
" );
$firstName = "";
$lastName = "";
$descriptionName = "";
// get the name and description
foreach ( $person->children as $personAttribute )
{
switch ( $personAttribute->name )
{
case "firstname" :
{
$firstName = getAttrValue( $personAttribute, "value" );
}break;
case "lastname" :
{
$lastName = getAttrValue( $personAttribute, "value" );
}break;
case "description" :
{
// get the description text
foreach ( $personAttribute->children as $description )
{
if ( $description->type == 3 )
{
$description = $description->content;
}
}
}break;
}
}
print( "The persons firstname is: $firstName
" );
print( "The persons lastname is: $lastName
" );
print( "The persons description is: $description
" );
}
}
}
}
/*!
Function to fetch an attribute value.
Will return the value of the attribute if found. False if not found.
*/
function getAttrValue( $node, $attrName )
{
$ret = false;
foreach ( $node->attributes as $nodeAttr )
{
if ( $nodeAttr->name == $attrName )
{
$ret = $nodeAttr->content;
}
}
return $ret;
}
This code will produce the following output:
Found document with version: 42
Found a new person
The persons firstname is: Bård
The persons lastname is: Farstad
The persons description is: Coder.
Found a new person
The persons firstname is: Christoffer A.
The persons lastname is: Elo
The persons description is: Coder2.
Using XML is simple and straightforward with the eZ xml class. You don't need any external libraries, the class produce the same document tree as you would get from the XML functions in PHP (which all need external libraries). It is also the only PHP XML parser class which returns the same object tree as the library functions, making it easy to use your programs both on sites where XML is compiled into PHP and sites where it isn't.
In this example we have a XML file which consists of a document with a specific version. The document contains persons with the attributes firstname, lastname and description. The code snippet finds the document version and prints it. It's pretty straight forward you just check all the top nodes of the document and look for the one named document.
foreach ( $tree->children as $document )
{
// parse the document
if ( $document->name == "document" )
{
// get the document version attribute
foreach ( $document->attributes as $documentAttr )
{
if ( $documentAttr->name == "version" )
{
print( "Found document with version: " . $documentAttr->content . "
" );
}
}
// find persons here
}
}
When you've found the document node you can start looking for persons. This is done in the same manner, check the children nodes and look for person.
To make the process of getting the attribute values simpler we write a helper function to fetch the attribute value from a node.
function getAttrValue( $node, $attrName )
{
$ret = false;
foreach ( $node->attributes as $nodeAttr )
{
if ( $nodeAttr->name == $attrName )
{
$ret = $nodeAttr->content;
}
}
return $ret;
}
Now we're ready to parse the information describing the persons in this example. When we've found the person node we check all the subnodes and look for the nodes we want and fetch the information from these nodes.
// parse all persons
foreach ( $document->children as $person )
{
if ( $person->name == "person" )
{
print( "Found a new person
" );
$firstName = "";
$lastName = "";
$descriptionName = "";
// get the name and description
foreach ( $person->children as $personAttribute )
{
switch ( $personAttribute->name )
{
case "firstname" :
{
$firstName = getAttrValue( $personAttribute, "value" );
}break;
case "lastname" :
{
$lastName = getAttrValue( $personAttribute, "value" );
}break;
case "description" :
{
// get the description text
foreach ( $personAttribute->children as $description )
{
if ( $description->type == 3 )
{
$description = $description->content;
}
}
}break;
}
}
print( "The persons firstname is: $firstName
" );
print( "The persons lastname is: $lastName
" );
print( "The persons description is: $description
" );
}
}
include_once( "ezxml/classes/ezxml.php" );
$xmlDocument =
"
Coder.
Coder.
$tree =& eZXML::domTree( $xmlDocument, array( "TrimWhiteSpace" => true ) );
foreach ( $tree->children as $document )
{
// parse the document
if ( $document->name == "document" )
{
// get the document version attribute
foreach ( $document->attributes as $documentAttr )
{
if ( $documentAttr->name == "version" )
{
print( "Found document with version: " . $documentAttr->content . "
" );
}
}
// parse all persons
foreach ( $document->children as $person )
{
if ( $person->name == "person" )
{
print( "Found a new person
" );
$firstName = "";
$lastName = "";
$descriptionName = "";
// get the name and description
foreach ( $person->children as $personAttribute )
{
switch ( $personAttribute->name )
{
case "firstname" :
{
$firstName = getAttrValue( $personAttribute, "value" );
}break;
case "lastname" :
{
$lastName = getAttrValue( $personAttribute, "value" );
}break;
case "description" :
{
// get the description text
foreach ( $personAttribute->children as $description )
{
if ( $description->type == 3 )
{
$description = $description->content;
}
}
}break;
}
}
print( "The persons firstname is: $firstName
" );
print( "The persons lastname is: $lastName
" );
print( "The persons description is: $description
" );
}
}
}
}
/*!
Function to fetch an attribute value.
Will return the value of the attribute if found. False if not found.
*/
function getAttrValue( $node, $attrName )
{
$ret = false;
foreach ( $node->attributes as $nodeAttr )
{
if ( $nodeAttr->name == $attrName )
{
$ret = $nodeAttr->content;
}
}
return $ret;
}
This code will produce the following output:
Found document with version: 42
Found a new person
The persons firstname is: Bård
The persons lastname is: Farstad
The persons description is: Coder.
Found a new person
The persons firstname is: Christoffer A.
The persons lastname is: Elo
The persons description is: Coder2.
Using XML is simple and straightforward with the eZ xml class. You don't need any external libraries, the class produce the same document tree as you would get from the XML functions in PHP (which all need external libraries). It is also the only PHP XML parser class which returns the same object tree as the library functions, making it easy to use your programs both on sites where XML is compiled into PHP and sites where it isn't.
Wednesday, September 26, 2007
PuTTY: A Free Telnet/SSH Client
I finally build putty for my Motorola Q VGA Phone Windows Mobile 5 Phone. The migration from eVC to visual studio 8 went well. I just need to fix some minor issue with some keyboard key not interpreted correctly when the host name need to be provided and figure out how to send specific keyboard combination key at once such as ctrl+c from a menu. Finally provide an installer to be able to have an icon in the tray. Without icons it is not possible for me to come back to putty once i need to answer a call.
Friday, August 31, 2007
Scene Cut Detection
Back in time i use to integrate on Windows the following software issued by
this famous French Laboratory. The purpose was to perform scene cut detection
thanks to a frame by frame video engine written for Windows and DirectShow.
The algorithm was very CPU intensive but pretty interesting.
http://www.irisa.fr/vista/Themes/Logiciel/MdShots/MdShots.english.html
this famous French Laboratory. The purpose was to perform scene cut detection
thanks to a frame by frame video engine written for Windows and DirectShow.
The algorithm was very CPU intensive but pretty interesting.
http://www.irisa.fr/vista/Themes/Logiciel/MdShots/MdShots.english.html
Monday, August 13, 2007
HTTP performance testing with httperf, autobench
HTTP performance testing with httperf, autobench
* httperf is a benchmarking tool that measures the HTTP request throughput of a web server. The way it achieves this is by sending requests to the server at a fixed rate and measuring the rate at which replies arrive. Running the test several times and with monotonically increasing request rates, one can see the reply rate level off when the server becomes saturated, i.e., when it is operating at its full capacity.
* autobench is a Perl wrapper around httperf. It runs httperf a number of times against a Web server, increasing the number of requested connections per second on each iteration, and extracts the significant data from the httperf output, delivering a CSV format file which can be imported directly into a spreadsheet for analysis/graphing.
I ran a series of autobench/httperf and openload tests against a Web site I'll call site2 in the following discussion (site2 is a beta version of a site I'll call site1). For comparison purposes, I also ran similar tests against site1 and against www.example.com. The machine I ran the tests from is a Red Hat 9 Linux server co-located in downtown Los Angeles.
Here is an example of running httperf against www.example.com:
# httperf --server=www.example.com --rate=10 --num-conns=500
httperf --client=0/1 --server=www.example.com --port=80 --uri=/ --rate=10 --send-buffer=4096 --recv-buffer=16384 --num-conns=500 --num-calls=1
Maximum connect burst length: 1
Total: connections 500 requests 500 replies 500 test-duration 50.354 s
Connection rate: 9.9 conn/s (100.7 ms/conn, <=8 concurrent connections)
Connection time [ms]: min 449.7 avg 465.1 max 2856.6 median 451.5 stddev 132.1
Connection time [ms]: connect 74.1
Connection length [replies/conn]: 1.000
Request rate: 9.9 req/s (100.7 ms/req)
Request size [B]: 65.0
Reply rate [replies/s]: min 9.2 avg 9.9 max 10.0 stddev 0.3 (10 samples)
Reply time [ms]: response 88.1 transfer 302.9
Reply size [B]: header 274.0 content 54744.0 footer 2.0 (total 55020.0)
Reply status: 1xx=0 2xx=500 3xx=0 4xx=0 5xx=0
CPU time [s]: user 15.65 system 34.65 (user 31.1% system 68.8% total 99.9%)
Net I/O: 534.1 KB/s (4.4*10^6 bps)
Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0
The 3 arguments I specified on the command line are:
* server: the name or IP address of your Web site (you can also specify a particular URL via the --uri argument)
* rate: specifies the number of HTTP requests/second sent to the Web server -- indicates the number of concurrent clients accessing the server
* num-conns: specifies how many total HTTP connections will be made during the test run -- this is a cumulative number, so the higher the number of connections, the longer the test run
Here is a detailed interpretation of an httperf test run. In short, the main numbers to look for are the connection rate, the request rate and the reply rate. Ideally, you would like to see that all these numbers are very close to the request rate specified on the command line. If the actual request rate and the reply rate start to decline, that's a sign your server became saturated and can't handle any new connections. That could also be a sign that your client became saturated, so that's why it's better to test your client against a fast Web site in order to gauge how many outgoing HTTP requests can be sustained by your client.
Autobench is a simple Perl script that facilitates multiple runs of httperf and automatically increases the HTTP request rate. Configuration of autobench can be achieved for example by means of the ~/.autobench.conf file. Here is how my file looks like:
# Autobench Configuration File
# host1, host2
# The hostnames of the servers under test
# Eg. host1 = iis.test.com
# host2 = apache.test.com
host1 = testhost1
host2 = testhost2
# uri1, uri2
# The URI to test (relative to the document root). For a fair comparison
# the files should be identical (although the paths to them may differ on the
# different hosts)
uri1 = /
uri2 = /
# port1, port2
# The port number on which the servers are listening
port1 = 80
port2 = 80
# low_rate, high_rate, rate_step
# The 'rate' is the number of number of connections to open per second.
# A series of tests will be conducted, starting at low rate,
# increasing by rate step, and finishing at high_rate.
# The default settings test at rates of 20,30,40,50...180,190,200
low_rate = 10
high_rate = 50
rate_step = 10
# num_conn, num_call
# num_conn is the total number of connections to make during a test
# num_call is the number of requests per connection
# The product of num_call and rate is the the approximate number of
# requests per second that will be attempted.
num_conn = 200
#num_call = 10
num_call = 1
# timeout sets the maximimum time (in seconds) that httperf will wait
# for replies from the web server. If the timeout is exceeded, the
# reply concerned is counted as an error.
timeout = 60
# output_fmt
# sets the output type - may be either "csv", or "tsv";
output_fmt = csv
## Config for distributed autobench (autobench_admin)
# clients
# comma separated list of the hostnames and portnumbers for the
# autobench clients. No whitespace can appear before or after the commas.
# clients = bench1.foo.com:4600,bench2.foo.com:4600,bench3.foo.com:4600
clients = localhost:4600
The only variable I usually tweak from one test run to another is num_conn, which I set to the desired number of total HTTP connections to the server for that test run. In the example file above it is set to 200.
I changed the default num_call value from 10 to 1 (num_call specifies the number of HTTP requests per connection; I like to set it to 1 to keep things simple). I started my test runs with low_rate set to 10, high_rate set to 50 and rate_step set to 10. What this means is that autobench will run httperf 5 times, starting with 10 requests/sec and going up to 50 requests/sec in increments of 10.
When running the following command line...
# autobench --single_host --host1=www.example.com --file=example.com.csv
* httperf is a benchmarking tool that measures the HTTP request throughput of a web server. The way it achieves this is by sending requests to the server at a fixed rate and measuring the rate at which replies arrive. Running the test several times and with monotonically increasing request rates, one can see the reply rate level off when the server becomes saturated, i.e., when it is operating at its full capacity.
* autobench is a Perl wrapper around httperf. It runs httperf a number of times against a Web server, increasing the number of requested connections per second on each iteration, and extracts the significant data from the httperf output, delivering a CSV format file which can be imported directly into a spreadsheet for analysis/graphing.
I ran a series of autobench/httperf and openload tests against a Web site I'll call site2 in the following discussion (site2 is a beta version of a site I'll call site1). For comparison purposes, I also ran similar tests against site1 and against www.example.com. The machine I ran the tests from is a Red Hat 9 Linux server co-located in downtown Los Angeles.
Here is an example of running httperf against www.example.com:
# httperf --server=www.example.com --rate=10 --num-conns=500
httperf --client=0/1 --server=www.example.com --port=80 --uri=/ --rate=10 --send-buffer=4096 --recv-buffer=16384 --num-conns=500 --num-calls=1
Maximum connect burst length: 1
Total: connections 500 requests 500 replies 500 test-duration 50.354 s
Connection rate: 9.9 conn/s (100.7 ms/conn, <=8 concurrent connections)
Connection time [ms]: min 449.7 avg 465.1 max 2856.6 median 451.5 stddev 132.1
Connection time [ms]: connect 74.1
Connection length [replies/conn]: 1.000
Request rate: 9.9 req/s (100.7 ms/req)
Request size [B]: 65.0
Reply rate [replies/s]: min 9.2 avg 9.9 max 10.0 stddev 0.3 (10 samples)
Reply time [ms]: response 88.1 transfer 302.9
Reply size [B]: header 274.0 content 54744.0 footer 2.0 (total 55020.0)
Reply status: 1xx=0 2xx=500 3xx=0 4xx=0 5xx=0
CPU time [s]: user 15.65 system 34.65 (user 31.1% system 68.8% total 99.9%)
Net I/O: 534.1 KB/s (4.4*10^6 bps)
Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0
The 3 arguments I specified on the command line are:
* server: the name or IP address of your Web site (you can also specify a particular URL via the --uri argument)
* rate: specifies the number of HTTP requests/second sent to the Web server -- indicates the number of concurrent clients accessing the server
* num-conns: specifies how many total HTTP connections will be made during the test run -- this is a cumulative number, so the higher the number of connections, the longer the test run
Here is a detailed interpretation of an httperf test run. In short, the main numbers to look for are the connection rate, the request rate and the reply rate. Ideally, you would like to see that all these numbers are very close to the request rate specified on the command line. If the actual request rate and the reply rate start to decline, that's a sign your server became saturated and can't handle any new connections. That could also be a sign that your client became saturated, so that's why it's better to test your client against a fast Web site in order to gauge how many outgoing HTTP requests can be sustained by your client.
Autobench is a simple Perl script that facilitates multiple runs of httperf and automatically increases the HTTP request rate. Configuration of autobench can be achieved for example by means of the ~/.autobench.conf file. Here is how my file looks like:
# Autobench Configuration File
# host1, host2
# The hostnames of the servers under test
# Eg. host1 = iis.test.com
# host2 = apache.test.com
host1 = testhost1
host2 = testhost2
# uri1, uri2
# The URI to test (relative to the document root). For a fair comparison
# the files should be identical (although the paths to them may differ on the
# different hosts)
uri1 = /
uri2 = /
# port1, port2
# The port number on which the servers are listening
port1 = 80
port2 = 80
# low_rate, high_rate, rate_step
# The 'rate' is the number of number of connections to open per second.
# A series of tests will be conducted, starting at low rate,
# increasing by rate step, and finishing at high_rate.
# The default settings test at rates of 20,30,40,50...180,190,200
low_rate = 10
high_rate = 50
rate_step = 10
# num_conn, num_call
# num_conn is the total number of connections to make during a test
# num_call is the number of requests per connection
# The product of num_call and rate is the the approximate number of
# requests per second that will be attempted.
num_conn = 200
#num_call = 10
num_call = 1
# timeout sets the maximimum time (in seconds) that httperf will wait
# for replies from the web server. If the timeout is exceeded, the
# reply concerned is counted as an error.
timeout = 60
# output_fmt
# sets the output type - may be either "csv", or "tsv";
output_fmt = csv
## Config for distributed autobench (autobench_admin)
# clients
# comma separated list of the hostnames and portnumbers for the
# autobench clients. No whitespace can appear before or after the commas.
# clients = bench1.foo.com:4600,bench2.foo.com:4600,bench3.foo.com:4600
clients = localhost:4600
The only variable I usually tweak from one test run to another is num_conn, which I set to the desired number of total HTTP connections to the server for that test run. In the example file above it is set to 200.
I changed the default num_call value from 10 to 1 (num_call specifies the number of HTTP requests per connection; I like to set it to 1 to keep things simple). I started my test runs with low_rate set to 10, high_rate set to 50 and rate_step set to 10. What this means is that autobench will run httperf 5 times, starting with 10 requests/sec and going up to 50 requests/sec in increments of 10.
When running the following command line...
# autobench --single_host --host1=www.example.com --file=example.com.csv
Wednesday, August 08, 2007
Source Code Beautifier for C, C++, C#, D, Java, and Pawn
A must to use !
http://uncrustify.sourceforge.net/
http://uncrustify.sourceforge.net/
how to convert IIS log to apache log using perl
# http://www.jammed.com/~jwa/hacks/iis2apache/iis2apache
# 1. Go to Start -> Control Panel -> Administrative Tools
# 2. Run Internet Information Services (IIS).
# 3. Find your Web site under the tree on the left.
# 4. Right-click on it and choose Properties.
# 5. On the Web site tab, you will see an option near the bottom that says "Active #Log Format." Click on the Properties button.
# 6. At the bottom of the General Properties tab, you will see a box that contains #the log file directory and the log file name. The full log path is comprised of the #log file directory plus the first part of the log file name.
#
#For example, if the dialog box displayed the following values:
#
# * Log file directory: C:\Windows\System32\LogFiles
# * Log file name: W3SVC1\exyymmdd.log
#Then your full log path would be:
#C:\Windows\System32\LogFiles\W3SVC1
#!/usr/bin/perl
#
# make an IIS log look strikingly like an apache log
# we make a vague attempt to interpret the Fields: header
# in an IIS logfile and use that to make IIS fields match up
# with apache fields.
#
# jwa@jammed.com 12 Dec 2000
#
while ($arg = shift @ARGV) {
$tzoffset = shift @ARGV if ($arg eq "--faketz");
$vhost = shift @ARGV if ($arg eq "--vhost");
$debug = 1 if ($arg eq "--debug"); # show field interpretation
}
if ($tzoffset eq "") {
print STDERR "Will use -0000 as a fake tzoffset\n";
$tzoffset = "-0000";
}
# build month hash
@m = ('Jan','Feb','Mar','Apr','May','Jun','Jul','Aug','Sep','Oct','Nov','Dec');
while ($m = shift @m) {
$month{++$n} = $m;
}
# an IIS log adheres to what's defined by 'Fields;'
# attempt to parse this, tagging
LINE:
while ($line =) {
$line =~ s/\r|\n//g; # cooky DOS format
if ($line =~ /^#Fields: /) {
@line = split(" ", $line);
shift @line; # shifts of #Fields
# build a hash so we can look up a fieldname and
# have it return a position in the string
undef %fieldh;
$n = 0; # zero-based array for split
while ($l = shift @line) {
$fieldh{$l} = $n++;
print STDERR "$l is position $fieldh{$l}\n" if ($debug);
}
}
next LINE if ($line =~ /^#/);
#Fields: date time c-ip cs-username s-sitename s-computername s-ip cs-method cs-
#uri-stem cs-uri-query sc-status sc-win32-status sc-bytes cs-bytes time-taken
#s-port cs-version cs(User-Agent) cs(Cookie) cs(Referer)
# this is really slow.
$date = yankfield("date");
$time = yankfield("time");
$ip = yankfield("c-ip");
$username = yankfield("cs-username");
$method = yankfield("cs-method");
$stem = yankfield("cs-uri-stem");
$query = yankfield("cs-uri-query");
$status = yankfield("sc-status");
#$bytes = yankfield("cs-bytes");
# Which is it? sc-bytes or cs-bytes?
# cs-bytes only appears in some of the IIS logs I've seen.
# I'll assume that sc-bytes is "server->client bytes", which
# is what we want anyway.
$bytes = yankfield("sc-bytes"); # I'm gonna go with this.
$useragent = yankfield("cs(User-Agent)");
$referer = yankfield("cs(Referer)");
$useragent =~ s/\+/\ /g;
# our modified CLF sez:
# IP - - [DD/MMM/YYYY:HH:MM:SS TZOFFSET] "method stem[?query]" status bytes "referer" "user agent" "vhost"
# convert date
# 2000-07-19 00:00:01
($y, $m, $d) = split("-", $date);
$m =~ s/^0//g;
$mname = $month{$m};
# build url
$url = $stem;
if ($query ne "-") {
$url .= "?$query";
}
# all done, print it out
print "$ip - - [${d}/${mname}/${y}:${time} ${tzoffset}] \"$method $url\" $status $bytes \"$referer\" \"$useragent\" \"${vhost}\"\n";
}
# return the proper field, or "-" if it's not defined.
# (unfortunately ($date) = (split(" ", $line))[$fieldh{date}];
# will return element 0 if $fieldh{date} is undefined . . .)
sub yankfield {
my ($field) = shift @_;
print STDERR "Looking at $field; position [$fieldh{$field}]\n" if ($debug);
if ($fieldh{$field} ne "") {
return (split(" ", $line))[$fieldh{$field}];
} else {
print STDERR "$field undefined\n" if ($debug);
return "-";
}
}
# 1. Go to Start -> Control Panel -> Administrative Tools
# 2. Run Internet Information Services (IIS).
# 3. Find your Web site under the tree on the left.
# 4. Right-click on it and choose Properties.
# 5. On the Web site tab, you will see an option near the bottom that says "Active #Log Format." Click on the Properties button.
# 6. At the bottom of the General Properties tab, you will see a box that contains #the log file directory and the log file name. The full log path is comprised of the #log file directory plus the first part of the log file name.
#
#For example, if the dialog box displayed the following values:
#
# * Log file directory: C:\Windows\System32\LogFiles
# * Log file name: W3SVC1\exyymmdd.log
#Then your full log path would be:
#C:\Windows\System32\LogFiles\W3SVC1
#!/usr/bin/perl
#
# make an IIS log look strikingly like an apache log
# we make a vague attempt to interpret the Fields: header
# in an IIS logfile and use that to make IIS fields match up
# with apache fields.
#
# jwa@jammed.com 12 Dec 2000
#
while ($arg = shift @ARGV) {
$tzoffset = shift @ARGV if ($arg eq "--faketz");
$vhost = shift @ARGV if ($arg eq "--vhost");
$debug = 1 if ($arg eq "--debug"); # show field interpretation
}
if ($tzoffset eq "") {
print STDERR "Will use -0000 as a fake tzoffset\n";
$tzoffset = "-0000";
}
# build month hash
@m = ('Jan','Feb','Mar','Apr','May','Jun','Jul','Aug','Sep','Oct','Nov','Dec');
while ($m = shift @m) {
$month{++$n} = $m;
}
# an IIS log adheres to what's defined by 'Fields;'
# attempt to parse this, tagging
LINE:
while ($line =
$line =~ s/\r|\n//g; # cooky DOS format
if ($line =~ /^#Fields: /) {
@line = split(" ", $line);
shift @line; # shifts of #Fields
# build a hash so we can look up a fieldname and
# have it return a position in the string
undef %fieldh;
$n = 0; # zero-based array for split
while ($l = shift @line) {
$fieldh{$l} = $n++;
print STDERR "$l is position $fieldh{$l}\n" if ($debug);
}
}
next LINE if ($line =~ /^#/);
#Fields: date time c-ip cs-username s-sitename s-computername s-ip cs-method cs-
#uri-stem cs-uri-query sc-status sc-win32-status sc-bytes cs-bytes time-taken
#s-port cs-version cs(User-Agent) cs(Cookie) cs(Referer)
# this is really slow.
$date = yankfield("date");
$time = yankfield("time");
$ip = yankfield("c-ip");
$username = yankfield("cs-username");
$method = yankfield("cs-method");
$stem = yankfield("cs-uri-stem");
$query = yankfield("cs-uri-query");
$status = yankfield("sc-status");
#$bytes = yankfield("cs-bytes");
# Which is it? sc-bytes or cs-bytes?
# cs-bytes only appears in some of the IIS logs I've seen.
# I'll assume that sc-bytes is "server->client bytes", which
# is what we want anyway.
$bytes = yankfield("sc-bytes"); # I'm gonna go with this.
$useragent = yankfield("cs(User-Agent)");
$referer = yankfield("cs(Referer)");
$useragent =~ s/\+/\ /g;
# our modified CLF sez:
# IP - - [DD/MMM/YYYY:HH:MM:SS TZOFFSET] "method stem[?query]" status bytes "referer" "user agent" "vhost"
# convert date
# 2000-07-19 00:00:01
($y, $m, $d) = split("-", $date);
$m =~ s/^0//g;
$mname = $month{$m};
# build url
$url = $stem;
if ($query ne "-") {
$url .= "?$query";
}
# all done, print it out
print "$ip - - [${d}/${mname}/${y}:${time} ${tzoffset}] \"$method $url\" $status $bytes \"$referer\" \"$useragent\" \"${vhost}\"\n";
}
# return the proper field, or "-" if it's not defined.
# (unfortunately ($date) = (split(" ", $line))[$fieldh{date}];
# will return element 0 if $fieldh{date} is undefined . . .)
sub yankfield {
my ($field) = shift @_;
print STDERR "Looking at $field; position [$fieldh{$field}]\n" if ($debug);
if ($fieldh{$field} ne "") {
return (split(" ", $line))[$fieldh{$field}];
} else {
print STDERR "$field undefined\n" if ($debug);
return "-";
}
}
Tuesday, July 17, 2007
Microsoft PhotoTours
This is an amazing piece of technology, presented at a TED conference.
http://www.collegehumor.com/video:1762315
More information is available on the Microsoft Research Website.
http://research.microsoft.com/IVM/PhotoTours/
There is also a live preview of this software available online. All you need to do is install a 5.5 MB Photosynth ActiveX. The address is :
http://labs.live.com/photosynth/
http://www.crazyleafdesign.com/blog/photosynth-prototype/
http://www.collegehumor.com/video:1762315
More information is available on the Microsoft Research Website.
http://research.microsoft.com/IVM/PhotoTours/
There is also a live preview of this software available online. All you need to do is install a 5.5 MB Photosynth ActiveX. The address is :
http://labs.live.com/photosynth/
http://www.crazyleafdesign.com/blog/photosynth-prototype/
Friday, July 13, 2007
Peter Löthberg mother's Sigbritt, 75, has world's fastest broadband
Sigbritt, 75, has world's fastest broadband
Published: 12th July 2007 11:07 CET
Online: http://www.thelocal.se/7869/
A 75 year old woman from Karlstad in central Sweden has been thrust into the IT history books - with the world's fastest internet connection.
Sigbritt Löthberg's home has been supplied with a blistering 40 Gigabits per second connection, many thousands of times faster than the average residential link and the first time ever that a home user has experienced such a high speed.
But Sigbritt, who had never had a computer until now, is no ordinary 75 year old. She is the mother of Swedish internet legend Peter Löthberg who, along with Karlstad Stadsnät, the local council's network arm, has arranged the connection.
"This is more than just a demonstration," said network boss Hafsteinn Jonsson.
"As a network owner we're trying to persuade internet operators to invest in faster connections. And Peter Löthberg wanted to show how you can build a low price, high capacity line over long distances," he told The Local.
Sigbritt will now be able to enjoy 1,500 high definition HDTV channels simultaneously. Or, if there is nothing worth watching there, she will be able to download a full high definition DVD in just two seconds.
The secret behind Sigbritt's ultra-fast connection is a new modulation technique which allows data to be transferred directly between two routers up to 2,000 kilometres apart, with no intermediary transponders.
According to Karlstad Stadsnät the distance is, in theory, unlimited - there is no data loss as long as the fibre is in place.
"I want to show that there are other methods than the old fashioned ways such as copper wires and radio, which lack the possibilities that fibre has," said Peter Löthberg, who now works at Cisco.
Cisco contributed to the project but the point, said Hafsteinn Jonsson, is that fibre technology makes such high speed connections technically and commercially viable.
"The most difficult part of the whole project was installing Windows on Sigbritt's PC," said Jonsson.
The Local (news@thelocal.se/08 656 6518)
Published: 12th July 2007 11:07 CET
Online: http://www.thelocal.se/7869/
A 75 year old woman from Karlstad in central Sweden has been thrust into the IT history books - with the world's fastest internet connection.
Sigbritt Löthberg's home has been supplied with a blistering 40 Gigabits per second connection, many thousands of times faster than the average residential link and the first time ever that a home user has experienced such a high speed.
But Sigbritt, who had never had a computer until now, is no ordinary 75 year old. She is the mother of Swedish internet legend Peter Löthberg who, along with Karlstad Stadsnät, the local council's network arm, has arranged the connection.
"This is more than just a demonstration," said network boss Hafsteinn Jonsson.
"As a network owner we're trying to persuade internet operators to invest in faster connections. And Peter Löthberg wanted to show how you can build a low price, high capacity line over long distances," he told The Local.
Sigbritt will now be able to enjoy 1,500 high definition HDTV channels simultaneously. Or, if there is nothing worth watching there, she will be able to download a full high definition DVD in just two seconds.
The secret behind Sigbritt's ultra-fast connection is a new modulation technique which allows data to be transferred directly between two routers up to 2,000 kilometres apart, with no intermediary transponders.
According to Karlstad Stadsnät the distance is, in theory, unlimited - there is no data loss as long as the fibre is in place.
"I want to show that there are other methods than the old fashioned ways such as copper wires and radio, which lack the possibilities that fibre has," said Peter Löthberg, who now works at Cisco.
Cisco contributed to the project but the point, said Hafsteinn Jonsson, is that fibre technology makes such high speed connections technically and commercially viable.
"The most difficult part of the whole project was installing Windows on Sigbritt's PC," said Jonsson.
The Local (news@thelocal.se/08 656 6518)
Monday, July 09, 2007
chown on linux using c++
lchown
NAME
lchown - change the owner and group of a symbolic link
#include
int lchown(const char *path, uid_t owner, gid_t group);
DESCRIPTION
The lchown() function shall be equivalent to chown(), except in the case where the named file is a symbolic link. In this case, lchown() shall change the ownership of the symbolic link file itself, while chown() changes the ownership of the file or directory to which the symbolic link refers.
RETURN VALUE
Upon successful completion, lchown() shall return 0. Otherwise, it shall return -1 and set errno to indicate an error.
ERRORS
The lchown() function shall fail if:
EACCES
Search permission is denied on a component of the path prefix of path.
EINVAL
The owner or group ID is not a value supported by the implementation.
ELOOP
A loop exists in symbolic links encountered during resolution of the path argument.
ENAMETOOLONG
The length of a pathname exceeds {PATH_MAX} or a pathname component is longer than {NAME_MAX}.
ENOENT
A component of path does not name an existing file or path is an empty string.
ENOTDIR
A component of the path prefix of path is not a directory.
EOPNOTSUPP
The path argument names a symbolic link and the implementation does not support setting the owner or group of a symbolic link.
EPERM
The effective user ID does not match the owner of the file and the process does not have appropriate privileges.
EROFS
The file resides on a read-only file system.
The lchown() function may fail if:
EIO
An I/O error occurred while reading or writing to the file system.
EINTR
A signal was caught during execution of the function.
ELOOP
More than {SYMLOOP_MAX} symbolic links were encountered during resolution of the path argument.
ENAMETOOLONG
Pathname resolution of a symbolic link produced an intermediate result whose length exceeds {PATH_MAX}.
The following sections are informative.
EXAMPLES
Changing the Current Owner of a File
The following example shows how to change the ownership of the symbolic link named /modules/pass1 to the user ID associated with "jones" and the group ID associated with "cnd".
The numeric value for the user ID is obtained by using the getpwnam() function. The numeric value for the group ID is obtained by using the getgrnam() function.
#include
#include
#include
#include
struct passwd *pwd;
struct group *grp;
char *path = "/modules/pass1";
...
pwd = getpwnam("Sri_Chinmoy");
grp = getgrnam("guru");
lchown(path, pwd->pw_uid, grp->gr_gid);
NAME
lchown - change the owner and group of a symbolic link
#include
int lchown(const char *path, uid_t owner, gid_t group);
DESCRIPTION
The lchown() function shall be equivalent to chown(), except in the case where the named file is a symbolic link. In this case, lchown() shall change the ownership of the symbolic link file itself, while chown() changes the ownership of the file or directory to which the symbolic link refers.
RETURN VALUE
Upon successful completion, lchown() shall return 0. Otherwise, it shall return -1 and set errno to indicate an error.
ERRORS
The lchown() function shall fail if:
EACCES
Search permission is denied on a component of the path prefix of path.
EINVAL
The owner or group ID is not a value supported by the implementation.
ELOOP
A loop exists in symbolic links encountered during resolution of the path argument.
ENAMETOOLONG
The length of a pathname exceeds {PATH_MAX} or a pathname component is longer than {NAME_MAX}.
ENOENT
A component of path does not name an existing file or path is an empty string.
ENOTDIR
A component of the path prefix of path is not a directory.
EOPNOTSUPP
The path argument names a symbolic link and the implementation does not support setting the owner or group of a symbolic link.
EPERM
The effective user ID does not match the owner of the file and the process does not have appropriate privileges.
EROFS
The file resides on a read-only file system.
The lchown() function may fail if:
EIO
An I/O error occurred while reading or writing to the file system.
EINTR
A signal was caught during execution of the function.
ELOOP
More than {SYMLOOP_MAX} symbolic links were encountered during resolution of the path argument.
ENAMETOOLONG
Pathname resolution of a symbolic link produced an intermediate result whose length exceeds {PATH_MAX}.
The following sections are informative.
EXAMPLES
Changing the Current Owner of a File
The following example shows how to change the ownership of the symbolic link named /modules/pass1 to the user ID associated with "jones" and the group ID associated with "cnd".
The numeric value for the user ID is obtained by using the getpwnam() function. The numeric value for the group ID is obtained by using the getgrnam() function.
#include
#include
#include
#include
struct passwd *pwd;
struct group *grp;
char *path = "/modules/pass1";
...
pwd = getpwnam("Sri_Chinmoy");
grp = getgrnam("guru");
lchown(path, pwd->pw_uid, grp->gr_gid);
Advanced Unix Programming Source
A set of handy function and class to perform some basics
UNIX tasks in c++
http://basepath.com/aup/ex/index.html
UNIX tasks in c++
http://basepath.com/aup/ex/index.html
Thursday, July 05, 2007
Thread POSIX Thread basic
Further Threads Programming:Thread Attributes (POSIX)
PART I
API covered is for POSIX threads only. Otherwise, the functionality for Solaris threads and pthreads is largely the same.
Attributes
Attributes are a way to specify behavior that is different from the default. When a thread is created with pthread_create() or when a synchronization variable is initialized, an attribute object can be specified. Note: however that the default atributes are usually sufficient for most applications.
Important Note: Attributes are specified only at thread creation time; they cannot be altered while the thread is being used.
Thus three functions are usually called in tandem
* Thread attibute intialisation -- pthread_attr_init() create a default pthread_attr_t tattr
* Thread attribute value change (unless defaults appropriate) -- a variety of pthread_attr_*() functions are available to set individual attribute values for the pthread_attr_t tattr structure. (see below).
* Thread creation -- a call to pthread_create() with approriate attribute values set in a pthread_attr_t tattr structure.
The following code fragment should make this point clearer:
#include
pthread_attr_t tattr;
pthread_t tid;
void *start_routine;
void arg
int ret;
/* initialized with default attributes */
ret = pthread_attr_init(&tattr);
/* call an appropriate functions to alter a default value */
ret = pthread_attr_*(&tattr,SOME_ATRIBUTE_VALUE_PARAMETER);
/* create the thread */
ret = pthread_create(&tid, &tattr, start_routine, arg);
In order to save space, code examples mainly focus on the attribute setting functions and the intializing and creation functions are ommitted. These must of course be present in all actual code fragtments.
An attribute object is opaque, and cannot be directly modified by assignments. A set of functions is provided to initialize, configure, and destroy each object type. Once an attribute is initialized and configured, it has process-wide scope. The suggested method for using attributes is to configure all required state specifications at one time in the early stages of program execution. The appropriate attribute object can then be referred to as needed. Using attribute objects has two primary advantages:
* First, it adds to code portability. Even though supported attributes might vary between implementations, you need not modify function calls that create thread entities because the attribute object is hidden from the interface. If the target port supports attributes that are not found in the current port, provision must be made to manage the new attributes. This is an easy porting task though, because attribute objects need only be initialized once in a well-defined location.
* Second, state specification in an application is simplified. As an example, consider that several sets of threads might exist within a process, each providing a separate service, and each with its own state requirements. At some point in the early stages of the application, a thread attribute object can be initialized for each set. All future thread creations will then refer to the attribute object initialized for that type of thread. The initialization phase is simple and localized, and any future modifications can be made quickly and reliably.
Attribute objects require attention at process exit time. When the object is initialized, memory is allocated for it. This memory must be returned to the system. The pthreads standard provides function calls to destroy attribute objects.
Initializing Thread Attributes
The function pthread_attr_init() is used to initialize object attributes to their default values. The storage is allocated by the thread system during execution.
The function is prototyped by:
int pthread_attr_init(pthread_attr_t *tattr);
An example call to this function is:
#include
pthread_attr_t tattr;
int ret;
/* initialize an attribute to the default value */
ret = pthread_attr_init(&tattr);
The default values for attributes (tattr) are:
Attribute Value Result
scope PTHREAD_SCOPE_PROCESS New thread is unbound - not permanently attached to LWP.
detachstate PTHREAD_CREATE_JOINABLE Exit status and thread are preserved after the thread
terminates.
stackaddr NULL New thread has system-allocated stack address.
stacksize 1 megabyte New thread has system-defined stack size.
priority New thread inherits parent thread priority.
inheritsched PTHREAD_INHERIT_SCHED New thread inherits parent thread scheduling priority.
schedpolicy SCHED_OTHER New thread uses Solaris-defined fixed priority scheduling;
threads run until preempted by a higher-priority thread or until they block or yield.
This function zero after completing successfully. Any other returned value indicates that an error occurred. If the following condition occurs, the function fails and returns an error value (to errno).
Destroying Thread Attributes
The function pthread_attr_destroy() is used to remove the storage allocated during initialization. The attribute object becomes invalid. It is prototyped by:
int pthread_attr_destroy(pthread_attr_t *tattr);
A sample call to this functions is:
#include
pthread_attr_t tattr;
int ret;
/* destroy an attribute */
ret = pthread_attr_destroy(&tattr);
Attribites are declared as for pthread_attr_init() above.
pthread_attr_destroy() returns zero after completing successfully. Any other returned value indicates that an error occurred.
Thread's Detach State
When a thread is created detached (PTHREAD_CREATE_DETACHED), its thread ID and other resources can be reused as soon as the thread terminates.
If you do not want the calling thread to wait for the thread to terminate then call the function pthread_attr_setdetachstate().
When a thread is created nondetached (PTHREAD_CREATE_JOINABLE), it is assumed that you will be waiting for it. That is, it is assumed that you will be executing a pthread_join() on the thread. Whether a thread is created detached or nondetached, the process does not exit until all threads have exited.
pthread_attr_setdetachstate() is prototyped by:
int pthread_attr_setdetachstate(pthread_attr_t *tattr,int detachstate);
pthread_attr_setdetachstate() returns zero after completing successfully. Any other returned value indicates that an error occurred. If the following condition occurs, the function fails and returns the corresponding value.
An example call to detatch a thread with this function is:
#include
pthread_attr_t tattr;
int ret;
/* set the thread detach state */
ret = pthread_attr_setdetachstate(&tattr,PTHREAD_CREATE_DETACHED);
Note - When there is no explicit synchronization to prevent it, a newly created, detached thread can die and have its thread ID reassigned to another new thread before its creator returns from pthread_create(). For nondetached (PTHREAD_CREATE_JOINABLE) threads, it is very important that some thread join with it after it terminates -- otherwise the resources of that thread are not released for use by new threads. This commonly results in a memory leak. So when you do not want a thread to be joined, create it as a detached thread.
It is quite common that you will wish to create a thread which is detatched from creation. The following code illustrates how this may be achieved with the standard calls to initialise and set and then create a thread:
#include
pthread_attr_t tattr;
pthread_t tid;
void *start_routine;
void arg
int ret;
/* initialized with default attributes */
ret = pthread_attr_init(&tattr);
ret = pthread_attr_setdetachstate(&tattr,PTHREAD_CREATE_DETACHED);
ret = pthread_create(&tid, &tattr, start_routine, arg);
The function pthread_attr_getdetachstate() may be used to retrieve the thread create state, which can be either detached or joined. It is prototyped by:
int pthread_attr_getdetachstate(const pthread_attr_t *tattr, int *detachstate);
pthread_attr_getdetachstate() returns zero after completing successfully. Any other returned value indicates that an error occurred.
An example call to this fuction is:
#include
pthread_attr_t tattr;
int detachstate;
int ret;
/* get detachstate of thread */
ret = pthread_attr_getdetachstate (&tattr, &detachstate);
Full Article here
http://www.cs.cf.ac.uk/Dave/C/node30.html
PART II
Meanwhile, the second thread will do something like this:
Assuming both threads use the same mutex, we are assured that after they both ran through this code, either both variables are set to '0', or both are set to '1'. You'd note this requires some work from the programmer - If a third thread was to access these variables via some code that does not use this mutex, it still might mess up the variable's contents. Thus, it is important to enclose all the code that accesses these variables in a small set of functions, and always use only these functions to access these variables.
One note should be made here: This type of initialization creates a mutex called 'fast mutex'. This means that if a thread locks the mutex and then tries to lock it again, it'll get stuck - it will be in a deadlock.
There is another type of mutex, called 'recursive mutex', which allows the thread that locked it, to lock it several more times, without getting blocked (but other threads that try to lock the mutex now will get blocked). If the thread then unlocks the mutex, it'll still be locked, until it is unlocked the same amount of times as it was locked. This is similar to the way modern door locks work - if you turned it twice clockwise to lock it, you need to turn it twice counter-clockwise to unlock it. This kind of mutex can be created by assigning the constant
After the thread did what it had to (change variables or data structures, handle file, or whatever it intended to do), it should free the mutex, using the
After this call, this variable (a_mutex) may not be used as a mutex any more, unless it is initialized again. Thus, if one destroys a mutex too early, and another thread tries to lock or unlock it, that thread will get a
Two copies of the program are supplied. One that uses a mutex, and one that does not. Try them both, to see the differences, and be convinced that mutexes are essential in a multi-threaded environment.
The programs themselves are in the files accompanying this tutorial. The one that uses a mutex is employee-with-mutex.c. The one that does not use a mutex is employee-without-mutex.c. Read the comments inside the source files to get a better understanding of how they work.
Again we should remember that
The pthread library might, however, figure out a "deadlock". A deadlock is a situation in which a set of threads are all waiting for resources taken by other threads, all in the same set. Naturally, if all threads are blocked waiting for a mutex, none of them will ever come back to life again. The pthread library keeps track of such situations, and thus would fail the last thread trying to call
All these examples require the ability to send notifications between threads. This is where condition variables are brought into the picture.
Note that a condition variable does not provide locking. Thus, a mutex is used along with the condition variable, to provide the necessary locking when accessing this condition variable.
This defines a condition variable named 'got_request', and initializes it.
Note: since the
Or by using the broadcast function:
When either function returns, 'rc' is set to 0 on success, and to a non-zero value on failure. In such a case (failure), the return value denotes the error that occured (
Note: success of a signaling operation does not mean any thread was awakened - it might be that no thread was waiting on the condition variable, and thus the signaling does nothing (i.e. the signal is lost).
It is also not remembered for future use - if after the signaling function returns another thread starts waiting on this condition variable, a further signal is required to wake it up.
The only difference between these two functions is that
Here is how to use these two functions. We make the assumption that 'got_request' is a properly initialized condition variable, and that 'request_mutex' is a properly initialized mutex. First, we try the
Now an example using the
As you can see, the timed wait version is way more complex, and thus better be wrapped up by some function, rather then being re-coded in every necessary location.
Note: it might be that a condition variable that has 2 or more threads waiting on it is signaled many times, and yet one of the threads waiting on it never awakened. This is because we are not guaranteed which of the waiting threads is awakened when the variable is signaled. It might be that the awakened thread quickly comes back to waiting on the condition variables, and gets awakened again when the variable is signaled again, and so on. The situation for the un-awakened thread is called 'starvation'. It is up to the programmer to make sure this situation does not occur if it implies bad behavior. Yet, in our server example from before, this situation might indicate requests are coming in a very slow pace, and thus perhaps we have too many threads waiting to service requests. In this case, this situation is actually good, as it means every request is handled immediately when it arrives.
Note 2: when the mutex is being broadcast (using pthread_cond_broadcast), this does not mean all threads are running together. Each of them tries to lock the mutex again before returning from their wait function, and thus they'll start running one by one, each one locking the mutex, doing their work, and freeing the mutex before the next thread gets its chance to run.
What if some thread is still waiting on this variable? depending on the case, it might imply some flaw in the usage of this variable, or just lack of proper thread cleanup code. It is probably good to alert the programmer, at least during debug phase of the program, of such a case. It might mean nothing, but it might be significant.
However, what if all threads are busy handling previous requests, when a new one arrives? the signaling of the condition variable will do nothing (since all threads are busy doing other things, NOT waiting on the condition variable now), and after all threads finish handling their current request, they come back to wait on the variable, which won't necessarily be signaled again (for example, if no new requests arrive). Thus, there is at least one request pending, while all handling threads are blocked, waiting for a signal.
In order to overcome this problem, we may set some integer variable to denote the number of pending requests, and have each thread check the value of this variable before waiting on the variable. If this variable's value is positive, some request is pending, and the thread should go and handle it, instead of going to sleep. Further more, a thread that handled a request, should reduce the value of this variable by one, to make the count correct.
Lets see how this affects the waiting code we have seen above.
PART I
API covered is for POSIX threads only. Otherwise, the functionality for Solaris threads and pthreads is largely the same.
Attributes
Attributes are a way to specify behavior that is different from the default. When a thread is created with pthread_create() or when a synchronization variable is initialized, an attribute object can be specified. Note: however that the default atributes are usually sufficient for most applications.
Important Note: Attributes are specified only at thread creation time; they cannot be altered while the thread is being used.
Thus three functions are usually called in tandem
* Thread attibute intialisation -- pthread_attr_init() create a default pthread_attr_t tattr
* Thread attribute value change (unless defaults appropriate) -- a variety of pthread_attr_*() functions are available to set individual attribute values for the pthread_attr_t tattr structure. (see below).
* Thread creation -- a call to pthread_create() with approriate attribute values set in a pthread_attr_t tattr structure.
The following code fragment should make this point clearer:
#include
pthread_attr_t tattr;
pthread_t tid;
void *start_routine;
void arg
int ret;
/* initialized with default attributes */
ret = pthread_attr_init(&tattr);
/* call an appropriate functions to alter a default value */
ret = pthread_attr_*(&tattr,SOME_ATRIBUTE_VALUE_PARAMETER);
/* create the thread */
ret = pthread_create(&tid, &tattr, start_routine, arg);
In order to save space, code examples mainly focus on the attribute setting functions and the intializing and creation functions are ommitted. These must of course be present in all actual code fragtments.
An attribute object is opaque, and cannot be directly modified by assignments. A set of functions is provided to initialize, configure, and destroy each object type. Once an attribute is initialized and configured, it has process-wide scope. The suggested method for using attributes is to configure all required state specifications at one time in the early stages of program execution. The appropriate attribute object can then be referred to as needed. Using attribute objects has two primary advantages:
* First, it adds to code portability. Even though supported attributes might vary between implementations, you need not modify function calls that create thread entities because the attribute object is hidden from the interface. If the target port supports attributes that are not found in the current port, provision must be made to manage the new attributes. This is an easy porting task though, because attribute objects need only be initialized once in a well-defined location.
* Second, state specification in an application is simplified. As an example, consider that several sets of threads might exist within a process, each providing a separate service, and each with its own state requirements. At some point in the early stages of the application, a thread attribute object can be initialized for each set. All future thread creations will then refer to the attribute object initialized for that type of thread. The initialization phase is simple and localized, and any future modifications can be made quickly and reliably.
Attribute objects require attention at process exit time. When the object is initialized, memory is allocated for it. This memory must be returned to the system. The pthreads standard provides function calls to destroy attribute objects.
Initializing Thread Attributes
The function pthread_attr_init() is used to initialize object attributes to their default values. The storage is allocated by the thread system during execution.
The function is prototyped by:
int pthread_attr_init(pthread_attr_t *tattr);
An example call to this function is:
#include
pthread_attr_t tattr;
int ret;
/* initialize an attribute to the default value */
ret = pthread_attr_init(&tattr);
The default values for attributes (tattr) are:
Attribute Value Result
scope PTHREAD_SCOPE_PROCESS New thread is unbound - not permanently attached to LWP.
detachstate PTHREAD_CREATE_JOINABLE Exit status and thread are preserved after the thread
terminates.
stackaddr NULL New thread has system-allocated stack address.
priority New thread inherits parent thread priority.
inheritsched PTHREAD_INHERIT_SCHED New thread inherits parent thread scheduling priority.
schedpolicy SCHED_OTHER New thread uses Solaris-defined fixed priority scheduling;
threads run until preempted by a higher-priority thread or until they block or yield.
This function zero after completing successfully. Any other returned value indicates that an error occurred. If the following condition occurs, the function fails and returns an error value (to errno).
Destroying Thread Attributes
The function pthread_attr_destroy() is used to remove the storage allocated during initialization. The attribute object becomes invalid. It is prototyped by:
int pthread_attr_destroy(pthread_attr_t *tattr);
A sample call to this functions is:
#include
pthread_attr_t tattr;
int ret;
/* destroy an attribute */
ret = pthread_attr_destroy(&tattr);
Attribites are declared as for pthread_attr_init() above.
pthread_attr_destroy() returns zero after completing successfully. Any other returned value indicates that an error occurred.
Thread's Detach State
When a thread is created detached (PTHREAD_CREATE_DETACHED), its thread ID and other resources can be reused as soon as the thread terminates.
If you do not want the calling thread to wait for the thread to terminate then call the function pthread_attr_setdetachstate().
When a thread is created nondetached (PTHREAD_CREATE_JOINABLE), it is assumed that you will be waiting for it. That is, it is assumed that you will be executing a pthread_join() on the thread. Whether a thread is created detached or nondetached, the process does not exit until all threads have exited.
pthread_attr_setdetachstate() is prototyped by:
int pthread_attr_setdetachstate(pthread_attr_t *tattr,int detachstate);
pthread_attr_setdetachstate() returns zero after completing successfully. Any other returned value indicates that an error occurred. If the following condition occurs, the function fails and returns the corresponding value.
An example call to detatch a thread with this function is:
#include
pthread_attr_t tattr;
int ret;
/* set the thread detach state */
ret = pthread_attr_setdetachstate(&tattr,PTHREAD_CREATE_DETACHED);
Note - When there is no explicit synchronization to prevent it, a newly created, detached thread can die and have its thread ID reassigned to another new thread before its creator returns from pthread_create(). For nondetached (PTHREAD_CREATE_JOINABLE) threads, it is very important that some thread join with it after it terminates -- otherwise the resources of that thread are not released for use by new threads. This commonly results in a memory leak. So when you do not want a thread to be joined, create it as a detached thread.
It is quite common that you will wish to create a thread which is detatched from creation. The following code illustrates how this may be achieved with the standard calls to initialise and set and then create a thread:
#include
pthread_attr_t tattr;
pthread_t tid;
void *start_routine;
void arg
int ret;
/* initialized with default attributes */
ret = pthread_attr_init(&tattr);
ret = pthread_attr_setdetachstate(&tattr,PTHREAD_CREATE_DETACHED);
ret = pthread_create(&tid, &tattr, start_routine, arg);
The function pthread_attr_getdetachstate() may be used to retrieve the thread create state, which can be either detached or joined. It is prototyped by:
int pthread_attr_getdetachstate(const pthread_attr_t *tattr, int *detachstate);
pthread_attr_getdetachstate() returns zero after completing successfully. Any other returned value indicates that an error occurred.
An example call to this fuction is:
#include
pthread_attr_t tattr;
int detachstate;
int ret;
/* get detachstate of thread */
ret = pthread_attr_getdetachstate (&tattr, &detachstate);
Full Article here
http://www.cs.cf.ac.uk/Dave/C/node30.html
What Is a Thread? Why Use Threads
A thread is a semi-process, that has its own stack, and executes a given piece of code. Unlike a real process, the thread normally shares its memory with other threads (where as for processes we usually have a different memory area for each one of them). A Thread Group is a set of threads all executing inside the same process. They all share the same memory, and thus can access the same global variables, same heap memory, same set of file descriptors, etc. All these threads execute in parallel (i.e. using time slices, or if the system has several processors, then really in parallel).
The advantage of using a thread group instead of a normal serial program is that several operations may be carried out in parallel, and thus events can be handled immediately as they arrive (for example, if we have one thread handling a user interface, and another thread handling database queries, we can execute a heavy query requested by the user, and still respond to user input while the query is executed).
The advantage of using a thread group over using a process group is that context switching between threads is much faster then context switching between processes (context switching means that the system switches from running one thread or process, to running another thread or process). Also, communications between two threads is usually faster and easier to implement then communications between two processes.
On the other hand, because threads in a group all use the same memory space, if one of them corrupts the contents of its memory, other threads might suffer as well. With processes, the operating system normally protects processes from one another, and thus if one corrupts its own memory space, other processes won't suffer. Another advantage of using processes is that they can run on different machines, while all the threads have to run on the same machine (at least normally).
Creating And Destroying Threads
When a multi-threaded program starts executing, it has one thread running, which executes the main() function of the program. This is already a full-fledged thread, with its own thread ID. In order to create a new thread, the program should use the
pthread_create() function. Here is how to use it:
#include /* standard I/O routines */
#include /* pthread functions and data structures */
/* function to be executed by the new thread */
void*
do_loop(void* data)
{
int i;
int i; /* counter, to print numbers */
int j; /* counter, for delay */
int me = *((int*)data); /* thread identifying number */
for (i=0; i<10 color="brown" font="" for="" i="" j="">/* delay loop */10>
;
printf("'%d' - Got '%d'\n", me, i);
}
/* terminate the thread */
pthread_exit(NULL);
}
/* like any C program, program's execution begins in main */
int
main(int argc, char* argv[])
{
int thr_id; /* thread ID for the newly created thread */
pthread_t p_thread; /* thread's structure */
int a = 1; /* thread 1 identifying number */
int b = 2; /* thread 2 identifying number */
/* create a new thread that will execute 'do_loop()' */
thr_id = pthread_create(&p_thread, NULL, do_loop, (void*)&a);
/* run 'do_loop()' in the main thread as well */
do_loop((void*)&b);
/* NOT REACHED */
return 0;
}
A few notes should be mentioned about this program:
- Note that the main program is also a thread, so it executes the
do_loop()function in parallel to the thread it creates. pthread_create()gets 4 parameters. The first parameter is used bypthread_create()to supply the program with information about the thread. The second parameter is used to set some attributes for the new thread. In our case we supplied a NULL pointer to tellpthread_create()to use the default values. The third parameter is the name of the function that the thread will start executing. The forth parameter is an argument to pass to this function. Note the cast to a 'void*'. It is not required by ANSI-C syntax, but is placed here for clarification.- The delay loop inside the function is used only to demonstrate that the threads are executing in parallel. Use a larger delay value if your CPU runs too fast, and you see all the printouts of one thread before the other.
- The call to
pthread_exit()Causes the current thread to exit and free any thread-specific resources it is taking. There is no need to use this call at the end of the thread's top function, since when it returns, the thread would exit automatically anyway. This function is useful if we want to exit a thread in the middle of its execution.
In order to compile a multi-threaded program using
The source code for this program may be found in the pthread_create.c file.
gcc, we need to link it with the pthreads library. Assuming you have this library already installed on your system, here is how to compile our first program:gcc pthread_create.c -o pthread_create -lpthread The source code for this program may be found in the pthread_create.c file.
Synchronizing Threads With Mutexes
One of the basic problems when running several threads that use the same memory space, is making sure they don't "step on each other's toes". By this we refer to the problem of using a data structure from two different threads.
For instance, consider the case where two threads try to update two variables. One tries to set both to 0, and the other tries to set both to 1. If both threads would try to do that at the same time, we might get with a situation where one variable contains 1, and one contains 0. This is because a context-switch (we already know what this is by now, right?) might occur after the first tread zeroed out the first variable, then the second thread would set both variables to 1, and when the first thread resumes operation, it will zero out the second variable, thus getting the first variable set to '1', and the second set to '0'.
What Is A Mutex?
A basic mechanism supplied by the pthreads library to solve this problem, is called a mutex. A mutex is a lock that guarantees three things:
- Atomicity - Locking a mutex is an atomic operation, meaning that the operating system (or threads library) assures you that if you locked a mutex, no other thread succeeded in locking this mutex at the same time.
- Singularity - If a thread managed to lock a mutex, it is assured that no other thread will be able to lock the thread until the original thread releases the lock.
- Non-Busy Wait - If a thread attempts to lock a thread that was locked by a second thread, the first thread will be suspended (and will not consume any CPU resources) until the lock is freed by the second thread. At this time, the first thread will wake up and continue execution, having the mutex locked by it.
From these three points we can see how a mutex can be used to assure exclusive access to variables (or in general critical code sections). Here is some pseudo-code that updates the two variables we were talking about in the previous section, and can be used by the first thread:
lock mutex 'X1'. set first variable to '0'. set second variable to '0'. unlock mutex 'X1'.
Meanwhile, the second thread will do something like this:
lock mutex 'X1'. set first variable to '1'. set second variable to '1'. unlock mutex 'X1'.
Assuming both threads use the same mutex, we are assured that after they both ran through this code, either both variables are set to '0', or both are set to '1'. You'd note this requires some work from the programmer - If a third thread was to access these variables via some code that does not use this mutex, it still might mess up the variable's contents. Thus, it is important to enclose all the code that accesses these variables in a small set of functions, and always use only these functions to access these variables.
Creating And Initializing A Mutex
In order to create a mutex, we first need to declare a variable of typepthread_mutex_t , and then initialize it. The simplest way it by assigning it the PTHREAD_MUTEX_INITIALIZER constant. So we'll use a code that looks something like this:
pthread_mutex_t a_mutex = PTHREAD_MUTEX_INITIALIZER;
One note should be made here: This type of initialization creates a mutex called 'fast mutex'. This means that if a thread locks the mutex and then tries to lock it again, it'll get stuck - it will be in a deadlock.
There is another type of mutex, called 'recursive mutex', which allows the thread that locked it, to lock it several more times, without getting blocked (but other threads that try to lock the mutex now will get blocked). If the thread then unlocks the mutex, it'll still be locked, until it is unlocked the same amount of times as it was locked. This is similar to the way modern door locks work - if you turned it twice clockwise to lock it, you need to turn it twice counter-clockwise to unlock it. This kind of mutex can be created by assigning the constant
PTHREAD_RECURSIVE_MUTEX_INITIALIZER_NP to a mutex variable.Locking And Unlocking A Mutex
In order to lock a mutex, we may use the functionpthread_mutex_lock(). This function attempts to lock the mutex, or block the thread if the mutex is already locked by another thread. In this case, when the mutex is unlocked by the first process, the function will return with the mutex locked by our process. Here is how to lock a mutex (assuming it was initialized earlier):
int rc = pthread_mutex_lock(&a_mutex);
if (rc) { /* an error has occurred */
perror("pthread_mutex_lock");
pthread_exit(NULL);
}
/* mutex is now locked - do your stuff. */
.
.
After the thread did what it had to (change variables or data structures, handle file, or whatever it intended to do), it should free the mutex, using the
pthread_mutex_unlock() function, like this:
rc = pthread_mutex_unlock(&a_mutex);
if (rc) {
perror("pthread_mutex_unlock");
pthread_exit(NULL);
}
Destroying A Mutex
After we finished using a mutex, we should destroy it. Finished using means no thread needs it at all. If only one thread finished with the mutex, it should leave it alive, for the other threads that might still need to use it. Once all finished using it, the last one can destroy it using thepthread_mutex_destroy() function:
rc = pthread_mutex_destroy(&a_mutex);
After this call, this variable (a_mutex) may not be used as a mutex any more, unless it is initialized again. Thus, if one destroys a mutex too early, and another thread tries to lock or unlock it, that thread will get a
EINVAL error code from the lock or unlock function.Using A Mutex - A Complete Example
After we have seen the full life cycle of a mutex, lets see an example program that uses a mutex. The program introduces two employees competing for the "employee of the day" title, and the glory that comes with it. To simulate that in a rapid pace, the program employs 3 threads: one that promotes Danny to "employee of the day", one that promotes Moshe to that situation, and a third thread that makes sure that the employee of the day's contents is consistent (i.e. contains exactly the data of one employee).Two copies of the program are supplied. One that uses a mutex, and one that does not. Try them both, to see the differences, and be convinced that mutexes are essential in a multi-threaded environment.
The programs themselves are in the files accompanying this tutorial. The one that uses a mutex is employee-with-mutex.c. The one that does not use a mutex is employee-without-mutex.c. Read the comments inside the source files to get a better understanding of how they work.
Starvation And Deadlock Situations
pthread_mutex_lock() might block for a non-determined duration, in case of the mutex being already locked. If it remains locked forever, it is said that our poor thread is "starved" - it was trying to acquire a resource, but never got it. It is up to the programmer to ensure that such starvation won't occur. The pthread library does not help us with that.The pthread library might, however, figure out a "deadlock". A deadlock is a situation in which a set of threads are all waiting for resources taken by other threads, all in the same set. Naturally, if all threads are blocked waiting for a mutex, none of them will ever come back to life again. The pthread library keeps track of such situations, and thus would fail the last thread trying to call
pthread_mutex_lock(), with an error of type EDEADLK. The programmer should check for such a value, and take steps to solve the deadlock somehow.Refined Synchronization - Condition Variables
As we've seen before with mutexes, they allow for simple coordination - exclusive access to a resource. However, we often need to be able to make real synchronization between threads:- In a server, one thread reads requests from clients, and dispatches them to several threads for handling. These threads need to be notified when there is data to process, otherwise they should wait without consuming CPU time.
- In a GUI (Graphical User Interface) Application, one thread reads user input, another handles graphical output, and a third thread sends requests to a server and handles its replies. The server-handling thread needs to be able to notify the graphics-drawing thread when a reply from the server arrived, so it will immediately show it to the user. The user-input thread needs to be always responsive to the user, for example, to allow her to cancel long operations currently executed by the server-handling thread.
What Is A Condition Variable?
A condition variable is a mechanism that allows threads to wait (without wasting CPU cycles) for some even to occur. Several threads may wait on a condition variable, until some other thread signals this condition variable (thus sending a notification). At this time, one of the threads waiting on this condition variable wakes up, and can act on the event. It is possible to also wake up all threads waiting on this condition variable by using a broadcast method on this variable.Note that a condition variable does not provide locking. Thus, a mutex is used along with the condition variable, to provide the necessary locking when accessing this condition variable.
Creating And Initializing A Condition Variable
Creation of a condition variable requires defining a variable of typepthread_cond_t, and initializing it properly. Initialization may be done with either a simple use of a macro named PTHREAD_COND_INITIALIZER or the usage of the pthread_cond_init() function. We will show the first form here:pthread_cond_t got_request = PTHREAD_COND_INITIALIZER; This defines a condition variable named 'got_request', and initializes it.
Note: since the
PTHREAD_COND_INITIALIZER is actually a structure, it may be used to initialize a condition variable only when it is declared. In order to initialize it during runtime, one must use the pthread_cond_init() function.Signaling A Condition Variable
In order to signal a condition variable, one should either thepthread_cond_signal() function (to wake up a only one thread waiting on this variable), or the pthread_cond_broadcast() function (to wake up all threads waiting on this variable). Here is an example using signal, assuming 'got_request' is a properly initialized condition variable:int rc = pthread_cond_signal(&got_request); Or by using the broadcast function:
int rc = pthread_cond_broadcast(&got_request); When either function returns, 'rc' is set to 0 on success, and to a non-zero value on failure. In such a case (failure), the return value denotes the error that occured (
EINVAL denotes that the given parameter is not a condition variable. ENOMEM denotes that the system has run out of memory.Note: success of a signaling operation does not mean any thread was awakened - it might be that no thread was waiting on the condition variable, and thus the signaling does nothing (i.e. the signal is lost).
It is also not remembered for future use - if after the signaling function returns another thread starts waiting on this condition variable, a further signal is required to wake it up.
Waiting On A Condition Variable
If one thread signals the condition variable, other threads would probably want to wait for this signal. They may do so using one of two functions,pthread_cond_wait() or pthread_cond_timedwait(). Each of these functions takes a condition variable, and a mutex (which should be locked before calling the wait function), unlocks the mutex, and waits until the condition variable is signaled, suspending the thread's execution. If this signaling causes the thread to awake (see discussion of pthread_cond_signal() earlier), the mutex is automagically locked again by the wait funciton, and the wait function returns.pthread_cond_timedwait() allows the programmer to specify a timeout for the waiting, after which the function always returns, with a proper error value (ETIMEDOUT) to notify that condition variable was NOT signaled before the timeout passed. The pthread_cond_wait() would wait indefinitely if it was never signaled.Here is how to use these two functions. We make the assumption that 'got_request' is a properly initialized condition variable, and that 'request_mutex' is a properly initialized mutex. First, we try the
pthread_cond_wait() function:
/* first, lock the mutex */
int rc = pthread_mutex_lock(&a_mutex);
if (rc) { /* an error has occurred */
perror("pthread_mutex_lock");
pthread_exit(NULL);
}
/* mutex is now locked - wait on the condition variable. */
/* During the execution of pthread_cond_wait, the mutex is unlocked. */
rc = pthread_cond_wait(&got_request, &a_mutex);
if (rc == 0) { /* we were awakened due to the cond. variable being signaled */
/* The mutex is now locked again by pthread_cond_wait() */
/* do your stuff... */
.
}
/* finally, unlock the mutex */
pthread_mutex_unlock(&a_mutex);
Now an example using the
pthread_cond_timedwait() function:
#include /* struct timeval definition */
#include /* declaration of gettimeofday() */
struct timeval now; /* time when we started waiting */
struct timespec timeout; /* timeout value for the wait function */
int done; /* are we done waiting? */
/* first, lock the mutex */
int rc = pthread_mutex_lock(&a_mutex);
if (rc) { /* an error has occurred */
perror("pthread_mutex_lock");
pthread_exit(NULL);
}
/* mutex is now locked */
/* get current time */
gettimeofday(&now);
/* prepare timeout value */
timeout.tv_sec = now.tv_sec + 5
timeout.tv_nsec = now.tv_usec * 1000; /* timeval uses microseconds. */
/* timespec uses nanoseconds. */
/* 1 nanosecond = 1000 micro seconds. */
/* wait on the condition variable. */
/* we use a loop, since a Unix signal might stop the wait before the timeout */
done = 0;
while (!done) {
/* remember that pthread_cond_timedwait() unlocks the mutex on entrance */
rc = pthread_cond_timedwait(&got_request, &a_mutex, &timeout);
switch(rc) {
case 0: /* we were awakened due to the cond. variable being signaled */
/* the mutex was now locked again by pthread_cond_timedwait. */
/* do your stuff here... */
.
.
done = 0;
break;
case ETIMEDOUT: /* our time is up */
done = 0;
break;
default: /* some error occurred (e.g. we got a Unix signal) */
break; /* break this switch, but re-do the while loop. */
}
}
/* finally, unlock the mutex */
pthread_mutex_unlock(&a_mutex);
As you can see, the timed wait version is way more complex, and thus better be wrapped up by some function, rather then being re-coded in every necessary location.
Note: it might be that a condition variable that has 2 or more threads waiting on it is signaled many times, and yet one of the threads waiting on it never awakened. This is because we are not guaranteed which of the waiting threads is awakened when the variable is signaled. It might be that the awakened thread quickly comes back to waiting on the condition variables, and gets awakened again when the variable is signaled again, and so on. The situation for the un-awakened thread is called 'starvation'. It is up to the programmer to make sure this situation does not occur if it implies bad behavior. Yet, in our server example from before, this situation might indicate requests are coming in a very slow pace, and thus perhaps we have too many threads waiting to service requests. In this case, this situation is actually good, as it means every request is handled immediately when it arrives.
Note 2: when the mutex is being broadcast (using pthread_cond_broadcast), this does not mean all threads are running together. Each of them tries to lock the mutex again before returning from their wait function, and thus they'll start running one by one, each one locking the mutex, doing their work, and freeing the mutex before the next thread gets its chance to run.
Destroying A Condition Variable
After we are done using a condition variable, we should destroy it, to free any system resources it might be using. This can be done using thepthread_cond_destroy(). In order for this to work, there should be no threads waiting on this condition variable. Here is how to use this function, again, assuming 'got_request' is a pre-initialized condition variable:
int rc = pthread_cond_destroy(&got_request);
if (rc == EBUSY) { /* some thread is still waiting on this condition variable */
/* handle this case here... */
.
.
}
What if some thread is still waiting on this variable? depending on the case, it might imply some flaw in the usage of this variable, or just lack of proper thread cleanup code. It is probably good to alert the programmer, at least during debug phase of the program, of such a case. It might mean nothing, but it might be significant.
A Real Condition For A Condition Variable
A note should be taken about condition variables - they are usually pointless without some real condition checking combined with them. To make this clear, lets consider the server example we introduced earlier. Assume that we use the 'got_request' condition variable to signal that a new request has arrived that needs handling, and is held in some requests queue. If we had threads waiting on the condition variable when this variable is signaled, we are assured that one of these threads will awake and handle this request.However, what if all threads are busy handling previous requests, when a new one arrives? the signaling of the condition variable will do nothing (since all threads are busy doing other things, NOT waiting on the condition variable now), and after all threads finish handling their current request, they come back to wait on the variable, which won't necessarily be signaled again (for example, if no new requests arrive). Thus, there is at least one request pending, while all handling threads are blocked, waiting for a signal.
In order to overcome this problem, we may set some integer variable to denote the number of pending requests, and have each thread check the value of this variable before waiting on the variable. If this variable's value is positive, some request is pending, and the thread should go and handle it, instead of going to sleep. Further more, a thread that handled a request, should reduce the value of this variable by one, to make the count correct.
Lets see how this affects the waiting code we have seen above.
/* number of pending requests, initially none */
int num_requests = 0;
.
.
/* first, lock the mutex */
int rc = pthread_mutex_lock(&a_mutex);
if (rc) { /* an error has occurred */
perror("pthread_mutex_lock");
pthread_exit(NULL);
}
/* mutex is now locked - wait on the condition variable */
/* if there are no requests to be handled. */
rc = 0;
if (num_requests == 0)
rc = pthread_cond_wait(&got_request, &a_mutex);
if (num_requests > 0 && rc == 0) { /* we have a request pending */
/* do your stuff... */
.
.
/* decrease count of pending requests */
num_requests--;
}
}
/* finally, unlock the mutex */
pthread_mutex_unlock(&a_mutex);
Full Article here
http://www.cs.kent.edu/~ruttan/sysprog/lectures/multi-thread/multi-thread.html
Wednesday, June 20, 2007
Listing All Registered ActiveXControls / COM object based on there category
//Initialise COM libraries
CoInitialize (NULL);
//The Component Category Manager implemented by System implements
//this interface
ICatInformation *pCatInfo=NULL;
//Create an instance of standard Component Category Manager
HRESULT hr=CoCreateInstance (CLSID_StdComponentCategoriesMgr ,
NULL,
CLSCTX_INPROC_SERVER,
IID_ICatInformation ,
(void **)&pCatInfo);
//Increase ref count on interface
pCatInfo->AddRef ();
//IEnumGUID interface provides enumerator for enumerating through
//the collection of COM objects
IEnumGUID *pEnumGUID=NULL;
//We are intersted in finding out only controls so put CATID_Control
//in the array
CATID pcatidImpl[1];
CATID pcatidReqd[1];
// Want only my Plugin Category
pcatidImpl[0]=CATID_MyPlugin;
pcatidReqd[1]=CATID_MyPlugin;
// Want all Active X Control
//pcatidImpl[0]=CATID_Control;
//Now enumerate the classes i.e. COM objects of this type.
pCatInfo->EnumClassesOfCategories (1,
pcatidImpl,
0,
pcatidReqd ,
&pEnumGUID);
//Enumerate as long as you get S_OK
CLSID clsid;
while( (hr= pEnumGUID->Next( 1, &clsid, NULL ))==S_OK )
{
BSTR bstrClassName; //Get the information of class
//This is what MSDN says about the parameters
/*-----------------------------------------------
USERCLASSTYPE_FULL The full type name of the class.
USERCLASSTYPE_SHORT A short name (maximum of 15 characters) that
is used for popup menus and the Links dialog
box.
USERCLASSTYPE_APPNAME The name of the application servicing the class
and is used in the Result text in dialog boxes.
-----------------------------------------------*/
OleRegGetUserType (clsid,USERCLASSTYPE_FULL,&bstrClassName);
CString strControlName(bstrClassName);
//Add string in our listbox
m_list1.AddString (strControlName);
}
//we are done so now release the interface ptr
pCatInfo->Release ();
CoUninitialize ();
CoInitialize (NULL);
//The Component Category Manager implemented by System implements
//this interface
ICatInformation *pCatInfo=NULL;
//Create an instance of standard Component Category Manager
HRESULT hr=CoCreateInstance (CLSID_StdComponentCategoriesMgr ,
NULL,
CLSCTX_INPROC_SERVER,
IID_ICatInformation ,
(void **)&pCatInfo);
//Increase ref count on interface
pCatInfo->AddRef ();
//IEnumGUID interface provides enumerator for enumerating through
//the collection of COM objects
IEnumGUID *pEnumGUID=NULL;
//We are intersted in finding out only controls so put CATID_Control
//in the array
CATID pcatidImpl[1];
CATID pcatidReqd[1];
// Want only my Plugin Category
pcatidImpl[0]=CATID_MyPlugin;
pcatidReqd[1]=CATID_MyPlugin;
// Want all Active X Control
//pcatidImpl[0]=CATID_Control;
//Now enumerate the classes i.e. COM objects of this type.
pCatInfo->EnumClassesOfCategories (1,
pcatidImpl,
0,
pcatidReqd ,
&pEnumGUID);
//Enumerate as long as you get S_OK
CLSID clsid;
while( (hr= pEnumGUID->Next( 1, &clsid, NULL ))==S_OK )
{
BSTR bstrClassName; //Get the information of class
//This is what MSDN says about the parameters
/*-----------------------------------------------
USERCLASSTYPE_FULL The full type name of the class.
USERCLASSTYPE_SHORT A short name (maximum of 15 characters) that
is used for popup menus and the Links dialog
box.
USERCLASSTYPE_APPNAME The name of the application servicing the class
and is used in the Result text in dialog boxes.
-----------------------------------------------*/
OleRegGetUserType (clsid,USERCLASSTYPE_FULL,&bstrClassName);
CString strControlName(bstrClassName);
//Add string in our listbox
m_list1.AddString (strControlName);
}
//we are done so now release the interface ptr
pCatInfo->Release ();
CoUninitialize ();
Waiting for the Prj2make# Visual Studio Add-in For Mono
Right now Novell provide a command line utility to migrate a Microsoft Prj
to Mono.
http://forge.novell.com/modules/xfcontent/downloads.php/prj2make-sharp
Looking forward to see this project.
I have completed a Visual Studio Doxygen Add On if the author want
to contact me.
http://www.mfconsulting.com/product/gtks-inst4win/vsprj2make_Proposal.html
to Mono.
http://forge.novell.com/modules/xfcontent/downloads.php/prj2make-sharp
Looking forward to see this project.
I have completed a Visual Studio Doxygen Add On if the author want
to contact me.
http://www.mfconsulting.com/product/gtks-inst4win/vsprj2make_Proposal.html
Tuesday, June 19, 2007
java to c# and c# to java
Microsoft has the Java Language Conversion Assistant (JLCA) that enables
developers to migrate their existing Java code to C# on the .NET Framework.
It can be found at:
http://msdn2.microsoft.com/en-us/vstudio/aa718346.aspx
However, the reverse, a C# to Java is more difficlut to achieve
http://jsc.sourceforge.net/
developers to migrate their existing Java code to C# on the .NET Framework.
It can be found at:
http://msdn2.microsoft.com/en-us/vstudio/aa718346.aspx
However, the reverse, a C# to Java is more difficlut to achieve
http://jsc.sourceforge.net/
Thursday, June 14, 2007
Direct Show Filters Rules
APE Source Filter: DirectShow audio decoder filter used to decode Monkey's Audio files (APE files)
AC3 Filter: DirectShow audio decoder and processor filter used to decode audio tracks in movies (DVD, MPEG4 and others)
AVI Splitter: Replacement for DirectShow's default Audio Video Interleave (AVI) Splitter (It can open many broken files and can reindex when needed)
CDDA Reader Filter: DirectShow audio decoder capable of reading Audio CDs
CDXA Reader Filter: DirectShow video decoder capable of reading (S)VCDs and XCDs
CoreAAC Filter: Advanced Audio Coding (AAC) DirectShow audio decoder capable of reading AAC files
CoreFlac Decoder: DirectShow audio decoder for the Flac audio codec
FFDShow: DirectShow decoding filter for decompressing DivX, XviD, H.264, FLV1, WMV, MPEG-1 and MPEG-2, MPEG-4 and various other audio and video formats. It uses libavcodec from ffmpeg project for video decompression, postprocessing code from mplayer to enhance visual quality of low bitrate movies, and is based on original DirectShow filter from XviD, which is GPL'ed educational implementation of MPEG4 encoder
FLV Splitter: Capable of splitting Flash Video files (FLV) and decoding On2 VP62 Video streams (notable users of the FLV format include YouTube, Google Video, Reuters.com, Yahoo! Video and MySpace.)
OGG Splitter: DirectShow OGG Splitter capable of splitting OGG and OGM file(s) and enables to play / create .ogm wrapped video
GPL MPEG-1/2 Decoder: DirectShow MPEG decoder filter can be used to play MPEG-1 and MPEG-2 streams
MOD Source Filter: DirectShow filter decoder capable of decoding Tracker files (MO3, IT, XM, S3M, MTM, MOD and UMX)
MPEG Splitter: DirectShow splitter capable of splitting MPEG1/MPEG2 files (MP(E)G, VOB, DAT, TS, TP, etc.)
MPV Decoder: DirectShow decoder capable of decoding MPEG1/MPEG2 Video streams, and DVDs
MPA Splitter: DirectShow Splitter capable of splitting MPEG Audio files (MP3) and MPEG4 Audio files (AAC)
MPC Source Filter: DirectShow decoder filter capable of decoding MusePack Audio files (MPC).
MP4 Splitter: DirectShow Splitter capable of splitting MP4, M4A, 3GP and HD-MOV files
Matroska Splitter: DirectShow splitter capable of splitting Matroska files (MKV and MKA)
Media Player Classic: an open source media player that looks just like Windows Media Player, but has many additional features. It has a built in DVD player with real-time zoom, support for AVI subtitles, QuickTime and RealVideo support
OFR Source Filter: DirectShow filter decoder capable of decoding OptimFROG Audio files (OFR)
RealMedia Splitter: DirectShow splitter that allows you to play RealNetworks audio/video files.
SHOUTcast Source Filter: DirectShow decoder filter capable of handling internet radio stations streams (http://www.shoutcast.com)
WV Splitter / Decoder: DirectShow WV Splitter and decoder capable of splitting and decoding WavPack Audio files (WV)
VobSub Filter: DirectShow Vob Subtitles filter capable of displaying (embedded) Subtitle files (SRT, SUB, IDX, etc)
AC3 Filter: DirectShow audio decoder and processor filter used to decode audio tracks in movies (DVD, MPEG4 and others)
AVI Splitter: Replacement for DirectShow's default Audio Video Interleave (AVI) Splitter (It can open many broken files and can reindex when needed)
CDDA Reader Filter: DirectShow audio decoder capable of reading Audio CDs
CDXA Reader Filter: DirectShow video decoder capable of reading (S)VCDs and XCDs
CoreAAC Filter: Advanced Audio Coding (AAC) DirectShow audio decoder capable of reading AAC files
CoreFlac Decoder: DirectShow audio decoder for the Flac audio codec
FFDShow: DirectShow decoding filter for decompressing DivX, XviD, H.264, FLV1, WMV, MPEG-1 and MPEG-2, MPEG-4 and various other audio and video formats. It uses libavcodec from ffmpeg project for video decompression, postprocessing code from mplayer to enhance visual quality of low bitrate movies, and is based on original DirectShow filter from XviD, which is GPL'ed educational implementation of MPEG4 encoder
FLV Splitter: Capable of splitting Flash Video files (FLV) and decoding On2 VP62 Video streams (notable users of the FLV format include YouTube, Google Video, Reuters.com, Yahoo! Video and MySpace.)
OGG Splitter: DirectShow OGG Splitter capable of splitting OGG and OGM file(s) and enables to play / create .ogm wrapped video
GPL MPEG-1/2 Decoder: DirectShow MPEG decoder filter can be used to play MPEG-1 and MPEG-2 streams
MOD Source Filter: DirectShow filter decoder capable of decoding Tracker files (MO3, IT, XM, S3M, MTM, MOD and UMX)
MPEG Splitter: DirectShow splitter capable of splitting MPEG1/MPEG2 files (MP(E)G, VOB, DAT, TS, TP, etc.)
MPV Decoder: DirectShow decoder capable of decoding MPEG1/MPEG2 Video streams, and DVDs
MPA Splitter: DirectShow Splitter capable of splitting MPEG Audio files (MP3) and MPEG4 Audio files (AAC)
MPC Source Filter: DirectShow decoder filter capable of decoding MusePack Audio files (MPC).
MP4 Splitter: DirectShow Splitter capable of splitting MP4, M4A, 3GP and HD-MOV files
Matroska Splitter: DirectShow splitter capable of splitting Matroska files (MKV and MKA)
Media Player Classic: an open source media player that looks just like Windows Media Player, but has many additional features. It has a built in DVD player with real-time zoom, support for AVI subtitles, QuickTime and RealVideo support
OFR Source Filter: DirectShow filter decoder capable of decoding OptimFROG Audio files (OFR)
RealMedia Splitter: DirectShow splitter that allows you to play RealNetworks audio/video files.
SHOUTcast Source Filter: DirectShow decoder filter capable of handling internet radio stations streams (http://www.shoutcast.com)
WV Splitter / Decoder: DirectShow WV Splitter and decoder capable of splitting and decoding WavPack Audio files (WV)
VobSub Filter: DirectShow Vob Subtitles filter capable of displaying (embedded) Subtitle files (SRT, SUB, IDX, etc)
Subscribe to:
Posts (Atom)