Friday, January 25, 2008

How to detect Mobile device hitting your Web Server in PHP

by Andrea Trasatti

http://wurfl.sourceforge.net/index.php

PHP is a great platform for WAP development. Thousands of developers worldwide love PHP for its performance and for the semplicity of its model.
It should come as no surprise that some PHP developer quickly built the tools to tap the WURFL power from PHP.

One easy way to play with the WURFL, is to use what I call the "WURFL PHP Library". The package includes the library and some support files such as a readme (I STRONGLY SUGGEST EVERYONE TO READ IT), check_wurfl.php that will let you quickly read all the capabilities of a selected device and update_cache.php which will be better described later.

* wurfl_parser.php: parse the xml and put it in an Array and the other to work with the data collected. Before starting, make sure you have compiled PHP properly ( http://www.php.net/manual/en/ref.xml.php ), wurfl_parser.php uses the basic XML functions implemented compiling with expat.
* wurfl_class.php: this script lets you access the data in the wurfl array (see previous bullet) in an object-oriented fashion.
* wurfl_config.php: with the growing number of possible configurations in the PHP library, I am now introducing a single file for configuration. This should hopefully make it clearer where you need to configure the scripts and maybe also make it easier to integrate it with your central configuration files (if your application has any)


wurfl_parser.php

wurfl_parser.php: This is a VERY simple XML parser that reads WURFL and puts all the needed data into an array. Considering the time the parser was taking to parse the XML, I thought that using some kind of caching mechanism was probably a good idea. This will be discussed more later.
The generated Array is an associative array that looks like this:

$wurfl["devices"]["ericsson_generic"]["fall_back"]="generic";

The first key is "devices". Other possible values such as authors and contributors are probably not so interesting to you.

The second key is the UNIQUE ID assigned to each user agent. All other keys are related to the attributes or groups and capabilities of the device.

Of course, this may seem not so flexible in practice, since devices tell us their user-agent strings, not their WURFL IDs.
This is solved thanks to an additional array called $wurfl_agents, which is simply an associative array of user agents and the relative unique id. This array makes your life much simpler (and your search much faster). $wurfl_agents is cached too.
Let's look at a concrete example. Someone visits you site with, say, a Siemens S45 (user agent: "SIE-S45/24 UP.Browser/5.0"). In order to find the device ID, you would normally be required to cycle most of the $wurfl array and retrieve the ID, before you can look up the actual capabilities.
Thanks to $wurfl_agent, this is as simple as:

if ( in_array($id, $wurfl_agents) ) {
echo "The ID $id is known
\n";
}

Example to search an ID knowing the user agent:

$wurfl_user_agents = array_keys($wurfl_agents);
while ( $x = each($wurfl_user_agents) ) {
if ( $x[1] == $user_agent ) {
return $wurfl_agents($x[1]);
}
}


wurfl_class.php

wurfl_class.php was created to make our life simpler. Once we had the data ready in an array, we needed an easy way to access it and some methods to manipulate it. By looking closely at what had been done with the Java API, I reproduced many of those useful methods in PHP.

The class loads the parser automatically. To load the parser a defined must be set appropriately in the config file.

The class is initialized calling the constructor, wurfl_class and passing two variables that may be empty. The former is the full XML parsed (like the parser does) and the latter is the array of user agents and id's as generated by the parser. Pass two empty variables (or nothing) if you want them to be filled (when needed) or the real values if you already have them. The class will check the values and the cache files (if enabled) and decide what to do. Also check your configuration, because the behaviour will change.
Use the public method GetDeviceCapabilitiesFromAgent() passing the user agent to make the class search for the best fit and fill the object's properties. Once again, what the class does depends on your configuration, if you enabled cache files and so on.
Once you have instantiated the object and passed a user agent you may use all the class' methods.
Here is a list of properties:

$wurfl_class->_wurfl is the WURFL array (all of it)

$wurfl_class->_wurfl_agents is the associative array made of the user agents
and unique id's

$wurfl_class->user_agent the visitor's user agent

$wurfl_class->wurfl_agent the WURFL's best fitting user agent

$wurfl_class->id the corresponding id

$wurfl_class->GUI true if the device supports Openwave's GUI
extensions

$wurfl_class->browser_is_wap true if the device is WAP capable. It is here
only for legacy support, you should use the
is_wireless_device capability from WURFL.
browser_is_wap now has the same value as the
capability, when found. If you want to take full
advantage of this capability you should download
the web patch from the WURFL site or CVS!
$wurfl_class->capabilities the array of device's capabilities

Note: PHP (up to version 4.3) does not have any distinction between private and public methods. In the wurfl_class implementation, I named all private methods with a leading underscore to distinguish them from public methods without the underscore. If you are interested in knowing the details, just open the class, there are beautiful JAVADOC-like comments for each variable and method.
These libraries don't require register_globals anymore (from version 2 and up), but will not work with versions before 4.1.

wurfl_class($wurfl, $wurfl_agents) is the constructor. Built to work
best with the wurfl_parser.

GetDeviceCapabilitiesFromAgent($ua) given a user agent it will search WURFL
for the best fit

getDeviceCapability($capability) given a capability it will tell you the
value. Remember that capabilities might be string,
integer or boolean.

wurfl_config.php

The scope of this file is straight forward, modify it at your wish to configure the library to act as you like it best.
Please check the paragraph about caching for more info about cache files.
Here is a quick explanation of all the fields:

WURFL_CONFIG boolean, this is set to true by default, it's used as a simple
check to make sure the configuration was included. Add this to your
configuration files if you won't use wurfl_config.php, otherwise just
leave it as it is

DATADIR string, where all data is stored (wurfl.xml, cache file, logs, etc)

WURFL_FILE string, full path and filename of wurfl.xml

WURFL_PARSER_FILE string, full path and filename of wurfl_parser.php

WURFL_CLASS_FILE string, full path and filename of wurfl_class.php

WURFL_USE_CACHE boolean, true if you want to use a cache file (strongly
suggested). If only this parameter is set to true will be used
cache.php.

WURFL_USE_MULTICACHE boolean, true if you want to use Multicache files
instead of a single BIG cache file (cache.php)

MULTICACHE_DIR string, used only if you enabled Multicache, defines where
the cache files will be stored. WARNING: while cache.php will grow
in size but remain a single file, here the files will grow in
number. Expect more than 5000 tiny files.

MULTICACHE_SUFFIX string, suffix for the files generated using Multicache.
Useful if you use a caching system and don't want to load your
shared memory with a ton of tiny files.

CACHE_FILE string, with full path and filename of the cache file to use
(refreshed when a new WURFL is found, if WURFL_CACHE_AUTOUPDATE is
set to true)

WURFL_CACHE_AUTOUPDATE boolean, tells the class to automatically update the
cached files with a new XML is found. This is NOT suggested when
using MULICACHE because of the high number of files to be updated.
Race conditions are highly possible to happen. The use of
update_cache.php is strongly suggested for production
environments

WURFL_PATCH_FILE string, optional patch file for WURFL

WURFL_AGENT2ID_FILE string, used by wurfl_class.php. Used only when
WURFL_USE_CACHE is set to true

MAX_UA_CACHE integer, max number of user agents to store in
WURFL_AGENT2ID_FILE. Too high limits might give the opposite effect.

WURFL_LOG_FILE string, defines full path and filename for logging

LOG_LEVEL integer, desired logging level. Use the same constants as for PHP
logging

WURFL_AUTOLOAD boolean, true if you want the XML to be loaded at every
startup. If not, the XML will be loaded when needed.


Caching

Considering how slow PHP can be when parsing a big XML file, caching was a must.
Currently there are two caching systems. The older is activated when setting WURFL_USE_CACHE to true and uses DATADIR to store its files. The concept is quite simple, dump the array generated by the parsers in a big file, by default called cache.php (set by the define CACHE_FILE). In this file we also store the array called $wurfl_agents and a timestamp, useful to check if a new XML was deployed.
This system is very simple in its concept and worked well for quite some time. Considering the big size that cache.php was reaching it became a need (and in fact I always strongly suggested it) to use a caching system at PHP-level, such as Zend Accelerator, Turck cache, APC 2.0. Using such tools lets you store the cache file (cache.php) into shared memory and provides really good performances from the third hit on (first hit the XML is parsed: slow, second hit the cache is stored in shared memory: slow, third hit and on the cache is read from the shared memory: fast!).

The new caching system was dubbed "multicache" because instead of generating a single big cache file it generates 1 cache file for every device in WURFL. For this reason you will need to create a directory for this (or at least this is suggested) because the library will generate about 6000 tiny files when this feature is activated.
To activate the multicache system you will need to set WURFL_USE_CACHE to true.
CACHE_FILE will still be used, but the file will only contain the array $wurfl_agents and the timestamp.
Also set WURFL_USE_MULTICACHE to true, set the appropriate path for MULTICACHE_DIR, an absolute path is suggested. Don't forget the ending slash (for example '/tmp/cache/multicache/').
MULTICACHE_SUFFIX should be left unchanged in most cases. This will define the extension of the tiny files. You might want to set some strange extension if you want to avoid that those files are cached in shared memory by any PHP-cache, for example. Change it only if you know what you're doing!
The multicache system provides a MUCH faster data retrival, files are smaller and so it will take a way less time to read them. This will also mean a higher I/O on your system, consider returning to the older cache system if you have problems. Generating many tiny files also involves possible race conditions if a new XML is deployed and the library is configured to update automatically the cache. Read on for more info.

If WURFL_CACHE_AUTOUPDATE is set to true the library (specifically wurfl_class.php) will check the timestamp in cache.php against the file mtime of wurfl.xml. If the XML is newer than the cache it is reloaded. This is not suggested for production environments, if you have many concurrent hits you might have more than one process trying to refresh the same cache wasting a lot of resources. If you would like to avoid this you can set the automatic update to false and use the 'ad hoc' script called update_cache.php. This script was created to be called from command line (or a hidden URL if you'd like, but the command line is suggested when available) and force a cache update. This way the cache update will be prepared and the file will be changed at the very last second saving a lot of resources and having a single process do it. On sites with MANY hits you might also consider preparing the cache files on a separated system and moving them to the production server at once. There isn't a sccript to do this automatically at this time, but the new update_cache.php is already a step in that direction.

When setting WURFL_USE_CACHE to true you also enable another simple caching system (active both when using standard cache and multicache). When a user agent hits your site, this will most likely hit it again a few times. It would be stupid to search for all its capabilities at every second hit. For this reason we store the user agent and its capabilities in a file named after the value set in WURFL_AGENT2ID_FILE, this will make every second hit A LOT faster.
Storing all the user agents hitting your site will end up having a second full cache file and the benefit would reach zero. For this reason you can define a limit of user agents stored using MAX_UA_CACHE. The "perfect" value will change depending on your server's performances on the variety of user agents visiting your site and so on. A good number is between 30 and 50. I suggest you to start with 30 and maybe check the logs and see how often the cache is cleaned of the elder user agents and how often a user agent that is still visiting your site is cleaned and researched.

It's a direct consequence of the cache system that you will not need to read the entire XML and parse it every time you start the wurfl object. You may still want to force this for debug reasons for example. You can do this setting to true the define named WURFL_AUTOLOAD. If you are using any of the caching systems, I suggest you to disable this. If you're not using any cache, the XML will be loaded anyway, so just set this to true, if you'd like.


Logging

While logging is out of the scope of the WURFL PHP Libraries and I suggest you to integrate the libraries with your logging system (if you have any), a basic logging feature is included. This should work fine on Linux, Solaris and Windows.
Logging is done on a file as configured with WURFL_LOG_FILE. The log level is defined following the PHP contants and using the define named LOG_LEVEL. It used to be buggy in previous releases, check out how it has changed and it is now supposed to work properly. When set with the highest level the library might generate a log of logs.
Logs should anyway give you all the info you might need. This is a sample log when set at the highest detail level:

[LuckyTitan.local 327][constructor] Class Initiated
[LuckyTitan.local 327][GetDeviceCapabilitiesFromAgent] searching for SonyEricssonZ600/R601
[LuckyTitan.local 327][_cacheIsValid] cache file is outdated
[LuckyTitan.local 327][GetDeviceCapabilitiesFromAgent] cache enabled, WURFL is not loaded, now loading
[LuckyTitan.local 327][GetDeviceCapabilitiesFromAgent] loading WURFL from XML
[LuckyTitan.local 327][parse] No XML patch file defined
[LuckyTitan.local 327][GetDeviceCapabilitiesFromAgent] Searching in the agent database
[LuckyTitan.local 327][_GetFullCapabilities] searching for sonyericsson_z600_ver1_subr601
[LuckyTitan.local 327][_GetDeviceCapabilitiesFromId] reading id:sonyericsson_z600_ver1_subr601
[LuckyTitan.local 327][_GetDeviceCapabilitiesFromId] I have it in wurfl_agents cache, done

Wednesday, December 19, 2007

How to detect if Visual Studio 2005 SP1 is installed on Windows

If you don't need to be specific about a particular Visual Studio 2005 SKU, you can detect SP1 for a particular written language (for VSTS and TFS) or for a particular project language (for Express SKUs) using an older registry key under,

HKEY_LOCAL_MACHINE\Software\Microsoft\Active Setup\Installed Components\{PatchCode}

You can find the {PatchCode} in the .msp file. In Windows 2003 and earlier platforms, you'll see this as the Revision number property in the properties Summary tab in Windows Explorer. In all cases, you can query the Summary Information stream or open the .msp file in Orca, then click the View -> Summary Information menu item.

For the English Visual Studio 2005 Service Pack 1 for Standard, Professional, and Team editions (VS80sp1-KB926601-X86-ENU) an example of the registry key above follows.

HKEY_LOCAL_MACHINE\Software\Microsoft\Active Setup\Installed Components\{D93F9C7C-AB57-44C8-BAD6-1494674BCAF7}

You can also determine the service pack level of a more general, larger scope by reading a REG_DWORD registry value named SP from under,

HKEY_LOCAL_MACHINE\Software\Microsoft\DevDiv\[ProductFamily]\Servicing\8.0

…and for a specific language SKU using a sub-key,

HKEY_LOCAL_MACHINE\Software\Microsoft\DevDiv\[ProductFamily]\Servicing\8.0\[ProductEdition]\[ProductLanguage]

Product families are pretty broad in scope, and include for "Whidbey":

URT (.NET Framework, once known as the Universal Runtime)
VB (Microsoft Visual Basic 2005 Express)
VC (Microsoft Visual C++ 2005 Express)
VCS (Microsoft Visual C# 2005 Express)
VJS (Microsoft Visual J# 2005 Express)
VNS (Microsoft Visual Web Developer 2005 Express)
VS (Visual Studio 2005 Standard, Professional, Team Suite, etc.)
VSTF (Visual Studio 2005 Team Foundation Services)
An example using the English Visual Studio 2005 Team Suite Service Pack 1 of the registry keys above follows. Under each key you would find a registry value named SP set to 1.

HKEY_LOCAL_MACHINE\Software\Microsoft\DevDiv\VS\Servicing\8.0
HKEY_LOCAL_MACHINE\Software\Microsoft\DevDiv\VS\Servicing\8.0\VSTS\1033

Visual Studio 2005 Express SKUs will always use the ProductEdition of EXP, so you can combine the ProductFamily values documented above with EXP to form registry keys like the following, for Visual C# 2005 Express.

HKEY_LOCAL_MACHINE\Software\Microsoft\DevDiv\VCS\Servicing\8.0
HKEY_LOCAL_MACHINE\Software\Microsoft\DevDiv\VCS\Servicing\8.0\EXP\1033

Sunday, December 16, 2007

Locking header control in MFC to prevent the column to be resized

The way to prevent sizing is to eat this notification (without passing it to the header control), but as with many other Windows® messages and notifications, HDN_BEGINTRACK comes in two flavors: HDN_BEGINTRACKW (wide-character, Unicode) and HDN_BEGINTRACKA (ANSI). The "neuter" symbol is #defined to one or the other of these, based on the value of UNICODE defined in your project, as shown here:

// From commctrl.h
#ifdef UNICODE
#define HDN_BEGINTRACK HDN_BEGINTRACKW
#else
#define HDN_BEGINTRACK HDN_BEGINTRACKA
#endif


So when you implement a handler for HDN_BEGINTRACK, you're actually implementing it for HDN_BEGINTRACKA or HDN_BEGINTRACKW, depending on the value of UNICODE. But which message does the header control actually send? Remember, the header control is part of Windows, one of the common controls in comctl32.dll. Since the DLL is already compiled into executable code, changing the value of UNICODE in your project has absolutely no effect on its operation. How does the header control know which flavor of notification to send—A or W?

The answer lies in an oft-forgotten message, WM_NOTIFYFORMAT. When a control is first created, it sends a message to its parent, in effect asking, "do you want ANSI or Unicode notifications?" The parent responds with NFR_ANSI or NFR_UNICODE. If the parent doesn't handle WM_NOTIFYFORMAT, the Windows DefWindowProc responds based on the preference of the parent window or dialog itself. The default is Unicode. So I suspect the reason you had no luck trapping HDN_BEGINTRACK is that you compiled an ANSI program without handling WM_NOTIFYFORMAT. Your application is looking for HDN_BEGINTRACKA (ANSI) while the header control is sending HDN_BEGINTRACKW (Unicode).

One way to fix the problem is to implement a WM_NOTIFYFORMAT handler for your list control, one that returns NFR_ANSI. When I tried this, the header control did indeed send HDN_BEGINTRACKA and I was able to prevent sizing. But using NFR_ANSI broke other features. For example, the list control no longer repaints its columns while sizing.

A simpler, more reliable way to prevent sizing header columns is to implement handlers for both HDN_BEGINTRACKA and HDN_BEGINTRACKW. This not only obviates the need to process WM_NOTIFYFORMAT, it lets your code work in both ANSI and Unicode modes.



The header control sends HDN_XXX notifications to the parent (list control) window, but when using MFC you can use message reflection to handle the notifications in the header itself. Since the "lockable columns" feature is more a property of the header than the list control, this is the approach I chose. If you're not using MFC, you'll have to handle these notifications in the list control. To do message reflection, you can use ON_NOTIFY_REFLECT in your header control's message map or simply override the virtual function OnChildNotify, as shown here:

BOOL CLockableHeader::OnChildNotify(
UINT msg, WPARAM wp, LPARAM lp, LRESULT* pRes)
{
NMHDR& nmh = *(NMHDR*)lp;
if (nmh.code==HDN_BEGINTRACKW || nmg.code==HDN_BEGINTRACKA)
return *pRes=TRUE;
•••
}


Since OnChildNotify is virtual, there's no need for message map entries. All you have to do is implement it. In any given application, the header will send one or the other, not both. Either way, CLockableHeader eats the notification—that is, returns TRUE (handled) without passing on to the default header control. CLockableHeader controls locking through a flag m_bLocked which the app can set by calling CLockableHeader::Lock.

If you're going to prevent sizing, you should also disable the size cursor. Otherwise users may think your app is either broken or lame. Fortunately, it's trivial:

BOOL
CLockableHeader::OnSetCursor(
CWnd* pWnd, UINT nHit, UINT msg)
{
return m_bLocked ? TRUE :
CHeaderCtrl::OnSetCursor(pWnd, nHit, msg);
}


In other words: if the columns are locked, OnSetCursor returns TRUE without setting the cursor; otherwise, let the header control do its size cursor thing. Now when the columns are locked, Windows displays its standard arrow cursor instead of displaying the left-right sizing cursor.

Once you've implemented your custom header control by deriving from CHeaderCtrl, how do you get Windows to use it? The same way you would for any dialog control, by subclassing. The right place is in the parent window's OnCreate handler:

// CMyView is derived from CListView
int CMyView::OnCreate(LPCREATESTRUCT lpcs)
{
VERIFY(CListView::OnCreate(lpcs)==0);
return m_header.SubclassDlgItem(0,this) ? 0 : -1;
}


This works because the header control always has ID = 0. With all this in place, the only thing left to do is implement the command and UI update handlers for the View | Lock Columns command.

Sunday, December 02, 2007

How lucky you are if you read this text

Docteur Phillip M Harter, Professeur à l'école de Médecine de l'Université de Stanford .

Si on pouvait réduire la population du monde en un village de 100 personnes tout en maintenant les proportions de tous les peuples existants sur la Terre, ce village serait ainsi composé :

57 asiatiques,

21 européens,

14 américains (Nord, Centre et Sud)

8 africains.

Il y aurait :

52 femmes et 48 hommes

30 blancs et 70 non blancs

30 chrétiens et 70 non chrétiens

89 hétérosexuels et 11 homosexuels

6 personnes posséderaient 59% de la richesse totale et tous les 6 seraient américains

80 vivraient dans des maisons vétustes

70 seraient analphabètes

50 souffriraient de malnutrition

1 serait en train de mourir

1 serait en train de naître

1 posséderait un ordinateur

1 (oui, un seulement) aurait un diplôme universitaire

Si on considère le monde de cette manière, le besoin d'accepter et de comprendre devient évident.

Prenez en considération aussi ceci :

Si vous vous êtes levé ce matin avec plus de santé que de maladie, vous êtes plus chanceux que le million de personnes qui ne verra pas la semaine prochaine.

Si vous n'avez jamais été dans le danger d'une bataille, la solitude de l'emprisonnement, l'agonie de la torture, l'étau de la faim, vous êtes mieux que 500 millions de personnes.

Si vous pouvez aller à l'église ou dans un temple sans peur d'être menacé, torturé ou tué, vous avez plus de chance que 3 milliards de personnes.

Si vous avez de la nourriture dans votre réfrigérateur, des habits sur vous, un toit sur votre tête et un endroit pour dormir, vous êtes plus riche que 75% des habitants de la Planète.

Si vous avez de l'argent à la banque, dans votre portefeuille ou de la monnaie dans une petite boite, vous faite partie des 8% les plus privilégiés du monde.

Si vos parents sont encore vivants et toujours mariés, vous êtes des personnes réellement rares.

Si, enfin, vous lisez ce message, vous venez de recevoir un double cadeau parce que quelqu'un a pensé à vous et parce que vous ne faites pas partie des deux milliards de personnes qui ne savent pas lire...

Monday, November 26, 2007

How to create a def file for a dll

you can still create a def file using the output from the objdump program (from the MinGW distribution). Here's an example.

objdump -p mydll.dll > mydll.fil

Search for "[Ordinal/Name Pointer] Table" in mydll.fil and use the list of functions following it to create your def file.

Wednesday, November 21, 2007

Upload Video to Youtube

I am finally on the verge to integrate a youtube uploader tool to one of my side project. I just need to figure out how to log to gmail instead of youtube since there is two ways to authenticate with youtube.

About youtube :

When you upload a video to YouTube, it automatically converts it to a flash video file (.flv) with specific settings (see below, standard column). The contrast of the video is increased and video noise is suppressed. YouTube appears to respect the framerate of the uploaded file; Framerates will varie between 15 (minimum framerate to tell it is a video...), 24 frames (pal) and 30 frames per second (ntsc with no drop frame). It is hard to tell how long you have to wait for the video to be available but it is pretty fast !!. Once the video has been uploaded you can log to your account and from there you can check the status of your video. Usually you will see Uploaded (processing, please wait).


Video Profile

standard recommended possible
file extension : .flv (flash video)
total bitrate : ~320 kbits/sec 804.7 kbits/sec 2147.2 kbits/sec
audio codec : MPEG layer 3 (MP3)
audio format : Mono, 22.050 kHz Stereo, 44.100 kHz Stereo, 44.100 kHz
audio bitrate : ~67 kpbs/sec 128 kpbs/sec 128 kpbs/sec
video codec : Sorenson Spark (H.263)
video format : 320 x 240 px, 15-30 fps 320 x 240 px, 29.97 fps 640 x 480 px, 29.97 fps
video bitrate : ~250 kbps/sec 677.7 kbps/sec 2019.2 kbps/sec

You can save your own .flv files and upload them to YouTube. You can encode video with higher bitrates and resolution than YouTube compresses with. The only limitations I can tell are that the file must be under 100 MB and you must use the Sorenson Spark video codec and MP3 audio codec.

Monday, November 19, 2007

How to take advantage of Silverlight Marker to create hypervideo

Interesting article from Jesse Liberty also i worked on Hypervideo back in 1999.... Silverlight is finally well suited to turn HD video into interactive sequences.

http://silverlight.net/blogs/jesseliberty/archive/2007/10/24/hypervideo-2.aspx

Wednesday, November 14, 2007

silent install of the VC 8.0 runtime (vcredist) packages

if you have downloaded the standalone VC 8.0 redistributable packages, you will need to modify the command lines slightly. The following command lines can be used to install the original release of the standalone VC 8.0 redistributable packages:

For x86: vcredist_x86.exe /q:a /c:"VCREDI~1.EXE /q:a /c:""msiexec /i vcredist.msi /qn"" "
For x64: vcredist_x64.exe /q:a /c:"VCREDI~2.EXE /q:a /c:""msiexec /i vcredist.msi /qn"" "
For ia64: vcredist_ia64.exe /q:a /c:"VCREDI~3.EXE /q:a /c:""msiexec /i vcredist.msi /qn"" "
The following command lines can be used to install the Visual Studio 2005 SP1 release of the standalone VC 8.0 redistributable packages:

For x86: vcredist_x86.exe /q:a /c:"VCREDI~3.EXE /q:a /c:""msiexec /i vcredist.msi /qn"" "
For x64: vcredist_x64.exe /q:a /c:"VCREDI~2.EXE /q:a /c:""msiexec /i vcredist.msi /qn"" "
For ia64: vcredist_ia64.exe /q:a /c:"VCREDI~1.EXE /q:a /c:""msiexec /i vcredist.msi /qn"" "
If you would like to install the VC runtime packages in unattended mode (which will show a small progress bar but not require any user interaction), you can change the /qn switch above to /qb. If you would like the progress bar to not show a cancel button, then you can change the /qn switch above to /qb!

Some folks ask me what vcredist.msi..
Iy means that you want to extract the embedded msi from the self exe package and be able to run msiexec which is used to install the package. msiexec is the Microsoft Installer runtime.
MSI is the prefered format for microsoft installer. Installshield and Visual Studio 2005 and higher can generate MSI. Using the msiexec command. You can even log all the installer session to a file and you can verify and explore the content of the MSI using ORCA in order to look at the installer sequence, string table and in some cases embedded Visual Basic Script. Yes Visual Basic is not dead. Still used and convnient enough for scripting installer code :)

If you want to install the runtime from a Null Soft Installer NSI script with minimal UI and progress bar, here is what you want to do

;extract selft exe to temp
SetOutPath "$TEMP"
; uncompress
File "${RUNTIME_VCREDIST}"



;-------------------------------
; Test if Visual Studio Redistributables 2005+ SP1 installed
; Returns -1 if there is no VC redistributables intstalled
Function CheckVCRedist
Push $R0
ClearErrors
ReadRegDword $R0 HKLM "SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\{7299052b-02a4-4627-81f2-1818da5d550d}" "Version"

; if VS 2005+ redist SP1 not installed, install it
IfErrors 0 VSRedistInstalled
StrCpy $R0 "-1"

VSRedistInstalled:
Exch $R0
FunctionEnd


Section "Installer Sections" SecDummy
SetOverwrite ifnewer
Call CheckVCRedist

;IntCmp $0 5 is5 lessthan5 morethan5
IntCmp $R0 -1 noVCRuntime noVCRuntime VCRuntime
;force install
; IntCmp $R0 -1 noVCRuntime noVCRuntime noVCRuntime

noVCRuntime:


;extract VCRuntime to Temp
SetOutPath "$TEMP"
File "${RUNTIME_VCREDIST}"

;/qn for complete silent install
;/qb small progress dialog but does not require any user interaction
;StrCpy $VC_RUNTIME "$\"$TEMP\vcredist_x86.exe$\" /q:a /c:$\"VCREDI~3.EXE /q:a /c:$\"$\"msiexec /i vcredist.msi /qb!$\""
StrCpy $VC_RUNTIME "$\"$TEMP\vcredist_x86.exe$\" /q:a"

;MessageBox MB_ICONINFORMATION 'Need to install VCRuntime {$R0} {$VC_RUNTIME}'

ExecWait '$VC_RUNTIME' $0
;MessageBox MB_ICONINFORMATION 'Return Code {$0}'
;IntCmp $0 0 done

;MessageBox MB_ICONINFORMATION 'install complete'

Delete "$TEMP\vcredist_x86.exe"

Goto VCRuntime

VCRuntime:

SectionEnd


http://blogs.msdn.com/astebner/archive/2007/02/07/update-regarding-silent-install-of-the-vc-8-0-runtime-vcredist-packages.aspx

Sunday, November 11, 2007

Friday, October 12, 2007

Parsing XML in PHP

Traverse the document tree
In this example we have a XML file which consists of a document with a specific version. The document contains persons with the attributes firstname, lastname and description. The code snippet finds the document version and prints it. It's pretty straight forward you just check all the top nodes of the document and look for the one named document.


foreach ( $tree->children as $document )
{
// parse the document
if ( $document->name == "document" )
{
// get the document version attribute
foreach ( $document->attributes as $documentAttr )
{
if ( $documentAttr->name == "version" )
{
print( "Found document with version: " . $documentAttr->content . "
" );
}
}

// find persons here
}
}


When you've found the document node you can start looking for persons. This is done in the same manner, check the children nodes and look for person.

To make the process of getting the attribute values simpler we write a helper function to fetch the attribute value from a node.


function getAttrValue( $node, $attrName )
{
$ret = false;

foreach ( $node->attributes as $nodeAttr )
{
if ( $nodeAttr->name == $attrName )
{
$ret = $nodeAttr->content;
}
}
return $ret;
}


Now we're ready to parse the information describing the persons in this example. When we've found the person node we check all the subnodes and look for the nodes we want and fetch the information from these nodes.


// parse all persons
foreach ( $document->children as $person )
{
if ( $person->name == "person" )
{
print( "Found a new person
" );

$firstName = "";
$lastName = "";
$descriptionName = "";

// get the name and description
foreach ( $person->children as $personAttribute )
{
switch ( $personAttribute->name )
{
case "firstname" :
{
$firstName = getAttrValue( $personAttribute, "value" );
}break;

case "lastname" :
{
$lastName = getAttrValue( $personAttribute, "value" );
}break;

case "description" :
{
// get the description text
foreach ( $personAttribute->children as $description )
{
if ( $description->type == 3 )
{
$description = $description->content;
}
}
}break;
}
}

print( "The persons firstname is: $firstName
" );
print( "The persons lastname is: $lastName
" );
print( "The persons description is: $description
" );
}
}


include_once( "ezxml/classes/ezxml.php" );

$xmlDocument =
"





Coder.






Coder.


";

$tree =& eZXML::domTree( $xmlDocument, array( "TrimWhiteSpace" => true ) );

foreach ( $tree->children as $document )
{
// parse the document
if ( $document->name == "document" )
{
// get the document version attribute
foreach ( $document->attributes as $documentAttr )
{
if ( $documentAttr->name == "version" )
{
print( "Found document with version: " . $documentAttr->content . "
" );
}
}

// parse all persons
foreach ( $document->children as $person )
{
if ( $person->name == "person" )
{
print( "Found a new person
" );

$firstName = "";
$lastName = "";
$descriptionName = "";

// get the name and description
foreach ( $person->children as $personAttribute )
{
switch ( $personAttribute->name )
{
case "firstname" :
{
$firstName = getAttrValue( $personAttribute, "value" );
}break;

case "lastname" :
{
$lastName = getAttrValue( $personAttribute, "value" );
}break;

case "description" :
{
// get the description text
foreach ( $personAttribute->children as $description )
{
if ( $description->type == 3 )
{
$description = $description->content;
}
}
}break;
}
}

print( "The persons firstname is: $firstName
" );
print( "The persons lastname is: $lastName
" );
print( "The persons description is: $description
" );
}
}
}
}

/*!
Function to fetch an attribute value.
Will return the value of the attribute if found. False if not found.
*/
function getAttrValue( $node, $attrName )
{
$ret = false;

foreach ( $node->attributes as $nodeAttr )
{
if ( $nodeAttr->name == $attrName )
{
$ret = $nodeAttr->content;
}
}
return $ret;
}


This code will produce the following output:

Found document with version: 42
Found a new person
The persons firstname is: Bård
The persons lastname is: Farstad
The persons description is: Coder.
Found a new person
The persons firstname is: Christoffer A.
The persons lastname is: Elo
The persons description is: Coder2.


Using XML is simple and straightforward with the eZ xml class. You don't need any external libraries, the class produce the same document tree as you would get from the XML functions in PHP (which all need external libraries). It is also the only PHP XML parser class which returns the same object tree as the library functions, making it easy to use your programs both on sites where XML is compiled into PHP and sites where it isn't.

Wednesday, September 26, 2007

PuTTY: A Free Telnet/SSH Client

I finally build putty for my Motorola Q VGA Phone Windows Mobile 5 Phone. The migration from eVC to visual studio 8 went well. I just need to fix some minor issue with some keyboard key not interpreted correctly when the host name need to be provided and figure out how to send specific keyboard combination key at once such as ctrl+c from a menu. Finally provide an installer to be able to have an icon in the tray. Without icons it is not possible for me to come back to putty once i need to answer a call.

Friday, August 31, 2007

Scene Cut Detection

Back in time i use to integrate on Windows the following software issued by
this famous French Laboratory. The purpose was to perform scene cut detection
thanks to a frame by frame video engine written for Windows and DirectShow.
The algorithm was very CPU intensive but pretty interesting.

http://www.irisa.fr/vista/Themes/Logiciel/MdShots/MdShots.english.html

Monday, August 13, 2007

HTTP performance testing with httperf, autobench

HTTP performance testing with httperf, autobench


* httperf is a benchmarking tool that measures the HTTP request throughput of a web server. The way it achieves this is by sending requests to the server at a fixed rate and measuring the rate at which replies arrive. Running the test several times and with monotonically increasing request rates, one can see the reply rate level off when the server becomes saturated, i.e., when it is operating at its full capacity.
* autobench is a Perl wrapper around httperf. It runs httperf a number of times against a Web server, increasing the number of requested connections per second on each iteration, and extracts the significant data from the httperf output, delivering a CSV format file which can be imported directly into a spreadsheet for analysis/graphing.


I ran a series of autobench/httperf and openload tests against a Web site I'll call site2 in the following discussion (site2 is a beta version of a site I'll call site1). For comparison purposes, I also ran similar tests against site1 and against www.example.com. The machine I ran the tests from is a Red Hat 9 Linux server co-located in downtown Los Angeles.

Here is an example of running httperf against www.example.com:

# httperf --server=www.example.com --rate=10 --num-conns=500

httperf --client=0/1 --server=www.example.com --port=80 --uri=/ --rate=10 --send-buffer=4096 --recv-buffer=16384 --num-conns=500 --num-calls=1
Maximum connect burst length: 1

Total: connections 500 requests 500 replies 500 test-duration 50.354 s

Connection rate: 9.9 conn/s (100.7 ms/conn, <=8 concurrent connections)
Connection time [ms]: min 449.7 avg 465.1 max 2856.6 median 451.5 stddev 132.1
Connection time [ms]: connect 74.1
Connection length [replies/conn]: 1.000

Request rate: 9.9 req/s (100.7 ms/req)
Request size [B]: 65.0

Reply rate [replies/s]: min 9.2 avg 9.9 max 10.0 stddev 0.3 (10 samples)
Reply time [ms]: response 88.1 transfer 302.9
Reply size [B]: header 274.0 content 54744.0 footer 2.0 (total 55020.0)
Reply status: 1xx=0 2xx=500 3xx=0 4xx=0 5xx=0

CPU time [s]: user 15.65 system 34.65 (user 31.1% system 68.8% total 99.9%)
Net I/O: 534.1 KB/s (4.4*10^6 bps)

Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0

The 3 arguments I specified on the command line are:

* server: the name or IP address of your Web site (you can also specify a particular URL via the --uri argument)
* rate: specifies the number of HTTP requests/second sent to the Web server -- indicates the number of concurrent clients accessing the server
* num-conns: specifies how many total HTTP connections will be made during the test run -- this is a cumulative number, so the higher the number of connections, the longer the test run

Here is a detailed interpretation of an httperf test run. In short, the main numbers to look for are the connection rate, the request rate and the reply rate. Ideally, you would like to see that all these numbers are very close to the request rate specified on the command line. If the actual request rate and the reply rate start to decline, that's a sign your server became saturated and can't handle any new connections. That could also be a sign that your client became saturated, so that's why it's better to test your client against a fast Web site in order to gauge how many outgoing HTTP requests can be sustained by your client.

Autobench is a simple Perl script that facilitates multiple runs of httperf and automatically increases the HTTP request rate. Configuration of autobench can be achieved for example by means of the ~/.autobench.conf file. Here is how my file looks like:

# Autobench Configuration File

# host1, host2
# The hostnames of the servers under test
# Eg. host1 = iis.test.com
# host2 = apache.test.com

host1 = testhost1
host2 = testhost2

# uri1, uri2
# The URI to test (relative to the document root). For a fair comparison
# the files should be identical (although the paths to them may differ on the
# different hosts)

uri1 = /
uri2 = /

# port1, port2
# The port number on which the servers are listening

port1 = 80
port2 = 80

# low_rate, high_rate, rate_step
# The 'rate' is the number of number of connections to open per second.
# A series of tests will be conducted, starting at low rate,
# increasing by rate step, and finishing at high_rate.
# The default settings test at rates of 20,30,40,50...180,190,200

low_rate = 10
high_rate = 50
rate_step = 10

# num_conn, num_call
# num_conn is the total number of connections to make during a test
# num_call is the number of requests per connection
# The product of num_call and rate is the the approximate number of
# requests per second that will be attempted.

num_conn = 200
#num_call = 10
num_call = 1

# timeout sets the maximimum time (in seconds) that httperf will wait
# for replies from the web server. If the timeout is exceeded, the
# reply concerned is counted as an error.

timeout = 60

# output_fmt
# sets the output type - may be either "csv", or "tsv";

output_fmt = csv

## Config for distributed autobench (autobench_admin)
# clients
# comma separated list of the hostnames and portnumbers for the
# autobench clients. No whitespace can appear before or after the commas.
# clients = bench1.foo.com:4600,bench2.foo.com:4600,bench3.foo.com:4600

clients = localhost:4600

The only variable I usually tweak from one test run to another is num_conn, which I set to the desired number of total HTTP connections to the server for that test run. In the example file above it is set to 200.

I changed the default num_call value from 10 to 1 (num_call specifies the number of HTTP requests per connection; I like to set it to 1 to keep things simple). I started my test runs with low_rate set to 10, high_rate set to 50 and rate_step set to 10. What this means is that autobench will run httperf 5 times, starting with 10 requests/sec and going up to 50 requests/sec in increments of 10.

When running the following command line...

# autobench --single_host --host1=www.example.com --file=example.com.csv

Wednesday, August 08, 2007

Source Code Beautifier for C, C++, C#, D, Java, and Pawn

A must to use !

http://uncrustify.sourceforge.net/

how to convert IIS log to apache log using perl

# http://www.jammed.com/~jwa/hacks/iis2apache/iis2apache

# 1. Go to Start -> Control Panel -> Administrative Tools
# 2. Run Internet Information Services (IIS).
# 3. Find your Web site under the tree on the left.
# 4. Right-click on it and choose Properties.
# 5. On the Web site tab, you will see an option near the bottom that says "Active #Log Format." Click on the Properties button.
# 6. At the bottom of the General Properties tab, you will see a box that contains #the log file directory and the log file name. The full log path is comprised of the #log file directory plus the first part of the log file name.
#
#For example, if the dialog box displayed the following values:
#
# * Log file directory: C:\Windows\System32\LogFiles
# * Log file name: W3SVC1\exyymmdd.log
#Then your full log path would be:
#C:\Windows\System32\LogFiles\W3SVC1

#!/usr/bin/perl
#
# make an IIS log look strikingly like an apache log
# we make a vague attempt to interpret the Fields: header
# in an IIS logfile and use that to make IIS fields match up
# with apache fields.
#
# jwa@jammed.com 12 Dec 2000
#

while ($arg = shift @ARGV) {
$tzoffset = shift @ARGV if ($arg eq "--faketz");
$vhost = shift @ARGV if ($arg eq "--vhost");
$debug = 1 if ($arg eq "--debug"); # show field interpretation
}

if ($tzoffset eq "") {
print STDERR "Will use -0000 as a fake tzoffset\n";
$tzoffset = "-0000";
}

# build month hash
@m = ('Jan','Feb','Mar','Apr','May','Jun','Jul','Aug','Sep','Oct','Nov','Dec');
while ($m = shift @m) {
$month{++$n} = $m;
}

# an IIS log adheres to what's defined by 'Fields;'
# attempt to parse this, tagging


LINE:
while ($line = ) {
$line =~ s/\r|\n//g; # cooky DOS format
if ($line =~ /^#Fields: /) {
@line = split(" ", $line);
shift @line; # shifts of #Fields
# build a hash so we can look up a fieldname and
# have it return a position in the string
undef %fieldh;
$n = 0; # zero-based array for split
while ($l = shift @line) {
$fieldh{$l} = $n++;
print STDERR "$l is position $fieldh{$l}\n" if ($debug);
}

}
next LINE if ($line =~ /^#/);

#Fields: date time c-ip cs-username s-sitename s-computername s-ip cs-method cs-
#uri-stem cs-uri-query sc-status sc-win32-status sc-bytes cs-bytes time-taken
#s-port cs-version cs(User-Agent) cs(Cookie) cs(Referer)

# this is really slow.

$date = yankfield("date");
$time = yankfield("time");
$ip = yankfield("c-ip");
$username = yankfield("cs-username");
$method = yankfield("cs-method");
$stem = yankfield("cs-uri-stem");
$query = yankfield("cs-uri-query");
$status = yankfield("sc-status");
#$bytes = yankfield("cs-bytes");
# Which is it? sc-bytes or cs-bytes?
# cs-bytes only appears in some of the IIS logs I've seen.
# I'll assume that sc-bytes is "server->client bytes", which
# is what we want anyway.
$bytes = yankfield("sc-bytes"); # I'm gonna go with this.
$useragent = yankfield("cs(User-Agent)");
$referer = yankfield("cs(Referer)");

$useragent =~ s/\+/\ /g;

# our modified CLF sez:
# IP - - [DD/MMM/YYYY:HH:MM:SS TZOFFSET] "method stem[?query]" status bytes "referer" "user agent" "vhost"

# convert date
# 2000-07-19 00:00:01
($y, $m, $d) = split("-", $date);
$m =~ s/^0//g;
$mname = $month{$m};

# build url
$url = $stem;
if ($query ne "-") {
$url .= "?$query";
}

# all done, print it out
print "$ip - - [${d}/${mname}/${y}:${time} ${tzoffset}] \"$method $url\" $status $bytes \"$referer\" \"$useragent\" \"${vhost}\"\n";
}


# return the proper field, or "-" if it's not defined.
# (unfortunately ($date) = (split(" ", $line))[$fieldh{date}];
# will return element 0 if $fieldh{date} is undefined . . .)

sub yankfield {
my ($field) = shift @_;

print STDERR "Looking at $field; position [$fieldh{$field}]\n" if ($debug);

if ($fieldh{$field} ne "") {
return (split(" ", $line))[$fieldh{$field}];
} else {
print STDERR "$field undefined\n" if ($debug);
return "-";
}
}

Tuesday, July 17, 2007

Microsoft PhotoTours

This is an amazing piece of technology, presented at a TED conference.
http://www.collegehumor.com/video:1762315

More information is available on the Microsoft Research Website.

http://research.microsoft.com/IVM/PhotoTours/

There is also a live preview of this software available online. All you need to do is install a 5.5 MB Photosynth ActiveX. The address is :

http://labs.live.com/photosynth/



http://www.crazyleafdesign.com/blog/photosynth-prototype/

Friday, July 13, 2007

Peter Löthberg mother's Sigbritt, 75, has world's fastest broadband

Sigbritt, 75, has world's fastest broadband
Published: 12th July 2007 11:07 CET
Online: http://www.thelocal.se/7869/

A 75 year old woman from Karlstad in central Sweden has been thrust into the IT history books - with the world's fastest internet connection.

Sigbritt Löthberg's home has been supplied with a blistering 40 Gigabits per second connection, many thousands of times faster than the average residential link and the first time ever that a home user has experienced such a high speed.

But Sigbritt, who had never had a computer until now, is no ordinary 75 year old. She is the mother of Swedish internet legend Peter Löthberg who, along with Karlstad Stadsnät, the local council's network arm, has arranged the connection.

"This is more than just a demonstration," said network boss Hafsteinn Jonsson.

"As a network owner we're trying to persuade internet operators to invest in faster connections. And Peter Löthberg wanted to show how you can build a low price, high capacity line over long distances," he told The Local.

Sigbritt will now be able to enjoy 1,500 high definition HDTV channels simultaneously. Or, if there is nothing worth watching there, she will be able to download a full high definition DVD in just two seconds.

The secret behind Sigbritt's ultra-fast connection is a new modulation technique which allows data to be transferred directly between two routers up to 2,000 kilometres apart, with no intermediary transponders.

According to Karlstad Stadsnät the distance is, in theory, unlimited - there is no data loss as long as the fibre is in place.

"I want to show that there are other methods than the old fashioned ways such as copper wires and radio, which lack the possibilities that fibre has," said Peter Löthberg, who now works at Cisco.

Cisco contributed to the project but the point, said Hafsteinn Jonsson, is that fibre technology makes such high speed connections technically and commercially viable.

"The most difficult part of the whole project was installing Windows on Sigbritt's PC," said Jonsson.

The Local (news@thelocal.se/08 656 6518)

Monday, July 09, 2007

chown on linux using c++

lchown
NAME
lchown - change the owner and group of a symbolic link

#include

int lchown(const char *path, uid_t owner, gid_t group);

DESCRIPTION
The lchown() function shall be equivalent to chown(), except in the case where the named file is a symbolic link. In this case, lchown() shall change the ownership of the symbolic link file itself, while chown() changes the ownership of the file or directory to which the symbolic link refers.

RETURN VALUE
Upon successful completion, lchown() shall return 0. Otherwise, it shall return -1 and set errno to indicate an error.

ERRORS
The lchown() function shall fail if:

EACCES
Search permission is denied on a component of the path prefix of path.
EINVAL
The owner or group ID is not a value supported by the implementation.
ELOOP
A loop exists in symbolic links encountered during resolution of the path argument.
ENAMETOOLONG
The length of a pathname exceeds {PATH_MAX} or a pathname component is longer than {NAME_MAX}.
ENOENT
A component of path does not name an existing file or path is an empty string.
ENOTDIR
A component of the path prefix of path is not a directory.
EOPNOTSUPP
The path argument names a symbolic link and the implementation does not support setting the owner or group of a symbolic link.
EPERM
The effective user ID does not match the owner of the file and the process does not have appropriate privileges.
EROFS
The file resides on a read-only file system.
The lchown() function may fail if:

EIO
An I/O error occurred while reading or writing to the file system.
EINTR
A signal was caught during execution of the function.
ELOOP
More than {SYMLOOP_MAX} symbolic links were encountered during resolution of the path argument.
ENAMETOOLONG
Pathname resolution of a symbolic link produced an intermediate result whose length exceeds {PATH_MAX}.
The following sections are informative.

EXAMPLES
Changing the Current Owner of a File
The following example shows how to change the ownership of the symbolic link named /modules/pass1 to the user ID associated with "jones" and the group ID associated with "cnd".

The numeric value for the user ID is obtained by using the getpwnam() function. The numeric value for the group ID is obtained by using the getgrnam() function.

#include
#include
#include
#include

struct passwd *pwd;
struct group *grp;
char *path = "/modules/pass1";
...
pwd = getpwnam("Sri_Chinmoy");
grp = getgrnam("guru");
lchown(path, pwd->pw_uid, grp->gr_gid);

Advanced Unix Programming Source

A set of handy function and class to perform some basics
UNIX tasks in c++

http://basepath.com/aup/ex/index.html

Thursday, July 05, 2007

Thread POSIX Thread basic

Further Threads Programming:Thread Attributes (POSIX)

PART I
API covered is for POSIX threads only. Otherwise, the functionality for Solaris threads and pthreads is largely the same.

Attributes

Attributes are a way to specify behavior that is different from the default. When a thread is created with pthread_create() or when a synchronization variable is initialized, an attribute object can be specified. Note: however that the default atributes are usually sufficient for most applications.

Important Note: Attributes are specified only at thread creation time; they cannot be altered while the thread is being used.

Thus three functions are usually called in tandem

* Thread attibute intialisation -- pthread_attr_init() create a default pthread_attr_t tattr
* Thread attribute value change (unless defaults appropriate) -- a variety of pthread_attr_*() functions are available to set individual attribute values for the pthread_attr_t tattr structure. (see below).
* Thread creation -- a call to pthread_create() with approriate attribute values set in a pthread_attr_t tattr structure.

The following code fragment should make this point clearer:

#include

pthread_attr_t tattr;
pthread_t tid;
void *start_routine;
void arg
int ret;

/* initialized with default attributes */
ret = pthread_attr_init(&tattr);

/* call an appropriate functions to alter a default value */
ret = pthread_attr_*(&tattr,SOME_ATRIBUTE_VALUE_PARAMETER);

/* create the thread */
ret = pthread_create(&tid, &tattr, start_routine, arg);

In order to save space, code examples mainly focus on the attribute setting functions and the intializing and creation functions are ommitted. These must of course be present in all actual code fragtments.

An attribute object is opaque, and cannot be directly modified by assignments. A set of functions is provided to initialize, configure, and destroy each object type. Once an attribute is initialized and configured, it has process-wide scope. The suggested method for using attributes is to configure all required state specifications at one time in the early stages of program execution. The appropriate attribute object can then be referred to as needed. Using attribute objects has two primary advantages:

* First, it adds to code portability. Even though supported attributes might vary between implementations, you need not modify function calls that create thread entities because the attribute object is hidden from the interface. If the target port supports attributes that are not found in the current port, provision must be made to manage the new attributes. This is an easy porting task though, because attribute objects need only be initialized once in a well-defined location.
* Second, state specification in an application is simplified. As an example, consider that several sets of threads might exist within a process, each providing a separate service, and each with its own state requirements. At some point in the early stages of the application, a thread attribute object can be initialized for each set. All future thread creations will then refer to the attribute object initialized for that type of thread. The initialization phase is simple and localized, and any future modifications can be made quickly and reliably.

Attribute objects require attention at process exit time. When the object is initialized, memory is allocated for it. This memory must be returned to the system. The pthreads standard provides function calls to destroy attribute objects.

Initializing Thread Attributes


The function pthread_attr_init() is used to initialize object attributes to their default values. The storage is allocated by the thread system during execution.

The function is prototyped by:

int pthread_attr_init(pthread_attr_t *tattr);

An example call to this function is:

#include
pthread_attr_t tattr;
int ret;
/* initialize an attribute to the default value */
ret = pthread_attr_init(&tattr);

The default values for attributes (tattr) are:

Attribute Value Result
scope PTHREAD_SCOPE_PROCESS New thread is unbound - not permanently attached to LWP.
detachstate PTHREAD_CREATE_JOINABLE Exit status and thread are preserved after the thread
terminates.
stackaddr NULL New thread has system-allocated stack address. 

stacksize 1 megabyte New thread has system-defined stack size.
priority New thread inherits parent thread priority.
inheritsched PTHREAD_INHERIT_SCHED New thread inherits parent thread scheduling priority.
schedpolicy SCHED_OTHER New thread uses Solaris-defined fixed priority scheduling;
threads run until preempted by a higher-priority thread or until they block or yield.

This function zero after completing successfully. Any other returned value indicates that an error occurred. If the following condition occurs, the function fails and returns an error value (to errno).

Destroying Thread Attributes

The function pthread_attr_destroy() is used to remove the storage allocated during initialization. The attribute object becomes invalid. It is prototyped by:

int pthread_attr_destroy(pthread_attr_t *tattr);

A sample call to this functions is:

#include
pthread_attr_t tattr;
int ret;
/* destroy an attribute */
ret = pthread_attr_destroy(&tattr);

Attribites are declared as for pthread_attr_init() above.

pthread_attr_destroy() returns zero after completing successfully. Any other returned value indicates that an error occurred.

Thread's Detach State

When a thread is created detached (PTHREAD_CREATE_DETACHED), its thread ID and other resources can be reused as soon as the thread terminates.

If you do not want the calling thread to wait for the thread to terminate then call the function pthread_attr_setdetachstate().

When a thread is created nondetached (PTHREAD_CREATE_JOINABLE), it is assumed that you will be waiting for it. That is, it is assumed that you will be executing a pthread_join() on the thread. Whether a thread is created detached or nondetached, the process does not exit until all threads have exited.

pthread_attr_setdetachstate() is prototyped by:

int pthread_attr_setdetachstate(pthread_attr_t *tattr,int detachstate);

pthread_attr_setdetachstate() returns zero after completing successfully. Any other returned value indicates that an error occurred. If the following condition occurs, the function fails and returns the corresponding value.

An example call to detatch a thread with this function is:

#include
pthread_attr_t tattr;
int ret;
/* set the thread detach state */
ret = pthread_attr_setdetachstate(&tattr,PTHREAD_CREATE_DETACHED);

Note - When there is no explicit synchronization to prevent it, a newly created, detached thread can die and have its thread ID reassigned to another new thread before its creator returns from pthread_create(). For nondetached (PTHREAD_CREATE_JOINABLE) threads, it is very important that some thread join with it after it terminates -- otherwise the resources of that thread are not released for use by new threads. This commonly results in a memory leak. So when you do not want a thread to be joined, create it as a detached thread.

It is quite common that you will wish to create a thread which is detatched from creation. The following code illustrates how this may be achieved with the standard calls to initialise and set and then create a thread:

#include
pthread_attr_t tattr;
pthread_t tid;
void *start_routine;
void arg
int ret;

/* initialized with default attributes */
ret = pthread_attr_init(&tattr);
ret = pthread_attr_setdetachstate(&tattr,PTHREAD_CREATE_DETACHED);
ret = pthread_create(&tid, &tattr, start_routine, arg);

The function pthread_attr_getdetachstate() may be used to retrieve the thread create state, which can be either detached or joined. It is prototyped by:

int pthread_attr_getdetachstate(const pthread_attr_t *tattr, int *detachstate);

pthread_attr_getdetachstate() returns zero after completing successfully. Any other returned value indicates that an error occurred.

An example call to this fuction is:

#include
pthread_attr_t tattr;
int detachstate;
int ret;

/* get detachstate of thread */
ret = pthread_attr_getdetachstate (&tattr, &detachstate);

Full Article here

http://www.cs.cf.ac.uk/Dave/C/node30.html


PART II

What Is a Thread? Why Use Threads

A thread is a semi-process, that has its own stack, and executes a given piece of code. Unlike a real process, the thread normally shares its memory with other threads (where as for processes we usually have a different memory area for each one of them). A Thread Group is a set of threads all executing inside the same process. They all share the same memory, and thus can access the same global variables, same heap memory, same set of file descriptors, etc. All these threads execute in parallel (i.e. using time slices, or if the system has several processors, then really in parallel).
The advantage of using a thread group instead of a normal serial program is that several operations may be carried out in parallel, and thus events can be handled immediately as they arrive (for example, if we have one thread handling a user interface, and another thread handling database queries, we can execute a heavy query requested by the user, and still respond to user input while the query is executed).
The advantage of using a thread group over using a process group is that context switching between threads is much faster then context switching between processes (context switching means that the system switches from running one thread or process, to running another thread or process). Also, communications between two threads is usually faster and easier to implement then communications between two processes.
On the other hand, because threads in a group all use the same memory space, if one of them corrupts the contents of its memory, other threads might suffer as well. With processes, the operating system normally protects processes from one another, and thus if one corrupts its own memory space, other processes won't suffer. Another advantage of using processes is that they can run on different machines, while all the threads have to run on the same machine (at least normally).

Creating And Destroying Threads

When a multi-threaded program starts executing, it has one thread running, which executes the main() function of the program. This is already a full-fledged thread, with its own thread ID. In order to create a new thread, the program should use the pthread_create() function. Here is how to use it:


#include        /* standard I/O routines                 */
#include      /* pthread functions and data structures */

/* function to be executed by the new thread */
void*
do_loop(void* data)
{
    int i;

    int i;   /* counter, to print numbers */
    int j;   /* counter, for delay        */
    int me = *((int*)data);     /* thread identifying number */

    for (i=0; i<10 color="brown" font="" for="" i="" j="">/* delay loop */
; printf("'%d' - Got '%d'\n", me, i); } /* terminate the thread */ pthread_exit(NULL); } /* like any C program, program's execution begins in main */ int main(int argc, char* argv[]) { int thr_id; /* thread ID for the newly created thread */ pthread_t p_thread; /* thread's structure */ int a = 1; /* thread 1 identifying number */ int b = 2; /* thread 2 identifying number */ /* create a new thread that will execute 'do_loop()' */ thr_id = pthread_create(&p_thread, NULL, do_loop, (void*)&a); /* run 'do_loop()' in the main thread as well */ do_loop((void*)&b); /* NOT REACHED */ return 0; }
A few notes should be mentioned about this program:
  1. Note that the main program is also a thread, so it executes the do_loop() function in parallel to the thread it creates.
  2. pthread_create() gets 4 parameters. The first parameter is used by pthread_create() to supply the program with information about the thread. The second parameter is used to set some attributes for the new thread. In our case we supplied a NULL pointer to tellpthread_create() to use the default values. The third parameter is the name of the function that the thread will start executing. The forth parameter is an argument to pass to this function. Note the cast to a 'void*'. It is not required by ANSI-C syntax, but is placed here for clarification.
  3. The delay loop inside the function is used only to demonstrate that the threads are executing in parallel. Use a larger delay value if your CPU runs too fast, and you see all the printouts of one thread before the other.
  4. The call to pthread_exit() Causes the current thread to exit and free any thread-specific resources it is taking. There is no need to use this call at the end of the thread's top function, since when it returns, the thread would exit automatically anyway. This function is useful if we want to exit a thread in the middle of its execution.
In order to compile a multi-threaded program using gcc, we need to link it with the pthreads library. Assuming you have this library already installed on your system, here is how to compile our first program:

gcc pthread_create.c -o pthread_create -lpthread 

The source code for this program may be found in the pthread_create.c file.

Synchronizing Threads With Mutexes

One of the basic problems when running several threads that use the same memory space, is making sure they don't "step on each other's toes". By this we refer to the problem of using a data structure from two different threads.
For instance, consider the case where two threads try to update two variables. One tries to set both to 0, and the other tries to set both to 1. If both threads would try to do that at the same time, we might get with a situation where one variable contains 1, and one contains 0. This is because a context-switch (we already know what this is by now, right?) might occur after the first tread zeroed out the first variable, then the second thread would set both variables to 1, and when the first thread resumes operation, it will zero out the second variable, thus getting the first variable set to '1', and the second set to '0'.

What Is A Mutex?

A basic mechanism supplied by the pthreads library to solve this problem, is called a mutex. A mutex is a lock that guarantees three things:
  1. Atomicity - Locking a mutex is an atomic operation, meaning that the operating system (or threads library) assures you that if you locked a mutex, no other thread succeeded in locking this mutex at the same time.
  2. Singularity - If a thread managed to lock a mutex, it is assured that no other thread will be able to lock the thread until the original thread releases the lock.
  3. Non-Busy Wait - If a thread attempts to lock a thread that was locked by a second thread, the first thread will be suspended (and will not consume any CPU resources) until the lock is freed by the second thread. At this time, the first thread will wake up and continue execution, having the mutex locked by it.
From these three points we can see how a mutex can be used to assure exclusive access to variables (or in general critical code sections). Here is some pseudo-code that updates the two variables we were talking about in the previous section, and can be used by the first thread:
lock mutex 'X1'.
set first variable to '0'.
set second variable to '0'.
unlock mutex 'X1'.


Meanwhile, the second thread will do something like this:

lock mutex 'X1'.
set first variable to '1'.
set second variable to '1'.
unlock mutex 'X1'.


Assuming both threads use the same mutex, we are assured that after they both ran through this code, either both variables are set to '0', or both are set to '1'. You'd note this requires some work from the programmer - If a third thread was to access these variables via some code that does not use this mutex, it still might mess up the variable's contents. Thus, it is important to enclose all the code that accesses these variables in a small set of functions, and always use only these functions to access these variables.



Creating And Initializing A Mutex

In order to create a mutex, we first need to declare a variable of type pthread_mutex_t , and then initialize it. The simplest way it by assigning it the PTHREAD_MUTEX_INITIALIZER constant. So we'll use a code that looks something like this:


pthread_mutex_t a_mutex = PTHREAD_MUTEX_INITIALIZER;


One note should be made here: This type of initialization creates a mutex called 'fast mutex'. This means that if a thread locks the mutex and then tries to lock it again, it'll get stuck - it will be in a deadlock.


There is another type of mutex, called 'recursive mutex', which allows the thread that locked it, to lock it several more times, without getting blocked (but other threads that try to lock the mutex now will get blocked). If the thread then unlocks the mutex, it'll still be locked, until it is unlocked the same amount of times as it was locked. This is similar to the way modern door locks work - if you turned it twice clockwise to lock it, you need to turn it twice counter-clockwise to unlock it. This kind of mutex can be created by assigning the constantPTHREAD_RECURSIVE_MUTEX_INITIALIZER_NP to a mutex variable.

Locking And Unlocking A Mutex

In order to lock a mutex, we may use the function pthread_mutex_lock(). This function attempts to lock the mutex, or block the thread if the mutex is already locked by another thread. In this case, when the mutex is unlocked by the first process, the function will return with the mutex locked by our process. Here is how to lock a mutex (assuming it was initialized earlier):


int rc = pthread_mutex_lock(&a_mutex);
if (rc) { /* an error has occurred */
    perror("pthread_mutex_lock");
    pthread_exit(NULL);
}
/* mutex is now locked - do your stuff. */
.
.



After the thread did what it had to (change variables or data structures, handle file, or whatever it intended to do), it should free the mutex, using the pthread_mutex_unlock() function, like this:


rc = pthread_mutex_unlock(&a_mutex);
if (rc) {
    perror("pthread_mutex_unlock");
    pthread_exit(NULL);
}


Destroying A Mutex

After we finished using a mutex, we should destroy it. Finished using means no thread needs it at all. If only one thread finished with the mutex, it should leave it alive, for the other threads that might still need to use it. Once all finished using it, the last one can destroy it using the pthread_mutex_destroy() function:


rc = pthread_mutex_destroy(&a_mutex);


After this call, this variable (a_mutex) may not be used as a mutex any more, unless it is initialized again. Thus, if one destroys a mutex too early, and another thread tries to lock or unlock it, that thread will get a EINVAL error code from the lock or unlock function.



Using A Mutex - A Complete Example

After we have seen the full life cycle of a mutex, lets see an example program that uses a mutex. The program introduces two employees competing for the "employee of the day" title, and the glory that comes with it. To simulate that in a rapid pace, the program employs 3 threads: one that promotes Danny to "employee of the day", one that promotes Moshe to that situation, and a third thread that makes sure that the employee of the day's contents is consistent (i.e. contains exactly the data of one employee).
Two copies of the program are supplied. One that uses a mutex, and one that does not. Try them both, to see the differences, and be convinced that mutexes are essential in a multi-threaded environment.
The programs themselves are in the files accompanying this tutorial. The one that uses a mutex is employee-with-mutex.c. The one that does not use a mutex is employee-without-mutex.c. Read the comments inside the source files to get a better understanding of how they work.

Starvation And Deadlock Situations

Again we should remember that pthread_mutex_lock() might block for a non-determined duration, in case of the mutex being already locked. If it remains locked forever, it is said that our poor thread is "starved" - it was trying to acquire a resource, but never got it. It is up to the programmer to ensure that such starvation won't occur. The pthread library does not help us with that.

The pthread library might, however, figure out a "deadlock". A deadlock is a situation in which a set of threads are all waiting for resources taken by other threads, all in the same set. Naturally, if all threads are blocked waiting for a mutex, none of them will ever come back to life again. The pthread library keeps track of such situations, and thus would fail the last thread trying to call pthread_mutex_lock(), with an error of type EDEADLK. The programmer should check for such a value, and take steps to solve the deadlock somehow.

Refined Synchronization - Condition Variables

As we've seen before with mutexes, they allow for simple coordination - exclusive access to a resource. However, we often need to be able to make real synchronization between threads:
  • In a server, one thread reads requests from clients, and dispatches them to several threads for handling. These threads need to be notified when there is data to process, otherwise they should wait without consuming CPU time.
  • In a GUI (Graphical User Interface) Application, one thread reads user input, another handles graphical output, and a third thread sends requests to a server and handles its replies. The server-handling thread needs to be able to notify the graphics-drawing thread when a reply from the server arrived, so it will immediately show it to the user. The user-input thread needs to be always responsive to the user, for example, to allow her to cancel long operations currently executed by the server-handling thread.
All these examples require the ability to send notifications between threads. This is where condition variables are brought into the picture.


What Is A Condition Variable?

A condition variable is a mechanism that allows threads to wait (without wasting CPU cycles) for some even to occur. Several threads may wait on a condition variable, until some other thread signals this condition variable (thus sending a notification). At this time, one of the threads waiting on this condition variable wakes up, and can act on the event. It is possible to also wake up all threads waiting on this condition variable by using a broadcast method on this variable.
Note that a condition variable does not provide locking. Thus, a mutex is used along with the condition variable, to provide the necessary locking when accessing this condition variable.

Creating And Initializing A Condition Variable

Creation of a condition variable requires defining a variable of type pthread_cond_t, and initializing it properly. Initialization may be done with either a simple use of a macro named PTHREAD_COND_INITIALIZER or the usage of the pthread_cond_init() function. We will show the first form here:

pthread_cond_t got_request = PTHREAD_COND_INITIALIZER; 

This defines a condition variable named 'got_request', and initializes it.
Note: since the PTHREAD_COND_INITIALIZER is actually a structure, it may be used to initialize a condition variable only when it is declared. In order to initialize it during runtime, one must use the pthread_cond_init() function.

Signaling A Condition Variable

In order to signal a condition variable, one should either the pthread_cond_signal() function (to wake up a only one thread waiting on this variable), or the pthread_cond_broadcast() function (to wake up all threads waiting on this variable). Here is an example using signal, assuming 'got_request' is a properly initialized condition variable:

int rc = pthread_cond_signal(&got_request); 

Or by using the broadcast function:

int rc = pthread_cond_broadcast(&got_request); 

When either function returns, 'rc' is set to 0 on success, and to a non-zero value on failure. In such a case (failure), the return value denotes the error that occured (EINVAL denotes that the given parameter is not a condition variable. ENOMEM denotes that the system has run out of memory.
Note: success of a signaling operation does not mean any thread was awakened - it might be that no thread was waiting on the condition variable, and thus the signaling does nothing (i.e. the signal is lost).
It is also not remembered for future use - if after the signaling function returns another thread starts waiting on this condition variable, a further signal is required to wake it up.


Waiting On A Condition Variable

If one thread signals the condition variable, other threads would probably want to wait for this signal. They may do so using one of two functions, pthread_cond_wait() or pthread_cond_timedwait(). Each of these functions takes a condition variable, and a mutex (which should be locked before calling the wait function), unlocks the mutex, and waits until the condition variable is signaled, suspending the thread's execution. If this signaling causes the thread to awake (see discussion of pthread_cond_signal() earlier), the mutex is automagically locked again by the wait funciton, and the wait function returns.

The only difference between these two functions is that pthread_cond_timedwait() allows the programmer to specify a timeout for the waiting, after which the function always returns, with a proper error value (ETIMEDOUT) to notify that condition variable was NOT signaled before the timeout passed. The pthread_cond_wait() would wait indefinitely if it was never signaled.

Here is how to use these two functions. We make the assumption that 'got_request' is a properly initialized condition variable, and that 'request_mutex' is a properly initialized mutex. First, we try the pthread_cond_wait() function:


/* first, lock the mutex */
int rc = pthread_mutex_lock(&a_mutex);
if (rc) { /* an error has occurred */
    perror("pthread_mutex_lock");
    pthread_exit(NULL);
}
/* mutex is now locked - wait on the condition variable.             */
/* During the execution of pthread_cond_wait, the mutex is unlocked. */
rc = pthread_cond_wait(&got_request, &a_mutex);
if (rc == 0) { /* we were awakened due to the cond. variable being signaled */
               /* The mutex is now locked again by pthread_cond_wait()      */
    /* do your stuff... */
    .
}
/* finally, unlock the mutex */ 
pthread_mutex_unlock(&a_mutex);


Now an example using the pthread_cond_timedwait() function:


#include      /* struct timeval definition           */
#include        /* declaration of gettimeofday()       */

struct timeval  now;            /* time when we started waiting        */
struct timespec timeout;        /* timeout value for the wait function */
int             done;           /* are we done waiting?                */

/* first, lock the mutex */
int rc = pthread_mutex_lock(&a_mutex);
if (rc) { /* an error has occurred */
    perror("pthread_mutex_lock");
    pthread_exit(NULL);
}
/* mutex is now locked */

/* get current time */ 
gettimeofday(&now);
/* prepare timeout value */
timeout.tv_sec = now.tv_sec + 5
timeout.tv_nsec = now.tv_usec * 1000; /* timeval uses microseconds.         */
                                      /* timespec uses nanoseconds.         */
                                      /* 1 nanosecond = 1000 micro seconds. */

/* wait on the condition variable. */
/* we use a loop, since a Unix signal might stop the wait before the timeout */
done = 0;
while (!done) {
    /* remember that pthread_cond_timedwait() unlocks the mutex on entrance */
    rc = pthread_cond_timedwait(&got_request, &a_mutex, &timeout);
    switch(rc) {
        case 0:  /* we were awakened due to the cond. variable being signaled */
                 /* the mutex was now locked again by pthread_cond_timedwait. */
            /* do your stuff here... */
            .
            .
            done = 0;
            break;
        case ETIMEDOUT: /* our time is up */
            done = 0;
            break;
        default:        /* some error occurred (e.g. we got a Unix signal) */
            break;      /* break this switch, but re-do the while loop.   */
    }
}
/* finally, unlock the mutex */
pthread_mutex_unlock(&a_mutex);


As you can see, the timed wait version is way more complex, and thus better be wrapped up by some function, rather then being re-coded in every necessary location.


Note: it might be that a condition variable that has 2 or more threads waiting on it is signaled many times, and yet one of the threads waiting on it never awakened. This is because we are not guaranteed which of the waiting threads is awakened when the variable is signaled. It might be that the awakened thread quickly comes back to waiting on the condition variables, and gets awakened again when the variable is signaled again, and so on. The situation for the un-awakened thread is called 'starvation'. It is up to the programmer to make sure this situation does not occur if it implies bad behavior. Yet, in our server example from before, this situation might indicate requests are coming in a very slow pace, and thus perhaps we have too many threads waiting to service requests. In this case, this situation is actually good, as it means every request is handled immediately when it arrives.
Note 2: when the mutex is being broadcast (using pthread_cond_broadcast), this does not mean all threads are running together. Each of them tries to lock the mutex again before returning from their wait function, and thus they'll start running one by one, each one locking the mutex, doing their work, and freeing the mutex before the next thread gets its chance to run.

Destroying A Condition Variable

After we are done using a condition variable, we should destroy it, to free any system resources it might be using. This can be done using the pthread_cond_destroy(). In order for this to work, there should be no threads waiting on this condition variable. Here is how to use this function, again, assuming 'got_request' is a pre-initialized condition variable:


int rc = pthread_cond_destroy(&got_request);
if (rc == EBUSY) { /* some thread is still waiting on this condition variable */
    /* handle this case here... */
    .
    .
}


What if some thread is still waiting on this variable? depending on the case, it might imply some flaw in the usage of this variable, or just lack of proper thread cleanup code. It is probably good to alert the programmer, at least during debug phase of the program, of such a case. It might mean nothing, but it might be significant.



A Real Condition For A Condition Variable

A note should be taken about condition variables - they are usually pointless without some real condition checking combined with them. To make this clear, lets consider the server example we introduced earlier. Assume that we use the 'got_request' condition variable to signal that a new request has arrived that needs handling, and is held in some requests queue. If we had threads waiting on the condition variable when this variable is signaled, we are assured that one of these threads will awake and handle this request.
However, what if all threads are busy handling previous requests, when a new one arrives? the signaling of the condition variable will do nothing (since all threads are busy doing other things, NOT waiting on the condition variable now), and after all threads finish handling their current request, they come back to wait on the variable, which won't necessarily be signaled again (for example, if no new requests arrive). Thus, there is at least one request pending, while all handling threads are blocked, waiting for a signal.
In order to overcome this problem, we may set some integer variable to denote the number of pending requests, and have each thread check the value of this variable before waiting on the variable. If this variable's value is positive, some request is pending, and the thread should go and handle it, instead of going to sleep. Further more, a thread that handled a request, should reduce the value of this variable by one, to make the count correct.
Lets see how this affects the waiting code we have seen above.



/* number of pending requests, initially none */
int num_requests = 0;
.
.
/* first, lock the mutex */
int rc = pthread_mutex_lock(&a_mutex);
if (rc) { /* an error has occurred */
    perror("pthread_mutex_lock");
    pthread_exit(NULL);
}
/* mutex is now locked - wait on the condition variable */
/* if there are no requests to be handled.              */
rc = 0;
if (num_requests == 0)
    rc = pthread_cond_wait(&got_request, &a_mutex);
if (num_requests > 0 && rc == 0) { /* we have a request pending */
        /* do your stuff... */
        .
        .
        /* decrease count of pending requests */
        num_requests--;
    }
}
/* finally, unlock the mutex */
pthread_mutex_unlock(&a_mutex);

Full Article here

http://www.cs.kent.edu/~ruttan/sysprog/lectures/multi-thread/multi-thread.html