Thursday, July 05, 2007

Thread POSIX Thread basic

Further Threads Programming:Thread Attributes (POSIX)

PART I
API covered is for POSIX threads only. Otherwise, the functionality for Solaris threads and pthreads is largely the same.

Attributes

Attributes are a way to specify behavior that is different from the default. When a thread is created with pthread_create() or when a synchronization variable is initialized, an attribute object can be specified. Note: however that the default atributes are usually sufficient for most applications.

Important Note: Attributes are specified only at thread creation time; they cannot be altered while the thread is being used.

Thus three functions are usually called in tandem

* Thread attibute intialisation -- pthread_attr_init() create a default pthread_attr_t tattr
* Thread attribute value change (unless defaults appropriate) -- a variety of pthread_attr_*() functions are available to set individual attribute values for the pthread_attr_t tattr structure. (see below).
* Thread creation -- a call to pthread_create() with approriate attribute values set in a pthread_attr_t tattr structure.

The following code fragment should make this point clearer:

#include

pthread_attr_t tattr;
pthread_t tid;
void *start_routine;
void arg
int ret;

/* initialized with default attributes */
ret = pthread_attr_init(&tattr);

/* call an appropriate functions to alter a default value */
ret = pthread_attr_*(&tattr,SOME_ATRIBUTE_VALUE_PARAMETER);

/* create the thread */
ret = pthread_create(&tid, &tattr, start_routine, arg);

In order to save space, code examples mainly focus on the attribute setting functions and the intializing and creation functions are ommitted. These must of course be present in all actual code fragtments.

An attribute object is opaque, and cannot be directly modified by assignments. A set of functions is provided to initialize, configure, and destroy each object type. Once an attribute is initialized and configured, it has process-wide scope. The suggested method for using attributes is to configure all required state specifications at one time in the early stages of program execution. The appropriate attribute object can then be referred to as needed. Using attribute objects has two primary advantages:

* First, it adds to code portability. Even though supported attributes might vary between implementations, you need not modify function calls that create thread entities because the attribute object is hidden from the interface. If the target port supports attributes that are not found in the current port, provision must be made to manage the new attributes. This is an easy porting task though, because attribute objects need only be initialized once in a well-defined location.
* Second, state specification in an application is simplified. As an example, consider that several sets of threads might exist within a process, each providing a separate service, and each with its own state requirements. At some point in the early stages of the application, a thread attribute object can be initialized for each set. All future thread creations will then refer to the attribute object initialized for that type of thread. The initialization phase is simple and localized, and any future modifications can be made quickly and reliably.

Attribute objects require attention at process exit time. When the object is initialized, memory is allocated for it. This memory must be returned to the system. The pthreads standard provides function calls to destroy attribute objects.

Initializing Thread Attributes


The function pthread_attr_init() is used to initialize object attributes to their default values. The storage is allocated by the thread system during execution.

The function is prototyped by:

int pthread_attr_init(pthread_attr_t *tattr);

An example call to this function is:

#include
pthread_attr_t tattr;
int ret;
/* initialize an attribute to the default value */
ret = pthread_attr_init(&tattr);

The default values for attributes (tattr) are:

Attribute Value Result
scope PTHREAD_SCOPE_PROCESS New thread is unbound - not permanently attached to LWP.
detachstate PTHREAD_CREATE_JOINABLE Exit status and thread are preserved after the thread
terminates.
stackaddr NULL New thread has system-allocated stack address. 

stacksize 1 megabyte New thread has system-defined stack size.
priority New thread inherits parent thread priority.
inheritsched PTHREAD_INHERIT_SCHED New thread inherits parent thread scheduling priority.
schedpolicy SCHED_OTHER New thread uses Solaris-defined fixed priority scheduling;
threads run until preempted by a higher-priority thread or until they block or yield.

This function zero after completing successfully. Any other returned value indicates that an error occurred. If the following condition occurs, the function fails and returns an error value (to errno).

Destroying Thread Attributes

The function pthread_attr_destroy() is used to remove the storage allocated during initialization. The attribute object becomes invalid. It is prototyped by:

int pthread_attr_destroy(pthread_attr_t *tattr);

A sample call to this functions is:

#include
pthread_attr_t tattr;
int ret;
/* destroy an attribute */
ret = pthread_attr_destroy(&tattr);

Attribites are declared as for pthread_attr_init() above.

pthread_attr_destroy() returns zero after completing successfully. Any other returned value indicates that an error occurred.

Thread's Detach State

When a thread is created detached (PTHREAD_CREATE_DETACHED), its thread ID and other resources can be reused as soon as the thread terminates.

If you do not want the calling thread to wait for the thread to terminate then call the function pthread_attr_setdetachstate().

When a thread is created nondetached (PTHREAD_CREATE_JOINABLE), it is assumed that you will be waiting for it. That is, it is assumed that you will be executing a pthread_join() on the thread. Whether a thread is created detached or nondetached, the process does not exit until all threads have exited.

pthread_attr_setdetachstate() is prototyped by:

int pthread_attr_setdetachstate(pthread_attr_t *tattr,int detachstate);

pthread_attr_setdetachstate() returns zero after completing successfully. Any other returned value indicates that an error occurred. If the following condition occurs, the function fails and returns the corresponding value.

An example call to detatch a thread with this function is:

#include
pthread_attr_t tattr;
int ret;
/* set the thread detach state */
ret = pthread_attr_setdetachstate(&tattr,PTHREAD_CREATE_DETACHED);

Note - When there is no explicit synchronization to prevent it, a newly created, detached thread can die and have its thread ID reassigned to another new thread before its creator returns from pthread_create(). For nondetached (PTHREAD_CREATE_JOINABLE) threads, it is very important that some thread join with it after it terminates -- otherwise the resources of that thread are not released for use by new threads. This commonly results in a memory leak. So when you do not want a thread to be joined, create it as a detached thread.

It is quite common that you will wish to create a thread which is detatched from creation. The following code illustrates how this may be achieved with the standard calls to initialise and set and then create a thread:

#include
pthread_attr_t tattr;
pthread_t tid;
void *start_routine;
void arg
int ret;

/* initialized with default attributes */
ret = pthread_attr_init(&tattr);
ret = pthread_attr_setdetachstate(&tattr,PTHREAD_CREATE_DETACHED);
ret = pthread_create(&tid, &tattr, start_routine, arg);

The function pthread_attr_getdetachstate() may be used to retrieve the thread create state, which can be either detached or joined. It is prototyped by:

int pthread_attr_getdetachstate(const pthread_attr_t *tattr, int *detachstate);

pthread_attr_getdetachstate() returns zero after completing successfully. Any other returned value indicates that an error occurred.

An example call to this fuction is:

#include
pthread_attr_t tattr;
int detachstate;
int ret;

/* get detachstate of thread */
ret = pthread_attr_getdetachstate (&tattr, &detachstate);

Full Article here

http://www.cs.cf.ac.uk/Dave/C/node30.html


PART II

What Is a Thread? Why Use Threads

A thread is a semi-process, that has its own stack, and executes a given piece of code. Unlike a real process, the thread normally shares its memory with other threads (where as for processes we usually have a different memory area for each one of them). A Thread Group is a set of threads all executing inside the same process. They all share the same memory, and thus can access the same global variables, same heap memory, same set of file descriptors, etc. All these threads execute in parallel (i.e. using time slices, or if the system has several processors, then really in parallel).
The advantage of using a thread group instead of a normal serial program is that several operations may be carried out in parallel, and thus events can be handled immediately as they arrive (for example, if we have one thread handling a user interface, and another thread handling database queries, we can execute a heavy query requested by the user, and still respond to user input while the query is executed).
The advantage of using a thread group over using a process group is that context switching between threads is much faster then context switching between processes (context switching means that the system switches from running one thread or process, to running another thread or process). Also, communications between two threads is usually faster and easier to implement then communications between two processes.
On the other hand, because threads in a group all use the same memory space, if one of them corrupts the contents of its memory, other threads might suffer as well. With processes, the operating system normally protects processes from one another, and thus if one corrupts its own memory space, other processes won't suffer. Another advantage of using processes is that they can run on different machines, while all the threads have to run on the same machine (at least normally).

Creating And Destroying Threads

When a multi-threaded program starts executing, it has one thread running, which executes the main() function of the program. This is already a full-fledged thread, with its own thread ID. In order to create a new thread, the program should use the pthread_create() function. Here is how to use it:


#include        /* standard I/O routines                 */
#include      /* pthread functions and data structures */

/* function to be executed by the new thread */
void*
do_loop(void* data)
{
    int i;

    int i;   /* counter, to print numbers */
    int j;   /* counter, for delay        */
    int me = *((int*)data);     /* thread identifying number */

    for (i=0; i<10 color="brown" font="" for="" i="" j="">/* delay loop */
; printf("'%d' - Got '%d'\n", me, i); } /* terminate the thread */ pthread_exit(NULL); } /* like any C program, program's execution begins in main */ int main(int argc, char* argv[]) { int thr_id; /* thread ID for the newly created thread */ pthread_t p_thread; /* thread's structure */ int a = 1; /* thread 1 identifying number */ int b = 2; /* thread 2 identifying number */ /* create a new thread that will execute 'do_loop()' */ thr_id = pthread_create(&p_thread, NULL, do_loop, (void*)&a); /* run 'do_loop()' in the main thread as well */ do_loop((void*)&b); /* NOT REACHED */ return 0; }
A few notes should be mentioned about this program:
  1. Note that the main program is also a thread, so it executes the do_loop() function in parallel to the thread it creates.
  2. pthread_create() gets 4 parameters. The first parameter is used by pthread_create() to supply the program with information about the thread. The second parameter is used to set some attributes for the new thread. In our case we supplied a NULL pointer to tellpthread_create() to use the default values. The third parameter is the name of the function that the thread will start executing. The forth parameter is an argument to pass to this function. Note the cast to a 'void*'. It is not required by ANSI-C syntax, but is placed here for clarification.
  3. The delay loop inside the function is used only to demonstrate that the threads are executing in parallel. Use a larger delay value if your CPU runs too fast, and you see all the printouts of one thread before the other.
  4. The call to pthread_exit() Causes the current thread to exit and free any thread-specific resources it is taking. There is no need to use this call at the end of the thread's top function, since when it returns, the thread would exit automatically anyway. This function is useful if we want to exit a thread in the middle of its execution.
In order to compile a multi-threaded program using gcc, we need to link it with the pthreads library. Assuming you have this library already installed on your system, here is how to compile our first program:

gcc pthread_create.c -o pthread_create -lpthread 

The source code for this program may be found in the pthread_create.c file.

Synchronizing Threads With Mutexes

One of the basic problems when running several threads that use the same memory space, is making sure they don't "step on each other's toes". By this we refer to the problem of using a data structure from two different threads.
For instance, consider the case where two threads try to update two variables. One tries to set both to 0, and the other tries to set both to 1. If both threads would try to do that at the same time, we might get with a situation where one variable contains 1, and one contains 0. This is because a context-switch (we already know what this is by now, right?) might occur after the first tread zeroed out the first variable, then the second thread would set both variables to 1, and when the first thread resumes operation, it will zero out the second variable, thus getting the first variable set to '1', and the second set to '0'.

What Is A Mutex?

A basic mechanism supplied by the pthreads library to solve this problem, is called a mutex. A mutex is a lock that guarantees three things:
  1. Atomicity - Locking a mutex is an atomic operation, meaning that the operating system (or threads library) assures you that if you locked a mutex, no other thread succeeded in locking this mutex at the same time.
  2. Singularity - If a thread managed to lock a mutex, it is assured that no other thread will be able to lock the thread until the original thread releases the lock.
  3. Non-Busy Wait - If a thread attempts to lock a thread that was locked by a second thread, the first thread will be suspended (and will not consume any CPU resources) until the lock is freed by the second thread. At this time, the first thread will wake up and continue execution, having the mutex locked by it.
From these three points we can see how a mutex can be used to assure exclusive access to variables (or in general critical code sections). Here is some pseudo-code that updates the two variables we were talking about in the previous section, and can be used by the first thread:
lock mutex 'X1'.
set first variable to '0'.
set second variable to '0'.
unlock mutex 'X1'.


Meanwhile, the second thread will do something like this:

lock mutex 'X1'.
set first variable to '1'.
set second variable to '1'.
unlock mutex 'X1'.


Assuming both threads use the same mutex, we are assured that after they both ran through this code, either both variables are set to '0', or both are set to '1'. You'd note this requires some work from the programmer - If a third thread was to access these variables via some code that does not use this mutex, it still might mess up the variable's contents. Thus, it is important to enclose all the code that accesses these variables in a small set of functions, and always use only these functions to access these variables.



Creating And Initializing A Mutex

In order to create a mutex, we first need to declare a variable of type pthread_mutex_t , and then initialize it. The simplest way it by assigning it the PTHREAD_MUTEX_INITIALIZER constant. So we'll use a code that looks something like this:


pthread_mutex_t a_mutex = PTHREAD_MUTEX_INITIALIZER;


One note should be made here: This type of initialization creates a mutex called 'fast mutex'. This means that if a thread locks the mutex and then tries to lock it again, it'll get stuck - it will be in a deadlock.


There is another type of mutex, called 'recursive mutex', which allows the thread that locked it, to lock it several more times, without getting blocked (but other threads that try to lock the mutex now will get blocked). If the thread then unlocks the mutex, it'll still be locked, until it is unlocked the same amount of times as it was locked. This is similar to the way modern door locks work - if you turned it twice clockwise to lock it, you need to turn it twice counter-clockwise to unlock it. This kind of mutex can be created by assigning the constantPTHREAD_RECURSIVE_MUTEX_INITIALIZER_NP to a mutex variable.

Locking And Unlocking A Mutex

In order to lock a mutex, we may use the function pthread_mutex_lock(). This function attempts to lock the mutex, or block the thread if the mutex is already locked by another thread. In this case, when the mutex is unlocked by the first process, the function will return with the mutex locked by our process. Here is how to lock a mutex (assuming it was initialized earlier):


int rc = pthread_mutex_lock(&a_mutex);
if (rc) { /* an error has occurred */
    perror("pthread_mutex_lock");
    pthread_exit(NULL);
}
/* mutex is now locked - do your stuff. */
.
.



After the thread did what it had to (change variables or data structures, handle file, or whatever it intended to do), it should free the mutex, using the pthread_mutex_unlock() function, like this:


rc = pthread_mutex_unlock(&a_mutex);
if (rc) {
    perror("pthread_mutex_unlock");
    pthread_exit(NULL);
}


Destroying A Mutex

After we finished using a mutex, we should destroy it. Finished using means no thread needs it at all. If only one thread finished with the mutex, it should leave it alive, for the other threads that might still need to use it. Once all finished using it, the last one can destroy it using the pthread_mutex_destroy() function:


rc = pthread_mutex_destroy(&a_mutex);


After this call, this variable (a_mutex) may not be used as a mutex any more, unless it is initialized again. Thus, if one destroys a mutex too early, and another thread tries to lock or unlock it, that thread will get a EINVAL error code from the lock or unlock function.



Using A Mutex - A Complete Example

After we have seen the full life cycle of a mutex, lets see an example program that uses a mutex. The program introduces two employees competing for the "employee of the day" title, and the glory that comes with it. To simulate that in a rapid pace, the program employs 3 threads: one that promotes Danny to "employee of the day", one that promotes Moshe to that situation, and a third thread that makes sure that the employee of the day's contents is consistent (i.e. contains exactly the data of one employee).
Two copies of the program are supplied. One that uses a mutex, and one that does not. Try them both, to see the differences, and be convinced that mutexes are essential in a multi-threaded environment.
The programs themselves are in the files accompanying this tutorial. The one that uses a mutex is employee-with-mutex.c. The one that does not use a mutex is employee-without-mutex.c. Read the comments inside the source files to get a better understanding of how they work.

Starvation And Deadlock Situations

Again we should remember that pthread_mutex_lock() might block for a non-determined duration, in case of the mutex being already locked. If it remains locked forever, it is said that our poor thread is "starved" - it was trying to acquire a resource, but never got it. It is up to the programmer to ensure that such starvation won't occur. The pthread library does not help us with that.

The pthread library might, however, figure out a "deadlock". A deadlock is a situation in which a set of threads are all waiting for resources taken by other threads, all in the same set. Naturally, if all threads are blocked waiting for a mutex, none of them will ever come back to life again. The pthread library keeps track of such situations, and thus would fail the last thread trying to call pthread_mutex_lock(), with an error of type EDEADLK. The programmer should check for such a value, and take steps to solve the deadlock somehow.

Refined Synchronization - Condition Variables

As we've seen before with mutexes, they allow for simple coordination - exclusive access to a resource. However, we often need to be able to make real synchronization between threads:
  • In a server, one thread reads requests from clients, and dispatches them to several threads for handling. These threads need to be notified when there is data to process, otherwise they should wait without consuming CPU time.
  • In a GUI (Graphical User Interface) Application, one thread reads user input, another handles graphical output, and a third thread sends requests to a server and handles its replies. The server-handling thread needs to be able to notify the graphics-drawing thread when a reply from the server arrived, so it will immediately show it to the user. The user-input thread needs to be always responsive to the user, for example, to allow her to cancel long operations currently executed by the server-handling thread.
All these examples require the ability to send notifications between threads. This is where condition variables are brought into the picture.


What Is A Condition Variable?

A condition variable is a mechanism that allows threads to wait (without wasting CPU cycles) for some even to occur. Several threads may wait on a condition variable, until some other thread signals this condition variable (thus sending a notification). At this time, one of the threads waiting on this condition variable wakes up, and can act on the event. It is possible to also wake up all threads waiting on this condition variable by using a broadcast method on this variable.
Note that a condition variable does not provide locking. Thus, a mutex is used along with the condition variable, to provide the necessary locking when accessing this condition variable.

Creating And Initializing A Condition Variable

Creation of a condition variable requires defining a variable of type pthread_cond_t, and initializing it properly. Initialization may be done with either a simple use of a macro named PTHREAD_COND_INITIALIZER or the usage of the pthread_cond_init() function. We will show the first form here:

pthread_cond_t got_request = PTHREAD_COND_INITIALIZER; 

This defines a condition variable named 'got_request', and initializes it.
Note: since the PTHREAD_COND_INITIALIZER is actually a structure, it may be used to initialize a condition variable only when it is declared. In order to initialize it during runtime, one must use the pthread_cond_init() function.

Signaling A Condition Variable

In order to signal a condition variable, one should either the pthread_cond_signal() function (to wake up a only one thread waiting on this variable), or the pthread_cond_broadcast() function (to wake up all threads waiting on this variable). Here is an example using signal, assuming 'got_request' is a properly initialized condition variable:

int rc = pthread_cond_signal(&got_request); 

Or by using the broadcast function:

int rc = pthread_cond_broadcast(&got_request); 

When either function returns, 'rc' is set to 0 on success, and to a non-zero value on failure. In such a case (failure), the return value denotes the error that occured (EINVAL denotes that the given parameter is not a condition variable. ENOMEM denotes that the system has run out of memory.
Note: success of a signaling operation does not mean any thread was awakened - it might be that no thread was waiting on the condition variable, and thus the signaling does nothing (i.e. the signal is lost).
It is also not remembered for future use - if after the signaling function returns another thread starts waiting on this condition variable, a further signal is required to wake it up.


Waiting On A Condition Variable

If one thread signals the condition variable, other threads would probably want to wait for this signal. They may do so using one of two functions, pthread_cond_wait() or pthread_cond_timedwait(). Each of these functions takes a condition variable, and a mutex (which should be locked before calling the wait function), unlocks the mutex, and waits until the condition variable is signaled, suspending the thread's execution. If this signaling causes the thread to awake (see discussion of pthread_cond_signal() earlier), the mutex is automagically locked again by the wait funciton, and the wait function returns.

The only difference between these two functions is that pthread_cond_timedwait() allows the programmer to specify a timeout for the waiting, after which the function always returns, with a proper error value (ETIMEDOUT) to notify that condition variable was NOT signaled before the timeout passed. The pthread_cond_wait() would wait indefinitely if it was never signaled.

Here is how to use these two functions. We make the assumption that 'got_request' is a properly initialized condition variable, and that 'request_mutex' is a properly initialized mutex. First, we try the pthread_cond_wait() function:


/* first, lock the mutex */
int rc = pthread_mutex_lock(&a_mutex);
if (rc) { /* an error has occurred */
    perror("pthread_mutex_lock");
    pthread_exit(NULL);
}
/* mutex is now locked - wait on the condition variable.             */
/* During the execution of pthread_cond_wait, the mutex is unlocked. */
rc = pthread_cond_wait(&got_request, &a_mutex);
if (rc == 0) { /* we were awakened due to the cond. variable being signaled */
               /* The mutex is now locked again by pthread_cond_wait()      */
    /* do your stuff... */
    .
}
/* finally, unlock the mutex */ 
pthread_mutex_unlock(&a_mutex);


Now an example using the pthread_cond_timedwait() function:


#include      /* struct timeval definition           */
#include        /* declaration of gettimeofday()       */

struct timeval  now;            /* time when we started waiting        */
struct timespec timeout;        /* timeout value for the wait function */
int             done;           /* are we done waiting?                */

/* first, lock the mutex */
int rc = pthread_mutex_lock(&a_mutex);
if (rc) { /* an error has occurred */
    perror("pthread_mutex_lock");
    pthread_exit(NULL);
}
/* mutex is now locked */

/* get current time */ 
gettimeofday(&now);
/* prepare timeout value */
timeout.tv_sec = now.tv_sec + 5
timeout.tv_nsec = now.tv_usec * 1000; /* timeval uses microseconds.         */
                                      /* timespec uses nanoseconds.         */
                                      /* 1 nanosecond = 1000 micro seconds. */

/* wait on the condition variable. */
/* we use a loop, since a Unix signal might stop the wait before the timeout */
done = 0;
while (!done) {
    /* remember that pthread_cond_timedwait() unlocks the mutex on entrance */
    rc = pthread_cond_timedwait(&got_request, &a_mutex, &timeout);
    switch(rc) {
        case 0:  /* we were awakened due to the cond. variable being signaled */
                 /* the mutex was now locked again by pthread_cond_timedwait. */
            /* do your stuff here... */
            .
            .
            done = 0;
            break;
        case ETIMEDOUT: /* our time is up */
            done = 0;
            break;
        default:        /* some error occurred (e.g. we got a Unix signal) */
            break;      /* break this switch, but re-do the while loop.   */
    }
}
/* finally, unlock the mutex */
pthread_mutex_unlock(&a_mutex);


As you can see, the timed wait version is way more complex, and thus better be wrapped up by some function, rather then being re-coded in every necessary location.


Note: it might be that a condition variable that has 2 or more threads waiting on it is signaled many times, and yet one of the threads waiting on it never awakened. This is because we are not guaranteed which of the waiting threads is awakened when the variable is signaled. It might be that the awakened thread quickly comes back to waiting on the condition variables, and gets awakened again when the variable is signaled again, and so on. The situation for the un-awakened thread is called 'starvation'. It is up to the programmer to make sure this situation does not occur if it implies bad behavior. Yet, in our server example from before, this situation might indicate requests are coming in a very slow pace, and thus perhaps we have too many threads waiting to service requests. In this case, this situation is actually good, as it means every request is handled immediately when it arrives.
Note 2: when the mutex is being broadcast (using pthread_cond_broadcast), this does not mean all threads are running together. Each of them tries to lock the mutex again before returning from their wait function, and thus they'll start running one by one, each one locking the mutex, doing their work, and freeing the mutex before the next thread gets its chance to run.

Destroying A Condition Variable

After we are done using a condition variable, we should destroy it, to free any system resources it might be using. This can be done using the pthread_cond_destroy(). In order for this to work, there should be no threads waiting on this condition variable. Here is how to use this function, again, assuming 'got_request' is a pre-initialized condition variable:


int rc = pthread_cond_destroy(&got_request);
if (rc == EBUSY) { /* some thread is still waiting on this condition variable */
    /* handle this case here... */
    .
    .
}


What if some thread is still waiting on this variable? depending on the case, it might imply some flaw in the usage of this variable, or just lack of proper thread cleanup code. It is probably good to alert the programmer, at least during debug phase of the program, of such a case. It might mean nothing, but it might be significant.



A Real Condition For A Condition Variable

A note should be taken about condition variables - they are usually pointless without some real condition checking combined with them. To make this clear, lets consider the server example we introduced earlier. Assume that we use the 'got_request' condition variable to signal that a new request has arrived that needs handling, and is held in some requests queue. If we had threads waiting on the condition variable when this variable is signaled, we are assured that one of these threads will awake and handle this request.
However, what if all threads are busy handling previous requests, when a new one arrives? the signaling of the condition variable will do nothing (since all threads are busy doing other things, NOT waiting on the condition variable now), and after all threads finish handling their current request, they come back to wait on the variable, which won't necessarily be signaled again (for example, if no new requests arrive). Thus, there is at least one request pending, while all handling threads are blocked, waiting for a signal.
In order to overcome this problem, we may set some integer variable to denote the number of pending requests, and have each thread check the value of this variable before waiting on the variable. If this variable's value is positive, some request is pending, and the thread should go and handle it, instead of going to sleep. Further more, a thread that handled a request, should reduce the value of this variable by one, to make the count correct.
Lets see how this affects the waiting code we have seen above.



/* number of pending requests, initially none */
int num_requests = 0;
.
.
/* first, lock the mutex */
int rc = pthread_mutex_lock(&a_mutex);
if (rc) { /* an error has occurred */
    perror("pthread_mutex_lock");
    pthread_exit(NULL);
}
/* mutex is now locked - wait on the condition variable */
/* if there are no requests to be handled.              */
rc = 0;
if (num_requests == 0)
    rc = pthread_cond_wait(&got_request, &a_mutex);
if (num_requests > 0 && rc == 0) { /* we have a request pending */
        /* do your stuff... */
        .
        .
        /* decrease count of pending requests */
        num_requests--;
    }
}
/* finally, unlock the mutex */
pthread_mutex_unlock(&a_mutex);

Full Article here

http://www.cs.kent.edu/~ruttan/sysprog/lectures/multi-thread/multi-thread.html

Wednesday, June 20, 2007

Listing All Registered ActiveXControls / COM object based on there category

//Initialise COM libraries
CoInitialize (NULL);

//The Component Category Manager implemented by System implements
//this interface
ICatInformation *pCatInfo=NULL;

//Create an instance of standard Component Category Manager
HRESULT hr=CoCreateInstance (CLSID_StdComponentCategoriesMgr ,
NULL,
CLSCTX_INPROC_SERVER,
IID_ICatInformation ,
(void **)&pCatInfo);

//Increase ref count on interface
pCatInfo->AddRef ();

//IEnumGUID interface provides enumerator for enumerating through
//the collection of COM objects
IEnumGUID *pEnumGUID=NULL;

//We are intersted in finding out only controls so put CATID_Control
//in the array
CATID pcatidImpl[1];
CATID pcatidReqd[1];
// Want only my Plugin Category
pcatidImpl[0]=CATID_MyPlugin;
pcatidReqd[1]=CATID_MyPlugin;

// Want all Active X Control
//pcatidImpl[0]=CATID_Control;







//Now enumerate the classes i.e. COM objects of this type.
pCatInfo->EnumClassesOfCategories (1,
pcatidImpl,
0,
pcatidReqd ,
&pEnumGUID);

//Enumerate as long as you get S_OK
CLSID clsid;

while( (hr= pEnumGUID->Next( 1, &clsid, NULL ))==S_OK )
{
BSTR bstrClassName; //Get the information of class

//This is what MSDN says about the parameters
/*-----------------------------------------------
USERCLASSTYPE_FULL The full type name of the class.
USERCLASSTYPE_SHORT A short name (maximum of 15 characters) that
is used for popup menus and the Links dialog
box.
USERCLASSTYPE_APPNAME The name of the application servicing the class
and is used in the Result text in dialog boxes.
-----------------------------------------------*/
OleRegGetUserType (clsid,USERCLASSTYPE_FULL,&bstrClassName);
CString strControlName(bstrClassName);
//Add string in our listbox
m_list1.AddString (strControlName);
}

//we are done so now release the interface ptr
pCatInfo->Release ();

CoUninitialize ();

Waiting for the Prj2make# Visual Studio Add-in For Mono

Right now Novell provide a command line utility to migrate a Microsoft Prj
to Mono.

http://forge.novell.com/modules/xfcontent/downloads.php/prj2make-sharp

Looking forward to see this project.

I have completed a Visual Studio Doxygen Add On if the author want
to contact me.

http://www.mfconsulting.com/product/gtks-inst4win/vsprj2make_Proposal.html

Tuesday, June 19, 2007

java to c# and c# to java

Microsoft has the Java Language Conversion Assistant (JLCA) that enables
developers to migrate their existing Java code to C# on the .NET Framework.
It can be found at:


http://msdn2.microsoft.com/en-us/vstudio/aa718346.aspx

However, the reverse, a C# to Java is more difficlut to achieve

http://jsc.sourceforge.net/

Thursday, June 14, 2007

Direct Show Filters Rules

APE Source Filter: DirectShow audio decoder filter used to decode Monkey's Audio files (APE files)
AC3 Filter: DirectShow audio decoder and processor filter used to decode audio tracks in movies (DVD, MPEG4 and others)
AVI Splitter: Replacement for DirectShow's default Audio Video Interleave (AVI) Splitter (It can open many broken files and can reindex when needed)
CDDA Reader Filter: DirectShow audio decoder capable of reading Audio CDs
CDXA Reader Filter: DirectShow video decoder capable of reading (S)VCDs and XCDs
CoreAAC Filter: Advanced Audio Coding (AAC) DirectShow audio decoder capable of reading AAC files
CoreFlac Decoder: DirectShow audio decoder for the Flac audio codec
FFDShow: DirectShow decoding filter for decompressing DivX, XviD, H.264, FLV1, WMV, MPEG-1 and MPEG-2, MPEG-4 and various other audio and video formats. It uses libavcodec from ffmpeg project for video decompression, postprocessing code from mplayer to enhance visual quality of low bitrate movies, and is based on original DirectShow filter from XviD, which is GPL'ed educational implementation of MPEG4 encoder
FLV Splitter: Capable of splitting Flash Video files (FLV) and decoding On2 VP62 Video streams (notable users of the FLV format include YouTube, Google Video, Reuters.com, Yahoo! Video and MySpace.)
OGG Splitter: DirectShow OGG Splitter capable of splitting OGG and OGM file(s) and enables to play / create .ogm wrapped video
GPL MPEG-1/2 Decoder: DirectShow MPEG decoder filter can be used to play MPEG-1 and MPEG-2 streams
MOD Source Filter: DirectShow filter decoder capable of decoding Tracker files (MO3, IT, XM, S3M, MTM, MOD and UMX)
MPEG Splitter: DirectShow splitter capable of splitting MPEG1/MPEG2 files (MP(E)G, VOB, DAT, TS, TP, etc.)
MPV Decoder: DirectShow decoder capable of decoding MPEG1/MPEG2 Video streams, and DVDs
MPA Splitter: DirectShow Splitter capable of splitting MPEG Audio files (MP3) and MPEG4 Audio files (AAC)
MPC Source Filter: DirectShow decoder filter capable of decoding MusePack Audio files (MPC).
MP4 Splitter: DirectShow Splitter capable of splitting MP4, M4A, 3GP and HD-MOV files
Matroska Splitter: DirectShow splitter capable of splitting Matroska files (MKV and MKA)
Media Player Classic: an open source media player that looks just like Windows Media Player, but has many additional features. It has a built in DVD player with real-time zoom, support for AVI subtitles, QuickTime and RealVideo support
OFR Source Filter: DirectShow filter decoder capable of decoding OptimFROG Audio files (OFR)
RealMedia Splitter: DirectShow splitter that allows you to play RealNetworks audio/video files.
SHOUTcast Source Filter: DirectShow decoder filter capable of handling internet radio stations streams (http://www.shoutcast.com)
WV Splitter / Decoder: DirectShow WV Splitter and decoder capable of splitting and decoding WavPack Audio files (WV)
VobSub Filter: DirectShow Vob Subtitles filter capable of displaying (embedded) Subtitle files (SRT, SUB, IDX, etc)

Friday, June 08, 2007

Manifest hell

I thought Manifests were an end to DLL hell - but it just seems that now we have manifest hell


A solution to two references to different versions of CRT, MFC, ATL in one application manifest file
I have received several questions about a case when developers find two or more references to different versions of CRT or MFC or ALT libraries in application manifest. Usually application manifest would look similar to the following:



Notice underlined version of assembly. This manifest tells Windows loader to load two different versions of the same assembly. The loader is going to do what it is told to. But the problem arises when it can only find one version (50608) of them. There are two cases when this may happen:
1) The oldest version of VC++ library is installed in WinSxS folder. For example, author of the application may have redistributed VS2005 RTM version of CRT library using MSMs or VCRedist.EXE, but ones parts of application were rebuilt with VS2005 SP1 and deployed to the same machine, SP1 versions of MSMs or VCRedist.EXE were not deployed to the same machine.
2) It is not possible to have two versions of a private assembly in one folder. For example, an author of an application has deployed the RTM version of CRT library as a private assembly in application’s local folder. After application is rebuilt with SP1, the author faces a challenge of having two copies of same files/folders in application local folder. If she keeps RTM version, Windows loader is going to complain about SP1 version missing. If she keeps SP1 version, Windows loaders is going to fail to find RTM version. Windows shows a message box which says :”The application has failed to start because of side-by-side configuration is incorrect,” and event viewer shows error details similar to “Activation context generation failed for "…\foo.exe".Error in manifest or policy file. A component version required by the application conflicts with another component version already active.” Bottom line, application does not start.
The root cause of the issue is that not all parts of application’s source code are built with the same version of VC++ libraries and tools. The linker catches some cases and reports errors when it tries to link objects and libraries built with different versions of compiler. However it is still possible to get pass the linker and an application may get these two dependencies in its manifest.
There is one solution to the problem and two workarounds.
The solution. To fix the problem, you must rebuild all parts of your code with the newest toolset and libraries. Make sure you have cleaned up all binaries built by the old toolset and started full rebuild of the whole source base. Also check that old versions of headers and import libraries for VC++ libraries are not on INCLUDE and LIB path and they are not used during the build.
If you cannot rebuild all your code and use one version of VC libraries, there are two ways to work around this problem:
Workaround#1: Install the newer version (8.0.50727.762 in this case) of VC++ MSMs or VCRedist.EXE on a machine where your application is going to run. Once policy for VC++ assemblies is installed on that machine they are going to redirect all loads of older versions (8.0.50608.0) to the newest version available on the machine.
Workaround#2: If you are redistributing VC++ libraries in application’s local folder, you need to add an application configuration file that redirects an attempt to load 8.0.50608.0 version to 8.0.50727.762 version. Configuration file has to have same name as the exe plus .config extension and has to be right next to exe or embedded into the EXE. Here is an example of a configuration file that one would use to resolve issue with the manifest from above:
inferior and superior char have been replace by { and } because google blogs text editor is retarded

{configuration}
{windows}
{assemblybinding xmlns="urn:schemas-microsoft-com:asm.v1"}
{dependentassembly}
{assemblyidentity publickeytoken="1fc8b3b9a1e18e3b" processorarchitecture="x86" name="Microsoft.VC80.CRT" type="win32"}{/assemblyidentity}
{bindingredirect newversion="8.0.50727.762" oldversion="8.0.41204.256-8.0.50727.762"}
{/dependentassembly}
{/assemblybinding}
{/windows}
{/configuration}
This file basically redirects attempts to load any version of VC CRT greater or equal to 8.0.41204.256 (VS2005 Beta 1) and less than 8.0.50727.762(VS2005 SP1) to load VS2005 SP1 version of VC CRT. It is similar to what the policy file does for a case when CRT is installed into WinSxS folder.
Overall my recommendation is to never use static libraries for which you do not have a source code. Using a static library without its source always puts a set of restrictions on build configuration of a binary that consumes it. There are ways to build a “clean” static library, but they are not well known among developers and it is very rare to find a static library that can be consumed in a code built with different versions of compiler, linker and libraries. If source code is not provided, I would only take a dependency on a DLL and only if its APIs are cleanly designed to obey rules of data exchange across DLL boundary. COM and .Net class assemblies are designed to address this problem and they are the easiest technologies to use in this scenario.


http://msdn2.microsoft.com/en-us/library/aa374182.aspx

Visual Studio 2005 SP1
http://www.microsoft.com/downloads/thankyou.aspx?familyId=bb4a75ab-e2d4-4c96-b39d-37baf6b5b1dc&displayLang=en

VCRedist x86
http://www.microsoft.com/downloads/details.aspx?familyid=32bc1bee-a3f9-4c13-9c99-220b62a191ee&displaylang=en

Visual Studio 2005 SP1

Installation fails on Windows Server 2003 editions with Windows Server 2003 SP1 Installed:
loadTOCNode(3, 'moreinformation');
The error reported is
Error 1718. File Filename was rejected by digital signature policy.This problem occurs when the computer has insufficient contiguous memory for Windows Server 2003 or Windows XP to verify that the .msi package or the .msp package is correctly signed.To resolve this issue:

Refer to KB article 925336 (http://support.microsoft.com/kb/925336).

VS2005 SP1 and Microsoft.VC80.CRT in applocal mode

problem running some of our applications in "applocal" mode since we have upgraded to SP1.
We use some external libraries (we don't have the source code) that were compiled with the original VS2005.
The following manifest is therefore generated:













The application runs fine when the CRT in present in WinSxS (the Policy does the correct redirection). However, if you want to use the "applocal" mode for this particular application.

When running "app.exe" if you get this message: "This application has failed to start because the application configuration is incorrect. Reinstalling the application may fix this problem.

Choose external manifest

_USE_RTM_VERSION is newly introduced in SP1 (and previously also in hotfix 919280). It is found in crtassem.h and crtdefs.h. If you use _USE_RTM_VERSION - it must be done in all EXEs and DLLs the app uses. The _USE_RTM_VERSION define determines which CRT version is placed in your manifest file. If it is defined, then the old version number is put in.

If not defined, then it puts the new version in (what ever version ends up shipping with SP1)And no, using _USE_RTM_VERSION does not avoid the runtime rebinding to the new version. The fact is, it only gives the apps the ability to use the originals, but if the new ones are there on the machine in WinSxS, then it uses the new ones instead. Without _USE_RTM_VERSION, if the originals are there and the new ones aren't, then the app doesn't even run.Without using WinSxS, even if you install applocal and if same or newer versions exist in WinSxS, then those in WinSxS will be used instead. However, there is one way to trick the system into never using the new ones from WinSxS, and only using your applocal (original) ones.

For more info see http://blog.kalmbachnet.de/?postid=80

I know of no way to force an app to use a specific version out of WinSxS, if a redirection policy has been installed.

Monday, June 04, 2007

Including QuickTime In A Web Page

http://www.apple.com/quicktime/tutorials/embed.html

Adobe Apollo

Apollo is the code name for a cross-operating system runtime being developed by Adobe that allows developers to leverage their existing web development skills (Flash, Flex, HTML, JavaScript, Ajax) to build and deploy rich Internet applications (RIAs) to the desktop.
Apollo enables developers to create applications that combine the benefits of web applications – network and user connectivity, rich media content, ease of development, and broad reach – with the strengths of desktop applications – application interactions, local resource access, personal settings, powerful functionality, and rich interactive experiences.

http://labs.adobe.com/technologies/apollo/

Darwing Streaming Server basics

Streaming Server Setup

Introduction

The Streaming Server comes in two flavors: Darwin Streaming Server is an Open Source project from Apple. Binaries of Darwin Streaming Server are available for Mac OS X, Red Hat Linux 9, Solaris 9 and Windows 2000/2003/XP. Apple includes QuickTime Streaming Server, a commercially supported version of the server, with Mac OS X Server. QuickTime Streaming Server adds:

QTSS Publisher, content management for streaming media.
QuickTime Broadcaster, a live encoder for live streaming from video/audio sources.
Server Admin, the graphical administration tool for managing Mac OS X Server including the Streaming Server, File Services, Web Services, Mail Services, Directory Services, etc.

QuickTime Streaming Server is formally supported by Apple, while support for the Open Source version is informally provided via mailing lists and on-line resources.

Support and Documentation
Documentation on QuickTime Streaming Server in pdf format is available here (most of the documentation applies to Darwin as well).

A QuickTime/Darwin Streaming Server technology brief in pdf format is available here. Developer documentation is available here. Apple has also posted a good summary of the use and advantages streaming versus http fast-start.
Mailing list support for Darwin Streaming Server and QuickTime Streaming Server is provided informally via the streaming server mailing lists (the users list or the developers list).
Online support resources include the QuickTime section of Apple's support site, and the Streaming Server FAQ.

Download and Install

If you are using Mac OS X Server, QuickTime Streaming Server is already installed. The procedures outlined on this page use the web administration front end for QuickTime Streaming Server. To use web administration you must enable this capability using the Server Admin application in Panther Server. Select the QuickTime Streaming Service and navigate to Access-Settings. Check off "Enable web-based administration", set the password for web administration and save your settings.

If you want to use Darwin Streaming Server, it can be downloaded here. Windows does not include Perl which is required for web administration of Darwin Streaming Server. You can download Active State Perl for Windows here. Make sure to have Perl installed before running the installer on Windows.

Once the Darwin binaries are downloaded, unpack the archive and review the Read Me file, then start the server. On Mac OS X, the server will automatically start after installation. The best way to start the server on Solaris, Linux and Windows is to start up the Perl script that provides web administration:

On Solaris and Linux: /usr/local/sbin/streamingadminserver.pl

On Windows: C:\Perl\bin\perl "C:\Program Files\Darwin Streaming Server\streamingadminserver.pl"

Note: On Windows it may be necessary to reset the administrative password before running streamingserveradmin.pl, the Perl administrative server.

You can create/reset the administrative password on Windows with the following command: C:\Perl\bin\perl "C:\Program Files\Darwin Streaming Server\WinPasswdAssistant.pl"

Once the Perl process is running you can access the web administrative interface by opening your web browser on the server and opening the URL:
http://127.0.0.1:1220

The Streaming Server can reflect live Icecast/Shoutcast broadcasts if you configure a MP3 Broadcast password.

SSL certificate. The server can be administered over SSL if a valid SSL certificate installed. Usually this feature is not enabled during installation.

Media Folder.
This is the "document root" for the streaming server. Media files will be placed in this folder or subdirectories of this folder.

Port 80 Streaming.
The server can be configured to stream over Port 80. If you are not running a web server on port 80 on the same system, it is a good idea to enable this option. This will allow clients to connect to your server through most firewalls.

Once you have navigated through the setup screens, you should see the main window for web server administration on your server. The options configured during setup can be changed in the general settings and port settings panes of the administrative web interface.

Note: You can set up the streamingserveradmin.pl to automatically execute when your server is started if you always want to have the server running (i.e. by editing rc.local on Linux). On Windows 2000/2003 Server the installer adds Darwin Streaming Server as a service in the Service Manager.

Test the Server

To test your server, go to another system on your network with QuickTime installed. Open QuickTime Player and select "Open URL..." from the File Menu:
Enter the rtsp url to your server using it's IP address and the filename sample_100kbit.mov (a sample movie installed in the Movies directory on the server):
The player should open up a stream from your server. Compressed and hinted movies can now be placed in the Movies directory and streamed.

Thursday, May 31, 2007

Microsoft Surface

Who say that apple is the best company for user interface.
Look at how microsoft is giving a lesson to apple

http://www.microsoft.com/surface/


Impressive ! Worth the virtual tour.

Seams to me be the next operating system !

Microsoft CEO Steve Ballmer showed off Surface, novel interface technology that powers a touch-screen tabletop on which you can create and manipulate images and content with gestures of your hands. The initial units will sell for between $5,000 and $10,000 each and are likely to find homes in hotels, casinos and retail stores (along with the dens of wealthy geeks). Consensus of onlookers: Pretty cool.

Wednesday, May 16, 2007

Unix Thread Versus Windows Thread

A good article for people who are trying to build efficient cross platform threading class
to be build on Linux or Windows


http://www.microsoft.com/technet/interopMigration/taskstools/migrate/unix/ucamg/ch03uav3.mspx

Tuesday, May 15, 2007

Visual Studio Project to Makefile for cross platform project

Visual Studio 6

To export a makefile
From the Project menu, select Export Makefile.
In the Export Makefile(s) dialog box, select the project(s) for which you want to create a .mak file.
A separate .mak file is created for each project you select.
If you want Visual C++ to update dependency information, check the Write dependencies when writing makefiles checkbox.
If you know that you want to automatically export the .mak file each time you update this project, use the following procedure.
To automatically export a makefile when the project is updated

1 From the Tools menu, choose Options.
2 On the Build tab, check the Export makefile when saving project file checkbox.
Selecting this option increases the time it takes to save the project.
3 To automatically write dependency information every time you export a makefile
From the Tools menu, choose Options.
4 On the Build tab, check the Always write dependencies when writing makefiles checkbox.
Selecting this option considerably increases the time it takes to write the .mak file.
5 To specify per-configuration dependency information
From the Project menu, choose Settings.
On the General tab, check the Allow per-configuration dependencies checkbox.
Selecting this option increases the time it takes to write the dependency information, according to the number of build configurations in the project.

http://msdn2.microsoft.com/en-us/library/aa233950(VS.60).aspx


Visual Studio .NET

Comvert the project to Visual Studio 6 using
stephane rodriguez prjconverter then export to makefile using Visual Studio 6

http://www.codeproject.com/tools/prjconverter.asp?df=100&forumid=10069&exp=0&select=1288997


WineMaker

The Winelib development toolkit

http://www.winehq.org/site/docs/winelib-guide/winelib-toolkit


Other

would be nice to implement a plugin for Visual Studio 2005.
I am using one to generate doxygen automatically. Merging code from winemaker
into a VS2005 plugin would be a great idea.

What about exporting the project towards those fancy building script as well

SConstruct
ANT
etc...

Monday, May 07, 2007

Using mouse and clipboard between two separate Operating Systems running on two separate boxes

Have you ever wanted to be able to cut and past text between two separate Operating System running on two diffrence box. Have you ever wanted to use one mouse on multiple Operating System.

What you are looking for is Synergy.

Synergy lets you easily share a single mouse and keyboard between multiple computers with different operating systems, each with its own display, without special hardware. It's intended for users with multiple computers on their desk since each system uses its own monitor(s).
Redirecting the mouse and keyboard is as simple as moving the mouse off the edge of your screen. Synergy also merges the clipboards of all the systems into one, allowing cut-and-paste between systems. Furthermore, it synchronizes screen savers so they all start and stop together and, if screen locking is enabled, only one screen requires a password to unlock them all. Learn more about how it works.

http://synergy2.sourceforge.net/

Tuesday, May 01, 2007

Show TIB Under The Hood From Matt Pietrek

Matt Pietrek is the author of Windows 95 System Programming Secrets (IDG Books, 1995).

http://www.microsoft.com/msj/archive/s2ce.aspx

While Windows NT™ and Windows® 95 are quite different under the hood, they both share a key system data structure that many programmers aren't aware of. To be a bit more precise, certain fields of this data structure are shared. Regardless of the differences, this structure is used extensively, and is even accessed by compiler-generated code. That's right, your C++ compiler generates code to access system level information directly.

What structure am I talking about? It goes by at least two different names. The Windows 95 code calls it a Thread Information Block (TIB).

In Windows NT, it's called the Thread Environment Block (TEB). However, I've seen it referred to as a TIB in some Windows NT header files so, for the purposes of this column, I'll refer to it as a TIB.

What's in a Thread Information Block that makes it so special? As its name implies, the data found in a TIB relates to threads, and there's a TIB for each thread in the system.

In fields shared by Windows NT and Windows 95, you'll find information like a pointer to the thread's structured exception handler list, the location of the thread's stack, and the location of the thread local storage slots. Other fields in the TIB differ between Windows NT and Windows 95.

You might be surprised to learn that the TIB didn't first appear in Windows NT or Windows 95. The TIB has its ancestry in OS/2, before Microsoft created Windows NT, and it still exists in OS/2 today. In fact, the OS/2 TIB shares much of the same format, and is accessed in the same manner as under Win32®. There's even a line in a Microsoft header file (NTDDK.H) that says <<>>

// This structure MUST MATCH OS/2 V2.0!
<<>>

How would you get at the TIB if you were inclined to go poking around? It's as easy as looking at what the FS register points to. Hey! Didn't segments and segment registers go away in Win32? For the most part, that's true. However, in all Intel-based Win32 implementations (even the forgotten stepchild, Win32s), the FS register points to the TIB. Thus, all of the structure offsets that I'll detail later can be used as offsets into the segment pointed to by the FS register. For example, FS:[0] points to the structured exception handling chain, while FS:[2C] points to the thread's thread local storage array.

I just mentioned that compilers access the TIB structure directly. Let's look at a small example to see this in action. This is a very small C program that uses structured exception handling:

int main()
{
__try
{ int i = 0;

}
__except( 1 )
{ int j = 1: }
return 0;
}

When compiled, the first part of the resulting code looks like this: 401000: PUSH EBP

401001: MOV EBP,ESP
401003: PUSH FF
401005: PUSH 00404000
40100A: PUSH 00401140
40100F: MOV EAX,FS:[00000000]
401015: PUSH EAX
401016: MOV DWORD PTR FS:[00000000],ESP

The thing to notice is the series of PUSH instructions. They create a data structure on the stack (sort of like an invisible local variable). The instruction at offset 0x_40100F retrieves the head of the structured exception handling chain out of the thread information block and stores it into EAX-this is where the FS:[00000000] part of the instruction comes from. The code then pushes the current head of the structured exception handling chain list onto the stack. This finishes the process of creating a local data structure on the stack. Finally, the MOV DWORD PTR FS:[00000000],ESP instruction changes the head of the structured exception handling chain to point at the newly created data structure.
The key point is that Win32 compilers implicitly know about the TIB and generate code that accesses it. Because the compiler can't know which Win32 system the code will run on, you can safely assume that any compiler-generated code that references the FS segment uses fields common between Win32 platforms.

Common Fields in the TIB You just saw one example of a TIB structure field that's common to all Win32 implementations. In this section, I'll list all of the common fields, along with a short description. The fields that differ between Win32 implementations are described later.

There are several different header files floating around that define the fields of a TIB. Unfortunately, they're not always consistent with each other or complete. In the Windows NT DDK, you'll find a structure called an NT_TIB defined in NTDDK.H.

In the Windows NT 3.51 service pack 3 SDK update, a new WINNT.H was added that also defines an NT_TIB structure.

In addition, someone from the Windows 95 team posted online a snippet from an .H file that described a TIB.

In my descriptions, I've tried to use the most descriptive name. You'll see these names in the TIB.H file included with the SHOWTIB program I'll present later on.

The 00h DWORD pvExcept field contains a pointer to the head of the thread's structured exception handling chain. The chain is a linked list of EXCEPTION_REGISTRATION_RECORD structures (which unfortunately are not defined in any official .H file).

For more information on the structured exception handling chain, you might refer to chapter 3 of my book "Windows 95 System Programming Secrets."
The 04h DWORD pvStackUserTop field contains the linear address of the topmost address of the thread's stack. Put another way, at no point will this thread have a stack pointer value that's greater than or equal to the value of this field.
The 08h DWORD pvStackUserBase field contains the linear address of the lowest committed page in the thread's user mode stack. As the thread uses successively lower addresses in the stack, those pages will be committed, and this field will be updated accordingly.

The 14h DWORD pvArbitrary field is theoretically available for applications to use however they want. It's almost like an extra thread local storage slot for you to use, although I've never seen an application use it.

The 18h DWORD ptibSelf field holds the linear address of the TIB. Put another way, the TIB block contains a pointer to itself. Why bother doing this? If the TIB's fields are used extensively, it makes sense for 32-bit code to read and write the TIB using regular pointers rather than using a segment register override (for example, by using FS:[xxxxxxxx]). The SHOWTIB program presented later uses this field.

The 2Ch DWORD pvTLSArray field contains a pointer to the thread local storage (TLS) slots for the thread. For example, if you had a TLS index value of 4, you could take this pointer, add 10h to it ( 4 * sizeof(DWORD) ), and retrieve the TLS value directly. Knowing that this field points to the thread's TLS slots, you could write your own versions of TlsSetValue and TlsGetValue easily. If you use _ _declspec(thread) variables in your code, check out the ASM code emitted by your compiler.

You'll find that it uses this field.
The location of the TLS slots is quite different between Windows NT and Windows 95. In Windows NT, this field contains a null pointer until the first time a TLS slot is used in the thread, then the system allocates memory for the TLS slots out of the default process heap. (In Windows NT, a buffer overrun of a HeapAlloc'ed block could trash your TLS data.)

Under Windows 95, this field always points to the TLS slots that are kept as part of the Ring 3 thread database.

Windows NT TIB fields
The meaning of some TIB data differs depending on whether you're running under Windows NT or Windows 95. This section of the OS/2 subsystem-Windows NT support running OS/2 1.X applications. For regular Win32 apps, this field appears to always be zero.

The 10h DWORD FiberData field's meaning depends on what version of Windows NT the thread is running. In the Windows NT 3.51 service pack 3 SDK update, WINNT.H describes this field as pointing to fiber data. Fibers are described in the accompanying HLP file as lightweight threads that are scheduled manually. Prior to the service pack update, this field was named "Version". Presumably this means what version of the system the thread expects to be running on, but I was unable to make sense of the values in this field.

The 20h DWORD processID field holds the process ID of the thread. The GetCurrentProcessId function in Windows NT 3.51 simply returns whatever is in this field.

The 24h DWORD threadID field holds the thread's ID. The GetCurrentThreadId function in Windows NT 3.51 returns the value in this field.

The segment pointed at by the Windows NT TIB actually extends far beyond the fields that I've described here. I've only mentioned the fields that fall within the first 34h bytes (the size of a Windows 95 TIB).

Windows 95 TIB fields
While the Windows NT TIB fields are relatively sedate, the Windows 95 TIB contains a fair amount of intriguing information. This section covers fields specific to the Windows 95 TIB.

The 0CH WORD pvTDB contains the task database selector for the task associated with the thread. The task database is a segment allocated from the 16-bit global heap, and the handle is known as an HTASK.

In Windows 95, every process (even a Win32 process) has a task database created for it. The 0EH WORD pvThunkSS field contains the selector that Windows 95 uses as the 16-bit stack selector when a thread thunks from 32-bit code to 16-bit code.
The 1CH WORD TIBFlags field is intended to hold various bit flags. The only known value is TIB_WIN32 (that is, 1).

If the low bit of this value is set, it's a 32-bit thread, otherwise it's a thread from a 16-bit process. The 1Eh WORD Win16MutexCount field is related to the thread's ownership of the Win16Mutex, which is a global critical section that only allows one thread at a time to be in 16-bit code.

Normally, this field's value is -1, which indicates that the thread doesn't own the Win16Mutex. As the thread enters and leaves thinking code, the value of this field is incremented and decremented accordingly.

The 20h DWORD DebugContext field normally contains the value zero. However, when you're debugging the thread's process, this field contains a pointer to a structure that contains register values and is similar to, but not the same as, the CONTEXT structure defined in WINNT.H.

The 24h DWORD pCurrentPriority field points to a DWORD containing the thread's scheduling priority. This will be some value between zero (lowest) and 31 (highest). The DWORD pointed to by this field is above 3GB in linear memory, which places it in VxD land. This makes sense, as threads are scheduled by the ring 0 Virtual Machine Manager (VMM). For normal priority threads, the priority DWORD will contain 9.

The 28h DWORD pvQueue field contains the message queue selector assigned to the thread.

Message queues are the means by which window messages get to the appropriate windows. In Windows 95, each thread can have its own message queue, but it is initially created without one. Therefore, this field may contain zero.

The 30h PVOID* pProcess field contains a linear address for the process database representing the process that owns the thread. However, this is not the same as a process handle or process ID.

Some Random Notes on TIBs
As I was experimenting with TIBs for this column, I came across some tidbits of information worth passing on.

First, in Windows 95, at offset 52h in each task database segment, you'll find the TIB selector for the primary thread in the process. At offset 54h in the task database, you'll find the linear address of the TIB. This is particularly interesting in that task databases are used by the 16-bit components of Windows 95. It appears that the 16-bit components may occasionally access thread-specific data in the TIB.

I also noticed the different uses of the FS register between Windows NT and Windows 95. Under Windows NT, the FS register is always the same for each thread's TIB. This implies that the linear address for the FS selector has to change whenever a thread switch occurs. In contrast, Windows 95 dedicates a different selector for each TIB (and hence, for each thread). The linear address of a Windows 95 TIB selector doesn't change.

I'll let you guess which method is kinder to system resources.
The SHOWTIB program
To bring the TIB to life, I wrote the SHOWTIB program (see Figure 1). SHOWTIB is a simple command-line program with two goals. The first is to create one or more threads. (You specify the actual number of threads on the command line. For example, "SHOWTIB 5" tells SHOWTIB to spin off five threads and display their TIBs.) The second goal is to display the various fields of each thread's TIB structure once all the threads are running. I show only the TIB structures for threads created by the primary thread, not for the primary thread itself.
In displaying the TIB for each thread, SHOWTIB first displays the fields common to all Win32 operating systems, then decides whether it's running on Windows NT or Windows 95. Depending on which system is running, SHOWTIB displays specific fields in the TIB relevant to the operating system. To make each TIB display come out coherent and in one piece, the DisplayTIB function guards the display code with a critical section.

There are two interesting pieces of code in SHOWTIB.CPP. Near the start of the DisplayTIB function, the code uses a bit of in-line assembler to grab the field at offset 18h in the TIB and stash it away into a pointer. Offset 18h is the linear address of the TIB. I did this so I could access the rest of the TIB with a regular pointer.

The alternative would have meant using in-line assembler and FS segment overrides to retrieve all of the values. Win32 compilers simply don't have a way to let you easily read from any segment other than the data segment (the DS register).
The second interesting piece of code is near the end of main.

After creating all the threads and storing all the corresponding thread handles into an array, I call WaitForMultipleObjects, passing in the array of thread handles. If I didn't do this, the primary thread could return from the main routine and call the exit function before the worker-bee threads had terminated. The result would be an incomplete display of all the various TIBs.

While Windows NT and Windows 95 are quite different under the hood, the TIB is one of the few areas you can rely on to be the same.

This isn't a coincidence; since threading is such an integral part of both operating systems, it's only natural that some common method of supplying thread-specific information would be needed.

The TIB is not described in any official documentation other than .H files. Nonetheless, it's an integral part of the Win32 specification that all Win32 implementations must conform to.

Friday, April 27, 2007

Doxygen to html to chm

Windows For those who like to use Doxygen to create some cool html help and who use source control you probably would like to create a unique file to be able to commit the file to your source control. The reason is that every time you will generate Doxygen new files will be generated based on your code change and you don't want to re-post all the html files every time. Doxygen itself can almost directly generate such a file!

The step-by-step howto:> Download the Microsoft HTML Help Workshop here: Microsoft MSDN page (the 'Download Htmlhelp.exe' link)

Install above program.

Edit the doxygen configuration file (located at 'doc\doxygen\html\doxygen.html.cfg') and change the following entries:

GENERATE_HTMLHELP = NO --> GENERATE_HTMLHELP = YES
BINARY_TOC = NO --> BINARY_TOC = YES
TOC_EXPAND = NO --> TOC_EXPAND = YES
HHC_LOCATION = --> HHC_LOCATION = "{path to MS HTML Help Workshop}\hhc.exe">

Follow the steps of generating the HTML version of the documentation: Online documentation page -- Or, if you don't like command line mucking about, ignore the above documentation link and use the Doxygen wizard called doxywizard which comes with the latest win32/linux doxygen version 1.3.9.1 (and maybe with earlier versions).

Run doxywizard>
Load the edited Doxygen configuration file (with the 'Load..' button)
Change the working directory to your project root dir (just strip off the 'doc\doxygen\html\') Click the 'Start' button
Let it munch a while-- And voila!

A nice and snappy myapp.chm file is waiting for you in the documentation directory!
Linux
On Linux such as Ubuntu if you need a solution to convert Doxygen to CHM you will have to use chmcmd
In this case you can't enable the CHM creation inside the doxy file because it won't work, doxygen will fail on executing hhc.exe on your Linux system. You couild use wine to emulate but this is like using a sledgehammer to kill a fly.... So the recommendation is to duplicate the doxyfile and disable all related CHM flag. Run doxygen once using the Windows doxy file to produce index.hhp, index.hhc and, index.hhk then edit the hhp file and remove unsupported options by chmcmd. See sample simple hhp file.
Maintain the hhp, hhc and hhk file in your source repository and each time you build the documentation on windows make sure to update the files to keep them in sync for your Linux CHM build Download fpc-2.6.0.x86_64-linux.tar from http://sourceforge.net/projects/freepascal/files/Linux/2.6.0/

android@U64:~$ cd pascal/
android@U64:~/pascal$ tar xvf fpc-2.6.0.x86_64-linux.tar
android@U64:~/pascal$ ls
fpc-2.6.0.x86_64-linux fpc-2.6.0.x86_64-linux.tar
android@U64:~/pascal$ cd fpc-2.6.0.x86_64-linux
android@U64:~/pascal/fpc-2.6.0.x86_64-linux$ ls
binary.x86_64-linux.tar demo.tar.gz doc-pdf.tar.gz install.sh
android@U64:~/pascal/fpc-2.6.0.x86_64-linux$
Install prefix (/usr or /usr/local) [/usr] : /usr/local
Installing compiler and RTL for x86_64-linux...
Installing utilities...
Install Textmode IDE (Y/n) ? Install FCL (Y/n) ? Y
Installing fcl-async
Installing fcl-base
Installing fcl-db
....
Install packages (Y/n) ? Y
Installing a52
Installing aspell
Installing bfd
Installing bzip2
Installing cairo
...
Installing zorba
Done.


Install documentation (Y/n) ? Y
Installing documentation in /usr/local/share/doc/fpc-2.6.0 ...
Done.

Install demos (Y/n) ? Y
Install demos in [/usr/local/share/doc/fpc-2.6.0/examples] :
Installing demos in /usr/local/share/doc/fpc-2.6.0/examples ...
Done.

Write permission in /etc.
Writing sample configuration file to /etc/fpc.cfg
Writing sample configuration file to /usr/local/lib/fpc/2.6.0/ide/text/fp.cfg
Writing sample configuration file to /usr/local/lib/fpc/2.6.0/ide/text/fp.ini
Writing sample configuration file to /etc/fppkg.cfg
Writing sample configuration file to /etc/fppkg/default

End of installation.

Refer to the documentation for more information.


Examples
This link has information about the TOC and Index files in chms: http://www.nongnu.org/chmspec/latest/Sitemap.html
These formats are based on HTML and use the following doctype:


The tag contains a tag providing information on the program that generated the files and a comment indicating the version of the file, e.g.:


The tag contains an tag that stores properties of the file in tags, followed by a
    tag, whose
  • tags have tags that store the properties of the Contents/Index items in tags. e.g.:


    Note that the Property Names and Property Values and tags are not case-sensitive, but HHW will always write all three in the default capitalization, when appropriate.
    Note that the tags are mostly in uppercase and the
  • tag is not closed; this is in compliance with the doctype.
    Some properties that were seen in HHA.dll that may or may not be used are Background Image, NumberImages, InformationTypeDecl, Secondary, Icon, Display, Keyword, Instruction, Section Title,
    Favorites, QueryType, SendEvent, SendMessage, HHI, Inclusive & Exclusive.
    This the beginning chunk of an autogenerated (the autogenerated TOC stinks) hhc (Table of Contents/TOC) file for the RTL: .hhk are in the same format but are for the Index pane and do not have
    subitems.


    For anybody determined to "roll their own", an absolutely minimal .hhp file reads something like


    chmcmd - creates a Compressed HTML help file (chm) using the hhp file

    android@U64:~/html/chmcmd index.hhp

    Will produce a nice index.chm file on an Android build system and use yoru favorite CHM APK viewer on your Mobile phone!

    Tuesday, March 27, 2007

    MIO 310 pocket PC GPS running Microsoft Pocket PC 4.2 with SIRF

    I love the MIO digiwalker 310 from mio-tech http://www.mio-tech.com/.
    This is such a fantastic device. It is running Microsoft Pocket PC 4.2 and this device shows how great and stable Microsoft OS can be when it is integrated the right way !
    When i say right way it is because the device is running some really good open source library. Also because there is no need to use some fancy overkill c# UI.
    The decision to choose c# or not on embedded device has to be driven by a good balance between scalability and speed. The faster the better....
    Using c# for the beauty of the code is not always good...
    The experience is the key to have a great product and the fact that the code has to be clean enough to maintain and understand is not always a must have... I mean clean is good but balancing managed and unmanaged code is important.

    A bried list of the open source libraries and sdk's

    Firefly embedded application engine www.pdamill.com
    FFUI embeded user interface www.pdamill.com
    Hekkus sound system www.shlzero.com
    Sqlite database Engine www.sqlite.org
    AGG graphics library www.antigrain.com
    Redblack tree
    Freetype font rendering library www.freetype.org
    Tremor OGG library www.Xiph.org

    The human interface is slick ! The device has a touch screen with buttons big enough not to use any stylus. SD/MMC reader, USB adapter, stereo headset plug and a way to lock the device.
    Your finger works fine but be carrefull when you leave an in and out burger place..
    Touched screens are not really fat friendly by nature.... After using the device for 3 days, i love it !
    i didn't see any negative point except maybe that i would have like to be able to pick a female english voice instead of the men english voice choosen by default for US. The french voice is female by defaut which is a nice french touch.

    Pros

    + Zoom is fantastic and realtime.
    + POI are great and intelligent. Based on your location and always sorted by distance.
    + POI can be browsed by categories as well.
    + Fantastic auto completion when you type anything using the keyboard
    + Voice is clear and loud
    + Night vision mode is great and the device switch automatically based on the time.
    + The device engine is smart enough to tell you to take the left lane on a free way
    when your driving segment is long enough
    + Calculation time are fast
    + Screen size is big
    + MP3 player

    Cons

    - no indoor outlet adapter to charge the device. Just a car adapter so if you have a bike
    you will need a USB charger at home.

    After all i think that a portable GPS device with additional audio features is the way to go. Using a phone which require activation seams to have more inconvenient than advantage. I have tons of ideas around GPS now by combining the POI database with mood and also by performing datamining and beeing able to analyse route history to use them as preferred route. I can't wait to see some video player on this device.

    Thursday, January 11, 2007

    CES 2007

    Hot products at CES 2007

    Number 1:

    My first kudos will naturally goes to the OQO model 02
    check out http://www.oqo.com/ this device is a kick ass
    PC with : hold your breath...

    1.5 GHZ VIA C7N ULV processor
    60 GB harddrive
    1GB DDR2 SDRAM
    Microsoft Vista capable
    integrated 3G EV-DO WWAN
    WIFI 802.11bg
    integrated bluetooth 2.0
    VX700 graphics chipset with integrated GPU
    VGA/Ethernet adapter
    3 hours of battery life standard
    6 hours double capacity
    illuminated keyboard with mouse
    USB 2.0
    HDMI output digital video port

    I want this device sssoooo bad

    Number 2:
    Certainly the All in one LG Super Multi Blue
    Blue Ray Disc Rewriter & HD DVD drom Drive
    GGW-H10N

    Number 3:
    Motorola new bluetooth stereo headset. Super thin.
    I want one soooo bad to listen to my music on my Q

    Number 4:
    VIDA BOX Slim High Definition Windows Media Center
    http://www.vidabox.com/

    Number 5:
    Toshiba HDDVD internal burner drives.

    Number 6:
    Those cool USB picture digital frames
    such as http://www.ceiva.com/ as a cool gift for you familly and friends

    Number 7:
    Niveus HDTV box using intel VIIV and those interesting
    Digital Wireless media adapter to stream from one place
    to an other. Not sure if this will let the user stream
    anything or just on demand pay per view content ???

    Number 8:
    The demo of local.live.com for the city of LasVegas.

    Number 9:
    Visteon wireless charger for portable electronics devices
    such as moto Q
    http://www.mobilemag.com/content/100/358/C11152/

    Number 10:
    Yet an other Windows laptop but close to a MacBook Design
    LG Mobile WiMAX/HSDPA laptop


    Why CES is always after christmas ::))

    iPhone talk

    A really great article explaining the pros and cons of the iphone

    http://www.oreillynet.com/xml/blog/2007/01/why_iphone_is_not_for_me.html

    BTW Steve i really like what you do but come on... not to provide the ability to replace battery on those iPod is a bummer for all of us.

    Tuesday, December 05, 2006

    How do I create a C++ application that embeds the Flash Player

    "How do I create a C++ application that embeds the Flash Player, and how can the embedded Flash Player communicate with the C++ application?"

    Mike Chambers wrote a terrific article on the subject (his was written inC#), but the DevNet Resource Kit it was a part of is no longer available.

    The article is still up for legacy purposes though:http://www.adobe.com/devnet/flash/articles/stock_history.html

    Basically, you embed the Flash Player ActiveX control in your application, and then there's an API exposed through the ActiveX control.

    Here's a page that details the methods and events exposed by the Flash Player ActiveX control scripting interface

    http://www.adobe.com/cfusion/knowledgebase/index.cfm?id=tn_12059

    Here's a PDF about
    how to write applications in VB which embed the FlashPlayer:

    http://www.adobe.com/devnet/flash/articles/flash_vb.pdf

    And, for your reference, Mike's blog post, which is result #1 when you Google "Embed Flash Player in C++ app"

    http://weblogs.macromedia.com/mesh/archives/2003/07/embedding_flash.html


    Mike Chambers Flash C# integration

    http://www.adobe.com/devnet/flash/articles/stock_history.html


    Macromedia Flash Slide Show (VB Flash sample application)