In linux, why a normal process can still run while real time process doesn't exit?

Question

I write a c program to test linux scheduler. this is my code:

#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>
#include <pthread.h>

void Thread1()
{
    sleep(1);
    int i,j;
    int policy;
    struct sched_param param;
    pthread_getschedparam(pthread_self(),&policy,&param);
    if(policy == SCHED_OTHER)
        printf("SCHED_OTHER\n");
    if(policy == SCHED_RR)
        printf("SCHED_RR 1 \n");
    if(policy==SCHED_FIFO)
         printf("SCHED_FIFO\n");
    /* for(i=1;i<100;i++) */
    while(1)
    {
         for(j=1;j<5000000;j++)
        {
        }
        printf("thread 1\n");
    }
    printf("Pthread 1 exit\n");
}

void Thread2()
{
    sleep(1);
    int i,j,m;
    int policy;
    struct sched_param param;
    pthread_getschedparam(pthread_self(),&policy,&param);
    if(policy == SCHED_OTHER)
        printf("SCHED_OTHER\n");
    if(policy == SCHED_RR)
        printf("SCHED_RR\n");
    if(policy==SCHED_FIFO)
        printf("SCHED_FIFO\n");
    /* for(i=1;i<10;i++) */
    while(1)
    {
        for(j=1;j<5000000;j++)
        {
        }
        printf("thread 2\n");
    }
    printf("Pthread 2 exit\n");
}

void Thread3()
{
    sleep(1);
    int i,j;
    int policy;
    struct sched_param param;
    pthread_getschedparam(pthread_self(),&policy,&param);
    if(policy == SCHED_OTHER)
        printf("SCHED_OTHER\n");
    if(policy == SCHED_RR)
        printf("SCHED_RR \n");
    if(policy==SCHED_FIFO)
        printf("SCHED_FIFO\n");

    /* for(i=1;i<10;i++) */
    while(1)
     {
        for(j=1;j<5000000;j++)
         {
         }
         printf("thread 3\n");
     }
     printf("Pthread 3 exit\n");
}

int main()
{
    int i;
    i = getuid();
    if(i==0)
        printf("The current user is root\n");
    else
        printf("The current user is not root\n");

    pthread_t ppid1,ppid2,ppid3;
    struct sched_param param;

    pthread_attr_t attr3,attr1,attr2;
    pthread_attr_init(&attr1);
    pthread_attr_init(&attr3);
    pthread_attr_init(&attr2);

    param.sched_priority = 97;
     pthread_attr_setschedpolicy(&attr1,SCHED_RR);
    pthread_attr_setschedparam(&attr1,&param);
    pthread_attr_setinheritsched(&attr1,PTHREAD_EXPLICIT_SCHED);

    param.sched_priority = 98;
    pthread_attr_setschedpolicy(&attr2,SCHED_RR);
    pthread_attr_setschedparam(&attr2,&param);
    pthread_attr_setinheritsched(&attr2,PTHREAD_EXPLICIT_SCHED);

    pthread_create(&ppid3,&attr3,(void *)Thread3,NULL);
    pthread_create(&ppid2,&attr2,(void *)Thread2,NULL);
    pthread_create(&ppid1,&attr1,(void *)Thread1,NULL);

    pthread_join(ppid3,NULL);
    pthread_join(ppid2,NULL);
    pthread_join(ppid1,NULL);
    pthread_attr_destroy(&attr2);
    pthread_attr_destroy(&attr1);
    return 0;
}

In this program, I create one thread with default attribute and two thread whose schedule policy is SCHED_RR and specific priority. My question is: When I run the program, I can barly see the output from thread 1. How can this happen ? I think that thread 1 and thread 2 are real time process and thread 3 is a normal process. So thread 3 will never run until thread 1 and thread 2 exit. But In my program thread 1 and thread 2 never exit, so I expect that only thread 2 can actually run. Why I can see the output of thread 2 and thread 3 and can't see the output of thread 1?

Dunno. Does thread1 get executed, at all? If you break on the first line, does it fire? — Martin James, Sep 17 '17 at 10:06
Also, how many cores do you have available? I mean, if you have 4 cores, you will probably see different behaviour than with one. — Martin James, Sep 17 '17 at 10:08
Actually , I run linux on virtual machine . I use lscpu command and there is one CPU present. I try to stop the program during its running and I don't see any output of thread 1. I try outputting blank line at the entrance of thread 1 to see whether thread 1 function is called or not and I see blank lines at the beginning of program's running. What's the problem? — Troy922, Sep 17 '17 at 13:00
regarding: `pthread_attr_destroy(&attr2); pthread_attr_destroy(&attr1);` what about `attr3`? — user3629249, Sep 17 '17 at 13:59
regarding these kinds of statement: `void Thread3()` the correct signature for a pthread is: `void *Thread3( void *data)` then if the `data` is not going to be used: `(void)data;` as the first line in the body of the function. — user3629249, Sep 17 '17 at 14:02
when compiling, enable the warnings, then fix those warnings. (for `gcc`, at a minimum use: `-Wall -Wextra -pedantic -Wconversion -std=gnu11` ) — user3629249, Sep 17 '17 at 14:06
the function: `getuid()` returns a `__uid_t` (a unsigned int), not a `int` — user3629249, Sep 17 '17 at 14:11
when calling system functions, like `pthread_create()`, always check the returned value to assure the operation was successful. — user3629249, Sep 17 '17 at 14:23
Finally I know why there is a output of blank line. pthread_create() will call Thread1() and immediately return . So if I add some code at the beginning of Thread1(), these codes will be executed. — Troy922, Sep 19 '17 at 01:17

myaut · Accepted Answer · 2017-09-18T15:53:23.640

Thread 3 can be run because 0.05s is reserved for non-runtime tasks, disable it via echo -1 > /proc/sys/kernel/sched_rt_runtime_us:

The default values for sched_rt_period_us (1000000 or 1s) and sched_rt_runtime_us (950000 or 0.95s). This gives 0.05s to be used by SCHED_OTHER (non-RT tasks). These defaults were chosen so that a run-away realtime tasks will not lock up the machine but leave a little time to recover it. By setting runtime to -1 you'd get the old behaviour back.

sched-rt-group.txt

Thread 1 cannot be run because thread 2 has higher priority (note that 99 is the highest RT priority in pthread calls, this is contradictory to internal Linux numbering and explained here), and round-robin scheduling is only performed within a queue of process with the same priority:

SCHED_RR tasks are scheduled by priority, and within a certain priority they are scheduled in a round-robin fashion. Each SCHED_RR task within a certain priority runs for its allotted timeslice, and then returns to the bottom of the list in its priority array queue.

Understanding the Linux 2.6.8.1 CPU Scheduler

Some notes on how to to find out why this is happening. To do so, we will need source code and dynamic tracer like SystemTap. Switching threads (or more precisely context switch) can be traced via scheduler.ctxswitch probe which is wrapper around sched_switch tracepoint.

Checking source code around that tracepoint says that new task is handled by __schedule function which calls pick_next_task:

3392    next = pick_next_task(rq, prev, cookie);
...
3397    if (likely(prev != next)) {
...
3402        trace_sched_switch(preempt, prev, next);

Crawling into source code leads us to pick_next_task_rt, which, under certain conditions returns NULL instead of our threads. It's SystemTap time!

# stap -e 'probe kernel.function("pick_next_task_rt").return { 
    if ($return == 0) {
        println($rq->rt$) } }' -c ./a.out
...
{.active={...}, .rt_nr_running=2, .highest_prio={...}, .rt_nr_migratory=2, .rt_nr_total=2, 
 .overloaded=1, .pushable_tasks={...}, .rt_throttled=1, .rt_time=950005330, .rt_runtime=950000000, 
 .rt_runtime_lock={...}, .rt_nr_boosted=0, .rq=0xffff8801bfc16c40, 
 .leaf_rt_rq_list={...}, .tg=0xffffffff81e3d480}
SCHED_OTHER

So, it seems that rt_time is greater than 950ms and rt_throttled flag is set when we switch to SCHED_OTHER. Further googling leads to this answer and documentation linked above.

Thanks for your answer! Now I understand why thread 3 can run . But why thread 1 doesn't run? — Troy922, Sep 18 '17 at 01:05

score -1 · Answer 2 · answered Sep 17 '17 at 15:17

the following proposed code:

which only checks some of the error status values from system functions
shows why the OP is having a problem
follows the axiom: only one statement per line and (at most) one variable declaration per statement.
corrects many of the problems noted by the compiler in the OPs code

Note: this was run on ubuntu linux 16.04

#include <stdio.h>
//#include <unistd.h> -- contents not used
#include <stdlib.h>
#include <pthread.h>


void *Thread1( void *data );
void *Thread2( void *data );
void *Thread3( void *data );


void *Thread1( void *data )
{
    (void)data;
    sleep(1);

    int policy;

    struct sched_param param;

    pthread_getschedparam(pthread_self(),&policy,&param);

    if(policy == SCHED_OTHER)
        printf("SCHED_OTHER\n");

    if(policy == SCHED_RR)
        printf("SCHED_RR 1 \n");

    if(policy==SCHED_FIFO)
         printf("SCHED_FIFO\n");

    for( size_t i=0; i<1000; i++ )
    {
        printf("thread 1\n");
    }

    printf("Pthread 1 exit\n");
    pthread_exit( NULL );
}


void *Thread2( void *data )
{
    (void)data;
    sleep(1);

    int policy;
    struct sched_param param;

    pthread_getschedparam(pthread_self(),&policy,&param);

    if(policy == SCHED_OTHER)
        printf("SCHED_OTHER\n");

    if(policy == SCHED_RR)
        printf("SCHED_RR\n");

    if(policy==SCHED_FIFO)
        printf("SCHED_FIFO\n");

    for( size_t i = 0; i< 1000; i++ )
    {
        printf("thread 2\n");
    }

    printf("Pthread 2 exit\n");
    pthread_exit( NULL );
}


void *Thread3( void *data )
{
    (void)data;
    sleep(1);

    int policy;
    struct sched_param param;

    pthread_getschedparam(pthread_self(),&policy,&param);

    if(policy == SCHED_OTHER)
        printf("SCHED_OTHER\n");

    if(policy == SCHED_RR)
        printf("SCHED_RR \n");

    if(policy==SCHED_FIFO)
        printf("SCHED_FIFO\n");

     for( size_t i = 0; i<1000; i++ )
     {
         printf("thread 3\n");
     }

     printf("Pthread 3 exit\n");
     pthread_exit( NULL );
}


int main( void )
{
    unsigned i;
    i = getuid();

    if(i==0)
        printf("The current user is root\n");
    else
        printf("The current user is not root\n");

    pthread_t ppid1;
    pthread_t ppid2;
    pthread_t ppid3;
    struct sched_param param;

    pthread_attr_t attr3;
    pthread_attr_t attr1;
    pthread_attr_t attr2;

    pthread_attr_init(&attr1);
    pthread_attr_init(&attr3);
    pthread_attr_init(&attr2);

    param.sched_priority = 97;
    pthread_attr_setschedpolicy(&attr1,SCHED_RR);
    pthread_attr_setschedparam(&attr1,&param);
    pthread_attr_setinheritsched(&attr1,PTHREAD_EXPLICIT_SCHED);

    param.sched_priority = 98;
    pthread_attr_setschedpolicy(&attr2,SCHED_RR);
    pthread_attr_setschedparam(&attr2,&param);
    pthread_attr_setinheritsched(&attr2,PTHREAD_EXPLICIT_SCHED);

    int create3 = pthread_create( &ppid3, &attr3, Thread3, NULL );
    if( create3 )
    {
        fprintf( stderr, "pthread_create for thread 3 failed with status: %d\n", create3 );
        //exit( EXIT_FAILURE );
    }

    int create2 = pthread_create( &ppid2, &attr2, Thread2, NULL );
    if( create2 )
    {
        fprintf( stderr, "pthread_create for thread 2 failed with status: %d\n", create2 );
        //exit( EXIT_FAILURE );
    }

    int create1 = pthread_create( &ppid1, &attr1, Thread1, NULL );
    if( create1 )
    {
        fprintf( stderr, "pthread_create for thread 1 failed with status: %d\n", create1 );
        //exit( EXIT_FAILURE );
    }


    pthread_join(ppid3,NULL);
    pthread_join(ppid2,NULL);
    pthread_join(ppid1,NULL);


    pthread_attr_destroy(&attr3);
    pthread_attr_destroy(&attr2);
    pthread_attr_destroy(&attr1);


    return 0;
}

running the program as a normal user results in:

The current user is not root
pthread_create for thread 2 failed with status: 1
pthread_create for thread 1 failed with status: 1
SCHED_OTHER
thread 3  <-- repeats 1000 times
Pthread 3 exit

running the program as root results in:

The current user is root
SCHED_RR
thread 2
thread 2
thread 2
thread 2
thread 2
thread 2
thread 2
thread 2
thread 2
thread 2
thread 2
thread 2
thread 2
thread 2
thread 2
thread 2
thread 2
thread 2
SCHED_OTHER
thread 3
thread 3
thread 3
thread 3
thread 3
thread 3
thread 2
SCHED_RR 1 
thread 1
thread 1
thread 2
thread 3
thread 3
thread 3
thread 2
thread 1
thread 1
thread 1
thread 1
thread 1
... continues for a total of 3000 lines and
... as each thread completes it outputs:
pthread 1 exit -OR- pthread 2 exit -OR- pthread 3 exit

To answer the OPs questions:

Every time the REAL TIME threads call printf() other threads get a chance to run. In part, this is due to the context switching being performed including the context switching to switch between user and OS processes.

`printf` may switch between user mode and kernel mode if it flushes its output buffer and performs a `write` system call, but that's not the same thing as a context switch between processes. — John Kugelman, Sep 17 '17 at 15:30
There is also the 'time slice' context switching and the RTC updates context switching, and and and. I was just stating a simple example — user3629249, Sep 17 '17 at 15:44
Thanks for your answer. Your code looks good, but it changes the trouble maker of my code. I purposely make the loop of every thread permanently so my problem can be seen. I am working on a project which needs every thread to loop forever. So I want to know how threads are scheduled under this specific circumstance。 — Troy922, Sep 18 '17 at 01:12
@Troy922, Every thread of execution gets a 'time slice' including the `main` thread. each thread causes a context switch. Each call to `printf()` is an I/O operation, which will result in the current line of execution to be blocked. Each block (can and usually will) cause a context switch. So there are LOTS of context switches and at each context switch, another thread (including the OS) gets a chance to run. — user3629249, Sep 20 '17 at 17:13
@Troy922, You can easily modify the proposed code to have infinite loops rather than the proposed counted loops. It will make no difference in the overall operation, except the code would never exit — user3629249, Sep 20 '17 at 17:15
@JohnKugelman, the function `printf()` uses interrupts to actually output the data to the display terminal. Each such interrupt is a context switch. — user3629249, Sep 20 '17 at 17:18

In linux, why a normal process can still run while real time process doesn't exit?

2 Answers2