Exetools - Flushing input stream

Exetools (https://forum.exetools.com/index.php)

- General Discussion (https://forum.exetools.com/forumdisplay.php?f=2)

- - Flushing input stream (https://forum.exetools.com/showthread.php?t=13648)

Flushing input stream

I've a little programming related problem and I don't know how to solve it.

A console program (lets call it "prog1.exe") has the following pseudo-code:

Code:

while not (Ctrl_Break)

{

  x = fetch_stats();     // Read some info

  writeconsole(x);       // Output the info to the console

  wait(60);              // Wait 60 seconds and repeat the whole process

}

I want to log the output of the program both to screen and to a file. So I wrote a little program ("tee.exe") which does:

Code:

while not (EndOfFile(StdIn))

{

  c = read(StdIn);     // Read one char from StdIn

  write(StdOut, c);    // Write one char to StdOut

  write(FILE, c);      // Write the same char to a file

}

In theory, executing "prog1.exe | tee.exe" would redirect all output from "prog1.exe" as input to "tee.exe", while "tee.exe" would write the input it gets to both the console and a file.

The problem is the output buffer. For performance reasons, the OS doesn't write the output of "prog1.exe" to "tee.exe" char-by-char, but waits until the output buffer is full and then writes the whole buffer to "tee.exe". Since "prog1.exe" does only output a few bytes every minute, it takes about 40 minutes before the output buffer is full and transfered to "tee.exe".

Running "prog1.exe >> output.txt" has a similar result: There is no output to the console any more und the file is also only updated when the output buffer is full.

I don't have access to "prog1.exe" or it's source code, so I can't flush the output buffer. Flushing the input buffer is not possible for some reason. The only workarounds I found for the problem describe how to flush the keyboard buffer.

Does anybody have an idea how to solve this problem?

My goal is simple:
1. See the console output in real time
2. Write the console output to a file

There is an ANSI C _flushall() command that causes all open buffers to be flushed to file. There is also a stream.flush() function if you are using C++ stream i/o. They may help, but I am not quite sure what you mean by flushing an input buffer, but if you don't have the source code to prog1 I don't see how you can affect it's behaviour. Unless.. if you can redirect stdout to a pipe then i/o can be made non-buffered, but I am still unsure if it can work.

Git

"prog1.exe" writes to the console. The default behavior ist unbuffered. The output will be written to the console every 60 seconds.

As soon as the output is redirected (to file ">>" or to pipeline "|"), the OS (both Windows and Linux) switch to buffered output. This means the output will be only written when the output buffer is full, which takes about 40 minutes each time. Somebody could "flush" the output buffer, which would write the contents of the buffer at once, even if it isn't full yet. Or somebody could enable unbuffered output again. But both changes would need to change "prog1.exe", something which is not possible in this situation.

The other end of the pipeline belongs to "tee.exe" (full source available). It reads all data from the standard input. The problem is that "tee.exe" only gets something to read when "prog1.exe" sends something, which will only happen when its output buffer is full.

Flushing the input buffer would mean something like "Dear OS, please send me all data from prog1.exe, even if the buffer is not full yet". Unfortunately this kind of command is not supported for some reason.

I also tried to set "stdin" to unbuffered without success. Using a very small buffer for "stdin" didn't help either, data was only available for read when the output buffer of "prog1.exe" was full, not earlier.

would a standard close(FILE) not work with stderr and stdout ?

Here is something which I hope it helps:

First, the program that writes to stdout. Since you've written that you don't have access to the source, this one is exchangeable:

Code:

#include <windows.h>



int main()

{

    char c[] = "";

    unsigned long count;

    HANDLE stdout = GetStdHandle(STD_OUTPUT_HANDLE);

    for(int i = 0; i < 10; i++)

    {

        c[0] = i + 0x30;

        WriteFile(stdout, c, 1, &count, NULL);

        Sleep(1000);

    }

    c[0] = 0x0;

    WriteFile(stdout, c, 1, &count, NULL);

    CloseHandle(stdout);

    return 0;

}

Now the one that reads from stdin and writes through at once:

Code:

#include <windows.h>



int main()

{

    char c[] = " ";

    unsigned long incount, outcount, foutcount;

    HANDLE stdin = GetStdHandle(STD_INPUT_HANDLE);

    HANDLE stdout = GetStdHandle(STD_OUTPUT_HANDLE);

    HANDLE File;

    

    File = CreateFile("test.txt", 

                       GENERIC_WRITE, 

                       0, 

                       NULL, 

                       CREATE_NEW, 

                       FILE_ATTRIBUTE_NORMAL|FILE_FLAG_WRITE_THROUGH, 

                       NULL);

    while(c[0])

    {

        ReadFile(stdin, &c[0], 1, &incount, NULL);

        FlushFileBuffers(stdin);

        WriteConsole(stdout, c, 1, &outcount, NULL);

        WriteFile(File, c, 1, &foutcount, NULL);

        FlushFileBuffers(File);

    }

    CloseHandle(File);

    CloseHandle(stdout);

    CloseHandle(stdin);

    return 0;

}

I'm pretty sure you can adjust it to your needs.
If that doesn't help you, I'm wrong and I didn't get you right.
I'm sorry then.

Best regards
*RemedY*

I'm so sorry.
I just saw that this thread is actually seven month old :eek:
I was fooled by redbull's reply and didn't recognise the real age of this thread.
Shame on me.

*RemedY*

Anyways, maybe my reply above is useful for someone.

The thread is not old to me, the problem still exists. ;)

Thank you for the code you posted, but it doesn't work either.
"ReadFile(stdin)" will not return anything until the output buffer is full.
"FlushFileBuffers(stdin)" will not change anything about this behaviour.

And your code contains a small error: "while(c[0])" will generate an endless loop, since c[0] will contain the last char which was read, which won't be "0" (false condition) when reaching EndOfFile or closing the handle.

@redbull:
While CloseHandle(stdout) might work, it would close the handle, making any further output impossible. "prog1.exe" is also running at higher privileges than "tee.exe", so even if I wanted to close the handle, I wouldn't have permission to do so.

have you tried mtee?

Code:

http://www.commandline.co.uk/mtee/index.html

@Kerlingen

no, there is no error.
Did you try the code? I guess you didn't.
In the first program the last char written is 0x0, thus there is no endless loop. Also, it writes directly to stdout and to the file, not only when the buffer is full. At least, it works with the two small programs above under Win7. As I've written, I'm pretty sure you can adjust it to your needs. If you can't get it to work, I'd be interessted in your input file.

Best regards
*RemedY*

There is no such thing like an "input file", there is just a service daemon which outputs his log messages to the console.

I want to see the messages in real time on the console and additionally log them to a file.

Ah, OK.

Win or *nix?

I want to know that, because I will set up a small daemon then and try it with that. I hope I can help you.

Best regards
*RemedY*

That really doesn't sound too much to ask for, does it?. it must be possible!

Git

It should be possible to change file handler to be non-blocking. Not sure of windows calls, but on *nix this can be done by
flags = fcntl(fd, F_GETFL, 0);
fcntl(fd, F_SETFL, flags | O_NONBLOCK);