You are not logged in.

Writing Binary files at high speed

Dear visitor, welcome to QtForum.org. If this is your first visit here, please read the Help. It explains in detail how this page works. To use all features of this page, you should consider registering. Please use the registration form, to register here or read more information about the registration process. If you are already registered, please login here.

ksi

Beginner

  • "ksi" started this thread

Posts: 11

  • Send private message

1

Wednesday, December 28th 2011, 2:11pm

Writing Binary files at high speed

Hello everyone,

Im having trouble saving information into a binary file, and i guess thats a problem concerning speed. Mi program recieves from an UDP SOCKET 1280 bytes every 2.5 ms. From those 1280 bytes, i only need to save half of them, 640 bytes. To do this, i save a specific amount into a static variable, and I save them after some time (to avoid opening and closing a file really often).

What im saving is the following struct.

MySQL queries

 
1
2
3
4
5

struct info 

{ 
unsigned char trama [CANT_PAQ][640]; 
}


Where CANT_PAQ is a define, for example, if I set it into 1000, i will store 1000 arrays of 640 bytes before saving them into my file. The problem is that when the file is created and i loose some information. For example, i wont have any trouble saving the first 999 arrays, bue then, if i recieve 10 more, only 5 will be saved. Apparently creating the file takes some time, and im loosing some packages. The code I use to save the binary file is the following:

MySQL queries

 
1
2
3
4
5
6
7
8

QString path; 
path="Package_"; 
path=path+path.number(numer_files); /*number_files tells how many files i have already saved, so the first one will be package_1.dat, 
and the second one package_2.dat, etc */ 
path.append(".dat"); 
ofstream file (path.toascii(),ios::out|ios::binary); 
file.write(reinterpret_cast<char*>(&temporal_struct),sizeof(info)); /*Temporal struct is where i have saved the information and its a "info"struct */ 
file.close();


Thats how i save information, but apparenlty its not fast enough. Ive tired varying CANT_PAQ, so i save larger files less often, or smaler files really fast, but im always loosing some information. I would appreciate if someone could tell me if theres a faster way to do this, or a way to give a higher priorty to this part of the program.

Thanks in advance!

This post has been edited 1 times, last edit by "ksi" (Dec 28th 2011, 2:17pm)

Amleto

Professional

Posts: 1,430

  • Send private message

2

Wednesday, December 28th 2011, 2:22pm

your method is inherently flawed as you have found out.

you need to save the data until it is written. when it is written, you can then clear the data cache.

Also, you REALLY shouldn't be needing reinterpret cast for this simple problem. what is wrong with temporal_struct.trama ?


You have a typical producer/consumer problem - you will find lots of threading examples as a solution to this problem. This would hopefully suggest to you that using a static variable for your data cache is not a good solution.
If you have a problem, CUT and PASTE your code. Do not retype or simplify it. Give a COMPLETE and COMPILABLE example of your problem. Otherwise we are all guessing the problem from a fabrication where relevant details are often missing.

ksi

Beginner

  • "ksi" started this thread

Posts: 11

  • Send private message

3

Wednesday, December 28th 2011, 2:47pm

Actually thats what i do. Every time i recieve a package I save it into the static temporal_struct, when this struct gets full, i write the binary file, erase the struct, and start all over again. I didnt copy that part of the code because its quite basic, and the problem is not there...

Amleto

Professional

Posts: 1,430

  • Send private message

4

Wednesday, December 28th 2011, 2:53pm

So tell me this, with your static variable for your network data, what happens when you are writing data (the variable is full) and then more data is sent?


how do you save the data when there is no where to save it? That is your problem, not slow write speed. You havent correctly recognised your problem and you are therefore looking at totally the wrong place in your code.
If you have a problem, CUT and PASTE your code. Do not retype or simplify it. Give a COMPLETE and COMPILABLE example of your problem. Otherwise we are all guessing the problem from a fabrication where relevant details are often missing.

ksi

Beginner

  • "ksi" started this thread

Posts: 11

  • Send private message

5

Wednesday, December 28th 2011, 3:10pm

MySQL queries

 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

void MainWindow::processPendingDatagrams() 
{ 
     QString path; 
     QByteArray datagram; 
     static info temporal_struct; 
     static int index=0; 
     datagram.resize(udpsocket->pendingdatagramsize()); 
     udpsocket->readDatagram(datagram.data(), datagram.size()); 
     for (int i =0; i<1280; i+=2) 
          temporal_struct.trama[index][i/2]=datagram[i]; /*Here i erase half of the information, that i do not need*/ 
     index++; 
     if((index%CANT_PAQ)==0) /*Here i know i have completed the buffer*/ 
     { 
          number_files++; 
          index=0; 
          path="Package_"; 
          path=path+path.number(number_files); /*number_files tells how many files i have already saved, so the first one will be package_1.dat, 
          and the second one package_2.dat, etc */ 
          path.append(".dat"); 
          ofstream file (path.toascii(),ios::out|ios::binary); 
          file.write(reinterpret_cast<char*>(&temporal_struct),sizeof(info)); /*Temporal struct is where i have saved the information and its a "info"struct */ 
          file.close(); 
} 
}


The program enters here (ProcessPendingDatagrams) everytime it recieves a package

This post has been edited 1 times, last edit by "ksi" (Dec 28th 2011, 3:16pm)

ksi

Beginner

  • "ksi" started this thread

Posts: 11

  • Send private message

6

Wednesday, December 28th 2011, 3:21pm

I cant think of a faster way to do it. I mean, if i recieve a package while the program is still writing the file, then this obviously wont work. But, how could i fix it?

Amleto

Professional

Posts: 1,430

  • Send private message

7

Wednesday, December 28th 2011, 3:45pm

where does your struct data get reset? where does index get reset? Please post you actual code otherwise I don't know which issues to ignore or which to bring up.


If your method is a slot, it wont be called for a second time when it is still executing, so that is not your problem either (unless you call it directly from another thread, but qt wont do this).

Your problem is that you may throw away data here

Source code

 
1
2
3

     udpsocket->readDatagram(datagram.data(), datagram.size()); 
     for (int i =0; i<1280; i+=2) 
          temporal_struct.trama[index][i/2]=datagram[i];


what if there is another packet's worth in datagram? You have just ignored it.

From a adesign perspective, you method does too many things - it is saving data, mostly, and sometimes writing data. This is a good circumstance where you separate tasks by using another thread for writing to disk.
If you have a problem, CUT and PASTE your code. Do not retype or simplify it. Give a COMPLETE and COMPILABLE example of your problem. Otherwise we are all guessing the problem from a fabrication where relevant details are often missing.

This post has been edited 1 times, last edit by "Amleto" (Dec 28th 2011, 3:52pm)

ksi

Beginner

  • "ksi" started this thread

Posts: 11

  • Send private message

8

Wednesday, December 28th 2011, 3:50pm

The struct data doesnt get reseted, i just put Index=0, so next information will be overwritten (thats in the code i posted)

MySQL queries

 
1
2
3
4
5
6

if((index%CANT_PAQ)==0) /*Here i know i have completed the buffer*/ 
     { 
          number_files++; 
          index=0;                   /* HERE THE BUFFER GETS RESETED*/
          path="Package_"; 
          ....


But what you have said about loosing information its a good idea, so im going to check if im loosing information there...

Amleto

Professional

Posts: 1,430

  • Send private message

9

Wednesday, December 28th 2011, 3:53pm

ah yes, missed that bit. It is probably a good idea to reset the data - it makes debugging a lot easier.
If you have a problem, CUT and PASTE your code. Do not retype or simplify it. Give a COMPLETE and COMPILABLE example of your problem. Otherwise we are all guessing the problem from a fabrication where relevant details are often missing.

ksi

Beginner

  • "ksi" started this thread

Posts: 11

  • Send private message

10

Wednesday, December 28th 2011, 3:55pm

When i just started the program, i used to do that, but as i was having speed problems, i tried to reduce the code to make it as fast as i could

Amleto

Professional

Posts: 1,430

  • Send private message

11

Wednesday, December 28th 2011, 5:29pm

but now we know your problems aren't speed ;)
If you have a problem, CUT and PASTE your code. Do not retype or simplify it. Give a COMPLETE and COMPILABLE example of your problem. Otherwise we are all guessing the problem from a fabrication where relevant details are often missing.

Similar threads

Used tags

Binary files write read socket UDP