You are not logged in.

Dear visitor, welcome to QtForum.org. If this is your first visit here, please read the Help. It explains in detail how this page works. To use all features of this page, you should consider registering. Please use the registration form, to register here or read more information about the registration process. If you are already registered, please login here.

Tom

Trainee

  • "Tom" is male
  • "Tom" started this thread

Posts: 48

Location: Darmstadt, Germany

  • Send private message

1

Wednesday, September 8th 2004, 3:30pm

Reading chunks from the byte stream - TCP

Hi everyone,

does anyone know how to read chunks from the byte stream of a socket connection (TCP)? Or maybe someone has a good tutorial? :D

Would be great, I am getting dumb trying it!
The problem is, I have to send objects. Therefore I cut them down to their attributes, send the attributes and put them back into an object. The problem is that not only one object is sent at one time. Sometimes TCP packs more than one object and sends it. Here I need to read one part of the message, create an object, process it, I have to do this as many times, as objects are in the packet.

Any help appreciated.

Many thanks in advance

Ciao
Tom
---
There are only 10 types of people in the world:
Those who understand binary and those who don't.
---

  • "wysota" is male

Posts: 4,276

Location: Warsaw, POLAND

  • Send private message

2

Wednesday, September 8th 2004, 4:07pm

RE: Reading chunks from the byte stream - TCP

Put some kind of separators into the stream, then at the receiving end read the stream until you reach the separator, process the object and carry on reaing until the next separator.

Tom

Trainee

  • "Tom" is male
  • "Tom" started this thread

Posts: 48

Location: Darmstadt, Germany

  • Send private message

3

Wednesday, September 8th 2004, 6:30pm

@wysota,

damn it, that would be too simple! 8)
The more you code, the more you start thinking complicated.

Thank you for answering.

Ciao
Tom
---
There are only 10 types of people in the world:
Those who understand binary and those who don't.
---

  • "wysota" is male

Posts: 4,276

Location: Warsaw, POLAND

  • Send private message

4

Thursday, September 9th 2004, 12:09am

Simple solutions are almost always the best ones - you can't make so many mistakes as with more complex ideas. :D

Tom

Trainee

  • "Tom" is male
  • "Tom" started this thread

Posts: 48

Location: Darmstadt, Germany

  • Send private message

5

Thursday, September 9th 2004, 9:18am

Hi,

I got the following code:
Client Send()

Source code

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
	else if(m_Status == SYNC || m_Status == START)
	{
		if(identifier == "event")
		{
			//Send identifier to let the client know what is being send
			clientSocket->writeBlock((char*) &identifierLength, sizeof(identifierLength));
			clientSocket->writeBlock(const_cast<char*>(identifier.data()), identifierLength);
			
			qDebug("CLIENT<%d>: Sending Event: comp<%d> elem<%d> event<%d> name<%s>", m_Status, event.componentID, event.elementID, event.eventID, event.name.data());

			clientSocket->writeBlock((char*) &event.elementID, sizeof(event.elementID));
			clientSocket->writeBlock((char*) &event.eventID, sizeof(event.eventID));
			clientSocket->writeBlock((char*) &event.componentID, sizeof(event.componentID));

			//send events name
			clientSocket->writeBlock((char*) &eventNameLength, sizeof(eventNameLength));
			if( eventNameLength > 1 )
			{
				clientSocket->writeBlock(const_cast<char*>(eventName.data()), eventNameLength);
			}

			//check size before sending parameter
			clientSocket->writeBlock((char*) &parameterLength, sizeof(parameterLength) );
			if( parameterLength > 1 )
			{
				clientSocket->writeBlock(const_cast<char*>(event.parameter.data()), parameterLength);
			}

			//send value list
			clientSocket->writeBlock((char*)&valuesLength,sizeof(valuesLength));
			if( valuesLength > 1)
			{
				for(int i=0; i<valuesLength; i++)
				{
					clientSocket->writeBlock((char*)&event.values[i],sizeof(event.values[i]));
				}
			}
			clientSocket->flush();
		}
	}


the server accepts it like this:

Source code

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
	else if(identifier == "event")
	{
		ReadStateEvent current = READ_ELEMENTID;
		
		socket->readBlock((char*) &elementID, sizeof(elementID));
		socket->readBlock((char*) &eventID, sizeof(eventID));
		socket->readBlock((char*) &componentID, sizeof(componentID));

		socket->readBlock((char*) &eventNamelength, sizeof(eventNamelength));
		if(eventNamelength > 1)
			eventName =	convertStreamToString(socket, eventNamelength);

		qDebug("SERVER<%d>: Name is <%s>", m_Status, eventName.data());

		//check size before reading valuelist
		socket->readBlock((char*) &parameterlength, sizeof(parameterlength));
		if(parameterlength > 1)
			parameter =	convertStreamToString(socket, parameterlength);
		
		//check size before reading values
		socket->readBlock((char*) &valuesLength, valuesLength);		
		if(valuesLength > 1)
		{
			int value;
			for(int x=0;x<valuesLength;x++)
			{
				socket->readBlock((char*)&value,sizeof(value));
				values.push_front(value);
			}
		}
		
		qDebug("SERVER: Event received: comp<%d> elem<%d> event<%d> name<%s>", componentID, elementID, eventID, eventName.data());
		//Send event to MMIManager
		MMIEvent event( componentID, elementID, eventID, values, parameter, eventName);
		MMIManager::getMMIObject()->delegateEvent( event );
	}


I send an identifier to let the recipient know what is coming. The problem is, when a package arrives with more than one event, it crashes or reads only the first event. When the sender sends another package, the recipient reads the remaining events from the package before!

I tried to work with QSocket's bytesAvailable() and waitForMore(), but that ends up in an infinite loop.
Any hints, tricks, critic ?
Many thanks in advance.

Regards
Tom
---
There are only 10 types of people in the world:
Those who understand binary and those who don't.
---

  • "wysota" is male

Posts: 4,276

Location: Warsaw, POLAND

  • Send private message

6

Thursday, September 9th 2004, 10:51am

I don't know what kind of data you send through sockets, but it'd the best if you used \n as a separator. Then, you could read line by line and each line would be a separate object.

Another approach is to read byte by byte and check each character wheather it is the separator. If not, push it into a buffer and carry on. If it is, discard it and process the object in buffer or just put it somewhere. Then carry on reading. If there is no more data to read, process queued objects and wait for an event telling you there is more to do. If you don't have a complete object in buffer, just wait for an event that tells you there is data to read. You should be able to do the whole thing without waitForMore(). Just process the data as it comes. Remember, it can happen that an object never gets completed (for example the connection breaks) and you have to react to it. If an object is not completed for some time, you can guess that something went wrong and you have to react. You can't wait for infinity.

You could also use UDP instead of TCP and read the data datagram at a time. Remember though that UDP can fail, so you have to notify the other end that you received the data. If some data doesn't get acknowledged, send it again. This way you don't need separators of any kind, because each datagram denotes an object.

Tom

Trainee

  • "Tom" is male
  • "Tom" started this thread

Posts: 48

Location: Darmstadt, Germany

  • Send private message

7

Thursday, September 9th 2004, 3:04pm

hi wysota,

nice thoughts, thank you.
I am realizing it, using a memory for each packet, for not complete packets. When the next part of such a packet arrives, I simply start reading, where I stopped before.

The next step is to handle more than one client.

Pretty challenging.

Ciao
Tom
---
There are only 10 types of people in the world:
Those who understand binary and those who don't.
---

  • "wysota" is male

Posts: 4,276

Location: Warsaw, POLAND

  • Send private message

8

Friday, September 10th 2004, 12:10am

If you have any problems, feel free to ask.

Posts: 2,162

Location: Graz, Austria

Occupation: Student

  • Send private message

9

Friday, September 10th 2004, 12:17pm

If both sides of the connection use Qt you can make your life easier by using QDataStream

Then you could use some kind of layered transfer mechanism:
one layer that can transfer packets of binary data and one layer that assembles and disassbles such packets.

(code untested, just to give an idea)

Source code

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
class PacketLayer : public QObject
{
    Q_OBJECT
public:
    PacketLayer(QSocket* socket, QObject* parent=0, const char* name=0);
    ~PacketLayer();    

    void sendPacket(const QByteArray& data);

signals:
    void newPacket(const QByteArray& data);

private slots:
    void slotNewData();

private:
    QSocket* m_socket;
    QBuffer* m_buffer;
}


Source code

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
PacketLayet(QSocket* socket, QObject* parent, const char* name)
    : QObject(parent, name), m_socket(socket)
{
    m_buffer = new QBuffer();

    QObject::connect(socket, SIGNAL(readyRead()), this, SLOT(slotNewData()));
}

PacketLayer::~PacketLayer()
{
    delete m_buffer;
}

void PacketLayer::sendPacket(const QByteArray& data)
{
    Q_UINT32 len = data.size();
    QDataStream stream(m_socket);
    stream << len;
    stream.witeRawBytes(data.data(), data.size());
}

void PacketLayer::slotNewData()
{
    // read data from socket, write into buffer
    // when at least 4 bytes are available, read size
    // when size is known wait until size+4 bytes are in buffer
    // read size bytes from buffer into temp byte array, emit signal
}


Cheers,
_
Qt/KDE Developer
Debian User

Tom

Trainee

  • "Tom" is male
  • "Tom" started this thread

Posts: 48

Location: Darmstadt, Germany

  • Send private message

10

Friday, September 10th 2004, 7:02pm

@anda_skoa

thanks for your help.

I got it working and it works correct without any crash. I am working right now on the issue of handling more than one client. I am trying to realize that with the QSignalMapper. In addition I am solving the problem that a package doesn't have to come in in one whole piece. What I do here is to use some kind of state machine to save the last chunk read and to proceed exactly at the last position.

Somehow it is fun :)

Ciao
Tom
---
There are only 10 types of people in the world:
Those who understand binary and those who don't.
---

Posts: 2,162

Location: Graz, Austria

Occupation: Student

  • Send private message

11

Saturday, September 11th 2004, 4:52pm

Do you have different clients on different connections or is this some kind of multiplexing?

If the first, you could build a complete message assembly/disassembly stack on each socket and have the top layer emit signals which have some kind of connection identifier as an additional parameter (could be the socket file descriptor, or the socket pointer or some name)

Cheers,
_
Qt/KDE Developer
Debian User

Tom

Trainee

  • "Tom" is male
  • "Tom" started this thread

Posts: 48

Location: Darmstadt, Germany

  • Send private message

12

Saturday, September 11th 2004, 5:37pm

@anda_skoa

you are right. I have different clients on different connections. I started sending an identifier string and the ID of the client when sending an object. The problem here is that when a packet is not send complete, the second part of the packet will not have the ID, as it has the remaining data.
Therefore I have to memorize which client sent which packet and how much of the packet arrived, so I can start at the correct point, when the remaining data arrives.

Ciao
Tom
---
There are only 10 types of people in the world:
Those who understand binary and those who don't.
---

  • "wysota" is male

Posts: 4,276

Location: Warsaw, POLAND

  • Send private message

13

Saturday, September 11th 2004, 5:44pm

I'm not experienced with Qt sockets, but normally it is done in the following way:

The server listens to connections on one socket. When the client connects, another socket is spawned by the server and the client is being connected to that socket. This way each client has its own socket for communication and the server still maintains the listening socket waiting for other connections. This way each client is identified by the socket it is connected to. Of course it all works for connection-based transmissions (like TCP).

Posts: 2,162

Location: Graz, Austria

Occupation: Student

  • Send private message

14

Monday, September 13th 2004, 2:47am

Right, if you don't disconnect between packets, you don't need any per packet ID at all.
Once the partner is known, for example by sending an ID at the beginning of communication, it stays the same during the rest of the communication.

Cheers,
_
Qt/KDE Developer
Debian User