It seems that people are impatient in reading or listening. So, I just make it short and quick.

1. How long do you expect to implement my project?

I calculate in tutorial like this. You have total less than one month from deadline. However, you have to deduct at least one week for final. Another week of debugging and documentation also need to be removed. Before starting coding you may need one week to think it over, discuss with group member. So, maybe you can only allow yourself for less one week for coding. Am I exaggerating?

2. What is a monitor?

In simple sense, it is an echo server which responses to whatever request it receives.

3. How to implement a reliable communication channel?

As I said in tutorial, reliable communication channel is an "exactly once plus FIFO". Or in plain English, you need to deliver a packet eventually only once and keep the delivery sequence of all packets exactly what they are sent.

4. How to implement exactly once?

To prove x==1, you only need to prove x<=1 AND x>=1 because the only intersection of two data set is exactly x=1. So, in terms of network, you need to implement both at least once "and" at most once.

5. How to implement at least once?

Simply keep sending until you get acknowledgement.

6. How to implement at most once?

By using id for each packet so that you can recognize the repeated packet.

7. How to implement FIFO?

Deliver packet in the same sequence that they are sent by acquiring the "sequence". In simple English, just use natural number as a sequence of sending and receiving.

8. Do I need to maintain connections?

Yes, because your server need to listen multiple clients at the same time which implies your server must maintain multiple sequence for each connection.

9. What is essentially a connection? Or how to identify it?

Each connection is simply a combination of IP plus port number for each client socket which is unique in network.

10. Do I need an extra thread for server socket?

Probably yes because your failure detection is relying on the response time of your server socket. If your server socket is running in single threaded mode your monitor maybe repeatedly report fake failure which is quite annoying.

11. Is there any other way than time out for failure detection?

No. This is the only way.

12. What if the failure detection is wrong?

Then make it true by killing the assumed non-response server so that it won't interfere your system by "split-brain" or "ghost server" whatever.

13. What if the monitor receives some strange reply from detected server? For example, monitor expects the server to reply "I am alive.". However, the server answers "what a good day"?

This is a kind of question whether we are going to handle "benign failure" or not. Just imagine the famous puzzle by assuming one person is always telling the truth. And you ask his such a contradictive question: "suppose you are dead and what will you reply when I ask you if you are alive."

If the person is dead, he cannot even reply. If he replies, it implies that he is not dead. Then how can he say he is not alive when you assume he is telling the truth. So, this failure is somewhat a Byzantine failure which we cannot handle.

14. Do I need to implement an internal buffer for server socket?

Probably yes because your socket only "delivers" packet when its user program calls its "deliver" function. So, without buffering your received packet, you are continually losing packet which will be quite difficult to handle. i.e. resending rate will be too high.

here comes one comment:

I don't agree with the answer about the following question:
Because almost all TCP/IP stack including embedded system has realized internal receiving buffer for socket interface. For example, Linux with full TCP/IP stack supports enough big buffer to store packets temporarily for sending and receiving and has some algorithms to prevent buffer overflow from Denial of Service attack. So, if you implement an internal buffer in application level, just adds buffer latency and decrease system performance. Of course, recently some open projects want to transform TCP/IP from kernel level to application level in order to avoid overhead of system call to improve performance of the whole system. Under the condition, a application will realize an internal buffer and fetch data packets directly from network interface so that the application can handle them whatever it wants for example just raw or with TCP/IP stack. The approach will make some network applications very happy such as firewall.
　

here goes my explanation:

You are quite right about implementation of TCP/IP. However, just remember now you are
supposed to implement a TCP-like socket based on UDP socket which means that you are
mimic the behaviour of TCP socket. I agree with you that you can take advantage of
internal UDP buffer so that you simply retrieve data when you "accept" the incoming call.
However, what I really mean is that the "sorted" packets need to be presented to user in
a kind of waiting queue. i.e. you only "deliver" those non-repeated packets in order of
its sequence id and awaiting your user program to "retrieve" them by calling some
function you expose to user, say "retrieve". Then this buffer is a kind of application
level queue for delivering.
　

here comes the further comments:

Honestly,I don't understand why you need implement a TCP-like socket based on UPD socket. Actually,in the project, we only need add "sequence number" and "session id" for every request and response packet,and design a state machine to handle some events such as out of order,time out etc. "session id" should be UNIQUE in the real world. So,it will avoid any additional buffers involved in the project. The method can be seen widely in the VOIP field.
　

Here goes my further explanations:

I sent the reply and it is lost due to expiry of time which is really annoying. So, I am not sure if I am patient to re-write it again.
1. When I say TCP-like socket, I mean FIFO, deliver exactly once which is a part of requirement of project.
2. When you say "session id", I am very interested in how you are going to acquire it. To me, it is more like a high-level concept. However, I am not familiar with java. Suppose you are given a UDP socket in java, can you acquire some kind of "session id". I am awaiting for your solution.
3. When I mentioned internal buffer, I mean the buffer for those "ready-delivered" packets. Assume you have implemented your socket and you will let your user to retrieve the packet by calling a method "deliver". See below is one possible implementation:
class MyPacket
{
public int custID;
public float balance;
public char stockSym[10];
...
} //this is just a struct for sending and receiving.

class MySocket
{
public MyPacket deliver();
private MyPacket internalBuffer[MaxBufferLength];
private MyThread myThread=new MyThread();
...
}

See above the class "MySocket" need to deliver one packet when its method "deliver" is called. And at the same time, the internal thread is keeping running receiving packet or asking resending of out of order packets. Therefore for all those "sorted" packets, or in order packets, you need to store them before user call "deliver" next time. Since you cannot predict when user call this "deliver" method, you have to buffer the "ready-to-deliver" packets. Otherwise your socket has to either overwrite current packet or reject furthre new packet. Think about it and you realize that what I am talking about are all common simple issues which may not be significent for experienced programmers. However, still I think it is worthwhile to mention here.

I am open for arguing.

regards,

nick
　

15. Do I make my client socket also as server socket?

It definitely helps because you see in java you can do the two-way communication by retrieving the "inputstream" and "outputstream". Just imagine if you don't make your socket both maintain an outgoing sequence of packet and incoming sequence of packets, how can you achieve this? However, it is not easy and it is not mandate to implement such a complex socket.

16. What is the format of log entry?

This is one of the biggest question you have to answer in this project. Before that you have to answer a series of questions like what is essentially my check point? What is a transaction? What is my recovery scheme, redo or undo?

17. Do I have to use two-phase-commit protocol in our "tradeStock" operation?

Indeed when we implement this function in assignment 2 we assume there is no failure. Now we are assume there are failures anywhere do we have to modify our implementation? Probably not because we are using fault-tolerance mechanism to deal with failure so that we don't modify specific function in our previous implementation. Instead the problem will be taken care universally. By the way, what I understand is that our current implementation implicitly uses "2pc" by waiting for result of remote server after we protect the local data.

18. Do we have to introduce roll-back scheme?

Definitely we would not like to do that because it only makes things extremely complicated.

19. Why do we need to introduce the request ID for each operation like "buyStock, tradeStock"?

Because the server will use this request ID to check if the client's request has been finished or not when failure happened at the last transaction.

20. Is the request ID necessary to be globally unique?

Yes because "tradeStock" can happen between different broker server. So, this becomes a question how you can aquire a globally unique ID. i.e. among multiple client program running in multiple hosts.

21. Why don't we just let server assign request ID for each client request?

First it doesn't solve problem because it now becomes a globally unique id among multiple servers running in multiple hosts. Secondly it is not convenient. For example, it may need client program to explicitly call server first to get an ID before each operation because it is possible that server may fail even before return result to client.

22. What is recovery scheme?

It is like optimistic and pessimistic theme in which one always does things assuming there is a failure so that he writes down log before he operates and another one always assumes failure is such a rare case that he simply does operation as if there is no failure. However, one makes performance bad when recovery becomes easier. And the other one makes performance happy while recovery becomes a painful job. It depends on your design choice to view if recovery is a frequent job or not.

23. What is essentially a check point?

As our active data are all locating in memory and each operation is only modifying data structure in memory, we don't have the luxury to synchronize in-memory data structure with database. So, you may need to set up a point such that you are sure in-memory data structure is already consistent with database. It sounds trivial. However, what is the contents of check point? It may include a point in your log file plus some image file of in-memory data structure. When I say "log" file, I mean it is such a kind of file which records all operation details requested by client in the exact sequence it happens. By assuming "piece-wise-deterministic", it is possible to re-construct in-memory data structure from scratch. However, it is really time-consuming job and that is why we need a check point such that we can re-construct on basis of those image file of in-memory data structure while those log entries are simply incremental changes.

24. Since we have a group manager, why do we still need a leader in replication group?

It is possible that group leader can act on behalf whole group. For example, in trade stock operation it is possible that only the group leader issues trade stock request and later broadcast the result to other member of group. Another example is happening when synchronization is needed for new member of group. Only the leader sends its data to new member of group without every member doing the same thing.

25. Can I find an API in windows to start a remote process in other host?

As far as I know, it is impossible to start a remote process with windows API. The only way is by an agent or running service in that host. So, you may assume this service is a kind of reliable or part of system. i.e. it is started by operating system when it restarts from power failure.

26. Can I assume monitor reliable?

Yes or no. In fact, nothing in this world is really reliable. The so-called reliable thing is only made by us. In one sense your system must run on some kind of reliable foundation like reliable OS, CORBA, nameservice, etc. Otherwise nothing makes senses. Therefore you may be able to assume monitor is part of our middleware which is reliable. However, viewing from another perspective your monitor is simply part of your application which by no means reliable at all. Then what? Maybe you can borrow idea of the  implementation of group manager which is using multiple replica to assure its reliability. So, by using a group of monitors which monitors each other you can achieve a reliable monitoring system. And of course this is not mandatory. However, my team-mate implemented such a system in our project.

27. What if my local server detects failure of remote server during "tradestock" operation?

You simply keep sending the same request until remote server recovers. So does client program when it doesn't receive reply from server. The big idea is that our system is a "lock-step" system such that all members (primary and backup) are acting in a "synchronized" manner.

28. Does my reliable communication channel implementation enough for the so-called "atomic FIFO broadcast"?

No because what you have implemented is only a reliable FIFO for single connection and it is not "atomic" for broadcast. i.e. It is possible that your sender will fail during its broadcast which will create inconsistency between receivers or replicas. Here you may need to implement a 2pc to guarantee "atomic" broadcast.

29. How to elect a new leader?

The simplest scheme is to choose the oldest replica or the one with smallest index. But the problem is how to know which one has the smallest index. Maybe a simple solution is to leave group manager to decide after all replica report its own index to it.

30. Then when to elect a new leader?

The question maybe able to rephrase as who is going to start election operation. Obvious solution is the one who detects failure of leader. So, monitor is one candidate. However, group manager is also a possible choice, depending on how your system is monitoring the group member.

31. How tough do you expect the project could be?

In my personal perspective, it is the most toughest project I have ever experienced with in Concordia.

32. What is one of the biggest assumption in primary-backup system?

There is no double-failure or our system cannot tolerate simultaneous failure of both primary and backup. Similarly I expect this kind of assumption or limitation of your system appears in your report before any systematic analysis begins.

33. What is checkpoint?

The essential of project is the design of checkpoint and recovery log mechanism because all operations are basically executing in memory which will have no effect on persistent data. i.e. After any power failure the states in memory will be gone and these states in fact haven't been updated to database EXCEPT ONE OPERATION "logout". After discussing with one student, I think I have a clearer picture about checkpoint now.

1) Fault-tolerant mechanism should not alter our original business logic. That is, only "logout" customer updates its data back to database. So, the checkpoint is simply an image file of memory at certain point independent from our database.

2) At restart, system will load image file or checkpoint data into memory. Then by redo operation messages stored in log file step by step, we can restore crashed system back to the last state before crashing.

3) Log file not only record message about requests from client, but also need to specify the point when check pointing is finished.