The submit deadline is midnight of due day.

When you submit your program code, please also submit a simple "readme.txt" file specifying the most important thing to compile and run your program. When I say the most important thing is something unusual. For example,

1. You are using "rmic" instead of "javac".

2. You setup a special environment variable.

3. Your program must be running from directories other than "classpath".

4. You assume your program are running in two machines with computer name like "Tom" and "Jerry".

...

So, the best way is still to write two simple "makefile" in "batch" file format. i.e. "compile.bat" which compiles your source code and "runclient.bat", "runserver.bat" which runs client and server respectively.  Of course this is not mandatory but a suggestion only. (Am I repeating myself? :) )

Yesterday is really an irony for this course because the second title of this class is "reliable distributed system design". However, my mailbox was bombarded by a dozen of assignment submission due to failure of EAS. Not only the file names varies from whatever you can imagine, but the file formats are also different. I really prefer a simple protocol for file name using your ID+A+ no of assignment.doc. i.e. 1234567A1.doc. I prefer doc to pdf because it makes me easier to add some comments.

Of course I cannot access EAS either. So for those who submit through electronic submission system, you don't have to send them by email to me because I really hope it can sustain longer before it explodes.

Index

Announcement:

May 25

I will give a simple tutorial on Thursday lab session on topic of synchronization based on the slides I made today. And I think it will both help you review the concept and give you tips on practical problem in both assignments and project. You are welcome to drop by at 5:30pm on Thursday at H-929.

May 25

If you find any difficulty compiling and running the example "shapelist", check here.

I add a simple explanation of what you are expected to submit.

May 26

I add more FAQ of assignment 1.

May 27

Do we assume different brokers are located in servers running in different hosts?

May 30

I posted a small example of how to read/write object into file. This would help those who are first time in using java.

And I would like to repeat one thing I mentioned in last Thursday tutorial. That is, the last thing I expect to see in your code is that you declare all your methods to be "synchronized" which is guarantee with "thread-safe" but defeat the purpose of concurrency.

As we are using lab H929 "concurrently" with comp471 in Tuesday session, I am not allowed to use projector in the lab. So, I will only give tutorial on lab session on Thursday(17:30--19:00). And during the lab time of Tuesday, I will try to talk on some general issue of assignment.

I also add a few missing slides in my previous tutorial about "synchronization". And I also noticed that it is not appropriate to give such a quantity of tutorial within one lab session. So, on Thursday lab, I will try to pick up some light topic like file operations or stream techniques.

A couple of interesting questions concerning programming assignment.

May 31

Probably the example of "shapelist" is a little bit too complicated to be a "hello world" example. So, check this.

Generally speaking, there exists no stupid questions. However there does exist difficult questions and simple questions. So, in case you have same doubts about some basic requirement, please just go ahead by sending me emails. Don't feel shy and I am paid to help you on that.

June 2

In this afternoon tutorial, I made a mistake by suggesting you to use "writeObject" for string. It will only work for "reading" or "writing" once. It will have trouble when you modify the record because "String" is a variable length of bytes. So, I made a new example here to demo how to modify a record in files. Of course there is no synchronization code at all. It is your duty to make them correct. Once again I have to point out the code example is only acting as an example. You are not limited by following the example. You can always implement your own idea.

Another reason for not using database in your assignment is that it will make marking difficult because setup a similar environment is really a painful job, if I want to make any test on your assignment. So, I do suggest you guys write your own simple file database by yourself. After all, it is really a good experience for your programming life.　

It seems that nobody is asking such a common question: Can two RMI server run in same host with same port number?

June 3

I just wonder how many people have ever started programming because this is a really practical question. Should we use average price for each stock if the customer have made more than one purchase at different price?

And in case someone might run into similar situation I have run into before.　

Can you guys do things like this for me?

June 4

Phew... I was confusing myself for a whole morning for a simple example. However, there does exist some lesson you might be interested except they are almost useless for this assignment. So, don't waste your time on it unless you finished your programming part. Briefly the question is how to make return value a self-defined class. If you happen to allow your class inherit or extend from "RemoteObject", you are dead like me.  

How about RemoteException? Does it cost when you take advantage to use it as return value?　

June 5

Who can solve the puzzle of "Serializable"?

Has anybody asked the question: Is methods of "vector" thread-safe?

June 6

I am hesitating what to talk in tomorrow's tutorial. And I am thinking about giving some analysis about assignment 2. Of course if somebody is interested in any topic, please let me know by email.

June 8

Here I list what I know to be thread safe containers: Vector, Hashtable. And the unsafe class are ArrayList, TreeSet.

To check by yourself, download and try.

June 10

I repeat some discussion in last tutorial about assignment 2. And I add another similar test for HashMap which is also thread safe. (Actually this is boring and any one can do the testing by changing the declaration and test if it runs properly.)

I will try to mark assignment 1 asap so that you can get some feedback when doing assignment 2. Otherwise what is the meaning of learning? However, believe me, it is really a time-consuming job. Suppose I give half hour for each report and program, I will have to work 25 hours without sleeping, eating etc. And half hour for you guys hours and days of working is the minimum for reading source code and report. So, maybe I will simply post some common mistakes or bad designs for you to see before I can finish all markings.

Here I list a few mistakes in your assignments. And I promise to add more.

June 11

I accidentally discovered an old example which I forgot to post. It is about how to write fixed record and it should have been posted long time ago.

Before this morning, I have a wrong concept about java "container" classes. That is from pure C++ STL perspective view, everything you get from vector, map, list is purely "copy" of data. So, when you are manipulating the data you get from vector, you are actually operating on separate copy. Finally when you are going to write it back to STL container, you maybe overwrite it. However, in java everything is as simple as reference which means you really get the reference of element in container. So, no matter how many threads are retrieving the same element from vector, you are manipulating the same object. This exposes a big synchronization problem. Here is my simple test to see if I am operating on copy or reference of element of both Vector and Hashtable. In order to test Hashtable, you may need to modify the code a little bit from the comment.

So, in Tuesday tutorial, I think the best thing is to explain what kind of mistake I discovered in your assignment. I really hope more people can show up since the mistakes are quite common. And if you don't correct them you will have more trouble in assignment 2 , 3 project. And I will promise you that I will deduct the same points from your future submission on these kind of mistakes, or inefficiencies.

Good Lord! HashMap is not thread-safe! Thank you so much for Mr. L. who points out my mistake in claiming HashMap is thread-safe. First Mr. L told me a way to see whether it is thread-safe or not. It is simple. Check the source code of java in  "jdk1.5/src.zip/Java/util/HashMap.java". And you can see the comments which is much more detailed than "sun.com". See the function are not declared as "synchronized". While if you check "Hashtable.java" in source code, you will find they are all synchronized!

Secondly, I modified my little testing code by increasing the number of testing threads to 100 and 100% I got an exception in accessing those corrupted data. This is a big lesson to learn today. Don't jump to conclusion too fast and you may run your program very well when you are using a small number of threads. The real problem will severely punish you if your program is not correctly synchronized! (I am answering the question of some students: why do I keep mentioning synchronization from time to time. If you check "google", this small website has already been placed on some rank about this repetition of "synchronized". It is as simple as this: This is not something exaggerating or bullshitting. It is the basic, basic requirement to make things right.)

And I have to modify my comments in "commonMistakes" again today! phewwwwwwwwwwwwwww......

When "synchronized" key word describes a method of a class, it means for each "instance" of such class there is only one thread which can execute any "synchronized" method. In other words, it is "instance-level" not "class-level". For example, if my class MyClass has two synchronized methods funcA() and funcB() and I create only one instance 'A' of MyClass.  I create two threads to call A.funcA() and A.funcB(), then only one thread can successfully be executed at any moment. But if I create two instances A,B of  MyClass, then the two threads call A.funcA and B.funcA() respectively. Both threads will be able to be executed. So, in one word, the 'token' for synchronization is for each instance. See example here.

The example of socket programming in web site is a bit too complicated. You can certainly use a single-threaded TCP Server to handle your tradeStock function like this.

More questions about "tradeStock".

June 12

How to run "Hello World" from multiple server?

What if I have different interpretation of requirement of assignment?

June 13

Let me repeat it again here. HashMap is NOT thread-safe class. And how do you deal with thread-safe class and non-thread-safe class?

Is this inconsistency of CORBA or RMI?

June 15

People are keeping asking the same question: what kind of testing case am I talking about? So, I will give a test case which I used in my assignment last term. Of course this is only a suggestion. However, if your program can run correctly with this test case, you would be probably pretty much confident about the synchronization part of your program.

June 18

I talk some general problems in openAccount, login, and logout. And I personally think "logout" is the most difficult one.

June 20

I finally finished marking of assignment 1 and please go here to download. Actually I also marked some comments on some theory assignments. However, later I feel so tired to find and change the file name of marked version. I give up uploading. This is really a boring and tedious job for me to place each file into particular folder and change its name then uploading.

What I have done in programming part is to try to read source code of every one and find out possible problem. I write down briefly whatever I found in your report and hope you can modify in your second assignment. Of course, it is very hard for me to accurately understand all people's source code and there might be some mis-understandings. However, believe me it is a painful job for me to work for dozens of hours, quite a few sleepless nights. I do hope you guys take a look at those comments and don't make repeated mistakes in A2.

As for the mark of A1, I roughly feel like this.

For those roughly above 50, you are generally speaking quite good in synchronization.

For those roughly 30-50, your implementation may have some big problems, say without following requirement.

For those below 30, you might miss a document.

For those with negative points, you might submit a wrong program....

For those higher than 60, you might be creating some arts.

Here is some more about what I found in submissions.

An interesting question and I hope everybody can give it a thought.

June 22

Somebody says the Assignment 2 is comparatively easy than Assignment 1 because we only need to implement one extra function "tradeStock". It is true if your assignment 1 is done very well. However, there is a complicated situation of deadlock problem in "tradeStock". How does it happen? And how can you solve it?

June 25

More and more I like to write some pseudo code because it is easy and quick. Here I add a little more discussion about dead lock in assignment 2. However, I should recommend for those who did not do very well in assignment one it is better for you to correct mistakes of assignment 1 and only concentrate in a basic solution of "tradestock". It is much more complicated issue of dead lock than mutual exclusion.

June 26

Most of you guys might be fed up with my repeating talk with correct synchronizations. However, what can I say about this? I mean I emphasis this with my own experience in last term when we are supposed to implement a huge fault-tolerance project. Without 90% or 100% sure of your synchronization implementation, your fault-tolerance project would be definitely like a chaos. When you debug the tricky fail-safe server, how do you know it is due to your server recovery part or your basic synchronization implementation? Or in other words, it seems to really a "luxury" to talk about fault-tolerance when you cannot even make it basically correct!

I want to add a few comments about OpenAccount. 

And seldom do people pay attention to efficiency issue of OpenAccount.

July 5, 2006

The web service assignment is a kind of "99% efforts of setting up environment, 1% of efforts of coding". And what makes things worse or better is that there are multiple solutions for the same problem, depending on what tools kit you are using. So, try to choose an application server like Tomcat, JBose etc and install them by following the setup documents. In case of Tomcat, read "setupclasspath.bat" to see what environment variable you need. And if you are using Eclips or some other tool kit maybe this job has been done by the IDE by installing some plug-in.

The only synchronization issue about web service is here.

July 7, 2006

The above comment on web service is based on my experience of last term. However, as I suspected there are countless ways to work around the "multiple servlets" issue. Thanks for advice from Mr. Fu who showed me the way to configure "serverconfig.xml" file by adding a "scope" property of "application". Then the Tomcat will NOT generate new servlets for each request. And our synchronization issue here disappears which means we don't have modify a single line of code from assignment 2. And another possible way is like this. In "HttpServlet" declare our implementation object as static data member, say "Broker", so that even though we generate multiple "HttpServlets" still our implementation object is still single one. i.e.

class MyHttpServlet extends HttpServlet implements BrokerInterface

{  private static Broker broker= new Broker();

    public String buyStock(String custID, String stockID, int qty)

    {    return broker.buyStock(custID, stockID, qty); }

    ...

}

Of course this method is purely my imagination and I hope some one can make it try to see if it eliminates the extra synchronization issue arising from "multiple servlets".

July 10, 2006

Probably it is too late to announce this. What I hope is that in the report you can make some discussion about the synchronization difference between web service and CORBA and RMI. For example, if your implementation of web service results in multiple servlets then surely the synchronization mechanism will be a little bit different. It is impossible for me to know how you configure your system. So, simply state in your report that if your system will create one servlet for each request or not. Then if yes, how you will deal with it. i.e. what part of code you make any adjustment from assignment 2. Of course it will make me easier for marking so that I don't have to look into source code. (Basically I would rather read report only. So, if you don't like me to read your source code, spend some time to explain more in your report.)

July 11, 2006

The lab time is changed! The session on Tuesday is combined with the session on Thursday. And the new combined lab is:

Thursday 18:20---21:10  H929

I was thinking to start to give some tutorial on project since there is not too much time to start it. On Thursday I plan to talk about something about reliable communication and its implementation.

July 20, 2006

Particularly I like what inspector Grim says in <the thin blue line>, "Time waits for no man and in CID crime waits for no man. We simply don't do anything funny about." So, let me be quick and short in those simple tips of project before you fall asleep.

July 24, 2006

I highly suspect that every student already knows that we need to modify the interface of all methods in project. That is, you need to add an extra parameter for all function calls in all interface except for "getquote". i.e. buyStock(String requestID, String customerID, String stockSym, int qty). Without this request ID, our fault tolerance cannot solve repeated requests from cases of either client or remote server. i.e. A client sends a "buyStock" request and finds out server is down. However, the server probably already finished its operation before crash. When client sends in its request again, without using this unique request id the server cannot realize that the same request has already been executed before crash. Similar thing can happen in "tradeStock" when remote broker failed before replying. I think this modified requirement in our project is very important and hope everybody realizes it.

Here comes a little comment and reply about question 14.

July 25, 2006

I cannot finish marking A2 in time. So, just like A1 some tips and hints are posted in "common mistakes" under title "assignment 2".

Here comes further comments.

Here I add more comments about marking of assignment 2 and I expect to finish marking within these two days. Sorry for delay. However, I hope you guys understand that I am unable to do the marking for full time.

July 28, 2006

The essential of project is the design of checkpoint and recovery log mechanism because all operations are basically executing in memory which will have no effect on persistent data. i.e. After any power failure the states in memory will be gone and these states in fact haven't been updated to database EXCEPT ONE OPERATION "logout". After discussing with one student, I think I have a clearer picture about checkpoint now.

1) Fault-tolerant mechanism should not alter our original business logic. That is, only "logout" customer updates its data back to database. So, the checkpoint is simply an image file of memory at certain point independent from our database.

2) At restart, system will load image file or checkpoint data into memory. Then by redo operation messages stored in log file step by step, we can restore crashed system back to the last state before crashing.

3) Log file not only record message about requests from client, but also need to specify the point when check pointing is finished.

And there is one more thing I forget to mention. The business logic of "tradeStock" is NOT equal to "buyStock + sellStock"! That is to say, "tradeStock" doesn't touch the balance and it is purely an "exchange" of stock between two customers' account.

Aug 3,  2006

Finally there is a chance for all of you to say about myself. Go to the TA evaluation site to give your vote.

https://eas.encs.concordia.ca/eas/ta-eval-authentication.jsp

Aug 4,  2006

It takes me almost one hour to search my old notes and finally I found the one which puzzles me for quite some time. It was solved by my groupmate last term. The question is how to start a new java process by "exec". See below.

Runtime.getRuntime().exec("cmd /c start cmd /c java Test"); //Test is your class to run

1. It is very important that you need two "cmd" because it is like the "shell" in Linux. Without a command interpreter or "shell", your command like "start" or "java" can never be executed.

2. It is also important that you have "/c" which is like a "return" key. Without this "return", you cannot "execute" the command.

3. Without "start", you will only create a java "thread" within same "jvm".

Aug 12,  2006

Please download the "demo sheet" and fill in the form by group leader and mail back to me at qingz_hu at cse.concordia.ca.

Also I noticed that some groups are still not very clear about the usage of "request ID". Basically it is like this:

1. It is an extra parameter in all your methods of interface. i.e. BuyStock(requestID, custNo, stockSym, qty). This implies that you need to modify your IDL file.

2. It is useful for fault tolerance project when client program notices failure of server or some broker server notices failure of remote broker server during "tradeStock" operation. Then both client program and the broker server needs to "lock" itself by repleating sending the same request until the failed server recovers. Otherwise there may be inconsistency between states of client-server or server-server. i.e. Server finishes an operation and fails before reply to client or a server finishes part of "tradestock" and fails before reply to remote server. Without checking request ID in log, definitely your server will make repeated operations like executing "buystock" twice with same client or trading one bunch of stock with other bunch of stock twice.

3. The client program is responsible to generate the global unique request ID. One simple solution is using the following combination of IP+ port number + time stamp + other parameter (optional) as seed of Random class. Or you can use other global synchronization method to guarantee no conflicting.

Another thing I noticed is that some groups are still not grasping the exact core parts of "failure-recovery". You see, the current project requires us to do all operations in memory except "logout" operation which writes memory contents back to database. And in fault-tolerance it only matters when you write data back to disk because all memory contents will be gone at power failure. So, the only critical period is when you write information back to disk. This can happen when you write down some "log" message or you write "memory image" back to disk or you write back to database. Therefore what you need to do is to try to guarantee safe recovery when failure happens at these periods.

1. Professor explicitly designs to force you to avoid making check point after every operation which is very slow in performance. i.e. Last term we are free to write result back to database after every operation like "buy, sell" etc. Now you must make your own "check pointing" timed at your own choice. i.e. After a certain number of operations you decide to store all memory back to an intermediate file in disk or when a certain kind of operation happens like "logout" operation you try to synchronize memory with database.

2. The procedure of writing back to database or disk must be done in a couscous manner. i.e. For each writing you need to make a "log" entry in your log file so that it is recoverable when your system recovers from power failure.

3. For remote transaction, you must be very careful to make sure consistency between two parties of transaction. For example, you may need to consider the extreme case when one or both two parties fails during "tradestock". As I mentioned above you will find the usage of "request ID".

All in one sentence, if you haven't discovered how request ID is going to play an important role in this project then it simply you haven't even STARTED with your project which is already a bit too late.

About the report, I will recommend you guys to place absolutely enough time and energy into it. Otherwise your project maybe only achieve not more than one third of total. And a general suggested contents includes at least following:

1. Structure of design.

2. Analysis of hardcore of fault tolerance. (one of most important thing!)

3. How does system work and how do you prove it is fault-proof fault-tolerant system. (scenarios, timelines, step-by-step etc.)

4. Limitations and assumptions. (This maybe one of the first thing you need to write in report at the beginning. )

5. Implementation details. (It depends. You maybe willing to share some shining point with others. Or you may discuss some interesting algorithm in details.)

6. Testing cases design. (Fault-tolerance are even harder to be tested than those deeply-hidden synchronization or deadlock bugs! A carefully designed testing cases usually shows how well you understand the problem. i.e. If you don't know what is the possible problem, seldom can you design a testing case to test it. )

7. Your choices to add more.

Aug 13,  2006

The marking of assignment 3 is finished and please check your mark here.

1. Theory part are all done very well with probable help of internet search engine, I believe.

2. Programming part are also done well because at this stage the survivors are basically experts in programming, more or less.

3. I believe some students' submission are missed because of submitting under different category in EAS. i.e. submitting assignment 3 under project 3 etc. Please send me email if you think you have submitted your assignment 3.

4. Again I setup a maximum limit for assignment 3 which is 65 if you don't tell me what is the difference between A2 and A3 in perspective of synchronization because in web service if you don't configure properly your servlet container will generate a new servlet for each incoming request. And if you don't let me know how you handle this, the maxim mark is 65.  Basically you have at least three options to deal with this issue. a) Make your implementation class a static member of its enclosing servlet; b) Carefully design those global resources to make them static so that they will be only initialized once in servlet constuctor and will be synchronized properly in assessing; c) Configure properly that web service is running in so-called "application" mode instead of "request-mode" such that only one single servlet instance will be generated.

The reason I give this issue 5 points is that it is simply an issue of correctness and if you don't mention this in report there is no way for me to figure out.

5. Some students may find their marks lower than 60 which indicates some big flaws in reports. i.e. missing claimed snapshots, missing necessary analysis or explanation. Compared with those reports whose authors have been invested enormous time and energy, the marks I deducted this time are comparatively small.

6. There is no way for me to help if you simply don't submit either report or source code which will be 0.

7. Please report any disagreement within this week. Otherwise it is hard for me to send modification to professor again and again. Hope you understand.

Another thing about project is again related with request ID. I am quite reluctant to post the following because I think it is a bit too late or too complicated for most groups which was originally mentioned by one group. However, it should be helpful for those who are willing to face challenge.

... And I think roughly the client
program simply keep re-sending its request when a failure is noticed. By using the
"request ID", the server will be able to avoid repetition when recovered from last
failure. However, some group do propose a hard question for this issue when client
re-sending request. That is, there is no way to guarantee all clients re-sending their
requests at exact sequence immediately before server fails. i.e. Just before server
fails, client A sends a "tradeStock" request which can be fulfilled when customer B has
enough stock for trading. What if the server fails before this "trade" request  return to
client A (the remote server finishes this trade request already. ) and another client C
comes before client A when server recovers. This client C is a "sell" request which will
reduce customer's stock to 0 so that when client A comes in the recovered server cannot
fulfill the unfinished transaction while the remote server already finished. This issue
is also possible to solve if you properly design the "log" file entry so that a
"transaction begins" plus "transaction commits" are both recorded in log so that when
recovering from failure the server will search for those unfinished transactions and make
sure they have priority to finish first.
I am not sure if any other group have this consideration or not because it seems to me a
bit too advanced for most groups.
　

The following issue is proposed by one group and I think it is fair to share among all groups.

Another small issue is about timing of synchronization between in-memory data and database. As we have discussed before that all in-memory data must be made an image file in disk from time to time so that this image file acts as a "checkpoint" for "failure-recovery" plus an independent "log-file" which records all operations like "buy, sell, trade" etc. And by our logic, only "logout" operation is supposed to directly update database from this image file. Therefore whenever there is an "logout" operation, we need to first write in-memory logged-out customer data back to image file and then begin to update database immediately. This is necessary because it is possible that a "login" may follow immediately after a "logout" and if we don't update database immediately the "login" will load inconsistent data into memory.

Aug 14,  2006

I just read your demo sheet, and I have some thing comfused. first, I don't know
what the flexibility and methods mean. Second, what's the different with "batch run"
and "complex operation".thanks for your reply.
　

The complexity and methods of deployment is a rough standard for my own marking usage.
However, still I would like to give some explanation here.
1. The deployment of distributed system is one of the most important part. Basically in
our project, you need to write down the following information at least(in CORBA):
a) The broker list of your system. i.e. how many brokers are there in my system?
b) Within each broker, the host list, or processes list or server list (replica in
project D).
c) The port number of each server. Here it is possible that the data communication and
monitor are using different port for each server so that you can run multiple server in
same host.
d) Database file path is also possible if you want to run your server in same host with
not current working directory.

2. The deployment methods may vary from system to system. The most common one is write
all information into a configure file. However, there is big difference if you don't
hardcode the host name as a fixed list.
So, the other good combination is to take advantage of ORBD which is a global database
and combine with configure file so that you can "dynamically" configure your system. i.e.
At least the server name list will be dynamically generated which makes deployment much
flexible. However, it is quite complex because you need to identify each server when it
runs. i.e. by passing parameter at command line, it will tell the server how to find its
broker ID.
So, the third important part is "batch" file which may be another method for deployment.
I mean a simple solution is to write a series of complex .bat file so that by running the
.bat file you can run your server with different port number, broker id etc.

3. Again "complex operation" is referring to "trade stock" and "batch run" means your
testing is running with large number of operations. And I think these are just basic part
in your demo.