I have done a lot more testing and investigation and I cannot find any identifiable problems.
Symptoms are as follows:
1) When starting from dead cold (overnight), if the engine doesn't start on the first crank, then it is very hard to start. While cranking, there are frequent 'backfires' that cause the engine to stop dead. It is as though the ignition fires at the wrong time in the cycle and causes the spinning engine to freeze at that point.
2) Once started from dead cold (which often happens on the first crank, despite the above), the engine idles very smoothly for the first 60 seconds or so. After that, the engine starts to miss while idling, which gets worse as the engine warms up.
3) Once at normal operating temperature, the engine misses badly while idling. It idles smoothly for a second or two, then misses, which causes the revs to drop suddenly, and then the control system seems to increase the idle speed to counteract the drop in revs for a second, and then the cyle repeats. Turning on the air conditioning makes this cycle much worse. That is, the miss is worse with the air conditioning engaged, and will frequently stall the engine. Listening at the exhaust pipe, you can hear a 'phut, phut, phut' noise when the engine is missing.
4) After idling at intersections and the like, there is a very bad hesitation or stumble when accelerating away. The engine will nearly stall for 2 to 4 seconds, and then take off. Occasionally it does stall. This bad hesitation also happens when driving along at normal road speeds if the foot is taken off the accelerator for a few moments (throttle fully closed) and then accelerate again.
5) While driving at normal road speeds (after the hesitation) or accelerating and the throttle is not fully closed, there is no miss whatsoever, and the engine performs very smoothly. While ever the throttle is partially or full open, the engine performs very smoothly and with normal power, apart from the initial hesitation.
6) While stationary with the air conditioning off, if the accelerator is depressed a tiny amount to increase the revs to about 1000 rpm (ie 200-300 rpm above normal idle) the engine will run very smoothly with no miss. This may be because the throttle position sensor is signalling just off idle.
7) If the throttle position sensor is disconnected the engine will idle very smoothly with no miss. It will also run at slightly higher revs, but if the idel speed is adjusted down to about 700rpm again it still runs perfectly smoothly with no miss. I discovered that one of the effects of disconnecting the TPS is that the spark advance is increased significantly by the ECU. It looks as though it goes from the normal 12 degrees up to about 20 to 25 degrees BTDC. I don't know what other compensating action is taken by the ECU when it detects a fault with the TPS after it is disconnected.
Replaced plugs, spark plug leads, distributor cap and rotor, fitted brand new distributor. Checked for any vacuum leaks. Removed throttle body and thoroughly cleaned and checked it. Checked many other sensors for correct operation and voltages (where possible) at the ECU. This include the water temperature sensor, air flow sensor, air temperature sensor, throttle position sensor.
Checked that the spark plugs are connected to the distributor in the right order, checked that the TDC mark on the crank is actually top dead centre (measured piston position through the spark plug hole), and TDC is aligned in the distributor with spark plug for cylinder number 1.
Checked that the TPS voltages are correct at the ECU, just in case any wires broken between the ECU and the TPS - everything as expected. Thanks for that suggestion.
Checked that the exhaust gas recirculation valve is operating by disconnecting the normal vacuum hose and applying vacuum while idling. This caused the engine to idle worse, although only a bit worse than it already was. The change was noticeable, however. Again, thanks for that suggestion.
Fitted a different ECU, with no change to symptoms at all. Fuel pressure was checked by the mechanic.
I was worried that the problem may be caused by low or uneven cylinder compression. However because the engine will idle perfectly smoothly in some cases (e.g. TPS disconnected) this would not be the case if the problem was low compression. Similarly if the problem was broken leads or damaged spark plugs, again doing simple things like very slightly opening the throttle would not cause the symptoms to disappear. Also, because the engine runs so well (smooth and with plenty of power) right up through the rev range, even under heavy load, it would seem unlikely that the problem is simple spark plug miss-firing.
Some ideas about what could possibly display such symptoms:
1) Igniter partially failing. I can not test the igniter, and I have not tried a replacement. While it obviously still works to some degree, I wondered whether it is sometimes firing the coil at the wrong time. Because it is partially controlled by the ECU, sensing changes such as the TPS off idle could potentially affect the operation of the igniter. Firing the coil at the wrong time would explain the bad backfire while cranking.
2) Cold start injector. If the cold start injector continued to operate instead of shutting off, then too much fuel would be injected while idling (and throughout the rev range), but the effects would probably be worse when there is very little air flow such as when at idle. This would also explain why while driving at normal speed, a period of time coasting with the throttle closed could build up the fuel mixture in the manifold, which causes the hesitation when the throttle is opened again. It could also explain why if the engine does not start first time, it is very hard to start afterwards.
3) EGR staying partially open. This is supposed to shut off at idle, and in some other conditions. If it was staying partially or fully open at idle, the mixture would be wrong (too lean?) causing the bad idle. Even though my test of the EGR makes the engine idle worse, it wasn't that much worse so it could be partially open all the time.
Does anybody have any suggestions about these ideas or anything else to try?