An algorithm is a set routing, or combination, of the operators. Routing the modulation in different ways gives you several different methods of applying modulation and different sonic results.
Pictured above is a 2 operator algorithm, which is FM synthesis in its most basic form. The Digitone has four operators available which can be routed in many different ways. Different algorithms dramatically changes the characteristics of the output - for example stacking four operators can lead to very complex timbres.
The Digitone has eight different algorithms where the four operators are routed in different ways. Each algorithm has two carrier outputs (X and Y) that come from two different operators depending on what algorithm you chose. It is possible to use the MIX parameter to mix between these outputs so that you can cross-fade between two separate timbres.
The lines going to X and Y indicate the output from a carrier. There are two different ways a carrier output is sent to the X/Y outputs: 1. Its amplitude is unaffected by the operator envelope (dotted line). 2. Its amplitude is affected by the operator envelope (filled line). The sound from the X/Y outputs is then routed via the overdrive to the filters.
Read more about the Digitone FM synthesis here:
Digitone FM synthesis overview.