Non linear progression is a progression system in a game that allows the player to choose upgrades and abilities they unlock, so that the order in which they receive abilities and upgrades is up to them. This is different than games which give you abilities in a specific order, usually because the game developers want to control what you do and because they design the game to be played in that way.
Nonlinear progression can cause balance issues in a game, since the game developer does not know what a player will be able to do at a given point in the game. Some permutations of upgrades and abilities may make a challenge in the game far easier or far harder than another permutation, which could make the game difficulty rise and drop unintentionally, or in the worst case may make the game too easy or impossible to finish.
If the option of purchasing both upgrades and abilities are available, then it is also possible that purchasing too many or not enough of one of the two might also cause balance issues.
Some ways to avoid this are to make sure the nonlinear upgrades available during a fight are known and that they all offer a different way to approach the challenge rather than make the fight easier or harder.
This could be done by limiting the amount of nonlinear abilities available at any point in the game, this is called “gating”. There are “gates” that are unlocked at certain points in the game. This is not necessarily liked though since there might be specific abilities and upgrades that the player wants earlier, but has to wait to get.
Balance through limitations
An alternative method is to make sure that none of the abilities and upgrade in the game and their permutations would change the difficulty of the game, but this might change the nature of the game by changing or limiting the progression that was already developed. Game content should not be substantially changed for the sake of balance, unless it is absolutely necessary.
One way to avoid doing anything to the progression options is to have the challenges’ difficulty scale with the purchases the player makes. This could either be a general difficulty increase such as an increase to enemy statistics, an improvement to their AI, or giving the enemy more weapons, units, or abilities, or instead of a general upgrade the enemy could receive something to specifically counter or balance the upgrades and abilities the player purchased. The latter is a bit more time consuming, but both make sense in certain situations. Naturally it would make sense for some games to have their enemies improve, but it depends on the situation, such as getting new weapons.
If the player however has become much stronger than they were at the start of the game, yet the enemies at the start of the game have also become extremely powerful simply by their statistics increasing, it may feel strange for the player to still have trouble with them even after having become far more powerful. If there is no justification within the game’s world for the enemies to become stronger, it may feel out of place and even be less fun. This specific situation is actually inadvisable for me, since I think it is fun to go back to the starting units and be able to easily defeat them while still having more difficult opponents in other parts of the game.
Unlock Tokens and Farming
A major problem for balance can arise from giving the player a potentially unlimited amount of money without progressing the game, such as receiving awards for replaying previous levels or challenges. If the player can “farm” to further their progression without the rest of the game progressing, they could unlock everything and be far too overpowered, making the game too easy.
An obvious solution is “sidegrades”, where even if you unlock everything, you do not have an advantage and the game does not become too easy.
Another solution is to implement unlock tokens that limit how many things you can have unlocked at a time, but still letting the player choose which upgrades and abilities to get.
Talent trees are generally this type of system, where you get a talent point for leveling up and can choose where to put it.
Farming can also be a good thing, since it allows the player to manually make the game easier if it is too hard, or avoid farming to make the game harder.
One way to balance too many upgrades or too many abilities is to have loadouts, where only a specific maximum number of upgrades or abilities can be used. This is common in games like Armored Core, where you can buy as many mech parts as you want but you can only use one head, torso, arms, legs, left and right arm weapons, and so forth at a time. The game also gives you points to upgrade speed, attack, shields, etc., but you only have a limited number of points to distribute. You can get more points, but it is still very limited, which is not a bad thing because it makes you strategically choose between various upgrades without making you overpowered.
Diablo 3 also limits you to only use four abilities at a time, which has the same effect, and makes you choose between various talents.
Loadouts are generally best done as sidegrades.
I recommend trying to balance the statistics of upgrades and abilities to avoid changing their nature or limiting them. Any number of the above solutions can be implemented together, such as having some systems be gated, some having unlock tokens, or both, in addition to other systems having other combinations of solutions.
Whatever system is used should always make the game work as intended while limiting it as little as possible.