Unfortunately we do not have any Goji berries planted at the garden so I cannot speak from experience here but Goji berries are generally not difficult to grow (as you have been finding out) but getting them to fruit usually takes a bit of time, usually 2-3 years. As you have had yours planted for 4 years now I agree you should be getting fruit.
I’m just wondering if you have been pruning them correctly? It’s possible that you have been cutting them back too hard and therefore reducing the fruiting.
Flowers and fruit are formed on the stems that grew in the previous year, so pruning aims to encourage the production of this wood. It is only necessary to prune lightly in early spring, removing dead and badly-placed shoots. If necessary, cut overlong stems back to a well-placed branch and remove some of the oldest wood. If you need to renovate your plants the time to do this is in early spring, reducing the plant to a low framework of branches or cutting close to the base. This will cause vigorous re-growth, but initially reduces the fruiting.
Don’t give up on them yet, you may have missed out on fruit this year, but if you stick to the pruning regime above and feed in Spring with a general fertilizer such as Growmore and every two weeks during the growing season with a potassium-rich fertilizer such as Tomorite hopefully you will see some fruit next year.
Lynn Burton - Horticultural Adviser