Every time a setting is changed in the app, you need to force refresh (CTRL + SHIFT + R on Windows) to clear your browser's cache. The browser cache will clear itself out every 30 minutes, but force refreshing / clearing the cache allows you to see the changes instantly.
You have a notification about this in the app and also detailed instructions on how to clear cache on various browsers / operating systems. Check this knowledge base article to find out more about how are caching works and how you can bypass it to see the changes faster.
We also wrote a bit about here about how we optimize our app and use caching to speed up your store in our case-study for VITALS.