The problem seems to be with the "Display modules to all groups always". I turned it off, and assigned the "login_box" module to the "Anonymous" group. I get the login box before I login, but it dissapears after I log in.
I added the "Registered" group, but the login box doesn't always appear. With a test user I created, that has only ever belonged to just the "registered" group, the login box appears. To another user, that has other groups besides the "registered" group, no login box.