How does Google AdSense know what's on pages behind a login?

You are correct that until rather recently, Google very clearly said that it wasn't okay to have AdSense ads on pages that were protected by passwords or limited in accessibility. In addition to violating the rules, the other reason should be obvious: if AdSense can't scan the page to identify content, it can't possibly match that content and the ads it places will be irrelevant and ignored.

Digging around in the Google AdSense system, however you can find exactly where it lets you sign up for AdSense for protected pages, and explains how it works.

The URL in question is https://www.google.com/adsense/list-auth, or you can navigate there (once you've logged in to your AdSense account), by going to:

Got it? Now, read what's on that page, and you'll learn:

"Use this section to allow the AdSense crawler to access pages that are behind a login. Our crawler will access these pages only to determine content for ad targeting purposes and will fully comply with Google's privacy policy."

So, you're right. There is no way to have AdSense on private, protected or limited access pages without letting Google AdSense's engine scan and archive the main keywords of the page or pages in question.

At this point, Google recommends you take three steps to get things set up properly:

1. Enter an authentication rule: Specify an authentication URL and a set of parameters that will allow Google's web crawlers to access your content.
2. Verify your site ownership using Google Webmaster Tools: We will direct you to Google Webmaster Tools, where you can verify that you own the site by uploading a file to it or changing the META tag of a file.
3. View your improved ad targeting: When your authentication rule is verified, you should see better ad targeting on your pages. You can check the status of your authentication rules at any time.

Hope that gets you going!