Secrets, URLs, and Logging, Oh My!

It’s interesting how as you move from “programmer” to “software engineer” so many little details evolve and start to take over.

The last few weeks I keep getting issues from my security team about logging “secrets” in URLs. So I want to explore how my view on this has changed over time.

1. college student – just put your secrets anywhere in the HTTP request and GSD
2. new grad – ok, I’ll use standard auth protocols (“Secure” Cookies, HTTP Basic, OAuth, etc) where I can.
3. devops – no secrets in URLs ever! These get logged/recorded all over the place. Always put secrets in HTTP headers.
4. senior eng – well actually… email validation, password reset, and similar APIs use one-time use tokens from email clients, so the tokens have to be in the URL. But as long as they’re truly one-time use tokens, they can be exempt from the above “no secrets in URLs” rule. Right? 😅
5. today – security is concerned even with one-time use tokens, because if they’re not 100% guaranteed to be marked as used as soon as they enter our system (and they’re not), then there’s a chance they weren’t used and those are “live” tokens in some third-party system.

So now I’m trying to figure out the new, more evolved guiding rules here.

• Security would like the rule “never log URLs” but that’s hard, because URLs are very useful for actually running and monitoring your service
• Also, it’s not just logging. Both OpenCensus and OpenTelemetry automatically include the full URL (including query params) as span attributes.
• So you’re really fighting an uphill battle assuming that you can actually eradicate all instances of URLs ending up where they shouldn’t.

But then again, is there a safe set of best practices here? At first I thought it might be “log the URL path and allow one-time use tokens the query”, but unfortunately that doesn’t satisfy my security team AND many systems you use probably log the full URL including query params already.

To make matters extra fun, there are also “hidden” examples like “return_to” query params (e.g., to go back to the previous page after logging in) which might also have secrets in them if the original URL had one in it. And there are broken clients that could send secrets as query params to APIs that don’t even support those query params, and they end up in your logs/monitoring anyway.

There is no easy answer here that I see. For now, my rules are

1. no secrets in URLs, except one-time tokens for email-client use cases
   • prefer to put these in the query param
2. minimize the number of places where you record the URL
   • prefer to just record the URL path and exclude the query string
   • more specifically, prefer to record just the URL path template (so people who mess rule #1 still see /validate_email/{token} instead of /validate_email/abcd12345)
3. prioritize security over user experience in your one-time use tokens
   • think about the tradeoffs
     • do I mark the token used and then use it? any errors I get will be returned and require the user to ask for a new link; or
     • do I use the token and then mark it used? any errors I get will mean that the token is live and likely in a third-party system (security leak!)
   • realize that if you’re successful and start with “user experience”, you’ll later on have a security team who will force you to change your choice
4. understand that a sufficiently paranoid security team won’t really care if its a one-time use token or a long-lived secret
   • unless you can guarantee that the token is one-time use and there is no possible way that a token might still be usable, they’re going to treat it as a usable leaked secret
   • if you use any third-party systems which involve tokens, your job of proving and guaranteeing the above is even harder
5. acknowledge that all your rules will be violated somewhere
   • even if all your first-party systems only record the URL path template… some third-party system will record the entire URL
   • your best bet is to figure out how to avoid sending the URL to the third-party in the first place

This list of rules is far too nuanced for putting on a bulletin board and expecting an army of engineers to adopt. This needs to be tightened up significantly.

I’ve love to hear your thoughts and how you’re handling this challenge in your company. What are your rules for URLs, secrets, and logging?

PS – This very common and super useful practice of one-time tokens in emails (e.g., user verification, password reset, magic links for sign-on) is particularly troublesome for software engineers. Links must be GET requests, and the HTTP spec says that GET requests must be safe. But the link changes meaningful state on the server (e.g., user is now verified!) which is decidedly unsafe. The only thing I’ve found in my (admittedly cursory) search is this one StackExchange question.