Cartographic questions

Some strategies and tactics I've come up with to ask good questions for technological roadmapping, podcasts, (and life?)

Note: I published this accidentally on June 2 2022 on an unused substack which I will be deleting soon, but if you’ve seen that there is nothing new here. This is the canonical version for commenting, etc.

As I’ve noted before — a big component of how I learn about new areas is by asking questions. Currently I primarily do this in two contexts: technological roadmapping as part of PARPA program design and podcasting. Prompted by the unduplicable José Luis Ricón, I realized it might be useful for me to write down how I tactically go about this.

Roadmapping Questions

Before jumping into the exact questions — a few notes on the general philosophy on questions.

I’m insistent that while thoughts should be written, questions should be verbal. Face-to-face is helpful: body language can tell you when someone is distracted, doesn’t really buy what they’re saying, wants to interject, etc. But it’s not the most important thing. The ability to interject, real-time clarify, go on tangents, or change what you’re saying mid-sentence is essential. As a result, in-person is ideal, zoom is ok, but voice-only can work too. In order to embrace the nature of conversation, it’s important to not be too prescriptive about the wording of a question or getting to all of them. Similarly, questions should enable situations where you want to dig into something unexpected. To that end they should be short and open-ended.

Another essential part of making questions work is to always go into a conversation having done your homework. Nothing shuts people down like asking a question that you would have known the answer to if you had read their work. Nothing opens people up like asking a question that you would only be able to ask if you had delved deeply into their work.¹ Part of the reason for this is that it’s boring for someone to reiterate what they’ve already written down and part of it is simply hard-to-fake demonstrations that you are actually interested in their work and the topic and aren’t just being lazy (which is what many question-askers are).    

Cunningham’s Law also works in conversations. It’s often more productive to throw out a wrong hypothesis and be corrected than to ask the other person to come up with a hypothesis on the spot. Be warned, this tactic runs the danger of anchoring someone in an unproductive area of idea space – another reason to do your homework so you’re suggesting something wrong, but not too wrong. (‘Wrong’ is perhaps not the right word here.)

People are terrible at answering questions like Richard Hamming’s “What are the most important problems in your field?”² As an outsider, you might think of a field in this way but that’s rarely how people inside of them think about their field, if they even think of themselves as being in a single, well-defined field.³ Additionally, I suspect that Hamming didn’t randomly walk up to people he had never met before and ask them his famous question — he was asking people in the Bell Labs cafeterias who he probably had preexisting relationships with or didn’t ask in exactly that way.

With those preliminaries out of the way, my goal for roadmapping questions is to figure out:

• In a given area, what are the compelling goals? (In as much detail as possible.)

• What are the paths to get to that goal? Ideally in as MECE (Mutually Exclusive and Collectively Exhaustive) a way as possible. 

• What (roughly indivisible) projects/intermediate goals need to be done along each of those paths?

• What do the dependencies between each of those projects look like?

• Which people+organizations are best suited to do each of those projects

  • Note here that “best suited” is not just about skill, but about alignment around the end goal, willingness to do it, etc.

The trick is that in almost every situation, you cannot ask those questions. People do not think in this way, especially practitioners in a field.

• Questions after doing this for almost two (!? Oh Lord) years

  • I was very impressed by <thing in paper that shows you read it> where did the idea for that come from?

    • Notes: this only works if you are actually impressed. People can smell fake interest a mile away.

    • This question is much less about the answer than signaling that you actually did your homework, making them feel special, and helping them understand which aspect of their work you’re most interested in.

  • It seems to me based on your work that <X> might be promising but I haven’t found anybody working on it. Have you thought about that?

  • Why isn’t anybody doing that work?

  • What would enable them to do it?

  • Have you considered working with <other discipline informed>? why not?

    • Note that this question can make you sound extremely dumb in the bad way if you don’t have a strong hypothesis for why they would want to work with that sort of person.

  • So it sounds like <X> is particularly <hard/expensive> is that right? Why do you think that is?

  • Do you have a sense of the big audacious goal for your work?

    • This question is less meant to answer the question but to see how people react to it.

  • Other people have told me <devil’s advocate position> what do you think about that?

    • Sam Arbesman points out that more broadly, it’s easier (once you reach a certain level of trust) for people to tell you what or who they think you should steer clear of, which directions are dead ends, or why an idea won’t work. The trick then becomes figuring out whether they are right in your particular context.       

  • <at the end of a conversation> Who else do you think I should talk to about this?

• Questions other people like

  • (From Pritha Ghosh) What are your “white whales?” What projects didn’t quite work out but are still nagging you?

  • (From Sam Arbesman) What kinds of grant proposals have you written that get rejected? What grants have you not even bothered writing because you know they’ll be rejected?

For contrast, here is a list of what, a few years ago, I thought the right questions were going to be:

• What are the most important problems in your field?

• Why haven’t they been solved?

• What do you think is the most promising thing in your field that is not being worked on?

• Why not?

• What is the limiting factor keeping the system from being more widely used?

• Can this idea/technology/technique be connected to that idea/technology/technique ? Why or why not?

• If this piece were changed, would any other pieces need to change?

• Do you have a sense of what the budget would be to tackle these big problems?

• What is the minimum amount to fund a project in a lab?

• Is there a minimum effective budget to generate outlier results?

What I’ve realized is that these are the questions that I want answered, not the questions that, when asked, help me answer them (I realize that’s a bit convoluted). Put differently, you need to map between the questions you want answered and the questions you ask. The mapping is not consistent and not 1:1. I do not think this process can become more efficient through some magical tool or technique — that’s just the nature of communication.

Podcasting questions

Podcasting questions are more idiosyncratic. There are two major differences between roadmapping questions and podcasting questions. First, my goals for podcast questions are less clear — the primary goal is just to have an interesting conversation. Second, not all the questions are for me: as the interviewer I think of my role as being the voice of the audience. That isn’t to say I don’t ask wonky questions from my own interest (I do try to fall closer to Tyler Cowen than a CNN reporter on that spectrum) but I do have in the back of my head what would be worthwhile to a listener.

That being said, some ways that I think about constructing podcast questions that I think are more broadly useful:

• Every question is a balance between giving enough context and constraints that the answer goes where you want it to and talking too much or leaving no room for the interviewee to say something interesting.

• Never ask the questions you’ve written down exactly as you wrote them unless it’s the opening question. It’s weirdly disconcerting for everybody and ruins the illusion that you’re just having a conversation.

• Like roadmapping (and I think in general) — show the other person you’ve done your homework. It both prevents boring recaps of things people can find elsewhere and makes the interaction more friendly. “You once said …” “In your piece, you wrote that …” Of course this can also backfire when people forget something they said or wrote…

Some other observations about questions

I’ve repeatedly run into situations where I craft what I think is a very precise technical question for a professor and they go off on an extremely high level tangent that really just sounds like philosophical mumbo jumbo. For example:

Q: “What are some specific experiments or projects that would drive systems integration in this one technology?”

A: “You need to think about thermal energy.”

I’m not sure why this happens. One explanation is that they’re so far from the experiments/code/equations that they’ve gotten manager-brain. Another is that making specific assertions is scary because they might be wrong. But really, my meta question is why does this happens so often?

I’m still learning how to ask effective questions well, so this is very much a “working notes” type post. If you’ve found tactics or questions that are particularly effective, please comment about it or send me an email!

Thanks to José Luis Ricón, Sam Arbesman, and Pritha Ghosh for discussion, reading drafts of this piece, and suggesting other questions. And thanks to everybody who has put up with me asking them questions!

1. Tyler Cowen is the master of this.

2. If you’re reading this piece, you should probably also read Hamming’s talk/essay.

3. There are of course exceptions here! Keep in mind that this is very much based on my anecdotal experience.