How to Build a System to Engage Subject Matter Experts

“I find it much more useful to pick a certain collection of trusted sources that have a demonstrated history of adding value around a given topic and subscribing to those.”

– Marshall Kirkpatrick

What you will learn

• Why source selection is essential in working with information (02:02)
• Why source size depends on the topic (04:01)
• Marshall’s guide to advanced Twitter search (09:31)
• How to maximise the benefits of a news aggregator (12:30)
• How to create a news article when you find a subject with high engagement (17:11)
• How to store and catalogue content you want to consume (20:12)
• Marshall’s method for connecting which he calls Triangle Thinking (26:41)
• How to read a book for maximum synthesis (30:15)
• How to create and use STEEP analysis (32:10)
• Why it is increasingly important to search for and listen to people on the margins of political power (38:38)

Episode Resources

• Anki Flashcard App
• Symphonic Thinking
• Doc Searls
• Steve Gillmor
• Walt Whitman
• ReadWriteWeb
• Richard MacManus
• PostRank
• Delicious
• Magpie RSS
• Feedly
• Techmeme
• Memeorandum
• Metaweb
• Pocket
• If This Then That (IFTT)
• Bruce McTague
• Roam App
• April Dunford
• Tom Cheesewright
• Daniel Pink
• How To Read A Book by Mortimer Adler
• HyperWrite
• STEEP Analysis
• Incasting
• John Hagel
• Damon Centola
• McKinsey
• Gartner
• Deloitte
• Forrester
• Accenture

Transcript

Marshall Kirkpatrick: Ross, thank you so much for having me on the show. What a great opportunity for you and me to meet, and to compare notes. I can’t wait to listen to all the episodes.

Ross: They are coming soon. Marshall, you have always thrived on information as a journalist, at one point as a tech journalist, so you got to keep on top of stuff there. You’ve built a very interesting startup. A lot of other guys you’ve been across, and all sorts of change. What’s the essence of that? How do you do that?

Marshall: The essence of it, I believe, is that I focus on a few fundamental steps. The first is source selection. I am careful and deliberate about building out a library of sources on a topic that I want to follow. Then I set up an interface for myself, that makes it easy for me to capture both, the most important pieces of information coming from those sources and a serendipitous mix of other information coming from those sources. Then finally, I try to process the information that I get through tools like spaced repetition flashcards, and linked notes taking, a database, paper and pen, symphonic thinking, and drawing of connections between various things that I’ve read over the years. That combination of source selection, interface creation, and post-processing for synthesis has been the fundamental story of how I have worked with information over the years.

Ross: Yes, a lot more people should learn to do it like you, I’d say.

Marshall: Thanks. I hope that this show can help. I want to share some of the practices that I have developed through trial and error over the years. I’d like to tell people about those practices, tools, and strategies, so that some of them may be useful to some people, and/or they may just make people feel freer to experiment themselves, and come up with methods that work well for them.

Ross: Okay, let’s dig into it. So source selection, is this explicit? Do you have a certain number of sources that you’re around? How many sources are there? And how does that evolve? How do you develop that list and evolve it?

Marshall: It really depends on the topic. I’d say that the most important step for me in source selection was just deciding to focus on that. I was inspired in large part by something that Doc Searls said, almost 20 years ago, when talking about using an RSS reader, and I believe, Doc Searls and Steve Gillmor were discussing keyword search versus source subscription, and Doc said, If I listened to everything that was published, that contained certain keywords that were of interest to me, it would just be really noisy, a very mixed quality. He said I find it to be much more useful to pick a certain collection of trusted sources that have a demonstrated history of adding value around a given topic and subscribing to those.
Now Steve took a different perspective. Steve Gillmor back in the day was so set on the serendipity that he refused to even share links to things with people, he’d say, go Google for it, go find it yourself because he knew that as you searched around for something, you were liable to come across all kinds of other magical unforeseen insights, as you finally made your way to your destination. I do think that that combination is really delightful but I think that the source selection reminds me of something that I read, Walt Whitman once said about writing poetry in iambic pentameter; he said, there is a special beauty to creating art within constraints; so having a finite set of sources I monitor, it might be 10 people. If I’m building a custom search engine, for example, to search in the archival content of an organization, these days, I limit that to 10 sources, largely because Google Custom Search now limits to only 10 sources that it’ll index, but that’s fine. There are ways to work around that.
I usually don’t bother, because 10 canonical sources on a topic are great. But when I was covering various industries, as a journalist, or various sectors of the industry, I would go out and build a collection of 300 sources on, say, big data, or geolocation, or mobile, whatever the trend was. When I am watching climate change, for example, I’ve got a collection of 1000 people and organizations who are specialists in climate, that I monitor. Different topics and different circumstances warrant different sizes of the source list.

Ross: So your sources are individuals more than media entities?

Marshall: Not necessarily, no. I really like a mix of both. I’d say that there’s a different flavor to the sources that you get. Say in a Twitter list, you’ll get a good mix, but perhaps a preponderance of individuals. There’s a certain tone to the conversation, there are lots of replies. It’s easy to sort tweets by engagement. I can say, for example, search inside this Twitter list of 1000, climate experts, individuals, and organizations, but largely individuals for anything containing the words indigenous land rights, that has more than 20 favorites, and sort by recency, and I can find that content, then click and open those tweets and see the dialogue that has occurred around them in replies. That is one flavor of research.
But if I am using an RSS folder in an OPML file, a collection of RSS feeds, then perhaps I’m monitoring climate or other organizations, maybe innovation organizations, or management consulting organizations, and reading just the official posts that have come out of that organization’s feed reader. Not many individuals these days are able to blog as consistently as you do, Ross. The fact that your RSS feed just keeps on bringing the hits is amazing. But RSS is largely dominated by organizations these days. Not all the content that gets published over an RSS feed gets tweeted about, so there’s a different tone and a different type of update that can be found in those sources.

Ross: How do you search within the way you were describing a moment ago? What tool do you use for that?

Marshall: Twitter. I just use a little-known but freely available advanced search protocol in the Twitter interface. Some of your listeners may or may not be familiar with the standard search engine protocol of site:domain space keyword to search, not the web at large as they go into Google and say. For example, while I was on a run here before this podcast, I searched site: Ross Dawson.com space post-capitalism, because I had seen you had used that phrase, and I wanted to see what you had written. I was about three kilometers into my jog, so I had to do it with my thumb on my phone and my phone was shaking all around, but it worked out well enough so that I was able to pull up your article about capital market efficiency and post finance reporting, and it was really intriguing.
Similarly, you can use that same search protocol or protocol like it on Twitter to search just inside of a specific Twitter list. A Twitter list is a feature that makes it easy for people to collect a bounded set of topical sources, and that set can be queried, just like a single website can be queried, or a group of websites can be queried using Google Custom Search or another custom search engine. When searching Twitter, the protocol is list: then there’s a number in the list URL. This changed about a year ago, but these days, you view the list, look up at the number in the URL of the list, search inside of that, so list:number space keyword. Then it will bring you back all of the tweets just inside of that specific list of people.
You can say, show me what my collection of climate people have said about a given topic. Now show me what my favorite futurists have said about a given topic. If you’ve got a Twitter list of my favorite people, search inside of those people’s tweets alone, and that closely constrained context really changes the search experience.

Ross: Yes, that’s really powerful, this idea of search within as in, don’t search the universe, but choose your own curated sources and then search within that.

Marshall: It’s so funny, it’s a simple thing but I think there’s something counterintuitive or unfamiliar with the idea, it’s not terribly complicated, but it’s not widely done. It’s even a little bit challenging to explain sometimes. When I worked as a journalist, I put all of these things together, for example, and a little bit more, and the central hub of our research. At ReadWriteWeb, when I was the editor there, or co-editor rather, with founder Richard MacManus, we had a dashboard, where we took the 300 top sources in big data, the 300 top sources in geolocation, we had 12 different topics we were monitoring, we went out and through link analysis and using free off the shelf tools, built these collections of 300. Then we ran those 300, each of their RSS feeds, we ran through a startup called PostRank, that would score each item in the feed on a scale of 1 to 10 by the relative number vis a vis. relative to other items in the same feed, it would score them by the number of comments, shares on social, bookmarks, Delicious, and add it.
Then you could say give me an RSS feed of just the 9.0s and above from Ross’s blog, for example. Then I might take 300 other futurists and say give me the 9.0s, the relative breakout hits from all those sources, and I take all of those RSS feeds, we splice them together and Yahoo! Pipes to make one RSS feed of the big breakout hits in the given sector, then we ran them through an open-source tool called Magpie RSS, that was really simple, just PHP display of our items in an RSS feed, so we could have the 10 most recent breakout hits in a given sector, open up the dashboard, view all those, and whenever we saw something that among deep subject matter experts had spiked, and their audience, we’d say, Hmm, would that be of interest to a general-purpose audience? If so, let’s grab it and consider writing about it. When we see that, then we go into… hope you don’t mind if I tell you the parts two and three of the process?

Ross: I was just going to get to one question. We have dumb aggregators as in just things that pull RSS feeds, but also algorithmic aggregators. I’m just wondering if you use any of the algorithmic aggregators?

Marshall: These days, the closest I do for that, I do use Feedly’s today tab, where they will serve us the most read content in a given folder in your collection. I’ve got an Art folder, for example, in my Feedly account, and when I go and I click that, it’ll show me presumably the URLs that not everyone has the same art collection, or art feed collection, as I do, but the URLs have been clicked through in other people’s subscriptions lesson. Those URLs that have been clicked through the most, up here, up at the top, and then down below that are just a raw river of news in reverse chronological order. I have built and used wherever possible two sides of the coin, where I say I want the top news from these sources, and I want the latest news from these sources. I’m going to try to scan as much as I can at the top, and I’m just going to dip my head into the latest because it’s much more high volume and we’ll see what comes at it.

Ross: Great.

Marshall: What about you? Can you recommend an algorithmic feed reader?

Ross: Techmeme and Memeorandum are part of my daily use.

Marshall: Me too. Are we counting those? For goodness sakes? Back when I was working as a tech journalist, I would check Techmeme 10 times a day, it’s probably down to two or three times these days, and Memeorandum, I’ll check a good five to 10 times a day.

Ross: Yes, whereas your Google News or Apple news are far less interesting. Anyway, so back to the process. So you’ve got your feeds…

Marshall: So something has broken out among subject matter experts that’s got a relatively high engagement among their audiences, and we take a look at it, and we say, Oh, we want to write about that. Then, step two is that we built a system, where we scraped the Twitter bios of everyone who followed anyone on our staff. We made a simple little search engine where you could search for keywords in people’s bios who were following anyone on our staff.
For example, we had a semantic web widget in our dashboard, because that was just the rage 10-12 years ago, and we saw that Google acquired this company called Metaweb, and we said, Oh, that’s a general interest story right there, so we searched inside of our Twitter follower network for the word Semantic Web, we had them ranked by a number of followers, we saw who on staff they were following, so I was able to jump into our Slack predecessor into a Skype chat room and say, Hey, Sarah, would you DM this lady, she follows you, and tell her we need urgent expert comment on this news. We’d fire off three, four, or five of those requests, and people would say, Oh, my goodness, you thought of me to ask for input on this. I followed you two years ago, or three years ago, and of all people, you thought of me? Sometimes we’d tell them that we have a system, but most of the time, it’s a Yep, we thought of you, we thought you’d be the perfect person to ask for comment.
Then, as we waited for those comments to come back, we would also go over to our custom search engine, where we had indexed the archives of all of those top 300 blogs that we were monitoring for breakout hits, so we could search for whatever Semantic Web experts said about Metaweb in the past? Then we were able to say, well in the industry, Metaweb has a reputation for this, that, and the other thing; and people would say, you just happen to have read that blog post 18 months ago and have recall of it now to be able to link to it in your new coverage. Before you know it within, usually our goal was to be in less than 30 minutes, we get a blog post up that covered the news before our competitors that had multiple real-time expert feedback in it, and cited multiple archival pieces of research from subject matter experts, and that was how we competed with other tech publications. It was awesome. It was really fun.

Ross: Yes, that creates value, obviously, for the people reading the articles but in a way, whether you’re sharing or not, this is processed to be able to get some real insight. The next phase is the pulling and storing, or cataloging, or linking? What’s that process?

Marshall: Yes, these days, I consume this content and process it through a couple of different methods. I take in a lot of content through my ears. I really love Pocket on my iPhone and its text-to-speech capabilities. I’ve got an If This Then That applet set up to say anytime I like a tweet that has a link in it, send that link to Pocket. Then I can go for a jog, or do my dishes, or clean my bathroom, or do whatever mundane activities, the jogging is not mundane, it’s beautiful, but having just all of the articles that I’ve bookmarked read as a playlist. I also got a good PDF text-to-speech app that works fairly well. There are lots of them out there. I’ll go and I’ll do that, then I’ll either stop or after I finish running. I also have the other If This Then That, there are a few select sources where I take the whole RSS feed, and just pipe it straight into Pocket.
There’s a short number of people whose content is just so remarkable, lately, Bruce McTague, calls himself the gentleman thief of business ideas, writes these long really thought-provoking blog posts, and every one of them goes straight into my phone, and then straight into my ears, and straight into my brain. Then I’ll stop, and I’ll take note of things that caught my interest, and I’ll put them into Roam Research. For more than a year now, I’ve been using Roam Research. I like that pretty well.
I do all my note-taking in Roam, but reading notes, in particular, I tag as reading, and then once every one to four weeks, depending on when I’m able to take the time on the weekend, I’ll go in, I’ll load up all of the things that I’ve tagged reading, many of which are from paper books, but some of which are from digital as well, and listening, and I will put them into Anki Spaced Repetition mobile flashcard app. Then each morning, I’ll spend between five and 15 minutes, reviewing flashcards of the things that I have read. Then they’ll come back up when I’m working with someone, I’ll say, oh this reminds me of, and I’ll either have it recall, or I’ll just remember that it’s there, and then I’ll go search in Roam, or in Anki, to be able to pull up an original quote, to sites.

Ross: For the flashcards, you’re accepting a phrase, or a quote, or a fact?

Marshall: Yep, whatever I want to try to remember. If I open up my Anki flashcard app right now, for example, I read “most products are exceptional only”, now I have to identify what’s on the flip side of that flashcard. On the other side, what it’s going to say is most products are exceptional only when seen within their very best frame of reference, said April Dunford, in her book, Obviously Awesome. Let’s see if I got it right. Flip it over, when we understand them within their best frame of reference, April Dunford. I got it. Now I’m going to hit the green button, and it says, this is a new one to you, so we’re going to show it to you again in 10 minutes. But if you’ve got it wrong, we would have shown it to you later in one minute, so that’s good.
Then I get another one that says Tom Cheesewright from Book of the Future, once said curation is my shorthand term for the dual skills of discovery and qualification. That was a hard one, I didn’t really get it, so I’m going to self-report and now it’s going to show me that one a little bit earlier, so those are the kinds of things that I have.

Ross: So the idea is these are things which you use in conversation or that are just feeding your views or ideas?

Marshall: A lot of them, yes. Some of them say things like if you do X, and the flipside says your life will fall apart, then I’ll be, that’s right, I don’t want to do that again. There are certainly some life lessons learned as well as things I’ve read.

Ross: In terms of getting connections have you found Roam research useful for drawing together connections between ideas or do you use other tools or other frames of mind?

Marshall: Roam is pretty good. I don’t use it as thoroughly as some people do for that. I have a random plugin there that I really like that, at the top of each day’s notes, will randomly print three of my previous notes tagged either reading, best practices, or lessons learned. Often, I connect those, but probably my favorite method for connections is something I came up with myself. It’s a method that I call Triangle Thinking and the way that I do it is I make a list of three different things. They can be related or unrelated, or random, but they’re usually things that I’m working on or thinking about, inspired by Daniel Pink’s book, A Whole New Mind, where he says if your work is in routine cognitive labor, you’re in trouble because computers are going to eat your lunch, but there are non-routine cognitive and emotional social labor, especially human, that’s where the future lies in terms of future work, and in particular, what he calls symphonic thinking, or the ability to draw connections between seemingly disconnected ideas or entities is a key skill for the future.
When I read that, I said, alright, I’m going to take three things, I’m going to put them on a piece of paper, I’m going to draw connections, I’m going to label them A, B, and C, and I’m going to ask, standing from the vantage point of A, what would A have to offer B? And then I’d flip it around and say, Now, if I was looking from the vantage point of B, what would B offer A? And then B to C, and C to A, so I’d write out six unidirectional connecting thoughts between things. Without fail, at least one and usually two or three of them, I’ll look at afterward and say, Now, that is a keeper. That’s a good one right there. I hadn’t thought of that, and I’d like it, I’m going to use it. Sometimes I’ll bring in a wildcard as well, and add that on and say, Now, what does that thought mean for climate change? Or what does that thought mean for innovation? That idea generation practice is something that I try to do almost every day.

Ross: That’s fantastic. Have you written about this?

Marshall: I have not, no. My buddy Bill Johnston has been telling me I should for a while.

Ross: You should. I’ll add my voice to that.

Marshall: Thanks. Awesome.

Ross: I think that’s really insightful. I actually haven’t read A Whole New Mind, so I think, synthesis is a foundation of the way I think about the world. I love that frame, symphonic.

Marshall: Symphonic thinking, I think other people refer to similar things with the phrase associative creativity.

Ross: Yes. I think that’s just a layer below. Association is one thing. The synthesis is the overarching thing when you’re bringing it all together. So associative is one, that’s the foundation for creativity and innovation, but the symphony or the synthesis is bringing that all together into a whole, so that’s a higher order.

Marshall: Yes, I like that a lot. It’s going to be my 45th birthday tomorrow. I think it was…

Ross: Happy birthday.

Marshall: Yes, thanks. I’m celebrating all month long. I can’t remember what birthday it was a while ago but a few years ago, I bought myself a copy of Mortimer Adler’s book called How to read a book. I really like the way that he talks about, you take something you’re reading, and there are a few basic questions that are good to ask about it. Like, what is it saying? What’s its argument? Do you believe that argument? Do you think that it’s credible? What are the constituent parts inside of the thing? How are they woven together? When I take the time to be mindful about it and deliberately think about it, that can also be super helpful. But there are so many different possibilities.
I feel like there are opportunities to find new ways to relate to synthesis. I feel like we’re just starting to scratch the surface of it. A couple of new things that I’ve started doing recently are using AI, specifically, GPT-3 based startup called HyperWrite to finish sentences and paragraphs based on a piece of information, or to help direct where I should go and search. I’ll start typing a couple of sentences and then say go, and it will use open AI’s corpus of the Internet to come up with the next logical sentences, based on things that I have no domain knowledge of, but that will then help direct where I should go and look, and what terms I should search for, and what some arguments might be to consider. That I really enjoy.

Ross: That’s very interesting. You’re a practitioner of future thinking and foresight would love to hear about how do you look for, find, make sense of, and make value from information.

Marshall: That too, I feel like I’m just beginning to scratch the surface of. I realized a few weeks, months ago that I have this big collection of custom search engines. For example, I’ve got the 10 best sources on artificial intelligence, the 10 best sources on climate change, the 10 best sources on indigenous matters, all on my homepage on Marshallk.com. I’ve made them publicly available to anyone else. I go back to that myself. It’s my little mini-library of dynamic reference books. It’s essentially like the Encyclopedia of English language Chinese news coverage in that custom search engine. I realized a few weeks ago, I had never created a STEEP, or PESTEL, or added collection of custom searches, and now I have.

Ross: Define those for the listeners.

Marshall: That is common foresight model that says when we look at a thing, and anticipate its prospective futures, let’s look at it from a number of different vantage points in terms of how social change might impact or be impacted by this thing, technological change, economic, environmental, and political, and sometimes people include legal, or other ways of doing things, so I just built a collection of custom search engines for myself on each of those different facets of the STEEP analysis. When I’m doing research on, say, travel and hospitality, I did some research on it the other day, I went and searched for some relevant keywords inside of the archives, looked with a focus on recent but not exclusively at all, articles from the top 10 websites covering culture and anthropology. Then I searched for travel and hospitality in the top 10 technology websites, everything from TechCrunch to MIT Technology Review. There’s so much content being published every day, all we can take is just a little tiny spoon to pull up a spoonful from the giant cauldron of soup that’s out there. But a precisely shaped and considered spoon perhaps can come up with some real gold, some real nuggets. That’s one way, is through custom search engines focused on STEEP in particular.
PwC did a STEEP analysis on the future of the insurance industry 10 years ago, my favorite STEEP analysis I’ve ever seen. Because they did probably a Delphi model, where they went out and interviewed a whole bunch of subject matter experts about a series of questions, then obfuscated their identities and language use, then re-shared with everyone what their peers had said in response to the question and said, So what do you think about this? And did three rounds of obfuscated but expert-driven discussions about various topics that they then used to populate a STEEP model on what the insurance industry might look like in 2020. They did this in 2010, but they did it not just in one column, social, technological, environmental, economic, and political but they did it as a continuum, where they said if things trend conservative, if they trend very conservative, here’s what it’ll look like in terms of social; moderately conservative, just straight up, no change; moderately progressive, very progressive. It was just a really rich map of prospective futures. I have done a lightweight reproduction of some of that in some analysis that I’ve done recently.

Ross: Fantastic.

Marshall: Then I love incasting. It’s simplest thing Wendy Schultz calls the little black dress of foresight, real simple tool, where you take a prospective future scenario, and you ask yourself if the world ends up looking like this, what’s a big win that hits the front page of this thing called a newspaper that we used to have, answer that question then, what’s newly prohibited or illegal? And a negative consequence of that future scenario? And then finally, how do you get to work each day in that world? What does just the practical utility around this matter look like? Oftentimes, I’ll go and say, Okay, I’m going to build a set of domain experts that have an extensive published history, on websites or Twitter, and then I will search for keywords to surface content that they have written related to this topic that I’m researching. Then I’ll categorize some of the things that I’ve found into that model and say, Oh, this one looks like one that could inform a big win, Oh, that sounds like a problem, that would help me illustrate a newly prohibited activity, or how I would get to work each day, so combine that information, gathering the foresight models in that way.

Ross: Just to round out, is there anything else other than what we’ve talked about that you think would be a particularly valuable insight, or perspective, or practice that you have?

Marshall: I think that, that one of the often missing elements to these kinds of analyses, and one that I strive to add more and more in my work is a matter of source selection, but in particular with political power in mind. John Hagel talks about how the value creation is increasingly occurring on the margins of power inside of an organization, where frontline workers are solving real-time novel problems, and the insights generated from that work is really one of the best sources of new value, so listening to and empowering those people on the margins of power is essential. Damon Centola says that innovation occurs in a Goldilocks zone best, where you are not so close to the center of power that you’ll be crushed by the immune system of the network, so you want to be out closer to the margins, but not so far outside of the margins, where you don’t have access to the flow of resources for the ability, for your innovative ideas to get traction.
Of course, every organization is just a fractal of the universe in general. When I do this analysis, I increasingly try to look for people who are on the margins of traditional political power in a wildly unjust world. I’ll search inside of the tweets of women futurists or women in tech, or I’ve got a Black Twitter, people of color, influencers on Twitter, or indigenous organizations, Custom Search engine, so I don’t want to just search inside of management consulting firms, they produce incredible stuff, the content that McKinsey, and Gartner, and Deloitte, and Forrester, and Accenture publish is just amazing, but I don’t want to live in a world that is entirely defined by those kinds of organizations.
There are ways to build tools to make sure to look for perspectives from sources outside of the traditional power structures who have been marginalized. That power dynamic is changing dramatically these days. If you are not paying attention to, learning from, and flowing some resources towards people who have been unjustly marginalized, and are increasingly empowered, moving towards the center in a period of disruption, then you’re going to be caught flat-footed and not feel really great about yourself. There are incredibly awesome opportunities coming from newly empowered traditionally marginalized populations who are publishing their thoughts and their work. We can stop and listen to those. We can set up tools to listen, and there’s a lot of innovation and inspiration out there, so I want to encourage folks to do that as well.

Ross: That’s a fantastic point. It’s so important for so many reasons. If you just look at the middle the core, then you’re missing a lot of what’s important and you sometimes just need to take some deliberate frames, all right, this is what’s excluded voices.

Marshall: Practically and ethically, I think, it’s essential.

Ross: Fantastic, thank you so much for your time and your insight, Marshall. It has been really valuable.

Marshall: Thank you for including me, Ross. I can’t wait to listen to the rest of the episodes and just learn from you in this. I really feel like we’re in a period of such dramatic information overload that dealing with that, both in terms of consumption and publishing effectively, and getting a hold of people through dignified, diligent, shirtsleeve tugging, is such a new set of skills. I think it’s a service to all of us that you’re doing here to interview folks so we can all learn from each other.

Ross: Thank you. It was great talking to you, Marshall.

Marshall: You too, Ross.