The Era of Small Data Begins
Posted on 6 Mar 2012 // personal
This is the first post in a series on the rise of small data* and the new platforms, tools and rules to empower people with their data. It was written for “The Rise of Big Data” panel at the Stanford Graduate School of Business E-Conference on March 6, 2012.
More data is created every year or so than has been created in all of human history. In this always-on, always-connected world, where even things are being plugged into the Web, the amount of data is growing exponentially.
The collection, storage, analysis, use and monetization of all that data is called “big data.” Corporations and governments are hyper-focused on becoming big data experts to avoid being permanently left behind. The first movers to master the art and science of big data are already changing the way we live, while disrupting industries and amassing fortunes at speeds never before seen.
Given the stakes, massive investments are being made every year to build the technology and expertise required to succeed in big data, optimized, of course, around the needs of companies and governments, not individuals. Industry experts have likened this big data boom to the early days of “big oil,” and refer to data as the “new oil.” Just as oil was essential to building the modern industrial economy, data has become the lifeblood of the new digital economy.
Companies must learn to compete in big data regardless of their industry, or else face obsolescence. This is a tough challenge and touches all aspects of the operations, strategy and culture of companies. At the same time, opportunities abound as entirely new industries are emerging around data as they did around oil -- sourcing, extracting, refining, mining, analyzing, distributing, and selling large sets of data.
Big data creates big problems
With its insatiable appetite for digital bits and bytes on each of us, big data is driving a virtual arms race to capture and exploit information about our every move. Big data will log the life of a child born in 2012 in such a way that the person’s activities will be able to be reconstructed not just by the day, but by the hour or minute. In the hands of bad actors, the potential for wrongdoing with these permanent and growing archives of our lives is real and rightfully concerning.
Yet, until recently, people had virtually no idea of big data’s existence as its tools and marketplaces remained largely hidden. The next generation of tracking and data mining technologies are being created based on the assumption that individuals do not care enough to change their online and mobile behavior, which confuses lack of interest with the current lack of alternatives.
But with privacy and security concerns now front-page news, and the financial triumphs of companies built entirely from personal data such as Facebook, Google and LinkedIn, people are waking up and starting to ask tough questions. While companies and government regulators negotiate over how to curb the most egregious risks and abuses, a new and more powerful model is emerging that is designed around the needs and interests of people, providing them a far better, more sustainable alternative to the status quo.
Enter small data
Small data puts the power and tools of big data into the hands of people. It is based on the assumption that people have a significant long-term competitive advantage over companies and governments at aggregating and curating the best and most complete set of structured, machine-readable data about themselves and their lives – the “golden copy”. With proper tools, protections and incentives, small data allows each person to become the ultimate gatekeeper and beneficiary of their own data.
Built on privacy by design and security by design principles, small data can help people become smarter, healthier, and make better, faster decisions. It can help people discover new experiences more easily, reclaim time in their busy lives, and enjoy deeper, more positive relationships with others.
Small data can also greatly improve the capacity and performance of governments and non-governmental institutions, from eliminating time-consuming forms and other inefficient data practices, to improving public health and education by leveraging the power of more accurate and complete data provided with an individual’s permission. Such institutions can also help share important data with individuals, allowing them to have a copy for their own use.
Applied to commerce, small data holds the promise of connecting people with the best and most relevant products and services in a safe and anonymous environment. It can transform advertising into a more respectful, less disruptive industry that rewards people for their time and engagement with their messages and for their purchases. Small data offers customers the opportunity to better balance and assert their interests with companies (some have called this model Vendor Relationship Management (VRM)). Companies who play by these new rules and earn the trust of individuals will be rewarded with access to rich and robust data otherwise unavailable, giving them instant competitive advantages over companies who choose to go it alone.
The first small data platform – a data vault, private network and apps
Personal has spent over two years designing, building and launching the first scalable small data platform. At its core is a secure data vault to aggregate and store structured and unstructured data from just about any source. A private, personal network sits on top to set permissions for data to enter or exit the vault. People are able to connect with other people through the network, and soon with companies, apps, and private or public institutions, and decide which, if any, of their data they are willing to grant them permission to access.
We have put equal weight on privacy and security, and on helping people leverage their own data in exciting, new ways. These concepts are inextricably linked in small data, which requires a high degree of trust to function properly. Similarly, we have rewritten the legal rules of data ownership to protect and empower users, who we call owners. And, because we know relationships can sometimes end, we have built what we believe is the most complete data portability and deletion capabilities in a data platform. Trust doesn’t work unless you are truly free to leave.
In addition to launching our own apps in the coming months, we are inviting developers to apply for early access to build apps on our platform to show off the power and benefits of small data. Individuals have never imagined the magic of running apps on reusable, structured data about the most important things in their lives, while developers have never assumed having access to such high quality data on which to innovate. The possibilities are limitless.
We are excited to help usher in this new era where permission, transparency and privacy become the norm, and where companies and governments have to align around new rules and provide clear and compelling benefits in order to earn access.
At Personal, we see the future through the lens of small data -- and we think it will change everything.
*I first heard the term "small data" in a talk by Jeremie Miller, co-founder of Singly, at the Web 2.0 Summit in November 2011, and it's one that stuck with me. Doc Searls included a passage from the talk in his new book, The Intention Economy (Boston: Harvard Business Review Press, 2012), p. 196:
You need to have a home for your data. I'm trying aggressively to define this home, in… the best software, the best technology, the best legal terms. This home is yours — that you own, that you control. And this home is for your data.
This ability for you to have it and share it out, is going to transform our industry, over the next ten years. There is going to be this tectonic shift, as everything sort of re-shapes and re-centers itself around people, around individuals, and around the mountains of data that they have… Everybody talks about 'big data.' This isn't big data. This is going to be the era of small data, of my data.