Digital life

New systems may allow humans to record everything they see or hear—even things they can't feel—and store all that data in a personal digital archive.

By Gordon Bell and Jim Gemmell
The elusiveness of human memory can sometimes be mind boggling. We encounter its limitations every day, when we forget a friend's phone number, the name of a person with whom we have a work relationship or the name of a favorite book. People have developed a variety of strategies to deal with forgetfulness - from messages scribbled on sticky notes to portable electronic address books - but despite everything, there will always be important information that evaporates into thin air. However, not long ago our team in Microsoft's research department embarked on a mission to digitally document every aspect of a person's life, and we're starting with the life of one of us (Bel). For the past six years, we have tried to document all of Bell's communications with humans and machines, the sights he sees, the sounds he hears and the websites he visits, and store it all in a personal and secure digital archive that can be searched.

Digital memories can be used as more than just an aid to remember events, conversations and projects from the past. Mobile sensors can collect information about things that are not even perceived by humans, such as blood oxygen levels or the amount of carbon dioxide in the air. Computers can scan this data over time to identify patterns. For example, they may determine which environmental conditions make a particular child's shortness of breath worse. The sensors can also record the approximately three billion heartbeats in a person's life, combined with other physiological signs, and warn of a possible heart attack. The information will allow doctors to identify irregularities as soon as possible and warn before the disease worsens. Your doctors will have access to a continuous and detailed medical record and you will no longer have to strain your memory to answer questions like "When did you feel this way for the first time?".

Our research project, called MyLifeBits, provided some of the tools needed to build a digital archive of a lifetime. We discovered that the digital memories allow a person to experience and relive events with the help of sounds and images, and increase the ability of personal observation similar to the way the Internet helps scientific research. With a few keystrokes you can find again every word you've ever read in an e-mail, computer document or website. Computers can analyze your digital memories to help with time management and tell you when you're not devoting enough time to your top priorities. It is possible to record where you have been at regular intervals and create animated maps of your movements in space. And perhaps most importantly, the digital memories can allow all people to tell their descendants the story of their lives in a detailed and attractive way, which until now was the exclusive property of the rich and famous.
A network of paths

The vision of extended memory through machines was first described at the end of World War II by Wenver Bush, who was then the director of the US government office that oversaw the study of the war. Bush proposed a device called "Memex" (short for memory extender, "memory extender" in English) that would be based on microfilm that would store all of a person's books, records and communications. The Memex was supposed to fit inside the desk and be equipped with a keyboard, microphone and several monitors. According to his proposal, the person behind the desk could use the camera to make microfilm copies of photos and documents or create new documents by writing on a touch-sensitive screen. The Memex user could also attach a camera to his forehead to take pictures of things when away from the desk. One of Bush's most far-reaching ideas was that the Memex should be designed to play off the associative thought of the human brain, which he described graphically: pathways inherent in the brain cells."

In the fifty years that followed, far-sighted pioneers of computer science, including Ted Nelson and Douglas Engelbert, developed some of Bush's ideas, and the inventors of the Internet borrowed the idea of ​​a "network of paths" to build their system for linking sites. However, the Memex itself remains technologically elusive. However, in recent years, rapid innovations in storage, sensing and processing technologies have paved the way for new digital recording and retrieval systems, which may eventually go far beyond Bush's vision.

The growth in digital storage capacity is staggering: Today, a $600 hard drive can store a terabyte of data, enough space to store everything you read (including e-mail, web pages, articles and books), every piece of music you buy, eight hours of Recorded speech and ten pictures per day for 60 years [see table on page 64]. If the current trends continue, within ten years you will be able to have this amount of information in the flash memory of your mobile phone and connect wirelessly to a 4 terabyte drive that will cost $100 and will be installed on your personal computer. In 20 years, the same $600 will be enough to purchase 250 terabytes of storage, which will be enough to save tens of thousands of hours of video and tens of millions of photos. This capacity should satisfy everyone's documentation needs for more than 100 years.

At the same time, manufacturers are producing a new generation of inexpensive sensors that will soon become common. Some of them can record a lot of information about the user's health and physical movements. Other sensors can sample the temperature, humidity, air pressure and amount of light in the environment and even detect hot bodies nearby. Some of the detectors are designed to be worn, and others are designed to be placed in rooms or installed inside appliances, such as refrigerators (a sensor will be able to track your snacking habits by measuring the number of times you have opened the refrigerator door). Inexpensive microphones and cameras are now enough to be installed almost everywhere - especially in mobile phones, the cameras installed in which have already become standard, as voice recording will soon be.

Finally, the dramatic increase in computing power over the past decade has resulted in the emergence of processors capable of retrieving, analyzing and displaying enormous amounts of information. A typical laptop is capable of running a database nearly 100 times more powerful and larger than that of a major bank in the 80s. An inexpensive mobile phone can surf the web, display videos and even understand a little speech.

As digital recording hardware improved, more and more people began to create an electronic record of their lives. The advent of cheap, high-quality digital cameras (including those integrated into mobile phones) has led to a boom in photography. Photo blogs are more common today than personal websites. The young people - especially them - embrace blogging and the use of mobile devices. The fact that this boom in digital documentation is happening with even the most basic of tools demonstrates how deep the passion to document is. The interest will undoubtedly increase even more when the digital documentation becomes simpler and more comprehensive.
Memories of one man

Our experiment with digital memories began in 1998, when Bell decided to stop using paper and get rid of the messy mountain of articles, books, cards, letters, notes, posters and photos in his possession. To transfer this pile of memories to a digital archive, Bell compulsively scanned all the documents and certificates he had collected in his personal life and throughout his long career in the computer business. (He even went so far as to scan the logo printed on coffee mugs and T-shirts.) He also began digitally recording home movies, videotaped lectures, and audio recordings. Bell is now a paperless man, but the price was high: completing the task required the work of a personal assistant for several years. (The inclusion of more recent items in the archive did not require as much effort, since the vast majority of documents, images and video are now produced in digital format, and their capture is done automatically.)

However, after scanning all this information, Bell felt frustrated. The limitations of the software available at the time did not allow him to use real usage information. This frustration led to the MyLifeBits project. When we started the project in 2001, PC search tools were clunky. Thus, we set ourselves the goal of creating a database that would allow not only full-text searching on a personal computer (a common capability now), but also quickly retrieving digital memories through properties known as metadata, for example, the date, place, and subject of an image, or written or recorded notes that a database The data is attached to the file. Metadata is often an essential component of the recall process. A person searching, for example, for a particular e-mail address, may remember that it was sent to him at a particular time during the year. By linking metadata, most of which is stored automatically, with digital memories, users can efficiently scan even the largest databases.

The MyLifeBits system also provided Bell with a new toolbox for capturing his interactions with people and machines. The system records his phone calls and the programs broadcast on radio and television. When he works on his personal computer, the system automatically stores a copy of every web page he visits, and the transcript of every instant message he sends or receives. It also records the files he opens, the songs he plays and the searches he makes. The system even checks which windows are at the front of the screen at any given moment, and how much keyboard and mouse activity is taking place. When Bell is in motion, the system continuously determines his position using a mobile GPS device and transmits the information wirelessly to his archive. This geographic tracking allows the software to automatically associate the photos Bell takes with the place where he took them according to the time of the photo.

To create a visual record of his day, Bell wears the SenseCam, a camera developed by Microsoft's research department. The camera automatically takes a picture whenever its sensors think the user wants to take a picture. For example, if the infrared sensor installed in the camera detects a body emitting heat in the environment, it photographs the person. If the lighting level changes a noticeable change - a sign that the user has entered or left the room and moved to a different environment - the camera takes another picture. A recent study conducted by researchers at Addenbrooke's Hospital in Cambridge, England showed that a patient suffering from memory impairment, who went through the SenseCam images every night, was able to retain memories for a period of more than two months (for comparison, reading a personal diary every night hardly improved the ability The recall.) Neuropsychologist Martin Conway from the University of Leeds in England estimates that the SenseCam may become "the first effective memory stimulator of the 21st century."

After six years, Bell has amassed a digital archive of more than 300,000 records, occupying about 150 gigabytes of memory. The information is stored on Bell's two-disk laptop and his assistant's PC, and these are backed up with a local backup and another backup elsewhere. Video files take up most of the storage volume - over 60 GB - the pictures consume about 25 GB and the sound files (mostly music) take up 18 GB. The remainder is divided between 100,000 web pages, 100,000 e-mails, 15,000 text files, 2000 PowerPoint presentations, and so on. Bell found the system particularly useful for contacting old acquaintances and locating other people he needed to contact. He also used MyLifeBits to retrieve websites for citations in his research papers, provide doctors with data on bypass surgery from 25 years ago, and obtain a photo of a friend who died for a newspaper obituary.

Some of MyLifeBits' features, such as full-text search, have already been incorporated into commercial products. However, the system as a whole needs further development to improve ease of use and data management. Better speech-to-text software would greatly improve the system by allowing users to search for words or phrases in phone calls and other voice recordings. Similarly, automatic face detection will solve the vexing problem of image tagging. Retrieving the information would become easier if the system could automatically identify the nature of each of the hundreds of types of documents, perhaps by analyzing their structure and content. But our project has already dramatically illustrated the evolution of the personal computer from a word processor and number cruncher to a processor of interrelationships capable of documenting every detail of the user's life in very accurate multimedia. Many experts predicted the death of the personal computer, but it is clear that the "personal" part of it is not going to pass away. In fact, computers may become even more personal. What will change is the "computer". Our machines will evolve into computerized ecosystems, which will contain not only computers but also Internet storage services, new access devices (such as mobile phones and entertainment units) and sensors that will be found in every corner. It is thought that the bits of our lives will eventually be stored on a home server that will be connected to various internet services.
the realization of the vision

To illustrate the potential impact of digital memories, we imagined a day in the life of a fictitious family making full use of this technology in the not-so-distant future [see box on pages 60 and 61]. Different parts of the family's digital memory are stored on their personal devices - phones, laptops, personal computers and more - but all the information is also securely transmitted over the Internet to a host server operated by an imaginary company called LifeBits. The company manages the storage of the data, performs regular backups (so that it is possible to retrieve material that was inadvertently deleted) and keeps copies of the archive in different places to ensure that it is not destroyed in the event of a natural disaster or human tampering.

Since most of their information is available through secure Internet access, family members can retrieve it from anywhere and anytime. Particularly sensitive information, which could put someone in legal jeopardy, can be stored in a remote data storage account—a Swiss data bank, so to speak—to keep it out of the reach of local courts. The children in the family can encrypt their recordings, but the LifeBits service will give parents access to the data in case of emergency. Similarly, parents' employment contracts will ensure that some of their digital memories containing information related to their work will belong to employers. When these employees leave the workplace, they may have to perform a "partial lobectomy" on their memory copies, which will strip them of anything that is considered company property.

Some of the scenarios we described are not futuristic at all. Sensor platforms that can be worn on the body, collect medical data and monitor vital indicators, such as heart rate, breathing and the number of calories burned, are already sold by companies such as VivoMetrics of Ventura, California, and BodyMedia of Pittsburgh. At the same time, Dust Networks from Hayward, California developed a wireless switchboard for transmitting signals in a network of sensors. The human speechhome project (on the weight of the "human genome"), led by Deb Roy of the Media Lab of the Massachusetts Institute of Technology (MIT), deals with recording almost every waking hour in the first three years of a child's life - Roy's son, now one year old – to learn how people acquire language. Kiyoharu Aizawa and his colleagues at the University of Tokyo are working on wearable video recording systems that will be able to identify interesting moments and capture them by monitoring the alpha waves in the user's brain.

Microsoft's research department supports 14 universities that are engaged in a variety of projects in the field of digital memories. One of these is MyHealthBits, led by Bambang Parmento of the University of Pittsburgh. This study faces the challenge of recording huge amounts of medical data and managing the huge records that are created as a result. Recent studies conducted at the University of Washington have shown the benefits of continuous medical monitoring of diabetics and people suffering from sleep disorders.

This initial progress is encouraging, but ushering in the age of digital memories will not be without problems. Certain countries and certain US states currently restrict the recording of conversations or photographing people. Many people are equally troubled by the recording of information that could be used against them in a court of law. Digital memories, unlike those in our brains, will serve as admissible evidence in any legal proceeding. Richard Nixon's famous advice to his aides was to say "I don't remember" when testifying before the grand jury, but tapes of his own conversations led to his downfall. Those of us who see digital memories as an extension of the mind, believe that the use of such materials in court is actually self-incrimination. However, new technologies can help reduce the potential dangers. For example, when recording others, it would be possible to blur their images or voices to prevent illegal recording.

Preserving the privacy of digital memories will be crucial. The idea that identity thieves, gossips or totalitarian states would gain access to such records is a terrifying one. However, most people already have quite a bit of sensitive information on their personal computers, and its security is an important consideration, even going as far with the idea of ​​digital memories. (Though archiving a lifetime's worth of personal data makes the problem quantitatively, if not qualitatively, more serious.) Furthermore, even if it were possible to make our computer systems as secure as Fort Knox, users would have to be very careful when sharing information their; One wrong push of a button can spread a person's entire medical history around the world. To avoid such errors, the user interfaces of the digital memories must be better than the ones we have now, and we will need intelligent software to warn us when sharing the information seems dangerous.

Another technical challenge will be ensuring that users can open their digital files decades after they are stored. We have already encountered cases where we could not access documents because they were stored in an outdated format. Digital archivists will need to constantly convert their files to current formats, and in some cases will need to run imaging software for older systems to retrieve the data. It is conceivable that a small industry will emerge whose purpose will be solely to prevent people from losing information due to the development of formats.

An even greater challenge would be to create software that would enable computers to perform useful tasks by tapping into this vast pool of accumulated knowledge. The final goal is a machine that can function as a personal assistant and anticipate the user's needs in advance. At the very least, computers will have to do a better job of organizing information. Search strategies that are appropriate for a few bookshelves may not be appropriate at all for a collection the size of the Library of Congress. Most of us are not interested in becoming the librarians of our digital archives - we want the computer to do the dirty work!

Because of this, our research group is very interested in applying artificial intelligence (AI) to digital memories. Although many experts are skeptical about this, we believe that such a software could provide practical results if it could extract data from the huge database in the personal archives. There is no doubt that an artificial intelligence system that is designed to operate with a wealth of information will do its job better than software that must choose a recommended course of action based on individual data. We have started working in this area and developing software that can organize files based on their content, but we still have a lot of work ahead of us.

In a sense, the age of digital memories is inevitable. Even those who are horrified by our vision will own a much larger storage volume in the coming years, and will expect their software to provide more and more help in using it. The idea of ​​all-encompassing recording may scare some people, but for us the excitement far outweighs the fear. Digital memories will bring benefits in a wide variety of fields and provide a treasure trove of information about the thoughts and feelings of humans. The continuous monitoring of patients' health will allow the doctors of the future to develop better treatments for heart disease, cancer and other diseases. Scientists will be able to peer into the thought processes of their predecessors, and future historians will be able to examine the past at an unprecedented level of detail. Only the limitations of our imagination prevent us from estimating the number of opportunities that digital memory has in store for us.

A day in the digital life

Access to digital memories throughout life can improve work productivity, medical care, academic achievement and many other things. A glimpse into the daily life of the Digital Family, a fictitious family living in the not-so-distant future, demonstrates some of the possible benefits.

Ann Digital, a professor of chemistry, needs to review an article written by a co-worker, but cannot remember the author's name. However, she remembers seeing the article while on the phone with a student last summer. She narrows her document search to those observed in the phone conversations with the student, which were automatically recorded at that time, and immediately finds the article.

Dave Digital, a stockbroker, archives all the documents, e-mails, phone calls and websites he visits during the workday. When Dave writes an email, his time management software alerts him that he is spending too much time communicating with an unimportant client. He gives up on polishing the message, sends it as it is and goes to work on an account with a higher priority. Dave also reviews recent records of his weight, heart rate and calorie intake to determine the negative effects of two consecutive days of stressful work meetings.

Dave takes Laura, his seven-year-old daughter, to the doctor. Laura's medical analysis software recommends a general examination because her weight gain in the last six months (as recorded by the daily home scales) is lower than expected. The doctor says the symptom may be a side effect of Laura's asthma medication. Since her breathing has been excellent in recent months, the doctor suggests stopping the use of the medicine for a while.

During dinner, Dave and Ann argue with Steve, their 14-year-old son. Ann is frustrated that Steve leaves his homework to the last minute and wants him to start the next essay right away. But Steve shows his parents the results of his study analysis software, which show that his grades are just as high when he prepares his lessons late. Steve's digital memories also reveal that his learning style is auditory, meaning that he benefits more from group discussions than from reading.

Ann's mother, Jean, is a retiree living in a nursing home. Nursing home staff have access to some of Jean's digital memories; They will receive an automatic alert if abnormalities in her heart or breathing are detected, or if the sensors she wears show that she is not taking her daily walks. Anne found that watching her mother's dishwasher records could teach her a lot: when Jean wasn't feeling well, she tended to neglect washing the dishes. When Jean goes to bed, she watches old photos and movies from her digital archive, and uses an interactive display to take a long walk down memory lane.
Overview/Digital Memories:
Because human memory is prone to error, researchers are trying to develop systems that can automatically record communications, documents and video images and store everything in a searchable archive.
Developments already taking place in the fields of sensors and data storage promise to make digital recording easier. The bigger challenge is to come up with software that can organize the information. Digital memories can bring benefits in medical treatments, work productivity and other areas, but the developers must ensure that the archives are secure.

About the authors:
Gordon Bell (Bell) and Jim Gemmell (Gemmell) have been working together on the MyLifeBits project in Microsoft's research department since 2001. Bell, one of the pioneers of the computer industry, managed the development of Digital's famous VAX minicomputer in the 70s. In the 80s he became involved in public policy on computer science, and in 1995 he joined Microsoft as a senior researcher in the eSciences research group in San Francisco. Gamel is a senior researcher at Microsoft's Next Media research group. His current research focus is lifetime personal storage, but his interests also include personal media management, telepresence, and reliable transmission of messages to multiple recipients.

More on:
As We May Think. Vannevar Bush in Atlantic Monthly, Vol. 176, no. 1, pages 101-108; July 1945.
A Personal Digital Store. G. Bell in Communications of the ACM, Vol. 44, no. 1, pages 86-91; January 2001.

Digital Memories in an Era of Ubiquitous Computing and Abundant Storage. Mary Czerwinski, Douglas W. Gage, Jim Gemmell, Catherine C. Marshall, Manuel A. Pérez-Quiñonesis, Meredith M. Skeels and Tiziana Catarci in Communications of the ACM, Vol. 49, no. 1, pages 44-50; January 2006.

MyLifeBits: A Personal Database for Everything. Jim Gemmell, Gordon Bell and Roger Lueder in Communications of the ACM, Vol. 49, no. 1, pages 88-95; January 2006.

Information about MyLifeBits can be found at www.mylifebits.com