Persistent Collaboration Tools
When attempts are made to develop "distributed
persistent collaboration" tools, a variety of technical issues stand
in the way. This article examines several of these impediments to progress.
by Eric Armstrong
What "Distributed Collaboration" Means
These days, multiple vendors offer "collaboration tools". Of course,
"distributed collaboration" is implied, because we already have the
meeting rooms and whiteboards we need to work with the guy next door. The available
tools work with varying degrees of success, but what they have in common is
a focus on distributed temporal collaboration. In other words, they attempt
to make it possible to "have a meeting" online.
Such tools are valuable, and useful. But I suspect that eventually, cameras
and projectors will be sufficient to solve that problem. Today, it's possible
to have a video meeting with someone on a distant coast. It works pretty well,
too. The only limitation is the size of the video screen. But with a large enough
screen, folks in another location could appear to occupy the other end of the
table. LCD screen technology is growing to meet that demand, along with those
of the television-viewing audience, so we can expect to see them arrive in the
The two-location meeting is therefore very nearly a solved problem. Producing
a virtual meeting from multiple locations is harder, but it can be seen in television
shows like the MacNeil/Lehr news hour, where each person is on camera, and each
has a monitor for all of the others. But, given a large enough LCD screen, all
that is required is the software necessary to slice and dice it into multiple
virtual monitors, so different portions of the real estate can be allocated
to a single individual, to a several small rooms with a few people in each,
or to a single large room that contains all of the remote participants.
Of course, the issues of sharing presentation materials and whiteboard space
still remain, but these issues, too, are well on their way to being solved:
- Systems exist that recognize what you draw on a whiteboard, saving the image
and it's location in the computer. When the "whiteboard" you draw
on is a clear sheet of plexiglass in front of an LCD screen, then the recorded
images need only be transmitted to the remote sites and displayed on LCD screens
there. A button click can then record a snapshot, after which material can
be erased and redrawn.
- Most slide-presentations are delivered from a computer, these days. The
technology already exists to send slides to remote viewers. The only reason
its not in wider use is that the technology for video-based meetings is still
expensive, and not really satisfying, given the size of LCD monitors and the
problem of sharing a whiteboard. But as those problems are solved, video meetings
will come to be greatly desired. Soon after that, the price will begin to
drop, hastening the acceptance curve still further.
- Even videos are more and more being stored on computers and delivered from
there. The same kind of limited broadcast technology will make it possible
to share video-based presentations. (Basically, a video is just a stored form
of the video stream that records a participant in the meeting. So the same
solution that allocates a portion of the LCD monitor applies here.)
The Goal: Distributed Persistent Collaboration
A problem which is much further away from being solved, however, is that having
a distributed collaboration that persists -- one that continues over
time, and which is reflected in a variety of documents and materials that result
from the collaboration.
In Requirements for a Collaborative Design/Discussion/Decision
System, I described a number of features that are needed to make such a
system work. (Eugene Kim produced a similar list of requirements which focused
more deeply on technical impediments, at __________.)
The goal of this article is to identify the technical problems that stand in
the way of building such systems. The motivation for each entry will also be
described -- that is, the desirable goal or feature that the issue impedes.
Along the way, therefore, you should be able to a decent picture of how a persistent
collaboration tool would work, even if you haven't considered them before.
Many of the issues discussed here are discussed at greater length in the paper,
What's Wrong with Email?. But here, we'll
try to lift the issues up to center stage and make them the focal point of the
conversation. This list follows a general progression from hard, low-level requirements
-- basic functions which are indispensably necessary -- to softer, high-level,
wish-list entries -- desirable features which would affect the pace of development
and adoption, without necessarily preventing it.
- Global Identification
- The most basic requirement of all is the ability to locate and access information
on remote systems. Web hyperlinks give us one way to do that. But they're
long, and they are too intimately tied to the information presentation
layer. We've been using the wrong metaphor -- addressing, which describes
a location, rather than identification, which is a unique tag of some
kind. (Even so-called Universal Resource Identifiers (URIs) have fallen prey
to that problem.)
- The plague of broken links that infests the Web points out the inadequacy
of links that are tightly tied to information presentation. A link consists
of the name of a system, plus a path to a document on it. Now, the name of
the system has a desirable layer of indirection. The name is used to look
up a system identification number on a DNS host (dynamic name server), and
that ID is used to find the system. But paths do not even have that layer
The whole system depends on what names my system call things. As a result,
any of these operations will break the link:
--change the name of the system
--change the name of a directory in the path
--move a directory to another location
--change the name of the file
--move the file to another location
These are syntactical changes to the syntax -- changes to how the information
is presented on my system. A deeper level of linking is needed, so
that when you link to X, at the very least the link remains valid no matter
where X lives on that system. Ideally, the link would remain valid
even if you moved it to another system. That is a harder problem to
solve, but it is necessary for truly global, truly semantic addressing
- Dynamic Links / Active Links
- The display of a Web path provides useful information, however. The names
of directories show how material is organized, while the name of a file suggests
the content it contains. So, having found a link to some document on the web,
it is an easy matter to browse the directories that contain that document.
So the ability to see the path to a document is desirable. But what you see,
and what the computer is linking to, obviously need to be decoupled. What
needs to happen is this:
1. You make a link, using the path you know about.
2. That path is displayed, but the system looks up the *actual* ID and stores
that, as well.
3. The system uses that ID to find the document, rather than the path
4. If the path changes, the path displayed in the document must change, as
(Perhaps the old and new values should be shown, but the very least the "current
address" must be displayed, rather than the old address.)
- Localized Paths
- A path-address to a location in Japan does me no good whatever. It might
as well be a series of numbers. In fact, that would be better -- I could read
the numbers. But if the semantic ID for the information causes the path address
to be displayed, then it would also be possible to view that path through
a "localization filter" that shows me path names in a language I
- Granular Addressing
- As useful as document-level identification is, it is not nearly granular
enough. We need to access information down to the level of paragraphs and
headings, at least. A paragraph is supposed to be a single unit of thought,
so in a sense it is the "atomic" level of document. That makes sentences
and words the subatomic level, and letters the quantum level, I suppose. Addressing
that deeply might well be useful -- or it might introduce inordinate levels
of complexity. But at the very least, uniform paragraph-level addressing is
necessary -- without requiring the author to add a tag for each and every
paragraph in the document.
- If a links remain valid when you move or rename a document, that's a major
step forward. But it still leaves the problem of deleting a document, or editing
it beyond all recognition. A versioning system is required to solve that problem.
Either remote documents must be copied to a local cache, or else the remote
site must be trusted to do the versioning. Either way, it is necessary to
ensure that information your document depends on is still available.
Again, "dynamic links" are needed, so you are informed when the
information has been modified. You then the ability to review the new material,
and the ability to decide whether to keep your link to the old material, or
upgrade to the new.
- Typed Links
- While manual review of versioned material is desirable, it must also be
possible to automate the process. It should be possible to specify "always
update", "never update", or "manual update" for a
Typed links are also necessary for a global ratings system. Rating a web entity
Amazon-style, for example, would mean giving it 1 to 5 stars, and attaching
a note to explain my rating. A google-style server could then prioritize search
results using global ratings, in addition to the number of links. That process
that becomes more difficult in hierarchies, where the sublevel ratings must
add to the ratings of parent entities, and sublevel entities must "share"
parent ratings. But however the semantics of ratings systems are defined,
a "rating link" type is a necessary requirement to make it work.
- Composite Documents
- A composite document is built from multiple segments, using a process that
Ted Nelson calls "transclusion". For example, the shared whiteboard
in the temporary collaboration scenario consists of material written in multiple
locations. Erasing the board in one location must remove material written
in a completely different location.
At first, I thought of document composition as simply requiring a type of
link, which was identified as a transcluding link. But that is an implementation
detail. The more difficult problem is creating facilities for operating on
such documents. It must be possible to edit transcluded material as though
it had been originated locally. In other words, the result of the composition
is a true "document". Whether copies are made locally for editing,
or versions are maintained on the remote system where the material originated,
this problem must be solved.
- Meta Data
- Collaboration is infinitely easier when you can find out what the heck the
other guy is talking about. Even before that, it only becomes possible when
you can find out that the other guy said something relevant in the first place.
Semantic meta data lets you find relevant information, and it also lets you
find related and explanatory material for the results of your search. (Topic
maps, in particular, hold out great promise for this.)
But systematic, wide spread meta data requires meta data standards, the ability
to rapidly find the appropriate tag and easily apply it to your data, and
the ability to searching using meta data.
At its most sophisticated, a search should be capable of finding isopmorphic
patterns of information that have one or two entries in common. If
the structure of an African bee's social system is very similar to the organization
of your file system, that would be worth knowing, wouldn't it? Semantic-level
meta data is a necessary first step.
- Tower of Babel
- Programs I write in Java can't talk to programs written in Perl. C hackers
live on one island, SmallTalk hackers on another. The world needs transducers
so that, no matter what language I coded in, the work products can be used
in other languages. (Then, the lifetime I have spent coding would have a lifetime
of code to show for it, instead of being limited to the most recent applications
on the most recent hardware platform I have been using.)
- Malleable Archives
- In collaboration based on email, an archive of the dialog is maintained.
That is a desirable level of persistence. But the conclusions reached during
the long, drawn-out dialogs remain buried at the bottom of the heap, where
they are inaccessible to all but the most determined archaeologists.
The persistent layer layer -- the "conversation log" Douglas Engelbart's
NLS and Augment systems, needs a more dynamic, malleable layer above it. In
that layer, it must be possible to take the conclusion, promote it to the
highest level of the archive, and subsume under it all of the dialog
that led up to that conclusion.
Such an archive will have the current tendency towards expansion and addition
balanced by a tendency towards reduction and simplification. It must be possible
to introduce new conclusions or new questions at any level of the hierarchy,
where new questions provide a way of framing a dialog, helping to resolve
disputes by identifying the crucial question(s) that lie at their heart (see
The IBIS Manual.)
An interesting aspect of this process is that people both compete and cooperate.
They may compete for "the most accurate conclusion" or for the "question
at the heart of the issue". As part of the competition, the subsumed
material will be referenced by each candidate. People may also cooperate,
using composite documents to refine the presentation of the conclusion or
question. Darwinian selection then applies, with the eventual winner being
hoisted to a position of prominence.
The nature of the "malleable archive", coupled with ratings, is
in essence the process of science. Theories come and go, rising and falling
in popularity. Complexity grows over time, until some simplifying hypothesis
resolves differences and causes material to coalesce. The same simplification
process occurs in the design and construction of computer systems (and, unfortunately,
much less frequently in legal and social systems).
- Authoring Tools
- XHTML has been around for years, now. It's a tremendous improvement over
HTML, in that is well-formed and easy to deal with, with sophisticated capabilities
like transcluding links. But XML authoring tools are still either abysmally
deficient or overwhelmingly expensive. So few original documents are being
produced in XHTML. (The lack of good display components for components to
use -- in particular, tree-structure viewers that are capable of displaying
mult-line entries -- is the major limiting factor in creating authoring tools
using today's technologies.
- Component-level Programming
- The concept of component-level programming is still nascent, after all these
years. So an editor for an a new paradigm like XHTML is still a complete write-from-scratch
project. So it takes many years before new editors come on line. With serious
component-level programming, it is (at least conceptually) possible to throw
together a bunch of existing components and create a solid program in a fortnight.
Sure would be nice.
- Extensible, Integrated Systems
- The world of collaboration systems and knowlege-based systems, like malleable
archives, is marked by shifting concepts and changing needs. The best editor
I ever used was one in which virtually all of the editing operations were
available for programming, so I could create powerful customized routines
to get the work done.
In the world of knowledge-based systems and collaboration systems, the tools
must first be integrated into a single, coherent system. It should
not require one program to view a document, another to author it, and yet
another to act as a server. All of these operations need to exist as a single,
unified whole, so that the act of authoring leads to the act of publishing,
merely by flicking a switch. Similarly, there should be smooth, indistinguishable
continuum of behavior that runs from browsing for information, through commenting
on information and rating it, up to authoring new documents which incorporate
and reference information that has been viewed.
It should be possible to send a feedback message to the author of any page
viewed on the web, without having to hunt for a email address that may or
may not exist somewhere on the site. (It should be possible to attach a rating
to any page or product viewed on the web, and do so in the sure knowledge
that unscrupulous vendors won't shake bad ratings by changing addresses!)
The Minimum Necessary Configuration
A final problem, worthy of a section unto itself, is the fact that the Minimum
Necessary Configuration (MNC) consists of several interacting components, each
of which must exist for the system to be effective. Ideally, the components
should be seamlessly integrated. But whether they are integrated or separated,
all of the components must exist to create a usable system.
In this section, we'll use the internet as our model, examine how it grew,
and contrast it with our ability to develop a usable collaboration system.
How the Web Grew
The "Web" as we know it as actually a composite of 5 different software
systems. If any of them were missing, the web would not exist:
- HTTP protocol
A mechanism for transmitting messages between systems, using available hardware.
- HTML language
A standard for encoding pages to make them viewable as documents, rather than
as simple text.
- Server programs
To deliver HTML pages to users.
- Browser programs
To decipher the HTML page and render its contents for display.
- Production systems
To create the HTML pages.
The Web grew and evolved because it started small, and grew from there. The
first production systems were simple text editors. The original HTML language
contained a basic, minimal set of tag. Browsers were equally simple, and server
programs delivered only static web pages. Originally, there were none of the
animated images or interactive displays that we see today.
But the web only grew to it's size and scope by virtue of the fact that all
of the necessary facets, no matter how minimal, were in place at the outset.
The result was a minimal system in which it was possible to create a simple
web page, and get it delivered to an end user.
As usage grew, each of the components grew and expanded, acquiring more functionality
and getting more complex, until the web as we know it came into existence. But
that expansion would never have happened had not the Minimum Necessary Configuration
existed at the outset.
Can a Collaboration System Grow the Same Way?
The thrust of this article has been, in the main, to identify problems in
the existing web infrastructure that tend to preclude the development of a viable
collaboration system. With those defects in the available infrastructure
remedied, it should be possible to create a minimal collaboration system, and
evolve it over time.
The evidence that irremediably defects exist in the infrastructure is the fact
that we still don't have good engines for remote, persistent collaboration,
despite great effort from many brilliant minds.
But if the infrastructure problems were solved, would it then be possible to
organically evolve major functionality from simple beginnings? It's an open
question, certainly. And it may be in the nature of the desired system that
it is simply not possible.
When we look at the web, we see a nice separation between the authoring process
and the delivery process. While the kinds of things we can author have grown
in scope and complexity, the authoring process itself is still a single-individual,
single-system process. Similarly, the delivery is (in general) to a single-user
on a single system. (Various broadcast models have been attempted, but none
has so far met with stunning success, to my knowledge, a fact which may also
be attributable to infrastructure problems.)
In a collaboration system, however, complexity is immediately and inevitably
part of the process of from the start. Global identification and versioning,
malleable archives and document composition -- all raise the bar of complexity
much higher than was necessary to display a simple web page.
On the other hand (and here is the cause for hope), what now looks like a "simple"
web system undoubtedly looked like major complexity, at the time. It is entirely
possible that even the most complex system we can imagine and build today will
still grow and evolve beyond our wildest imagination.
Maybe future generations will look back on the overwhelmingly complexity of
the systems we are designing today and say, "that was just a simple collaboration
§ Home ·
Health · Software
· Dance ·
Essays · Links
Copyright © 2003
by Eric Armstrong. All rights reserved.
Contact me to send feedback,
register for updates, or make a donation to help support the site.