408 lines
14 KiB
Plaintext
408 lines
14 KiB
Plaintext
<!doctype article PUBLIC "-//Davenport//DTD DocBook V3.0//EN" []>
|
|
|
|
<!-- SGMLized by Bertrand <Bertrand.Guiheneuf@aful.org> -->
|
|
|
|
<article id="index">
|
|
<artheader>
|
|
<authorgroup>
|
|
<author>
|
|
<firstname>Giao</firstname>
|
|
<surname>Nguyen</surname>
|
|
</author>
|
|
</authorgroup>
|
|
|
|
<title>An in-depth look at the virtual folder mechanism</title>
|
|
<abstract>
|
|
<para>
|
|
This document describes a different way of approaching mail
|
|
organization and how all things are possible in this brave new
|
|
world. This document does not describe physical storage issues
|
|
nor interface issues.
|
|
</para>
|
|
<para>
|
|
Historically mail has been organized into folders. These
|
|
folders usually mapped to a single storage medium. The
|
|
relationship between mail organization and storage medium was
|
|
one to one. There was one mail organization for every storage
|
|
medium. This scheme had its limitations.
|
|
</para>
|
|
<para>
|
|
Efforts at categorizations are only meaningful at the instance that
|
|
one categorized. To find any piece of data, regardless of how well
|
|
it was categorized, required some amount of searching. Therefore, any
|
|
attempts to nullify searching is doomed to fail. It's time to embrace
|
|
searching as a way of life.
|
|
</para>
|
|
<para>
|
|
These are the terms and their definitions. The example rules used are
|
|
based on the syntax for VM (http://www.wonderworks.com/vm/) by Kyle
|
|
Jones whose ideas form the basis for this. I'm only adding the
|
|
existence of summary files to aid in scaling. I currently use VM and
|
|
it's virtual-folder rules for my daily mail purposes. To date, my only
|
|
complaints are speed (it has no caches) and for the unitiated, it's
|
|
not very user-friendly.
|
|
</para>
|
|
<para>
|
|
Comments, questions, rants, etc. should be directed at Giao Nguyen
|
|
(grail@cafebabe.org) who will try to address issues in a timely
|
|
manner.
|
|
</para>
|
|
</abstract>
|
|
</artheader>
|
|
|
|
<!-- Definitions -->
|
|
<sect1 id="definitions">
|
|
<title>Definitions</title>
|
|
<sect2>
|
|
<title>Store</title>
|
|
<para>
|
|
A location where mail can be found. This may be a file (Berkeley
|
|
mbox), directory (MH), IMAP server, POP3 server, Exchange server,
|
|
Lotus Notes server, a stack of Post-Its by your monitor fed through
|
|
some OCR system.
|
|
</para>
|
|
</sect2>
|
|
|
|
<sect2>
|
|
<title>Message</title>
|
|
<para>
|
|
An individual mail message.
|
|
</para>
|
|
</sect2>
|
|
<sect2>
|
|
<title>Vfolder</title>
|
|
<para>
|
|
A group of messages sharing some commonality. This is the result of a
|
|
query. The vfolder maybe contained in a store, but it is not necessary
|
|
that a store holds only one vfolder. There is always an implicit
|
|
vfolder rule which matches all messages. A store contains the vfolder
|
|
which is the result of the query (any). It's short for virtual folder
|
|
or maybe view folder. I dunno.
|
|
</para>
|
|
</sect2>
|
|
<sect2>
|
|
<title>Default-vfolder</title>
|
|
<para>
|
|
The vfolder defined by (any) applied to the store. This is not the
|
|
inbox. The inbox could easily be defined by a query. A default rule
|
|
for the inbox could be (new) but it doesn't have to be. Mine happens
|
|
to be (or (unread) (new)).
|
|
</para>
|
|
</sect2>
|
|
<sect2>
|
|
<title>Folder</title>
|
|
<para>
|
|
The classical mail folder approach: one message organization per
|
|
store.
|
|
</para>
|
|
</sect2>
|
|
<sect2>
|
|
<title>Query</title>
|
|
<para>
|
|
A search for messages. The result of this is a vfolder. There are two
|
|
kinds of queries: named queries and lambda queries. More on this
|
|
later.
|
|
</para>
|
|
</sect2>
|
|
<sect2>
|
|
<title>Summary file </title>
|
|
<para>
|
|
An external file that contains pointers to messages which are matches
|
|
for a named query. In addition to pointers, the summary file should
|
|
also contain signatures of the store for sanity checks. When the term
|
|
"index" is used as a verb, it means to build a summary file for a
|
|
given name-value pair.
|
|
</para>
|
|
</sect2>
|
|
</sect1>
|
|
|
|
<!-- Queries -->
|
|
<sect1>
|
|
<title>Queries</title>
|
|
<para>
|
|
Named queries are analogous to classical mail folders. Because named
|
|
queries maybe reused, summary files are kept as caches to reduce
|
|
the overall cost of viewing a vfolder. Summary files are superior to
|
|
folders in that they allow for the same messages to appear in multiple
|
|
vfolders without message duplications. Duplications of messages
|
|
defeats attempts at tagging a message with additional user information
|
|
like annotations. Named queries will define folders.
|
|
</para>
|
|
<para>
|
|
Lambda queries are similar to named queries except that they have no
|
|
name. These are created on the fly by the user to filter out or
|
|
include certain messages.
|
|
</para>
|
|
<para>
|
|
All queries can be layered on top of each other. A lambda query can be
|
|
layered on a named query and a named query can be layered on a lambda
|
|
query. The possibilities are endless.
|
|
</para>
|
|
<para>
|
|
The layerings can be done as boolean operations (and, or, not). Short
|
|
circuiting should be used.
|
|
</para>
|
|
<para>
|
|
Examples:
|
|
<programlisting>
|
|
(and (author "Giao")
|
|
(unread))
|
|
</programlisting>
|
|
The (unread) query should only be evaluated on the results of (author
|
|
"Giao").
|
|
<programlisting>
|
|
(or (author "Giao")
|
|
(unread))
|
|
</programlisting>
|
|
Both of these queries should be evaluated. Any matches are added to the
|
|
resulting vfolder.
|
|
</para>
|
|
</sect1>
|
|
|
|
<!-- Summary files -->
|
|
<sect1>
|
|
<title>Summary files</title>
|
|
<para>
|
|
Summary files are only meaningful when applied to the context of the
|
|
default-vfolder of a store.
|
|
</para>
|
|
<para>
|
|
Summary files should be generated for queries of the form:
|
|
<programlisting>
|
|
(function "constant value")
|
|
</programlisting>
|
|
Summary files should never be generated for queries of the form:
|
|
<programlisting>
|
|
(function (function1))
|
|
|
|
(and (function "value")
|
|
(another-function "another value"))
|
|
</programlisting>
|
|
Given a query of the form:
|
|
<programlisting>
|
|
(and (function "value")
|
|
(another-function "another value"))
|
|
</programlisting>
|
|
The system should use one summary file for (function "value") and
|
|
another summary file for (another-function "another value"). I will
|
|
call the prior form the "plain form".
|
|
</para>
|
|
<para>
|
|
It should be noted that the signature of the store should be based on
|
|
the assumption that new data may have been added to the store since
|
|
the application generated the summary file. Signatures generated on
|
|
the entirety of the store will most likely be meaningless for things
|
|
like POP/IMAP servers.
|
|
</para>
|
|
</sect1>
|
|
|
|
<!-- Incremental Indexing -->
|
|
<sect1>
|
|
<title>Incremental indexing</title>
|
|
<para>
|
|
When new messages are detected, all known queries should be evaluated
|
|
on the new messages. vfolders should be notified of new messages that
|
|
are positive matches for their queries. The indexes generated by this
|
|
process should be merged into the current indexes for the vfolder.
|
|
</para>
|
|
</sect1>
|
|
|
|
<!-- Can I have multiple stores -->
|
|
<sect1>
|
|
<title>Can I have multiple stores?</title>
|
|
<para>
|
|
I don't see why not. Again, the inbox is a vfolder so you can get a
|
|
unified inbox consisting of all new mail sent to all your stores or
|
|
your can get inboxes for each store or any combination your heart
|
|
desire. You get your cake, eat it, and someone else cleans the dishes!
|
|
</para>
|
|
</sect1>
|
|
|
|
<!-- Why all this? -->
|
|
<sect1>
|
|
<title>Why all this?</title>
|
|
<para>
|
|
Consider the dynamic nature of the following query:
|
|
<programlisting>
|
|
(and (author "Giao")
|
|
(sent-after (today-midnight)))
|
|
</programlisting>
|
|
today-midnight would be a function that is evaluated at run-time to
|
|
calculate the appropriate object.
|
|
</para>
|
|
</sect1>
|
|
|
|
<!-- Scenarios of usage and their solutions -->
|
|
<sect1>
|
|
<title>Scenarios of usage and their solutions</title>
|
|
<sect2>
|
|
<title>Mesage alterations</title>
|
|
<para>
|
|
This is a fuzzy area that should be left to the UI to handle. Messages
|
|
are altered. Read status are altered when a new message is read for
|
|
example. How do we handle this if our query is for unread messages?
|
|
Upon viewing the state would change.
|
|
</para>
|
|
<para>
|
|
One idea is to not evaluate the queries unless we're changing between
|
|
vfolder views. This assumes that one can only view a particular
|
|
vfolder at a time. For multi-vfolder viewing, a message change should
|
|
propagate through the vfolder system. Certain effects (as in our
|
|
example) would not be intuitive.
|
|
</para>
|
|
<para>
|
|
It would not be a clean solution to make special cases but they may be
|
|
necessary where certain defined fields are ignored when they are
|
|
changed. Some combination of the above rules can be used. I don't
|
|
think it's an easy solution.
|
|
</para>
|
|
</sect2>
|
|
<sect2>
|
|
<title>Message inclusion and exclusion</title>
|
|
<para>
|
|
Messages are included and excluded also with queries. The final query
|
|
will have the form of:
|
|
<programlisting>
|
|
(and (author "Giao")
|
|
(criteria value)
|
|
(not (criteria other-value)))
|
|
</programlisting>
|
|
Userland criterias may be a label of some sort. These may be userland
|
|
labels or Message-IDs. What are the performance issues involved in
|
|
this? With short circuiting, it's not a major problem.
|
|
</para>
|
|
<para>
|
|
The criterias and values are determined by the UI. The vfolder
|
|
mechanism isn't concerned with such issues.
|
|
</para>
|
|
<para>
|
|
Messages can be included and excluded at will. The idea is often
|
|
called "arbitrary inclusion/exclusion". This can be done by
|
|
Message-IDs or other fields. It's been noted that Message-IDs are not
|
|
unique.
|
|
</para>
|
|
<para>
|
|
I propose that any given vfolder is allocated an inclusion label and an
|
|
exclusion label. These should be randomly generated. This should be
|
|
part of the vfolder description. It should be noted that the vfolder
|
|
description has not been drafted yet.
|
|
</para>
|
|
<para>
|
|
The result is such that the rules for a given named query is:
|
|
<programlisting>
|
|
(and (user-query)
|
|
(label inclusion-label)
|
|
(not exclusion-label))
|
|
</programlisting>
|
|
</para>
|
|
</sect2>
|
|
<sect2>
|
|
<title>Query scheduling</title>
|
|
<para>
|
|
Consider the following extremely dynamic queries:
|
|
<programlisting>
|
|
A:
|
|
(and (author "Giao")
|
|
(sent-after (today-midnight)))
|
|
|
|
B:
|
|
(and (sent-after (today-midnight))
|
|
(author "Giao"))
|
|
|
|
C:
|
|
(or (author "Giao")
|
|
(sent-after (today-midnight)))
|
|
</programlisting>
|
|
Query A would be significantly faster because (author "Giao") is not
|
|
dynamic. A summary file could be generated for this query. Query B is
|
|
slow and can be optimized if there was a query compiler of some
|
|
sort. Query C demonstrates a query in which there is no good
|
|
optimization which can be applied. These come with a certain amount of
|
|
baggage.
|
|
</para>
|
|
<para>
|
|
It seems then that for boolean 'and' operations, plain forms should be
|
|
moved forward and other queries should be moved such that they are
|
|
evaluated later. I would expect that the majority of queries would be
|
|
of the plain form.
|
|
</para>
|
|
<para>
|
|
First is that the summary file is tied to the query and the store
|
|
where the query originates from. Second, a hashing function for
|
|
strings needs to be calculated for the query so that the query and the
|
|
summary file can be associated. This hashing function could be similar
|
|
to the hashing function described in Rob Pike's "The Practice of
|
|
Programming". (FIXME: Stick page number here)
|
|
</para>
|
|
</sect2>
|
|
<sect2>
|
|
<title>Archives</title>
|
|
<para>
|
|
Many people are concerned that archives won't be preserved, archives
|
|
aren't supported, and many other archive related issues. This is the
|
|
short version.
|
|
</para>
|
|
<para>
|
|
Archives are just that, archives. Archives are stores. Take your
|
|
vfolder, export it to a store. You are done. If you load up the store
|
|
again, then the default-vfolder of that store is the view of the
|
|
vfolder, except the query is different.
|
|
</para>
|
|
<para>
|
|
The point to vfolder is not to do away with classical folder
|
|
representation but to move the queries to the front where it would
|
|
make data management easier for people who don't think in terms of
|
|
files but in terms of queries because ordinary people don't think in
|
|
terms of files.
|
|
</para>
|
|
</sect2>
|
|
</sect1>
|
|
|
|
<!-- Miscellany -->
|
|
<sect1>
|
|
<title>Miscellany</title>
|
|
<sect2>
|
|
<title>Annotations</title>
|
|
<para>
|
|
There should be a scheme to add annotations to messages. Common mail
|
|
user agents have used a tag in the message header to mark messages as
|
|
read/unread for example. Extending on this we have the ability to add
|
|
our own data to a message to add meaning to it. If we have a good
|
|
scheme for doing this, new possibilities are opened.
|
|
</para>
|
|
<sect3>
|
|
<title>Keywords</title>
|
|
<para>
|
|
When sending a message, a message could have certain keywords attached
|
|
to it. While this can be done with the subject line, the subject line
|
|
has a tendency to be munged by other mail applications. One popular
|
|
example is the "[rR]e:" prefix. Using the subject line also breaks the
|
|
"contract" with other mail user agents. Using keywords in another
|
|
field in the message header allows the sender to assist the recipient
|
|
in organizing data automatically. Note that the sender can only
|
|
provide hints as the sender is unlikely to know the organization
|
|
schemes of the recipient.
|
|
</para>
|
|
</sect3>
|
|
</sect2>
|
|
<sect2>
|
|
<title>Scope</title>
|
|
<para>
|
|
Let us assume that we have multiple stores. Does a query work on a
|
|
given store? Or does it work on all stores? Or is it configurable such
|
|
that a query can work on a user-selected list of stores?
|
|
</para>
|
|
</sect2>
|
|
</sect1>
|
|
|
|
<!-- Alternatives to the above -->
|
|
<sect1>
|
|
<title>Alternatives to the above</title>
|
|
<para>
|
|
Jim Meyer (purp@selequa.com) is putting some notes on where
|
|
annotations needs to be located. They'll be located here as well as
|
|
any contributions I may have to them.
|
|
</para>
|
|
</sect1>
|
|
</article>
|