torsdag den 2. april 2009

The characteristics of data persistence

Data is structured information. Persistence is steadyness: A persistent state is a steady state; this is particularly true when the word is used in the context of data. Here, I will be addressing persistence with regard to a particular type of data: Digital Text.

So...Persistent data is data you expect to stay the same.

Well duh, you say. Don't. Let me go over a concrete example of how the implementation of persistence may vary:

Programming languages deal with persistence in different ways; the most elegant way is, without a doubt, the functional programming languages, such as Scheme and F#. Functional languages do not accept mutation of data; that is, once data has been entered, it's there, forever. You may, at some point, lose the ability to address it; but you'll never be able to change the data at the end of an address.

This may seem like a bigger burden at first than it really is: Functional programming languages also allow you to create the (seemingly) same adress multiple times, so rather than changing the data at the address FOO, you can create a new address FOO and put the changed data there instead.

So even if you do not allow data at various addresses to mutate, you do not necessitate creativety in coming up with new meaningful address names; you can just reuse old ones.
This does make it more tricky to get at the old one (you need to get at it by writing something similar to "FOO; THE OLD ONE" in the address field), and it does make for confusing programs when the address FOO does not always refer to the same data (remember, all references to the old FOO within your program still point to the old data, because that's where they pointed before you introduced the new FOO).

The point is, however, that it's perfectly plausible to never erase anything, and to never change anything that anybody else has been working on, in the context of any advanced system. Contrast this to java, a complex programming language for modeling complex systems.

Here objects are passed by reference; that is, if a Salesman object passes a Car object reference to a Customer object, and the Customer object then makes a change (mutation) in the Car objects Wheeltype attribute, that change will also be apparent to the Salesman if he still has a reference to the Car object.

This behaviour can be destructive if the Salesman object was counting on the contents of the Wheeltype attribute to stay the same in the car object: If persistence was somehow important, then in this case, it would have been broken.

Alright, now that I've given some concrete examples of persistence, so you hopefully have a grasp of why it matters (at all), let's look at the how and why of the whole thing:

There are two defining questions to data persistence:

- Why persist?
- How do we persist?

The answer to the first one is: Because you might want to use it later.
The answer to the second one is: By making sure that we are able to address the data at the point where we want to use it.

Understand, then, that the relationship between not persisting and being able to address data is crucial: If you have 100 data articles, and you choose to only persist 50 of them, then you only have to keep track of 50 data addresses. Perhaps this seems like it's not that different from keeping track of 100 data addresses, but try to consider the receipts you get after purchasing groceries in the place of data articles. If you save 100 of those rather than the 50 most important ones, when the time comes to dig out the receipt for your new television set, you'll have to rummage through twice as many articles before coming upon it.

Not persisting the grocery receipts - or the data addresses - makes it quicker and easier to discern the useful ones from the useless ones; it makes it quicker and easier to address that which you are most likely to want to address, and as a concequence, it makes your collection of data more powerful and user friendly.

This is all well and good in a single user environment, but in a multi user environment, things tend to change. Whats important to one person could seem useless to another, and suddenly an important article is missing from the collection.

The answer is, of course, that deletion of any kind is a very primitive type of ordering in a persistent data set. It's actually hiding an article very very well; rarely does i actually phase permanently out of existence, it just becomes so hard to get at that it's no longer worth it to retrieve it. It probably only has such appeal to humans because we can safely forget about things we delete, just like things we throw out.

The same could be accomplished by using a less effective hide operation than deletion - instead of throwing the grocery receipts out, you could toss them all in an old shoebox, so that they're there at least, even if hidden behind a decidedly user-hostile interface. But even the shoebox will repressent a useless article to some, though, so it might get deleted eventually...unless it's impossible to delete, as it would be if it were an object in a functional programming language.

So now that we understand that the goal of persistence is the ability to use later, and that deletion is a type of ordering that simply makes us more _likely_ to use a collection later at all (because it becomes more powerful), we can toss the idea of deletion out the window for good...at least when it comes to small data.

Simply put, there are other types of maintenence that yeilds better results than deletion, often in much less time because there's far less finality to it (ie. it makes the collection more powerful, faster), so as long as size is not a factor, there should be no deletion. Since we're dealing with digital text in this article, it's verifiable that size is, indeed, not a factor.

This makes for a conclusion to this article: Everything should be persisted. Everything that is likely to be useful should be systematically ordered. And finally, efficient algorithms for ordering and retrieval are essentiel to maintain the power of a collection.

søndag den 29. marts 2009

Status on generic storyline editor, simplified:

Persistence:
For version 0.1, will be using user-activated object serialisation.
For version 1.0, will be using atomic timestamped patches that are pervasively stored, backed up, and synchronized when possible.

Platform dependence
All versions will likely be platform independent, both with regards to OS and processor. Python and clutter will be utilized for the bulk of the work.

Programming Language dependence
Python will be utilized for main development, allowing for mid-level portability.
For version 0.1, this is not a concern.
For version 1.0, XML export must be implemented and fully supported.

Network dependence
This will not be necessary for any versions. Version 0.1 will not support it.
For version 1.0, it will be necessary for pervasive synchrony.

Command response delay
No delay is acceptable in any versions.

User Interface
0.1: Basic experience with a basic font. Functional auto-placement and folding algorithm.
1.0: Pixel-precise user-customizable fonts, dynamic and fluid interface experience, with panning, zooming and scaling all incorporated, and dynamic folding and placement of everything. Placement structures and colour schemes incorporated to allow writers to use their visual memory.

Backwards and forwards compatibility
0.1: No focus is given here for this release.
1.0: Fully forwards and backwards compatible, as well as allowing for different branches when necessary. A simple tool for merging, splitting, and maintaining synchrony across versions and branches.

Programming language and graphics API for the generic storyline editor

I've been examining a couple of different programming languages and API's. The first is java and swing; java is the most widely used programming language, and almost always the most efficient one in terms of the number of necessary CPU cycles while maintaining stability.

It is, however, prone to several problems: Java garbage collection leads to annoying "jitter" because it isn't designed for maintaining a smooth user interface. Secondly, java swing looks weird on both windows and nix systems, being visually different from native window rendering.
Finally, java is...well, annoyingly heavy to write.

C# and windows presentation foundation is option number 2...C# is a smaller and less utilized language than Java, but it's only properly supported on microsoft platforms. Windows presentation foundation, while teoretically just what I'm looking for, is currently under-utilized as well. I don't quite dare jump on C# for these reasons...well that, and I don't like being limited to microsofts platform unless I have to be. C# is also a fairly heavy language to write.

Third option: Python and Clutter or Cairo. Clutter is a visual open source API intel is developing that can do all sorts of neat tricks. Python is google's favoured programming language, and they're currently actively developing solutions to it's biggest drawbacks. Cairo is a visual open source api that isn't being shouldered by any specific company, but it's big enough that the mozilla foundations gecko rendering engine, used in firefox, relies on it.
There are bindings from python to both api's, meaning that they should function with it; python is, itself, a language with a fairly large user base (bigger than C# and small than java), and it's dead simple to write and use. Very fast. And Clutter and Cairo look excellent; not necessarily like something that runs natively on any one system, but then, it's so fancy that it looks more next gen than simply different.

I'm hoping to use python and clutter, and if that doesn't work out, python and cairo.

lørdag den 28. marts 2009

Status on the generic storyline editor

My design needs to deal with the following:

Persistence
Syncrhonisation
Platform dependence
Programming Language dependence
Network dependence
Command response delay
User Interface
Backwards and forwards compatibility

Each area poses a unique challenge. Let's go over them one by one:

Persistence is necessary because an editor by definition needs to produce something persistent. But there are several kinds of persistence, and some are more valuable than others. One type is file serailization; this is a snapshot of the current results that is only readable by the current program and perhaps a hex editor. Another is XML-based serialisation; this allows for synchronization, limited interoperability with other programs, and limited readability by humans by means of standard text editors; but it's still limited to snapshots. The third, most advanced, and best option is persisting every change made in the editor along with a vector timestamp, an internet timestamp (if available), a local timestamp, and a user ID, and storing it as an atomic patch.

Synchronisation is necessary if more than one person uses the editor on the same project, or even if that one person works on the project from several locations. Backup is often considered an intrinsic part of syncrhonisation, because if you have access to synchronisation, you have access to backup.
There's a drastic difference between single-user and multi-user synchronisation; it's generally acceptable to use manual synchronisation between workstations, as well as manual backup, when there's only one user; further, it's also generally accepted that if that one user were to make different changes to his project at two different places without synchronising, all the changes made in one of the places would be lost upon synchronization: this limitation is accepted because a single user can generally "work around" this situation such that it never happens.
But for multi user synchronisation, it's generally necessary to allow simultaneous work in several places at once without having them manually reenter all the changes made in all but one place, in order to synchronize them.
Ideally, multiuser synchronization is either synchronous, or asynchronous but automatically run at set intervals and/or on demand, and when it occours, it usually does all, or almost all of the work without user intervention for solving conflicts.
Typically, synchronization also provides sync reports that go into detail on what was changed with each sync, so as to keep all users up to date. Synchronous synchronization is easy to implement, because everything is sync'ed all the time, but usually lags things down because all user clients need to be constantly chattering about whats happening all over the place, and demands solid network stability to the point of being always-on. Asynchronous sync is much harder to implement, but poses no strict network demands; instead, users must provide ID's, abd must help solve conflicts, but in return they're likely to get a snappy and free user experience.

Platform dependence is really OS dependence these days; all work done is done on PC's, so the only question is if the editor is able to run on all platforms natively, or if it has to happen on virtual machines. Platform independence is generally considered the more future-proof alternative, but it can pose some strict interface limitations because platform independent interface libraries generally look odd on most platforms.

Programming language dependence is closely associated to platform dependence; choosing a language will generally lock you in to using just that one language or having you incur rather tremendous costs, but there are different degrees of lockin. Python, for example, is often the second or third language that gets bindings to access libraries of another language, meaning you generally have several languages to fall back on. Some languages are also able to run on several different virtual machines, or interpreters, which can be a boon for future-proofing. Finally, some languages are exceedingly interoperable, allowing you to abandon them when a better one arises by means of automated translation.

Network dependence is just what it says; will the program need net access to function or not?

Command response delay is really a measure of how snappy the program is. If ever a user has to wait for the program, he becomes a _lot_ less productive; a short delay generally translates to a break in workflow, which makes a 10 second pause translate into several minutes of lost work. Some technologies _cost_ on this front; so it's one of those things that isn't an issue when you start out, but if you make the wrong tradeoff somewhere along the road, suddenly you have a useless program, or one with major drawback.

User Interface is just what it says; there are good user interfaces, there are clunky ones, there are bad ones, and there are ones that only a programmer can use, usually by command line. What type of user interface you need depend upon the target usergroup, and your ambitions with your application. The target audience of this program is fairly broad, since it simply means to get at creative individuals with no other qualifications, and they work better if they have a better interface. The ambitions are also high, of course, so that doesn't improve things. This makes the interface bit quite important.

Backwards and forwards compatibility is a matter of allowing the users to get at the program early while not hanging the sword of damocles over their head: If users are scared that future versions of the program won't be backwards compatible, they won't be inclined to use it.

So, now that we have some design parameters up, let me follow through with what I've been thinking, and how I've planned to tackle these issues so far...:

I've allowed the idea for a generic storyline editor to simmer for a while now, since about december 08 when I last did serious work on it, and have now come back to it.

And when I write idea, I mean plan. Traditionally, it's considered counterproductive to plan too far ahead concerning software development; rather, rapid prototyping and modular approaches tend to make for a far smoother creative process with far less grief over wasted effort.

It could be considered a mistake, then, that I've allowed my idea to linger for as long as I have. Thing is, allowing time to pass tends to reveal obvious design errors. One of which I feel I was close to making.

There is a reason, for example, that everybody hates the object-relational mismatch: The best way to store data is in databases, the best way to program is with objects. It's not just that these two methodologies are somewhat incompatible; it's that in spite of this incompatibility, using the two in conjunction is better than the alternatives for a wide range of software products.

Since a generic editor is by definition going to be a program with general utility, and since the projects meant to be edited by it are huge and may live to see several versions of it, such an editor closely mimics the utility pattern of enterprise software, and here's the point: It's precisely this type of software that usually needs both relational data storage and object based algorithms. Not because there aren't alternatives, but rather because the alternatives have been found wanting by the people making the decisions.

Fortunately, since the information needs to be available as objects in the application, I can start there, and simply be mindful that I only use plain serialization to disk if I need to persist anything at first. This means that the persistant data is, initially, bound to the programming language of the editor, but that's a small price to pay for utility before I have figured out how to write a database management module.

So, that means: persistence is going to be simple serialisation of objects for now, database stuff later, hopefully atomic patch-based.

Synchronisation will be solved by means of the database, again, later.

This means that, later, networking will be necessary for sync, but for nothing else.

This also means that there will be low command delay, since all the information will be available in local objects, always.

I've not made any final decisions on most of the other things, so that's the status for now.

torsdag den 5. februar 2009

Story Ownership continued

The reason I went into an exploration of this subject is because of a story a friend of mine, Jonas, related to me about how he felt when he explored the multi-layered game Braid.

Built into the game is an explanation of the lowest, most subtle layers - the game, being non linear as games are, allows the player to seek out this explanation or leave it alone at his leasure. But it still serves not just as an external message from the author, but as a part of the game itself, that insists on being the "right" manner of understanding the subtleties.

Any other interpretation has the unfortunate concequence of making the experience a lot worse, so it really gains authority in this fashion.

But what happens when the player disagrees with the "correct" interpretation of the subtleties? What happens if he hates it, as Jonas did?

There's a breach - it's like playing a game of pen and paper dungeons and dragons and the dungeon master suddenly decides to destroy the story for one of the players by denying him the ability to participate. It's an implicit rule that the dungeon master will not do that. It's self governed - so there's noone there to punish the dm if he does it - but it remains a rule.

So how does this rule work with the idea of story ownership? If the author truly owned the story as if it were his property, then how can there be room for rules which inhibit what he can do with it?

The answer must be that - well, at least if my understanding holds up - story ownership is shared between an author and a reciever, and the fact that the reciever traditionally has had few tools for defending his part of the property is unrelated to the sensations and feelings involved.

In other words, the ethics of story ownership take their basis in culture, rather than in law; it is culture that dictates that the DM doesn't destroy a story mid-game, and it is culture that dictates that Jonas felt his anger was righteous, and that something had been stolen from him by the author upon discovering the deepest depths of the game.

Now I'm not suggesting any changes to ownership laws in my writings on this subject - rather, I would just like to explore the ethics of what a author can and cannot reaonsably do without infringing on the story ownership his audience feels, at least in our current cultural climate. I hope to return to this subject at some later date, again.

onsdag den 4. februar 2009

Story Ownership

Who owns a story?

It's a simple question with a complex answer; there's certainly one naive answer that immediately comes up - the one who created it.

Which makes a great deal of sense when the author, and the person who performs the story, are one and the same; which is really the way the first stories were related, so in a historical context it'll often be true.

It makes less sense, though, when the performance of the story is seperate from the author of the story - does a dead author still own a story, for example?, Because that butchers the idea of ownership pretty badly; once you're dead, you don't own anything anymore. Certainly the ownership is not generally inherited - the children of JRR Tolkien couldn't simply add an additional book in Lord of the Rings series if they saw fit. Legally they could, but it would not be percieved as if it had been written by JRR - so at the very least, the exact type of ownership JRR possessed is not inherited. Perhaps a different type of ownership is what the children recieve, but that's really besides the point!

What I'm getting at is not that ownership of a story depends on the proximity of the author and the performance; it's rather that the question has an intuitively trivial answer given some very particular, but fairly common, circumstances. That is, it's trivial if a story is performed and made by one and the same person.

But if it is written, such that the story is detached from the author, it almost gains a life of its own; it sometimes lives beyond the mortal life of the author, at which point ownership becomes difficult to define. It becomes apparent that there are several kinds of ownership (of which I have aluded to at least 3 here), and that these depend upon a lot of different circumstances.

I hope to touch on this subject again later, because it's worth exploring in more detail.

torsdag den 15. januar 2009

Snake Webcams

Happy newyears! And now, snake cames!

(Being a mobile computing and hardware entusiast in addition to being a computer game theory geek means I will sometimes make blog posts such as these)

:

I've been on the lookout for a good webcam with a bendy mount for recording lectures and perhaps webcasts and notes, in an effort to improve my oral presentation skills. Requirements were integrated microphone, atleast 30fps 640x480 video, and a bendy usb mount.

My research turned up these models:

http://www.huehd.com/index-2.html
http://www0.shopping.com/xPO-Macally-IceCam
http://www0.shopping.com/xPO-Offspring-Technologies-CAM30
http://www0.shopping.com/xPO-MSI-StarCam-Mini
http://www0.shopping.com/xPO-Philips-PHILIPS-SPC610NC-USB-WEBCAM-FOR-PC-AND-LAPTOP
http://www0.shopping.com/xPO-Flex-CP-MPC-01
http://www0.shopping.com/xPO-Notebook-USB-Web-Cam
http://www0.shopping.com/xPO-A4tech-A4Tech-PK-7ma-Flexi-Webcam-Microphone-for-WebCam-Built-in-Microphone
http://www0.shopping.com/xPO-MSI-Starcam-370i
http://www0.shopping.com/xPO-Mustek-Mustek-Web-Cam-M6323
http://www0.shopping.com/xPO-Typhoon-Toto-TEL5GGC-60-EcoPower-Electronic-Faucet-Gooseneck-Thermal-Mixing-Chrome
http://www0.shopping.com/xPO-ClearOne-Communications-FlexCam-iCam-Digital
http://www0.shopping.com/xPO-AOC-PCCDC1120
http://www0.shopping.com/xPO-Solidtek-NoteCam-B100
http://www0.shopping.com/xPO-IC-Gear-Mini
http://www0.shopping.com/xPO-Compucessory-CCS42122
http://www0.shopping.com/xPO-BYTECC-350K
http://www0.shopping.com/xPO-Micro-Innovations-CPQ54CAM
http://www0.shopping.com/xPO-Logitech-430ML
http://www0.shopping.com/xPO-Logitech-Quick-Cam-530-Plus
http://www0.shopping.com/xPO-Micro-Innovations-IC70C
http://www0.shopping.com/xPO-SIIG-CE-UMCP12
http://www0.shopping.com/xPO-Sakar-Mini-WebCam-69352
http://www0.shopping.com/xPO-Iogear-GCA210U
http://www0.shopping.com/xPO-Canon-VIZCAM-1000
http://www0.shopping.com/xPO-Ezonics-CobraCam
http://www0.shopping.com/xPO-Sony-CCD-PC1

Now most of these are probably quite crappy, and don't even meet my demands, so proceed at your own risk, and sorry for not sorting it more thoroughly...but the above list should be current as of january 2009 with regards to everything available on the market, used or new. So it's a place to start looking for a solution that meets your needs.

Your best bet for getting a really good solution may be this though:

http://www.umpcportal.com/2009/01/digitus-flexi-webcam-ordered

It wasn't available and I've already purchased an excellent alternative from the list above, the Clique Hue HD:

http://www.amazon.com/Clique-HD-PC-Mac-Webcam/dp/B000TTIP8Q

But not at that abhorent price tag though, I purchased mine straight from Hong Kong on ebay for 17$ and 10$ shipping, brand new too. Only problem was getting the drivers from a reliable source.

Clique appears to have gone belly up, at least with regards to driver downloads! Which, if you do ebay it, kindof sucks because even if it does come with a driver cd, are you going to trust your computer to a cd that a random stranger sold on ebay?

Best drivers I have found for it are available here:
http://www.kentd.com/index.php?option=com_content&task=view&id=205&Itemid=1

Or they were at the time of writing. If the link is down, email me, and I'll try to upload them!

Exact size for the zip should be:
11.517.399 bytes.
At the time of writing Web Of Trust trusted the KetnD site. Not sure if that's worth much, but hey.

And neither avast nor any of the 4 firewalls I'm behind have complained about trojans or vira, so it appears completely legit. I don't recall if the drivers were digitally signed though, and they do install a server process for configuration, so use at your own risk.

I hope this proves useful to someone at some point =]