Comments on: Database Design Problem

By: Algorithm Blogs » Blog Archive » Actual Data Always Needs To Be Explicit

Fri, 10 Apr 2009 21:01:13 +0000

[...] encountered this issue when I implemented my multiple source db-design. A reminder: I had data collected from various sources, and then combined together to a final [...]

By: lorg

lorg — Wed, 25 Feb 2009 23:47:08 +0000

@Jay:
Actually, before I started working on the database, I considered using couchdb, and rejected it. I did so because:
1. It didn’t seem “ready enough” for me.
2. I did not find anyone I know who already worked with it (or knew about it, for that matter. I was the one telling them about it!).
3. My hosting doesn’t support it.
4. Relational databases are good enough.
Now, under other circumstances, being the first one “in my neighborhood” to use it could be really fun, but I thought that for my startup it isn’t the right way to go. It is an unnecessary risk, with not much to gain (See point no. 4).
So I decided that for this startup I’d keep to sql, and for my next fun project I would acquaint myself with couchdb.

As for your actual suggestion: I’m not well versed in couchdb usage, but I’m not sure I like it. Since when querying for records I usually don’t want to get the extra baggage of source records/data. I know I might be optimizing prematurely but I think this solution is a bit too heavy. Also, it seems to me this solution would keep source data and final data in different forms, which would make it harder to process.

By: Jay

Jay — Wed, 25 Feb 2009 17:43:17 +0000

Just something to consider, but this type of “non-normal” data structure is something that couchdb would be great at.
You could have a “generic entity” entry and then programmatically check for the existence of other document attributes and expand your behavior if they exist.

By: lorg

lorg — Mon, 23 Feb 2009 07:49:46 +0000

@jack9:
First, thank you for your comment and proposed solution.

Regarding normalization, I just didn’t describe it to clarify of the actual problem and proposed solutions.

Regarding your solution, since I ended up using solution 1**, I had a one-to-many mapping between items and sources, not many-to-many as you suggest. If I understand your idea correctly, you suggest keeping only the combined (final) data, and using a join table to keep track of all sources involved in creating a combined record.
There are two problems with this: (a) You don’t keep the source data, for later analysis. If you do, I don’t see a way to differentiate between source records and final records. (b) For a final record, you can’t seem to be able to tell for a single field what is its source. You can change your solution by adding a field_name to itemfromsource, but I don’t think the resulting solution is pretty elegant.

By: jack9

jack9 — Mon, 23 Feb 2009 03:33:58 +0000

Why not -

source
{
id
type (website or … can be normalized to type_id)
name
… (timestamp or whatever else is source appropriate)
}

item
{
id
name
…(whatever is unique across all sources)
}

itemfromsource
{
id
source_id
item_id
}