PDA

View Full Version : Merge problems, User ID auto entry, and FileMaker


Geoff Tani
07 May 2005, 06:32 AM
Since my explanation is rather long, here are my questions up front:
1. Is there any way to automatically populate the User ID field with unique IDs?
2. Is there any way to automatically copy the Person ID into to the User ID for every record?
3. After carefully doing a merge, many married people suddenly were listed as married twice: once to a person, and once to "none." Since I was merging newly imported records into existing records, I don't expect that I overwrote any of the IDs (person ID, family ID, parents ID). What am I doing wrong?

On an OS 10.3.9 environment, I have a Reunion 8.06 file with about 1500 people. On a regular basis, I want to be able to export the data out of Reunion, then into FileMaker Pro, then clean it up, then import it back into Reunion. After the import into Reunion, I would merge 1500 imported records into 1500 existing records. Because there are many people in my family file who have identical names, but who are actually different, I decided User ID matches would be the safest way to merge newly imported records. None of the records in my family file have a User ID.

So, the first step was to populate all the records with a User ID. Reunion appears not to have any feature to allow one to auto enter data into a field for a large number of records. (The move field feature doesn't work for ID numbers.) So I did the following:

I exported the following fields out of Reunion:
Person ID
Last Name
First Name
Birth Date
Death Date

I imported the TEXT file into FileMaker, created a new field for User ID, and mass copied the Person ID to the User ID. I then exported the FileMaker data to TEXT. The TEXT file was identical to the one I exported from Reunion, but with the User IDs.

I imported into Reunion. Then I merged the new records into the existing records. In the merge dialog box, I displayed the User ID field, so that I could confirm that the merge direction was correct: new records in the bottom window INTO old records in the top windo.

Everything seemed good after the merge. The records now had User IDs and they retained their original data. However, many people who were married once now suddenly were married twice. Their new spouse was listed as "none." Also, these people would appear as, for example, female in their original marriage, then as male in their "none" marriage. Finally, In the person's "none" marriage, he/she was connected to the wrong parent, a completely unrelated person.

Something got busted in the merge, but I cannot figure out what I did wrong. If there's a way I could automatically enter User IDs within Reunion, that would be the best.

If anyone can help with the above questions, I would be oh so grateful!

fmlyhntr
07 May 2005, 11:10 AM
I always have to *clean up* after a merge with much the same problems as you have, so I'm not sure you've done anything *wrong*--but I am very interested in solutions.

Christina

Eirik StrÝm
08 May 2005, 09:27 AM
3. After carefully doing a merge, many married people suddenly were listed as married twice: once to a person, and once to "none." Since I was merging newly imported records into existing records, I don't expect that I overwrote any of the IDs (person ID, family ID, parents ID). What am I doing wrong?
You certainly are having fun. - Personally I still consider the merging process too dangerous to be left to irresponsible computers, and I should say that someone editing Reunion data in other programs to be imported back into Reunion is doing so at his own risk. But then we must see if we can get you back on your feet.

Reunion has its own sample database (the Kennedys) that may be used as material to be submitted to controlled testing. In such cases we do just that. And if you export the sample file into a TEXT (tab-delimited) file by the same criteria as those used in your backfired FileMaker experience, and import this unedited text file back into Reunion, then you will see exactly the same phenomenon.

I exported the following fields out of Reunion:
Person ID
Last Name
First Name
Birth Date
Death Date
If this is the case, then you forgot a very important element: the sex of the exported persons. Which Reunion will not let you forget. You export a file of persons with no sex information, and imported back into Reunion these persons are stored as person of unknown sex. When you thereupon command Reunion to merge persons of male/female sex with persons of unknown sex, Reunion balks - in a diplomatic way of course.

Also, these people would appear as, for example, female in their original marriage, then as male in their "none" marriage.
You might check again. I should think that you find these persons of unknown/undetermined sex. However, Reunion prefers in most instances to place persons of unknown sex in the left-hand (male) section of a family card, so they do appear as males at first sight.

1. Is there any way to automatically populate the User ID field with unique IDs?
2. Is there any way to automatically copy the Person ID into to the User ID for every record?
As far as I know, not inside Reunion - currently...

- Eirik Str

Eirik StrÝm
08 May 2005, 09:29 AM
I always have to *clean up* after a merge with much the same problems as you have, so I'm not sure you've done anything *wrong*--but I am very interested in solutions.

Well, Mr Tani's case was a bit special, so I expect that any problems you might have are of a different nature. But as you see there are few conclusions to be drawn without quite specific descriptions of the procedures followed and the structures of the files in question.

You might be best served by personally acquiring an ability to analyze the problems involved in merging processes. Giving a crash course is way beyond the scope of the ReunionTalk Forum, but some introductory remarks may be appropriate to put you on the track.

Use a copy of the Reunion sample family file as your test object.

First step, assure yourself that a merge can be performed faultlessly. Open the copy of the samle file and import the original sample file into the copy. Now it has grown from 104 to 208 people. Next you perform a matching operation by comparing everybody to evetybody, identical first name and last name, and check the boxes for birth and death date criteria. When you get the result you perform a concluding merge operation. Study the ensuing family file yourself to spot any errors - but if all went well you will not find any.

Now we need some examples of errors. So into a copy of the sample file you import the original sample file once more, and you perform a new match and merge. This time you match by comparing identical first and last names, but do *not* check the boxes for birth and death dates. On the resulting list of possible matches you perform a merge operation by consecutively pressing the Merge button. You may occasionally help the selector bar in the top panel of the Merge window to move one line down, but *do not* change anything in the bottom panel in the merging process.

When you study the merged version of the sample file copy you may compare the errors that you observe (if our results are identical, check out for instance Joseph Patrick Kennedy (1888-) with parents, and grandson Patrick Joseph Kennedy (1967-) with parents) with the errors you use to encounter when you generally merge persons. Since we knew what produced the errors in our test you may also be able to formulate theories on how your general errors originate. And to show you why one should try to observe production of errors in controlled tests, you might perform still another match and merge operation.

This time we start with another fresh copy of the sample file and import the original sample file. As last time match by comparing names only. When the Merge people window appears displaying names you use a pop-up meny to add a display of birth dates. Then you pass down the upper list of names until you reach "BOUVIER, John Vernou" and select him by clicking once. Now you compare him with the list of names that appear in the bottom part of the window. Remember, this is a controlled test. We know that every person present in the file has a duplicate somewhere in the file. So why does BOUVIER, John Vernou (1865-) appear only once among the names? (I am not going to solve this mystery.)

To finish off with some hints: Some lessons learned from samples like this can be:
- It might be useful to perform several match and merge operations in succession, because you may be technically unable to perform all merging during one merging session.
- It might be useful to vary the criteria for matching when performing several operations, starting with the most detailed criteria.

- Eirik Str

Geoff Tani
29 August 2005, 12:17 PM
My apologies for replying to Eirik Strom so late. Thanks to his advice, I performed the merge again and was successful. I'd advise people with a similar problem to mine to read his reply, do the experiment merge he refers to, then re-read the Reunion 8 Help notes. The Help notes made more sense to me after I had actually done a merge. In particular, Mr. Strom made three points that were very helpful.

1. Always export/import the sex of the person. That alone cleared up 90% of my problems.
2. If you are merging a lot of people (in my case 1500 people), you should do it in stages. It took me 10 merge sessions to correctly merge everyone.
3. Do some finds on your pre and post merge files. The results should be the same, or the differences is find results should be explainable. In my case, I did the following finds in my Family file. Then I made a backup. Then, after I did the merge, I did the same finds. If I got unexplainable differences in the find results, I threw away the post-merge file and started over with the backup.

Num of Spouses > 1
Num of Parent Cards >1
Parent (name) = <empty>
Relationship = <empty>
Sex is male
Sex is female
Sex is unknown
Unlinked People
Marked People
With Relationship
Total Records
User ID = <empty>
Num of Family cards = 1
Num of Family cards = 2
Num of Family cards = 3

If you do not export/import the sex of the people, you will find dramatic differences in the results for the "number of family cards."

The advice I would add is to get User IDs into all records in your Family file. It is much easier to do a merge based on user ID matches.

And finally, I have a comment.

Mr. Strom wrote the following: "I still consider the merging process too dangerous to be left to irresponsible computers, and I should say that someone editing Reunion data in other programs to be imported back into Reunion is doing so at his own risk."

I agree that the user takes responsibility when he does a merge. But I think it is in Reunion's/Leister's interest to take an attitude of _encouraging_ imports and merges. Maybe the company does already (in which case, great!). But my comment is just in case the developers have an attitude like, "Reunion has an import and merge feature, but they're kinda dangerous so don't use them too much." The more Reunion can elegantly receive and merge data from other applications, the more people will keep using Reunion.

For example, there are many things that you can do to the data in FileMaker that you cannot do in Reunion. (Define a calculation field that displays names in a non-western order, "last name - first name," for example) If Reunion can elegantly merge that data, the more that other applications become an extension of Reunion.

As another example, I want to enter Japanese characters into Reunion. Reunion does not currently support Japanese text entry. FileMaker does, so I use FileMaker to quickly enter Japanese names for 1000 records. Then I import into Reunion and merge. And the Japanese displays correctly! Of course, Reunion does not officially support Japanese, and I am doing these imports and merges at my own risk. However, because Reunion can talk (though still in a clunky way) with FileMaker, I can do what I want to do with Reunion. Result: I want to keep using Reunion.

Ideally, I would be importing and merging all the time, because it is a much faster way to fill in records, as well as make my family file bi-lingual. However, because the merge feature is still a hassle, and because I can't control it as much as I would like to (for example, I would like to overwrite existing data in some cases, but the current merge does not allow overwriting), I often think to myself, "maybe I should scrap Reunon and just make my own FileMaker file."

So, if anything, Reunion should make the import and merge features more powerful and easier to use. Especially so for the merge feature. Compatibility, compatibility, compatibility.