Thursday, March 26, 2009

OD Archive fails due to "Keychain -25300" error

At my workplace, we have been having problems with Apple Open Directory. [I'm using OS X Server, 10.5.6.] One thing we noticed is that, if you go to Server Admin, and tell it to make an archive or your directory, it will appear to happily do so, but, if you check, you will not end up with an archive; no file will have been created, and, in looking at the "Configuration Log" (/Library/Logs/slapconfig.log), you will see at the end that there was some sort of mysterious keychain error at the end of step 5:

Error in backing up keychain -25300
Removed directory at path /tmp/slapconfig_backup_stage[funky-unique-name].
Searching the web for the keychain error message merely revealed other people who were having the problem, and had not come up with a solution.

At length, I found that Apple has open-sourced a number of their projects, including parts of Open Directory, and that the source is available for download.

When looking for the source for slapconfig (the tool used in creating the Open Directory archive), which does not appear to be available, I came across a posting where someone identified which keychain is missing.

The missing keychain is a System keychain called com.apple.opendirectory. Here is how you re-create it:

Run Keychain Access (it is in /Applications/Utilities).

Click on the "System" Keychain (on the left) and note that the com.apple.opendirectory keychain does not exist. [The picture below shows it after it has been created.]

You will need to add a new keychain item. Click on the "+" symbol at the bottom. [You may have to unlock your keychain first]

I did some testing. It is important that the account name be your server's hostname followed by a dollar sign. If you use something else -- for example, I tried the hostname of the server that used to be our Open Directory Master -- you will get keychain errors when you try to create the archive.

We went to some effort to recover the former password [cool enough, if you have a keychain, you can hit the 'i' button (or press Cmd-I) to see the info on it, and then tick the "Show password" checkbox to see the password.] This was needless; it turns out that the password you use really doesn't matter. I don't think you'd ever need to know it again, and while I don't know what purpose it serves, I'd use a reasonably strong password.

Having created your new key, can you make a backup? Yes, well, ..., erm, well, yes, but, not through the GUI tool. When you use Server Admin now (and I even tried rebooting to see if it would make a difference), things still do not work right. If you click on Open Directory and then on "Archive", it may say "loading information" for a couple of minutes, and, when you finally try to create the archive, it will fail silently, logging:

Error in backing up keychain -25308
a different error, which, I gather, means it couldn't communicate with the keychain. (Sigh)

But, you can now create the backup manually.

Open up a trusty Terminal, and issue a the backup command:

sudo slapconfig -backupdb Desktop/backup

The latter parameter is the path and filename of the backup to create. You will be asked for your admin password [by sudo], and then for a password for the sparseimage archive of your directory [by slapconfig]. It goes ahead and does it thing, even stopping to ask if it can access the keychain, and, you'll notice at the end of step 5 it says,

Backed Up Keycahin

[That is a direct cut-and-paste, by the way. Bugs come in all shapes and sizes.] You now have a successful backup!

Restoring the Backup

I went to a pristine server and tried to restore the backup. In Server Admin, select the server, choose "Open Directory", click on the "Archive" icon, and choose "Restore". It didn't work. The log said, "The directories were not merged because the kerberos realms are different."

Okay, fine. I changed the server from being an Open Directory Master to being a Standalone server. Then, I changed it back to being an Open Directory Master. This time I was sure to enter in the realm information used by our existing master, and to change the base path to correspond. Afterward, I was able to restore, right from the GUI tool.

I tested the restored master with Workgroup Manager and dscl, and su. It looks good. Making an Open Directory Archive from Server Admin worked, too!

Hooray!

Addendum

If the com.apple.opendirectory keychain does not exist, it will be created when you take a standalone server and make it into an Open Directory Master.  

35 comments:

  1. After you recreate the com.apple.opendirectory file, double click it and change the Access Policy settings to allow all applications to use it and you will be able to use Server Admin to make OD backups, not just slapconfig.

    By the way this article really helped me out. Thanks.

    ReplyDelete
  2. thanx, that helps me to!

    ReplyDelete
  3. I still having trouble backing up my OD post trying the above mentioned steps. Anymore thoughts ????

    ReplyDelete
  4. Anonymous,

    I'm afraid I don't have any more ideas on what to do beyond posting a question to, say, www.serverfault.com .

    ReplyDelete
  5. Thanks alot, did get the new error
    Error in backing up keychain -25300

    But i did open the Keychain Object and changed

    Access Control-Check- Allow all applications to access this item.



    And now it works! =)

    ReplyDelete
  6. I´ve reinstalled my server (10.6.3), mainly to see if I could get rid of this error. (Your workaround didn´t make it for me).

    To begin with, everything went well after reinstall - I could make a couple of backups.

    Then I activated the Web-DAV-Digest Authentication Method from Server Admin -> Open Directory. (This is needed for Wiki login).

    After resetting the passwords to make the logins work I tried to archive my OD again, and the bug was back - the archive routine quits at "backing up password server database", and the same old error appears again in the logs.

    It seems like the OD Archive simply can´t manage webDAV passwords.

    ReplyDelete
  7. Same issue, finally resolved. In 10.6.3 I got the same problem as listed above. All efforts to use the above directions failed. After much mind bending, I realized that the server was referring to itself as "blankety-blank.local" in the Server Admin and "blankety-blank.blah.com" in the LDAP reference. Poking into the com.apple.opendirectory keychain showed that it was pointed to the blah.com$ name in Account Name, not the .local$. I changed the Account Name to point to .local$, allowed all programs access, fired it up, and BAM, worked as expected. Yay.

    ReplyDelete
    Replies
    1. Anonymous, you are right on the money. Way smarter than I am anyway. The key chain came up as bla-bla-bla.com$ and worked as soon as I changed it to the .local$ name. Thanks man! I am running 10.6.8 that was upgraded from 10.4 to 10.5. The error I was getting was "error in backing up keychain -25300" when creating the sparse image of the LDAP database.

      Delete
  8. My server already had the keychain entry and was already set to allow all programs.
    I ran hostname and compared entries. In my case the case was wrong in the keychain entry.
    hostname: SERVERone.company.com
    keychain: serverone.company.com
    Once i changed the keychain to SERVERone.company.com everything worked.

    ReplyDelete
  9. This has been incredibly helpful, thank you. In my case, the com.apple.opendirectory keychain existed, but for some reason the account field was empty before fixing it as per above.

    ReplyDelete
  10. I'm having this problem after moving two servers to a new subnet (changeip was run and worked correctly).

    If I look at the "com.apple.opendirectory" system keychain entry, the "Account" line is blank as the above user indicated?

    Is the fix really just to put in $

    ReplyDelete
  11. er, the above didn't come through right.

    Is the fix to put in: my.server.hostname$

    in the now-empty Account section?

    ReplyDelete
  12. Worked like a charm for me also.
    Thank you so much!

    ReplyDelete
  13. Yes, Shriner, I believe that is the correct fix. I'm not sure if you'll need the current hostname or the previous one.

    I'm glad to hear the technique has been helpful to so many people.

    ReplyDelete
  14. Well, it partially worked for me. I put $ in for the account in the keychain and that does let me make the archive now (no more error in step 6 of the process). So that's a good thing!

    But filling that field in didn't do anything to address the pauses I have trying to *make* the backup. Every click in Server Admin --> Open Directory --> Archive (or Settings) takes 2 minutes to respond on one server and 4 minutes to respond on my other!

    I'm hopeful 10.6.5 will do something about this. Everything was fine until I moved my servers to their new subnet apparently...

    ReplyDelete
  15. Shriner,

    have you tried backing up your directory from the command line, as mentioned in the post?

    Also, does Console show anything interesting when problems occur?

    ReplyDelete
  16. Since I got a successful (I think) backup from Server Admin, I've not tried the command-line backup. That's not really a problem any more.

    I fired up both servers showing this problem and waited the 2-4 minutes after clicking "Archive" until things changed from "Loading Information..." to "You can archive..."

    Viewing Console and showing "All Logs".

    One only had a line from com.apple.passwordreset flushing expired sessions (which only happened once -- not a second time) and the other had nothing at all during the pauses.

    I could certainly sample the hung process with Activity Monitor, but that's not really going to help me figure out anything, unfortunately, as I don't know how to read them.

    ReplyDelete
  17. Shriner,

    just want to confirm -- you no longer has a problem, and your system is happy?

    ReplyDelete
  18. I had two problems:

    1) couldn't archive the OD database via Server Admin (which your suggestion fixed)

    2) A 2-4 minute pause every time I click something in Server Admin --> Open Directory --> Archive/Settings.

    You referenced this (sort of) in your original blog post: "If you click on Open Directory and then on "Archive", it may say "loading information" for a couple of minutes"

    I see the "couple of minutes" after every step ("Archive", "Choose", select a directory, click the "Archive" *button*, etc...)


    This problem -- didn't go away with the fix for #1, unfortunately.

    ReplyDelete
  19. Well, *that's* weird. The last time I tried this was on 11/3. I just tried again on 11/8...

    And the pauses are gone.


    No idea why. I didn't even reboot the servers.

    Unless it had something to do with the DST changes -- but that wouldn't have explained why things were working back in August.

    Weird.

    ReplyDelete
  20. Hello, I do not agree with the previous commentator - not so simple

    ReplyDelete
  21. Wow. This has been so helpful. I was caught out by this bug after destroying my open directory only to find my daily backs were not creating any files.
    with error 25308 it still didnt create the archive however after allowing all access to the key chain item. At first it also was not created under the system items it finally worked... next step recovery

    ReplyDelete
  22. Fantastic.. this works for me but only if i change the name to .local$

    ReplyDelete
  23. I've had this same problem with Snow Leopard Server 10.6.8 - everything seemed correct, including the server hostname in the com.apple.opendirectory entry in the System keychain.

    Same -25300 error due to inaccessible keychain.

    Well... having been a professional coder in a previous life I'll just bite my tongue here, but my particular problem (there are several, by the looks of things) was that if you've given your server a *display name* with any upper case letters, then this will be used as the account in the keychain item. Not the actual hostname (i.e. the output from 'hostname' at the CLI), but whatever display name you choose. Mind-boggling idiocy.

    For example, if your server is called 'catfish' and your domain is 'doggerpillar' - 'short names' of course, then a call to 'hostname' at the CLI will give you 'catfish.doggerpillar'. Looking in the directory will get you 'catfish.doggerpillar$' in the short name section... but let's say you want the server name to be displayed as 'Catfish'? OS X Server lets you do it. There's a 'Name', and a 'Short Name'. The short name is the system identifier. The 'Name' is merely a descriptive text... e.g. the Unix user 'root' has the 'Name' of 'System Administrator' - mixed case, spaces and all. You can change it to 'Giant Dog Eating Catfish' but the 'short name' of 'root' is what matters. This should be the case for computer records just as much as user records in a directory, surely???

    But instead, in the above case, the account in the com.apple.opendirectory keychain will contain 'Catfish.doggerpillar$'. And the code that checks this keychain must be case sensitive. AAAARRRRRGGGGHHHHHHHHHH. Who the HELL uses display names in system keys and then makes case-sensitive hostname checks?

    Anyway this may be a bit late now but it fixed my problem (Server Admin created the archive successfully after making the account all lower case in the keychain item) - hopefully this will now turn up on searches for others with similar problems...

    ReplyDelete
  24. Just to say thanks, this information helped me solve my - more or less - identical problem. Although the item com.apple.opendirectory existed in my keychain, I could not backup. But changing accountname from "server.local$" to "server.domain.local$" it all worked again.

    ReplyDelete
  25. You rock. Identical problem here--though I had the keychain, but wrong account name. Thanks for posting this.

    ReplyDelete
  26. WORKS LIKE A CHARM! OS 10.6.7 server

    My account was the local host name serverx.local$ changed to FQDN serverx.domain.net$ and we're archiving once again.

    Thanks!

    ReplyDelete
  27. I am trying to move the db to a new server. Old server is named tahome.thet.net, new server is home.thet.net - Everything above worked as far as allowing me to create the archive. When I try to use restore to move the db to the new server, I get indication that it is working and it asks if I want to merge or replace. I selected merge and it seemed to do its thing but no accounts added to db. Looking at the slapconfig.log shows messages indicating that "the directory archive was not merged because the Kerberos realms are different" It gives options for ways around this but most of them indicate a probable loss of passwords (I am trying to avoid this).

    Would choosing "replace" when asked work? I only have a few test accounts on the new server anyway.

    Thanks for the tips so far. I would still be pulling my hair out over the phantom archives without this.

    ReplyDelete
    Replies
    1. I am afraid I don't know, Glen. You might ask on the Mac Enterprise mailing list, or perhaps Apple has some documentation on moving from one realm to another.

      Good luck.

      Delete
  28. Thanks a lot ! I stumbled on this issue and your post was really a great help :-)

    ReplyDelete
  29. On my OD Master, with Mountain Lion, I had to add "/LDAPv3/127.0.0.1" rather than "com.apple.opendirectory". The access settings listed above were also crucial.

    ReplyDelete
    Replies
    1. Thanks! This helped me while upgrading from 10.6 to 10.10. The install process had created the "/LDAPv3/127.0.0.1" item required, but it somehow filled in the wrong Account Name.

      Delete
  30. This worked for me, the com.apple.opendirectory instruction didn't work for me on mountain lion.

    ReplyDelete
  31. On 10.9 my "/LDAPv3/127.0.0.1" (instead of "com.apple.opendirectory") keychain had an empty Account name. When I changed it to my server name (as it appears in Server.app) and added $ (example: "example.com$"), it worked great. Thank you for this post.

    ReplyDelete