Recovering A Corrupt OpenLDAP Database On OSX Server

Comments

44 responses to “Recovering A Corrupt OpenLDAP Database On OSX Server”

1. 

   2009.07.08

   Paul

   This tip saved my ass! Thank you.
2. 

   2009.07.09

   preston.lee

   @Paul
   You’re welcome!
3. 

   2009.07.09

   Terrance

   Followed your instructions, worked like a champ!! Thanks!!
4. 

   2009.08.12

   jan

   Thank you! Weird enough the configuration tester already fixed it for me!
5. 

   2009.08.20

   Bob

   Add me to the grateful throngs. The LDAP db on my Leopard Server was corrupted after a restart that didn’t. After a panicked morning, I found your post. Worked like a champ!

   (Does 4 count as a throng?)
6. 

   2009.09.08

   Craig

   This worked great but it was corrupted again this morning after a restart, any reason why this would keep happening? Thanks!
7. 

   2009.09.08

   preston.lee

   @Craig
   I haven’t seen that happen before. The only time this has happened to me is when I’ve had to hard reset the machine and/or manually kill the daemons: in other words not shutting stuff down properly. My best guess is that there may be something external to LDAP that is corrupting the files on disk, or perhaps a disk issue itself.
8. 

   2009.09.08

   Craig

   Hi,
   Thanks for your quick response, and actually we are having to force shutdown the machine as it is getting stuck on shutdown. Thanks so much for the insight.

   @preston.lee
9. 

   2009.09.08

   preston.lee

   @Craig
   No problem. If you’re consistently needing to force shutdown, try running `sync` at the command line before hitting the power button. That should force flush the disks write buffers. Hopefully that’ll at least keep LDAP from getting corrupted.
10. 

    2009.09.21

    Marco Papa

    Preston, another great thanks from me. Anyway, my situation was identical to yours. Same identical messages, and corruption generated by hitting the power button.

    I did the db_recover and it worked. Only thing, Kerberos is still listed as “Stopped”. Any ideas why? Everything seems to work.
11. 

    2009.09.22

    preston.lee

    @Marco Papa
    If you’ve already tried restarting the “Open Directory” service a couple times, I’m not sure. Is there anything in the system logs that seems relevant? Could it be a DNS problem?
12. 

    2009.09.29

    Adam

    Thank you!
13. 

    2009.11.10

    Mel

    Preston, yet another big vote of thanks for this – you saved our bacon big time!
14. 

    2009.11.11

    Adam

    MAN YOU SAVED OUR BACON!! Thanks a million we owe you a PINT. 🙂
15. 

    2009.12.01

    Quentin

    Great post & instructions; like the others, it got me out of a jam after an update hung the restart and left me with a corrupt db.
16. 

    2009.12.07

    Ranj

    Thanks a lot this assisted me with a similar problem I was having.

    I had to run a few more commands in terminal to get it working though

    1) sudo to root

    sudo -i
    2) shutdown the open directory server

    service org.openldap.slapd stop

    3) dump a copy of the Open Directory database to an LDIF format text file

    mkdir /var/root/opendirectory
    cd /var/root/opendirectory
    slapcat -l dir.ldif
    4) move the old (corrupt) database files out of the way (or remove them).

    cd /var/db/openldap/openldap-data
    mkdir SAVE
    mv *.bdb SAVE/

    be sure you don’t move, rename or delete the file named DB_CONFIG. It’s needed.

    5) recreate the database from the LDIF format file

    cd /var/root/opendirectory
    slapadd -l dir.ldif
    slapindex
    You will see some harmless warnings during slapadd. Ignore them.

    6) restart open directory

    service org.openldap.slapd start
17. 

    2009.12.08

    Jon Zgoda

    Thank you! Thank you! Thank you!!!

    I thought I was being good, doing a backup through ServerAdmin before any updates…but then I wasn’t able to restore through ServerAdmin when this problem occurred.

    Your solution worked perfectly, and was back up in minutes.
18. 

    2009.12.16

    john lewis

    Ranj’s instructions worked for me. THANKS GUYS! Phew!
19. 

    2009.12.23

    Moshik

    Ranj 10x Alot…!!! saved us also…
20. 

    2010.03.11

    Daniel

    Thank you.

    This has happened to to me three times.
    Two times I rebuilt from scratch.

    This time – thank you!
21. 

    2010.05.14

    Junjun

    This works! Won’t know db_recovery is the right tool without reading your post.

    On my Xserve, I have to use this to recover the database.

    sudo db_recover -cev -h /var/db/openldap/openldap-data/
22. 

    2010.05.14

    MacDave

    Worked great for me — thanks!
23. 

    2010.06.29

    BradDS

    I may try this seeing all the positive responses and the fact that I am currently in the same boat, but did you lose all of your LDAP setting or did they stay intact, because I see you said you carefully backed up the openldap directory but does this require you to replace it? Or does it fix the existing LDAP
24. 

    2010.06.29

    admin

    For *me* it fixed my existing database.
25. 

    2010.06.29

    BradDS

    That would be wonderful. One last question if I may. You also mentioned you had to restart services such as AFP after. What was the reason? Did it prevent the LDAP from fully starting or running that recovery caused those services to stop?
26. 

    2010.06.29

    BradDS

    Chalk up another Success Story! Thanks!
27. 

    2010.07.25

    BigClay

    Thanks for the post it saved my ‘bacon’ as well. Now on to make a better back up approach for the OS X server!!!
28. 

    2010.08.17

    Gill

    KARMA is a boomerang and you have a lot of good KARMA coming your way my friend. This fix is fantastic! and a life saver for all who lose theri ldap data!!

    TY TY TY !!
29. 

    2010.08.17

    admin

    No problem and I certainly hope so! 🙂
30. 

    2010.08.18

    Brian Jønch

    Followed your instructions, worked like a champ, Thanks alot.

    /BJ
31. 

    2010.08.23

    Eric

    You are the man!!! You saved me from a LOT of work, time, frustration, profanity, ulcers, high blood pressure, etc.
32. 

    2010.08.25

    Geoff Smyth

    Elevated to legend status you are.
33. 

    2010.10.05

    Michael

    Just about to poo my pants until I came across this article. Many thanks Guys!
34. 

    2011.01.14

    Kenny

    LDAP Server up and running again.
    Thanks a lot, this did it, great !!!!!
35. 

    2011.10.04

    Steve

    Preston, thanks for the article. I’m experiencing a similar problem, though OD reports that LDAP, Password Server and Kerberos are all running. But in my LDAP log I get the similar messages as you:

    Oct 4 11:07:36 s1 slapd[26448]: bdb(cn=accesslog): DB_ENV->lock_id interface requires an environment configured for the locking subsystem
    Oct 4 11:07:36: — last message repeated 3 times —
    Oct 4 11:07:36 s1 slapd[26448]: findbase failed! 80

    When I tested the config, I get this:

    s1:~ sadmin$ sudo /usr/libexec/slapd -Tt
    bdb_monitor_db_open: monitoring disabled; configure monitor database to enable
    bdb_db_open: database “cn=accesslog”: unclean shutdown detected; attempting recovery.
    bdb_db_open: database “cn=accesslog”: recovery skipped in read-only mode. Run manual recovery if errors are encountered.
    bdb_db_open: database “cn=accesslog”: alock_recover failed
    bdb_db_open: could not restore bdb backend -1config file testing succeeded
    bdb_db_close: database “cn=accesslog”: alock_close failed

    Do you think I should follow your steps above in using the db_recovery tool, or is that not applicable to my situation? Authentication and all services are working on the server, but I’m afraid this problem could manifest itself in ugly ways if I let it continue.
36. 

    2011.10.04

    Steve

    After some other troubleshooting I tried the db_recover tool, and rather than recovering the database, I received these messages:

    Oct 4 11:46:18 s1 slapd[1116]: bdb(dc=*****,dc=*****): PANIC: fatal region error detected; run recovery
    Oct 4 11:46:18 s1 slapd[1116]: SASL [conn=29] Failure: no user in database _ldap_replicator

    I’ve rebooted the machine, manually unloaded & loaded slapd, and ran the recovery tool multiple times, and nothing has worked. Any other ideas? Getting pretty nervous now.
37. 

    2011.10.05

    Steve

    Not sure if anyone is checking this thread anymore, but just in case someone with the same problem reads it, here was my solution. None of the commands above worked in my situation – I suppose the corruption was too bad for the db_recover tool (or the manual .ldif steps), so I took one of my healthy OD Replicas and promoted it to Master. I then destroyed OD on the the previous Master, rebooted and made it a Replica of the new Master. This resulted in a fresh, clean database on the old Master, as well as not losing any records or passwords on the new one. Took me hours of troubleshooting and trial and error tests, but I should have done this as my very first step. There was minimal downtime and to most users the transition was transparent.
38. 

    2011.10.18

    leolor

    has exactly the same problem on leopard 10.5.8 server. thanks. it works again for me!!!!!!!
39. 

    2011.12.22

    MV

    Seriously, put up a Paypal link since you saved me a few hours of research. Or a mailing address.
40. 2012.02.12

    linux-blog – Fa. anracon – Dr. Mönchmeyer » Blog Archive » OX5 – LDAP Restaurierung

    […] http://www.prestonlee.com/2009/07/08/recovering-a-corrupt-openldap-database-on-osx-server/ http://serverfault.com/questions/87889/ldap-database-recovery-after-server-crash […]
41. 

    2012.09.06

    Sanjiv Singh

    Thanks man !!
42. 

    2013.08.18

    chris

    Just saved my Sunday. thanks!
43. 

    2013.10.07

    TD

    Thank you, problem I encountered was identical.
44. 

    2015.07.28

    Bryan

    Thank you for the help. It is 2015 but I am still running 10.6.8 on my server and this solved the problem.